vhost-user.rst: Explicitly allow front-end to write to kick FDs
PATCH 1/1 vhost-user.rst: Explicitly allow front-end to write to kick FDs
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm). Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
--- docs/interop/vhost-user.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
docs/interop/vhost-user.rst +4 -1
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection to a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value -as on the source side, so that operation can resume). +as on the source side, so that operation can resume). The vhost-user front-end +may also write to the kick FDs of vrings containing unused buffers or send +``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination +since the driver likely already kicked them in the source and won't do it again. Both on the source and on the destination side, after the respective front-end has seen all data transferred (when the transfer FD has been
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). While
It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."
But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".
Sending a kick from the frontend, seems more an hack here.
That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.
Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."
So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".
Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit 110b9463d5c820120c8311db79f55a64c9d81ebe
Author: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800
vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.
Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
Thanks,
Stefano
Quoted 25 lines
Signed-off-by Jorge E. Moreira <[email protected]>---
docs/interop/vhost-user.rst| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)docs/interop/vhost-user.rst +0
-0@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connectionto a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value
docs/interop/vhost-user.rst +4
-1-as on the source side, so that operation can resume). +as on the source side, so that operation can resume). The vhost-user front-end +may also write to the kick FDs of vrings containing unused buffers or send +``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination +since the driver likely already kicked them in the source and won't do it again.Both on the source and on the destination side, after the respective front-end has seen all data transferred (when the transfer FD has been
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Quoted 110 lines · Stefano Garzarella
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit
7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
Thanks,
StefanoQuoted 6 lines
Signed-off-by Jorge E. Moreira <[email protected]>---
docs/interop/vhost-user.rst| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines
docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662
Quoted 13 lines
docs/interop/vhost-user.rst +0
-0@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connectionto a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value
docs/interop/vhost-user.rst +4
-1-as on the source side, so that operation can resume). +as on the source side, so that operation can resume). The vhost-user front-end +may also write to the kick FDs of vrings containing unused buffers or send +``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination +since the driver likely already kicked them in the source and won't do it again.
After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).
So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)
WDYT?
Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.
Thanks,
Stefano
Quoted 137 lines · Stefano Garzarella
Quoted 110 lines · Stefano Garzarella
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit
7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
Thanks,
StefanoQuoted 6 lines
Signed-off-by Jorge E. Moreira <[email protected]>---
docs/interop/vhost-user.rst| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)Quoted 13 lines
docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662
Quoted 13 lines
docs/interop/vhost-user.rst +0
-0@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connectionto a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value
docs/interop/vhost-user.rst +4
-1-as on the source side, so that operation can resume). +as on the source side, so that operation can resume). The vhost-user front-end +may also write to the kick FDs of vrings containing unused buffers or send +``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination +since the driver likely already kicked them in the source and won't do it again.After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).
So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)
Quoted 6 lines
WDYT?
Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.
I don't agree. Here is my thinking about how to solve this:
Device state migration uses the ring state machine to suspend the device on the source host and start rings on the destination host after loading state.
Starting rings is not specific to migration, it is covered by the ring state machine that's also used when pausing and resuming a VM locally, for example. We need to fix the ring state machine section in the spec rather than making changes to the migration or device state fd sections.
Given that there are vhost-user back-ends like DPDK that do not rely on the kick fd (they do not use it when running in poll mode), injecting a kick cannot be necessary. Adding anything to the spec that requires the kick fd is at best a no-op and at worst could break those back-ends.
The spec must be updated to say that rings are started by VHOST_USER_SET_KICK, which is what implementations already do today.
If you want to add a note that front-ends may send a kick after VHOST_USER_SET_KICK completes to ensure that rings are started according to the old spec wording, then that is fine, but it should be clear that kicks are not the primary mechanism for starting rings since it would be dangerous to rely on that (it breaks poll-mode back-ends).
Stefan
Quoted 157 lines · Stefan Hajnoczi
Quoted 137 lines · Stefano Garzarella
Quoted 110 lines · Stefano Garzarella
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit
7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
Thanks,
StefanoQuoted 6 lines
Signed-off-by Jorge E. Moreira <[email protected]>---
docs/interop/vhost-user.rst| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)Quoted 13 lines
docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662
Quoted 13 lines
docs/interop/vhost-user.rst +0
-0@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connectionto a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value
docs/interop/vhost-user.rst +4
-1-as on the source side, so that operation can resume). +as on the source side, so that operation can resume). The vhost-user front-end +may also write to the kick FDs of vrings containing unused buffers or send +``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination +since the driver likely already kicked them in the source and won't do it again.After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).
So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)Quoted 6 lines
WDYT?
Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.
I don't agree. Here is my thinking about how to solve this:
Device state migration uses the ring state machine to suspend the device on the source host and start rings on the destination host after loading state.
Starting rings is not specific to migration, it is covered by the ring state machine that's also used when pausing and resuming a VM locally, for example. We need to fix the ring state machine section in the spec rather than making changes to the migration or device state fd sections.
Quoted 7 lines
Given that there are vhost-user back-ends like DPDK that do not rely on the kick fd (they do not use it when running in poll mode), injecting a kick cannot be necessary. Adding anything to the spec that requires the kick fd is at best a no-op and at worst could break those back-ends.
The spec must be updated to say that rings are started by
VHOST_USER_SET_KICK, which is what implementations already do today.
If we update the spec with this, I agree with you. I also think it's the right thing to do. Even though I would have preferred a specific message to get things started, which would require a new feature, etc., I agree that’s too much.
My reasoning was based on the fact that we always needed to inject a kick to start the rings, which is essentially what we currently state in the specification. Not sure why all the implementation fortunately didn't follow that, maybe because they got inspired by libvhost-user.
BTW, I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration).
If the driver hasn't been initialized, everything is clear.
Quoted 6 lines
If you want to add a note that front-ends may send a kick after
VHOST_USER_SET_KICKcompletes to ensure that rings are started according to the old spec wording, then that is fine, but it should be clear that kicks are not the primary mechanism for starting rings since it would be dangerous to rely on that (it breaks poll-mode back-ends).
Agree, after clarifying VHOST_USER_SET_KICK.
Thanks,
Stefano
Quoted 90 lines · Stefano Garzarella
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.
Quoted 6 lines
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.
QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.
All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.
The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.
The following changes to the vhost-user spec would make this clearer:
1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.
2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.
Stefan
Quoted 1 line
So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)
I admit I don't know what that LOG_FD is for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.
Quoted 1 line
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.
Starting rings when SET_VRING_KICK is received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.
Quoted 4 lines
Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable
Implementations having to do in practice what the spec says today is another reason not to change the it.
While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.
------
Thanks for the feedback! I'll wait for consensus before making further modifications.
Quoted 136 lines · Stefan Hajnoczi
Quoted 90 lines · Stefano Garzarella
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.Quoted 6 lines
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started byVHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped byVHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled byVHOST_USER_SET_VRING_ENABLE.The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.
The following changes to the vhost-user spec would make this clearer:
1. Mention that virtqueues are started by
VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.2. Add a clarification to "Migrating back-end state" that the device must be suspended (see
_suspended_device_state) whenVHOST_USER_SET_DEVICE_STATE_FDis sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.Stefan
Quoted 8 lines · Jorge Moreira
Quoted 2 lines
So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)
Quoted 6 lines
I admit I don't know what that
LOG_FDis for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.
I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.
Quoted 7 lines
Quoted 3 lines
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.
Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.
Quoted 6 lines
Starting rings when
SET_VRING_KICKis received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.
This is a good point and essentially what changed my mind.
However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.
As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once SET_VRING_KICK is received), so perhaps clarifying in this way isn't too risky.
Do you know other backends we should check?
Quoted 9 lines
Quoted 5 lines
Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable
Implementations having to do in practice what the spec says today is another reason not to change the it.
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.
Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.
Quoted 4 lines
------
Thanks for the feedback! I'll wait for consensus before making further modifications.
Thanks!
Stefano
Quoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.Quoted 6 lines
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started byVHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped byVHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled byVHOST_USER_SET_VRING_ENABLE.The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.
The following changes to the vhost-user spec would make this clearer:
1. Mention that virtqueues are started by
VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.2. Add a clarification to "Migrating back-end state" that the device must be suspended (see
_suspended_device_state) whenVHOST_USER_SET_DEVICE_STATE_FDis sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.Stefan
Quoted 1 line
Do you know other backends we should check?
CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend
Quoted 223 lines · Stefano Garzarella
Quoted 8 lines · Jorge Moreira
Quoted 2 lines
So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)Quoted 6 lines
I admit I don't know what that
LOG_FDis for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.
Quoted 7 lines
Quoted 3 lines
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.
Quoted 6 lines
Starting rings when
SET_VRING_KICKis received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.This is a good point and essentially what changed my mind.
However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.
As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once
SET_VRING_KICKis received), so perhaps clarifying in this way isn't too risky.Do you know other backends we should check?
Quoted 9 lines
Quoted 5 lines
Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable
Implementations having to do in practice what the spec says today is another reason not to change the it.
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.
Quoted 4 lines
------
Thanks for the feedback! I'll wait for consensus before making further modifications.Thanks!
StefanoQuoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.Quoted 6 lines
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started byVHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped byVHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled byVHOST_USER_SET_VRING_ENABLE.The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.
The following changes to the vhost-user spec would make this clearer:
1. Mention that virtqueues are started by
VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.2. Add a clarification to "Migrating back-end state" that the device must be suspended (see
_suspended_device_state) whenVHOST_USER_SET_DEVICE_STATE_FDis sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.Stefan
Quoted 4 lines · Jorge Moreira
Quoted 2 lines
Do you know other backends we should check?
CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend
Which do exactly the same
https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596
No?
Quoted 227 lines · Stefano Garzarella
Quoted 223 lines · Jorge Moreira
Quoted 8 lines
Quoted 2 lines
So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)Quoted 6 lines
I admit I don't know what that
LOG_FDis for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.
Quoted 7 lines
Quoted 3 lines
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.
Quoted 6 lines
Starting rings when
SET_VRING_KICKis received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.This is a good point and essentially what changed my mind.
However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.
As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once
SET_VRING_KICKis received), so perhaps clarifying in this way isn't too risky.Do you know other backends we should check?
Quoted 9 lines
Quoted 5 lines
Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable
Implementations having to do in practice what the spec says today is another reason not to change the it.
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.
Quoted 4 lines
------
Thanks for the feedback! I'll wait for consensus before making further modifications.Thanks!
StefanoQuoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines
Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.
Quoted 4 lines · Jorge E. Moreira
Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the
VHOST_USER_VRING_KICKin-band message if negotiated). WhileIt's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified byVHOST_USER_SET_VRING_KICKor receiving the in-band messageVHOST_USER_VRING_KICKif negotiated, and stop a ring upon receivingVHOST_USER_GET_VRING_BASE."But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".Sending a kick from the frontend, seems more an hack here.
That said, for example, in
subprojects/libvhost-user/libvhost-user.cIIUC the virtqueue is started when theSET_VRING_KICKis handled byvu_set_vring_kick_exec(), but not sure how compliant it is.Quoted 7 lines
these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.
Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.
For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."So, IMO we can use the
VHOST_USER_SET_DEVICE_STATE_FDchannel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".Quoted 3 lines
This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:
commit
110b9463d5c820120c8311db79f55a64c9d81ebeAuthor: Yongji Xie <[email protected]> Date: Wed Jun 6 21:24:48 2018 +0800vhost-user-blk: start vhost when guest kicks
Some old guests (before commit7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before settingVIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.Signed-off-by Yongji Xie <[email protected]>Signed-off-by Chai Wen <[email protected]>Signed-off-by Ni Xun <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>Reviewed-by Michael S. Tsirkin <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>Quoted 6 lines
Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.
As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.Quoted 6 lines
That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.
Any other opinion?
IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.
QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICKis received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started byVHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped byVHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled byVHOST_USER_SET_VRING_ENABLE.The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.
The following changes to the vhost-user spec would make this clearer:
1. Mention that virtqueues are started by
VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.2. Add a clarification to "Migrating back-end state" that the device must be suspended (see
_suspended_device_state) whenVHOST_USER_SET_DEVICE_STATE_FDis sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.Stefan
Quoted 10 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines
Do you know other backends we should check?
CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend
Which do exactly the same
https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596No?
To round this out, I looked at DPDK, one of the most popular vhost-user back-end implementations.
The DPDK back-end starts/stops the rings in the vhost-user message handler rather than upon reading the kick fd:
https://github.com/DPDK/dpdk/blob/main/lib/vhost/vhost_user.c#L3274
I think it would be nice to update the spec since it seems implementations aren't actually waiting for the kick fd. However, it is not central to the original discussion, so feel free to skip it if you want.
Stefan
Quoted 6 lines · Jorge Moreira
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.
This is an interesting question to clarify in the spec.
Stefan
Quoted 9 lines · Stefan Hajnoczi
Quoted 6 lines · Jorge Moreira
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.
Quoted 3 lines
This is an interesting question to clarify in the spec.
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."
Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.
Stefano
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.
Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?
Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.
You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
Stefan
Quoted 72 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say thatVHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.
Quoted 5 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
Sounds good to me
Quoted 2 lines
Stefan
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
Quoted 6 lines
Quoted 6 lines
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibility
Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.
IMHO especially new backend implementations shouldn't rely on the kick injection.
So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick
Quoted 4 lines
Quoted 4 lines
with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.
Yep, I think we are aligned.
Quoted 6 lines
Quoted 6 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
LGTM!
Quoted 3 lines
Sounds good to me
@Jorge do you want to propose this change?
Thanks,
Stefano
Quoted 98 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341Quoted 6 lines
Quoted 6 lines
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibilityOkay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.
IMHO especially new backend implementations shouldn't rely on the kick injection.
So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues afterVHOST_USER_SET_VRING_KICK, but it might also receive a kickQuoted 4 lines
Quoted 4 lines
with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.Yep, I think we are aligned.
Quoted 6 lines
Quoted 6 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
Quoted 7 lines
LGTM!
Quoted 3 lines
Sounds good to me
@Jorge do you want to propose this change?
You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.
Quoted 4 lines
Thanks,
Stefano
Quoted 102 lines · Jorge Moreira
Quoted 101 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341Quoted 6 lines
Quoted 6 lines
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibilityOkay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.
IMHO especially new backend implementations shouldn't rely on the kick injection.
So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues afterVHOST_USER_SET_VRING_KICK, but it might also receive a kickQuoted 4 lines
Quoted 4 lines
with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.Yep, I think we are aligned.
Quoted 6 lines
Quoted 6 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
LGTM!
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Sounds good to me
@Jorge do you want to propose this change?
You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.
Okay, I can try, but next week.
Thanks,
Stefano
Quoted 102 lines · Jorge Moreira
Quoted 101 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341Quoted 6 lines
Quoted 6 lines
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibilityOkay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.
IMHO especially new backend implementations shouldn't rely on the kick injection.
So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues afterVHOST_USER_SET_VRING_KICK, but it might also receive a kickQuoted 4 lines
Quoted 4 lines
with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.Yep, I think we are aligned.
Quoted 6 lines
Quoted 6 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
LGTM!
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Sounds good to me
@Jorge do you want to propose this change?
You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.
I can give it a shot and will CC you.
Also, I wanted to apologize for not being very forthcoming in this discussion. I had an intuition about this issue but lacked the time to research and double-check the details. As a result, I didn't respond to all your points in detail. Sorry if it was frustrating.
Stefan
Quoted 75 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341Quoted 6 lines
Quoted 6 lines
QEMU does not inject a kick. The back-end must check the rings itself.
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibilityOkay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.
Quoted 7 lines
IMHO especially new backend implementations shouldn't rely on the kick injection.
So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues afterVHOST_USER_SET_VRING_KICK, but it might also receive a kick
Yes, agreed.
Quoted 26 lines
Quoted 5 lines
Quoted 5 lines
with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.Yep, I think we are aligned.
Quoted 6 lines
Quoted 6 lines
That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
LGTM!
Quoted 3 lines
Sounds good to me
@Jorge do you want to propose this change?
Thanks,
Stefano
Quoted 53 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.
Quoted 8 lines
You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341QEMU does not inject a kick. The back-end must check the rings itself.
I would worry that there could be buggy drivers that expect buffers not consumed until the 1st kick.
A safer thing to do would be for source to record the fact there was a kick after reset and for destination to then inject/check.
Quoted 16 lines
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say thatVHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
Stefan
Quoted 58 lines · Michael S. Tsirkin
Quoted 57 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi
On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
While starting the vrings on
SET_VRING_KICKcould solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.This is an interesting question to clarify in the spec.
Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."Quoted 9 lines
Quoted 2 lines
Stefan
This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.
IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending theSET_VRING_KICKmessage to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.Quoted 7 lines
Quoted 2 lines
Stefano
Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again withVHOST_USER_SET_VRING_KICKto reactivate it?Hi Jorge and Stefano,
Yes,VHOST_USER_GET_VRING_BASE->VHOST_USER_SET_VRING_KICKoccurs when a VM is paused and then resumed.You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.
Quoted 8 lines
Quoted 4 lines
Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341QEMU does not inject a kick. The back-end must check the rings itself.
I would worry that there could be buggy drivers that expect buffers not consumed until the 1st kick.
This describes the behavior today. I'm not sure whether you are commenting on the paragraph I wrote below suggesting how to change things or whether you are commenting on the current state?
Quoted 2 lines
A safer thing to do would be for source to record the fact there was a kick after reset and for destination to then inject/check.
Can you rephrase that, it's unclear to me what "source" and "destination" mean?
Thanks,
Stefan
Quoted 18 lines
Quoted 16 lines
I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICKcompletes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say thatVHOST_USER_SET_VRING_KICKstarts the ring and back-ends SHOULD check the vring upon processing the message.That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.
What do you think?
Stefan