Skip to contentSkip to conversation

vhost-user.rst: Explicitly allow front-end to write to kick FDs

PATCH In review Jorge E. Moreira 4 participants 24 messages Apr 11 – Apr 29

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vhost-user.rst: Explicitly allow front-end to write to kick FDs

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm). Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

---
 docs/interop/vhost-user.rst | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
docs/interop/vhost-user.rst +4 -1
index 137c9f3669..ad5aba3430 100644
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
 to a vhost-user back-end: This includes, for example, setting up memory
 mappings and kick and call FDs as necessary, negotiating protocol
 features, or setting the initial vring base indices (to the same value
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.
 
 Both on the source and on the destination side, after the respective
 front-end has seen all data transferred (when the transfer FD has been
Signature · 1 line
2.53.0.1213.gd9a14994de-goog
Reviewed-by Stefan Hajnoczi Reviewed-by Michael S. Tsirkin on v1 1/1
#1 lore raw

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks

Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

Thanks,
Stefano

Quoted 25 lines

---
docs/interop/vhost-user.rst | 5 ++++-

1 file changed, 4 insertions(+), 1 deletion(-)
docs/interop/vhost-user.rst +0 -0
index 137c9f3669..ad5aba3430 100644
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection

to a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value

docs/interop/vhost-user.rst +4 -1
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.

Both on the source and on the destination side, after the respective front-end has seen all data transferred (when the transfer FD has been

Signature · 1 line
2.53.0.1213.gd9a14994de-goog
#2 lore raw
Quoted 110 lines · Stefano Garzarella

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks

Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

Thanks,
Stefano

Quoted 6 lines

---
docs/interop/vhost-user.rst | 5 ++++-

1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines

docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662

Quoted 13 lines
docs/interop/vhost-user.rst +0 -0
index 137c9f3669..ad5aba3430 100644
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection

to a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value

docs/interop/vhost-user.rst +4 -1
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.

After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).

So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

WDYT?

Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.

Thanks,
Stefano

#3 lore raw
Quoted 137 lines · Stefano Garzarella
Quoted 110 lines · Stefano Garzarella

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks

Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

Thanks,
Stefano

Quoted 6 lines

---
docs/interop/vhost-user.rst | 5 ++++-

1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines

docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662

Quoted 13 lines
docs/interop/vhost-user.rst +0 -0
index 137c9f3669..ad5aba3430 100644
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection

to a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value

docs/interop/vhost-user.rst +4 -1
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.

After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).

So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

Quoted 6 lines

WDYT?

Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.

I don't agree. Here is my thinking about how to solve this:

Device state migration uses the ring state machine to suspend the device on the source host and start rings on the destination host after loading state.

Starting rings is not specific to migration, it is covered by the ring state machine that's also used when pausing and resuming a VM locally, for example. We need to fix the ring state machine section in the spec rather than making changes to the migration or device state fd sections.

Given that there are vhost-user back-ends like DPDK that do not rely on the kick fd (they do not use it when running in poll mode), injecting a kick cannot be necessary. Adding anything to the spec that requires the kick fd is at best a no-op and at worst could break those back-ends.

The spec must be updated to say that rings are started by VHOST_USER_SET_KICK, which is what implementations already do today.

If you want to add a note that front-ends may send a kick after VHOST_USER_SET_KICK completes to ensure that rings are started according to the old spec wording, then that is fine, but it should be clear that kicks are not the primary mechanism for starting rings since it would be dangerous to rely on that (it breaks poll-mode back-ends).

Stefan

#4 lore raw
Quoted 157 lines · Stefan Hajnoczi
Quoted 137 lines · Stefano Garzarella
Quoted 110 lines · Stefano Garzarella

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end must start a ring upon receiving a kick (that is, detecting that file descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one back-end, called the source, to another back-end, called the destination. After migration, the destination transparently resumes operation without requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks

Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

Thanks,
Stefano

Quoted 6 lines

---
docs/interop/vhost-user.rst | 5 ++++-

1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines

docs/interop/vhost-user.rst destination, following the usual protocol for establishing a connection · line 662

Quoted 13 lines
docs/interop/vhost-user.rst +0 -0
index 137c9f3669..ad5aba3430 100644
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection

to a vhost-user back-end: This includes, for example, setting up memory mappings and kick and call FDs as necessary, negotiating protocol features, or setting the initial vring base indices (to the same value

docs/interop/vhost-user.rst +4 -1
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.

After discussing this with Hanna, we came to the conclusion that your idea of injecting the kick is the least invasive option for now and complies with the spec (even though I still don’t think it’s a nice thing to do).

So it’s fine to continue in this direction, but I might add these words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

Quoted 6 lines

WDYT?

Then instead of "may" I'd use "should". And I would refer to the fact that migration is transparent to the driver, so the front-end should kick all initialized vrings to comply with what we described in the "Ring states" section.

I don't agree. Here is my thinking about how to solve this:

Device state migration uses the ring state machine to suspend the device on the source host and start rings on the destination host after loading state.

Starting rings is not specific to migration, it is covered by the ring state machine that's also used when pausing and resuming a VM locally, for example. We need to fix the ring state machine section in the spec rather than making changes to the migration or device state fd sections.

Quoted 7 lines

Given that there are vhost-user back-ends like DPDK that do not rely on the kick fd (they do not use it when running in poll mode), injecting a kick cannot be necessary. Adding anything to the spec that requires the kick fd is at best a no-op and at worst could break those back-ends.

The spec must be updated to say that rings are started by VHOST_USER_SET_KICK, which is what implementations already do today.

If we update the spec with this, I agree with you. I also think it's the right thing to do. Even though I would have preferred a specific message to get things started, which would require a new feature, etc., I agree that’s too much.

My reasoning was based on the fact that we always needed to inject a kick to start the rings, which is essentially what we currently state in the specification. Not sure why all the implementation fortunately didn't follow that, maybe because they got inspired by libvhost-user.

BTW, I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration).

If the driver hasn't been initialized, everything is clear.

Quoted 6 lines

If you want to add a note that front-ends may send a kick after VHOST_USER_SET_KICK completes to ensure that rings are started according to the old spec wording, then that is fine, but it should be clear that kicks are not the primary mechanism for starting rings since it would be dangerous to rely on that (it breaks poll-mode back-ends).

Agree, after clarifying VHOST_USER_SET_KICK.

Thanks,
Stefano

#5 lore raw
In reply to Stefano Garzarella
Quoted 90 lines · Stefano Garzarella

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

Quoted 6 lines

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.

2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.

Stefan

#6 lore raw
Quoted 1 line

So it’s fine to continue in this direction, but I might add these

words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

I admit I don't know what that LOG_FD is for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.

Quoted 1 line

QEMU's libvhost-user implementation already starts the ring when

VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

Starting rings when SET_VRING_KICK is received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.

Quoted 4 lines

Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable

Implementations having to do in practice what the spec says today is another reason not to change the it.

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

------
Thanks for the feedback! I'll wait for consensus before making further modifications.

Quoted 136 lines · Stefan Hajnoczi
Quoted 90 lines · Stefano Garzarella

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

Quoted 6 lines

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.

2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.

Stefan

#7 lore raw
Quoted 8 lines · Jorge Moreira
Quoted 2 lines

So it’s fine to continue in this direction, but I might add these

words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

Quoted 6 lines

I admit I don't know what that LOG_FD is for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.

I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.

Quoted 7 lines
Quoted 3 lines

QEMU's libvhost-user implementation already starts the ring when

VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.

Quoted 6 lines

Starting rings when SET_VRING_KICK is received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.

This is a good point and essentially what changed my mind.

However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.

As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once SET_VRING_KICK is received), so perhaps clarifying in this way isn't too risky.

Do you know other backends we should check?

Quoted 9 lines
Quoted 5 lines

Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable

Implementations having to do in practice what the spec says today is another reason not to change the it.

Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.

Quoted 4 lines

------
Thanks for the feedback! I'll wait for consensus before making further modifications.

Thanks!
Stefano

Quoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

Quoted 6 lines

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.

2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.

Stefan

#8 lore raw
Quoted 1 line

Do you know other backends we should check?

CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend

Quoted 223 lines · Stefano Garzarella
Quoted 8 lines · Jorge Moreira
Quoted 2 lines

So it’s fine to continue in this direction, but I might add these

words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

Quoted 6 lines

I admit I don't know what that LOG_FD is for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.

I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.

Quoted 7 lines
Quoted 3 lines

QEMU's libvhost-user implementation already starts the ring when

VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.

Quoted 6 lines

Starting rings when SET_VRING_KICK is received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.

This is a good point and essentially what changed my mind.

However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.

As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once SET_VRING_KICK is received), so perhaps clarifying in this way isn't too risky.

Do you know other backends we should check?

Quoted 9 lines
Quoted 5 lines

Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable

Implementations having to do in practice what the spec says today is another reason not to change the it.

Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.

Quoted 4 lines

------
Thanks for the feedback! I'll wait for consensus before making further modifications.

Thanks!
Stefano

Quoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

Quoted 6 lines

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.

2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.

Stefan

#9 lore raw
Quoted 4 lines · Jorge Moreira
Quoted 2 lines

Do you know other backends we should check?

CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend

Which do exactly the same
https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596

No?

Quoted 227 lines · Stefano Garzarella
Quoted 223 lines · Jorge Moreira
Quoted 8 lines
Quoted 2 lines

So it’s fine to continue in this direction, but I might add these

words more in the "Migration" section than here, since we’re talking about an optional state migration here. For example after "No further update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a ring after a migration. Or in both :-)

Quoted 6 lines

I admit I don't know what that LOG_FD is for or what a "userfault-fd" is, so I don't know how they relate to vring state. The paragraph I modified discusses which part of the back-end state is in fact handled by the front-end, which somewhat matches the "vring-can/should-kick" idea. That being said, I don't have a strong preference for where this should go. Repeating it in "Ring states" makes sense to me.

I didn't express myself clearly, but my point is that discussing this in the "Migrating back-end state" section is misleading. We should address it first because it has nothing to do with state; it's also needed for devices that don't require any state transfer.

Quoted 7 lines
Quoted 3 lines

QEMU's libvhost-user implementation already starts the ring when

VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.

Quoted 6 lines

Starting rings when SET_VRING_KICK is received technically violates the vhost-user protocol as defined in the spec. QEMU may support back-ends behaving this way, but other VMMs may not. Changing the spec accordingly could potentially cause other implementations not depending on QEMU's libvhost-user to become non-compliant, I prefer to avoid this.

This is a good point and essentially what changed my mind.

However, this doesn't change anything for those frontends that also send the kick (I don't know which ones do this, but QEMU doesn't, and I don't think CH does either, since it uses the rust-vmm crates); they continue to work just the same.

As for the backends, we’ve seen that both libvhost-user in QEMU and vhost-user-backend in rust-vmm do exactly this (start vrings once SET_VRING_KICK is received), so perhaps clarifying in this way isn't too risky.

Do you know other backends we should check?

Quoted 9 lines
Quoted 5 lines

Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable

Implementations having to do in practice what the spec says today is another reason not to change the it.

Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

Once the vrings are set, the VMM and the device have exactly the same view of the vring state, so IMO the kick is just some extra step (I'm fine to add it just because the spec wasn't clear). The device can start in the same way without it.

Quoted 4 lines

------
Thanks for the feedback! I'll wait for consensus before making further modifications.

Thanks!
Stefano

Quoted 140 lines · Stefan Hajnoczi
Quoted 136 lines · Stefano Garzarella
Quoted 90 lines

Thanks for starting the discussion here, let me add also Hanna, German, and Stefan in CC that can help us.

Quoted 4 lines · Jorge E. Moreira

Migration of back-end state happens while the device is suspended (i.e all vrings are stopped). To resume normal operation on the destination, the vrings need to be started again with a kick (either a write on the FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
"Each ring is initialized in a stopped and disabled state. The back-end
must start a ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK or receiving the in-band message VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the new device running in the destination, IMO we should consider the ring already started or add some messages to tell to the device: "hey, the device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC the virtqueue is started when the SET_VRING_KICK is handled by vu_set_vring_kick_exec(), but not sure how compliant it is.

Quoted 7 lines

these notifications are typically sent by the driver, it has no reason to send them in the destination if it already sent them in the source as the driver is unaware that a migration took place. Therefore it should be the responsibility of the vhost-user front-end to ensure these vrings are started. This is particularly necessary for queues where data only flows from device to driver, such as those used by the vsock and input devices.

Exactly, so IMO we should not use the kick, but maybe add something new or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
"Migrating device state involves transferring the state from one
back-end, called the source, to another back-end, called the destination.
After migration, the destination transparently resumes operation without
requiring the driver to re-initialize the device at the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to inform the new device about the state: "there isn't any state to transfer, but I notify you that the device was already initialized, so the vrings can be started".

Quoted 3 lines

This behavior is already used by some qemu vhost-user front-ends (e.g vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see the code around the comment "/* Kick right away to begin processing requests already in vring */" but that one IIUC was introduced more to fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe

Author: Yongji Xie <[email protected]>
Date:   Wed Jun 6 21:24:48 2018 +0800

vhost-user-blk: start vhost when guest kicks
Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates the virtio spec. But virtio 1.0 transitional devices support this behaviour. So we should start vhost when guest kicks in this case.

Quoted 6 lines

Adding it to the vhost-user documentation makes it explicit that this strategy is permitted and suggest it to vhost-user front-end authors. Explicitly documenting it is necessary because vring kicks appear designed to originate in the driver, so having some originate in the front-end can be counterintuitive and cause developers to waste time looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that should be used only by the driver to notify the device about new available buffers.

Quoted 6 lines

That said, I agree that we need to clarify in the specifications exactly what the backend and frontend should do after a migration to start vrings if there is no need to exchange a state.

Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable.

2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section.

Stefan

#10 lore raw
Quoted 10 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines

Do you know other backends we should check?

CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend

Which do exactly the same
https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596

No?

To round this out, I looked at DPDK, one of the most popular vhost-user back-end implementations.

The DPDK back-end starts/stops the rings in the vhost-user message handler rather than upon reading the kick fd:
https://github.com/DPDK/dpdk/blob/main/lib/vhost/vhost_user.c#L3274

I think it would be nice to update the spec since it seems implementations aren't actually waiting for the kick fd. However, it is not central to the original discussion, so feel free to skip it if you want.

Stefan

#11 lore raw
In reply to Jorge Moreira
Quoted 6 lines · Jorge Moreira

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Stefan

#12 lore raw
Quoted 9 lines · Stefan Hajnoczi
Quoted 6 lines · Jorge Moreira

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

Quoted 3 lines

This is an interesting question to clarify in the spec.

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

#13 lore raw
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Stefano

#14 lore raw
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

#15 lore raw
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

Stefan

#16 lore raw
Quoted 72 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Quoted 5 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

Sounds good to me

Quoted 2 lines

Stefan

#17 lore raw
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

Quoted 6 lines
Quoted 6 lines

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.

IMHO especially new backend implementations shouldn't rely on the kick injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick

Quoted 4 lines
Quoted 4 lines

with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Yep, I think we are aligned.

Quoted 6 lines
Quoted 6 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

LGTM!

Quoted 3 lines

Sounds good to me

@Jorge do you want to propose this change?

Thanks,
Stefano

#18 lore raw
Quoted 98 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

Quoted 6 lines
Quoted 6 lines

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.

IMHO especially new backend implementations shouldn't rely on the kick injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick

Quoted 4 lines
Quoted 4 lines

with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Yep, I think we are aligned.

Quoted 6 lines
Quoted 6 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

Quoted 7 lines

LGTM!

Quoted 3 lines

Sounds good to me

@Jorge do you want to propose this change?

You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.

Quoted 4 lines

Thanks,
Stefano

#19 lore raw
Quoted 102 lines · Jorge Moreira
Quoted 101 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

Quoted 6 lines
Quoted 6 lines

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.

IMHO especially new backend implementations shouldn't rely on the kick injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick

Quoted 4 lines
Quoted 4 lines

with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Yep, I think we are aligned.

Quoted 6 lines
Quoted 6 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

LGTM!

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Sounds good to me

@Jorge do you want to propose this change?

You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.

Okay, I can try, but next week.

Thanks,
Stefano

#20 lore raw
In reply to Jorge Moreira
Quoted 102 lines · Jorge Moreira
Quoted 101 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

Quoted 6 lines
Quoted 6 lines

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.

IMHO especially new backend implementations shouldn't rely on the kick injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick

Quoted 4 lines
Quoted 4 lines

with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Yep, I think we are aligned.

Quoted 6 lines
Quoted 6 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

LGTM!

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Sounds good to me

@Jorge do you want to propose this change?

You have a much better idea of where and how this should be written, it will save us a few rounds of review if one of you makes that change.

I can give it a shot and will CC you.

Also, I wanted to apologize for not being very forthcoming in this discussion. I had an intuition about this issue but lacked the time to research and double-check the details. As a result, I didn't respond to all your points in detail. Sorry if it was frustrating.

Stefan

#21 lore raw
In reply to Stefano Garzarella
Quoted 75 lines · Stefano Garzarella
Quoted 62 lines · Jorge Moreira
Quoted 61 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

Quoted 6 lines
Quoted 6 lines

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, we need to make it clear that a backend shouldn't necessarily expect the kick injected from every frontend, but should support it in some way since some of them can inject it.

Quoted 7 lines

IMHO especially new backend implementations shouldn't rely on the kick injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, but it might also receive a kick

Yes, agreed.

Quoted 26 lines
Quoted 5 lines
Quoted 5 lines

with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

Yep, I think we are aligned.

Quoted 6 lines
Quoted 6 lines

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

LGTM!

Quoted 3 lines

Sounds good to me

@Jorge do you want to propose this change?

Thanks,
Stefano

#22 lore raw
In reply to Stefan Hajnoczi
Quoted 53 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

Quoted 8 lines

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

QEMU does not inject a kick. The back-end must check the rings itself.

I would worry that there could be buggy drivers that expect buffers not consumed until the 1st kick.

A safer thing to do would be for source to record the fact there was a kick after reset and for destination to then inject/check.

Quoted 16 lines

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

Stefan

#23 lore raw
Quoted 58 lines · Michael S. Tsirkin
Quoted 57 lines · Stefan Hajnoczi
Quoted 40 lines · Jorge Moreira
Quoted 39 lines · Stefano Garzarella
Quoted 4 lines · Jorge Moreira
Quoted 2 lines · Stefan Hajnoczi

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

While starting the vrings on SET_VRING_KICK could solve the state machine issue, it still won't notify the back-end that buffers are ready (the driver won't do this). Non-polling back-ends depend on this kick, especially for queues where data flows only from the driver to the back-end. Most implementations likely attempt to read from the queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do when the vrings are configured, but the driver has already been initialized (which is usually the case after migration)."

Quoted 9 lines
Quoted 2 lines

Stefan

This is the question that interests me most, to be honest. I'd rather have the discussion about when to activate the vrings in a different thread and keep this one focused on whether the front-end should send the kick or if the back-end is expected to check if there are "new" buffers in the vring after restore.

IMO we don't need anything from the VMM. When the device receives SET_VRING_KICK, it can check if the vring already contains buffers (and this is the part we might need to clarify) and wake-up the other threads (or always wake-ups them, as crosvm does IIUC, and let them perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has the exact same knowledge of the vring state as the device, therefore, it's still unclear to me why we need to inject that kick.

Quoted 7 lines
Quoted 2 lines

Stefano

Is it possible to activate a vring after it has been deactivated with VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the kick file descriptor again with VHOST_USER_SET_VRING_KICK to reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio (virtio-blk) inside the guest and sending 'stop'/'cont' commands to QEMU's monitor.

Quoted 8 lines
Quoted 4 lines

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

QEMU does not inject a kick. The back-end must check the rings itself.

I would worry that there could be buggy drivers that expect buffers not consumed until the 1st kick.

This describes the behavior today. I'm not sure whether you are commenting on the paragraph I wrote below suggesting how to change things or whether you are commenting on the current state?

Quoted 2 lines

A safer thing to do would be for source to record the fact there was a kick after reset and for destination to then inject/check.

Can you rephrase that, it's unclear to me what "source" and "destination" mean?

Thanks,
Stefan

Quoted 18 lines
Quoted 16 lines

I'm not sure that all vhost-user back-ends actually check the rings. I think back-ends should do it, but we should also update the spec with an front-end implementation note recommending injecting a kick after VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility with implementations that follow the current spec wording. And at the same time I think the spec should also be changed to say that VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the vring upon processing the message.

That seems like it would clean up the issues without introducing compatibility issues or making existing implementations non-compliant with the updated spec.

What do you think?

Stefan