vhost-user.rst: clarify when rings are started
v2 cover vhost-user.rst: clarify when rings are started
Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. This patch series updates the spec to clarify how the ring state machine works, makes libvhost-user spec-compliant, and improves the compatibility of QEMU's vhost-user front-end.
Stefan Hajnoczi (3):
vhost-user.rst: clarify when rings are started
libvhost-user: look for available vq buffers upon SET_VRING_KICK
vhost-user: inject kick after SET_VRING_KICK
docs/interop/vhost-user.rst | 26 +++++++++++++++++------ hw/virtio/vhost-user.c | 24 ++++++++++++++++++++- subprojects/libvhost-user/libvhost-user.c | 10 ++++----- 3 files changed, 48 insertions(+), 12 deletions(-)
Signature · 1 line
2.54.0
PATCH v2 1/3 vhost-user.rst: clarify when rings are started
Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. In the discussion that followed, we discovered that the spec says one thing and implementations do something else. This patch updates the spec to reflect how things are actually implemented across widely-used front-ends and back-ends including QEMU, crosvm, rust-vmm, and DPDK. Do this while taking care not to make any other existing implementations non-compliant by changing the sepc.
The spec says rings are started when a kick is received but the implementations actually start rings when VHOST_USER_SET_VRING_KICK is received.
Reconcile this as follows:
- Clarify that a ring can be stopped and then started again. The back-end must resume processing available requests when the ring is restarted.
- Update the spec to say rings are started when VHOST_USER_SET_VRING_KICK is received.
- Ensure compatibility by saying front-ends SHOULD inject a kick in case the back-end strictly implemented the old spec.
- Avoid future back-end dependencies on injected kicks by saying that back-ends SHOULD NOT expect a kick to start rings.
This way implementors have clarity on how things work while still allowing compatibility for existing implementations.
--- docs/interop/vhost-user.rst | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
docs/interop/vhost-user.rst +20 -6
@@ -462,12 +462,26 @@ Rings have two independent states: started/stopped, and enabled/disabled. * started and enabled: The back-end must process the ring normally, i.e. process all requests and execute them. -Each ring is initialized in a stopped and disabled state. The back-end -must start a ring upon receiving a kick (that is, detecting that file -descriptor is readable) on the descriptor specified by -``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message -``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving -``VHOST_USER_GET_VRING_BASE``. +Each ring is initialized in a stopped and disabled state. Rings are started +with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if +``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and stopped with +``VHOST_USER_GET_VRING_BASE``. A stopped ring enters the started state again +with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if +``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and the back-end +resumes processing requests. + +Note that previous versions of this specification stated that rings start when +the back-end receives a kick (that is, detecting that file descriptor is +readable) on the descriptor specified by ``VHOST_USER_SET_VRING_KICK`` or +receiving the in-band message ``VHOST_USER_VRING_KICK`` if negotiated. +Widely-used front-ends and back-ends did not implement this behavior and it +complicates poll mode back-ends that do not rely on the kick file descriptor. + +For compatibility with back-ends that implemented the start on kick behavior, +front-ends SHOULD inject a kick after ``VHOST_USER_SET_VRING_KICK``. This +ensures that the back-end processes any available requests in the ring. +Back-ends SHOULD NOT rely on receiving a kick after +``VHOST_USER_SET_VRING_KICK``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
Signature · 1 line
2.54.0
Quoted 7 lines · Stefan Hajnoczi
Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. In the discussion that followed, we discovered that the spec says one thing and implementations do something else. This patch updates the spec to reflect how things are actually implemented across widely-used front-ends and back-ends including QEMU, crosvm, rust-vmm, and DPDK. Do this while taking care not to make any other existing implementations non-compliant by changing the sepc.
s/sepc/spec
Quoted 19 lines
The spec says rings are started when a kick is received but the implementations actually start rings when
VHOST_USER_SET_VRING_KICKis received.Reconcile this as follows:
- Clarify that a ring can be stopped and then started again. The back-end must resume processing available requests when the ring is restarted.
- Update the spec to say rings are started whenVHOST_USER_SET_VRING_KICKis received.
- Ensure compatibility by saying front-ends SHOULD inject a kick in case the back-end strictly implemented the old spec.
- Avoid future back-end dependencies on injected kicks by saying that back-ends SHOULD NOT expect a kick to start rings.This way implementors have clarity on how things work while still allowing compatibility for existing implementations.
Quoted 7 lines
Reported-by Jorge Moreira <[email protected]>Cc "Michael S . Tsirkin" <[email protected]>Cc Stefano Garzarella <[email protected]>Signed-off-by Stefan Hajnoczi <[email protected]>---
docs/interop/vhost-user.rst| 26 ++++++++++++++++++++------1 file changed, 20 insertions(+), 6 deletions(-)
Quoted 10 lines · Stefano Garzarella
Quoted 7 lines · Stefan Hajnoczi
Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. In the discussion that followed, we discovered that the spec says one thing and implementations do something else. This patch updates the spec to reflect how things are actually implemented across widely-used front-ends and back-ends including QEMU, crosvm, rust-vmm, and DPDK. Do this while taking care not to make any other existing implementations non-compliant by changing the sepc.
s/sepc/spec
Will fix.
Stefan
PATCH v2 2/3 libvhost-user: look for available vq buffers upon SET_VRING_KICK
In reply to Stefan HajnocziWhen a vring is started the back-end must look for available vq buffers and process them. This scenario can happen if the back-end is stopped with unprocessed available buffers and then started again.
The inflight I/O tracking code already did this, but it should also be done when inflight I/O tracking is not enabled. Move the code.
--- subprojects/libvhost-user/libvhost-user.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
subprojects/libvhost-user/libvhost-user.c +5 -5
@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq) vq->counter = vq->resubmit_list[0 ].counter +1 ; } -/* in case of I/O hang after reconnecting */ -if (eventfd_write (vq->kick_fd,1 )) { -return -1 ; - } -return 0 ; }
@@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)vu_panic (dev,"Failed to check inflights for vq: %d\n" , index); } +/* Inject a kick to look for available vq buffers */ +if (eventfd_write (dev->vq[index].kick_fd,1 )) { +return -1 ; + } +return false; }
Signature · 1 line
2.54.0
Re: [PATCH v2 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK
Quoted 29 lines · Stefan Hajnoczi
When a vring is started the back-end must look for available vq buffers and process them. This scenario can happen if the back-end is stopped with unprocessed available buffers and then started again.
The inflight I/O tracking code already did this, but it should also be done when inflight I/O tracking is not enabled. Move the code.
Signed-off-by Stefan Hajnoczi <[email protected]>---
subprojects/libvhost-user/libvhost-user.c| 10 +++++-----1 file changed, 5 insertions(+), 5 deletions(-)subprojects/libvhost-user/libvhost-user.c +0
-5@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq) vq->counter = vq->resubmit_list[0 ].counter +1 ; } -/* in case of I/O hang after reconnecting */ -if (eventfd_write (vq->kick_fd,1 )) { -return -1 ; - } -return 0 ; }
subprojects/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1435
Quoted 5 lines
subprojects/libvhost-user/libvhost-user.c +2
-0@@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)vu_panic (dev,"Failed to check inflights for vq: %d\n" , index); } +/* Inject a kick to look for available vq buffers */ +if (eventfd_write (dev->vq[index].kick_fd,1 )) {
Should we check if kick_fd is not -1 like we already do in other places in this function? (mainly when nofd is true)
subprojects/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1436
Quoted 1 line
subprojects/libvhost-user/libvhost-user.c +1
-0+return -1 ;
This function returns boolean, -1 is true and IIUC means that we ask for a reply which is not the case here. We should fix it.
Thanks,
Stefano
Quoted 9 lines
subprojects/libvhost-user/libvhost-user.c +2
-0+ } +return false; }Signature · 2 lines
-- 2.54.0
Re: [PATCH v2 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK
Quoted 39 lines · Stefano Garzarella
Quoted 29 lines · Stefan Hajnoczi
When a vring is started the back-end must look for available vq buffers and process them. This scenario can happen if the back-end is stopped with unprocessed available buffers and then started again.
The inflight I/O tracking code already did this, but it should also be done when inflight I/O tracking is not enabled. Move the code.
Signed-off-by Stefan Hajnoczi <[email protected]>---
subprojects/libvhost-user/libvhost-user.c| 10 +++++-----1 file changed, 5 insertions(+), 5 deletions(-)subprojects/libvhost-user/libvhost-user.c +0
-5@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq) vq->counter = vq->resubmit_list[0 ].counter +1 ; } -/* in case of I/O hang after reconnecting */ -if (eventfd_write (vq->kick_fd,1 )) { -return -1 ; - } -return 0 ; }subprojects/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1435
Quoted 5 lines
subprojects/libvhost-user/libvhost-user.c +2
-0@@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)vu_panic (dev,"Failed to check inflights for vq: %d\n" , index); } +/* Inject a kick to look for available vq buffers */ +if (eventfd_write (dev->vq[index].kick_fd,1 )) {Should we check if
kick_fdis not -1 like we already do in other places in this function? (mainly whennofdis true)
Quoted 4 lines
subprojects/libvhost-user/libvhost-user.c vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) · line 1436
Quoted 1 line
subprojects/libvhost-user/libvhost-user.c +1
-0+return -1 ;This function returns boolean, -1 is
trueand IIUC means that we ask for a reply which is not the case here. We should fix it.
Thanks for catch these things. I'm going to redo this patch. Not sure what I was thinking :).
Stefan
PATCH v2 3/3 vhost-user: inject kick after SET_VRING_KICK
In reply to Stefan HajnocziThe vhost-user specification was updated to say that front-ends should inject a kick after SET_VRING_KICK in case the back-end implements the old spec wording which said vrings start when a kick is received. Do this in QEMU's front-end.
An example scenario where this behavior helps: the back-end fails to check if the vring has available buffers when SET_VRING_KICK is received and the front-end stopped and then restarted the vring. In the case the back-end may not notice the available buffers unless the front-end injects a kick.
--- hw/virtio/vhost-user.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
hw/virtio/vhost-user.c +23 -1
@@ -1365,7 +1365,29 @@ static int vhost_set_vring_file(struct vhost_dev *dev,static int vhost_user_set_vring_kick (struct vhost_dev *dev,struct vhost_vring_file *file) { -return vhost_set_vring_file (dev,VHOST_USER_SET_VRING_KICK , file); +int ret =vhost_set_vring_file (dev,VHOST_USER_SET_VRING_KICK , file); +if (ret <0 ) { +return ret; + } + +/* +* Inject a kick in case the back-end only starts vring processing upon +* receiving a kick. The spec suggests this to improve compatibility. +*/ +if (file->fd !=-1 ) { +uint64_t val =1 ; +ssize_t nwritten; + +do { + nwritten =write (file->fd, &val,sizeof (val)); + }while (nwritten <0 && errno ==EINTR ); + +if (nwritten <0 && errno !=EAGAIN /* back-end can already read */ ) { +return -errno; + } + } + +return 0 ; } static int vhost_user_set_vring_call(struct vhost_dev *dev,
Signature · 1 line
2.54.0
Re: [PATCH v2 3/3] vhost-user: inject kick after SET_VRING_KICK
Quoted 11 lines · Stefan Hajnoczi
The vhost-user specification was updated to say that front-ends should inject a kick after
SET_VRING_KICKin case the back-end implements the old spec wording which said vrings start when a kick is received. Do this in QEMU's front-end.An example scenario where this behavior helps: the back-end fails to check if the vring has available buffers when
SET_VRING_KICKis received and the front-end stopped and then restarted the vring. In the case the back-end may not notice the available buffers unless the front-end injects a kick.
Quoted 4 lines
Signed-off-by Stefan Hajnoczi <[email protected]>---
hw/virtio/vhost-user.c| 24 +++++++++++++++++++++++-1 file changed, 23 insertions(+), 1 deletion(-)