Skip to contentSkip to conversation

vhost-user.rst: clarify when rings are started

PATCH In review Stefan Hajnoczi 2 participants 5 messages Jun 4 – Jun 5

lore
Share & tools
lore
#0 lore raw

v3 cover vhost-user.rst: clarify when rings are started

v3:
- Fix sepc -> spec typo in Patch 1 [Stefano]
- Check for a -1 fd in Patch 2 [Stefano]
- Handle failure with vu_panic() instead of incorrect return -1 in Patch 2 [Stefano]
- Add eventfd_write() EINTR and EAGAIN handling in Patch 2

Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. This patch series updates the spec to clarify how the ring state machine works, makes libvhost-user spec-compliant, and improves the compatibility of QEMU's vhost-user front-end.

Stefan Hajnoczi (3):
vhost-user.rst: clarify when rings are started
libvhost-user: look for available vq buffers upon SET_VRING_KICK
vhost-user: inject kick after SET_VRING_KICK

 docs/interop/vhost-user.rst               | 26 +++++++++++++++++------
 hw/virtio/vhost-user.c                    | 24 ++++++++++++++++++++-
 subprojects/libvhost-user/libvhost-user.c | 19 ++++++++++++-----
 3 files changed, 57 insertions(+), 12 deletions(-)
Signature · 1 line
2.54.0
#1 lore raw

PATCH v3 1/3 vhost-user.rst: clarify when rings are started

Jorge Moreira <[email protected]> pointed out that the ring state machine is underspecified. In the discussion that followed, we discovered that the spec says one thing and implementations do something else. This patch updates the spec to reflect how things are actually implemented across widely-used front-ends and back-ends including QEMU, crosvm, rust-vmm, and DPDK. Do this while taking care not to make any other existing implementations non-compliant by changing the spec.

The spec says rings are started when a kick is received but the implementations actually start rings when VHOST_USER_SET_VRING_KICK is received.

Reconcile this as follows:
- Clarify that a ring can be stopped and then started again. The back-end must resume processing available requests when the ring is restarted.
- Update the spec to say rings are started when VHOST_USER_SET_VRING_KICK is received.
- Ensure compatibility by saying front-ends SHOULD inject a kick in case the back-end strictly implemented the old spec.
- Avoid future back-end dependencies on injected kicks by saying that back-ends SHOULD NOT expect a kick to start rings.

This way implementors have clarity on how things work while still allowing compatibility for existing implementations.

---
 docs/interop/vhost-user.rst | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
docs/interop/vhost-user.rst +20 -6
index 137c9f3669..6ced95e080 100644
@@ -462,12 +462,26 @@ Rings have two independent states: started/stopped, and enabled/disabled.
 * started and enabled: The back-end must process the ring normally, i.e.
   process all requests and execute them.
 
-Each ring is initialized in a stopped and disabled state.  The back-end
-must start a ring upon receiving a kick (that is, detecting that file
-descriptor is readable) on the descriptor specified by
-``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
-``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving
-``VHOST_USER_GET_VRING_BASE``.
+Each ring is initialized in a stopped and disabled state.  Rings are started
+with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
+``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and stopped with
+``VHOST_USER_GET_VRING_BASE``.  A stopped ring enters the started state again
+with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
+``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and the back-end
+resumes processing requests.
+
+Note that previous versions of this specification stated that rings start when
+the back-end receives a kick (that is, detecting that file descriptor is
+readable) on the descriptor specified by ``VHOST_USER_SET_VRING_KICK`` or
+receiving the in-band message ``VHOST_USER_VRING_KICK`` if negotiated.
+Widely-used front-ends and back-ends did not implement this behavior and it
+complicates poll mode back-ends that do not rely on the kick file descriptor.
+
+For compatibility with back-ends that implemented the start on kick behavior,
+front-ends SHOULD inject a kick after ``VHOST_USER_SET_VRING_KICK``.  This
+ensures that the back-end processes any available requests in the ring.
+Back-ends SHOULD NOT rely on receiving a kick after
+``VHOST_USER_SET_VRING_KICK``.
 
 Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
 
Signature · 1 line
2.54.0
#2 lore raw

PATCH v3 2/3 libvhost-user: look for available vq buffers upon SET_VRING_KICK

In reply to Stefan Hajnoczi

When a vring is started the back-end must look for available vq buffers and process them. This scenario can happen if the back-end is stopped with unprocessed available buffers and then started again.

The inflight I/O tracking code already did this, but it should also be done when inflight I/O tracking is not enabled.

Move the code and make it robust in case of EINTR or EAGAIN.

---
 subprojects/libvhost-user/libvhost-user.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
subprojects/libvhost-user/libvhost-user.c +14 -5
index 9c630c2170..1dbb774a76 100644
@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq)
         vq->counter = vq->resubmit_list[0].counter + 1;
     }
 
-    /* in case of I/O hang after reconnecting */
-    if (eventfd_write(vq->kick_fd, 1)) {
-        return -1;
-    }
-
     return 0;
 }
 
@@ -1436,6 +1431,20 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
         vu_panic(dev, "Failed to check inflights for vq: %d\n", index);
     }
 
+    /* Inject a kick to look for available vq buffers */
+    if (dev->vq[index].kick_fd != -1) {
+        int ret;
+
+        do {
+            ret = eventfd_write(dev->vq[index].kick_fd, 1);
+        } while (ret != 0 && errno == EINTR);
+
+        if (ret != 0 && errno != EAGAIN /* already readable */) {
+            vu_panic(dev, "Failed to inject kick during SET_VRING_KICK "
+                     "on vq: %d with error: %m\n", index);
+        }
+    }
+
     return false;
 }
 
Signature · 1 line
2.54.0
Reviewed-by on v3 2/3
#3 lore raw

Re: [PATCH v3 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK

Quoted 7 lines · Stefan Hajnoczi

When a vring is started the back-end must look for available vq buffers and process them. This scenario can happen if the back-end is stopped with unprocessed available buffers and then started again.

The inflight I/O tracking code already did this, but it should also be done when inflight I/O tracking is not enabled.

Quoted 6 lines

Move the code and make it robust in case of EINTR or EAGAIN.

---
subprojects/libvhost-user/libvhost-user.c | 19 ++++++++++++++-----

1 file changed, 14 insertions(+), 5 deletions(-)
#4 lore raw

PATCH v3 3/3 vhost-user: inject kick after SET_VRING_KICK

In reply to Stefan Hajnoczi

The vhost-user specification was updated to say that front-ends should inject a kick after SET_VRING_KICK in case the back-end implements the old spec wording which said vrings start when a kick is received. Do this in QEMU's front-end.

An example scenario where this behavior helps: the back-end fails to check if the vring has available buffers when SET_VRING_KICK is received and the front-end stopped and then restarted the vring. In the case the back-end may not notice the available buffers unless the front-end injects a kick.

---
 hw/virtio/vhost-user.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
hw/virtio/vhost-user.c +23 -1
index a8907cca74..7ef4105473 100644
@@ -1365,7 +1365,29 @@ static int vhost_set_vring_file(struct vhost_dev *dev,
 static int vhost_user_set_vring_kick(struct vhost_dev *dev,
                                      struct vhost_vring_file *file)
 {
-    return vhost_set_vring_file(dev, VHOST_USER_SET_VRING_KICK, file);
+    int ret = vhost_set_vring_file(dev, VHOST_USER_SET_VRING_KICK, file);
+    if (ret < 0) {
+        return ret;
+    }
+
+    /*
+     * Inject a kick in case the back-end only starts vring processing upon
+     * receiving a kick. The spec suggests this to improve compatibility.
+     */
+    if (file->fd != -1) {
+        uint64_t val = 1;
+        ssize_t nwritten;
+
+        do {
+            nwritten = write(file->fd, &val, sizeof(val));
+        } while (nwritten < 0 && errno == EINTR);
+
+        if (nwritten < 0 && errno != EAGAIN /* back-end can already read */) {
+            return -errno;
+        }
+    }
+
+    return 0;
 }
 
 static int vhost_user_set_vring_call(struct vhost_dev *dev,
Signature · 1 line
2.54.0