Skip to contentSkip to conversation

virtio fix up started checks

RFC PATCH In review Michael S. Tsirkin 1 participant 5 messages Nov 5, 2022

lore
Share & tools
lore
#0 lore raw

cover virtio fix up started checks

This is an attempt to fix up device started checks. Unfortunately this causes failures in CI and I could not figure it out.

The simplest way to test is to set QEMU_CI to 2
on gitlab, then push there.

Alternatively, push to gitlab, then
create pipeline while setting QEMU_CI to 1, then run amd64-fedora-container and then clang-system - that slows things down enough to make the failures trigger.

See: https://gitlab.com/mstredhat/qemu/-/jobs/3279537476

Alex, Viresh, need your help here. Thanks!

Alex, pls note that same failures are triggered by your RFC - if we know the root cause we can discuss solutions. So if you prefer pls go ahead and debug that. Thanks!

Michael S. Tsirkin (3):
virtio: distinguish between started and running
gpio: use virtio_device_running
virtio: revert changes to virtio_device_started

 include/hw/virtio/virtio.h   | 7 ++++++-
 hw/virtio/vhost-user-fs.c    | 2 +-
 hw/virtio/vhost-user-gpio.c  | 4 ++--
 hw/virtio/vhost-user-i2c.c   | 4 ++--
 hw/virtio/vhost-user-rng.c   | 4 ++--
 hw/virtio/vhost-user-vsock.c | 2 +-
 hw/virtio/vhost-vsock.c      | 2 +-
 7 files changed, 15 insertions(+), 10 deletions(-)
Signature · 1 line
MST
#1 lore raw

RFC PATCH 1/3 virtio: distinguish between started and running

virtio core often needs to know whether device is started, this is what virtio_device_started already did. However, backends want to know whether virtio is actually running which also depends on whether vm is running. To address this we moved the check to virtio_device_started, but this changes virtio core behavior which wasn't intentional. Let's add a new API virtio_device_running just for the backends. Follow up patch will revert the change to virtio_device_started.

Further, the old API was actually ignoring vm running state most of the time (when use_started property is set), The new API takes vm running state into account properly.

Fixes 9f6bcfd99f ("hw/virtio: move vm_running check to virtio_device_started")
Cc "Alex Bennée" <[email protected]>
---
 include/hw/virtio/virtio.h   | 9 +++++++++
 hw/virtio/vhost-user-fs.c    | 2 +-
 hw/virtio/vhost-user-i2c.c   | 4 ++--
 hw/virtio/vhost-user-rng.c   | 4 ++--
 hw/virtio/vhost-user-vsock.c | 2 +-
 hw/virtio/vhost-vsock.c      | 2 +-
 6 files changed, 16 insertions(+), 7 deletions(-)
include/hw/virtio/virtio.h +9 -0
index 1423dba379..634c24513f 100644
@@ -402,6 +402,15 @@ static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
     return status & VIRTIO_CONFIG_S_DRIVER_OK;
 }
 
+static inline bool virtio_device_running(VirtIODevice *vdev, uint8_t status)
+{
+    if (!vdev->vm_running) {
+        return false;
+    }
+
+    return virtio_device_started(vdev, status);
+}
+
 static inline void virtio_set_started(VirtIODevice *vdev, bool started)
 {
     if (started) {
hw/virtio/vhost-user-fs.c +1 -1
index ad0f91c607..174c968179 100644
@@ -123,7 +123,7 @@ static void vuf_stop(VirtIODevice *vdev)
 static void vuf_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostUserFS *fs = VHOST_USER_FS(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
 
     if (vhost_dev_is_started(&fs->vhost_dev) == should_start) {
         return;
hw/virtio/vhost-user-i2c.c +2 -2
index bc58b6c0d1..cf4fd26cf4 100644
@@ -93,7 +93,7 @@ static void vu_i2c_stop(VirtIODevice *vdev)
 static void vu_i2c_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostUserI2C *i2c = VHOST_USER_I2C(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
 
     if (vhost_dev_is_started(&i2c->vhost_dev) == should_start) {
         return;
@@ -157,7 +157,7 @@ static int vu_i2c_connect(DeviceState *dev)
     i2c->connected = true;
 
     /* restore vhost state */
-    if (virtio_device_started(vdev, vdev->status)) {
+    if (virtio_device_running(vdev, vdev->status)) {
         vu_i2c_start(vdev);
     }
 
hw/virtio/vhost-user-rng.c +2 -2
index bc1f36c5ac..ba548dc83c 100644
@@ -90,7 +90,7 @@ static void vu_rng_stop(VirtIODevice *vdev)
 static void vu_rng_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostUserRNG *rng = VHOST_USER_RNG(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
 
     if (vhost_dev_is_started(&rng->vhost_dev) == should_start) {
         return;
@@ -144,7 +144,7 @@ static void vu_rng_connect(DeviceState *dev)
     rng->connected = true;
 
     /* restore vhost state */
-    if (virtio_device_started(vdev, vdev->status)) {
+    if (virtio_device_running(vdev, vdev->status)) {
         vu_rng_start(vdev);
     }
 }
hw/virtio/vhost-user-vsock.c +1 -1
index 7b67e29d83..30228eaa21 100644
@@ -55,7 +55,7 @@ const VhostDevConfigOps vsock_ops = {
 static void vuv_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostVSockCommon *vvc = VHOST_VSOCK_COMMON(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
 
     if (vhost_dev_is_started(&vvc->vhost_dev) == should_start) {
         return;
hw/virtio/vhost-vsock.c +1 -1
index 7dc3c73931..1c768ee196 100644
@@ -70,7 +70,7 @@ static int vhost_vsock_set_running(VirtIODevice *vdev, int start)
 static void vhost_vsock_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostVSockCommon *vvc = VHOST_VSOCK_COMMON(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
     int ret;
 
     if (vhost_dev_is_started(&vvc->vhost_dev) == should_start) {
Signature · 1 line
MST
#2 lore raw

RFC PATCH 2/3 gpio: use virtio_device_running

In reply to Michael S. Tsirkin

Same as other vhost-user devices, vhost-user-gpio cares whether device is running not whether frontend is started. Switch to that.

Fixes 27ba7b027f ("hw/virtio: add boilerplate for vhost-user-gpio device")
Cc "Viresh Kumar" <[email protected]>
---
 hw/virtio/vhost-user-gpio.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-user-gpio.c +2 -2
index 8b40fe450c..f34ee59b6e 100644
@@ -152,7 +152,7 @@ static void vu_gpio_stop(VirtIODevice *vdev)
 static void vu_gpio_set_status(VirtIODevice *vdev, uint8_t status)
 {
     VHostUserGPIO *gpio = VHOST_USER_GPIO(vdev);
-    bool should_start = virtio_device_started(vdev, status);
+    bool should_start = virtio_device_running(vdev, status);
 
     trace_virtio_gpio_set_status(status);
 
@@ -228,7 +228,7 @@ static int vu_gpio_connect(DeviceState *dev, Error **errp)
     }
 
     /* restore vhost state */
-    if (virtio_device_started(vdev, vdev->status)) {
+    if (virtio_device_running(vdev, vdev->status)) {
         vu_gpio_start(vdev);
     }
 
Signature · 1 line
MST
#3 lore raw

RFC PATCH 3/3 virtio: revert changes to virtio_device_started

In reply to Michael S. Tsirkin

virtio core often needs to know whether device is started, this is what virtio_device_started already did. However, backends want to know whether virtio is actually running which also depends on whether vm is running. To address this we moved the check to virtio_device_started, but this changes virtio core behavior which wasn't intentional. Now that backends use the new virtio_device_running API, revert the change to virtio_device_started.

Fixes 9f6bcfd99f ("hw/virtio: move vm_running check to virtio_device_started")
Cc "Alex Bennée" <[email protected]>
---
 include/hw/virtio/virtio.h | 4 ----
 1 file changed, 4 deletions(-)
include/hw/virtio/virtio.h +0 -4
index 634c24513f..de8d78af12 100644
@@ -395,10 +395,6 @@ static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
         return vdev->started;
     }
 
-    if (!vdev->vm_running) {
-        return false;
-    }
-
     return status & VIRTIO_CONFIG_S_DRIVER_OK;
 }
 
Signature · 1 line
MST
#4 lore raw
In reply to Michael S. Tsirkin
Quoted 7 lines · Michael S. Tsirkin

This is an attempt to fix up device started checks. Unfortunately this causes failures in CI and I could not figure it out.

The simplest way to test is to set QEMU_CI to 2
on gitlab, then push there.

Quoted 7 lines

Alternatively, push to gitlab, then
create pipeline while setting QEMU_CI to 1, then run amd64-fedora-container and then clang-system - that slows things down enough to make the failures trigger.

See: https://gitlab.com/mstredhat/qemu/-/jobs/3279537476

And here is a backtrace:

――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
stderr:
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-PIIDV1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-PIIDV1/reconnect.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-U7IGV1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-U7IGV1/connect-fail.sock,server=on
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-U7IGV1/connect-fail.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-BUYEV1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-BUYEV1/flags-mismatch.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
vhost lacks feature mask 0x40000000 for backend
qemu-system-arm: failed to init vhost_net for queue 0
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==8747==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x0000000000fc (pc 0x55b8ada1276d bp 0x000000000007 sp 0x7ffd127cf5f0 T8747)
==8747==The signal is caused by a WRITE memory access.
==8747==Hint: address points to the zero page.

    #0 0x55b8ada1276d in virtio_bus_release_ioeventfd /builds/mstredhat/qemu/build/../hw/virtio/virtio-bus.c:216:30
    #1 0x55b8ade97b51 in vu_gpio_set_status /builds/mstredhat/qemu/build/../hw/virtio/vhost-user-gpio.c:172:9
    #2 0x55b8ade593f9 in virtio_set_status /builds/mstredhat/qemu/build/../hw/virtio/virtio.c:2442:9
    #3 0x55b8ada4d3d7 in vm_state_notify /builds/mstredhat/qemu/build/../softmmu/runstate.c:334:13
    #4 0x55b8ada4459a in do_vm_stop /builds/mstredhat/qemu/build/../softmmu/cpus.c:262:9
    #5 0x55b8ada4e2db in qemu_cleanup /builds/mstredhat/qemu/build/../softmmu/runstate.c:827:5
    #6 0x55b8ad6054fc in qemu_default_main /builds/mstredhat/qemu/build/../softmmu/main.c:38:5
    #7 0x7f3da8999eaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
    #8 0x7f3da8999f5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f)
    #9 0x55b8ad5dc094 in _start (/builds/mstredhat/qemu/build/qemu-system-arm+0xc17094)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV /builds/mstredhat/qemu/build/../hw/virtio/virtio-bus.c:216:30 in virtio_bus_release_ioeventfd
==8747==ABORTING
../tests/qtest/libqtest.c:179: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
**
ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8737]) failed unexpectedly (test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

https://gitlab.com/mstredhat/qemu/-/jobs/3279637541

gpio attempts to stop backend when notifiers are not enabled.

No clue how that triggers.

Quoted 24 lines

Alex, Viresh, need your help here. Thanks!

Alex, pls note that same failures are triggered by your RFC - if we know the root cause we can discuss solutions. So if you prefer pls go ahead and debug that. Thanks!

Michael S. Tsirkin (3):
virtio: distinguish between started and running
gpio: use virtio_device_running
virtio: revert changes to virtio_device_started

 include/hw/virtio/virtio.h   | 7 ++++++-
 hw/virtio/vhost-user-fs.c    | 2 +-
 hw/virtio/vhost-user-gpio.c  | 4 ++--
 hw/virtio/vhost-user-i2c.c   | 4 ++--
 hw/virtio/vhost-user-rng.c   | 4 ++--
 hw/virtio/vhost-user-vsock.c | 2 +-
 hw/virtio/vhost-vsock.c      | 2 +-
 7 files changed, 15 insertions(+), 10 deletions(-)
Signature · 1 line
MST