Skip to contentSkip to conversation

Fix scsi devices plug/unplug races w.r.t virtio-scsi iothread

PATCH Superseded Maxim Levitsky 2 participants 14 messages Jul 15, 2020 – Aug 13, 2020

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover Fix scsi devices plug/unplug races w.r.t virtio-scsi iothread

Hi!

This is a patch series that is a result of my discussion with Paulo on how to correctly fix the root cause of the BZ #1812399.

The root cause of this bug is the fact that IO thread is running mostly unlocked versus main thread on which device hotplug is done.

qdev_device_add first creates the device object, then places it on the bus, and only then realizes it.

However some drivers and currently only virtio-scsi enumerate its child bus devices on each request that is received from the guest and that can happen on the IO thread.

Thus we have a window when new device is on the bus but not realized and can be accessed by the virtio-scsi driver in that state.

Fix that by doing two things:

1. Add partial RCU protection to the list of a bus's child devices. This allows the scsi IO thread to safely enumerate the child devices while it races with the hotplug placing the device on the bus.

2. Make the virtio-scsi driver check .realized property of the scsi device and avoid touching the device if it isn't

Note that in the particular bug report the issue wasn't a race but rather due
to combination of things, the .realize code in the middle managed to trigger IO on the virtqueue which caused the virtio-scsi driver to access the half realized device. However since this can happen as well with real IO thread, this patch series was done, which fixes this as well.

Changes from V1:
* Patch 2 is new, as suggested by Stefan, added drain_call_rcu() to fix the failing unit test, make check pass now

* Patches 6,7 are new as well: I added scsi_device_get as suggested by Stefan as well, although this is more a refactoring that anything else as it doesn't solve
an existing race.

* Addressed most of the review feedback from V1
- still need to decide if we need QTAILQ_FOREACH_WITH_RCU_READ_LOCK

Changes from V2:

* No longer RFC
* Addressed most of the feedback from Stefan
* Fixed reference count leak in patch 7 when device is about to be unrealized
* Better testing

This series was tested by adding a virtio-scsi drive with iothread, then running fio stress job in the guest in a loop, and then adding/removing the scsi drive on the host in the loop.
This test was failing usually on 1st iteration withouth this patch series, and now it seems to work smoothly.

Best regards,
Maxim Levitsky

Maxim Levitsky (7):
scsi/scsi_bus: switch search direction in scsi_device_find Implement drain_call_rcu and use it in hmp_device_del
device-core: use RCU for list of childs of a bus
device-core: use atomic_set on .realized property
virtio-scsi: don't touch scsi devices that are not yet realized or
about to be un-realized
scsi: Add scsi_device_get
virtio-scsi: use scsi_device_get

 hw/core/bus.c          | 28 +++++++++++++--------
 hw/core/qdev.c         | 56 +++++++++++++++++++++++++++++++-----------
 hw/scsi/scsi-bus.c     | 48 +++++++++++++++++++++++++++++++-----
 hw/scsi/virtio-scsi.c  | 47 ++++++++++++++++++++++++++++-------
 include/hw/qdev-core.h | 11 +++++++++
 include/hw/scsi/scsi.h |  2 ++
 include/qemu/rcu.h     |  1 +
 qdev-monitor.c         | 22 +++++++++++++++++
 util/rcu.c             | 55 +++++++++++++++++++++++++++++++++++++++++
 9 files changed, 230 insertions(+), 40 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH v3 1/7 scsi/scsi_bus: switch search direction in scsi_device_find

This change will allow us to convert the bus children list to RCU, while not changing the logic of this function

---
 hw/scsi/scsi-bus.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
hw/scsi/scsi-bus.c +10 -2
index df65cc2223..f8adfbc2a5 100644
@@ -1572,7 +1572,7 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
     BusChild *kid;
     SCSIDevice *target_dev = NULL;
 
-    QTAILQ_FOREACH_REVERSE(kid, &bus->qbus.children, sibling) {
+    QTAILQ_FOREACH(kid, &bus->qbus.children, sibling) {
         DeviceState *qdev = kid->child;
         SCSIDevice *dev = SCSI_DEVICE(qdev);
 
@@ -1580,7 +1580,15 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
             if (dev->lun == lun) {
                 return dev;
             }
-            target_dev = dev;
+
+            /*
+             * If we don't find exact match (channel/bus/lun),
+             * we will return the first device which matches channel/bus
+             */
+
+            if (!target_dev) {
+                target_dev = dev;
+            }
         }
     }
     return target_dev;
Signature · 1 line
2.26.2
#2 lore raw

PATCH v3 2/7 Implement drain_call_rcu and use it in hmp_device_del

In reply to Maxim Levitsky

This allows to preserve the semantics of hmp_device_del, that the device is deleted immediatly which was changed by previos patch that delayed this to RCU callback

---
 include/qemu/rcu.h |  1 +
 qdev-monitor.c     | 22 +++++++++++++++++++
 util/rcu.c         | 55 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 78 insertions(+)
include/qemu/rcu.h +1 -0
index 570aa603eb..0e375ebe13 100644
@@ -133,6 +133,7 @@ struct rcu_head {
 };
 
 extern void call_rcu1(struct rcu_head *head, RCUCBFunc *func);
+extern void drain_call_rcu(void);
 
 /* The operands of the minus operator must have the same type,
  * which must be the one that we specify in the cast.
qdev-monitor.c +22 -0
index 71ebce19df..a889fbc509 100644
@@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, Error **errp)
         return;
     }
     dev = qdev_device_add(opts, errp);
+
+    /*
+     * Drain all pending RCU callbacks. This is done because
+     * some bus related operations can delay a device removal
+     * (in this case this can happen if device is added and then
+     * removed due to a configuration error)
+     * to a RCU callback, but user might expect that this interface
+     * will finish its job completely once qmp command returns result
+     * to the user
+     */
+    drain_call_rcu();
+
     if (!dev) {
         qemu_opts_del(opts);
         return;
@@ -894,6 +906,16 @@ void qmp_device_del(const char *id, Error **errp)
         }
 
         qdev_unplug(dev, errp);
+
+        /*
+         * Drain all pending RCU callbacks. This is done because
+         * some bus related operations can delay a device removal
+         * to a RCU callback, but user might expect that this interface
+         * will finish its job completely once qmp command returns result
+         * to the user
+         */
+
+        drain_call_rcu();
     }
 }
 
util/rcu.c +55 -0

61 lines not on this page · raw message

Signature · 1 line
2.26.2
#3 lore raw

PATCH v3 3/7 device-core: use RCU for list of childs of a bus

In reply to Maxim Levitsky

This fixes the race between device emulation code that tries to find a child device to dispatch the request to (e.g a scsi disk), and hotplug of a new device to that bus.

Note that this doesn't convert all the readers of the list but only these that might go over that list without BQL held.

This is a very small first step to make this code thread safe.

---
 hw/core/bus.c          | 28 ++++++++++++++++++----------
 hw/core/qdev.c         | 37 +++++++++++++++++++++++--------------
 hw/scsi/scsi-bus.c     | 17 ++++++++++++++---
 hw/scsi/virtio-scsi.c  |  6 +++++-
 include/hw/qdev-core.h |  9 +++++++++
 5 files changed, 69 insertions(+), 28 deletions(-)
hw/core/bus.c +18 -10
index 6b987b6946..385eb3ad5a 100644
@@ -49,12 +49,14 @@ int qbus_walk_children(BusState *bus,
         }
     }
 
-    QTAILQ_FOREACH(kid, &bus->children, sibling) {
-        err = qdev_walk_children(kid->child,
-                                 pre_devfn, pre_busfn,
-                                 post_devfn, post_busfn, opaque);
-        if (err < 0) {
-            return err;
+    WITH_RCU_READ_LOCK_GUARD() {
+        QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+            err = qdev_walk_children(kid->child,
+                                     pre_devfn, pre_busfn,
+                                     post_devfn, post_busfn, opaque);
+            if (err < 0) {
+                return err;
+            }
         }
     }
 
@@ -90,9 +92,13 @@ static void bus_reset_child_foreach(Object *obj, ResettableChildCallback cb,
     BusState *bus = BUS(obj);
     BusChild *kid;
 
-    QTAILQ_FOREACH(kid, &bus->children, sibling) {
+    rcu_read_lock();
+
+    QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
         cb(OBJECT(kid->child), opaque, type);
     }
+
+    rcu_read_unlock();
 }
 
 static void qbus_init(BusState *bus, DeviceState *parent, const char *name)
@@ -194,9 +200,11 @@ static void bus_set_realized(Object *obj, bool value, Error **errp)
 
         /* TODO: recursive realization */
     } else if (!value && bus->realized) {
-        QTAILQ_FOREACH(kid, &bus->children, sibling) {
-            DeviceState *dev = kid->child;
-            qdev_unrealize(dev);
+        WITH_RCU_READ_LOCK_GUARD() {
+            QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+                DeviceState *dev = kid->child;
+                qdev_unrealize(dev);
+            }
         }
         if (bc->unrealize) {
             bc->unrealize(bus);
hw/core/qdev.c +23 -14
index 01796823b4..ee69b11d11 100644
@@ -51,6 +51,12 @@ const VMStateDescription *qdev_get_vmsd(DeviceState *dev)
     return dc->vmsd;
 }
 
+static void bus_free_bus_child(BusChild *kid)
+{
+    object_unref(OBJECT(kid->child));
+    g_free(kid);
+}
+
 static void bus_remove_child(BusState *bus, DeviceState *child)
 {
     BusChild *kid;
@@ -60,15 +66,16 @@ static void bus_remove_child(BusState *bus, DeviceState *child)
             char name[32];
 
             snprintf(name, sizeof(name), "child[%d]", kid->index);
-            QTAILQ_REMOVE(&bus->children, kid, sibling);
+            QTAILQ_REMOVE_RCU(&bus->children, kid, sibling);
 
             bus->num_children--;
 
             /* This gives back ownership of kid->child back to us.  */
             object_property_del(OBJECT(bus), name);
-            object_unref(OBJECT(kid->child));
-            g_free(kid);
-            return;
+
+            /* free the bus kid, when it is safe to do so*/
+            call_rcu(kid, bus_free_bus_child, rcu);
+            break;
         }
     }
 }
@@ -83,7 +90,7 @@ static void bus_add_child(BusState *bus, DeviceState *child)
     kid->child = child;
     object_ref(OBJECT(kid->child));
 
-    QTAILQ_INSERT_HEAD(&bus->children, kid, sibling);
+    QTAILQ_INSERT_HEAD_RCU(&bus->children, kid, sibling);
 
     /* This transfers ownership of kid->child to the property.  */
     snprintf(name, sizeof(name), "child[%d]", kid->index);
@@ -689,17 +696,19 @@ DeviceState *qdev_find_recursive(BusState *bus, const char *id)
     DeviceState *ret;
     BusState *child;
 
-    QTAILQ_FOREACH(kid, &bus->children, sibling) {
-        DeviceState *dev = kid->child;
+    WITH_RCU_READ_LOCK_GUARD() {
+        QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+            DeviceState *dev = kid->child;
 
-        if (dev->id && strcmp(dev->id, id) == 0) {
-            return dev;
-        }
+            if (dev->id && strcmp(dev->id, id) == 0) {
+                return dev;
+            }
 
-        QLIST_FOREACH(child, &dev->child_bus, sibling) {
-            ret = qdev_find_recursive(child, id);
-            if (ret) {
-                return ret;
+            QLIST_FOREACH(child, &dev->child_bus, sibling) {
+                ret = qdev_find_recursive(child, id);
+                if (ret) {
+                    return ret;
+                }
             }
         }
     }
hw/scsi/scsi-bus.c +14 -3
index f8adfbc2a5..92d412b65c 100644
@@ -400,7 +400,10 @@ static bool scsi_target_emulate_report_luns(SCSITargetReq *r)
     id = r->req.dev->id;
     found_lun0 = false;
     n = 0;
-    QTAILQ_FOREACH(kid, &r->req.bus->qbus.children, sibling) {
+
+    rcu_read_lock();
+
+    QTAILQ_FOREACH_RCU(kid, &r->req.bus->qbus.children, sibling) {
         DeviceState *qdev = kid->child;
         SCSIDevice *dev = SCSI_DEVICE(qdev);
 
@@ -421,7 +424,7 @@ static bool scsi_target_emulate_report_luns(SCSITargetReq *r)
     memset(r->buf, 0, len);
     stl_be_p(&r->buf[0], n);
     i = found_lun0 ? 8 : 16;
-    QTAILQ_FOREACH(kid, &r->req.bus->qbus.children, sibling) {
+    QTAILQ_FOREACH_RCU(kid, &r->req.bus->qbus.children, sibling) {
         DeviceState *qdev = kid->child;
         SCSIDevice *dev = SCSI_DEVICE(qdev);
 
@@ -430,6 +433,9 @@ static bool scsi_target_emulate_report_luns(SCSITargetReq *r)
             i += 8;
         }
     }
+
+    rcu_read_unlock();
+
     assert(i == n + 8);
     r->len = len;
     return true;
@@ -1572,12 +1578,15 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
     BusChild *kid;
     SCSIDevice *target_dev = NULL;
 
-    QTAILQ_FOREACH(kid, &bus->qbus.children, sibling) {
+    rcu_read_lock();
+
+    QTAILQ_FOREACH_RCU(kid, &bus->qbus.children, sibling) {
         DeviceState *qdev = kid->child;
         SCSIDevice *dev = SCSI_DEVICE(qdev);
 
         if (dev->channel == channel && dev->id == id) {
             if (dev->lun == lun) {
+                rcu_read_unlock();
                 return dev;
             }
 
@@ -1591,6 +1600,8 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
             }
         }
     }
+
+    rcu_read_unlock();
     return target_dev;
 }
 
hw/scsi/virtio-scsi.c +5 -1
index b49775269e..d3cfc8dfb6 100644
@@ -367,12 +367,16 @@ static int virtio_scsi_do_tmf(VirtIOSCSI *s, VirtIOSCSIReq *req)
     case VIRTIO_SCSI_T_TMF_I_T_NEXUS_RESET:
         target = req->req.tmf.lun[1];
         s->resetting++;
-        QTAILQ_FOREACH(kid, &s->bus.qbus.children, sibling) {
+
+        rcu_read_lock();
+        QTAILQ_FOREACH_RCU(kid, &s->bus.qbus.children, sibling) {
              d = SCSI_DEVICE(kid->child);
              if (d->channel == 0 && d->id == target) {
                 qdev_reset_all(&d->qdev);
              }
         }
+        rcu_read_unlock();
+
         s->resetting--;
         break;
 
include/hw/qdev-core.h +9 -0
index fe78073c70..e252640c1a 100644
@@ -3,6 +3,8 @@
 
 #include "qemu/queue.h"
 #include "qemu/bitmap.h"
+#include "qemu/rcu.h"
+#include "qemu/rcu_queue.h"
 #include "qom/object.h"
 #include "hw/hotplug.h"
 #include "hw/resettable.h"
@@ -230,6 +232,7 @@ struct BusClass {
 };
 
 typedef struct BusChild {
+    struct rcu_head rcu;
     DeviceState *child;
     int index;
     QTAILQ_ENTRY(BusChild) sibling;
@@ -250,6 +253,12 @@ struct BusState {
     int max_index;
     bool realized;
     int num_children;
+
+    /*
+     * children is a RCU QTAILQ, thus readers must use RCU to access it,
+     * and writers must hold the big qemu lock
+     */
+
     QTAILQ_HEAD(, BusChild) children;
     QLIST_ENTRY(BusState) sibling;
     ResettableState reset;
Signature · 1 line
2.26.2
Reviewed-by on v3 3/7
#4 lore raw

Re: [PATCH v3 3/7] device-core: use RCU for list of childs of a bus

Quoted 11 lines · Maxim Levitsky

This fixes the race between device emulation code that tries to find a child device to dispatch the request to (e.g a scsi disk), and hotplug of a new device to that bus.

Note that this doesn't convert all the readers of the list but only these that might go over that list without BQL held.

This is a very small first step to make this code thread safe.

Quoted 7 lines
---
 hw/core/bus.c          | 28 ++++++++++++++++++----------
 hw/core/qdev.c         | 37 +++++++++++++++++++++++--------------
 hw/scsi/scsi-bus.c     | 17 ++++++++++++++---
 hw/scsi/virtio-scsi.c  |  6 +++++-
 include/hw/qdev-core.h |  9 +++++++++
 5 files changed, 69 insertions(+), 28 deletions(-)
#5 lore raw

PATCH v3 4/7 device-core: use atomic_set on .realized property

In reply to Maxim Levitsky

Some code might race with placement of new devices on a bus. We currently first place a (unrealized) device on the bus and then realize it.

As a workaround, users that scan the child device list, can check the realized property to see if it is safe to access such a device. Use an atomic write here too to aid with this.

A separate discussion is what to do with devices that are unrealized:
It looks like for this case we only call the hotplug handler's unplug callback and its up to it to unrealize the device.
An atomic operation doesn't cause harm for this code path though.

---
 hw/core/qdev.c         | 19 ++++++++++++++++++-
 include/hw/qdev-core.h |  2 ++
 2 files changed, 20 insertions(+), 1 deletion(-)
hw/core/qdev.c +18 -1
index ee69b11d11..2ec4971aec 100644
@@ -966,7 +966,25 @@ static void device_set_realized(Object *obj, bool value, Error **errp)
             }
        }
 
+       atomic_store_release(&dev->realized, value);
+
     } else if (!value && dev->realized) {
+
+        /*
+         * Change the value so that any concurrent users are aware
+         * that the device is going to be unrealized
+         *
+         * TODO: change .realized property to enum that states
+         * each phase of the device realization/unrealization
+         */
+
+        atomic_set(&dev->realized, value);
+        /*
+         * execute full memory barrier to ensure that concurrent users
+         * see this update prior to any other changes to the device
+         */
+        smp_mb();
+
         QLIST_FOREACH(bus, &dev->child_bus, sibling) {
             qbus_unrealize(bus);
         }
@@ -981,7 +999,6 @@ static void device_set_realized(Object *obj, bool value, Error **errp)
     }
 
     assert(local_err == NULL);
-    dev->realized = value;
     return;
 
 child_realize_fail:
include/hw/qdev-core.h +2 -0
index e252640c1a..5faa715449 100644
@@ -165,6 +165,8 @@ struct NamedClockList {
 /**
  * DeviceState:
  * @realized: Indicates whether the device has been fully constructed.
+ *            When accessed outsize big qemu lock, must be accessed with
+ *            atomic_load_acquire()
  * @reset: ResettableState for the device; handled by Resettable interface.
  *
  * This structure should not be accessed directly.  We declare it here
Signature · 1 line
2.26.2
Reviewed-by on v3 4/7
#6 lore raw

Re: [PATCH v3 4/7] device-core: use atomic_set on .realized property

Quoted 13 lines · Maxim Levitsky

Some code might race with placement of new devices on a bus. We currently first place a (unrealized) device on the bus and then realize it.

As a workaround, users that scan the child device list, can check the realized property to see if it is safe to access such a device. Use an atomic write here too to aid with this.

A separate discussion is what to do with devices that are unrealized:
It looks like for this case we only call the hotplug handler's unplug callback and its up to it to unrealize the device.
An atomic operation doesn't cause harm for this code path though.

Quoted 5 lines
---
 hw/core/qdev.c         | 19 ++++++++++++++++++-
 include/hw/qdev-core.h |  2 ++
 2 files changed, 20 insertions(+), 1 deletion(-)
#7 lore raw

PATCH v3 5/7 virtio-scsi: don't touch scsi devices that are not yet realized or about to be un-realized

In reply to Maxim Levitsky

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1812399

---
 hw/scsi/virtio-scsi.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
hw/scsi/virtio-scsi.c +18 -1
index d3cfc8dfb6..18952e0f1c 100644
@@ -35,13 +35,30 @@ static inline int virtio_scsi_get_lun(uint8_t *lun)
 
 static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI *s, uint8_t *lun)
 {
+    SCSIDevice *device = NULL;
+
     if (lun[0] != 1) {
         return NULL;
     }
     if (lun[2] != 0 && !(lun[2] >= 0x40 && lun[2] < 0x80)) {
         return NULL;
     }
-    return scsi_device_find(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
+
+    device = scsi_device_find(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
+
+    /*
+     * This function might run on the IO thread and we might race against
+     * main thread hot-plugging the device.
+     *
+     * We assume that as soon as .realized is set to true we can let
+     * the user access the device.
+     */
+
+    if (!device || !atomic_load_acquire(&device->qdev.realized)) {
+        return NULL;
+    }
+
+    return device;
 }
 
 void virtio_scsi_init_req(VirtIOSCSI *s, VirtQueue *vq, VirtIOSCSIReq *req)
Signature · 1 line
2.26.2
#8 lore raw

PATCH v3 6/7 scsi: Add scsi_device_get

In reply to Maxim Levitsky

Add scsi_device_get which finds the scsi device
and takes a reference to it.

---
 hw/scsi/scsi-bus.c     | 31 ++++++++++++++++++++++++-------
 include/hw/scsi/scsi.h |  2 ++
 2 files changed, 26 insertions(+), 7 deletions(-)
hw/scsi/scsi-bus.c +24 -7
index 92d412b65c..677074eaa6 100644
@@ -1572,8 +1572,13 @@ static char *scsibus_get_fw_dev_path(DeviceState *dev)
     return g_strdup_printf("channel@%x/%s@%x,%x", d->channel,
                            qdev_fw_name(dev), d->id, d->lun);
 }
-
-SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
+/*
+ * Finds a matching channel/id/lun device on scsi bus, and
+ * takes a reference to it and returns it.
+ * If we don't find exact match (channel/bus/lun),
+ * we will return the first device which matches channel/bus
+ */
+SCSIDevice *scsi_device_get(SCSIBus *bus, int channel, int id, int lun)
 {
     BusChild *kid;
     SCSIDevice *target_dev = NULL;
@@ -1586,25 +1591,37 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
 
         if (dev->channel == channel && dev->id == id) {
             if (dev->lun == lun) {
+                object_ref(OBJECT(dev));
                 rcu_read_unlock();
                 return dev;
             }
 
-            /*
-             * If we don't find exact match (channel/bus/lun),
-             * we will return the first device which matches channel/bus
-             */
-
             if (!target_dev) {
                 target_dev = dev;
             }
         }
     }
 
+    object_ref(OBJECT(target_dev));
     rcu_read_unlock();
     return target_dev;
 }
 
+/*
+ * This function works like scsi_device_get but doesn't take a reference
+ * to the returned object.
+ * Devices that run under the QEMU global mutex can use this function.
+ * Devices that run outside the QEMU global mutex must use
+ * scsi_device_get() instead.
+ */
+SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
+{
+    SCSIDevice *dev = scsi_device_get(bus, channel, id, lun);
+
+    object_unref(OBJECT(dev));
+    return dev;
+}
+
 /* SCSI request list.  For simplicity, pv points to the whole device */
 
 static int put_scsi_requests(QEMUFile *f, void *pv, size_t size,
include/hw/scsi/scsi.h +2 -0
index 2fc23e44ba..6e11ece641 100644
@@ -193,7 +193,9 @@ void scsi_generic_read_device_inquiry(SCSIDevice *dev);
 int scsi_device_get_sense(SCSIDevice *dev, uint8_t *buf, int len, bool fixed);
 int scsi_SG_IO_FROM_DEV(BlockBackend *blk, uint8_t *cmd, uint8_t cmd_size,
                         uint8_t *buf, uint8_t buf_size);
+
 SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int target, int lun);
+SCSIDevice *scsi_device_get(SCSIBus *bus, int channel, int target, int lun);
 
 /* scsi-generic.c. */
 extern const SCSIReqOps scsi_generic_req_ops;
Signature · 1 line
2.26.2
Reviewed-by on v3 6/7
#9 lore raw

Re: [PATCH v3 6/7] scsi: Add scsi_device_get

Quoted 9 lines · Maxim Levitsky

Add scsi_device_get which finds the scsi device
and takes a reference to it.

---
 hw/scsi/scsi-bus.c     | 31 ++++++++++++++++++++++++-------
 include/hw/scsi/scsi.h |  2 ++
 2 files changed, 26 insertions(+), 7 deletions(-)
#10 lore raw

PATCH v3 7/7 virtio-scsi: use scsi_device_get

In reply to Maxim Levitsky

This will help us to avoid the scsi device disappearing after we took a reference to it.

It doesn't by itself forbid case when we try to access an unrealized device

---
 hw/scsi/virtio-scsi.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
hw/scsi/virtio-scsi.c +16 -8
index 18952e0f1c..a4870b155d 100644
@@ -33,7 +33,7 @@ static inline int virtio_scsi_get_lun(uint8_t *lun)
     return ((lun[2] << 8) | lun[3]) & 0x3FFF;
 }
 
-static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI *s, uint8_t *lun)
+static inline SCSIDevice *virtio_scsi_device_get(VirtIOSCSI *s, uint8_t *lun)
 {
     SCSIDevice *device = NULL;
 
@@ -44,7 +44,7 @@ static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI *s, uint8_t *lun)
         return NULL;
     }
 
-    device = scsi_device_find(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
+    device = scsi_device_get(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
 
     /*
      * This function might run on the IO thread and we might race against
@@ -55,12 +55,15 @@ static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI *s, uint8_t *lun)
      */
 
     if (!device || !atomic_load_acquire(&device->qdev.realized)) {
+        object_unref(OBJECT(device));
         return NULL;
     }
 
     return device;
 }
 
+
+
 void virtio_scsi_init_req(VirtIOSCSI *s, VirtQueue *vq, VirtIOSCSIReq *req)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(s);
@@ -273,7 +276,7 @@ static inline void virtio_scsi_ctx_check(VirtIOSCSI *s, SCSIDevice *d)
  *  case of async cancellation. */
 static int virtio_scsi_do_tmf(VirtIOSCSI *s, VirtIOSCSIReq *req)
 {
-    SCSIDevice *d = virtio_scsi_device_find(s, req->req.tmf.lun);
+    SCSIDevice *d = virtio_scsi_device_get(s, req->req.tmf.lun);
     SCSIRequest *r, *next;
     BusChild *kid;
     int target;
@@ -387,10 +390,10 @@ static int virtio_scsi_do_tmf(VirtIOSCSI *s, VirtIOSCSIReq *req)
 
         rcu_read_lock();
         QTAILQ_FOREACH_RCU(kid, &s->bus.qbus.children, sibling) {
-             d = SCSI_DEVICE(kid->child);
-             if (d->channel == 0 && d->id == target) {
-                qdev_reset_all(&d->qdev);
-             }
+            SCSIDevice *d1 = SCSI_DEVICE(kid->child);
+            if (d1->channel == 0 && d1->id == target) {
+                qdev_reset_all(&d1->qdev);
+            }
         }
         rcu_read_unlock();
 
@@ -403,14 +406,17 @@ static int virtio_scsi_do_tmf(VirtIOSCSI *s, VirtIOSCSIReq *req)
         break;
     }
 
+    object_unref(OBJECT(d));
     return ret;
 
 incorrect_lun:
     req->resp.tmf.response = VIRTIO_SCSI_S_INCORRECT_LUN;
+    object_unref(OBJECT(d));
     return ret;
 
 fail:
     req->resp.tmf.response = VIRTIO_SCSI_S_BAD_TARGET;
+    object_unref(OBJECT(d));
     return ret;
 }
 
@@ -581,7 +587,7 @@ static int virtio_scsi_handle_cmd_req_prepare(VirtIOSCSI *s, VirtIOSCSIReq *req)
         }
     }
 
-    d = virtio_scsi_device_find(s, req->req.cmd.lun);
+    d = virtio_scsi_device_get(s, req->req.cmd.lun);
     if (!d) {
         req->resp.cmd.response = VIRTIO_SCSI_S_BAD_TARGET;
         virtio_scsi_complete_cmd_req(req);
@@ -597,10 +603,12 @@ static int virtio_scsi_handle_cmd_req_prepare(VirtIOSCSI *s, VirtIOSCSIReq *req)
             req->sreq->cmd.xfer > req->qsgl.size)) {
         req->resp.cmd.response = VIRTIO_SCSI_S_OVERRUN;
         virtio_scsi_complete_cmd_req(req);
+        object_unref(OBJECT(d));
         return -ENOBUFS;
     }
     scsi_req_ref(req->sreq);
     blk_io_plug(d->conf.blk);
+    object_unref(OBJECT(d));
     return 0;
 }
 
Signature · 1 line
2.26.2
#11 lore raw
In reply to Maxim Levitsky
Quoted 70 lines · Maxim Levitsky

Hi!

This is a patch series that is a result of my discussion with Paulo on how to correctly fix the root cause of the BZ #1812399.

The root cause of this bug is the fact that IO thread is running mostly unlocked versus main thread on which device hotplug is done.

qdev_device_add first creates the device object, then places it on the bus, and only then realizes it.

However some drivers and currently only virtio-scsi enumerate its child bus devices on each request that is received from the guest and that can happen on the IO thread.

Thus we have a window when new device is on the bus but not realized and can be accessed by the virtio-scsi driver in that state.

Fix that by doing two things:

1. Add partial RCU protection to the list of a bus's child devices. This allows the scsi IO thread to safely enumerate the child devices while it races with the hotplug placing the device on the bus.

2. Make the virtio-scsi driver check .realized property of the scsi device and avoid touching the device if it isn't

Note that in the particular bug report the issue wasn't a race but rather due
to combination of things, the .realize code in the middle managed to trigger IO on the virtqueue which caused the virtio-scsi driver to access the half realized device. However since this can happen as well with real IO thread, this patch series was done, which fixes this as well.

Changes from V1:
* Patch 2 is new, as suggested by Stefan, added drain_call_rcu() to fix the failing unit test, make check pass now

* Patches 6,7 are new as well: I added scsi_device_get as suggested by Stefan as well, although this is more a refactoring that anything else as it doesn't solve
an existing race.

* Addressed most of the review feedback from V1
- still need to decide if we need QTAILQ_FOREACH_WITH_RCU_READ_LOCK

Changes from V2:

* No longer RFC
* Addressed most of the feedback from Stefan
* Fixed reference count leak in patch 7 when device is about to be unrealized
* Better testing

This series was tested by adding a virtio-scsi drive with iothread, then running fio stress job in the guest in a loop, and then adding/removing the scsi drive on the host in the loop.
This test was failing usually on 1st iteration withouth this patch series, and now it seems to work smoothly.

Best regards,
Maxim Levitsky

Maxim Levitsky (7):
scsi/scsi_bus: switch search direction in scsi_device_find Implement drain_call_rcu and use it in hmp_device_del
device-core: use RCU for list of childs of a bus
device-core: use atomic_set on .realized property
virtio-scsi: don't touch scsi devices that are not yet realized or
about to be un-realized
scsi: Add scsi_device_get
virtio-scsi: use scsi_device_get

Quoted 14 lines
 hw/core/bus.c          | 28 +++++++++++++--------
 hw/core/qdev.c         | 56 +++++++++++++++++++++++++++++++-----------
 hw/scsi/scsi-bus.c     | 48 +++++++++++++++++++++++++++++++-----
 hw/scsi/virtio-scsi.c  | 47 ++++++++++++++++++++++++++++-------
 include/hw/qdev-core.h | 11 +++++++++
 include/hw/scsi/scsi.h |  2 ++
 include/qemu/rcu.h     |  1 +
 qdev-monitor.c         | 22 +++++++++++++++++
 util/rcu.c             | 55 +++++++++++++++++++++++++++++++++++++++++
 9 files changed, 230 insertions(+), 40 deletions(-)
Signature · 1 line
2.26.2

Very gentle ping about this patch series.

Best regards,
Maxim Levitsky

#13 lore raw
In reply to Maxim Levitsky
Quoted 70 lines · Maxim Levitsky

Hi!

This is a patch series that is a result of my discussion with Paulo on how to correctly fix the root cause of the BZ #1812399.

The root cause of this bug is the fact that IO thread is running mostly unlocked versus main thread on which device hotplug is done.

qdev_device_add first creates the device object, then places it on the bus, and only then realizes it.

However some drivers and currently only virtio-scsi enumerate its child bus devices on each request that is received from the guest and that can happen on the IO thread.

Thus we have a window when new device is on the bus but not realized and can be accessed by the virtio-scsi driver in that state.

Fix that by doing two things:

1. Add partial RCU protection to the list of a bus's child devices. This allows the scsi IO thread to safely enumerate the child devices while it races with the hotplug placing the device on the bus.

2. Make the virtio-scsi driver check .realized property of the scsi device and avoid touching the device if it isn't

Note that in the particular bug report the issue wasn't a race but rather due
to combination of things, the .realize code in the middle managed to trigger IO on the virtqueue which caused the virtio-scsi driver to access the half realized device. However since this can happen as well with real IO thread, this patch series was done, which fixes this as well.

Changes from V1:
* Patch 2 is new, as suggested by Stefan, added drain_call_rcu() to fix the failing unit test, make check pass now

* Patches 6,7 are new as well: I added scsi_device_get as suggested by Stefan as well, although this is more a refactoring that anything else as it doesn't solve
an existing race.

* Addressed most of the review feedback from V1
- still need to decide if we need QTAILQ_FOREACH_WITH_RCU_READ_LOCK

Changes from V2:

* No longer RFC
* Addressed most of the feedback from Stefan
* Fixed reference count leak in patch 7 when device is about to be unrealized
* Better testing

This series was tested by adding a virtio-scsi drive with iothread, then running fio stress job in the guest in a loop, and then adding/removing the scsi drive on the host in the loop.
This test was failing usually on 1st iteration withouth this patch series, and now it seems to work smoothly.

Best regards,
Maxim Levitsky

Maxim Levitsky (7):
scsi/scsi_bus: switch search direction in scsi_device_find Implement drain_call_rcu and use it in hmp_device_del
device-core: use RCU for list of childs of a bus
device-core: use atomic_set on .realized property
virtio-scsi: don't touch scsi devices that are not yet realized or
about to be un-realized
scsi: Add scsi_device_get
virtio-scsi: use scsi_device_get

Quoted 14 lines
 hw/core/bus.c          | 28 +++++++++++++--------
 hw/core/qdev.c         | 56 +++++++++++++++++++++++++++++++-----------
 hw/scsi/scsi-bus.c     | 48 +++++++++++++++++++++++++++++++-----
 hw/scsi/virtio-scsi.c  | 47 ++++++++++++++++++++++++++++-------
 include/hw/qdev-core.h | 11 +++++++++
 include/hw/scsi/scsi.h |  2 ++
 include/qemu/rcu.h     |  1 +
 qdev-monitor.c         | 22 +++++++++++++++++
 util/rcu.c             | 55 +++++++++++++++++++++++++++++++++++++++++
 9 files changed, 230 insertions(+), 40 deletions(-)
Signature · 1 line
2.26.2

Seems that this patch series doesn't have any disagreeements, so anybody wants to put it on an maintainer's tree, now that the freeze is over?

Best regards,
Maxim Levitsky