Skip to contentSkip to conversation

xen: fix a potential crash in xen-bus

PATCH Superseded Paul Durrant 2 participants 7 messages Sep 12, 2019 – Sep 13, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover xen: fix a potential crash in xen-bus

This series fixes a potential segfault caused by NotifierList corruption in xen-bus. The first two patches lay the groundwork and the third actually fixes the problem.

Paul Durrant (3):
xen / notify: introduce a new XenWatchList abstraction
xen: introduce separate XenWatchList for XenDevice objects
xen: perform XenDevice clean-up in XenBus watch handler

 hw/xen/trace-events      |   9 +-
 hw/xen/xen-bus.c         | 267 ++++++++++++++++++++++++++++-----------
 include/hw/xen/xen-bus.h |   7 +-
 include/qemu/notify.h    |   2 +
 util/notify.c            |   5 +
 5 files changed, 213 insertions(+), 77 deletions(-)

---
Cc: Anthony Perard <[email protected]>
Cc: Stefano Stabellini <[email protected]>

Signature · 7 lines
2.20.1.2.gb21ebb6


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v2 1/3 xen / notify: introduce a new XenWatchList abstraction

Xenstore watch call-backs are already abstracted away from XenBus using the XenWatch data structure but the associated NotifierList manipulation and file handle registration is still open coded in various xen_bus_...() functions.
This patch creates a new XenWatchList data structure to allow these interactions to be abstracted away from XenBus as well. This is in preparation for a subsequent patch which will introduce separate watch lists for XenBus and XenDevice objects.

NOTE: This patch also introduces a new notifier_list_empty() helper function
for the purposes of adding an assertion that a XenWatchList is not freed whilst its associated NotifierList is still occupied.

---
Cc: Stefano Stabellini <[email protected]>

---
 hw/xen/trace-events      |   5 +-
 hw/xen/xen-bus.c         | 117 +++++++++++++++++++++++++--------------
 include/hw/xen/xen-bus.h |   3 +-
 include/qemu/notify.h    |   2 +
 util/notify.c            |   5 ++
 5 files changed, 87 insertions(+), 45 deletions(-)
hw/xen/trace-events +2 -3
index bc82ecb1a5..ac8d9c20d2 100644
@@ -19,9 +19,8 @@ xen_bus_unrealize(void) ""
 xen_bus_enumerate(void) ""
 xen_bus_type_enumerate(const char *type) "type: %s"
 xen_bus_backend_create(const char *type, const char *path) "type: %s path: %s"
-xen_bus_add_watch(const char *node, const char *key, char *token) "node: %s key: %s token: %s"
-xen_bus_remove_watch(const char *node, const char *key, char *token) "node: %s key: %s token: %s"
-xen_bus_watch(const char *token) "token: %s"
+xen_bus_add_watch(const char *node, const char *key) "node: %s key: %s"
+xen_bus_remove_watch(const char *node, const char *key) "node: %s key: %s"
 xen_device_realize(const char *type, char *name) "type: %s name: %s"
 xen_device_unrealize(const char *type, char *name) "type: %s name: %s"
 xen_device_backend_state(const char *type, char *name, const char *state) "type: %s name: %s -> %s"
hw/xen/xen-bus.c +76 -41

156 lines not on this page · raw message

include/hw/xen/xen-bus.h +2 -1
index 1c2d9dfdb8..88b84e29bb 100644
@@ -14,6 +14,7 @@
 
 typedef void (*XenWatchHandler)(void *opaque);
 
+typedef struct XenWatchList XenWatchList;
 typedef struct XenWatch XenWatch;
 typedef struct XenEventChannel XenEventChannel;
 
@@ -63,7 +64,7 @@ typedef struct XenBus {
     BusState qbus;
     domid_t backend_id;
     struct xs_handle *xsh;
-    NotifierList watch_notifiers;
+    XenWatchList *watch_list;
     XenWatch *backend_watch;
 } XenBus;
 
include/qemu/notify.h +2 -0
index a3d73e4bc7..bcfa70fb2e 100644
@@ -40,6 +40,8 @@ void notifier_remove(Notifier *notifier);
 
 void notifier_list_notify(NotifierList *list, void *data);
 
+bool notifier_list_empty(NotifierList *list);
+
 /* Same as Notifier but allows .notify() to return errors */
 typedef struct NotifierWithReturn NotifierWithReturn;
 
util/notify.c +5 -0
index aee8d93cb0..76bab212ae 100644
@@ -40,6 +40,11 @@ void notifier_list_notify(NotifierList *list, void *data)
     }
 }
 
+bool notifier_list_empty(NotifierList *list)
+{
+    return QLIST_EMPTY(&list->notifiers);
+}
+
 void notifier_with_return_list_init(NotifierWithReturnList *list)
 {
     QLIST_INIT(&list->notifiers);
Signature · 7 lines
2.20.1.2.gb21ebb6


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH v2 2/3 xen: introduce separate XenWatchList for XenDevice objects

In reply to Paul Durrant

This patch uses the XenWatchList abstraction to add a separate watch list for each device. This is more scalable than walking a single notifier list for all watches and is also necessary to implement a bug-fix in a subsequent patch.

---
Cc: Stefano Stabellini <[email protected]>

---
 hw/xen/trace-events      |  2 ++
 hw/xen/xen-bus.c         | 72 ++++++++++++++++++++++++++++++++--------
 include/hw/xen/xen-bus.h |  2 ++
 3 files changed, 62 insertions(+), 14 deletions(-)
hw/xen/trace-events +2 -0
index ac8d9c20d2..80ce3dafad 100644
@@ -29,6 +29,8 @@ xen_device_backend_changed(const char *type, char *name) "type: %s name: %s"
 xen_device_frontend_state(const char *type, char *name, const char *state) "type: %s name: %s -> %s"
 xen_device_frontend_changed(const char *type, char *name) "type: %s name: %s"
 xen_device_unplug(const char *type, char *name) "type: %s name: %s"
+xen_device_add_watch(const char *type, char *name, const char *node, const char *key) "type: %s name: %s node: %s key: %s"
+xen_device_remove_watch(const char *type, char *name, const char *node, const char *key) "type: %s name: %s node: %s key: %s"
 
 # xen-bus-helper.c
 xs_node_create(const char *node) "%s"
hw/xen/xen-bus.c +58 -14

139 lines not on this page · raw message

include/hw/xen/xen-bus.h +2 -0
index 88b84e29bb..0d198148f6 100644
@@ -22,6 +22,8 @@ typedef struct XenDevice {
     DeviceState qdev;
     domid_t frontend_id;
     char *name;
+    struct xs_handle *xsh;
+    XenWatchList *watch_list;
     char *backend_path, *frontend_path;
     enum xenbus_state backend_state, frontend_state;
     Notifier exit;
Signature · 7 lines
2.20.1.2.gb21ebb6


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw

PATCH v2 3/3 xen: perform XenDevice clean-up in XenBus watch handler

In reply to Paul Durrant

Cleaning up offine XenDevice objects directly in xen_device_backend_changed() is dangerous as xen_device_unrealize() will modify the watch list that is being walked. Even the QLIST_FOREACH_SAFE() used in notifier_list_notify() is insufficient as *two* notifiers (for the frontend and backend watches) are removed, thus potentially rendering the 'next' pointer unsafe.

The solution is to use the XenBus backend_watch handler to do the clean-up instead, as it is invoked whilst walking a separate watch list.

This patch therefore adds a new 'offline_devices' list to XenBus, to which offline devices are added by xen_device_backend_changed(). The XenBus backend_watch registration is also changed to not only invoke xen_bus_enumerate() but also a new xen_bus_cleanup() function, which will walk 'offline_devices' and perform the necessary actions.
For safety a an extra 'online' check is also added to xen_bus_type_enumerate() to make sure that no attempt is made to create a new XenDevice object for a backend that is offline.

NOTE: This patch also include some cosmetic changes:
- substitute the local variable name 'backend_state'
in xen_bus_type_enumerate() with 'state', since there is no ambiguity with any other state in that context.
- change xen_device_state_is_active() to xen_device_frontend_is_active() (and pass a XenDevice directly) since the state tests contained therein only apply to a frontend.
- use 'state' rather then 'xendev->backend_state' in xen_device_backend_changed() to shorten the code.

---
Cc: Stefano Stabellini <[email protected]>
Cc: Anthony Perard <[email protected]>

v2:
- Make sure we don't try to add a XenDevice to 'offline_devices' more than once

---
 hw/xen/trace-events      |  2 +
 hw/xen/xen-bus.c         | 92 +++++++++++++++++++++++++++++-----------
 include/hw/xen/xen-bus.h |  2 +
 3 files changed, 71 insertions(+), 25 deletions(-)
hw/xen/trace-events +2 -0
index 80ce3dafad..e6885bc751 100644
@@ -17,8 +17,10 @@ xen_domid_restrict(int err) "err: %u"
 xen_bus_realize(void) ""
 xen_bus_unrealize(void) ""
 xen_bus_enumerate(void) ""
+xen_bus_cleanup(void) ""
 xen_bus_type_enumerate(const char *type) "type: %s"
 xen_bus_backend_create(const char *type, const char *path) "type: %s path: %s"
+xen_bus_device_cleanup(const char *type, char *name) "type: %s name: %s"
 xen_bus_add_watch(const char *node, const char *key) "node: %s key: %s"
 xen_bus_remove_watch(const char *node, const char *key) "node: %s key: %s"
 xen_device_realize(const char *type, char *name) "type: %s name: %s"
hw/xen/xen-bus.c +67 -25

141 lines not on this page · raw message

include/hw/xen/xen-bus.h +2 -0
index 0d198148f6..15d71aff70 100644
@@ -33,6 +33,7 @@ typedef struct XenDevice {
     xengnttab_handle *xgth;
     bool feature_grant_copy;
     QLIST_HEAD(, XenEventChannel) event_channels;
+    QLIST_ENTRY(XenDevice) list;
 } XenDevice;
 
 typedef char *(*XenDeviceGetName)(XenDevice *xendev, Error **errp);
@@ -68,6 +69,7 @@ typedef struct XenBus {
     struct xs_handle *xsh;
     XenWatchList *watch_list;
     XenWatch *backend_watch;
+    QLIST_HEAD(, XenDevice) offline_devices;
 } XenBus;
 
 typedef struct XenBusClass {
Signature · 7 lines
2.20.1.2.gb21ebb6


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

Re: [Qemu-devel] [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 1 line · Paul Durrant

Cleaning up offine XenDevice objects directly in

^ offline

Quoted 9 lines

xen_device_backend_changed() is dangerous as xen_device_unrealize() will modify the watch list that is being walked. Even the QLIST_FOREACH_SAFE() used in notifier_list_notify() is insufficient as *two* notifiers (for the frontend and backend watches) are removed, thus potentially rendering the 'next' pointer unsafe.

The solution is to use the XenBus backend_watch handler to do the clean-up instead, as it is invoked whilst walking a separate watch list.

Quoted 6 lines

This patch therefore adds a new 'offline_devices' list to XenBus, to which offline devices are added by xen_device_backend_changed(). The XenBus backend_watch registration is also changed to not only invoke xen_bus_enumerate() but also a new xen_bus_cleanup() function, which will walk 'offline_devices' and perform the necessary actions.
For safety a an extra 'online' check is also added to

^ one 'a' too many?

Quoted 29 lines

xen_bus_type_enumerate() to make sure that no attempt is made to create a new XenDevice object for a backend that is offline.

NOTE: This patch also include some cosmetic changes:
- substitute the local variable name 'backend_state'
in xen_bus_type_enumerate() with 'state', since there is no ambiguity with any other state in that context.
- change xen_device_state_is_active() to xen_device_frontend_is_active() (and pass a XenDevice directly) since the state tests contained therein only apply to a frontend.
- use 'state' rather then 'xendev->backend_state' in xen_device_backend_changed() to shorten the code.

---

v2:
- Make sure we don't try to add a XenDevice to 'offline_devices' more than once
---

/*
* If a backend is still 'online' then we should leave it alone but,

hw/xen/xen-bus.c +3 -2
-     * if a backend is not 'online', then the device should be destroyed
-     * once the state is Closed.
+     * if a backend is not 'online', then the device is a candidate
+     * for destruction. Hence add it to the 'offline' list to be cleaned
+     * by xen_bus_cleanup().
      */

hw/xen/xen-bus.c static void xen_device_backend_changed(void *opaque) · line 653

Quoted 10 lines
hw/xen/xen-bus.c +4 -6
-    if (!xendev->backend_online &&
-        (xendev->backend_state == XenbusStateClosed ||
-         xendev->backend_state == XenbusStateInitialising ||
-         xendev->backend_state == XenbusStateInitWait ||
-         xendev->backend_state == XenbusStateUnknown)) {
-        Error *local_err = NULL;
+    if (!online &&
+        (state == XenbusStateClosed ||  state == XenbusStateInitialising ||
+         state == XenbusStateInitWait || state == XenbusStateUnknown) &&
+        !QLIST_NEXT(xendev, list)) {

Could you add a note about this QLIST_NEXT? I don't think it's going to be obvious enough why we check that there are no next item. I might only understand it just because of your reply to the v1 of the patch. (Well the changelog of the patch also point out what it's for.)

Quoted 5 lines
hw/xen/xen-bus.c +1 -3
+        XenBus *xenbus = XEN_BUS(qdev_get_parent_bus(DEVICE(xendev)));
 
-        if (!xen_backend_try_device_destroy(xendev, &local_err)) {
-            object_unparent(OBJECT(xendev));
-        }

hw/xen/xen-bus.c static void xen_device_backend_changed(void *opaque) · line 662

Quoted 10 lines
hw/xen/xen-bus.c +6 -3
+        QLIST_INSERT_HEAD(&xenbus->offline_devices, xendev, list);
 
-        if (local_err) {
-            error_report_err(local_err);
-        }
+        /*
+         * Re-write the state to cause a XenBus backend_watch notification,
+         * resulting in a call to xen_bus_cleanup().
+         */
+        xen_device_backend_printf(xendev, "state", "%u", state);

It kind of feels wrong to rely on xenstore to notify QEMU's xenbus driver that a device needs cleanup. But that does the job.

With a note about the use of QLIST_NEXT and the few typo fixed, the patch will be good to go.

Thanks,

Signature · 6 lines
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

Re: [Qemu-devel] [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 68 lines

-----Original Message-----
From: Anthony PERARD <[email protected]>
Sent: 12 September 2019 16:04
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; Stefano Stabellini <[email protected]>
Subject: Re: [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 1 line · Paul Durrant

Cleaning up offine XenDevice objects directly in

^ offline

Quoted 9 lines

xen_device_backend_changed() is dangerous as xen_device_unrealize() will modify the watch list that is being walked. Even the QLIST_FOREACH_SAFE() used in notifier_list_notify() is insufficient as *two* notifiers (for the frontend and backend watches) are removed, thus potentially rendering the 'next' pointer unsafe.

The solution is to use the XenBus backend_watch handler to do the clean-up instead, as it is invoked whilst walking a separate watch list.

Quoted 6 lines

This patch therefore adds a new 'offline_devices' list to XenBus, to which offline devices are added by xen_device_backend_changed(). The XenBus backend_watch registration is also changed to not only invoke xen_bus_enumerate() but also a new xen_bus_cleanup() function, which will walk 'offline_devices' and perform the necessary actions.
For safety a an extra 'online' check is also added to

^ one 'a' too many?

Quoted 29 lines

xen_bus_type_enumerate() to make sure that no attempt is made to create a new XenDevice object for a backend that is offline.

NOTE: This patch also include some cosmetic changes:
- substitute the local variable name 'backend_state'
in xen_bus_type_enumerate() with 'state', since there is no ambiguity with any other state in that context.
- change xen_device_state_is_active() to xen_device_frontend_is_active() (and pass a XenDevice directly) since the state tests contained therein only apply to a frontend.
- use 'state' rather then 'xendev->backend_state' in xen_device_backend_changed() to shorten the code.

---

v2:
- Make sure we don't try to add a XenDevice to 'offline_devices' more than once
---

/*
* If a backend is still 'online' then we should leave it alone but,

hw/xen/xen-bus.c +3 -2
-     * if a backend is not 'online', then the device should be destroyed
-     * once the state is Closed.
+     * if a backend is not 'online', then the device is a candidate
+     * for destruction. Hence add it to the 'offline' list to be cleaned
+     * by xen_bus_cleanup().
      */

hw/xen/xen-bus.c static void xen_device_backend_changed(void *opaque) · line 653

Quoted 10 lines
hw/xen/xen-bus.c +4 -6
-    if (!xendev->backend_online &&
-        (xendev->backend_state == XenbusStateClosed ||
-         xendev->backend_state == XenbusStateInitialising ||
-         xendev->backend_state == XenbusStateInitWait ||
-         xendev->backend_state == XenbusStateUnknown)) {
-        Error *local_err = NULL;
+    if (!online &&
+        (state == XenbusStateClosed ||  state == XenbusStateInitialising ||
+         state == XenbusStateInitWait || state == XenbusStateUnknown) &&
+        !QLIST_NEXT(xendev, list)) {
Quoted 5 lines

Could you add a note about this QLIST_NEXT? I don't think it's going to be obvious enough why we check that there are no next item. I might only understand it just because of your reply to the v1 of the patch. (Well the changelog of the patch also point out what it's for.)

Sure, it's worth a comment.

Quoted 16 lines
Quoted 5 lines
hw/xen/xen-bus.c +1 -3
+        XenBus *xenbus = XEN_BUS(qdev_get_parent_bus(DEVICE(xendev)));
 
-        if (!xen_backend_try_device_destroy(xendev, &local_err)) {
-            object_unparent(OBJECT(xendev));
-        }

hw/xen/xen-bus.c static void xen_device_backend_changed(void *opaque) · line 662

Quoted 10 lines
hw/xen/xen-bus.c +6 -3
+        QLIST_INSERT_HEAD(&xenbus->offline_devices, xendev, list);
 
-        if (local_err) {
-            error_report_err(local_err);
-        }
+        /*
+         * Re-write the state to cause a XenBus backend_watch notification,
+         * resulting in a call to xen_bus_cleanup().
+         */
+        xen_device_backend_printf(xendev, "state", "%u", state);
Quoted 3 lines

It kind of feels wrong to rely on xenstore to notify QEMU's xenbus driver that a device needs cleanup. But that does the job.

I had originally considered setting up an event notifier but that seemed like more fds than were strictly necessary ;-)

Quoted 3 lines

With a note about the use of QLIST_NEXT and the few typo fixed, the patch will be good to go.

Cool. I'll tidy up and send a v3.

Paul

Quoted 4 lines

Thanks,

--
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

Re: [Qemu-devel] [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 76 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Paul Durrant
Sent: 12 September 2019 16:16
To: Anthony Perard <[email protected]>
Cc: [email protected]; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [Xen-devel] [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 5 lines

-----Original Message-----
From: Anthony PERARD <[email protected]>
Sent: 12 September 2019 16:04
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; Stefano Stabellini

<[email protected]>

Quoted 63 lines

Subject: Re: [PATCH v2 3/3] xen: perform XenDevice clean-up in XenBus watch handler

Quoted 1 line · Paul Durrant

Cleaning up offine XenDevice objects directly in

^ offline

Quoted 9 lines

xen_device_backend_changed() is dangerous as xen_device_unrealize() will modify the watch list that is being walked. Even the QLIST_FOREACH_SAFE() used in notifier_list_notify() is insufficient as *two* notifiers (for the frontend and backend watches) are removed, thus potentially rendering the 'next' pointer unsafe.

The solution is to use the XenBus backend_watch handler to do the clean-up instead, as it is invoked whilst walking a separate watch list.

Quoted 6 lines

This patch therefore adds a new 'offline_devices' list to XenBus, to which offline devices are added by xen_device_backend_changed(). The XenBus backend_watch registration is also changed to not only invoke xen_bus_enumerate() but also a new xen_bus_cleanup() function, which will walk 'offline_devices' and perform the necessary actions.
For safety a an extra 'online' check is also added to

^ one 'a' too many?

Quoted 39 lines

xen_bus_type_enumerate() to make sure that no attempt is made to create a new XenDevice object for a backend that is offline.

NOTE: This patch also include some cosmetic changes:
- substitute the local variable name 'backend_state'
in xen_bus_type_enumerate() with 'state', since there is no ambiguity with any other state in that context.
- change xen_device_state_is_active() to xen_device_frontend_is_active() (and pass a XenDevice directly) since the state tests contained therein only apply to a frontend.
- use 'state' rather then 'xendev->backend_state' in xen_device_backend_changed() to shorten the code.

---

v2:
- Make sure we don't try to add a XenDevice to 'offline_devices' more than once
---

/*
* If a backend is still 'online' then we should leave it alone but,

hw/xen/xen-bus.c +7 -8
-     * if a backend is not 'online', then the device should be destroyed
-     * once the state is Closed.
+     * if a backend is not 'online', then the device is a candidate
+     * for destruction. Hence add it to the 'offline' list to be cleaned
+     * by xen_bus_cleanup().
      */
-    if (!xendev->backend_online &&
-        (xendev->backend_state == XenbusStateClosed ||
-         xendev->backend_state == XenbusStateInitialising ||
-         xendev->backend_state == XenbusStateInitWait ||
-         xendev->backend_state == XenbusStateUnknown)) {
-        Error *local_err = NULL;
+    if (!online &&
+        (state == XenbusStateClosed ||  state == XenbusStateInitialising ||
+         state == XenbusStateInitWait || state == XenbusStateUnknown) &&
+        !QLIST_NEXT(xendev, list)) {
Quoted 7 lines
Quoted 5 lines

Could you add a note about this QLIST_NEXT? I don't think it's going to be obvious enough why we check that there are no next item. I might only understand it just because of your reply to the v1 of the patch. (Well the changelog of the patch also point out what it's for.)

Sure, it's worth a comment.

Actually, on closer inspection !QLIST_NEXT() is an insufficient checked. I had assumed that QLISTs were full doubly-linked lists but they are not; the last element on a list will still have a NULL next pointer. It will be sufficient, and also clearer to the reader, if I add a boolean to XenDevice which is set when it is added to the list.

Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel