Skip to contentSkip to conversation

DEVICE_UNPLUG_ERROR QAPI event

PATCH Superseded Daniel Henrique Barboza 4 participants 14 messages Jun 4, 2021 – Jun 17, 2021

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover DEVICE_UNPLUG_ERROR QAPI event

Hi,

This is the v2 of a series that started with 2 events, DEVICE_UNPLUG_ERROR and DEVICE_NOT_DELETED [1]. After discussions in v1 we reached the conclussion that the DEVICE_NOT_DELETED wasn't doing much of anything. It was an event that was trying to say 'I think something happen, but I'm not sure', forcing the QAPI listener to inspect the guest itself to see what went wrong, or just wait for some sort of internal timeout (as Libvirt will do) and fail the operation regardless.

During this period between v1 and this v2 the PowerPC kernel was changed to add a reliable error report mechanism in the device_removal path of CPUs, which in turn gave QEMU the opportunity to do the same. This made the DEVICE_UNPLUG_ERROR more relevant because now we can report CPU and DIMM hotunplug errors.

changes from v1:
- former patches 1 and 2: dropped
- patch 1 (former 3): changed the version to '6.1'
- patch 2 (former 4): add a DEVICE_UNPLUG_ERROR event in the device unplug error path of CPUs and DIMMs

[1] v1 link: https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg04682.html

Daniel Henrique Barboza (2):
qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event
spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

 hw/ppc/spapr.c     |  2 +-
 hw/ppc/spapr_drc.c | 15 +++++++++------
 qapi/machine.json  | 23 +++++++++++++++++++++++
 3 files changed, 33 insertions(+), 7 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v2 1/2 qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

---
 qapi/machine.json | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
qapi/machine.json +23 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
 ##
 { 'event': 'MEM_UNPLUG_ERROR',
   'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }
Signature · 1 line
2.31.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

Quoted 10 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

Quoted 35 lines
---
 qapi/machine.json | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
qapi/machine.json +23 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
 ##
 { 'event': 'MEM_UNPLUG_ERROR',
   'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#3 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

Quoted 6 lines · David Gibson
Quoted 5 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Quoted 7 lines
Quoted 5 lines

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

Markus, I'm happy to take this through my tree if that's convenient for you, but I'd like to get an ack.

Quoted 37 lines
Quoted 36 lines
---
 qapi/machine.json | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
qapi/machine.json +23 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
 ##
 { 'event': 'MEM_UNPLUG_ERROR',
   'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
Reviewed-by
#4 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

Quoted 11 lines · David Gibson
Quoted 10 lines · David Gibson
Quoted 9 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Markus, I'm happy to take this through my tree if that's convenient for you, but I'd like to get an ack.

I'm not Markus, but if you're happy with my interface review,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#5 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

In reply to Daniel Henrique Barboza
Quoted 35 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

---
 qapi/machine.json | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
qapi/machine.json +13 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
 ##
 { 'event': 'MEM_UNPLUG_ERROR',
   'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:

qapi/machine.json line 1299

Quoted 10 lines
qapi/machine.json +10 -0
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }

Missing: update of device_add's doc comment in qdev.json:

# Notes: When this command completes, the device may not be removed from the
# guest. Hot removal is an operation that requires guest cooperation.
# This command merely requests that the guest begin the hot removal
# process. Completion of the device removal process is signaled with a
# DEVICE_DELETED event. Guest reset will automatically complete removal
# for all devices.

This sure could use some polish.

If I understand things correctly, we're aiming for the following device unplug protocol:

Unplug the device with device_del (or possibly equivalent)

If we know we can't unplug the device, fail immediately. Also emit DEVICE_UNPLUG_ERROR.

If possible, unplug the device synchronously and succeed. Also emit DEVICE_DELETED.

Else, initiate unplug and succeed.

When unplug finishes, emit either DEVICE_DELETED or DEVICE_UNPLUG_ERROR.

For some machines and devices, unplug may never finish.

Correct?

Any particular reason for not putting event DEVICE_UNPLUG_ERROR next to DEVICE_DELETED in qdev.json?

#6 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

Quoted 50 lines · Markus Armbruster
Quoted 35 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

---
  qapi/machine.json | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)
qapi/machine.json +13 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
  ##
  { 'event': 'MEM_UNPLUG_ERROR',
    'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:

qapi/machine.json line 1299

Quoted 10 lines
qapi/machine.json +10 -0
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }

Missing: update of device_add's doc comment in qdev.json:

Quoted 6 lines

# Notes: When this command completes, the device may not be removed from the
# guest. Hot removal is an operation that requires guest cooperation.
# This command merely requests that the guest begin the hot removal
# process. Completion of the device removal process is signaled with a
# DEVICE_DELETED event. Guest reset will automatically complete removal
# for all devices.

Ok

Quoted 5 lines

This sure could use some polish.

If I understand things correctly, we're aiming for the following device unplug protocol:

One thing to note is that DEVICE_UNPLUG_ERROR isn't guaranteed to be send for every hotunplug error. The event depends on machine/architecture support to detect a guest side error.

Quoted 5 lines

Unplug the device with device_del (or possibly equivalent)

If we know we can't unplug the device, fail immediately. Also emit DEVICE_UNPLUG_ERROR.

I haven't predicted to use this event in those cases as well, although it seems reasonable to do so now that you mentioned it.

Quoted 8 lines

If possible, unplug the device synchronously and succeed. Also emit DEVICE_DELETED.

Else, initiate unplug and succeed.

When unplug finishes, emit either DEVICE_DELETED or DEVICE_UNPLUG_ERROR.

Since there's no 100% guarantee that DEVICE_UNPLUG_ERROR will be emitted for guest side errors, the wording here would be

"When unplug finishes, emit DEVICE_DELETED. A DEVICE_UNPLUG_ERROR can be emitted if a guest side error was detected"

Quoted 7 lines

For some machines and devices, unplug may never finish.

Correct?

Any particular reason for not putting event DEVICE_UNPLUG_ERROR next to DEVICE_DELETED in qdev.json?

Not really. I looked where MEM_UNPLUG_ERROR was declared and put it right after it. I can change it to qdev.json near DEVICE_DELETED.

Thanks,

Daniel

Quoted 2 lines
#7 lore raw

Re: [PATCH v2 1/2] qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event

Quoted 61 lines · Daniel Henrique Barboza
Quoted 50 lines · Markus Armbruster
Quoted 35 lines · Daniel Henrique Barboza

At this moment we only provide one event to report a hotunplug error, MEM_UNPLUG_ERROR. As of Linux kernel 5.12 and QEMU 6.0.0, the pseries machine is now able to report unplug errors for other device types, such as CPUs.

Instead of creating a (device_type)_UNPLUG_ERROR for each new device, create a generic DEVICE_UNPLUG_ERROR event that can be used by all unplug errors in the future.

---
  qapi/machine.json | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)
qapi/machine.json +13 -0
index 58a9c86b36..f0c7e56be0 100644
@@ -1274,3 +1274,26 @@
  ##
  { 'event': 'MEM_UNPLUG_ERROR',
    'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @DEVICE_UNPLUG_ERROR:
+#
+# Emitted when a device hot unplug error occurs.
+#
+# @device: device name
+#
+# @msg: Informative message
+#
+# Since: 6.1
+#
+# Example:

qapi/machine.json line 1299

Quoted 10 lines
qapi/machine.json +10 -0
+#
+# <- { "event": "DEVICE_UNPLUG_ERROR"
+#      "data": { "device": "dimm1",
+#                "msg": "Memory hotunplug rejected by the guest for device dimm1"
+#      },
+#      "timestamp": { "seconds": 1615570772, "microseconds": 202844 } }
+#
+##
+{ 'event': 'DEVICE_UNPLUG_ERROR',
+  'data': { 'device': 'str', 'msg': 'str' } }

Missing: update of device_add's doc comment in qdev.json:

Quoted 6 lines

# Notes: When this command completes, the device may not be removed from the
# guest. Hot removal is an operation that requires guest cooperation.
# This command merely requests that the guest begin the hot removal
# process. Completion of the device removal process is signaled with a
# DEVICE_DELETED event. Guest reset will automatically complete removal
# for all devices.

Ok

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

This sure could use some polish.

If I understand things correctly, we're aiming for the following device unplug protocol:

One thing to note is that DEVICE_UNPLUG_ERROR isn't guaranteed to be send for every hotunplug error. The event depends on machine/architecture support to detect a guest side error.

Yes. I tried to provide for that in my description of the protocol.

Quoted 9 lines
Quoted 5 lines

Unplug the device with device_del (or possibly equivalent)

If we know we can't unplug the device, fail immediately. Also emit DEVICE_UNPLUG_ERROR.

I haven't predicted to use this event in those cases as well, although it seems reasonable to do so now that you mentioned it.

I think this is a matter of taste. For what it's worth, we do emit DEVICE_DELETED on immediate success (see next paragraph).

Emitting DEVICE_DELETED always lets QMP clients track device deletion even when it's triggered by something else. Feature.

No such tracking is needed for unplug failure.

Quoted 5 lines
Quoted 5 lines

If possible, unplug the device synchronously and succeed. Also emit DEVICE_DELETED.

Else, initiate unplug and succeed.

Quoted 8 lines
Quoted 2 lines

When unplug finishes, emit either DEVICE_DELETED or DEVICE_UNPLUG_ERROR.

Since there's no 100% guarantee that DEVICE_UNPLUG_ERROR will be emitted for guest side errors, the wording here would be

"When unplug finishes, emit DEVICE_DELETED. A DEVICE_UNPLUG_ERROR can be emitted if a guest side error was detected"

My assumption is that QEMU can detect asynchronous success reliably, but not asynchronous failure.

From QEMU's point of view, "asynchronous unplug has not finished" is indistinguishable from "asynchronous unplug finished unsuccessfully, but we can't detect it". So I lumped these two cases together in the next paragraph:

Quoted 1 line
Quoted 1 line

For some machines and devices, unplug may never finish.

My goal is a clear description of the state machine as it can be observed in QMP.

Quoted 8 lines
Quoted 4 lines

Correct?

Any particular reason for not putting event DEVICE_UNPLUG_ERROR next to DEVICE_DELETED in qdev.json?

Not really. I looked where MEM_UNPLUG_ERROR was declared and put it right after it. I can change it to qdev.json near DEVICE_DELETED.

Yes, please.

#8 lore raw

PATCH v2 2/2 spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

In reply to Daniel Henrique Barboza

Linux Kernel 5.12 is now unisolating CPU DRCs in the device_removal error path, signalling that the hotunplug process wasn't successful. This allow us to send a DEVICE_UNPLUG_ERROR in drc_unisolate_logical() to signal this error to the management layer.

We also have another error path in spapr_memory_unplug_rollback() for configured LMB DRCs. Kernels older than 5.13 will not unisolate the LMBs in the hotunplug error path, but it will reconfigure them. Let's send the DEVICE_UNPLUG_ERROR event in that code path as well to cover the case of older kernels.

---
 hw/ppc/spapr.c     |  2 +-
 hw/ppc/spapr_drc.c | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
hw/ppc/spapr.c +1 -1
index c23bcc4490..29aa2f467d 100644
@@ -3639,7 +3639,7 @@ void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev)
      */
     qapi_error = g_strdup_printf("Memory hotunplug rejected by the guest "
                                  "for device %s", dev->id);
-    qapi_event_send_mem_unplug_error(dev->id, qapi_error);
+    qapi_event_send_device_unplug_error(dev->id, qapi_error);
 }
 
 /* Callback to be called during DRC release. */
hw/ppc/spapr_drc.c +9 -6
index a2f2634601..0e1a8733bc 100644
@@ -17,6 +17,8 @@
 #include "hw/ppc/spapr_drc.h"
 #include "qom/object.h"
 #include "migration/vmstate.h"
+#include "qapi/error.h"
+#include "qapi/qapi-events-machine.h"
 #include "qapi/visitor.h"
 #include "qemu/error-report.h"
 #include "hw/ppc/spapr.h" /* for RTAS return codes */
@@ -160,6 +162,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
          * means that the kernel is refusing the removal.
          */
         if (drc->unplug_requested && drc->dev) {
+            const char qapi_error_fmt[] = "Device hotunplug rejected by the "
+                                          "guest for device %s";
+            g_autofree char *qapi_error = NULL;
+
             if (spapr_drc_type(drc) == SPAPR_DR_CONNECTOR_TYPE_LMB) {
                 spapr = SPAPR_MACHINE(qdev_get_machine());
 
@@ -167,13 +173,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
             }
 
             drc->unplug_requested = false;
-            error_report("Device hotunplug rejected by the guest "
-                         "for device %s", drc->dev->id);
+            error_report(qapi_error_fmt, drc->dev->id);
 
-            /*
-             * TODO: send a QAPI DEVICE_UNPLUG_ERROR event when
-             * it is implemented.
-             */
+            qapi_error = g_strdup_printf(qapi_error_fmt, drc->dev->id);
+            qapi_event_send_device_unplug_error(drc->dev->id, qapi_error);
         }
 
         return RTAS_OUT_SUCCESS; /* Nothing to do */
Signature · 1 line
2.31.1
Reviewed-by on v2 2/2
#9 lore raw

Re: [PATCH v2 2/2] spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

Quoted 5 lines · Daniel Henrique Barboza

Linux Kernel 5.12 is now unisolating CPU DRCs in the device_removal error path, signalling that the hotunplug process wasn't successful. This allow us to send a DEVICE_UNPLUG_ERROR in drc_unisolate_logical() to signal this error to the management layer.

Quoted 7 lines

We also have another error path in spapr_memory_unplug_rollback() for configured LMB DRCs. Kernels older than 5.13 will not unisolate the LMBs in the hotunplug error path, but it will reconfigure them. Let's send the DEVICE_UNPLUG_ERROR event in that code path as well to cover the case of older kernels.

Quoted 59 lines
---
 hw/ppc/spapr.c     |  2 +-
 hw/ppc/spapr_drc.c | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
hw/ppc/spapr.c +1 -1
index c23bcc4490..29aa2f467d 100644
@@ -3639,7 +3639,7 @@ void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev)
      */
     qapi_error = g_strdup_printf("Memory hotunplug rejected by the guest "
                                  "for device %s", dev->id);
-    qapi_event_send_mem_unplug_error(dev->id, qapi_error);
+    qapi_event_send_device_unplug_error(dev->id, qapi_error);
 }
 
 /* Callback to be called during DRC release. */
hw/ppc/spapr_drc.c +9 -6
index a2f2634601..0e1a8733bc 100644
@@ -17,6 +17,8 @@
 #include "hw/ppc/spapr_drc.h"
 #include "qom/object.h"
 #include "migration/vmstate.h"
+#include "qapi/error.h"
+#include "qapi/qapi-events-machine.h"
 #include "qapi/visitor.h"
 #include "qemu/error-report.h"
 #include "hw/ppc/spapr.h" /* for RTAS return codes */
@@ -160,6 +162,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
          * means that the kernel is refusing the removal.
          */
         if (drc->unplug_requested && drc->dev) {
+            const char qapi_error_fmt[] = "Device hotunplug rejected by the "
+                                          "guest for device %s";
+            g_autofree char *qapi_error = NULL;
+
             if (spapr_drc_type(drc) == SPAPR_DR_CONNECTOR_TYPE_LMB) {
                 spapr = SPAPR_MACHINE(qdev_get_machine());
 
@@ -167,13 +173,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
             }
 
             drc->unplug_requested = false;
-            error_report("Device hotunplug rejected by the guest "
-                         "for device %s", drc->dev->id);
+            error_report(qapi_error_fmt, drc->dev->id);
 
-            /*
-             * TODO: send a QAPI DEVICE_UNPLUG_ERROR event when
-             * it is implemented.
-             */
+            qapi_error = g_strdup_printf(qapi_error_fmt, drc->dev->id);
+            qapi_event_send_device_unplug_error(drc->dev->id, qapi_error);
         }
 
         return RTAS_OUT_SUCCESS; /* Nothing to do */
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#10 lore raw

Re: [PATCH v2 2/2] spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

In reply to Daniel Henrique Barboza
Quoted 22 lines · Daniel Henrique Barboza

Linux Kernel 5.12 is now unisolating CPU DRCs in the device_removal error path, signalling that the hotunplug process wasn't successful. This allow us to send a DEVICE_UNPLUG_ERROR in drc_unisolate_logical() to signal this error to the management layer.

We also have another error path in spapr_memory_unplug_rollback() for configured LMB DRCs. Kernels older than 5.13 will not unisolate the LMBs in the hotunplug error path, but it will reconfigure them. Let's send the DEVICE_UNPLUG_ERROR event in that code path as well to cover the case of older kernels.

---
 hw/ppc/spapr.c     |  2 +-
 hw/ppc/spapr_drc.c | 15 +++++++++------
 2 files changed, 10 insertions(+), 7 deletions(-)
hw/ppc/spapr.c +0 -0
index c23bcc4490..29aa2f467d 100644

hw/ppc/spapr.c void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev) · line 3642

Quoted 5 lines
hw/ppc/spapr.c +1 -1
@@ -3639,7 +3639,7 @@ void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev)
      */
     qapi_error = g_strdup_printf("Memory hotunplug rejected by the guest "
                                  "for device %s", dev->id);
-    qapi_event_send_mem_unplug_error(dev->id, qapi_error);
+    qapi_event_send_device_unplug_error(dev->id, qapi_error);

Incompatible change: we now emit DEVICE_UNPLUG_ERROR instead of MEM_UNPLUG_ERROR. Intentional?

If yes, we need a release note.

To avoid the incompatible, we can emit both, and deprecate MEM_UNPLUG_ERROR.

What about the MEM_UNPLUG_ERROR in acpi_memory_hotplug_write()?

Quoted 34 lines
hw/ppc/spapr.c +0 -0
 }
 
 /* Callback to be called during DRC release. */
hw/ppc/spapr_drc.c +7 -2
index a2f2634601..0e1a8733bc 100644
@@ -17,6 +17,8 @@
 #include "hw/ppc/spapr_drc.h"
 #include "qom/object.h"
 #include "migration/vmstate.h"
+#include "qapi/error.h"
+#include "qapi/qapi-events-machine.h"
 #include "qapi/visitor.h"
 #include "qemu/error-report.h"
 #include "hw/ppc/spapr.h" /* for RTAS return codes */
@@ -160,6 +162,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
          * means that the kernel is refusing the removal.
          */
         if (drc->unplug_requested && drc->dev) {
+            const char qapi_error_fmt[] = "Device hotunplug rejected by the "
+                                          "guest for device %s";
+            g_autofree char *qapi_error = NULL;
+
             if (spapr_drc_type(drc) == SPAPR_DR_CONNECTOR_TYPE_LMB) {
                 spapr = SPAPR_MACHINE(qdev_get_machine());
 
@@ -167,13 +173,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
             }
 
             drc->unplug_requested = false;
-            error_report("Device hotunplug rejected by the guest "
-                         "for device %s", drc->dev->id);
+            error_report(qapi_error_fmt, drc->dev->id);

hw/ppc/spapr_drc.c static uint32_t drc_unisolate_logical(SpaprDrc *drc) · line 182

Quoted 10 lines
hw/ppc/spapr_drc.c +2 -4
 
-            /*
-             * TODO: send a QAPI DEVICE_UNPLUG_ERROR event when
-             * it is implemented.
-             */
+            qapi_error = g_strdup_printf(qapi_error_fmt, drc->dev->id);
+            qapi_event_send_device_unplug_error(drc->dev->id, qapi_error);
         }
 
         return RTAS_OUT_SUCCESS; /* Nothing to do */

Reporting both to stderr and QMP is odd. Can you describe a use case where the report to stderr is useful?

#11 lore raw

Re: [PATCH v2 2/2] spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

Quoted 33 lines · Markus Armbruster
Quoted 22 lines · Daniel Henrique Barboza

Linux Kernel 5.12 is now unisolating CPU DRCs in the device_removal error path, signalling that the hotunplug process wasn't successful. This allow us to send a DEVICE_UNPLUG_ERROR in drc_unisolate_logical() to signal this error to the management layer.

We also have another error path in spapr_memory_unplug_rollback() for configured LMB DRCs. Kernels older than 5.13 will not unisolate the LMBs in the hotunplug error path, but it will reconfigure them. Let's send the DEVICE_UNPLUG_ERROR event in that code path as well to cover the case of older kernels.

---
  hw/ppc/spapr.c     |  2 +-
  hw/ppc/spapr_drc.c | 15 +++++++++------
  2 files changed, 10 insertions(+), 7 deletions(-)
hw/ppc/spapr.c +0 -0
index c23bcc4490..29aa2f467d 100644

hw/ppc/spapr.c void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev) · line 3642

Quoted 5 lines
hw/ppc/spapr.c +1 -1
@@ -3639,7 +3639,7 @@ void spapr_memory_unplug_rollback(SpaprMachineState *spapr, DeviceState *dev)
       */
      qapi_error = g_strdup_printf("Memory hotunplug rejected by the guest "
                                   "for device %s", dev->id);
-    qapi_event_send_mem_unplug_error(dev->id, qapi_error);
+    qapi_event_send_device_unplug_error(dev->id, qapi_error);

Incompatible change: we now emit DEVICE_UNPLUG_ERROR instead of MEM_UNPLUG_ERROR. Intentional?

Quoted 6 lines

If yes, we need a release note.

To avoid the incompatible, we can emit both, and deprecate MEM_UNPLUG_ERROR.

What about the MEM_UNPLUG_ERROR in acpi_memory_hotplug_write()?

I'll emit DEVICE_UNPLUG_ERROR together with all MEM_UNPLUG_ERROR instances. Then we can deprecate MEM_UNPLUG_ERROR.

By the way, how do I mark MEM_UNPLUG_ERROR as deprecated? I see examples of command line options being documented as deprecated in docs/system/deprecated.rst and some deprecated QOM/QDEV properties are marked as deprecated directly in their .json files, but I didn't find any case where a whole event is deprecated. Would something like this be adequate?

$ git diff
qapi/machine.json +4 -0
index 58a9c86b36..ce3d873c64 100644
@@ -1261,6 +1261,10 @@
 #
 # @msg: Informative message
 #
+#
+# @deprecated: Starting in 6.1 this event has been replaced by
+#              DEVICE_UNPLUG_ERROR.
+#
 # Since: 2.4
 #
 # Example:

Thanks,

Daniel

Quoted 49 lines
Quoted 35 lines
hw/ppc/spapr.c +0 -0
  }
 
  /* Callback to be called during DRC release. */
hw/ppc/spapr_drc.c +7 -2
index a2f2634601..0e1a8733bc 100644
@@ -17,6 +17,8 @@
  #include "hw/ppc/spapr_drc.h"
  #include "qom/object.h"
  #include "migration/vmstate.h"
+#include "qapi/error.h"
+#include "qapi/qapi-events-machine.h"
  #include "qapi/visitor.h"
  #include "qemu/error-report.h"
  #include "hw/ppc/spapr.h" /* for RTAS return codes */
@@ -160,6 +162,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
           * means that the kernel is refusing the removal.
           */
          if (drc->unplug_requested && drc->dev) {
+            const char qapi_error_fmt[] = "Device hotunplug rejected by the "
+                                          "guest for device %s";
+            g_autofree char *qapi_error = NULL;
+
              if (spapr_drc_type(drc) == SPAPR_DR_CONNECTOR_TYPE_LMB) {
                  spapr = SPAPR_MACHINE(qdev_get_machine());
 
@@ -167,13 +173,10 @@ static uint32_t drc_unisolate_logical(SpaprDrc *drc)
              }
 
              drc->unplug_requested = false;
-            error_report("Device hotunplug rejected by the guest "
-                         "for device %s", drc->dev->id);
+            error_report(qapi_error_fmt, drc->dev->id);

hw/ppc/spapr_drc.c static uint32_t drc_unisolate_logical(SpaprDrc *drc) · line 182

Quoted 10 lines
hw/ppc/spapr_drc.c +2 -4
 
-            /*
-             * TODO: send a QAPI DEVICE_UNPLUG_ERROR event when
-             * it is implemented.
-             */
+            qapi_error = g_strdup_printf(qapi_error_fmt, drc->dev->id);
+            qapi_event_send_device_unplug_error(drc->dev->id, qapi_error);
          }
 
          return RTAS_OUT_SUCCESS; /* Nothing to do */

Reporting both to stderr and QMP is odd. Can you describe a use case where the report to stderr is useful?

#12 lore raw

Re: [PATCH v2 2/2] spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

Quoted 13 lines · Daniel Henrique Barboza
Quoted 9 lines

Incompatible change: we now emit DEVICE_UNPLUG_ERROR instead of MEM_UNPLUG_ERROR. Intentional?

If yes, we need a release note.

To avoid the incompatible, we can emit both, and deprecate MEM_UNPLUG_ERROR.

What about the MEM_UNPLUG_ERROR in acpi_memory_hotplug_write()?

I'll emit DEVICE_UNPLUG_ERROR together with all MEM_UNPLUG_ERROR instances. Then we can deprecate MEM_UNPLUG_ERROR.

Quoted 6 lines

By the way, how do I mark MEM_UNPLUG_ERROR as deprecated? I see examples of command line options being documented as deprecated in docs/system/deprecated.rst and some deprecated QOM/QDEV properties are marked as deprecated directly in their .json files, but I didn't find any case where a whole event is deprecated. Would something like this be adequate?

Almost. That documents the deprecation for readers, but you also need to mark it for viewing by machine code...

Quoted 8 lines

$ git diff
qapi/machine.json +0 -0
index 58a9c86b36..ce3d873c64 100644

qapi/machine.json line 1270

Quoted 10 lines
qapi/machine.json +4 -0
@@ -1261,6 +1261,10 @@
 #
 # @msg: Informative message
 #
+#
+# @deprecated: Starting in 6.1 this event has been replaced by
+#              DEVICE_UNPLUG_ERROR.
+#
 # Since: 2.4
 #
 # Example:

...do that by adding 'features':['deprecated'] to the QAPI event definition.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#13 lore raw
In reply to Daniel Henrique Barboza
Quoted 17 lines · Daniel Henrique Barboza

Hi,

This is the v2 of a series that started with 2 events, DEVICE_UNPLUG_ERROR and DEVICE_NOT_DELETED [1]. After discussions in v1 we reached the conclussion that the DEVICE_NOT_DELETED wasn't doing much of anything. It was an event that was trying to say 'I think something happen, but I'm not sure', forcing the QAPI listener to inspect the guest itself to see what went wrong, or just wait for some sort of internal timeout (as Libvirt will do) and fail the operation regardless.

During this period between v1 and this v2 the PowerPC kernel was changed to add a reliable error report mechanism in the device_removal path of CPUs, which in turn gave QEMU the opportunity to do the same. This made the DEVICE_UNPLUG_ERROR more relevant because now we can report CPU and DIMM hotunplug errors.

Quoted 8 lines

changes from v1:
- former patches 1 and 2: dropped
- patch 1 (former 3): changed the version to '6.1'
- patch 2 (former 4): add a DEVICE_UNPLUG_ERROR event in the device unplug error path of CPUs and DIMMs

[1] v1 link:
https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg04682.html

It would be nice to add a patch making x86 also issue the new error format (as well as the old) on memory hot unplug errors.

Quoted 12 lines

Daniel Henrique Barboza (2):
qapi/machine.json: add DEVICE_UNPLUG_ERROR QAPI event
spapr: use DEVICE_UNPLUG_ERROR to report unplug errors

 hw/ppc/spapr.c     |  2 +-
 hw/ppc/spapr_drc.c | 15 +++++++++------
 qapi/machine.json  | 23 +++++++++++++++++++++++
 3 files changed, 33 insertions(+), 7 deletions(-)
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson