Skip to contentSkip to conversation

libxl: When restricted, start QEMU paused

PATCH Superseded Anthony PERARD 3 participants 7 messages Jan 24, 2019 – Jan 31, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libxl: When restricted, start QEMU paused

Since libxl later during guest creation run the command "cont", it kind of expect that QEMU would not do any emulation, use the "-S" command option to make this effective. Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this options when we have a QEMU startup notification via QMP available, which is when dm_restrict is activated.

This have the side-effect of rendering ineffective the startup notification via xenstore, libxl will only have the notification via QMP.

It became important to rely only on QMP for notification when we have it, as cutting short that path may result in the QMP socket been blocked and have QEMU stop responding to upcoming connection even if none are active.

The QEMU bug that this patch works around is:
- libxl connect and hand-check with QEMU, then send the cmd "query-status".
- QEMU prepare and maybe try send the response,
while also writing "running" into xenstore.
- libxl see via xenstore that QEMU is running and disconnect from the QMP socket before receiving the response the cmd.
=> The QMP socket (monitor) is sometime blocked and will never reply
to commands on new connections.

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

---
Here is an example of what's happening in osstest with qemu mainline:
http://logs.test-lab.xenproject.org/osstest/logs/132403/test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict/info.html
http://logs.test-lab.xenproject.org/osstest/logs/132403/test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict/10.ts-debian-hvm-install.log
(there are a few "qmp_next: Domain 1:timeout", beside what's look like debian not finishing to install and reboot)

---
 tools/libxl/libxl_dm.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
tools/libxl/libxl_dm.c +14 -1
index b245956b77..2f19786bdd 100644
@@ -1183,6 +1183,14 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
         flexarray_append(dm_args,
             GCSPRINTF("socket,id=libxl-cmd,fd=%d,server,nowait",
                       state->dm_monitor_fd));
+
+        /*
+         * Start QEMU with its "CPU" paused, it will not start any emulation
+         * until the QMP command "cont" is used. This also prevent QEMU from
+         * writing "running" to the "state" xenstore node so we only use this
+         * flag when we have the QMP based startup notification.
+         * */
+        flexarray_append(dm_args, "-S");
     } else {
         flexarray_append(dm_args,
                          GCSPRINTF("socket,id=libxl-cmd,"
@@ -2702,6 +2710,7 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,
     libxl__dm_spawn_state *dmss = CONTAINER_OF(ev, *dmss, qmp);
     const libxl__json_object *o;
     const char *status;
+    const char *expected_state;
 
     libxl__ev_qmp_dispose(gc, ev);
 
@@ -2717,7 +2726,11 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,
         goto failed;
     }
     status = libxl__json_object_get_string(o);
-    if (strcmp(status, "running")) {
+    if (!dmss->build_state->saved_state)
+        expected_state = "prelaunch";
+    else
+        expected_state = "paused";
+    if (strcmp(status, expected_state)) {
         LOGD(ERROR, ev->domid, "Unexpected QEMU status: %s", status);
         rc = ERROR_NOT_READY;
         goto failed;
Signature · 7 lines
Anthony PERARD


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 26 lines · Anthony PERARD

Since libxl later during guest creation run the command "cont", it kind of expect that QEMU would not do any emulation, use the "-S" command option to make this effective. Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this options when we have a QEMU startup notification via QMP available, which is when dm_restrict is activated.

This have the side-effect of rendering ineffective the startup notification via xenstore, libxl will only have the notification via QMP.

It became important to rely only on QMP for notification when we have it, as cutting short that path may result in the QMP socket been blocked and have QEMU stop responding to upcoming connection even if none are active.

The QEMU bug that this patch works around is:
- libxl connect and hand-check with QEMU, then send the cmd "query-status".
- QEMU prepare and maybe try send the response,
while also writing "running" into xenstore.
- libxl see via xenstore that QEMU is running and disconnect from the QMP socket before receiving the response the cmd.
=> The QMP socket (monitor) is sometime blocked and will never reply
to commands on new connections.

Quoted 7 lines

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
In reply to Anthony PERARD

Anthony PERARD writes ("[PATCH for-4.12] libxl: When restricted, start QEMU paused"):

Quoted 1 line

[stuff]

Thanks for this. I think the code looks right but to make it easier to understand what was going on I have taken the liberty of trying to reword your commit message for English grammar.

Can you please check that this is true ?

libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.

Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when dm_restrict is activated.

The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.

It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.

The QEMU bug that this patch works around is as follows:
- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]

I added a couple of xxx where I was particularly unsure.

Regards,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 38 lines · Ian Jackson

Anthony PERARD writes ("[PATCH for-4.12] libxl: When restricted, start QEMU paused"):

Quoted 1 line

[stuff]

Thanks for this. I think the code looks right but to make it easier to understand what was going on I have taken the liberty of trying to reword your commit message for English grammar.

Can you please check that this is true ?

libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.

Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when dm_restrict is activated.

The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.

It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.

The QEMU bug that this patch works around is as follows:
- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.

Quoted 8 lines

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]

Sorry for this been vague. I'm not 100% sure of what QEMU do and when. The listing of QEMU's action are probably not accurate.

QEMU probably write the "running" state to xenstore before starting to handle the QMP connection (respond to command). I'm unsure of when QEMU does handle QMP, but that's probably in the main_loop(), which is started after the piece of code that write to xenstore.

Or maybe the issue here is: s/hand-checks/handshake/, I was trying to describe the context with as little words as possible before getting to the point where things fails.

But here is a little more detail of libxl's action, when the bug happen (and before that patch is applied):
- connect(qmp_socket)
. wait (for event, like something to read or xenstore updates)
- receive QMP greeting, then send 'qmp_capabilities' command . wait
- receive response to qmp_capability, then send 'query-status' . wait
- receive update from xenstore: qemu's state=runnning
then close connection to qmp_socket and keep going with domain creation

Is that better to understand the context? How much of this would actally be useful for the patch description?

Quoted 1 line

I added a couple of xxx where I was particularly unsure.

Thanks,

Signature · 6 lines
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

Anthony PERARD writes ("Re: [PATCH for-4.12] libxl: When restricted, start QEMU paused"):

Quoted 16 lines · Ian Jackson
Quoted 15 lines

- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

Quoted 5 lines
Quoted 2 lines

[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]

Sorry for this been vague. I'm not 100% sure of what QEMU do and when. The listing of QEMU's action are probably not accurate.

Hrm. Maybe you want to write roughly' or I think' or something, then. I think the description of the QEMU behaviour is useful even if not 100% accurate.

Quoted 3 lines

Or maybe the issue here is: s/hand-checks/handshake/, I was trying to describe the context with as little words as possible before getting to the point where things fails.

Yes, handshake' makes much more sense. Hand-check' would mean `check by hand'.
https://en.wiktionary.org/wiki/handshake
https://en.wiktionary.org/wiki/by_hand

Quoted 2 lines

Is that better to understand the context? How much of this would actally be useful for the patch description?

Probably what you wrote originally and I edited, with the xxx's fixed. Assuming it's actually true...

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

PATCH v2 1/1 libxl: When restricted, start QEMU paused

libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.

Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when dm_restrict is activated.

The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.

It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.

When the QEMU bug happens, the actions taken by both libxl and QEMU are roughly as follows:
- libxl connects and handshakes with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.

This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command has been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.

---
v2:
commit message reworked.

---
 tools/libxl/libxl_dm.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
tools/libxl/libxl_dm.c +14 -1
index b245956b77..2f19786bdd 100644
@@ -1183,6 +1183,14 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
         flexarray_append(dm_args,
             GCSPRINTF("socket,id=libxl-cmd,fd=%d,server,nowait",
                       state->dm_monitor_fd));
+
+        /*
+         * Start QEMU with its "CPU" paused, it will not start any emulation
+         * until the QMP command "cont" is used. This also prevent QEMU from
+         * writing "running" to the "state" xenstore node so we only use this
+         * flag when we have the QMP based startup notification.
+         * */
+        flexarray_append(dm_args, "-S");
     } else {
         flexarray_append(dm_args,
                          GCSPRINTF("socket,id=libxl-cmd,"
@@ -2702,6 +2710,7 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,
     libxl__dm_spawn_state *dmss = CONTAINER_OF(ev, *dmss, qmp);
     const libxl__json_object *o;
     const char *status;
+    const char *expected_state;
 
     libxl__ev_qmp_dispose(gc, ev);
 
@@ -2717,7 +2726,11 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,
         goto failed;
     }
     status = libxl__json_object_get_string(o);
-    if (strcmp(status, "running")) {
+    if (!dmss->build_state->saved_state)
+        expected_state = "prelaunch";
+    else
+        expected_state = "paused";
+    if (strcmp(status, expected_state)) {
         LOGD(ERROR, ev->domid, "Unexpected QEMU status: %s", status);
         rc = ERROR_NOT_READY;
         goto failed;
Signature · 7 lines
Anthony PERARD


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel