libxl: When restricted, start QEMU paused
PATCH 1/1 libxl: When restricted, start QEMU paused
Since libxl later during guest creation run the command "cont", it kind of expect that QEMU would not do any emulation, use the "-S" command option to make this effective. Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this options when we have a QEMU startup notification via QMP available, which is when dm_restrict is activated.
This have the side-effect of rendering ineffective the startup notification via xenstore, libxl will only have the notification via QMP.
It became important to rely only on QMP for notification when we have it, as cutting short that path may result in the QMP socket been blocked and have QEMU stop responding to upcoming connection even if none are active.
The QEMU bug that this patch works around is:
- libxl connect and hand-check with QEMU, then send the cmd "query-status".
- QEMU prepare and maybe try send the response,
while also writing "running" into xenstore.
- libxl see via xenstore that QEMU is running and disconnect from the QMP socket before receiving the response the cmd.
=> The QMP socket (monitor) is sometime blocked and will never reply
to commands on new connections.
This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
---
Here is an example of what's happening in osstest with qemu mainline:
http://logs.test-lab.xenproject.org/osstest/logs/132403/test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict/info.html
http://logs.test-lab.xenproject.org/osstest/logs/132403/test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict/10.ts-debian-hvm-install.log
(there are a few "qmp_next: Domain 1:timeout", beside what's look like debian not finishing to install and reboot)
--- tools/libxl/libxl_dm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
tools/libxl/libxl_dm.c +14 -1
@@ -1183,6 +1183,14 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,flexarray_append (dm_args,GCSPRINTF ("socket,id=libxl-cmd,fd=%d,server,nowait" , state->dm_monitor_fd)); + +/* +* Start QEMU with its "CPU" paused, it will not start any emulation +* until the QMP command "cont" is used. This also prevent QEMU from +* writing "running" to the "state" xenstore node so we only use this +* flag when we have the QMP based startup notification. +* */ +flexarray_append (dm_args,"-S" ); }else { flexarray_append(dm_args, GCSPRINTF("socket,id=libxl-cmd,"
@@ -2702,6 +2710,7 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,libxl__dm_spawn_state *dmss =CONTAINER_OF (ev, *dmss, qmp);const libxl__json_object *o;const char *status; +const char *expected_state;libxl__ev_qmp_dispose (gc, ev);
@@ -2717,7 +2726,11 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev, goto failed; } status =libxl__json_object_get_string (o); -if (strcmp (status,"running" )) { +if (!dmss->build_state->saved_state) + expected_state ="prelaunch" ; +else + expected_state ="paused" ; +if (strcmp (status, expected_state)) {LOGD (ERROR , ev->domid,"Unexpected QEMU status: %s" , status); rc =ERROR_NOT_READY ; goto failed;
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 26 lines · Anthony PERARD
Since libxl later during guest creation run the command "cont", it kind of expect that QEMU would not do any emulation, use the "-S" command option to make this effective. Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this options when we have a QEMU startup notification via QMP available, which is when
dm_restrictis activated.This have the side-effect of rendering ineffective the startup notification via xenstore, libxl will only have the notification via QMP.
It became important to rely only on QMP for notification when we have it, as cutting short that path may result in the QMP socket been blocked and have QEMU stop responding to upcoming connection even if none are active.
The QEMU bug that this patch works around is:
- libxl connect and hand-check with QEMU, then send the cmd "query-status".
- QEMU prepare and maybe try send the response,
while also writing "running" into xenstore.
- libxl see via xenstore that QEMU is running and disconnect from the QMP socket before receiving the response the cmd.
=> The QMP socket (monitor) is sometime blocked and will never reply
to commands on new connections.
Quoted 7 lines
This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
Signed-off-by Anthony PERARD <[email protected]>
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Anthony PERARD writes ("[PATCH for-4.12] libxl: When restricted, start QEMU paused"):
Quoted 1 line
[stuff]
Thanks for this. I think the code looks right but to make it easier to understand what was going on I have taken the liberty of trying to reword your commit message for English grammar.
Can you please check that this is true ?
libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.
Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when dm_restrict is activated.
The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.
It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.
The QEMU bug that this patch works around is as follows:
- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.
This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]
I added a couple of xxx where I was particularly unsure.
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 38 lines · Ian Jackson
Anthony PERARD writes ("[PATCH for-4.12] libxl: When restricted, start QEMU paused"):
Quoted 1 line
[stuff]
Thanks for this. I think the code looks right but to make it easier to understand what was going on I have taken the liberty of trying to reword your commit message for English grammar.
Can you please check that this is true ?
libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.
Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when
dm_restrictis activated.The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.
It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.
The QEMU bug that this patch works around is as follows:
- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.
Quoted 8 lines
This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]
Sorry for this been vague. I'm not 100% sure of what QEMU do and when. The listing of QEMU's action are probably not accurate.
QEMU probably write the "running" state to xenstore before starting to handle the QMP connection (respond to command). I'm unsure of when QEMU does handle QMP, but that's probably in the main_loop(), which is started after the piece of code that write to xenstore.
Or maybe the issue here is: s/hand-checks/handshake/, I was trying to describe the context with as little words as possible before getting to the point where things fails.
But here is a little more detail of libxl's action, when the bug happen (and before that patch is applied):
- connect(qmp_socket)
. wait (for event, like something to read or xenstore updates)
- receive QMP greeting, then send 'qmp_capabilities' command . wait
- receive response to qmp_capability, then send 'query-status' . wait
- receive update from xenstore: qemu's state=runnning
then close connection to qmp_socket and keep going with domain creation
Is that better to understand the context? How much of this would actally be useful for the patch description?
Quoted 1 line
I added a couple of xxx where I was particularly unsure.
Thanks,
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Anthony PERARD writes ("Re: [PATCH for-4.12] libxl: When restricted, start QEMU paused"):
Quoted 16 lines · Ian Jackson
Quoted 15 lines
- libxl connects and hand-checks [xxx???] with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command as been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
Quoted 5 lines
Quoted 2 lines
[xxx??? So, require, therefore, that when we get the QMP readiness notification, the qemu state is xenstore.]
Sorry for this been vague. I'm not 100% sure of what QEMU do and when. The listing of QEMU's action are probably not accurate.
Hrm. Maybe you want to write roughly' or I think' or something, then. I think the description of the QEMU behaviour is useful even if not 100% accurate.
Quoted 3 lines
Or maybe the issue here is: s/hand-checks/handshake/, I was trying to describe the context with as little words as possible before getting to the point where things fails.
Yes, handshake' makes much more sense. Hand-check' would mean `check by hand'.
https://en.wiktionary.org/wiki/handshake
https://en.wiktionary.org/wiki/by_hand
Quoted 2 lines
Is that better to understand the context? How much of this would actally be useful for the patch description?
Probably what you wrote originally and I edited, with the xxx's fixed. Assuming it's actually true...
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 1/1 libxl: When restricted, start QEMU paused
libxl runs the command "cont" later during guest creation; i.e. it is expecting that QEMU would not do any emulation. Use the "-S" command option to achieve this.
Unfortunately, when QEMU is started with "-S", it won't write QEMU's readiness into xenstore. So only activate this option when we have a QEMU startup notification via QMP available, i.e. when dm_restrict is activated.
The -S option has the side-effect of suppressing the startup notification via xenstore: libxl will only get the notification via QMP.
It is important to rely only on QMP for notification when we have QMP available, as (due to a qemu bug) not waiting for that QMP notification may result in the QMP socket becoming blocked, so that QEMU stops responding to new connections even if no existing ones are active.
When the QEMU bug happens, the actions taken by both libxl and QEMU are roughly as follows:
- libxl connects and handshakes with QEMU, then sends the cmd "query-status".
- QEMU prepares and maybe tries to send the response,
while also writing "running" into xenstore.
- libxl sees via xenstore that QEMU is running and disconnects from the QMP socket before receiving the response from the cmd.
=> The QMP socket (monitor) is thereby blocked and will never reply
to commands on new connections.
This is due to QEMU only responding to one command at a time, and suspending its monitor (QMP) until the command has been processed and sent. Disconnecting from the socket doesn't unsuspend the monitor. The race described here is very likely to happen with QEMU 3.1.50 (during 3.2 development), but can be reproduced with QEMU 3.1.
---
v2:
commit message reworked.
--- tools/libxl/libxl_dm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
tools/libxl/libxl_dm.c +14 -1
@@ -1183,6 +1183,14 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,flexarray_append (dm_args,GCSPRINTF ("socket,id=libxl-cmd,fd=%d,server,nowait" , state->dm_monitor_fd)); + +/* +* Start QEMU with its "CPU" paused, it will not start any emulation +* until the QMP command "cont" is used. This also prevent QEMU from +* writing "running" to the "state" xenstore node so we only use this +* flag when we have the QMP based startup notification. +* */ +flexarray_append (dm_args,"-S" ); }else { flexarray_append(dm_args, GCSPRINTF("socket,id=libxl-cmd,"
@@ -2702,6 +2710,7 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev,libxl__dm_spawn_state *dmss =CONTAINER_OF (ev, *dmss, qmp);const libxl__json_object *o;const char *status; +const char *expected_state;libxl__ev_qmp_dispose (gc, ev);
@@ -2717,7 +2726,11 @@ static void device_model_qmp_cb(libxl__egc *egc, libxl__ev_qmp *ev, goto failed; } status =libxl__json_object_get_string (o); -if (strcmp (status,"running" )) { +if (!dmss->build_state->saved_state) + expected_state ="prelaunch" ; +else + expected_state ="paused" ; +if (strcmp (status, expected_state)) {LOGD (ERROR , ev->domid,"Unexpected QEMU status: %s" , status); rc =ERROR_NOT_READY ; goto failed;
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Anthony PERARD writes ("[PATCH v2 for-4.12] libxl: When restricted, start QEMU paused"):
Quoted 2 lines
Signed-off-by Anthony PERARD <[email protected]>Release-acked-by Juergen Gross <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel