libxl: Enable save/restore/migration of a restricted QEMU + libxl__ev_qmp_*
v7 cover libxl: Enable save/restore/migration of a restricted QEMU + libxl__ev_qmp_*
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.libxl-ev-qmp-v
Changes in v7:
plenty, with new patches
Changes in v6:
Implementation of libxl__ev_qmp_* functions have been squashed to a single patch. And with that, a lot of changes in order to make it simpler to read the implementation, have better error reporting and a few bug fix.
Checkout more detail changelog in the notes of each patch, as there is many.
Changes in v5:
Plenty of patch have been applied.
Other changes mostly are coding style and other typos. Some bug fixes.
Details can be found in patch notes.
I have left aside the change to cdrom_insert until I can found what to do with the userdata lock.
In order for libxl to be able to manage QEMU while it is restricted, a few changes are needed. We need a new way to get a startup notification from QEMU as xenstore may not be accessible when QEMU is ready. We also need to a different way to have QEMU save it's state and to insert cdrom as a restricted QEMU doesn't have access to the file system.
For both, we can use QMP, we can use it to query QEMU's status, and we can use it to send a file descriptor through which QEMU can save its state, or it can be a cdrom.
We take this opportunity to rewrite the QMP client, and this time been asynchronous, the result is libxl__ev_qmp_*.
The plat de résistance in this patch series start with patch "libxl: Design of an async API to issue QMP commands to QEMU" which implement libxl__ev_qmp_* functions to turn the QMP client into asynchronous mode.
This comes with changes that uses the new interface.
* "libxl: QEMU startup sync based on QMP"
which can use QMP to find out when QEMU as started.
this requires: "libxl_dm: Pre-open QMP socket for QEMU"
But that only works with dm_restrict=1 as explain in the patch.
* "libxl: Re-implement domain_suspend_device_model using libxl__ev_qmp" Which rewrite libxl__qmp_save(), and adds the ability to have QEMU save its state to a file descriptor which libxl will have openned.
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.libxl-ev-qmp-v
Cheers,
Anthony PERARD (14):
libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK
libxl_qmp: Separate QMP message generation from qmp_send_prepare
libxl_qmp: Change qmp_qemu_check_version to compare version
libxl: Add wrapper around libxl__json_object_to_json JSON
libxl: Design of an async API to issue QMP commands to QEMU
libxl_qmp: Implementation of libxl__ev_qmp_*
libxl_exec: Add libxl__spawn_initiate_failure
libxl: Add init/dispose of for libxl__domain_build_state
libxl_dm: Pre-open QMP socket for QEMU
libxl: Add dmss_init/dispose for libxl__dm_spawn_state
libxl: QEMU startup sync based on QMP
libxl_qmp: Store advertised QEMU version in libxl__ev_qmp
libxl: Change libxl__domain_suspend_device_model() to be async
libxl: Re-implement domain_suspend_device_model using libxl__ev_qmp
tools/libxl/libxl_create.c | 54 +- tools/libxl/libxl_dm.c | 142 ++++- tools/libxl/libxl_dom_suspend.c | 37 +- tools/libxl/libxl_exec.c | 11 +- tools/libxl/libxl_internal.h | 172 +++++- tools/libxl/libxl_qmp.c | 992 ++++++++++++++++++++++++++++++-- tools/libxl/libxl_types.idl | 7 + tools/libxl/libxl_utils.c | 25 +- 8 files changed, 1334 insertions(+), 106 deletions(-)
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 1/14 libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK
This patch change the behavior of libxl__sendmsg_fds to retry sendmsg on EINTR error.
This patch also allow a caller of libxl__sendmsg_fds to deal with EWOULDBLOCK. The function now requires to send only 1 byte of data so that when dealing with non-blocking fds a EWOULDBLOCK error would mean that the fds haven't been sent yet. Current caller already send only 1 byte.
---
Notes:
v7:
always assert datalen == 1, but only fail when sendmsg haven't send
everything (r != datalen)
check sendmsg return value on success as well (check for short write)tools/libxl/libxl_internal.h | 5 ++++- tools/libxl/libxl_utils.c | 25 ++++++++++++++++++++----- 2 files changed, 24 insertions(+), 6 deletions(-)
tools/libxl/libxl_internal.h +4 -1
@@ -1864,7 +1864,10 @@ _hidden void libxl__qmp_cleanup(libxl__gc *gc, uint32_t domid); _hidden int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,const libxl_domain_config *guest_config); -/* on failure, logs */ +/* `datalen` should be 1 byte +* When dealing with a non-blocking fd, it returns +* ERROR_NOT_READY on EWOULDBLOCK +* logs on other failures. */ int libxl__sendmsg_fds (libxl__gc *gc,int carrier,const void *data,size_t datalen,int nfds,const int fds[],const char *what);
tools/libxl/libxl_utils.c +20 -5
@@ -1070,6 +1070,8 @@ int libxl__sendmsg_fds(libxl__gc *gc, int carrier,struct iovec iov;int r; +assert (datalen ==1 ); + iov.iov_base = (void *)data; iov.iov_len = datalen;
@@ -1088,11 +1090,24 @@ int libxl__sendmsg_fds(libxl__gc *gc, int carrier, msg.msg_controllen = cmsg->cmsg_len; - r =sendmsg (carrier, &msg,0 ); -if (r <0 ) { -LOGE (ERROR ,"failed to send fd-carrying message (%s)" , what); -return ERROR_FAIL ; - } +while (1 ) { + r =sendmsg (carrier, &msg,0 ); +if (r <0 ) { +if (errno ==EINTR ) +continue ; +if (errno ==EWOULDBLOCK ) { +return ERROR_NOT_READY ; + } +LOGE (ERROR ,"failed to send fd-carrying message (%s)" , what); +return ERROR_FAIL ; + } +if (r != datalen) { +LOG (ERROR ,"sendmsg have written %d instead of %ld" , + r, datalen); +return ERROR_FAIL ; + } +break ; + };return 0 ; }
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 01/14] libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK
Anthony PERARD writes ("[PATCH v7 01/14] libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK"):
Quoted 8 lines
This patch change the behavior of
libxl__sendmsg_fdsto retry sendmsg on EINTR error.This patch also allow a caller of
libxl__sendmsg_fdsto deal with EWOULDBLOCK. The function now requires to send only 1 byte of data so that when dealing with non-blocking fds a EWOULDBLOCK error would mean that the fds haven't been sent yet. Current caller already send only 1 byte.
Even with a blocking fd, sendmsg may in principle report a short write. (So the commit message should talk about short writes in general.)
Quoted 5 lines
Notes:
v7: always assert datalen == 1, but only fail when sendmsg haven't send everything (r != datalen) check sendmsg return value on success as well (check for short write)
Rather than having a function which takes an argument which mandatorily takes the value 1, how about simply deleting that argument ?
You can do that in a followup patch if you like. In the meantime:
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 01/14] libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK
Quoted 10 lines · Ian Jackson
Anthony PERARD writes ("[PATCH v7 01/14] libxl: Enhance
libxl__sendmsg_fdsto deal with EINTR and EWOULDBLOCK"):Quoted 8 lines
This patch change the behavior of
libxl__sendmsg_fdsto retry sendmsg on EINTR error.This patch also allow a caller of
libxl__sendmsg_fdsto deal with EWOULDBLOCK. The function now requires to send only 1 byte of data so that when dealing with non-blocking fds a EWOULDBLOCK error would mean that the fds haven't been sent yet. Current caller already send only 1 byte.
Quoted 3 lines
Even with a blocking fd, sendmsg may in principle report a short write. (So the commit message should talk about short writes in general.)
I have rewrite the commit message like this:
This patch change the behavior of libxl__sendmsg_fds to retry sendmsg on EINTR error and return an error on short writes.
This patch allow a caller of libxl__sendmsg_fds to deal with EWOULDBLOCK and short writes. The function now requires to send only 1 byte of data so that when dealing with non-blocking fds a EWOULDBLOCK error would mean that the fds haven't been sent yet. Current caller already send only 1 byte.
Quoted 9 lines
Quoted 5 lines
Notes:
v7: always assert datalen == 1, but only fail when sendmsg haven't send everything (r != datalen) check sendmsg return value on success as well (check for short write)Rather than having a function which takes an argument which mandatorily takes the value 1, how about simply deleting that argument ?
I'll add a new patch to do that at the end of the series.
Quoted 3 lines
You can do that in a followup patch if you like. In the meantime:
Acked-by Ian Jackson <[email protected]>
Thanks,
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 01/14] libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK
Anthony PERARD writes ("Re: [PATCH v7 01/14] libxl: Enhance libxl__sendmsg_fds to deal with EINTR and EWOULDBLOCK"):
Quoted 7 lines · Ian Jackson
Quoted 3 lines
Even with a blocking fd, sendmsg may in principle report a short write. (So the commit message should talk about short writes in general.)
I have rewrite the commit message like this:
Quoted 8 lines
This patch change the behavior of
libxl__sendmsg_fdsto retry sendmsg on EINTR error and return an error on short writes.This patch allow a caller of
libxl__sendmsg_fdsto deal with EWOULDBLOCK and short writes. The function now requires to send only 1 byte of data so that when dealing with non-blocking fds a EWOULDBLOCK error would mean that the fds haven't been sent yet. Current caller already send only 1 byte.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 2/14 libxl_qmp: Separate QMP message generation from qmp_send_prepare
In reply to Anthony PERARD.. to be able to re-use qmp_prepare_cmd with libxl__ev_qmp.
This patch also add the QMP end of command '\r\n' into the generated string as every caller will needs this.
There should be no functional change.
---
Notes:
v7:
got rid of len_r in qmp_prepare_cmd parameters
callers will need to call strlen.
The returned value (ret) is now allocated within gc (instead of NOGC)
v6:
comment about ownership of buf
use lib__sprintf instead of two strncpy
v5:
rename qmp_prepare_qmp_cmd to qmp_prepare_cmd fix coding style
tools/libxl/libxl_qmp.c | 49 ++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 18 deletions(-)
tools/libxl/libxl_qmp.c +31 -18
@@ -571,17 +571,16 @@ static int qmp_next(libxl__gc *gc, libxl__qmp_handler *qmp)return rc; } -static char *qmp_send_prepare (libxl__gc *gc,libxl__qmp_handler *qmp, -const char *cmd,libxl__json_object *args, -qmp_callback_t callback,void *opaque, -qmp_request_context *context) +static char *qmp_prepare_cmd (libxl__gc *gc,const char *cmd, +const libxl__json_object *args, +int id) { -const unsigned char *buf =NULL ; -char *ret =NULL ; -libxl_yajl_length len =0 ; +yajl_gen hand =NULL ; +/* memory for 'buf' is owned by 'hand' */ +const unsigned char *buf; +libxl_yajl_length len;yajl_gen_status s; -yajl_gen hand; -callback_id_pair *elm =NULL ; +char *ret =NULL ; hand =libxl_yajl_gen_alloc (NULL );
@@ -598,7 +597,7 @@ static char *qmp_send_prepare(libxl__gc *gc, libxl__qmp_handler *qmp,libxl__yajl_gen_asciiz (hand,"execute" );libxl__yajl_gen_asciiz (hand, cmd);libxl__yajl_gen_asciiz (hand,"id" ); -yajl_gen_integer (hand, ++qmp->last_id_used); +yajl_gen_integer (hand, id);if (args) {libxl__yajl_gen_asciiz (hand,"arguments" );libxl__json_object_to_yajl_gen (gc, hand, args);
@@ -607,7 +606,27 @@ static char *qmp_send_prepare(libxl__gc *gc, libxl__qmp_handler *qmp, s =yajl_gen_get_buf (hand, &buf, &len); -if (s) { +if (s != yajl_gen_status_ok) + goto out; + + ret =libxl__sprintf (gc,"%*.*s\r\n" , (int )len, (int )len, buf); + +out: +yajl_gen_free (hand); +return ret; +} + +static char *qmp_send_prepare (libxl__gc *gc,libxl__qmp_handler *qmp, +const char *cmd,libxl__json_object *args, +qmp_callback_t callback,void *opaque, +qmp_request_context *context) +{ +char *buf; +callback_id_pair *elm; + + buf =qmp_prepare_cmd (gc, cmd, args, ++qmp->last_id_used); + +if (!buf) {LOGD (ERROR , qmp->domid,"Failed to generate a qmp command" ); goto out; }
@@ -623,13 +642,10 @@ static char *qmp_send_prepare(libxl__gc *gc, libxl__qmp_handler *qmp, elm->context = context;LIBXL_STAILQ_INSERT_TAIL (&qmp->callback_list, elm, next); - ret =libxl__strndup (gc, (const char *)buf, len); -LOGD (DEBUG , qmp->domid,"next qmp command: '%s'" , buf); out: -yajl_gen_free (hand); -return ret; +return buf; } static int qmp_send(libxl__qmp_handler *qmp,
@@ -650,9 +666,6 @@ static int qmp_send(libxl__qmp_handler *qmp,if (libxl_write_exactly (qmp->ctx, qmp->qmp_fd, buf,strlen (buf),"QMP command" ,"QMP socket" )) goto out; -if (libxl_write_exactly (qmp->ctx, qmp->qmp_fd,"\r\n" ,2 , -"CRLF" ,"QMP socket" )) - goto out; rc = qmp->last_id_used; out:
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 02/14] libxl_qmp: Separate QMP message generation from qmp_send_prepare
Anthony PERARD writes ("[PATCH v7 02/14] libxl_qmp: Separate QMP message generation from qmp_send_prepare"):
Quoted 6 lines
.. to be able to re-use
qmp_prepare_cmdwithlibxl__ev_qmp.This patch also add the QMP end of command '\r\n' into the generated string as every caller will needs this.
There should be no functional change.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 3/14 libxl_qmp: Change qmp_qemu_check_version to compare version
In reply to Anthony PERARDThis patch makes the function simpler to read. It also add the ability for a caller to tell if QEMU is newer or have the exact version.
---
Notes:
v6:
new patchtools/libxl/libxl_qmp.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
tools/libxl/libxl_qmp.c +21 -7
@@ -392,13 +392,27 @@ static int qmp_handle_response(libxl__gc *gc, libxl__qmp_handler *qmp,return 0 ; } -static bool qmp_qemu_check_version (libxl__qmp_handler *qmp,int major, -int minor,int micro) +/* +* return values: +* < 0 if qemu's version < asked version +* = 0 if qemu's version == asked version +* > 0 if qemu's version > asked version +*/ +static int qmp_qemu_compare_version (libxl__qmp_handler *qmp,int major, +int minor,int micro) { -return qmp->version.major > major || - (qmp->version.major == major && - (qmp->version.minor > minor || - (qmp->version.minor == minor && qmp->version.micro >= micro))); +#define CHECK_VERSION (level) do { \ + if (qmp->version.level > (level)) return +1; \ + if (qmp->version.level < (level)) return -1; \ +} while (0) + +CHECK_VERSION (major); +CHECK_VERSION (minor); +CHECK_VERSION (micro); + +#undef CHECK_VERSION + +return 0 ; }/*
@@ -1012,7 +1026,7 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename, bool live)/* live parameter was added to QEMU 2.11. It signal QEMU that the save * operation is for a live migration rather that for taking a snapshot. */ -if (qmp_qemu_check_version (qmp,2 ,11 ,0 )) +if (qmp_qemu_compare_version (qmp,2 ,11 ,0 ) >=0 )qmp_parameters_add_bool (gc, &args,"live" , live); rc = qmp_synchronous_send(qmp, "xen-save-devices-state", args,
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 4/14 libxl: Add wrapper around libxl__json_object_to_json JSON
In reply to Anthony PERARDThat wrapper is going to be used to safely log a json_object, as libxl__json_object_to_json return NULL on error. In the error case, JSON() will return an invalid json string.
---
Notes:
v7:
new patch
There are no user yet because the first users is going to be in "libxl_qmp: Implementation of libxl__ev_qmp_*" which is already a huge patch, that 3 lines might get lost.
tools/libxl/libxl_internal.h | 3 +++ 1 file changed, 3 insertions(+)
tools/libxl/libxl_internal.h +3 -0
@@ -2034,6 +2034,9 @@ _hidden libxl__json_object *libxl__json_parse(libxl__gc *gc_opt, const char *s); _hidden char *libxl__json_object_to_json(libxl__gc *gc,const libxl__json_object *args); +/* Always return a valid string, but invalid json on error. */ +#define JSON (o) \ + (libxl__json_object_to_json(gc, (o)) ? : "<invalid-json-object>")/* Based on /local/domain/$domid/dm-version xenstore key * default is qemu xen traditional */
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 04/14] libxl: Add wrapper around libxl__json_object_to_json JSON
Anthony PERARD writes ("[PATCH v7 04/14] libxl: Add wrapper around libxl__json_object_to_json JSON"):
Quoted 3 lines
That wrapper is going to be used to safely log a
json_object, aslibxl__json_object_to_jsonreturn NULL on error. In the error case,JSON()will return an invalid json string.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 5/14 libxl: Design of an async API to issue QMP commands to QEMU
In reply to Anthony PERARDAll the functions will be implemented in later patches.
This patch includes the API that libxl can use to send QMP commands to QEMU.
---
Notes:
v7:
acked, but with:
fd field renamed to payload_fd
libxl__ao ao field added
added to _send that callback isn't called synchronously.
v6:
use libxl_domid type for domid instead of plain uin32_t avoid the work "chained", rewrite the paragraph about sending one
cmd after another
Rewrite the comment about the callback, and explain that on error,
the ev may be Idle or may still be Connected
Change the carefd to a simple int (field cfd -> fd)
v5:
some changes in the comment
tools/libxl/libxl_internal.h | 76 +++++++++++++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 2 deletions(-)
tools/libxl/libxl_internal.h +74 -2
@@ -186,6 +186,8 @@ typedef struct libxl__ao libxl__ao;typedef struct libxl__aop_occurred libxl__aop_occurred ;typedef struct libxl__osevent_hook_nexus libxl__osevent_hook_nexus ;typedef struct libxl__osevent_hook_nexi libxl__osevent_hook_nexi ; +typedef struct libxl__json_object libxl__json_object ; +typedef struct libxl__carefd libxl__carefd ;typedef struct libxl__domain_create_state libxl__domain_create_state ; typedef void libxl__domain_create_cb(struct libxl__egc *egc,
@@ -349,6 +351,76 @@ struct libxl__ev_child {LIBXL_LIST_ENTRY (struct libxl__ev_child ) entry; }; +/* +* QMP asynchronous calls +* +* This facility allows a command to be sent to QEMU, and the response +* to be handed to a callback function. +* +* Commands can be submited one after an other with the same +* connection (e.g. the result from the "add-fd" command need to be +* use in a follow-up command before disconnecting from QMP). A +* libxl__ev_qmp can be reused when the callback is been called in +* order to use the same connection. +* +* Only one connection at a time can be made to one QEMU, so avoid +* keeping a libxl__ev_qmp Connected for to long and call +* libxl__ev_qmp_dispose as soon as it is not needed anymore. +* +* Possible states of a libxl__ev_qmp: +* Undefined +* Might contain anything. +* Idle +* Struct contents are defined enough to pass to any +* libxl__ev_qmp_* function. +* The struct does not contain references to any allocated private +* resources so can be thrown away. +* Active +* Currently waiting for the callback to be called. +* _dispose must be called to reclaim resources. +* Connected +* Struct contain allocated ressources. +* Calling _send() with this same ev will use the same QMP connection. +* _dispose() must be called to reclaim resources. +* +* libxl__ev_qmp_init: Undefined/Idle -> Idle +* +* libxl__ev_qmp_send: Idle/Connected -> Active (on error: Idle) +* Sends a command to QEMU. +* callback will be called when a response is received or when an +* error as occured. +* callback isn't called synchronously. +* +* libxl__ev_qmp_dispose: Connected/Active/Idle -> Idle +* +* callback: When called: Active -> Connected (on error: Idle/Connected) +* When called, ev is Connected and can be reused or disposed of. +* On error, the callback is called with response == NULL and the +* error code in rc. The new state of ev depending on the value of rc: +* - rc == ERROR_QMP_*: This is an error associated with the cmd to +* run, ev is Connected. +* - otherwise: An other error happend, ev is now Idle. +* The callback is only called once. +*/ +typedef struct libxl__ev_qmp libxl__ev_qmp ; +typedef void libxl__ev_qmp_callback (libxl__egc *egc,libxl__ev_qmp *ev, +const libxl__json_object *response, +int rc); + +_hidden void libxl__ev_qmp_init(libxl__ev_qmp *ev); +_hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, +const char *cmd,libxl__json_object *args); +_hidden void libxl__ev_qmp_dispose(libxl__gc *gc, libxl__ev_qmp *ev); + +struct libxl__ev_qmp { +/* caller should include this in their own struct */ +/* caller must fill these in, and they must all remain valid */ +libxl__ao *ao; +libxl_domid domid; +libxl__ev_qmp_callback *callback; +int payload_fd;/* set to send a fd with the command, -1 otherwise */ +}; +/* * evgen structures, which are the state we use for generating
@@ -1897,7 +1969,7 @@ typedef enum { JSON_ANY = 255 /* this is a mask of all values above, adjust as needed */ } libxl__json_node_type; -typedef struct libxl__json_object { +struct libxl__json_object {libxl__json_node_type type;union {bool b;
@@ -1910,7 +1982,7 @@ typedef struct libxl__json_object {flexarray_t *map; } u;struct libxl__json_object *parent; -} libxl__json_object; +}; typedef int (*libxl__json_parse_callback)(libxl__gc *gc, libxl__json_object *o,
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 6/14 libxl_qmp: Implementation of libxl__ev_qmp_*
In reply to Anthony PERARDThis patch implement the API libxl__ev_qmp documented in the previous patch, "libxl: Design of an async API to issue QMP commands to QEMU".
Since this API is to interact with QEMU via the QMP protocol, it also implement a QMP client. The specification for the QEMU Machine Protocol (QMP) can be found in the QEMU repository at:
https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/qmp-spec.txt
---
Notes:
v7:
- Make use of the new ao field filled by the caller
(no more free(tx_buf))
- Have reworked the state table to be more accurate, easier to read and with better desc of the waiting_reply state with sub-states as well as better description of the state of the ev_fd efd.
- There is state transition changes, now we have cap.neg -> waiting_reply (before it was cap.neg -> connected -> waiting_reply), that makes the internal connected state the same as the external Connected state.
- there are now 3 different id variable instead of a single one, the currently sent one id, the next one next_id, and the one associated with the caller's command msg_id, id will always be the for the next expected message from QEMU.
- qmp_ev_fd_callback now check for POLLERR, and read async error with getsockopt(SO_ERROR).
- qmp_ev_callback_readable takes care once again to also parse messages. But it now first attempt to read messages that would be in the rx_buf before reading from the socket.
- Some functions state changes have been updated, qmp_ev_ensure_reading_writing, qmp_ev_set_state, qmp_ev_callback_writable, qmp_ev_callback_readable
- Have cleaned up qmp_ev_ensure_reading_writing
- Have cleaned up qmp_error_class_to_libxl_error_code, and log unknown error classes.
- In qmp_ev_callback_readable, the size increase of the buffer have been reworked to update both rx_buf_size and rx_buf at the same time, and remove the use of max()
- Return value of qmp_prepare_cmd calles is checked and logged on error.
- comments in RHS of struct
- New qmp_ev_tx_buf_clear func
- Add a link to the QMP spec.
- update callers of qmp_prepare_cmd, which doesn't provide the string lenght anymore also remove ev->msg_len
- rename internal fields qmp_cfd, qmp_efd, qmp_state to cfd, efd and state.
- change qmp_ev_connect to only allow disconnect on entry
- squash qmp_ev_prepare_cmd into libxl__ev_qmp_send
- reduce max rx buffer size to 1M
query-vcpus with 71 cpus active yield 14484 bytes.
- use JSON to print libxl__json_object
v6.2:
Add definition of the internal broken state updated comments about states
v6.1:
Adding some comment about possible internal state changes
v6:
This is a squash of 7 commits on the previous version:
- libxl_qmp: Connect to QMP socket
- libxl_qmp: Implement fd callback and read data
- libxl_qmp: Parse JSON input from QMP
- libxl_qmp: Prepare the command to be sent
- libxl_qmp: Handle write to QMP socket
- libxl_qmp: Handle messages from QEMU
- libxl_qmp: Respond to QMP greeting
General rework of the implementation.
Added more comment, with a description of allowed internal states. Check for EINPROGRESS after connect().
Read until EWOULDBLOCK.
Handle EWOULDBLOCK on write and sendmsg.
Using memmem instead of strstr.
Using memmove instead of having an offset in rx_buf. Rework buffer allocation
Don't feed \r into json parser anymore
Add a check for a maximum RX buffer size
Added more error messages
New error code ERROR_PROTOCOL_ERROR_QMP
Rewrite conversion of QMP ErrorClass to libxl_error code Added helpers: qmp_ev_ensure_reading_writing, qmp_ev_set_state Split some functions, squash others
Added ev->msg* to store generated user command as tx_buf is used during
connection (for qmp_capabilities)
Remove qmp_state_greeting
Added qmp_state_waiting_reply
v5:
nits
use a define instead of a static int for QMP_CAPABILITY_NEGOCIATION_MSGID use a switch in qmp_ev_callback_writable to check qmp_state Add a description of the different value of libxl__qmp_state enum. some cleanup
remove read loop that only handled EINTR, simply return initialize len and s at declaration time
remove old comment
rename buf_fd to send_fd
Adding default:abort() in qmp_ev_handle_message.
v4:
remove use of a linked list of receive buffer, and use realloc instead. simplification of the patch due to use of a single allocated space for the receive buffer.
tools/libxl/libxl_internal.h | 34 ++ tools/libxl/libxl_qmp.c | 740 +++++++++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + 3 files changed, 780 insertions(+)
tools/libxl/libxl_internal.h +34 -0
@@ -412,6 +412,19 @@ _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, const char *cmd, libxl__json_object *args); _hidden void libxl__ev_qmp_dispose(libxl__gc *gc, libxl__ev_qmp *ev); +typedef enum { +/* initial state */ + qmp_state_disconnected =1 , +/* connected to QMP socket, waiting for greeting message */ + qmp_state_connecting, +/* qmp_capabilities command sent, waiting for reply */ + qmp_state_capability_negotiation, +/* sending user's cmd and waiting for reply */ + qmp_state_waiting_reply, +/* ready to send commands */ + qmp_state_connected, +}libxl__qmp_state ; +struct libxl__ev_qmp {/* caller should include this in their own struct */ /* caller must fill these in, and they must all remain valid */
@@ -419,6 +432,27 @@ struct libxl__ev_qmp {libxl_domid domid;libxl__ev_qmp_callback *callback;int payload_fd;/* set to send a fd with the command, -1 otherwise */ + +/* +* remaining fields are private to libxl_ev_qmp_* +*/ + +libxl__carefd *cfd; +libxl__ev_fd efd; +libxl__qmp_state state; +int id; +int next_id;/* next id to use */ +/* receive buffer */ +char *rx_buf; +size_t rx_buf_size;/* current allocated size */ +size_t rx_buf_used;/* actual data in the buffer */ +/* sending buffer */ +char *tx_buf; +size_t tx_buf_len;/* tx_buf size */ +size_t tx_buf_off;/* already sent */ +/* The message to send when ready */ +char *msg; +int msg_id; };
tools/libxl/libxl_qmp.c +740 -0
757 lines not on this page · raw message
tools/libxl/libxl_types.idl +6 -0
@@ -69,6 +69,12 @@ libxl_error = Enumeration("error", [ (-23, "NOTFOUND"), (-24, "DOMAIN_DESTROYED"), # Target domain ceased to exist during op (-25, "FEATURE_REMOVED"), # For functionality that has been removed + (-26, "PROTOCOL_ERROR_QMP"), + (-27, "UNKNOWN_QMP_ERROR"), + (-28, "QMP_GENERIC_ERROR"), # unspecified qmp error + (-29, "QMP_COMMAND_NOT_FOUND"), # the requested command has not been found + (-30, "QMP_DEVICE_NOT_ACTIVE"), # a device has failed to be become active + (-31, "QMP_DEVICE_NOT_FOUND"), # the requested device has not been found ], value_namespace = "") libxl_domain_type = Enumeration("domain_type", [
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
Anthony PERARD writes ("[PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*"):
Quoted 7 lines
This patch implement the API
libxl__ev_qmpdocumented in the previous patch, "libxl: Design of an async API to issue QMP commands to QEMU".Since this API is to interact with QEMU via the QMP protocol, it also implement a QMP client. The specification for the QEMU Machine Protocol (QMP) can be found in the QEMU repository at:
https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/qmp-spec.txt
Thanks.
I have only fairly minor comments now. The biggest one remaining is about the use of EGC_GC which I think probably wants to become STATE_AO_GC throughout. See below...
tools/libxl/libxl_internal.h _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, · line 414
Quoted 8 lines
tools/libxl/libxl_internal.h +2
-0@@ -412,6 +412,19 @@ _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, +/* receive buffer */ +char *rx_buf;
rx_buf needs a comment saying it comes from NOGC since otherwise one would assume it came from the ao gc like the other buffers.
(Could it come from the ao gc instead?)
tools/libxl/libxl_internal.h _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, · line 417
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+static void qmp_ev_ensure_reading_writing (libxl__gc *gc,libxl__ev_qmp *ev) +/* Update the state of `efd` to match the permited state */ +{
This function is legal only in states other than disconnected. Needs to be documented.
Quoted 13 lines
tools/libxl/libxl_internal.h +13
-0+static void qmp_ev_set_state (libxl__gc *gc,libxl__ev_qmp *ev, +libxl__qmp_state new_state) +/* on entry: !broken and !disconnected */ +{ +switch (new_state) { +case qmp_state_disconnected: +break ; +case qmp_state_connecting: +assert (ev->state == qmp_state_disconnected); +break ; +case qmp_state_capability_negotiation: +assert (ev->state == qmp_state_connecting); +break ;
tools/libxl/libxl_internal.h _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev,
Quoted 10 lines
tools/libxl/libxl_internal.h +10
-0+case qmp_state_waiting_reply: +assert (ev->state == qmp_state_capability_negotiation || + ev->state == qmp_state_connected); +break ; +case qmp_state_connected: +assert (ev->state == qmp_state_waiting_reply); +break ; + } + + ev->state = new_state;
I think this function needs to update efd ? What am I missing ?
The comment doesn't say what the output state is but naturally I assume that it is precisely new_state, and not some transitional mixture. If it is intended to produce a transitional mixture that ought to be documented.
For a concrete example: if on entry, with new_state==disconnected, we are `connecting' then: efd will be looking for POLLIN. But it needs to become Idle.
tools/libxl/libxl_internal.h
Quoted 6 lines
tools/libxl/libxl_internal.h +6
-0+/* Setup connection */ + +static int qmp_ev_connect (libxl__gc *gc,libxl__ev_qmp *ev) +/* disconnected -> connecting but with `msg` free +* on error: broken */ +{
This function looks fine to me. However, earlier I wrote this:
Contrary to the state description, this function does not transition rx_buf from free to used. However, I think this would probably be more easily remedied by changing the definition of used' to permit NULL/0/0. You might want to use a different word to used', `inuse' perhaps ?
This is still true. That is, your state description for connecting' says that rx_buf is used'. And your description lower about what rx_buf being used' means says that rx_buf must be allocated'.
I think this would probably be best resolved by writing:
* free used
- * rx_buf NULL allocated + * rx_buf NULL NULL or allocated * rx_buf_size 0 allocation size of `rx_buf` * rx_buf_used 0 <= rx_buf_size, actual data in the * buffer
Ie just to change the internal spec. I am going to assume for the rest of the review that the code is right and the internal spec will be updated. (I don't think it is necessary to change the descriptions of rx_buf_size and rx_buf_used; it will be clear that the `allocation size' of a NULL must be 0.)
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+/* QMP FD callbacks */ + +static void qmp_ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev_fd,
...
Quoted 9 lines
tools/libxl/libxl_internal.h +9
-0+static int qmp_ev_callback_writable (libxl__gc *gc, +libxl__ev_qmp *ev,int fd) +/* on entry: !disconnected +* on return, one of these state transition: +* waiting_reply (with msg set) -> waiting_reply (with msg free) +* tx_buf set -> same state or tx_buf free +* tx_buf free -> no state change +* on error: broken */ +{
...
tools/libxl/libxl_internal.h
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+assert (ev->tx_buf); +if (!ev->tx_buf) +return 0 ;
I think the if is vestigial.
tools/libxl/libxl_internal.h
Quoted 12 lines
tools/libxl/libxl_internal.h +12
-0+while (ev->tx_buf_off < ev->tx_buf_len) { + r =write (fd, ev->tx_buf + ev->tx_buf_off, + ev->tx_buf_len - ev->tx_buf_off); +if (r <0 ) { +if (errno ==EINTR ) +continue ; +if (errno ==EWOULDBLOCK ) +break ; +LOGED (ERROR , ev->domid,"failed to write to QMP socket" ); +return ERROR_FAIL ; + } + ev->tx_buf_off += r;
Can you assert that the value of r was within range ? (Perhaps this is paranoia on my part, but, still.)
tools/libxl/libxl_internal.h
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+static int qmp_ev_get_next_msg (libxl__egc *egc,libxl__ev_qmp *ev, +libxl__json_object **o_r) +/* Find a JSON object and store it in o_r. +* return ERROR_NOTFOUND if no object is found. +* `o_r` is allocated within `egc`.
Why not allocate o_r from within AO_GC ?
ISTM that taking it from egc is a beartrap. If you do want to allocate it from egc, this should definitely be documented in the internal public api for libxl__ev_qmp_callback. Right now a caller might well reasonably assume that the libxl__json_object *response they get is useable for the whole ao. Indeed future callers might even need that semantic.
To do this, use STATE_AO_GC instead of EGC_GC. TBH I think you should probably do that throughout.
tools/libxl/libxl_internal.h
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+static int qmp_ev_parse_error_messages (libxl__egc *egc, +libxl__ev_qmp *ev, +const libxl__json_object *resp)
Personally I would have preferred to read this and qmp_ev_handle_message in the other order, so I would swap them. Up to you.
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+/* no state change */ +{ +EGC_GC ; +int rc; +const char *s;
tools/libxl/libxl_internal.h
Quoted 10 lines
tools/libxl/libxl_internal.h +10
-0+const libxl__json_object *o; +const libxl__json_object *err; + +/* +* { "error": { "class": string, "desc": string } } +*/ + + err =libxl__json_map_get ("error" , resp,JSON_MAP ); + + o =libxl__json_map_get ("class" , err,JSON_STRING );
I wondered: surely err can be NULL ? I didn't find any docs saying that it couldn't; nor that it tolerated NULL for o on input.
However, reading the implementation I see that libxl__json_map_get calls libxl__json_object_is_map which does indeed handle o==0. Could you perhaps add a comment (in libxl_internal.h near libxl__json_map_get) et al about this ?
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+static int qmp_ev_handle_message (libxl__egc *egc, +libxl__ev_qmp *ev, +const libxl__json_object *resp)
...
tools/libxl/libxl_internal.h
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+/* Prepare next message to send */ +assert (!ev->tx_buf); + ev->id = ev->next_id++; + buf =qmp_prepare_cmd (libxl__ao_inprogress_gc (ev->ao), +"qmp_capabilities" ,NULL , ev->id);
Erk, I don't like the open coded call to libxl__ao_inprogress_gc. I'm becoming more convinced you just wanted AO_GC or STATE_AO_GC everywhere.
tools/libxl/libxl_internal.h
Quoted 12 lines
tools/libxl/libxl_internal.h +12
-0+void libxl__ev_qmp_dispose (libxl__gc *gc,libxl__ev_qmp *ev) +/* * -> disconnected */ +{ +LOGD (DEBUG , ev->domid," ev %p" , ev); + +free (ev->rx_buf); + +libxl__ev_fd_deregister (gc, &ev->efd); +libxl__carefd_close (ev->cfd); + +libxl__ev_qmp_init (ev); +}
It's a small point, but it would be nicer to move the free of rx_buf nearer the call to libxl__ev_qmp_init which zeroes it.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
Ian Jackson writes ("Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*"):
Quoted 3 lines
I have only fairly minor comments now. The biggest one remaining is about the use of
EGC_GCwhich I think probably wants to becomeSTATE_AO_GCthroughout. See below...
I realise that I should state explicitly that libxl__ev_qmp_callback must still get an egc.
The egc should be passed through all the layers, but its gc should generally not be used.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
In reply to Ian JacksonQuoted 22 lines · Ian Jackson
Anthony PERARD writes ("[PATCH v7 06/14]
libxl_qmp: Implementation of libxl__ev_qmp_*"):Quoted 7 lines
This patch implement the API
libxl__ev_qmpdocumented in the previous patch, "libxl: Design of an async API to issue QMP commands to QEMU".Since this API is to interact with QEMU via the QMP protocol, it also implement a QMP client. The specification for the QEMU Machine Protocol (QMP) can be found in the QEMU repository at:
https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/qmp-spec.txtThanks.
I have only fairly minor comments now. The biggest one remaining is about the use of
EGC_GCwhich I think probably wants to becomeSTATE_AO_GCthroughout. See below...Quoted 5 lines
tools/libxl/libxl_internal.h +0
-0@@ -412,6 +412,19 @@ _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev,
Quoted 7 lines
tools/libxl/libxl_internal.h struct libxl__ev_qmp { · line 446
Quoted 2 lines
tools/libxl/libxl_internal.h +2
-0+/* receive buffer */ +char *rx_buf;
rx_bufneeds a comment saying it comes from NOGC since otherwise one would assume it came from the ao gc like the other buffers.(Could it come from the ao gc instead?)
I guess it's fine to have the rx_buf comes from ao gc. The buffer isn't that big (maybe 100kB in worse cases).
Quoted 7 lines
tools/libxl/libxl_internal.h _hidden int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev, · line 416
Quoted 4 lines
tools/libxl/libxl_internal.h +3
-0+static void qmp_ev_ensure_reading_writing (libxl__gc *gc,libxl__ev_qmp *ev) +/* Update the state of `efd` to match the permited state */ +{This function is legal only in states other than disconnected. Needs to be documented.
Will do.
Quoted 17 lines
Quoted 17 lines
tools/libxl/libxl_internal.h +17
-0+static void qmp_ev_set_state (libxl__gc *gc,libxl__ev_qmp *ev, +libxl__qmp_state new_state) +/* on entry: !broken and !disconnected */ +{ +switch (new_state) { +case qmp_state_disconnected: +break ; +case qmp_state_connecting: +assert (ev->state == qmp_state_disconnected); +break ; +case qmp_state_capability_negotiation: +assert (ev->state == qmp_state_connecting); +break ; +case qmp_state_waiting_reply: +assert (ev->state == qmp_state_capability_negotiation || + ev->state == qmp_state_connected); +break ;
Quoted 8 lines
tools/libxl/libxl_internal.h
Quoted 6 lines
tools/libxl/libxl_internal.h +6
-0+case qmp_state_connected: +assert (ev->state == qmp_state_waiting_reply); +break ; + } + + ev->state = new_state;I think this function needs to update efd ? What am I missing ?
Yes, I think it's fine to call qmp_ev_ensure_reading_writing here ( or even inline it) and qmp_ev_ensure_reading_writing won't needs to be call from other places.
Quoted 8 lines
The comment doesn't say what the output state is but naturally I assume that it is precisely
new_state, and not some transitional mixture. If it is intended to produce a transitional mixture that ought to be documented.For a concrete example: if on entry, with new_state==disconnected, we are `connecting' then: efd will be looking for POLLIN. But it needs to become Idle.
Once I update efd with this function, I think qmp_ev_set_state should always output precisely new_state. But new_state alone might not be enough in few cases (waiting_reply) to describe a full state, but it will be one of the possible internal state as describe in the state documentation of the implementation.
Quoted 20 lines
tools/libxl/libxl_internal.h
Quoted 7 lines
tools/libxl/libxl_internal.h +6
-0+/* Setup connection */ + +static int qmp_ev_connect (libxl__gc *gc,libxl__ev_qmp *ev) +/* disconnected -> connecting but with `msg` free +* on error: broken */ +{This function looks fine to me. However, earlier I wrote this:
Contrary to the state description, this function does not transition
rx_buffrom free to used. However, I think this would probably be more easily remedied by changing the definition ofused' to permit NULL/0/0. You might want to use a different word toused', `inuse' perhaps ?This is still true. That is, your state description for
connecting' says that rx_buf isused'. And your description lower about whatrx_bufbeingused' means says that rx_buf must beallocated'.
Quoted 8 lines
I think this would probably be best resolved by writing:
* free used
- * rx_buf NULL allocated + * rx_buf NULL NULL or allocated * rx_buf_size 0 allocation size of `rx_buf` * rx_buf_used 0 <= rx_buf_size, actual data in the * buffer
I'll make this change.
Quoted 22 lines
Ie just to change the internal spec. I am going to assume for the rest of the review that the code is right and the internal spec will be updated. (I don't think it is necessary to change the descriptions of
rx_buf_sizeandrx_buf_used; it will be clear that the `allocation size' of a NULL must be 0.)Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+/* QMP FD callbacks */ + +static void qmp_ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev_fd,...
Quoted 9 lines
tools/libxl/libxl_internal.h +9
-0+static int qmp_ev_callback_writable (libxl__gc *gc, +libxl__ev_qmp *ev,int fd) +/* on entry: !disconnected +* on return, one of these state transition: +* waiting_reply (with msg set) -> waiting_reply (with msg free) +* tx_buf set -> same state or tx_buf free +* tx_buf free -> no state change +* on error: broken */ +{...
Quoted 5 lines
tools/libxl/libxl_internal.h
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+assert (ev->tx_buf); +if (!ev->tx_buf) +return 0 ;I think the if is vestigial.
I'll remove it.
Quoted 13 lines
tools/libxl/libxl_internal.h
Quoted 12 lines
tools/libxl/libxl_internal.h +12
-0+while (ev->tx_buf_off < ev->tx_buf_len) { + r =write (fd, ev->tx_buf + ev->tx_buf_off, + ev->tx_buf_len - ev->tx_buf_off); +if (r <0 ) { +if (errno ==EINTR ) +continue ; +if (errno ==EWOULDBLOCK ) +break ; +LOGED (ERROR , ev->domid,"failed to write to QMP socket" ); +return ERROR_FAIL ; + } + ev->tx_buf_off += r;
Quoted 2 lines
Can you assert that the value of r was within range ? (Perhaps this is paranoia on my part, but, still.)
I do assert the value of tx_buf_off which does include r. But I can add
`assert(r > 0 && r <= (ev->rx_buf_size - ev->rx_buf_used));'.
Quoted 6 lines
tools/libxl/libxl_internal.h
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+static int qmp_ev_get_next_msg (libxl__egc *egc,libxl__ev_qmp *ev, +libxl__json_object **o_r) +/* Find a JSON object and store it in o_r. +* return ERROR_NOTFOUND if no object is found. +* `o_r` is allocated within `egc`.
Quoted 8 lines
Why not allocate
o_rfrom withinAO_GC?ISTM that taking it from egc is a beartrap. If you do want to allocate it from egc, this should definitely be documented in the internal public api for
libxl__ev_qmp_callback. Right now a caller might well reasonably assume that thelibxl__json_object*response they get is useable for the whole ao. Indeed future callers might even need that semantic.
Indeed, that can be an issue. I'll make the changes to use ao gc instead of egc.
Quoted 25 lines
To do this, use
STATE_AO_GCinstead ofEGC_GC. TBH I think you should probably do that throughout.Quoted 8 lines
tools/libxl/libxl_internal.h +8
-0+static int qmp_ev_parse_error_messages (libxl__egc *egc, +libxl__ev_qmp *ev, +const libxl__json_object *resp) +/* no state change */ +{ +EGC_GC ; +int rc; +const char *s;tools/libxl/libxl_internal.h
Quoted 10 lines
tools/libxl/libxl_internal.h +10
-0+const libxl__json_object *o; +const libxl__json_object *err; + +/* +* { "error": { "class": string, "desc": string } } +*/ + + err =libxl__json_map_get ("error" , resp,JSON_MAP ); + + o =libxl__json_map_get ("class" , err,JSON_STRING );I wondered: surely err can be NULL ? I didn't find any docs saying that it couldn't; nor that it tolerated NULL for o on input.
Quoted 4 lines
However, reading the implementation I see that
libxl__json_map_getcallslibxl__json_object_is_mapwhich does indeed handle o==0. Could you perhaps add a comment (inlibxl_internal.hnearlibxl__json_map_get) et al about this ?
I'll add the comments.
Quoted 14 lines
Quoted 3 lines
tools/libxl/libxl_internal.h +3
-0+static int qmp_ev_handle_message (libxl__egc *egc, +libxl__ev_qmp *ev, +const libxl__json_object *resp)...
tools/libxl/libxl_internal.h
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+/* Prepare next message to send */ +assert (!ev->tx_buf); + ev->id = ev->next_id++; + buf =qmp_prepare_cmd (libxl__ao_inprogress_gc (ev->ao), +"qmp_capabilities" ,NULL , ev->id);Erk, I don't like the open coded call to
libxl__ao_inprogress_gc. I'm becoming more convinced you just wantedAO_GCorSTATE_AO_GCeverywhere.
Quoted 15 lines
tools/libxl/libxl_internal.h
Quoted 12 lines
tools/libxl/libxl_internal.h +12
-0+void libxl__ev_qmp_dispose (libxl__gc *gc,libxl__ev_qmp *ev) +/* * -> disconnected */ +{ +LOGD (DEBUG , ev->domid," ev %p" , ev); + +free (ev->rx_buf); + +libxl__ev_fd_deregister (gc, &ev->efd); +libxl__carefd_close (ev->cfd); + +libxl__ev_qmp_init (ev); +}It's a small point, but it would be nicer to move the free of
rx_bufnearer the call tolibxl__ev_qmp_initwhich zeroes it.
As rx_buf is probably going to be allocated from ao gc, the free won't be needed anymore. Or, I could realloc with a new size of 0:
libxl__realloc(maybe_gc, ev->rx_buf, 0);
and that would free the memory. I've check realloc, libxl__realloc and libxl__free_all, and they all seems to do the right things when the new size is 0.
Thanks,
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
Quoted 6 lines · Anthony PERARD
Quoted 5 lines · Ian Jackson
Anthony PERARD writes ("[PATCH v7 06/14]
libxl_qmp: Implementation of libxl__ev_qmp_*"):Quoted 4 lines
tools/libxl/libxl_qmp.c +4
-0+static void qmp_ev_set_state (libxl__gc *gc,libxl__ev_qmp *ev, +libxl__qmp_state new_state) +/* on entry: !broken and !disconnected */ +{
[...]
Quoted 8 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + ev->state = new_state;I think this function needs to update efd ? What am I missing ?
Yes, I think it's fine to call
qmp_ev_ensure_reading_writinghere ( or even inline it) andqmp_ev_ensure_reading_writingwon't needs to be call from other places.
Actually, I still need to update efd after having transmit data to QEMU, as ev->state isn't updated in that case. So I'll have efd update when qmp_ev_set_state is called, and when the tx_buf is maybe sent.
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
In reply to Anthony PERARD(Skipping the parts where you were just agreeing with me...)
Anthony PERARD writes ("Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*"):
Quoted 6 lines · Ian Jackson
Quoted 5 lines
Quoted 5 lines
tools/libxl/libxl_internal.h +4
-0+static void qmp_ev_set_state (libxl__gc *gc,libxl__ev_qmp *ev, +libxl__qmp_state new_state) +/* on entry: !broken and !disconnected */ +{
...
Quoted 5 lines
Quoted 1 line
I think this function needs to update efd ? What am I missing ?
Yes, I think it's fine to call
qmp_ev_ensure_reading_writinghere ( or even inline it) andqmp_ev_ensure_reading_writingwon't needs to be call from other places.
Sure.
Quoted 9 lines
Quoted 8 lines
The comment doesn't say what the output state is but naturally I assume that it is precisely
new_state, and not some transitional mixture. If it is intended to produce a transitional mixture that ought to be documented.For a concrete example: if on entry, with new_state==disconnected, we are `connecting' then: efd will be looking for POLLIN. But it needs to become Idle.
Quoted 5 lines
Once I update efd with this function, I think
qmp_ev_set_stateshould always output preciselynew_state. Butnew_statealone might not be enough in few cases (waiting_reply) to describe a full state, but it will be one of the possible internal state as describe in the state documentation of the implementation.
That SGTM.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
tools/libxl/libxl_internal.h +12
-0+while (ev->tx_buf_off < ev->tx_buf_len) { + r =write (fd, ev->tx_buf + ev->tx_buf_off, + ev->tx_buf_len - ev->tx_buf_off); +if (r <0 ) { +if (errno ==EINTR ) +continue ; +if (errno ==EWOULDBLOCK ) +break ; +LOGED (ERROR , ev->domid,"failed to write to QMP socket" ); +return ERROR_FAIL ; + } + ev->tx_buf_off += r;
Quoted 6 lines
Quoted 2 lines
Can you assert that the value of r was within range ? (Perhaps this is paranoia on my part, but, still.)
I do assert the value of
tx_buf_offwhich does include r. But I can add
`assert(r > 0 && r <= (ev->rx_buf_size-ev->rx_buf_used));'.
Yes. As I say worrying that write() might return something insane is perhaps paranoia.
Do you mean `ev->tx_buf_len - ev->tx_buf_off' like in the write call ? I would factor the expression out into a variable so you only have to write it once.
Quoted 11 lines
Quoted 8 lines
Why not allocate
o_rfrom withinAO_GC?ISTM that taking it from egc is a beartrap. If you do want to allocate it from egc, this should definitely be documented in the internal public api for
libxl__ev_qmp_callback. Right now a caller might well reasonably assume that thelibxl__json_object*response they get is useable for the whole ao. Indeed future callers might even need that semantic.Indeed, that can be an issue. I'll make the changes to use ao gc instead of egc.
Thanks.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
tools/libxl/libxl_internal.h +7
-0+void libxl__ev_qmp_dispose (libxl__gc *gc,libxl__ev_qmp *ev) +/* * -> disconnected */ +{ +LOGD (DEBUG , ev->domid," ev %p" , ev); + +free (ev->rx_buf); +Quoted 8 lines
tools/libxl/libxl_internal.h
Quoted 5 lines
tools/libxl/libxl_internal.h +5
-0+libxl__ev_fd_deregister (gc, &ev->efd); +libxl__carefd_close (ev->cfd); + +libxl__ev_qmp_init (ev); +}It's a small point, but it would be nicer to move the free of
rx_bufnearer the call tolibxl__ev_qmp_initwhich zeroes it.
Quoted 2 lines
As
rx_bufis probably going to be allocated from ao gc, the free won't be needed anymore.
Yay, less code.
Quoted 5 lines
Or, I could realloc with a new size of 0:
libxl__realloc(maybe_gc, ev->rx_buf, 0);and that would free the memory. I've check realloc,
libxl__reallocandlibxl__free_all, and they all seems to do the right things when the new size is 0.
We can probably afford to leak it for the duration of the ao.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 06/14] libxl_qmp: Implementation of libxl__ev_qmp_*
Quoted 18 lines · Ian Jackson
Anthony PERARD writes ("Re: [PATCH v7 06/14]
libxl_qmp: Implementation of libxl__ev_qmp_*"):Quoted 17 lines · Ian Jackson
Quoted 15 lines
tools/libxl/libxl_qmp.c int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid, · line 1655
Quoted 12 lines
tools/libxl/libxl_qmp.c +12
-0+while (ev->tx_buf_off < ev->tx_buf_len) { + r =write (fd, ev->tx_buf + ev->tx_buf_off, + ev->tx_buf_len - ev->tx_buf_off); +if (r <0 ) { +if (errno ==EINTR ) +continue ; +if (errno ==EWOULDBLOCK ) +break ; +LOGED (ERROR , ev->domid,"failed to write to QMP socket" ); +return ERROR_FAIL ; + } + ev->tx_buf_off += r;Can you assert that the value of r was within range ? (Perhaps this is paranoia on my part, but, still.)
Quoted 8 lines
Quoted 3 lines
I do assert the value of
tx_buf_offwhich does include r. But I can add
`assert(r > 0 && r <= (ev->rx_buf_size-ev->rx_buf_used));'.Yes. As I say worrying that
write()might return something insane is perhaps paranoia.Do you mean `ev->tx_buf_len -
ev->tx_buf_off' like in the write call ?
I was looking at the wrong function :(.
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 7/14 libxl_exec: Add libxl__spawn_initiate_failure
In reply to Anthony PERARDThis function can be used by user of libxl__spawn_* when they setup a notification other than xenstore. The parent can already report success via libxl__spawn_initiate_detach(), this new function can be used for failure instead of waiting for the timeout.
---
Notes:
v7:
call spawn_fail from libxl__spawn_initiate_failure. modify spawn_fail to set ss->rc only once.
Fix description to tell caller of libxl__spawn_initiate_failure that
they have to log.
use an egc instead of gc with libxl__spawn_initiate_failure.
v6:
long line fix
typos fixed
fix leak of internal state into external doc
if the function is call multiple times, set rc only the first time.
v5:
fix typos
tools/libxl/libxl_exec.c | 11 ++++++++++- tools/libxl/libxl_internal.h | 23 ++++++++++++++++++++++- 2 files changed, 32 insertions(+), 2 deletions(-)
tools/libxl/libxl_exec.c +10 -1
@@ -373,13 +373,22 @@ void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state *ss)spawn_detach (gc, ss); } +void libxl__spawn_initiate_failure (libxl__egc *egc,libxl__spawn_state *ss, +int rc) +/* The spawn state must be Attached on entry and will be Attached Failed +* on return. */ +{ +spawn_fail (egc, ss, rc); +} +static void spawn_fail (libxl__egc *egc,libxl__spawn_state *ss,int rc)/* Caller must have logged. Must be last thing in calling function, * as it may make the callback. Precondition: Attached or Detaching. */ {EGC_GC ;assert (rc); - ss->rc = rc; +if (!ss->rc) + ss->rc = rc;spawn_detach (gc, ss); }
tools/libxl/libxl_internal.h +22 -1
@@ -1553,7 +1553,8 @@ _hidden void libxl__spawn_init(libxl__spawn_state*);* * The inner child must soon exit or exec. It must also soon exit or * notify the parent of its successful startup by writing to the -* xenstore path xspath. +* xenstore path xspath OR via other means that the parent will have +* to set up. * * The user (in the parent) will be called back (confirm_cb) every * time that xenstore path is modified.
@@ -1609,6 +1610,26 @@ _hidden int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *spawn);*/ _hidden void libxl__spawn_initiate_detach(libxl__gc *gc, libxl__spawn_state*); +/* +* libxl__spawn_initiate_failure - Propagate failure from the caller to the +* callee. +* +* Works by killing the intermediate process from spawn_spawn. +* After this function returns, a failure will be reported. +* +* This is not synchronous: there will be a further callback when +* the detach is complete. +* +* Caller must have logged a failure reason. +* +* The spawn state must be Attached on entry and will remain Attached. It +* is possible for a spawn to fail for multiple reasons, for example +* call(s) to libxl__spawn_initiate_failure and also for some other reason. +* In that case the first rc value from any source will take precedence. +*/ +_hidden void libxl__spawn_initiate_failure(libxl__egc *egc, +libxl__spawn_state *ss,int rc); +/* * If successful, this should return 0. *
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 07/14] libxl_exec: Add libxl__spawn_initiate_failure
Anthony PERARD writes ("[PATCH v7 07/14] libxl_exec: Add libxl__spawn_initiate_failure"):
Quoted 4 lines
This function can be used by user of libxl__spawn_* when they setup a notification other than xenstore. The parent can already report success via
libxl__spawn_initiate_detach(), this new function can be used for failure instead of waiting for the timeout.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 8/14 libxl: Add init/dispose of for libxl__domain_build_state
In reply to Anthony PERARDThese two new functions libxl__domain_build_state_{init,dispose} should be called every time a new libxl__domain_build_state comes to existance.
There seems to be two of them, one with the domain creation machinery, and one in the stub_dm_spawn.
---
Notes:
v7:
new patchtools/libxl/libxl_create.c | 14 ++++++++++++-- tools/libxl/libxl_dm.c | 3 +++ tools/libxl/libxl_internal.h | 3 +++ 3 files changed, 18 insertions(+), 2 deletions(-)
tools/libxl/libxl_create.c +12 -2
@@ -761,6 +761,16 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid, libxl_device_model_version_to_string(b_info->device_model_version)); } +void libxl__domain_build_state_init (libxl__domain_build_state *state) +{ +} + +void libxl__domain_build_state_dispose (libxl__domain_build_state *state) +{ +libxl__file_reference_unmap (&state->pv_kernel); +libxl__file_reference_unmap (&state->pv_ramdisk); +} +/*----- main domain creation -----*/ /* We have a linear control flow; only one event callback is
@@ -820,6 +830,7 @@ static void initiate_domain_create(libxl__egc *egc,const int restore_fd = dcs->restore_fd; domid = dcs->domid_soft_reset; +libxl__domain_build_state_init (&dcs->build_state);if (d_config->c_info.ssid_label) {char *s = d_config->c_info.ssid_label;
@@ -1592,8 +1603,7 @@ static void domcreate_complete(libxl__egc *egc,libxl_domain_config *const d_config = dcs->guest_config;libxl_domain_config *d_config_saved = &dcs->guest_config_saved; -libxl__file_reference_unmap (&dcs->build_state.pv_kernel); -libxl__file_reference_unmap (&dcs->build_state.pv_ramdisk); +libxl__domain_build_state_dispose (&dcs->build_state);if (!rc && d_config->b_info.exec_ssidref) rc =xc_flask_relabel_domain (CTX ->xch, dcs->guest_domid, d_config->b_info.exec_ssidref);
tools/libxl/libxl_dm.c +3 -0
@@ -1919,6 +1919,8 @@ void libxl__spawn_stub_dm(libxl__egc *egc, libxl__stub_dm_spawn_state *sdss)libxl__domain_build_state *const d_state = sdss->dm.build_state;libxl__domain_build_state *const stubdom_state = &sdss->dm_state; +libxl__domain_build_state_init (stubdom_state); +if (guest_config->b_info.device_model_version !=LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL ) { ret =ERROR_INVAL ;
@@ -2247,6 +2249,7 @@ static void stubdom_xswait_cb(libxl__egc *egc, libxl__xswait_state *xswait,if (strcmp (p,"running" ))return ; out: +libxl__domain_build_state_dispose (&sdss->dm_state);libxl__xswait_stop (gc, xswait); sdss->callback (egc, &sdss->dm, rc); }
tools/libxl/libxl_internal.h +3 -0
@@ -1251,6 +1251,9 @@ typedef struct {uint32_t clock_frequency; } libxl__domain_build_state; +_hidden void libxl__domain_build_state_init(libxl__domain_build_state *s); +_hidden void libxl__domain_build_state_dispose(libxl__domain_build_state *s); + _hidden int libxl__build_pre(libxl__gc *gc, uint32_t domid,libxl_domain_config *const d_config,libxl__domain_build_state *state);
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 08/14] libxl: Add init/dispose of for libxl__domain_build_state
Anthony PERARD writes ("[PATCH v7 08/14] libxl: Add init/dispose of for libxl__domain_build_state"):
Quoted 5 lines
These two new functions libxl__domain_build_state_{init,dispose} should be called every time a new
libxl__domain_build_statecomes to existance.There seems to be two of them, one with the domain creation machinery, and one in the
stub_dm_spawn.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 9/14 libxl_dm: Pre-open QMP socket for QEMU
In reply to Anthony PERARDThis patch moves the creation of the QMP unix socket from QEMU to libxl. But libxl doesn't rely on this yet.
When starting QEMU with dm_restrict=1, pre-open the QMP socket before exec QEMU. That socket will be useful to find out if QEMU is ready, and pre-opening it means that libxl can connect to it without waiting for QEMU to create it.
The pre-opening is conditional, based on the use of dm_restrict because it is using a new command line option of QEMU, and dm_restrict support in QEMU is newer.
-chardev socket,fd=X is available with QEMU 2.12, since commit:
Quoted 2 lines
char: allow passing pre-opened socket file descriptor at startup
0935700f8544033ebbd41e1f13cd528f8a58d24d
dm_restrict is available in QEMU 3.0.
---
Notes:
v7:
move domain_build_state init/dispose to a different patch
and close dm_monitor_fd in that new dispose function
assert that dm_monitor_fd should be -1 when using old qemu
v6:
move dm_monitor_fd into libxl_domain_build_info (or d_state) -> move the creation of the socket into libxl__spawn_local_dm
instead of libxl__build_device_model_args
Use libxl_domid type instead of int for libxl__pre_open_qmp_socket() Check function calls (bind and listen) return value in a separate statement. typo and other coding style issue fixes
v5:
use libxl__remove_file
few changes in coding style
remove stale includes (sys/socket, sys/un) which are now in libxl_internal.h
v4:
separate the logic to open a socket into a function. Use libxl__prepare_sockaddr_un() to check path size
tools/libxl/libxl_create.c | 5 +++ tools/libxl/libxl_dm.c | 70 +++++++++++++++++++++++++++++++++--- tools/libxl/libxl_internal.h | 1 + 3 files changed, 72 insertions(+), 4 deletions(-)
tools/libxl/libxl_create.c +5 -0
@@ -763,12 +763,17 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid,void libxl__domain_build_state_init (libxl__domain_build_state *state) { + state->dm_monitor_fd =-1 ; }void libxl__domain_build_state_dispose (libxl__domain_build_state *state) {libxl__file_reference_unmap (&state->pv_kernel);libxl__file_reference_unmap (&state->pv_ramdisk); +if (state->dm_monitor_fd >=0 ) { +close (state->dm_monitor_fd); + state->dm_monitor_fd =-1 ; + } }/*----- main domain creation -----*/
tools/libxl/libxl_dm.c +66 -4
94 lines not on this page · raw message
tools/libxl/libxl_internal.h +1 -0
@@ -1234,6 +1234,7 @@ typedef struct {char *console_tty;char *saved_state; +int dm_monitor_fd;libxl__file_reference pv_kernel;libxl__file_reference pv_ramdisk;
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 09/14] libxl_dm: Pre-open QMP socket for QEMU
Anthony PERARD writes ("[PATCH v7 09/14] libxl_dm: Pre-open QMP socket for QEMU"):
Quoted 2 lines
This patch moves the creation of the QMP unix socket from QEMU to libxl. But libxl doesn't rely on this yet.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 2 lines · Anthony PERARD
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.libxl-ev-qmp-v
That should have read: br.libxl-ev-qmp-v7
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 10/14 libxl: Add dmss_init/dispose for libxl__dm_spawn_state
In reply to Anthony PERARDThese two functions, dmss_init and dmss_dispose, need to be called to initialise the private parts of a libxl__dm_spawn_state (dmss) as well as dispose of them before giving back control to a caller.
There are 3 functions that can start using a dmss, the classic libxl__spawn_local_dm, the one for stubdom libxl__spawn_stub_dm and libxl__spawn_qdisk_backend. But there are only 2 exit path as libxl__spawn_qdisk_backend is using libxl__spawn_local_dm functions.
These two new functions are empty but will be used shortly.
---
Notes:
v7:
new patchtools/libxl/libxl_dm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
tools/libxl/libxl_dm.c +16 -0
@@ -1935,6 +1935,14 @@ retry_transaction:return 0 ; } +static void dmss_init (libxl__dm_spawn_state *dmss) +{ +} + +static void dmss_dispose (libxl__gc *gc,libxl__dm_spawn_state *dmss) +{ +} +static void spawn_stubdom_pvqemu_cb (libxl__egc *egc,libxl__dm_spawn_state *stubdom_dmss,int rc);
@@ -1973,6 +1981,7 @@ void libxl__spawn_stub_dm(libxl__egc *egc, libxl__stub_dm_spawn_state *sdss)libxl__domain_build_state *const stubdom_state = &sdss->dm_state;libxl__domain_build_state_init (stubdom_state); +dmss_init (&sdss->dm);if (guest_config->b_info.device_model_version !=LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL ) {
@@ -2304,6 +2313,7 @@ static void stubdom_xswait_cb(libxl__egc *egc, libxl__xswait_state *xswait, out:libxl__domain_build_state_dispose (&sdss->dm_state);libxl__xswait_stop (gc, xswait); +dmss_dispose (gc, &sdss->dm); sdss->callback (egc, &sdss->dm, rc); }
@@ -2345,6 +2355,8 @@ void libxl__spawn_local_dm(libxl__egc *egc, libxl__dm_spawn_state *dmss)const char *dm;int dm_state_fd =-1 ; +dmss_init (dmss); +if (libxl_defbool_val (b_info->device_model_stubdomain)) {abort (); }
@@ -2550,6 +2562,7 @@ static void device_model_spawn_outcome(libxl__egc *egc, } out: +dmss_dispose (gc, dmss); dmss->callback (egc, dmss, rc); }
@@ -2562,6 +2575,8 @@ void libxl__spawn_qdisk_backend(libxl__egc *egc, libxl__dm_spawn_state *dmss)int logfile_w, null =-1 , rc;uint32_t domid = dmss->guest_domid; +dmss_init (dmss); +/* Always use qemu-xen as device model */ dm =qemu_xen_path (gc);
@@ -2626,6 +2641,7 @@ void libxl__spawn_qdisk_backend(libxl__egc *egc, libxl__dm_spawn_state *dmss) rc =0 ; out: +dmss_dispose (gc, dmss);if (logfile_w >=0 )close (logfile_w);if (null >=0 )close (null);/* callback on error only, success goes via dmss->spawn.*_cb */
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 11/14 libxl: QEMU startup sync based on QMP
In reply to Anthony PERARDThis is only activated when dm_restrict=1, as explained in a previous patch "libxl_dm: Pre-open QMP socket for QEMU"
---
Notes:
v7:
fixed _dispose call in device_model_spawn_outcome move qmp field in libxl__dm_spawn_state into the private section added description about the new QEMU_API error value
v6:
invent ERROR_QEMU_API
return better rc: ERROR_QEMU_API or ERROR_NOT_READY enhance log messages (debug and error)
v5:
removed empty success branch in device_model_qmp_cb() call libxl__ev_qmp_init() earlier in libxl__spawn_local_dm.
otherwise the error path would use an uninitialised libxl__ev_qmp.
v4:
moved to libxl__dm_spawn_* from libxl__spawn_*
tools/libxl/libxl_dm.c | 53 ++++++++++++++++++++++++++++++++++++ tools/libxl/libxl_internal.h | 1 + tools/libxl/libxl_types.idl | 1 + 3 files changed, 55 insertions(+)
tools/libxl/libxl_dm.c +53 -0
@@ -1937,10 +1937,12 @@ retry_transaction:static void dmss_init (libxl__dm_spawn_state *dmss) { +libxl__ev_qmp_init (&dmss->qmp); }static void dmss_dispose (libxl__gc *gc,libxl__dm_spawn_state *dmss) { +libxl__ev_qmp_dispose (gc, &dmss->qmp); } static void spawn_stubdom_pvqemu_cb(libxl__egc *egc,
@@ -2325,6 +2327,9 @@ static void device_model_startup_failed(libxl__egc *egc, int rc);static void device_model_detached (libxl__egc *egc,libxl__spawn_state *spawn); +static void device_model_qmp_cb (libxl__egc *egc,libxl__ev_qmp *ev, +const libxl__json_object *response, +int rc);/* our "next step" function, called from those callbacks and elsewhere */ static void device_model_spawn_outcome(libxl__egc *egc,
@@ -2465,6 +2470,17 @@ retry_transaction: spawn->failure_cb = device_model_startup_failed; spawn->detached_cb = device_model_detached; +if (state->dm_monitor_fd >=0 ) { +/* There is a valid QMP socket available now, +* use it to find out when QEMU is ready */ + dmss->qmp.ao = ao; + dmss->qmp.callback = device_model_qmp_cb; + dmss->qmp.domid = domid; + dmss->qmp.payload_fd =-1 ; + rc =libxl__ev_qmp_send (gc, &dmss->qmp,"query-status" ,NULL ); +if (rc) goto out_close; + } + rc =libxl__spawn_spawn (egc, spawn);if (rc <0 ) goto out_close;
@@ -2538,6 +2554,43 @@ static void device_model_detached(libxl__egc *egc,device_model_spawn_outcome (egc, dmss,0 ); } +static void device_model_qmp_cb (libxl__egc *egc,libxl__ev_qmp *ev, +const libxl__json_object *response, +int rc) +{ +EGC_GC ; +libxl__dm_spawn_state *dmss =CONTAINER_OF (ev, *dmss, qmp); +const libxl__json_object *o; +const char *status; + +libxl__ev_qmp_dispose (gc, ev); + +if (rc) + goto failed; + + o =libxl__json_map_get ("status" , response,JSON_STRING ); +if (!o) { +LOGD (ERROR , ev->domid, +"Missing 'status' in response to 'query-status'" ); +LOGD (DEBUG , ev->domid,".. instead, got: %s" ,JSON (response)); + rc =ERROR_QEMU_API ; + goto failed; + } + status =libxl__json_object_get_string (o); +if (strcmp (status,"running" )) { +LOGD (ERROR , ev->domid,"Unexpected QEMU status: %s" , status); + rc =ERROR_NOT_READY ; + goto failed; + } + +libxl__spawn_initiate_detach (gc, &dmss->spawn); +return ; + +failed: +LOGD (ERROR , ev->domid,"QEMU did not start properly, rc=%d" , rc); +libxl__spawn_initiate_failure (egc, &dmss->spawn, rc); +} +static void device_model_spawn_outcome (libxl__egc *egc,libxl__dm_spawn_state *dmss,int rc)
tools/libxl/libxl_internal.h +1 -0
@@ -3902,6 +3902,7 @@ typedef void libxl__dm_spawn_cb(libxl__egc *egc, libxl__dm_spawn_state*,struct libxl__dm_spawn_state {/* mixed - spawn.ao must be initialised by user; rest is private: */ libxl__spawn_state spawn; +libxl__ev_qmp qmp;/* filled in by user, must remain valid: */ uint32_t guest_domid;/* domain being served */ libxl_domain_config *guest_config;
tools/libxl/libxl_types.idl +1 -0
@@ -75,6 +75,7 @@ libxl_error = Enumeration("error", [ (-29, "QMP_COMMAND_NOT_FOUND"), # the requested command has not been found (-30, "QMP_DEVICE_NOT_ACTIVE"), # a device has failed to be become active (-31, "QMP_DEVICE_NOT_FOUND"), # the requested device has not been found + (-32, "QEMU_API"), # QEMU's replies don't contains expected members ], value_namespace = "") libxl_domain_type = Enumeration("domain_type", [
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 11/14] libxl: QEMU startup sync based on QMP
Anthony PERARD writes ("[PATCH v7 11/14] libxl: QEMU startup sync based on QMP"):
Quoted 5 lines
This is only activated when dm_restrict=1, as explained in a previous patch "
libxl_dm: Pre-open QMP socket for QEMU"Signed-off-by Anthony PERARD <[email protected]>Reviewed-by Roger Pau Monné <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 12/14 libxl_qmp: Store advertised QEMU version in libxl__ev_qmp
In reply to Anthony PERARDThis will be used in a later patch.
---
Notes:
v7:
Add do{}while around the MACRO.
formating nits changes.
v6:
new local macro GRAB_VERSION
better definition of qemu_version field in libxl_internal.h
v5:
initialise qemu_version struct in libxl__ev_qmp_init
tools/libxl/libxl_internal.h | 8 ++++++++ tools/libxl/libxl_qmp.c | 25 +++++++++++++++++++++++++ 2 files changed, 33 insertions(+)
tools/libxl/libxl_internal.h +8 -0
@@ -433,6 +433,14 @@ struct libxl__ev_qmp {libxl__ev_qmp_callback *callback;int payload_fd;/* set to send a fd with the command, -1 otherwise */ +/* read-only when Connected +* and not to be accessed by the caller otherwise */ +struct { +int major; +int minor; +int micro; + } qemu_version; +/* * remaining fields are private to libxl_ev_qmp_* */
tools/libxl/libxl_qmp.c +25 -0
@@ -1869,6 +1869,27 @@ static int qmp_ev_handle_message(libxl__egc *egc,return ERROR_PROTOCOL_ERROR_QMP ; } +/* +* Store advertised QEMU version +* { "QMP": { "version": { +* "qemu": { "major": int, "minor": int, "micro": int } } } } +*/ + o =libxl__json_map_get ("QMP" , resp,JSON_MAP ); + o =libxl__json_map_get ("version" , o,JSON_MAP ); + o =libxl__json_map_get ("qemu" , o,JSON_MAP ); +#define GRAB_VERSION (level) do { \ + ev->qemu_version.level = libxl__json_object_get_integer( \ + libxl__json_map_get(#level, o, JSON_INTEGER)); \ + } while (0) +GRAB_VERSION (major); +GRAB_VERSION (minor); +GRAB_VERSION (micro); +#undef GRAB_VERSION +LOGD (DEBUG , ev->domid,"QEMU version: %d.%d.%d" , + ev->qemu_version.major, + ev->qemu_version.minor, + ev->qemu_version.micro); +/* Prepare next message to send */ assert (!ev->tx_buf); ev->id = ev->next_id++;
@@ -1988,6 +2009,10 @@ void libxl__ev_qmp_init(libxl__ev_qmp *ev) ev->msg =NULL ; ev->msg_id =0 ; + + ev->qemu_version.major =-1 ; + ev->qemu_version.minor =-1 ; + ev->qemu_version.micro =-1 ; } int libxl__ev_qmp_send(libxl__gc *gc, libxl__ev_qmp *ev,
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 12/14] libxl_qmp: Store advertised QEMU version in libxl__ev_qmp
Anthony PERARD writes ("[PATCH v7 12/14] libxl_qmp: Store advertised QEMU version in libxl__ev_qmp"):
Quoted 1 line
This will be used in a later patch.
tools/libxl/libxl_qmp.c static int qmp_ev_handle_message(libxl__egc *egc, · line 1886
Quoted 10 lines
tools/libxl/libxl_qmp.c +10
-0+ o =libxl__json_map_get ("QMP" , resp,JSON_MAP ); + o =libxl__json_map_get ("version" , o,JSON_MAP ); + o =libxl__json_map_get ("qemu" , o,JSON_MAP ); +#define GRAB_VERSION (level) do { \ + ev->qemu_version.level = libxl__json_object_get_integer( \ + libxl__json_map_get(#level, o, JSON_INTEGER)); \ + } while (0) +GRAB_VERSION (major); +GRAB_VERSION (minor); +GRAB_VERSION (micro);
Earlier I wrote:
I would prefer the indentation to be such that the statement inside the macro is indented like the ones outside.
Ie like this:
+#define GRAB_VERSION(level) do { \
+ ev->qemu_version.level = libxl__json_object_get_integer( \
+ libxl__json_map_get(#level, o, JSON_INTEGER)); \
+ } while (0)
+ GRAB_VERSION(major);
But up to you. My ack stands either way.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 13/14 libxl: Change libxl__domain_suspend_device_model() to be async
In reply to Anthony PERARDThis create an extra step for the two call sites of the function.
libxl__domain_suspend_device_model() in this patch gets an extra error variable (there is ret and rc), but ret goes away in the next patch.
---libxl_domain_soft_reset() haven't been tested, as it doesn't appear to possible to call the function from xl.
---
Notes:
v6:
fix multiple way to report errors, libxl__domain_suspend_device_model will now only report via callbacks, and return void
add rc in libxl__domain_suspend_device_model (ret isn't a proper rc as libxl__qmp_save don't return one)
tools/libxl/libxl_create.c | 35 +++++++++++++++++++++----------- tools/libxl/libxl_dom_suspend.c | 36 ++++++++++++++++++++------------- tools/libxl/libxl_internal.h | 7 +++++-- 3 files changed, 50 insertions(+), 28 deletions(-)
tools/libxl/libxl_create.c +23 -12
@@ -1771,6 +1771,9 @@ error:domcreate_complete (egc, &cdcs->dcs, rc); } +static void soft_reset_dm_suspended (libxl__egc *egc, +libxl__domain_suspend_state *dsps, +int rc); static int do_domain_soft_reset(libxl_ctx *ctx, libxl_domain_config *d_config, uint32_t domid_soft_reset,
@@ -1853,11 +1856,24 @@ static int do_domain_soft_reset(libxl_ctx *ctx, goto out; } - rc =libxl__domain_suspend_device_model (gc, &dss->dsps); -if (rc) { -LOGD (ERROR , domid_soft_reset,"failed to suspend device model." ); - goto out; - } + dss->dsps.ao = ao; + dss->dsps.callback_device_model_done = soft_reset_dm_suspended; +libxl__domain_suspend_device_model (egc, &dss->dsps);/* must be last */ + +return AO_INPROGRESS ; + + out: +return AO_CREATE_FAIL (rc); +} + +static void soft_reset_dm_suspended (libxl__egc *egc, +libxl__domain_suspend_state *dsps, +int rc) +{ +STATE_AO_GC (dsps->ao); +libxl__domain_soft_reset_state *srs = +CONTAINER_OF (dsps, *srs, dss.dsps); +libxl__app_domain_create_state *cdcs = &srs->cdcs;/* * Ask all backends to disconnect by removing the domain from
@@ -1865,18 +1881,13 @@ static int do_domain_soft_reset(libxl_ctx *ctx,* xenstore again with probably different store/console/... * channels. */ -xs_release_domain (ctx->xsh, cdcs->dcs.domid_soft_reset); +xs_release_domain (CTX ->xsh, cdcs->dcs.domid_soft_reset); srs->dds.ao = ao; - srs->dds.domid = domid_soft_reset; + srs->dds.domid = cdcs->dcs.domid_soft_reset; srs->dds.callback = domain_soft_reset_cb; srs->dds.soft_reset = true;libxl__domain_destroy (egc, &srs->dds); - -return AO_INPROGRESS ; - - out: -return AO_CREATE_FAIL (rc); } static void domain_create_cb(libxl__egc *egc,
tools/libxl/libxl_dom_suspend.c +22 -14
@@ -68,10 +68,12 @@ out:/*----- callbacks, called by xc_domain_save -----*/ -int libxl__domain_suspend_device_model (libxl__gc *gc, +void libxl__domain_suspend_device_model (libxl__egc *egc,libxl__domain_suspend_state *dsps) { +STATE_AO_GC (dsps->ao);int ret =0 ; +int rc =0 ;uint32_t const domid = dsps->domid;const char *const filename = dsps->dm_savefile;
@@ -83,18 +85,29 @@ int libxl__domain_suspend_device_model(libxl__gc *gc,break ; }case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN : -if (libxl__qmp_stop (gc, domid)) -return ERROR_FAIL ; + ret =libxl__qmp_stop (gc, domid); +if (ret) { + rc =ERROR_FAIL ; + goto out; + }/* Save DM state into filename */ ret =libxl__qmp_save (gc, domid, filename, dsps->live); -if (ret) +if (ret) { + rc =ERROR_FAIL ;unlink (filename); + goto out; + }break ;default : -return ERROR_INVAL ; + rc =ERROR_INVAL ; + goto out; } -return ret; +out: +if (rc) +LOGD (ERROR , dsps->domid, +"failed to suspend device model, rc=%d" , rc); + dsps->callback_device_model_done (egc, dsps, rc);/* must be last */ } static void domain_suspend_common_wait_guest(libxl__egc *egc,
@@ -371,20 +384,15 @@ static void domain_suspend_common_guest_suspended(libxl__egc *egc, libxl__domain_suspend_state *dsps) {STATE_AO_GC (dsps->ao); -int rc;libxl__ev_evtchn_cancel (gc, &dsps->guest_evtchn);libxl__ev_xswatch_deregister (gc, &dsps->guest_watch);libxl__ev_time_deregister (gc, &dsps->guest_timeout);if (dsps->type ==LIBXL_DOMAIN_TYPE_HVM ) { - rc =libxl__domain_suspend_device_model (gc, dsps); -if (rc) { -LOGD (ERROR , dsps->domid, -"libxl__domain_suspend_device_model failed ret=%d" , rc); -domain_suspend_common_done (egc, dsps, rc); -return ; - } + dsps->callback_device_model_done = domain_suspend_common_done; +libxl__domain_suspend_device_model (egc, dsps);/* must be last */ +return ; }domain_suspend_common_done (egc, dsps,0 ); }
tools/libxl/libxl_internal.h +5 -2
@@ -3423,6 +3423,8 @@ struct libxl__domain_suspend_state {libxl__ev_time guest_timeout;const char *dm_savefile; +void (*callback_device_model_done)(libxl__egc *, +struct libxl__domain_suspend_state *,int rc);void (*callback_common_done)(libxl__egc *,struct libxl__domain_suspend_state *,int ok); };
@@ -4044,8 +4046,9 @@ static inline bool libxl__save_helper_inuse(const libxl__save_helper_state *shs)return libxl__ev_child_inuse (&shs->child); } -/* Each time the dm needs to be saved, we must call suspend and then save */ -_hidden int libxl__domain_suspend_device_model(libxl__gc *gc, +/* Each time the dm needs to be saved, we must call suspend and then save +* calls dsps->callback_device_model_done when done */ +_hidden void libxl__domain_suspend_device_model(libxl__egc *egc,libxl__domain_suspend_state *dsps); _hidden const char *libxl__device_model_savefile(libxl__gc *gc, uint32_t domid);
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 13/14] libxl: Change libxl__domain_suspend_device_model() to be async
Anthony PERARD writes ("[PATCH v7 13/14] libxl: Change libxl__domain_suspend_device_model() to be async"):
Quoted 4 lines
This create an extra step for the two call sites of the function.
libxl__domain_suspend_device_model()in this patch gets an extra error variable (there is ret and rc), but ret goes away in the next patch.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 14/14 libxl: Re-implement domain_suspend_device_model using libxl__ev_qmp
In reply to Anthony PERARDThe re-implementation is done because we want to be able to send the file description that QEMU can use to save its state. When QEMU is restricted, it would not be able to write to a path.
This replace both libxl__qmp_stop() and libxl__qmp_save().
qmp_qemu_check_version() was only used by libxl__qmp_save(), so it is replace by a version using libxl__ev_qmp instead.
Coding style fixed in libxl__domain_suspend_device_model() for the return value.
---
Notes:
v7:
use libxl__remove_file instead of unlink.
comment that libxl__qmp_suspend_save can callback synchronously
v6:
extract open call from libxl__carefd_opened to respect coding style libxl__qmp_suspend_save now always report success/error
via dsps->callback_device_model_done
v5:
rename goto 'out' label to 'error', as it is use only for errors. re-add/keep the comment about the "live" parameter in dm_state_fd_ready use libxl__remove_file instead of plain unlink
v4:
This patch replace the patch "libxl_qmp: Have QEMU save its state to a file descriptor" from previous version of the serie.
It uses libxl__ev_qmp instead.
tools/libxl/libxl_dom_suspend.c | 19 +--- tools/libxl/libxl_internal.h | 11 ++- tools/libxl/libxl_qmp.c | 160 +++++++++++++++++++++++++------- 3 files changed, 139 insertions(+), 51 deletions(-)
tools/libxl/libxl_dom_suspend.c +5 -14
@@ -34,6 +34,7 @@ int libxl__domain_suspend_init(libxl__egc *egc,libxl__ev_evtchn_init (&dsps->guest_evtchn);libxl__ev_xswatch_init (&dsps->guest_watch);libxl__ev_time_init (&dsps->guest_timeout); +libxl__ev_qmp_init (&dsps->qmp);if (type ==LIBXL_DOMAIN_TYPE_INVALID ) goto out; dsps->type = type;
@@ -72,7 +73,6 @@ void libxl__domain_suspend_device_model(libxl__egc *egc, libxl__domain_suspend_state *dsps) {STATE_AO_GC (dsps->ao); -int ret =0 ;int rc =0 ;uint32_t const domid = dsps->domid;const char *const filename = dsps->dm_savefile;
@@ -85,19 +85,9 @@ void libxl__domain_suspend_device_model(libxl__egc *egc,break ; }case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN : - ret =libxl__qmp_stop (gc, domid); -if (ret) { - rc =ERROR_FAIL ; - goto out; - } -/* Save DM state into filename */ - ret =libxl__qmp_save (gc, domid, filename, dsps->live); -if (ret) { - rc =ERROR_FAIL ; -unlink (filename); - goto out; - } -break ; +/* calls dsps->callback_device_model_done when done */ +libxl__qmp_suspend_save (egc, dsps);/* must be last */ +return ;default : rc =ERROR_INVAL ; goto out;
@@ -406,6 +396,7 @@ static void domain_suspend_common_done(libxl__egc *egc,libxl__ev_evtchn_cancel (gc, &dsps->guest_evtchn);libxl__ev_xswatch_deregister (gc, &dsps->guest_watch);libxl__ev_time_deregister (gc, &dsps->guest_timeout); +libxl__ev_qmp_dispose (gc, &dsps->qmp); dsps->callback_common_done (egc, dsps, rc); }
tools/libxl/libxl_internal.h +6 -5
@@ -1951,13 +1951,8 @@ _hidden int libxl__qmp_pci_del(libxl__gc *gc, int domid, libxl_device_pci *pcidev);/* Resume hvm domain */ _hidden int libxl__qmp_system_wakeup(libxl__gc *gc, int domid); -/* Suspend QEMU. */ -_hidden int libxl__qmp_stop(libxl__gc *gc, int domid);/* Resume QEMU. */ _hidden int libxl__qmp_resume(libxl__gc *gc, int domid); -/* Save current QEMU state into fd. */ -_hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename, -bool live);/* Load current QEMU state from file. */ _hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);/* Set dirty bitmap logging status */
@@ -3421,6 +3416,7 @@ struct libxl__domain_suspend_state {libxl__xswait_state pvcontrol;libxl__ev_xswatch guest_watch;libxl__ev_time guest_timeout; +libxl__ev_qmp qmp;const char *dm_savefile; void (*callback_device_model_done)(libxl__egc*,
@@ -3432,6 +3428,11 @@ int libxl__domain_suspend_init(libxl__egc *egc,libxl__domain_suspend_state *dsps, libxl_domain_type type); +/* calls dsps->callback_device_model_done when done +* may synchronously calls this callback */ +_hidden void libxl__qmp_suspend_save(libxl__egc *egc, +libxl__domain_suspend_state *dsps); +struct libxl__domain_save_state {/* set by caller of libxl__domain_save */ libxl__ao *ao;
tools/libxl/libxl_qmp.c +128 -32
186 lines not on this page · raw message
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v7 14/14] libxl: Re-implement domain_suspend_device_model using libxl__ev_qmp
Anthony PERARD writes ("[PATCH v7 14/14] libxl: Re-implement domain_suspend_device_model using libxl__ev_qmp"):
Quoted 11 lines
The re-implementation is done because we want to be able to send the file description that QEMU can use to save its state. When QEMU is restricted, it would not be able to write to a path.
This replace both
libxl__qmp_stop()andlibxl__qmp_save().
qmp_qemu_check_version()was only used bylibxl__qmp_save(), so it is replace by a version usinglibxl__ev_qmpinstead.Coding style fixed in
libxl__domain_suspend_device_model()for the return value.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Anthony PERARD writes ("[PATCH v7 00/14] libxl: Enable save/restore/migration of a restricted QEMU + libxl__ev_qmp_*"):
Quoted 2 lines
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.libxl-ev-qmp-v
Thanks. Sorry for being so slow to review this. It is very nearly ready and I hope it will make 4.12.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel