libxl: New slow lock + fix libxl_cdrom_insert with QEMU depriv
v3 cover libxl: New slow lock + fix libxl_cdrom_insert with QEMU depriv
Hi,
Changes in v3:
- renamed libxl__ev_lock to libxl__ev_devlock
- rebased
- 1 patch not acked "libxl_internal: Introduce libxl__ev_devlock for devices hotplug via QMP"
- other patches have been updated for the new ev_devlock name and for the rebased
Changes in v2:
- New libxl__ev_lock, which actually respect lock hierarchy (it's outside of CTX_LOCK).
- some smaller changes detailed in patch notes.
This patch series fix libxl_cdrom_insert to work with a depriviledge QEMU. For that, we need to use libxl__ev_qmp. For that, we need a new lock because userdata_lock can't be used anymore.
FYI: I don't think that enough yet to migrate a depriviledged QEMU. We may need to open disks/cdrom in libxl before starting QEMU, similar to what this patch series do when inserting a new cdrom.
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.libxl-slow-lock-v3
Anthony PERARD (9):
libxl_internal: Remove lost comment
libxl: Pointer on usage of libxl__domain_userdata_lock
libxl_internal: Introduce libxl__ev_devlock for devices hotplug via
QMP
libxl: Add optimisation to ev_lock
libxl_disk: Reorganise libxl_cdrom_insert
libxl_disk: Cut libxl_cdrom_insert into steps ..
libxl_disk: Implement missing timeout for libxl_cdrom_insert
libxl: Move qmp_parameters_* prototypes to libxl_internal.h
libxl_disk: Use ev_qmp in libxl_cdrom_insert
tools/libxl/Makefile | 3 + tools/libxl/libxl_disk.c | 341 ++++++++++++++++++++++++++++------- tools/libxl/libxl_internal.c | 182 +++++++++++++++++++ tools/libxl/libxl_internal.h | 105 +++++++++-- tools/libxl/libxl_qmp.c | 89 ++++----- 5 files changed, 590 insertions(+), 130 deletions(-)
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 1/9 libxl_internal: Remove lost comment
That comment as been separated from the function it defines by 4197d3abbb3055d3798254eb7ba239bfb5824360, but then was not useful anymore when the libxl__device_disk_add() prototype was removed by 22ea8ad02e465e32cd40887c750b55c3a997a288.
--- tools/libxl/libxl_internal.h | 7 ------- 1 file changed, 7 deletions(-)
tools/libxl/libxl_internal.h +0 -7
@@ -2719,13 +2719,6 @@ struct libxl__multidev {* DONE. */ -/* AO operation to connect a disk device, called by -* libxl_device_disk_add and libxl__add_disks. This function calls -* libxl__wait_device_connection to wait for the device to -* finish the connection (might involve executing hotplug scripts). -* -* Once finished, aodev->callback will be executed. -*/ /* * As of Xen 4.5 we maintain various information, including hotplug * device information, in JSON files, so that we can use this JSON
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 2/9 libxl: Pointer on usage of libxl__domain_userdata_lock
In reply to Anthony PERARDIt is currently difficult to know how/when/why the userdata lock is supposed to be used. Add some pointers to the hotplug comments.
--- tools/libxl/libxl_internal.h | 6 ++++++ 1 file changed, 6 insertions(+)
tools/libxl/libxl_internal.h +6 -0
@@ -4477,6 +4477,12 @@ void libxl__unlock_domain_userdata(libxl__domain_userdata_lock *lock);* data store. The registry entry in libxl private data store * is "libxl-json". * Caller must hold user data lock. +* +* Other names used for this lock throughout the libxl code are json_lock, +* libxl__domain_userdata_lock, "libxl-json", data store lock. +* +* See the comment for libxl__ao_device, and "Algorithm for handling device +* removal", for information about using the libxl-json lock / json_lock. */ int libxl__get_domain_configuration (libxl__gc *gc,uint32_t domid,libxl_domain_config *d_config);
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 3/9 libxl_internal: Introduce libxl__ev_devlock for devices hotplug via QMP
In reply to Anthony PERARDThe current lock domain_userdata_lock' can't be used when modification to a guest is done by sending command to QEMU, this is a slow process and requires to call CTX_UNLOCK, which is not possible while holding the domain_userdata_lock'.
To resolve this issue, we create a new lock which can take over part of the job of the json_lock.
This lock is outside CTX_LOCK in the lock hierarchy. libxl__ev_devlock_lock will have CTX_UNLOCK before trying to grab the ev_devlock. The callback is used to notify when the ev_devlock have been acquired.
---
Notes:
v3:
- fix use of LOGED where errno numeric value was printed.
- change comment in libxl__ev_unlock to refer to the one in libxl__unlock_domain_userdata() instead of making a copy of it.
- rename ev_lock to ev_devlock. (And _get to _lock.)
v2:
- new patch, to replace 2 patch
implement a different lock
tools/libxl/libxl_internal.c | 163 +++++++++++++++++++++++++++++++++++ tools/libxl/libxl_internal.h | 76 +++++++++++++++- 2 files changed, 235 insertions(+), 4 deletions(-)
tools/libxl/libxl_internal.c +163 -0
@@ -575,6 +575,169 @@ void libxl__update_domain_configuration(libxl__gc *gc, dst->b_info.video_memkb = src->b_info.video_memkb; } +void libxl__ev_devlock_init (libxl__ev_devlock *lock) +{ +libxl__ev_child_init (&lock->child); + lock->path =NULL ; + lock->fd =-1 ; + lock->held = false; +} + +static void ev_lock_prepare_fork (libxl__egc *egc,libxl__ev_devlock *lock); +static void ev_lock_child_callback (libxl__egc *egc,libxl__ev_child *child, +pid_t pid,int status); + +void libxl__ev_devlock_lock (libxl__egc *egc,libxl__ev_devlock *lock) +{ +STATE_AO_GC (lock->ao); +const char *lockfile; + + lockfile =libxl__userdata_path (gc, lock->domid, +"libxl-device-changes-lock" ,"l" ); +if (!lockfile) goto out; + lock->path =libxl__strdup (NOGC , lockfile); + +ev_lock_prepare_fork (egc, lock); +return ; +out: + lock->callback (egc, lock,ERROR_LOCK_FAIL ); +} + +static void ev_lock_prepare_fork (libxl__egc *egc,libxl__ev_devlock *lock) +{ +STATE_AO_GC (lock->ao); +pid_t pid; +int fd; + +/* Convenience aliases */ +libxl_domid domid = lock->domid; +const char *lockfile = lock->path; + + lock->fd =open (lockfile,O_RDWR |O_CREAT ,0666 ); +if (lock->fd <0 ) { +LOGED (ERROR , domid,"cannot open lockfile %s" , lockfile); + goto out; + } + fd = lock->fd; + + pid =libxl__ev_child_fork (gc, &lock->child, ev_lock_child_callback); +if (pid <0 ) + goto out; +if (!pid) { +/* child */ +int exit_val =0 ; + +/* Lock the file in exclusive mode, wait indefinitely to +* acquire the lock */ +while (flock (fd,LOCK_EX )) { +switch (errno) { +case EINTR : +/* Signal received, retry */ +continue ; +default : +/* All other errno: EBADF, EINVAL, ENOLCK, EWOULDBLOCK */ +LOGED (ERROR , domid, +"unexpected error while trying to lock %s, fd=%d" , + lockfile, fd); + exit_val =1 ; +break ; + } + } +_exit (exit_val); + } + +/* Now that the child has the fd, set cloexec in the parent to prevent +* more leakage than necessary */ +libxl_fd_set_cloexec (CTX , fd,1 ); +return ; +out: +libxl__ev_devlock_unlock (gc, lock); + lock->callback (egc, lock,ERROR_LOCK_FAIL ); +} + +static void ev_lock_child_callback (libxl__egc *egc,libxl__ev_child *child, +pid_t pid,int status) +{ +EGC_GC ; +libxl__ev_devlock *lock =CONTAINER_OF (child, *lock, child); +struct stat stab, fstab; +int rc =ERROR_LOCK_FAIL ; + +/* Convenience aliases */ +int fd = lock->fd; +const char *lockfile = lock->path; +libxl_domid domid = lock->domid; + +if (status) { +libxl_report_child_exitstatus (CTX ,XTL_ERROR ,"flock child" , + pid, status); + goto out; + } + +if (fstat (fd, &fstab)) { +LOGED (ERROR , domid,"cannot fstat %s, fd=%d" , lockfile, fd); + goto out; + } +if (stat (lockfile, &stab)) { +if (errno !=ENOENT ) { +LOGED (ERROR , domid,"cannot stat %s" , lockfile); + goto out; + } + }else { +if (stab.st_dev == fstab.st_dev && stab.st_ino == fstab.st_ino) { +/* We held the lock */ + lock->held = true; + rc =0 ; + goto out; + } + } + +/* We didn't grab the lock, let's try again */ +flock (lock->fd,LOCK_UN ); +close (lock->fd); + lock->fd =-1 ; +ev_lock_prepare_fork (egc, lock); +return ; + +out: +if (lock->held) { +/* Check the domain is still there, if not we should release the +* lock and clean up. */ +if (libxl_domain_info (CTX ,NULL , domid)) + rc =ERROR_LOCK_FAIL ; + } +if (rc) { +LOGD (ERROR , domid,"Failed to grab qmp-lock" ); +libxl__ev_devlock_unlock (gc, lock); + } + lock->callback (egc, lock, rc); +} + +void libxl__ev_devlock_unlock (libxl__gc *gc,libxl__ev_devlock *lock) +{ +int r; + +assert (!libxl__ev_child_inuse (&lock->child)); + +/* See the rationale in libxl__unlock_domain_userdata() +* about why we do unlink() before unlock(). */ + +if (lock->path && lock->held) +unlink (lock->path); + +if (lock->fd >=0 ) { +/* We need to call unlock as the fd may have leaked into other +* processes */ + r =flock (lock->fd,LOCK_UN ); +if (r) +LOGED (ERROR , lock->domid,"failed to unlock fd=%d, path=%s" , + lock->fd, lock->path); +close (lock->fd); + } +free (lock->path); +libxl__ev_devlock_init (lock); +} +/* * Local variables: * mode: C
tools/libxl/libxl_internal.h +72 -4
101 lines not on this page · raw message
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v3 3/9] libxl_internal: Introduce libxl__ev_devlock for devices hotplug via QMP
Anthony PERARD writes ("[PATCH v3 3/9] libxl_internal: Introduce libxl__ev_devlock for devices hotplug via QMP"):
Quoted 8 lines
The current lock
domain_userdata_lock' can't be used when modification to a guest is done by sending command to QEMU, this is a slow process and requires to call CTX_UNLOCK, which is not possible while holding thedomain_userdata_lock'.To resolve this issue, we create a new lock which can take over part of the job of the
json_lock.
Quoted 6 lines
This lock is outside
CTX_LOCKin the lock hierarchy.libxl__ev_devlock_lockwill haveCTX_UNLOCKbefore trying to grab theev_devlock. The callback is used to notify when theev_devlockhave been acquired.Signed-off-by Anthony PERARD <[email protected]>
Thanks.
And committed the series.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 4/9 libxl: Add optimisation to ev_lock
In reply to Anthony PERARDIt will often be the case that the lock is free to grab. So we first try to grab it before we have to fork. Even though in this case the locks are grabbed in the wrong order in the lock hierarchy (ev_lock should be outside of CTX_LOCK), it is fine to try without blocking. If that failed, we will release CTX_LOCK and try to grab both lock again in the right order.
That optimisation is only enabled in releases (debug=n) so the more complicated code with fork is actually exercised.
---
Notes:
v2:
- new patch
tools/libxl/Makefile | 3 +++ tools/libxl/libxl_internal.c | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+)
tools/libxl/Makefile +3 -0
@@ -35,6 +35,9 @@ ifeq ($(CONFIG_LIBNL),y) CFLAGS_LIBXL += $(LIBNL3_CFLAGS) endif CFLAGS_LIBXL += -Wshadow +ifeq ($(debug),y) +CFLAGS_LIBXL += -DCONFIG_DEBUG +endif LIBXL_LIBS-$(CONFIG_ARM) += -lfdt
tools/libxl/libxl_internal.c +19 -0
@@ -620,6 +620,25 @@ static void ev_lock_prepare_fork(libxl__egc *egc, libxl__ev_devlock *lock) } fd = lock->fd; +/* Enable this optimisation only in releases, so the fork code is +* exercised while libxl is built with debug=y. */ +#ifndef CONFIG_DEBUG +/* +* We try to grab the lock before forking as it is likely to be free. +* Even though we are supposed to CTX_UNLOCK before attempting to grab +* the ev_lock, it is fine to do a non-blocking request now with the +* CTX_LOCK held as if that fails we'll try again in a fork (CTX_UNLOCK +* will be called in libxl), that will avoid deadlocks. +*/ +int r =flock (fd,LOCK_EX |LOCK_NB ); +if (!r) { +libxl_fd_set_cloexec (CTX , fd,1 ); +/* We held a lock, no need to fork but we need to check it. */ +ev_lock_child_callback (egc, &lock->child,0 ,0 ); +return ; + } +#endif + pid =libxl__ev_child_fork (gc, &lock->child, ev_lock_child_callback);if (pid <0 ) goto out;
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 5/9 libxl_disk: Reorganise libxl_cdrom_insert
In reply to Anthony PERARDThis is in preparation of cutting libxl_cdrom_insert into several functions to allow asynchronous callbacks.
No functional changes.
--- tools/libxl/libxl_disk.c | 58 ++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 26 deletions(-)
tools/libxl/libxl_disk.c +32 -26
@@ -647,7 +647,7 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, {AO_CREATE (ctx, domid, ao_how);int num =0 , i; -libxl_device_disk *disks =NULL , disk_saved, disk_empty; +libxl_device_disk *disks =NULL , disk_saved;libxl_domain_config d_config;int rc, dm_ver;libxl__device device;
@@ -658,16 +658,9 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,flexarray_t *insert =NULL , *empty =NULL ;libxl_domain_config_init (&d_config); -libxl_device_disk_init (&disk_empty);libxl_device_disk_init (&disk_saved);libxl_device_disk_copy (ctx, &disk_saved, disk); - disk_empty.format =LIBXL_DISK_FORMAT_EMPTY ; - disk_empty.vdev =libxl__strdup (NOGC , disk->vdev); - disk_empty.pdev_path =libxl__strdup (NOGC ,"" ); - disk_empty.is_cdrom =1 ; -libxl__device_disk_setdefault (gc, domid, &disk_empty, false); -libxl_domain_type type =libxl__domain_type (gc, domid);if (type ==LIBXL_DOMAIN_TYPE_INVALID ) { rc =ERROR_FAIL ;
@@ -721,23 +714,6 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, be_path =libxl__device_backend_path (gc, &device); libxl_path =libxl__device_libxl_path (gc, &device); - insert =flexarray_make (gc,4 ,1 ); - -flexarray_append_pair (insert,"type" , -libxl__device_disk_string_of_backend (disk->backend)); -if (disk->format !=LIBXL_DISK_FORMAT_EMPTY ) -flexarray_append_pair (insert,"params" , -GCSPRINTF ("%s:%s" , -libxl__device_disk_string_of_format (disk->format), - disk->pdev_path)); -else -flexarray_append_pair (insert,"params" ,"" ); - - empty =flexarray_make (gc,4 ,1 ); -flexarray_append_pair (empty,"type" , -libxl__device_disk_string_of_backend (disk->backend)); -flexarray_append_pair (empty,"params" ,"" ); -/* Note: CTX lock is already held at this point so lock hierarchy * is maintained. */
@@ -750,11 +726,27 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,/* We need to eject the original image first. This is implemented * by inserting empty media. JSON is not updated. */ +if (dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { +libxl_device_disk disk_empty; + +libxl_device_disk_init (&disk_empty); + disk_empty.format =LIBXL_DISK_FORMAT_EMPTY ; + disk_empty.vdev =libxl__strdup (NOGC , disk->vdev); + disk_empty.pdev_path =libxl__strdup (NOGC ,"" ); + disk_empty.is_cdrom =1 ; +libxl__device_disk_setdefault (gc, domid, &disk_empty, false); + rc =libxl__qmp_insert_cdrom (gc, domid, &disk_empty); +libxl_device_disk_dispose (&disk_empty);if (rc) goto out; } + empty =flexarray_make (gc,4 ,1 ); +flexarray_append_pair (empty,"type" , +libxl__device_disk_string_of_backend (disk->backend)); +flexarray_append_pair (empty,"params" ,"" ); +for (;;) { rc =libxl__xs_transaction_start (gc, &t);if (rc) goto out;
@@ -781,6 +773,10 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,if (rc <0 ) goto out; } +/* +* Now that the drive is empty, we can insert the new media. +*/ + rc =libxl__get_domain_configuration (gc, domid, &d_config);if (rc) goto out;
@@ -794,6 +790,17 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,if (rc) goto out; } + insert =flexarray_make (gc,4 ,1 ); +flexarray_append_pair (insert,"type" , +libxl__device_disk_string_of_backend (disk->backend)); +if (disk->format !=LIBXL_DISK_FORMAT_EMPTY ) +flexarray_append_pair (insert,"params" , +GCSPRINTF ("%s:%s" , +libxl__device_disk_string_of_format (disk->format), + disk->pdev_path)); +else +flexarray_append_pair (insert,"params" ,"" ); +for (;;) { rc =libxl__xs_transaction_start (gc, &t);if (rc) goto out;
@@ -831,7 +838,6 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, out:libxl__xs_transaction_abort (gc, &t);libxl__device_list_free (&libxl__disk_devtype, disks, num); -libxl_device_disk_dispose (&disk_empty);libxl_device_disk_dispose (&disk_saved);libxl_domain_config_dispose (&d_config);
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 6/9 libxl_disk: Cut libxl_cdrom_insert into steps ..
In reply to Anthony PERARD.. and use a new "slow" lock to avoid holding the userdata lock across several functions.
This patch cuts libxl_cdrom_insert into different step/function but there are still called synchronously. (Taking the ev_lock is the only step that might be asynchronous.) A later patch will call them asynchronously when QMP is involved.
Thee userdata lock (json_lock) use to protect against concurrent change of cdrom is replaced by an ev_lock which can be held across different CTX_LOCK sections. The json_lock is still used when reading/modifying the domain userdata (mandatory) and update xenstore (mostly because it's updated as the same time as the userdata).
---
Notes:
v2:
- rewrite patch description
- rework use of the new lock
tools/libxl/libxl_disk.c | 196 +++++++++++++++++++++++++++++++-------- 1 file changed, 155 insertions(+), 41 deletions(-)
tools/libxl/libxl_disk.c +155 -41
@@ -642,24 +642,43 @@ int libxl_device_disk_getinfo(libxl_ctx *ctx, uint32_t domid,return rc; } +typedef struct { +libxl__ao *ao; +libxl_domid domid; +libxl_device_disk *disk; +libxl_device_disk disk_saved; +libxl__ev_devlock qmp_lock; +int dm_ver; +}libxl__cdrom_insert_state ; + +static void cdrom_insert_lock_acquired (libxl__egc *,libxl__ev_devlock *, +int rc); +static void cdrom_insert_ejected (libxl__egc *egc, +libxl__cdrom_insert_state *cis); +static void cdrom_insert_inserted (libxl__egc *egc, +libxl__cdrom_insert_state *cis); +static void cdrom_insert_done (libxl__egc *egc, +libxl__cdrom_insert_state *cis, +int rc); +int libxl_cdrom_insert (libxl_ctx *ctx,uint32_t domid,libxl_device_disk *disk,const libxl_asyncop_how *ao_how) {AO_CREATE (ctx, domid, ao_how);int num =0 , i; -libxl_device_disk *disks =NULL , disk_saved; -libxl_domain_config d_config; -int rc, dm_ver; -libxl__device device; -const char *be_path, *libxl_path; -char * tmp; -libxl__domain_userdata_lock *lock =NULL ; -xs_transaction_t t =XBT_NULL ; -flexarray_t *insert =NULL , *empty =NULL ; - -libxl_domain_config_init (&d_config); -libxl_device_disk_init (&disk_saved); -libxl_device_disk_copy (ctx, &disk_saved, disk); +libxl_device_disk *disks =NULL ; +int rc; +libxl__cdrom_insert_state *cis; + +GCNEW (cis); + cis->ao = ao; + cis->domid = domid; + cis->disk = disk; +libxl_device_disk_init (&cis->disk_saved); +libxl_device_disk_copy (ctx, &cis->disk_saved, disk); +libxl__ev_devlock_init (&cis->qmp_lock); + cis->qmp_lock.ao = ao; + cis->qmp_lock.domid = domid;libxl_domain_type type =libxl__domain_type (gc, domid);if (type ==LIBXL_DOMAIN_TYPE_INVALID ) {
@@ -678,8 +697,8 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, goto out; } - dm_ver =libxl__device_model_version_running (gc, domid); -if (dm_ver ==-1 ) { + cis->dm_ver =libxl__device_model_version_running (gc, domid); +if (cis->dm_ver ==-1 ) {LOGD (ERROR , domid,"Cannot determine device model version" ); rc =ERROR_FAIL ; goto out;
@@ -708,40 +727,82 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, disk->format =LIBXL_DISK_FORMAT_EMPTY ; } - rc =libxl__device_from_disk (gc, domid, disk, &device); -if (rc) goto out; +out: +libxl__device_list_free (&libxl__disk_devtype, disks, num); +if (rc) { +cdrom_insert_done (egc, cis, rc);/* must be last */ + }else { + cis->qmp_lock.callback = cdrom_insert_lock_acquired; +libxl__ev_devlock_lock (egc, &cis->qmp_lock);/* must be last */ + } +return AO_INPROGRESS ; +} - be_path =libxl__device_backend_path (gc, &device); - libxl_path =libxl__device_libxl_path (gc, &device); +static void cdrom_insert_lock_acquired (libxl__egc *egc, +libxl__ev_devlock *lock, +int rc) +{ +libxl__cdrom_insert_state *cis =CONTAINER_OF (lock, *cis, qmp_lock); +STATE_AO_GC (cis->ao); -/* Note: CTX lock is already held at this point so lock hierarchy -* is maintained. -*/ - lock =libxl__lock_domain_userdata (gc, domid); -if (!lock) { - rc =ERROR_LOCK_FAIL ; - goto out; - } +if (rc) goto out;/* We need to eject the original image first. This is implemented * by inserting empty media. JSON is not updated. */ -if (dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { +if (cis->dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) {libxl_device_disk disk_empty;libxl_device_disk_init (&disk_empty); disk_empty.format =LIBXL_DISK_FORMAT_EMPTY ; - disk_empty.vdev =libxl__strdup (NOGC , disk->vdev); + disk_empty.vdev =libxl__strdup (NOGC , cis->disk->vdev); disk_empty.pdev_path =libxl__strdup (NOGC ,"" ); disk_empty.is_cdrom =1 ; -libxl__device_disk_setdefault (gc, domid, &disk_empty, false); +libxl__device_disk_setdefault (gc, cis->domid, &disk_empty, false); - rc =libxl__qmp_insert_cdrom (gc, domid, &disk_empty); + rc =libxl__qmp_insert_cdrom (gc, cis->domid, &disk_empty);libxl_device_disk_dispose (&disk_empty);if (rc) goto out; } +cdrom_insert_ejected (egc, cis);/* must be last */ +return ; + +out: +cdrom_insert_done (egc, cis, rc);/* must be last */ +} + +static void cdrom_insert_ejected (libxl__egc *egc, +libxl__cdrom_insert_state *cis) +{ +EGC_GC ; +int rc; +libxl__domain_userdata_lock *data_lock =NULL ; +libxl__device device; +const char *be_path, *libxl_path; +flexarray_t *empty =NULL ; +xs_transaction_t t =XBT_NULL ; +char *tmp; +libxl_domain_config d_config; + +/* convenience aliases */ +libxl_domid domid = cis->domid; +libxl_device_disk *disk = cis->disk; + +libxl_domain_config_init (&d_config); + + rc =libxl__device_from_disk (gc, domid, disk, &device); +if (rc) goto out; + be_path =libxl__device_backend_path (gc, &device); + libxl_path =libxl__device_libxl_path (gc, &device); + + data_lock =libxl__lock_domain_userdata (gc, domid); +if (!data_lock) { + rc =ERROR_LOCK_FAIL ; + goto out; + } + empty =flexarray_make (gc,4 ,1 );flexarray_append_pair (empty,"type" ,libxl__device_disk_string_of_backend (disk->backend));
@@ -780,16 +841,66 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, rc =libxl__get_domain_configuration (gc, domid, &d_config);if (rc) goto out; -device_add_domain_config (gc, &d_config, &libxl__disk_devtype, &disk_saved); +device_add_domain_config (gc, &d_config, &libxl__disk_devtype, + &cis->disk_saved); rc =libxl__dm_check_start (gc, &d_config, domid);if (rc) goto out; -if (dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { +if (cis->dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { rc =libxl__qmp_insert_cdrom (gc, domid, disk);if (rc) goto out; } + rc =0 ; + +out: +libxl__xs_transaction_abort (gc, &t); +libxl_domain_config_dispose (&d_config); +if (data_lock)libxl__unlock_domain_userdata (data_lock); +if (rc) { +cdrom_insert_done (egc, cis, rc);/* must be last */ + }else { +cdrom_insert_inserted (egc, cis);/* must be last */ + } +} + +static void cdrom_insert_inserted (libxl__egc *egc, +libxl__cdrom_insert_state *cis) +{ +EGC_GC ; +int rc; +libxl__domain_userdata_lock *data_lock =NULL ; +libxl_domain_config d_config; +flexarray_t *insert =NULL ; +xs_transaction_t t =XBT_NULL ; +libxl__device device; +const char *be_path, *libxl_path; +char *tmp; + +/* convenience aliases */ +libxl_domid domid = cis->domid; +libxl_device_disk *disk = cis->disk; + +libxl_domain_config_init (&d_config); + + rc =libxl__device_from_disk (gc, domid, disk, &device); +if (rc) goto out; + be_path =libxl__device_backend_path (gc, &device); + libxl_path =libxl__device_libxl_path (gc, &device); + + data_lock =libxl__lock_domain_userdata (gc, domid); +if (!data_lock) { + rc =ERROR_LOCK_FAIL ; + goto out; + } + + rc =libxl__get_domain_configuration (gc, domid, &d_config); +if (rc) goto out; + +device_add_domain_config (gc, &d_config, &libxl__disk_devtype, + &cis->disk_saved); + insert =flexarray_make (gc,4 ,1 );flexarray_append_pair (insert,"type" ,libxl__device_disk_string_of_backend (disk->backend));
@@ -830,21 +941,24 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,if (rc <0 ) goto out; } -/* success, no actual async */ -libxl__ao_complete (egc, ao,0 ); - rc =0 ; out:libxl__xs_transaction_abort (gc, &t); -libxl__device_list_free (&libxl__disk_devtype, disks, num); -libxl_device_disk_dispose (&disk_saved);libxl_domain_config_dispose (&d_config); +if (data_lock)libxl__unlock_domain_userdata (data_lock); +cdrom_insert_done (egc, cis, rc);/* must be last */ +} -if (lock)libxl__unlock_domain_userdata (lock); +static void cdrom_insert_done (libxl__egc *egc, +libxl__cdrom_insert_state *cis, +int rc) +{ +EGC_GC ; -if (rc)return AO_CREATE_FAIL (rc); -return AO_INPROGRESS ; +libxl__ev_devlock_unlock (gc, &cis->qmp_lock); +libxl_device_disk_dispose (&cis->disk_saved); +libxl__ao_complete (egc, cis->ao, rc); }/* libxl__alloc_vdev only works on the local domain, that is the domain
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 7/9 libxl_disk: Implement missing timeout for libxl_cdrom_insert
In reply to Anthony PERARDAfter the patch "libxl_disk: Use ev_qmp in libxl_cdrom_insert" there will not be any kind of timeout, add one back.
---
Notes:
Ian, in this patch, the timeout is setup after we have aquired the lock. Should we change that to also have a timeout waiting for the lock to be released?
v2:
- patch move earlier in the series to keep bisectability.
tools/libxl/libxl_disk.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
tools/libxl/libxl_disk.c +21 -0
@@ -649,6 +649,7 @@ typedef struct {libxl_device_disk disk_saved;libxl__ev_devlock qmp_lock;int dm_ver; +libxl__ev_time time; } libxl__cdrom_insert_state; static void cdrom_insert_lock_acquired(libxl__egc *, libxl__ev_devlock *,
@@ -657,6 +658,9 @@ static void cdrom_insert_ejected(libxl__egc *egc, libxl__cdrom_insert_state *cis);static void cdrom_insert_inserted (libxl__egc *egc,libxl__cdrom_insert_state *cis); +static void cdrom_insert_timout (libxl__egc *egc,libxl__ev_time *ev, +const struct timeval *requested_abs, +int rc);static void cdrom_insert_done (libxl__egc *egc,libxl__cdrom_insert_state *cis,int rc);
@@ -679,6 +683,7 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk,libxl__ev_devlock_init (&cis->qmp_lock); cis->qmp_lock.ao = ao; cis->qmp_lock.domid = domid; +libxl__ev_time_init (&cis->time);libxl_domain_type type =libxl__domain_type (gc, domid);if (type ==LIBXL_DOMAIN_TYPE_INVALID ) {
@@ -747,6 +752,11 @@ static void cdrom_insert_lock_acquired(libxl__egc *egc,if (rc) goto out; + rc =libxl__ev_time_register_rel (ao, &cis->time, + cdrom_insert_timout, +LIBXL_HOTPLUG_TIMEOUT *1000 ); +if (rc) goto out; +/* We need to eject the original image first. This is implemented * by inserting empty media. JSON is not updated. */
@@ -950,12 +960,23 @@ static void cdrom_insert_inserted(libxl__egc *egc,cdrom_insert_done (egc, cis, rc);/* must be last */ } +static void cdrom_insert_timout (libxl__egc *egc,libxl__ev_time *ev, +const struct timeval *requested_abs, +int rc) +{ +EGC_GC ; +libxl__cdrom_insert_state *cis =CONTAINER_OF (ev, *cis, time); +LOGD (ERROR , cis->domid,"cdrom insertion timed out" ); +cdrom_insert_done (egc, cis, rc); +} +static void cdrom_insert_done (libxl__egc *egc,libxl__cdrom_insert_state *cis,int rc) {EGC_GC ; +libxl__ev_time_deregister (gc, &cis->time);libxl__ev_devlock_unlock (gc, &cis->qmp_lock);libxl_device_disk_dispose (&cis->disk_saved);libxl__ao_complete (egc, cis->ao, rc);
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 8/9 libxl: Move qmp_parameters_* prototypes to libxl_internal.h
In reply to Anthony PERARD.. and rename them to libxl__qmp_param_*.
This is to allow other files than libxl_qmp.c to make QMP calls with parameters.
--- tools/libxl/libxl_internal.h | 15 ++++++++ tools/libxl/libxl_qmp.c | 75 +++++++++++++++++------------------- 2 files changed, 51 insertions(+), 39 deletions(-)
tools/libxl/libxl_internal.h +15 -0
@@ -470,6 +470,21 @@ struct libxl__ev_qmp {int msg_id; }; +/* QMP parameters helpers */ + +_hidden void libxl__qmp_param_add_string(libxl__gc *gc, +libxl__json_object **param, +const char *name,const char *s); +_hidden void libxl__qmp_param_add_bool(libxl__gc *gc, +libxl__json_object **param, +const char *name,bool b); +_hidden void libxl__qmp_param_add_integer(libxl__gc *gc, +libxl__json_object **param, +const char *name,const int i); +#define QMP_PARAMETERS_SPRINTF (args, name, format, ...) \ + libxl__qmp_param_add_string(gc, args, name, \ + GCSPRINTF(format, __VA_ARGS__)) +/* * evgen structures, which are the state we use for generating
tools/libxl/libxl_qmp.c +36 -39
201 lines not on this page · raw message
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 9/9 libxl_disk: Use ev_qmp in libxl_cdrom_insert
In reply to Anthony PERARDMake libxl_cdrom_insert asynchronous when QEMU is involved. And have the cdrom opened by libxl, sending a file descriptor to QEMU.
The "opaque" parameter of the "add-fd" can help to figure out what a fdset in QEMU is used for. It can be queried by "query-fdsets".
---
Notes:
v2:
- acked
- assert that there's no payload_fd before openning a cdrom file
- renamed flag asynchronous_callback' -> has_callback'
tools/libxl/libxl_disk.c | 126 +++++++++++++++++++++++++++-------- tools/libxl/libxl_internal.h | 1 - tools/libxl/libxl_qmp.c | 18 ----- 3 files changed, 100 insertions(+), 45 deletions(-)
tools/libxl/libxl_disk.c +100 -26
@@ -650,14 +650,17 @@ typedef struct {libxl__ev_devlock qmp_lock;int dm_ver;libxl__ev_time time; +libxl__ev_qmp qmp; } libxl__cdrom_insert_state;static void cdrom_insert_lock_acquired (libxl__egc *,libxl__ev_devlock *,int rc); -static void cdrom_insert_ejected (libxl__egc *egc, -libxl__cdrom_insert_state *cis); -static void cdrom_insert_inserted (libxl__egc *egc, -libxl__cdrom_insert_state *cis); +static void cdrom_insert_ejected (libxl__egc *egc,libxl__ev_qmp *, +const libxl__json_object *,int rc); +static void cdrom_insert_addfd_cb (libxl__egc *egc,libxl__ev_qmp *, +const libxl__json_object *,int rc); +static void cdrom_insert_inserted (libxl__egc *egc,libxl__ev_qmp *, +const libxl__json_object *,int rc);static void cdrom_insert_timout (libxl__egc *egc,libxl__ev_time *ev,const struct timeval *requested_abs,int rc);
@@ -684,6 +687,10 @@ int libxl_cdrom_insert(libxl_ctx *ctx, uint32_t domid, libxl_device_disk *disk, cis->qmp_lock.ao = ao; cis->qmp_lock.domid = domid;libxl__ev_time_init (&cis->time); +libxl__ev_qmp_init (&cis->qmp); + cis->qmp.ao = ao; + cis->qmp.domid = domid; + cis->qmp.payload_fd =-1 ;libxl_domain_type type =libxl__domain_type (gc, domid);if (type ==LIBXL_DOMAIN_TYPE_INVALID ) {
@@ -757,26 +764,22 @@ static void cdrom_insert_lock_acquired(libxl__egc *egc, LIBXL_HOTPLUG_TIMEOUT * 1000);if (rc) goto out; -/* We need to eject the original image first. This is implemented -* by inserting empty media. JSON is not updated. +/* We need to eject the original image first. +* JSON is not updated. */ if (cis->dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { -libxl_device_disk disk_empty; - -libxl_device_disk_init (&disk_empty); - disk_empty.format =LIBXL_DISK_FORMAT_EMPTY ; - disk_empty.vdev =libxl__strdup (NOGC , cis->disk->vdev); - disk_empty.pdev_path =libxl__strdup (NOGC ,"" ); - disk_empty.is_cdrom =1 ; -libxl__device_disk_setdefault (gc, cis->domid, &disk_empty, false); +libxl__json_object *args =NULL ; +int devid =libxl__device_disk_dev_number (cis->disk->vdev, +NULL ,NULL ); - rc =libxl__qmp_insert_cdrom (gc, cis->domid, &disk_empty); -libxl_device_disk_dispose (&disk_empty); +QMP_PARAMETERS_SPRINTF (&args,"device" ,"ide-%i" , devid); + cis->qmp.callback = cdrom_insert_ejected; + rc =libxl__ev_qmp_send (gc, &cis->qmp,"eject" , args);if (rc) goto out; + }else { +cdrom_insert_ejected (egc, &cis->qmp,NULL ,0 );/* must be last */ } - -cdrom_insert_ejected (egc, cis);/* must be last */ return ; out:
@@ -784,10 +787,12 @@ static void cdrom_insert_lock_acquired(libxl__egc *egc, }static void cdrom_insert_ejected (libxl__egc *egc, -libxl__cdrom_insert_state *cis) +libxl__ev_qmp *qmp, +const libxl__json_object *response, +int rc) {EGC_GC ; -int rc; +libxl__cdrom_insert_state *cis =CONTAINER_OF (qmp, *cis, qmp);libxl__domain_userdata_lock *data_lock =NULL ;libxl__device device;const char *be_path, *libxl_path;
@@ -795,6 +800,7 @@ static void cdrom_insert_ejected(libxl__egc *egc,xs_transaction_t t =XBT_NULL ;char *tmp;libxl_domain_config d_config; +bool has_callback = false;/* convenience aliases */ libxl_domid domid = cis->domid;
@@ -802,6 +808,8 @@ static void cdrom_insert_ejected(libxl__egc *egc,libxl_domain_config_init (&d_config); +if (rc) goto out; + rc =libxl__device_from_disk (gc, domid, disk, &device);if (rc) goto out; be_path =libxl__device_backend_path (gc, &device);
@@ -857,9 +865,29 @@ static void cdrom_insert_ejected(libxl__egc *egc, rc =libxl__dm_check_start (gc, &d_config, domid);if (rc) goto out; -if (cis->dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN ) { - rc =libxl__qmp_insert_cdrom (gc, domid, disk); +if (cis->dm_ver ==LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN && + disk->format !=LIBXL_DISK_FORMAT_EMPTY ) { +libxl__json_object *args =NULL ; + +assert (qmp->payload_fd ==-1 ); + qmp->payload_fd =open (disk->pdev_path,O_RDONLY ); +if (qmp->payload_fd <0 ) { +LOGED (ERROR , domid,"Failed to open cdrom file %s" , + disk->pdev_path); + rc =ERROR_FAIL ; + goto out; + } + +/* This free form parameter is not use by QEMU or libxl. */ +QMP_PARAMETERS_SPRINTF (&args,"opaque" ,"%s:%s" , +libxl_disk_format_to_string (disk->format), + disk->pdev_path); + qmp->callback = cdrom_insert_addfd_cb; + rc =libxl__ev_qmp_send (gc, qmp,"add-fd" , args);if (rc) goto out; + has_callback = true; + }else { + has_callback = false; } rc =0 ;
@@ -870,16 +898,58 @@ static void cdrom_insert_ejected(libxl__egc *egc,if (data_lock)libxl__unlock_domain_userdata (data_lock);if (rc) {cdrom_insert_done (egc, cis, rc);/* must be last */ - }else { -cdrom_insert_inserted (egc, cis);/* must be last */ + }else if (!has_callback) { +/* Only called if no asynchronous callback are set. */ +cdrom_insert_inserted (egc, qmp,NULL ,0 );/* must be last */ + } +} + +static void cdrom_insert_addfd_cb (libxl__egc *egc, +libxl__ev_qmp *qmp, +const libxl__json_object *response, +int rc) +{ +EGC_GC ; +libxl__cdrom_insert_state *cis =CONTAINER_OF (qmp, *cis, qmp); +libxl__json_object *args =NULL ; +const libxl__json_object *o; +int devid; +int fdset; + +/* convenience aliases */ +libxl_device_disk *disk = cis->disk; + +close (qmp->payload_fd); + qmp->payload_fd =-1 ; + +if (rc) goto out; + + o =libxl__json_map_get ("fdset-id" , response,JSON_INTEGER ); +if (!o) { + rc =ERROR_FAIL ; + goto out; } + fdset =libxl__json_object_get_integer (o); + + devid =libxl__device_disk_dev_number (disk->vdev,NULL ,NULL ); +QMP_PARAMETERS_SPRINTF (&args,"device" ,"ide-%i" , devid); +QMP_PARAMETERS_SPRINTF (&args,"target" ,"/dev/fdset/%d" , fdset); +libxl__qmp_param_add_string (gc, &args,"arg" , +libxl__qemu_disk_format_string (disk->format)); + qmp->callback = cdrom_insert_inserted; + rc =libxl__ev_qmp_send (gc, qmp,"change" , args); +out: +if (rc) +cdrom_insert_done (egc, cis, rc);/* must be last */ }static void cdrom_insert_inserted (libxl__egc *egc, -libxl__cdrom_insert_state *cis) +libxl__ev_qmp *qmp, +const libxl__json_object *response, +int rc) {EGC_GC ; -int rc; +libxl__cdrom_insert_state *cis =CONTAINER_OF (qmp, *cis, qmp);libxl__domain_userdata_lock *data_lock =NULL ;libxl_domain_config d_config;flexarray_t *insert =NULL ;
@@ -894,6 +964,8 @@ static void cdrom_insert_inserted(libxl__egc *egc,libxl_domain_config_init (&d_config); +if (rc) goto out; + rc =libxl__device_from_disk (gc, domid, disk, &device);if (rc) goto out; be_path =libxl__device_backend_path (gc, &device);
@@ -977,6 +1049,8 @@ static void cdrom_insert_done(libxl__egc *egc,EGC_GC ;libxl__ev_time_deregister (gc, &cis->time); +libxl__ev_qmp_dispose (gc, &cis->qmp); +if (cis->qmp.payload_fd >=0 )close (cis->qmp.payload_fd);libxl__ev_devlock_unlock (gc, &cis->qmp_lock);libxl_device_disk_dispose (&cis->disk_saved);libxl__ao_complete (egc, cis->ao, rc);
tools/libxl/libxl_internal.h +0 -1
@@ -1960,7 +1960,6 @@ _hidden int libxl__qmp_resume(libxl__gc *gc, int domid); _hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);/* Set dirty bitmap logging status */ _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable); -_hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);/* Add a virtual CPU */ _hidden int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int index);/* Query the bitmap of CPUs */
tools/libxl/libxl_qmp.c +0 -18
@@ -1118,24 +1118,6 @@ int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable) NULL, NULL); } -int libxl__qmp_insert_cdrom (libxl__gc *gc,int domid, -const libxl_device_disk *disk) -{ -libxl__json_object *args =NULL ; -int dev_number =libxl__device_disk_dev_number (disk->vdev,NULL ,NULL ); - -QMP_PARAMETERS_SPRINTF (&args,"device" ,"ide-%i" , dev_number); - -if (disk->format ==LIBXL_DISK_FORMAT_EMPTY ) { -return qmp_run_command (gc, domid,"eject" , args,NULL ,NULL ); - }else { -libxl__qmp_param_add_string (gc, &args,"target" , disk->pdev_path); -libxl__qmp_param_add_string (gc, &args,"arg" , -libxl__qemu_disk_format_string (disk->format)); -return qmp_run_command (gc, domid,"change" , args,NULL ,NULL ); - } -} -int libxl__qmp_cpu_add (libxl__gc *gc,int domid,int idx) {libxl__json_object *args =NULL ;
Signature · 7 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel