Skip to contentSkip to conversation

(no subject)

PATCH In review Ian Jackson 3 participants 40 messages Dec 17, 2013 – Mar 13, 2014 Arm/actions semi architecture

lore
Share & tools
lore
#0 lore raw

This series removes the usleeps and waiting loops in libxl_dom.c and replaces them with event-callback code.

Firstly, some documentation of things I had to reverse-engineer:
01/23 xen: Document XEN_DOMCTL_subscribe
02/23 xen: Document that EVTCHNOP_bind_interdomain signals 03/23 docs: Document event-channel-based suspend protocol 04/23 libxc: Document xenctrl.h event channel calls
Arguably these might be 4.4 material (hence the CC to George).

We want some additional libxl event facilities:
05/23 libxl: init: Provide a gc later in libxl_ctx_alloc 06/23 libxl: init: libxl__poller_init and _get take gc 07/23 libxl: events: const-correct *_inuse, *_isregistered 08/23 libxl: events: Provide libxl__xswait_* 09/23 libxl: events: Use libxl__xswait_* in spawn code 10/23 libxl: events: Provide libxl__ev_evtchn*

We need to clean up some unfortunate code in libxc:
11/23 libxc: suspend: Rename, improve xc_suspend_evtchn_init 12/23 libxc: suspend: Fix suspend event channel locking

We do some shuffling around of the libxl suspend control flow:
13/23 libxl: suspend: Async libxl__domain_suspend_callback 14/23 libxl: suspend: Async domain_suspend_callback_common 15/23 libxl: suspend: Reorg domain_suspend_callback_common 16/23 libxl: suspend: New libxl__domain_pvcontrol_xspath 17/23 libxl: suspend: New domain_suspend_pvcontrol_acked
No functional change in those five. These changes are broken down just to make the changes reviewable.

Finally, we can start to work on the event code, removing the bugs, usleeps and loops one at a time:
18/23 libxl: suspend: domain_suspend_callback_common xs errs 19/23 libxl: suspend: Async xenstore pvcontrol wait 20/23 libxl: suspend: Abolish usleeps in domain suspend wait 21/23 libxl: suspend: Fix suspend wait corner cases 22/23 libxl: suspend: Async evtchn wait
23/23 libxl: suspend: Apply guest timeout in evtchn case

I have tested this with a Debian pvops kernel (xenstore pvcontrol suspend signalling) and OpenSUSE 11 (event channel suspend signalling).

Shriram: I haven't really touched the Remus-specific code here but this series ought to be suitable for you to base things on, assuming my co-maintainers like the general approach.

Thanks,
Ian.

#1 lore raw

PATCH 1/23 xen: Document XEN_DOMCTL_subscribe

Arguably this domctl is misnamed. But, for now, document its actual behaviour (reverse-engineered from the code and found in the commit message for 4539594d46f9) under its actual name.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
CC Shriram Rajagopalan <[email protected]>
CC Jan Beulich <[email protected]>
---
 tools/libxc/xenctrl.h       |    4 +++-
 xen/include/public/domctl.h |   16 ++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
tools/libxc/xenctrl.h +3 -1
index 6e58ebe..096a590 100644
@@ -1452,8 +1452,10 @@ int xc_version(xc_interface *xch, int cmd, void *arg);
 int xc_flask_op(xc_interface *xch, xen_flask_op_t *op);
 
 /*
- * Subscribe to state changes in a domain via evtchn.
+ * Subscribe to domain suspend via evtchn.
  * Returns -1 on failure, in which case errno will be set appropriately.
+ * Just calls XEN_DOMCTL_subscribe - see the caveats for that domctl
+ * (in its doc comment in domctl.h).
  */
 int xc_domain_subscribe_for_suspend(
     xc_interface *xch, domid_t domid, evtchn_port_t port);
xen/include/public/domctl.h +16 -0
index 01a3652..91f01fa 100644
@@ -619,6 +619,22 @@ typedef struct xen_domctl_cpuid xen_domctl_cpuid_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_cpuid_t);
 #endif
 
+/*
+ * Arranges that if the domain suspends (specifically, if it shuts
+ * down with code SHUTDOWN_suspend), this event channel will be
+ * notified.
+ *
+ * This is _instead of_ the usual notification to the global
+ * VIRQ_DOM_EXC.  (In most systems that pirq is owned by xenstored.)
+ *
+ * Only one subscription per domain is possible.  Last subscriber
+ * wins; others are silently displaced.
+ *
+ * NB that contrary to the rather general name, it only applies to
+ * domain shutdown with code suspend.  Shutdown for other reasons
+ * (including crash), and domain death, are notified to VIRQ_DOM_EXC
+ * regardless.
+ */
 /* XEN_DOMCTL_subscribe */
 struct xen_domctl_subscribe {
     uint32_t port; /* IN */
Signature · 1 line
1.7.10.4
#2 lore raw

PATCH 2/23 xen: Document that EVTCHNOP_bind_interdomain signals

In reply to Ian Jackson

EVTCHNOP_bind_interdomain signals the event channel. Document this.

Also explain the usual use pattern.

CC Keir Fraser <[email protected]>
CC Jan Beulich <[email protected]>
---
 xen/include/public/event_channel.h |   11 +++++++++++
 1 file changed, 11 insertions(+)
xen/include/public/event_channel.h +11 -0
index b78ee32..49ac8cc 100644
@@ -101,6 +101,17 @@ typedef struct evtchn_alloc_unbound evtchn_alloc_unbound_t;
  * a port that is unbound and marked as accepting bindings from the calling
  * domain. A fresh port is allocated in the calling domain and returned as
  * <local_port>.
+ *
+ * In case the peer domain has already tried to set our event channel
+ * pending, before it was bound, EVTCHNOP_bind_interdomain always sets
+ * the local event channel pending.
+ *
+ * The usual pattern of use, in the guest's upcall (or subsequent
+ * handler) is as follows: (Re-enable the event channel for subsequent
+ * signalling and then) check for the existence of whatever condition
+ * is being waited for by other means, and take whatever action is
+ * needed (if any).
+ *
  * NOTES:
  *  1. <remote_dom> may be DOMID_SELF, allowing loopback connections.
  */
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH 3/23 docs: Document event-channel-based suspend protocol

In reply to Ian Jackson

Document the event channel protocol in xenstore-paths.markdown, in the section for ~/device/suspend/event-channel.

Protocol reverse-engineered from commentary and commit messages of
4539594d46f9 Add facility to get notification of domain suspend ... 17636f47a474 Teach xc_save to use event-channel-based ...
and implementations in
xc_save (current version)
libxl (current version)
linux-2.6.18-xen (mercurial 1241:2993033a77ca)

CC Keir Fraser <[email protected]>
CC Shriram Rajagopalan <[email protected]>
---
 docs/misc/xenstore-paths.markdown |   23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
docs/misc/xenstore-paths.markdown +21 -2
index a0fc003..70ab7f4 100644
@@ -143,10 +143,29 @@ The size of the video RAM this domain is configured with.
 
 #### ~/device/suspend/event-channel = ""|EVTCHN [w]
 
-The domain's suspend event channel. The use of a suspend event channel
-is optional at the domain's discretion. The toolstack will create this
+The domain's suspend event channel. The toolstack will create this
 path with an empty value which the guest may choose to overwrite.
 
+If the guest overwrites this, it will be with the number of an unbound
+event channel port it has acquired.  The toolstack is expected to use
+an interdomain bind, and then, when it wishes to ask the guest to
+suspend, to signal the event channel.
+
+The guest does not need to explicitly acknowledge the request; indeed,
+there is no explicit signalling by the guest in the reverse direction.
+The guest, when it is ready, simply shuts down (`SCHEDOP_shutdown`)
+with reason code `SHUTDOWN_suspend`.  The toolstack is expected to use
+`XEN_DOMCTL_subscribe` to be alerted to guest state changes, and
+`XEN_SYSCTL_getdomaininfolist` to verify that the domain has
+suspended.
+
+Note that the use of this event channel suspend protocol is optional
+for both sides.  By writing a non-empty string to the node, the guest
+is advertising its support.  However, the toolstack is at liberty to
+use the xenstore-based protocol instead (see ~/control/shutdown,
+below) even if the guest has advertised support for the event channel
+protocol.
+
 #### ~/hvmloader/generation-id-address = ADDRESS [r,HVM,INTERNAL]
 
 The hexadecimal representation of the address of the domain's
Signature · 1 line
1.7.10.4
#4 lore raw

PATCH 4/23 libxc: Document xenctrl.h event channel calls

In reply to Ian Jackson

Provide semantic documentation for how the libxc calls relate to the hypervisor interface, and how they are to be used.

Also document the bug (present at least in Linux 3.12) that setting the evtchn fd to nonblocking doesn't in fact make xc_evtchn_pending nonblocking, and describe the appropriate workaround.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
CC Jan Beulich <[email protected]>
CC Konrad Rzeszutek Wilk <[email protected]>
---
 tools/libxc/xenctrl.h |   27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
tools/libxc/xenctrl.h +26 -1
index 096a590..13f816b 100644
@@ -1043,6 +1043,18 @@ int xc_evtchn_close(xc_evtchn *xce);
 
 /*
  * Return an fd that can be select()ed on.
+ *
+ * Note that due to bugs, setting this fd to non blocking may not
+ * work: you would hope that it would result in xc_evtchn_pending
+ * failing with EWOULDBLOCK if there are no events signaled, but in
+ * fact it may block.  (Bug is present in at least Linux 3.12, and
+ * perhaps on other platforms or later version.)
+ *
+ * To be safe, you must use poll() or select() before each call to
+ * xc_evtchn_pending.  If you have multiple threads (or processes)
+ * sharing a single xce handle this will not work, and there is no
+ * straightforward workaround.  Please design your program some other
+ * way.
  */
 int xc_evtchn_fd(xc_evtchn *xce);
 
@@ -1082,7 +1094,20 @@ int xc_evtchn_unbind(xc_evtchn *xce, evtchn_port_t port);
 
 /*
  * Return the next event channel to become pending, or -1 on failure, in which
- * case errno will be set appropriately.  
+ * case errno will be set appropriately.
+ *
+ * At the hypervisor level the event channel will have been masked,
+ * and then cleared, by the underlying machinery (evtchn kernel
+ * driver, or equivalent).  So if the event channel is signaled again
+ * after it is returned here, it will be queued up, and delivered
+ * again after you unmask it.  (See the documentation in the Xen
+ * public header event_channel.h.)
+ *
+ * On receiving the notification from xc_evtchn_pending, you should
+ * normally: check (by other means) what work needs doing; do the
+ * necessary work (if any); unmask the event channel with
+ * xc_evtchn_unmask (if you want to receive any further
+ * notifications).
  */
 evtchn_port_or_error_t
 xc_evtchn_pending(xc_evtchn *xce);
Signature · 1 line
1.7.10.4
#5 lore raw

PATCH 5/23 libxl: init: Provide a gc later in libxl_ctx_alloc

In reply to Ian Jackson

Provide libxl__gc *gc for the second half of libxl_ctx_alloc. (For the first half of the function, gc is in scope but set to NULL.)

This makes it possible to make gc-requiring calls. For example, it makes error logging more convenient.

Make use of this by changing the logging calls to use the LOG* convenience macros.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
tools/libxl/libxl.c +7 -4
index e8ad610..cfd1435 100644
@@ -25,6 +25,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags, xentoollog_logger * lg)
 {
     libxl_ctx *ctx = NULL;
+    libxl__gc gc_buf, *gc = NULL;
     int rc;
 
     if (version != LIBXL_VERSION) { rc = ERROR_VERSION; goto out; }
@@ -79,6 +80,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     }
 
     /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
+    LIBXL_INIT_GC(gc_buf,ctx);
+    gc = &gc_buf;
+    /* Now gc is useable */
 
     rc = libxl__atfork_init(ctx);
     if (rc) goto out;
@@ -88,8 +92,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 
     ctx->xch = xc_interface_open(lg,lg,0);
     if (!ctx->xch) {
-        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, errno,
-                        "cannot open libxc handle");
+        LOGEV(ERROR, errno, "cannot open libxc handle");
         rc = ERROR_FAIL; goto out;
     }
 
@@ -97,8 +100,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     if (!ctx->xsh)
         ctx->xsh = xs_domain_open();
     if (!ctx->xsh) {
-        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, errno,
-                        "cannot connect to xenstore");
+        LOGEV(ERROR, errno, "cannot connect to xenstore");
         rc = ERROR_FAIL; goto out;
     }
 
@@ -106,6 +108,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     return 0;
 
  out:
+    if (gc) libxl__free_all(gc);
     libxl_ctx_free(ctx);
     *pctx = NULL;
     return rc;
Signature · 1 line
1.7.10.4
#6 lore raw

Re: [PATCH 05/23] libxl: init: Provide a gc later in libxl_ctx_alloc

Quoted 2 lines · Ian Jackson

Provide libxl__gc *gc for the second half of libxl_ctx_alloc. (For the first half of the function, gc is in scope but set to NULL.)

libxl uses -Wno-declaration-after-statement so it would be valid to just declare it at the point it becomes valid, would avoid mistakes with people adding uses before this point.

Quoted 63 lines

This makes it possible to make gc-requiring calls. For example, it makes error logging more convenient.

Make use of this by changing the logging calls to use the LOG* convenience macros.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
tools/libxl/libxl.c +7 -4
index e8ad610..cfd1435 100644
@@ -25,6 +25,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags, xentoollog_logger * lg)
 {
     libxl_ctx *ctx = NULL;
+    libxl__gc gc_buf, *gc = NULL;
     int rc;
 
     if (version != LIBXL_VERSION) { rc = ERROR_VERSION; goto out; }
@@ -79,6 +80,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     }
 
     /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
+    LIBXL_INIT_GC(gc_buf,ctx);
+    gc = &gc_buf;
+    /* Now gc is useable */
 
     rc = libxl__atfork_init(ctx);
     if (rc) goto out;
@@ -88,8 +92,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 
     ctx->xch = xc_interface_open(lg,lg,0);
     if (!ctx->xch) {
-        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, errno,
-                        "cannot open libxc handle");
+        LOGEV(ERROR, errno, "cannot open libxc handle");
         rc = ERROR_FAIL; goto out;
     }
 
@@ -97,8 +100,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     if (!ctx->xsh)
         ctx->xsh = xs_domain_open();
     if (!ctx->xsh) {
-        LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, errno,
-                        "cannot connect to xenstore");
+        LOGEV(ERROR, errno, "cannot connect to xenstore");
         rc = ERROR_FAIL; goto out;
     }
 
@@ -106,6 +108,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     return 0;
 
  out:
+    if (gc) libxl__free_all(gc);
     libxl_ctx_free(ctx);
     *pctx = NULL;
     return rc;
#7 lore raw

Re: [PATCH 05/23] libxl: init: Provide a gc later in libxl_ctx_alloc

Ian Campbell writes ("Re: [PATCH 05/23] libxl: init: Provide a gc later in libxl_ctx_alloc"):

Quoted 7 lines · Ian Jackson
Quoted 2 lines

Provide libxl__gc *gc for the second half of libxl_ctx_alloc. (For the first half of the function, gc is in scope but set to NULL.)

libxl uses -Wno-declaration-after-statement so it would be valid to just declare it at the point it becomes valid, would avoid mistakes with people adding uses before this point.

However, that would involve changing the "goto out" in the malloc failure case to something else, because "goto out" would jump past the initialisation of gc to NULL.

The malloc failure case can't be handled with our usual fatal error logic because that depends on having a ctx. I guess we could have a dummy ctx but it would get fiddly.

Ian.

#8 lore raw

PATCH 6/23 libxl: init: libxl__poller_init and _get take gc

In reply to Ian Jackson

Change libxl__poller_init and libxl__poller__get to take a libxl__gc* rather than a libxl_ctx*. The gc is not used for memory allocation but simply to provide the standard local variable "gc" expected by the convenience macros. Doing this makes the error logging more convenient.

Hence, convert the logging calls to use the LOG* convenience macros.

And consequently, change the call sites, and the function bodies to use CTX rather than ctx.

Also convert a call to malloc() (with error check) in libxl__poller_get, to libxl__zalloc (no error check needed).

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl.c          |    2 +-
 tools/libxl/libxl_event.c    |   25 ++++++++++---------------
 tools/libxl/libxl_internal.h |    4 ++--
 3 files changed, 13 insertions(+), 18 deletions(-)
tools/libxl/libxl.c +1 -1
index cfd1435..85b56a9 100644
@@ -87,7 +87,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     rc = libxl__atfork_init(ctx);
     if (rc) goto out;
 
-    rc = libxl__poller_init(ctx, &ctx->poller_app);
+    rc = libxl__poller_init(gc, &ctx->poller_app);
     if (rc) goto out;
 
     ctx->xch = xc_interface_open(lg,lg,0);
tools/libxl/libxl_event.c +10 -15
index bdef7ac..9c4fe1c 100644
@@ -1274,7 +1274,7 @@ int libxl_event_check(libxl_ctx *ctx, libxl_event **event_r,
  * Manipulation of pollers
  */
 
-int libxl__poller_init(libxl_ctx *ctx, libxl__poller *p)
+int libxl__poller_init(libxl__gc *gc, libxl__poller *p)
 {
     int r, rc;
     p->fd_polls = 0;
@@ -1282,15 +1282,15 @@ int libxl__poller_init(libxl_ctx *ctx, libxl__poller *p)
 
     r = pipe(p->wakeup_pipe);
     if (r) {
-        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "cannot create poller pipe");
+        LOGE(ERROR, "cannot create poller pipe");
         rc = ERROR_FAIL;
         goto out;
     }
 
-    rc = libxl_fd_set_nonblock(ctx, p->wakeup_pipe[0], 1);
+    rc = libxl_fd_set_nonblock(CTX, p->wakeup_pipe[0], 1);
     if (rc) goto out;
 
-    rc = libxl_fd_set_nonblock(ctx, p->wakeup_pipe[1], 1);
+    rc = libxl_fd_set_nonblock(CTX, p->wakeup_pipe[1], 1);
     if (rc) goto out;
 
     return 0;
@@ -1308,25 +1308,20 @@ void libxl__poller_dispose(libxl__poller *p)
     free(p->fd_rindices);
 }
 
-libxl__poller *libxl__poller_get(libxl_ctx *ctx)
+libxl__poller *libxl__poller_get(libxl__gc *gc)
 {
     /* must be called with ctx locked */
     int rc;
 
-    libxl__poller *p = LIBXL_LIST_FIRST(&ctx->pollers_idle);
+    libxl__poller *p = LIBXL_LIST_FIRST(&CTX->pollers_idle);
     if (p) {
         LIBXL_LIST_REMOVE(p, entry);
         return p;
     }
 
-    p = malloc(sizeof(*p));
-    if (!p) {
-        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "cannot allocate poller");
-        return 0;
-    }
-    memset(p, 0, sizeof(*p));
+    p = libxl__zalloc(NOGC, sizeof(*p));
 
-    rc = libxl__poller_init(ctx, p);
+    rc = libxl__poller_init(gc, p);
     if (rc) {
         free(p);
         return NULL;
@@ -1446,7 +1441,7 @@ int libxl_event_wait(libxl_ctx *ctx, libxl_event **event_r,
     EGC_INIT(ctx);
     CTX_LOCK;
 
-    poller = libxl__poller_get(ctx);
+    poller = libxl__poller_get(gc);
     if (!poller) { rc = ERROR_FAIL; goto out; }
 
     for (;;) {
@@ -1622,7 +1617,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
     if (how) {
         ao->how = *how;
     } else {
-        ao->poller = libxl__poller_get(ctx);
+        ao->poller = libxl__poller_get(&ao->gc);
         if (!ao->poller) goto out;
     }
     libxl__log(ctx,XTL_DEBUG,-1,file,line,func,
tools/libxl/libxl_internal.h +2 -2
index 1bd23ff..2712005 100644
@@ -823,13 +823,13 @@ _hidden void libxl__event_disaster(libxl__egc*, const char *msg, int errnoval,
 
 /* Fills in, or disposes of, the resources held by, a poller whose
  * space the caller has allocated.  ctx must be locked. */
-_hidden int libxl__poller_init(libxl_ctx *ctx, libxl__poller *p);
+_hidden int libxl__poller_init(libxl__gc *gc, libxl__poller *p);
 _hidden void libxl__poller_dispose(libxl__poller *p);
 
 /* Obtain a fresh poller from malloc or the idle list, and put it
  * away again afterwards.  _get can fail, returning NULL.
  * ctx must be locked. */
-_hidden libxl__poller *libxl__poller_get(libxl_ctx *ctx);
+_hidden libxl__poller *libxl__poller_get(libxl__gc *gc);
 _hidden void libxl__poller_put(libxl_ctx*, libxl__poller *p /* may be NULL */);
 
 /* Notifies whoever is polling using p that they should wake up.
Signature · 1 line
1.7.10.4
#9 lore raw

Re: [PATCH 06/23] libxl: init: libxl__poller_init and _get take gc

tools/libxl/libxl_event.c libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid, · line 1620

Quoted 6 lines · Ian Jackson
tools/libxl/libxl_event.c +1 -1
@@ -1622,7 +1617,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
     if (how) {
         ao->how = *how;
     } else {
-        ao->poller = libxl__poller_get(ctx);
+        ao->poller = libxl__poller_get(&ao->gc);

this is a bit subtle, libxl__poller_get would previously have gotten the toplevel ctx gc and not the ao-gc, if it had thought to init a GC at all.

It only actually uses whatever it gets for logging, so that's ok, but I'm not sure which would be the correct/expected gc to use. I take it you considered this?

libxl__poller_get does a NOGC allocation and the associated manual memory mgmt -- now that it has the ao->gc in hand could it use that? Is the poller's lifecycle entirely contained within the ao?

Ian.

#10 lore raw

Re: [PATCH 06/23] libxl: init: libxl__poller_init and _get take gc

Ian Campbell writes ("Re: [PATCH 06/23] libxl: init: libxl__poller_init and _get take gc"):

Quoted 11 lines · Ian Jackson

tools/libxl/libxl_event.c libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid, · line 1620

Quoted 6 lines
tools/libxl/libxl_event.c +1 -1
@@ -1622,7 +1617,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
     if (how) {
         ao->how = *how;
     } else {
-        ao->poller = libxl__poller_get(ctx);
+        ao->poller = libxl__poller_get(&ao->gc);

this is a bit subtle, libxl__poller_get would previously have gotten the toplevel ctx gc and not the ao-gc, if it had thought to init a GC at all.

Yes.

Quoted 3 lines

It only actually uses whatever it gets for logging, so that's ok, but I'm not sure which would be the correct/expected gc to use. I take it you considered this?

Indeed.

Quoted 3 lines

libxl__poller_get does a NOGC allocation and the associated manual memory mgmt -- now that it has the ao->gc in hand could it use that? Is the poller's lifecycle entirely contained within the ao?

No, it isn't. The poller is longer-lived, mainly as a way to avoid having to recreate the pipe on each occasion.

Ian.

#11 lore raw

PATCH 7/23 libxl: events: const-correct *_inuse, *_isregistered

In reply to Ian Jackson

The comments for libxl__ev_time_isregistered and the corresponding watch function even say that these should be const. Make it so.

Also fix libxl__ev_child_inuse and libxl__ev_spawn_inuse.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_internal.h |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tools/libxl/libxl_internal.h +4 -4
index 2712005..51d6c6d 100644
@@ -716,7 +716,7 @@ _hidden int libxl__ev_fd_modify(libxl__gc*, libxl__ev_fd *ev,
 _hidden void libxl__ev_fd_deregister(libxl__gc*, libxl__ev_fd *ev);
 static inline void libxl__ev_fd_init(libxl__ev_fd *efd)
                     { efd->fd = -1; }
-static inline int libxl__ev_fd_isregistered(libxl__ev_fd *efd)
+static inline int libxl__ev_fd_isregistered(const libxl__ev_fd *efd)
                     { return efd->fd >= 0; }
 
 _hidden int libxl__ev_time_register_rel(libxl__gc*, libxl__ev_time *ev_out,
@@ -732,7 +732,7 @@ _hidden int libxl__ev_time_modify_abs(libxl__gc*, libxl__ev_time *ev,
 _hidden void libxl__ev_time_deregister(libxl__gc*, libxl__ev_time *ev);
 static inline void libxl__ev_time_init(libxl__ev_time *ev)
                 { ev->func = 0; }
-static inline int libxl__ev_time_isregistered(libxl__ev_time *ev)
+static inline int libxl__ev_time_isregistered(const libxl__ev_time *ev)
                 { return !!ev->func; }
 
 
@@ -772,7 +772,7 @@ _hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
                                  libxl__ev_child_callback *death);
 static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
                 { childw_out->pid = -1; }
-static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
+static inline int libxl__ev_child_inuse(const libxl__ev_child *childw_out)
                 { return childw_out->pid >= 0; }
 
 /* Useable (only) in the child to once more make the ctx useable for
@@ -1213,7 +1213,7 @@ struct libxl__spawn_state {
     libxl__ev_xswatch xswatch;
 };
 
-static inline int libxl__spawn_inuse(libxl__spawn_state *ss)
+static inline int libxl__spawn_inuse(const libxl__spawn_state *ss)
     { return libxl__ev_child_inuse(&ss->mid); }
 
 /*
Signature · 1 line
1.7.10.4
Acked-by on v1 7/23
#12 lore raw

Re: [PATCH 07/23] libxl: events: const-correct *_inuse, *_isregistered

Quoted 7 lines · Ian Jackson

The comments for libxl__ev_time_isregistered and the corresponding watch function even say that these should be const. Make it so.

Also fix libxl__ev_child_inuse and libxl__ev_spawn_inuse.

CC Stefano Stabellini <[email protected]>
Quoted 44 lines
---
 tools/libxl/libxl_internal.h |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tools/libxl/libxl_internal.h +4 -4
index 2712005..51d6c6d 100644
@@ -716,7 +716,7 @@ _hidden int libxl__ev_fd_modify(libxl__gc*, libxl__ev_fd *ev,
 _hidden void libxl__ev_fd_deregister(libxl__gc*, libxl__ev_fd *ev);
 static inline void libxl__ev_fd_init(libxl__ev_fd *efd)
                     { efd->fd = -1; }
-static inline int libxl__ev_fd_isregistered(libxl__ev_fd *efd)
+static inline int libxl__ev_fd_isregistered(const libxl__ev_fd *efd)
                     { return efd->fd >= 0; }
 
 _hidden int libxl__ev_time_register_rel(libxl__gc*, libxl__ev_time *ev_out,
@@ -732,7 +732,7 @@ _hidden int libxl__ev_time_modify_abs(libxl__gc*, libxl__ev_time *ev,
 _hidden void libxl__ev_time_deregister(libxl__gc*, libxl__ev_time *ev);
 static inline void libxl__ev_time_init(libxl__ev_time *ev)
                 { ev->func = 0; }
-static inline int libxl__ev_time_isregistered(libxl__ev_time *ev)
+static inline int libxl__ev_time_isregistered(const libxl__ev_time *ev)
                 { return !!ev->func; }
 
 
@@ -772,7 +772,7 @@ _hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
                                  libxl__ev_child_callback *death);
 static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
                 { childw_out->pid = -1; }
-static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
+static inline int libxl__ev_child_inuse(const libxl__ev_child *childw_out)
                 { return childw_out->pid >= 0; }
 
 /* Useable (only) in the child to once more make the ctx useable for
@@ -1213,7 +1213,7 @@ struct libxl__spawn_state {
     libxl__ev_xswatch xswatch;
 };
 
-static inline int libxl__spawn_inuse(libxl__spawn_state *ss)
+static inline int libxl__spawn_inuse(const libxl__spawn_state *ss)
     { return libxl__ev_child_inuse(&ss->mid); }
 
 /*
#13 lore raw

PATCH 8/23 libxl: events: Provide libxl__xswait_*

In reply to Ian Jackson

This is an ao utility for for conveniently doing a timed wait on xenstore. It handles setting up and cancelling the timeout, and also conveniently reads the key for you.

No callers yet in this patch.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_aoutils.c  |   77 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   52 ++++++++++++++++++++++++++++
 2 files changed, 129 insertions(+)
tools/libxl/libxl_aoutils.c +77 -0
index b4eb6e5..477717b 100644
@@ -16,6 +16,83 @@
 
 #include "libxl_internal.h"
 
+/*----- xswait -----*/
+
+static libxl__ev_xswatch_callback xswait_xswatch_callback;
+static libxl__ev_time_callback xswait_timeout_callback;
+static void xswait_report_error(libxl__egc*, libxl__xswait_state*, int rc);
+
+void libxl__xswait_init(libxl__xswait_state *xswa)
+{
+    libxl__ev_time_init(&xswa->time_ev);
+    libxl__ev_xswatch_init(&xswa->watch_ev);
+}
+
+void libxl__xswait_stop(libxl__gc *gc, libxl__xswait_state *xswa)
+{
+    libxl__ev_time_deregister(gc, &xswa->time_ev);
+    libxl__ev_xswatch_deregister(gc, &xswa->watch_ev);
+}
+
+bool libxl__xswait_inuse(const libxl__xswait_state *xswa)
+{
+    bool time_inuse = libxl__ev_time_isregistered(&xswa->time_ev);
+    bool watch_inuse = libxl__ev_xswatch_isregistered(&xswa->watch_ev);
+    assert(time_inuse == watch_inuse);
+    return time_inuse;
+}
+
+int libxl__xswait_start(libxl__gc *gc, libxl__xswait_state *xswa)
+{
+    int rc;
+
+    rc = libxl__ev_time_register_rel(gc, &xswa->time_ev,
+                                     xswait_timeout_callback, xswa->timeout_ms);
+    if (rc) goto err;
+
+    rc = libxl__ev_xswatch_register(gc, &xswa->watch_ev,
+                                    xswait_xswatch_callback, xswa->path);
+    if (rc) goto err;
+
+    return 0;
+
+ err:
+    libxl__xswait_stop(gc, xswa);
+    return rc;
+}
+
+void xswait_xswatch_callback(libxl__egc *egc, libxl__ev_xswatch *xsw,
+                             const char *watch_path, const char *event_path)
+{
+    EGC_GC;
+    libxl__xswait_state *xswa = CONTAINER_OF(xsw, *xswa, watch_ev);
+    int rc;
+    const char *data;
+
+    rc = libxl__xs_read_checked(gc, XBT_NULL, xswa->path, &data);
+    if (rc) { xswait_report_error(egc, xswa, rc); return; }
+
+    xswa->callback(egc, xswa, 0, data);
+}
+
+void xswait_timeout_callback(libxl__egc *egc, libxl__ev_time *ev,
+                             const struct timeval *requested_abs)
+{
+    EGC_GC;
+    libxl__xswait_state *xswa = CONTAINER_OF(ev, *xswa, time_ev);
+    LOG(DEBUG, "%s: xswait timeout (path=%s)", xswa->what, xswa->path);
+    xswait_report_error(egc, xswa, ERROR_TIMEDOUT);
+}
+
+static void xswait_report_error(libxl__egc *egc, libxl__xswait_state *xswa,
+                                int rc)
+{
+    EGC_GC;
+    libxl__xswait_stop(gc, xswa);
+    xswa->callback(egc, xswa, rc, 0);
+}
+
+
 /*----- data copier -----*/
 
 void libxl__datacopier_init(libxl__datacopier_state *dc)
tools/libxl/libxl_internal.h +52 -0

58 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#14 lore raw

Re: [PATCH 08/23] libxl: events: Provide libxl__xswait_*

Quoted 9 lines · Ian Jackson
tools/libxl/libxl_internal.h +9 -0
+/*----- xswait: wait for a xenstore node to be suitable -----*/
+
+typedef struct libxl__xswait_state libxl__xswait_state;
+
+/*
+ * rc describes the circumstances of this callback:
+ *
+ * rc==0
+ *

tools/libxl/libxl_internal.h _hidden int libxl__create_pci_backend(libxl__gc *gc, uint32_t domid, · line 1089

Quoted 10 lines
tools/libxl/libxl_internal.h +10 -0
+ *     The xenstore path (may have) changed.  It has been read for
+ *     you.  The result is in data (allocated from the ao gc).
+ *     data may be NULL, which means that the xenstore read gave
+ *     ENOENT.
+ *
+ *     If you are satisfied, you MUST call libxl__xswait_stop.
+ *     Otherwise, xswait will continue waiting and watching and
+ *     will call you back later.
+ *
+ * rc==ETIMEDOUT

Isn't rc actual a libxl error and hence ERROR_TIMEDOUT?

Quoted 6 lines

+ *

+ *     The specified timeout was reached.
+ *     This has NOT been logged (except to the debug log).
+ *     xswait will not continue (but calling libxl__xswait_stop is OK).

+ *
+ * rc!=0

also != E{,RROR_}TIMEDOUT </pedant>

Quoted 30 lines

+ *

+ *     Some other error occurred.
+ *     This HAS been logged.
+ *     xswait will not continue (but calling libxl__xswait_stop is OK).

+ *
+ */

tools/libxl/libxl_internal.h +21 -0
+typedef void libxl__xswait_callback(libxl__egc *egc,
+      libxl__xswait_state *xswa, int rc, const char *data);
+
+struct libxl__xswait_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ao *ao;
+    const char *what; /* for error msgs: noun phrase, what we're waiting for */
+    const char *path;
+    int timeout_ms; /* as for poll(2) */
+    libxl__xswait_callback *callback;
+    /* remaining fields are private to xswait */
+    libxl__ev_time time_ev;
+    libxl__ev_xswatch watch_ev;
+};
+
+void libxl__xswait_init(libxl__xswait_state*);
+void libxl__xswait_stop(libxl__gc*, libxl__xswait_state*); /*idempotent*/
+bool libxl__xswait_inuse(const libxl__xswait_state *ss);
+
+int libxl__xswait_start(libxl__gc*, libxl__xswait_state*);
+
 /*
  *----- spawn -----
  *
#15 lore raw

Re: [PATCH 08/23] libxl: events: Provide libxl__xswait_*

Ian Campbell writes ("Re: [PATCH 08/23] libxl: events: Provide libxl__xswait_*"):

Quoted 9 lines · Ian Jackson
Quoted 6 lines

+/*----- xswait: wait for a xenstore node to be suitable -----*/

+ *     If you are satisfied, you MUST call libxl__xswait_stop.
+ *     Otherwise, xswait will continue waiting and watching and
+ *     will call you back later.

+ *
+ * rc==ETIMEDOUT

Isn't rc actual a libxl error and hence ERROR_TIMEDOUT?

Err, yes.

Quoted 8 lines
Quoted 6 lines

+ *

+ *     The specified timeout was reached.
+ *     This has NOT been logged (except to the debug log).
+ *     xswait will not continue (but calling libxl__xswait_stop is OK).

+ *
+ * rc!=0

also != E{,RROR_}TIMEDOUT </pedant>

Both will be fixed in v2.

Ian.

#16 lore raw

PATCH 9/23 libxl: events: Use libxl__xswait_* in spawn code

In reply to Ian Jackson

Replace open-coded use of ev_time and ev_xswatch with xswait.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_exec.c     |   49 +++++++++++++++---------------------------
 tools/libxl/libxl_internal.h |    3 +--
 2 files changed, 18 insertions(+), 34 deletions(-)
tools/libxl/libxl_exec.c +17 -32
index b6efa0f..4b012dc 100644
@@ -257,10 +257,8 @@ err:
  */
 
 /* Event callbacks. */
-static void spawn_watch_event(libxl__egc *egc, libxl__ev_xswatch *xsw,
-                              const char *watch_path, const char *event_path);
-static void spawn_timeout(libxl__egc *egc, libxl__ev_time *ev,
-                          const struct timeval *requested_abs);
+static void spawn_watch_event(libxl__egc *egc, libxl__xswait_state *xswa,
+                              int rc, const char *xsdata);
 static void spawn_middle_death(libxl__egc *egc, libxl__ev_child *childw,
                                pid_t pid, int status);
 
@@ -274,8 +272,7 @@ static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss);
 void libxl__spawn_init(libxl__spawn_state *ss)
 {
     libxl__ev_child_init(&ss->mid);
-    libxl__ev_time_init(&ss->timeout);
-    libxl__ev_xswatch_init(&ss->xswatch);
+    libxl__xswait_init(&ss->xswait);
 }
 
 int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss)
@@ -288,12 +285,12 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss)
     libxl__spawn_init(ss);
     ss->failed = ss->detaching = 0;
 
-    rc = libxl__ev_time_register_rel(gc, &ss->timeout,
-                                     spawn_timeout, ss->timeout_ms);
-    if (rc) goto out_err;
-
-    rc = libxl__ev_xswatch_register(gc, &ss->xswatch,
-                                    spawn_watch_event, ss->xspath);
+    ss->xswait.ao = ao;
+    ss->xswait.what = GCSPRINTF("%s startup", ss->what);
+    ss->xswait.path = ss->xspath;
+    ss->xswait.timeout_ms = ss->timeout_ms;
+    ss->xswait.callback = spawn_watch_event;
+    rc = libxl__xswait_start(gc, &ss->xswait);
     if (rc) goto out_err;
 
     pid_t middle = libxl__ev_child_fork(gc, &ss->mid, spawn_middle_death);
@@ -350,8 +347,7 @@ int libxl__spawn_spawn(libxl__egc *egc, libxl__spawn_state *ss)
 static void spawn_cleanup(libxl__gc *gc, libxl__spawn_state *ss)
 {
     assert(!libxl__ev_child_inuse(&ss->mid));
-    libxl__ev_time_deregister(gc, &ss->timeout);
-    libxl__ev_xswatch_deregister(gc, &ss->xswatch);
+    libxl__xswait_stop(gc, &ss->xswait);
 }
 
 static void spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
@@ -362,8 +358,7 @@ static void spawn_detach(libxl__gc *gc, libxl__spawn_state *ss)
     int r;
 
     assert(libxl__ev_child_inuse(&ss->mid));
-    libxl__ev_time_deregister(gc, &ss->timeout);
-    libxl__ev_xswatch_deregister(gc, &ss->xswatch);
+    libxl__xswait_stop(gc, &ss->xswait);
 
     pid_t child = ss->mid.pid;
     r = kill(child, SIGKILL);
@@ -387,25 +382,15 @@ static void spawn_fail(libxl__egc *egc, libxl__spawn_state *ss)
     spawn_detach(gc, ss);
 }
 
-static void spawn_timeout(libxl__egc *egc, libxl__ev_time *ev,
-                          const struct timeval *requested_abs)
-{
-    /* Before event, was Attached. */
-    EGC_GC;
-    libxl__spawn_state *ss = CONTAINER_OF(ev, *ss, timeout);
-    LOG(ERROR, "%s: startup timed out", ss->what);
-    spawn_fail(egc, ss); /* must be last */
-}
-
-static void spawn_watch_event(libxl__egc *egc, libxl__ev_xswatch *xsw,
-                              const char *watch_path, const char *event_path)
+static void spawn_watch_event(libxl__egc *egc, libxl__xswait_state *xswa,
+                              int rc, const char *p)
 {
     /* On entry, is Attached. */
     EGC_GC;
-    libxl__spawn_state *ss = CONTAINER_OF(xsw, *ss, xswatch);
-    char *p = libxl__xs_read(gc, 0, ss->xspath);
-    if (!p && errno != ENOENT) {
-        LOG(ERROR, "%s: xenstore read of %s failed", ss->what, ss->xspath);
+    libxl__spawn_state *ss = CONTAINER_OF(xswa, *ss, xswait);
+    if (rc) {
+        if (rc == ERROR_TIMEDOUT)
+            LOG(ERROR, "%s: startup timed out", ss->what);
         spawn_fail(egc, ss); /* must be last */
         return;
     }
tools/libxl/libxl_internal.h +1 -2
index 36de135..5d2e651 100644
@@ -1261,8 +1261,7 @@ struct libxl__spawn_state {
     int detaching; /* we are in Detaching */
     int failed; /* might be true whenever we are not Idle */
     libxl__ev_child mid; /* always in use whenever we are not Idle */
-    libxl__ev_time timeout;
-    libxl__ev_xswatch xswatch;
+    libxl__xswait_state xswait;
 };
 
 static inline int libxl__spawn_inuse(const libxl__spawn_state *ss)
Signature · 1 line
1.7.10.4
#18 lore raw

PATCH 10/23 libxl: events: Provide libxl__ev_evtchn*

In reply to Ian Jackson

This also involves providing a gc for the latter part of libxl_ctx_alloc.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl.c          |    9 +++
 tools/libxl/libxl_event.c    |  152 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   48 +++++++++++++
 3 files changed, 209 insertions(+)
tools/libxl/libxl.c +9 -0
index 85b56a9..51158aa 100644
@@ -60,6 +60,10 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_SLIST_INIT(&ctx->watch_freeslots);
     libxl__ev_fd_init(&ctx->watch_efd);
 
+    ctx->xce = 0;
+    LIBXL_LIST_INIT(&ctx->evtchns_waiting);
+    libxl__ev_fd_init(&ctx->evtchn_efd);
+
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
@@ -104,6 +108,8 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
         rc = ERROR_FAIL; goto out;
     }
 
+    rc = libxl__ctx_evtchn_init(gc);
+
     *pctx = ctx;
     return 0;
 
@@ -147,16 +153,19 @@ int libxl_ctx_free(libxl_ctx *ctx)
     for (i = 0; i < ctx->watch_nslots; i++)
         assert(!libxl__watch_slot_contents(gc, i));
     libxl__ev_fd_deregister(gc, &ctx->watch_efd);
+    libxl__ev_fd_deregister(gc, &ctx->evtchn_efd);
     libxl__ev_fd_deregister(gc, &ctx->sigchld_selfpipe_efd);
 
     /* Now there should be no more events requested from the application: */
 
     assert(LIBXL_LIST_EMPTY(&ctx->efds));
     assert(LIBXL_TAILQ_EMPTY(&ctx->etimes));
+    assert(LIBXL_LIST_EMPTY(&ctx->evtchns_waiting));
 
     if (ctx->xch) xc_interface_close(ctx->xch);
     libxl_version_info_dispose(&ctx->version_info);
     if (ctx->xsh) xs_daemon_close(ctx->xsh);
+    if (ctx->xce) xc_evtchn_close(ctx->xce);
 
     libxl__poller_dispose(&ctx->poller_app);
     assert(LIBXL_LIST_EMPTY(&ctx->pollers_event));
tools/libxl/libxl_event.c +152 -0

158 lines not on this page · raw message

tools/libxl/libxl_internal.h +48 -0

66 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#19 lore raw

Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*

Quoted 2 lines · Ian Jackson

This also involves providing a gc for the latter part of libxl_ctx_alloc.

Didn't you do that bit in an earlier patch? I guess you forgot to update this message when you refactored.

tools/libxl/libxl_event.c void libxl__ev_xswatch_deregister(libxl__gc *gc, libxl__ev_xswatch *w) · line 642

Quoted 6 lines
tools/libxl/libxl_event.c +6 -0
+static void evtchn_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
+                               int fd, short events, short revents)
+{
+    EGC_GC;
+    libxl__ev_evtchn *evev;
+    int port, r, rc;

Should port be evtchn_port_or_error_t ? (from the use I don't think plain evtchn_port_t would be valid)

Quoted 6 lines

[...]

tools/libxl/libxl_internal.h +0 -0
index 5d2e651..c519abc 100644

tools/libxl/libxl_internal.h struct libxl__ev_xswatch { · line 205

Quoted 9 lines
tools/libxl/libxl_internal.h +6 -0
@@ -197,6 +197,17 @@ struct libxl__ev_xswatch {
     uint32_t counterval;
 };
 
+typedef struct libxl__ev_evtchn libxl__ev_evtchn;
+typedef void libxl__ev_evtchn_callback(libxl__egc *egc, libxl__ev_evtchn*);
+struct libxl__ev_evtchn {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ev_evtchn_callback *callback;
+    int port;

evtchn_port_t?

tools/libxl/libxl_internal.h struct libxl__ctx { · line 357

Quoted 13 lines
tools/libxl/libxl_internal.h +6 -0
+    /* remainder is private for libxl__ev_evtchn_... */
+    bool waiting;
+    LIBXL_LIST_ENTRY(libxl__ev_evtchn) entry;
+};
+
 /*
  * An entry in the watch_slots table is either:
  *  1. an entry in the free list, ie NULL or pointer to next free list entry
@@ -343,6 +354,10 @@ struct libxl__ctx {
     uint32_t watch_counter; /* helps disambiguate slot reuse */
     libxl__ev_fd watch_efd;
 
+    xc_evtchn *xce; /* for waiting use only libxl__ev_evtchn* */

The comment means "don't use directly, use libxl__ev-evtchn"?

Or does it imply that for uses other than waiting you may use it directly?

tools/libxl/libxl_internal.h struct libxl__ctx { · line 358

Quoted 1 line
tools/libxl/libxl_internal.h +1 -0
+    LIBXL_LIST_HEAD(, libxl__ev_evtchn) evtchns_waiting;

Are there any locking requirements relating to this list?

tools/libxl/libxl_internal.h static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw) · line 766

Quoted 10 lines
tools/libxl/libxl_internal.h +3 -0
+    libxl__ev_fd evtchn_efd;
+
     LIBXL_TAILQ_HEAD(libxl__evgen_domain_death_list, libxl_evgen_domain_death)
         death_list /* sorted by domid */,
         death_reported;
@@ -748,6 +763,39 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
 
 
 /*
+ * The evtchn facility is one-shot per call t libxl__ev_evtchn_wait.

s/ t / to /

Quoted 10 lines
tools/libxl/libxl_internal.h +10 -0
+ * You should call some suitable xc bind function on (or to obtain)
+ * the port, then libxl__ev_evtchn_wait.
+ *
+ * When the event is signaled then the callback will be made, once.
+ * Then you must call libxl__ev_evtchn_wait again, if desired.
+ *
+ * You must NOT call xc_evtchn_unmask.  wait will do that for you.
+ *
+ * Calling libxl__ev_evtchn_cancel will arrange for libxl to disregard
+ * future occurrences of event.  Both libxl__ev_evtchn_wait and

tools/libxl/libxl_internal.h static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw) · line 786

Quoted 10 lines
tools/libxl/libxl_internal.h +10 -0
+ * libxl__ev_evtchn_cancel are idempotent.
+ *
+ * (Note of course that an event channel becomes signaled when it is
+ * first bound, so you will get one call to libxl__ev_evtchn_wait
+ * "right away"; unless you have won a very fast race, the condition
+ * you were waiting for won't exist yet so when you check for it
+ * you'll find you need to call wait again.)
+ *
+ * You must not wait on the same port twice at once (that is, with
+ * two separate libxl__ev_evtchn's).

Doing so would require per wait xce_handles? Worth avoiding.

Quoted 15 lines
tools/libxl/libxl_internal.h +12 -0
+ */
+_hidden int libxl__ev_evtchn_wait(libxl__gc*, libxl__ev_evtchn *evev);
+_hidden void libxl__ev_evtchn_cancel(libxl__gc *gc, libxl__ev_evtchn *evev);
+
+static inline void libxl__ev_evtchn_init(libxl__ev_evtchn *evev)
+                { evev->waiting = 0; }
+static inline bool libxl__ev_evtchn_iswaiting(const libxl__ev_evtchn *evev)
+                { return evev->waiting; }
+
+_hidden int libxl__ctx_evtchn_init(libxl__gc *gc); /* for libxl_ctx_alloc */
+
+/*
  * For making subprocesses.  This is the only permitted mechanism for
  * code in libxl to do so.
  *
#20 lore raw

Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*

Ian Campbell writes ("Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*"):

Quoted 6 lines · Ian Jackson
Quoted 2 lines

This also involves providing a gc for the latter part of libxl_ctx_alloc.

Didn't you do that bit in an earlier patch? I guess you forgot to update this message when you refactored.

Yes.

Quoted 9 lines

tools/libxl/libxl_event.c void libxl__ev_xswatch_deregister(libxl__gc *gc, libxl__ev_xswatch *w) · line 642

Quoted 6 lines
tools/libxl/libxl_event.c +6 -0
+static void evtchn_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
+                               int fd, short events, short revents)
+{
+    EGC_GC;
+    libxl__ev_evtchn *evev;
+    int port, r, rc;

Should port be evtchn_port_or_error_t ? (from the use I don't think plain evtchn_port_t would be valid)

Fixed.

Quoted 7 lines

tools/libxl/libxl_internal.h struct libxl__ev_xswatch { · line 205

Quoted 5 lines

[...]

tools/libxl/libxl_internal.h +4 -0
+struct libxl__ev_evtchn {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ev_evtchn_callback *callback;
+    int port;

evtchn_port_t?

Leaving it as "int" means that the caller can put -1 in it if the struct isn't in use or the port not allocated. That avoids the caller needing to have a separate copy of the value, or a separate boolean. And later, we do.

Quoted 10 lines

tools/libxl/libxl_internal.h struct libxl__ctx { · line 357

Quoted 5 lines
tools/libxl/libxl_internal.h +1 -0
@@ -343,6 +354,10 @@ struct libxl__ctx {
     uint32_t watch_counter; /* helps disambiguate slot reuse */
     libxl__ev_fd watch_efd;
 
+    xc_evtchn *xce; /* for waiting use only libxl__ev_evtchn* */

The comment means "don't use directly, use libxl__ev-evtchn"?

Or does it imply that for uses other than waiting you may use it directly?

The latter. I'm open to suggestions for improved wording.

Quoted 3 lines

tools/libxl/libxl_internal.h struct libxl__ctx { · line 358

Quoted 1 line
tools/libxl/libxl_internal.h +1 -0
+    LIBXL_LIST_HEAD(, libxl__ev_evtchn) evtchns_waiting;

Are there any locking requirements relating to this list?

They are the same as all the other event-related data structures in the ctx: the ctx mutex protects them. This isn't particularly explicit here but I think it's fairly obvious. Only the event code looks at this list anyway and it all takes a gc.

Quoted 4 lines

tools/libxl/libxl_internal.h static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw) · line 766

Quoted 2 lines
tools/libxl/libxl_internal.h +1 -0
 /*
+ * The evtchn facility is one-shot per call t libxl__ev_evtchn_wait.

s/ t / to /

Fixed.

Quoted 4 lines

tools/libxl/libxl_internal.h static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw) · line 786

Quoted 2 lines
tools/libxl/libxl_internal.h +2 -0
+ * You must not wait on the same port twice at once (that is, with
+ * two separate libxl__ev_evtchn's).

Doing so would require per wait xce_handles? Worth avoiding.

It would be possible to improve the data structure and event code to support this. The obvious way would be to keep scanning the evtchns_waiting list after we found one waiter, but you'd have to do something fiddly to avoid reentering the same waiter again right away.

Thanks,
Ian.

#21 lore raw

Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*

Quoted 7 lines · Ian Jackson

Ian Campbell writes ("Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*"):

Quoted 6 lines
Quoted 5 lines

[...]

tools/libxl/libxl_internal.h +4 -0
+struct libxl__ev_evtchn {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ev_evtchn_callback *callback;
+    int port;
Quoted 6 lines
Quoted 1 line

evtchn_port_t?

Leaving it as "int" means that the caller can put -1 in it if the struct isn't in use or the port not allocated. That avoids the caller needing to have a separate copy of the value, or a separate boolean. And later, we do.

I suppose that's vaguely the same sort of thing as evtchn_port_or_error_t, but not quite.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
tools/libxl/libxl_internal.h +0 -0
@@ -343,6 +354,10 @@ struct libxl__ctx {
     uint32_t watch_counter; /* helps disambiguate slot reuse */
     libxl__ev_fd watch_efd;
 
Quoted 8 lines
Quoted 6 lines

tools/libxl/libxl_internal.h struct libxl__ctx { · line 357

Quoted 1 line
tools/libxl/libxl_internal.h +1 -0
+    xc_evtchn *xce; /* for waiting use only libxl__ev_evtchn* */

The comment means "don't use directly, use libxl__ev-evtchn"?

Or does it imply that for uses other than waiting you may use it directly?

The latter. I'm open to suggestions for improved wording.

"waiting must only be done via libxl__ev_..." not much better though.

Quoted 9 lines
Quoted 4 lines

tools/libxl/libxl_internal.h struct libxl__ctx { · line 358

Quoted 2 lines
tools/libxl/libxl_internal.h +1 -0
+    LIBXL_LIST_HEAD(, libxl__ev_evtchn) evtchns_waiting;

Are there any locking requirements relating to this list?

They are the same as all the other event-related data structures in the ctx: the ctx mutex protects them. This isn't particularly explicit here but I think it's fairly obvious. Only the event code looks at this list anyway and it all takes a gc.

OK.

Quoted 9 lines
Quoted 4 lines

tools/libxl/libxl_internal.h static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw) · line 786

Quoted 2 lines
tools/libxl/libxl_internal.h +2 -0
+ * You must not wait on the same port twice at once (that is, with
+ * two separate libxl__ev_evtchn's).

Doing so would require per wait xce_handles? Worth avoiding.

It would be possible to improve the data structure and event code to support this. The obvious way would be to keep scanning the evtchns_waiting list after we found one waiter, but you'd have to do something fiddly to avoid reentering the same waiter again right away.

Yes, easiest to avoid until the requirement occurs I think.

Ian.

#22 lore raw

Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*

Ian Campbell writes ("Re: [PATCH 10/23] libxl: events: Provide libxl__ev_evtchn*"):

Quoted 8 lines · Ian Jackson
Quoted 4 lines

Leaving it as "int" means that the caller can put -1 in it if the struct isn't in use or the port not allocated. That avoids the caller needing to have a separate copy of the value, or a separate boolean. And later, we do.

I suppose that's vaguely the same sort of thing as evtchn_port_or_error_t, but not quite.

Yes.

Quoted 3 lines
Quoted 1 line

The latter. I'm open to suggestions for improved wording.

"waiting must only be done via libxl__ev_..." not much better though.

I will change it to:

xc_evtchn *xce; /* waiting must be done only with libxl__ev_evtchn* */

Thanks,
Ian.

#23 lore raw

PATCH 11/23 libxc: suspend: Rename, improve xc_suspend_evtchn_init

In reply to Ian Jackson

xc_suspend_evtchn_init expects to eat the first event on the xce. If the xce is used for any other purpose then this can break. Document this fact and rename the function to xc_suspend_evtchn_init_exclusive. (I haven't checked the call sites for improper shared use of the xce.)

Provide a corresponding xc_suspend_evtchn_init_sane which doesn't try to eat an event, and instead leaves the caller the ability to demultiplex.

Also document that xc_await_suspend needs exclusive use of the xce.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
CC Shriram Rajagopalan <[email protected]>
---
 tools/libxc/xc_suspend.c                           |   21 ++++++++++++++++----
 tools/libxc/xenguest.h                             |   13 +++++++++++-
 tools/libxl/libxl_dom.c                            |    2 +-
 tools/misc/xen-hptool.c                            |    2 +-
 .../python/xen/lowlevel/checkpoint/libcheckpoint.c |    2 +-
 tools/xcutils/xc_save.c                            |    2 +-
 6 files changed, 33 insertions(+), 9 deletions(-)
tools/libxc/xc_suspend.c +17 -4
index 1ace411..7968a44 100644
@@ -14,6 +14,8 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
  */
 
+#include <assert.h>
+
 #include "xc_private.h"
 #include "xenguest.h"
 
@@ -102,7 +104,7 @@ int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int
     return unlock_suspend_event(xch, domid);
 }
 
-int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port)
 {
     int rc, suspend_evtchn = -1;
 
@@ -121,9 +123,6 @@ int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int por
         goto cleanup;
     }
 
-    /* event channel is pending immediately after binding */
-    xc_await_suspend(xch, xce, suspend_evtchn);
-
     return suspend_evtchn;
 
 cleanup:
@@ -132,3 +131,17 @@ cleanup:
 
     return -1;
 }
+
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+{
+    int suspend_evtchn;
+
+    suspend_evtchn = xc_suspend_evtchn_init_sane(xch, xce, domid, port);
+    if (suspend_evtchn < 0)
+        return suspend_evtchn;
+
+    /* event channel is pending immediately after binding */
+    xc_await_suspend(xch, xce, suspend_evtchn);
+
+    return suspend_evtchn;
+}
tools/libxc/xenguest.h +12 -1
index a0e30e1..ce5456c 100644
@@ -256,10 +256,21 @@ int xc_hvm_build_target_mem(xc_interface *xch,
 
 int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int suspend_evtchn);
 
-int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+/**
+ * This function eats the initial notification.
+ * xce must not be used for anything else
+ */
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port);
 
+/* xce must not be used for anything else */
 int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn);
 
+/**
+ * The port will be signaled immediately after this call
+ * The caller should check the domain status and look for the next event
+ */
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+
 int xc_get_bit_size(xc_interface *xch,
                     const char *image_name, const char *cmdline,
                     const char *features, int *type);
tools/libxl/libxl_dom.c +1 -1
index 55f74b2..4b42856 100644
@@ -1358,7 +1358,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
 
         if (port >= 0) {
             dss->suspend_eventchn =
-                xc_suspend_evtchn_init(CTX->xch, dss->xce, dss->domid, port);
+                xc_suspend_evtchn_init_exclusive(CTX->xch, dss->xce, dss->domid, port);
 
             if (dss->suspend_eventchn < 0)
                 LOG(WARN, "Suspend event channel initialization failed");
tools/misc/xen-hptool.c +1 -1
index 24c3e95..db76f79 100644
@@ -112,7 +112,7 @@ static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid, int *evtc
         fprintf(stderr, "DOM%d: No suspend port, try live migration\n", domid);
         goto failed;
     }
-    suspend_evtchn = xc_suspend_evtchn_init(xch, xce, domid, port);
+    suspend_evtchn = xc_suspend_evtchn_init_exclusive(xch, xce, domid, port);
     if (suspend_evtchn < 0)
     {
         fprintf(stderr, "Suspend evtchn initialization failed\n");
tools/python/xen/lowlevel/checkpoint/libcheckpoint.c +1 -1
index 01c0d47..817d272 100644
@@ -360,7 +360,7 @@ static int setup_suspend_evtchn(checkpoint_state* s)
     return -1;
   }
 
-  s->suspend_evtchn = xc_suspend_evtchn_init(s->xch, s->xce, s->domid, port);
+  s->suspend_evtchn = xc_suspend_evtchn_init_exclusive(s->xch, s->xce, s->domid, port);
   if (s->suspend_evtchn < 0) {
       s->errstr = "failed to bind suspend event channel";
       return -1;
tools/xcutils/xc_save.c +1 -1
index e34bd2c..974f706 100644
@@ -202,7 +202,7 @@ main(int argc, char **argv)
         else
         {
             si.suspend_evtchn =
-              xc_suspend_evtchn_init(si.xch, si.xce, si.domid, port);
+              xc_suspend_evtchn_init_exclusive(si.xch, si.xce, si.domid, port);
 
             if (si.suspend_evtchn < 0)
                 warnx("suspend event channel initialization failed, "
Signature · 1 line
1.7.10.4
Acked-by on v1 11/23
#24 lore raw

Re: [PATCH 11/23] libxc: suspend: Rename, improve xc_suspend_evtchn_init

Quoted 8 lines · Ian Jackson

xc_suspend_evtchn_init expects to eat the first event on the xce. If the xce is used for any other purpose then this can break. Document this fact and rename the function to xc_suspend_evtchn_init_exclusive. (I haven't checked the call sites for improper shared use of the xce.)

Provide a corresponding xc_suspend_evtchn_init_sane which doesn't try to eat an event, and instead leaves the caller the ability to demultiplex.

I'd much rather have fixed the existing one (and its callers as necessary) but I can see why you wouldn't have the appetite for that.

Quoted 145 lines

Also document that xc_await_suspend needs exclusive use of the xce.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
CC Shriram Rajagopalan <[email protected]>
---
 tools/libxc/xc_suspend.c                           |   21 ++++++++++++++++----
 tools/libxc/xenguest.h                             |   13 +++++++++++-
 tools/libxl/libxl_dom.c                            |    2 +-
 tools/misc/xen-hptool.c                            |    2 +-
 .../python/xen/lowlevel/checkpoint/libcheckpoint.c |    2 +-
 tools/xcutils/xc_save.c                            |    2 +-
 6 files changed, 33 insertions(+), 9 deletions(-)
tools/libxc/xc_suspend.c +17 -4
index 1ace411..7968a44 100644
@@ -14,6 +14,8 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
  */
 
+#include <assert.h>
+
 #include "xc_private.h"
 #include "xenguest.h"
 
@@ -102,7 +104,7 @@ int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int
     return unlock_suspend_event(xch, domid);
 }
 
-int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port)
 {
     int rc, suspend_evtchn = -1;
 
@@ -121,9 +123,6 @@ int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int por
         goto cleanup;
     }
 
-    /* event channel is pending immediately after binding */
-    xc_await_suspend(xch, xce, suspend_evtchn);
-
     return suspend_evtchn;
 
 cleanup:
@@ -132,3 +131,17 @@ cleanup:
 
     return -1;
 }
+
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+{
+    int suspend_evtchn;
+
+    suspend_evtchn = xc_suspend_evtchn_init_sane(xch, xce, domid, port);
+    if (suspend_evtchn < 0)
+        return suspend_evtchn;
+
+    /* event channel is pending immediately after binding */
+    xc_await_suspend(xch, xce, suspend_evtchn);
+
+    return suspend_evtchn;
+}
tools/libxc/xenguest.h +12 -1
index a0e30e1..ce5456c 100644
@@ -256,10 +256,21 @@ int xc_hvm_build_target_mem(xc_interface *xch,
 
 int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int suspend_evtchn);
 
-int xc_suspend_evtchn_init(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+/**
+ * This function eats the initial notification.
+ * xce must not be used for anything else
+ */
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port);
 
+/* xce must not be used for anything else */
 int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn);
 
+/**
+ * The port will be signaled immediately after this call
+ * The caller should check the domain status and look for the next event
+ */
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+
 int xc_get_bit_size(xc_interface *xch,
                     const char *image_name, const char *cmdline,
                     const char *features, int *type);
tools/libxl/libxl_dom.c +1 -1
index 55f74b2..4b42856 100644
@@ -1358,7 +1358,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
 
         if (port >= 0) {
             dss->suspend_eventchn =
-                xc_suspend_evtchn_init(CTX->xch, dss->xce, dss->domid, port);
+                xc_suspend_evtchn_init_exclusive(CTX->xch, dss->xce, dss->domid, port);
 
             if (dss->suspend_eventchn < 0)
                 LOG(WARN, "Suspend event channel initialization failed");
tools/misc/xen-hptool.c +1 -1
index 24c3e95..db76f79 100644
@@ -112,7 +112,7 @@ static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid, int *evtc
         fprintf(stderr, "DOM%d: No suspend port, try live migration\n", domid);
         goto failed;
     }
-    suspend_evtchn = xc_suspend_evtchn_init(xch, xce, domid, port);
+    suspend_evtchn = xc_suspend_evtchn_init_exclusive(xch, xce, domid, port);
     if (suspend_evtchn < 0)
     {
         fprintf(stderr, "Suspend evtchn initialization failed\n");
tools/python/xen/lowlevel/checkpoint/libcheckpoint.c +1 -1
index 01c0d47..817d272 100644
@@ -360,7 +360,7 @@ static int setup_suspend_evtchn(checkpoint_state* s)
     return -1;
   }
 
-  s->suspend_evtchn = xc_suspend_evtchn_init(s->xch, s->xce, s->domid, port);
+  s->suspend_evtchn = xc_suspend_evtchn_init_exclusive(s->xch, s->xce, s->domid, port);
   if (s->suspend_evtchn < 0) {
       s->errstr = "failed to bind suspend event channel";
       return -1;
tools/xcutils/xc_save.c +1 -1
index e34bd2c..974f706 100644
@@ -202,7 +202,7 @@ main(int argc, char **argv)
         else
         {
             si.suspend_evtchn =
-              xc_suspend_evtchn_init(si.xch, si.xce, si.domid, port);
+              xc_suspend_evtchn_init_exclusive(si.xch, si.xce, si.domid, port);
 
             if (si.suspend_evtchn < 0)
                 warnx("suspend event channel initialization failed, "
#25 lore raw

PATCH 12/23 libxc: suspend: Fix suspend event channel locking

In reply to Ian Jackson

Use fcntl F_SETLK, rather than writing our pid into a "lock" file. That way if we crash we don't leave the lockfile lying about. Callers now need to keep the fd for our lockfile. (We don't use flock because we don't want anyone who inherits this fd across fork to end up with a handle onto the lock.)

While we are here:
* Move the lockfile to /var/run/xen
* De-duplicate the calculation of the pathname
* Compute the buffer size for the pathname so that it will definitely not overrun (and use the computed value everywhere)
* Fix various error handling bugs

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
CC Shriram Rajagopalan <[email protected]>
---
 tools/libxc/xc_suspend.c                           |  159 +++++++++++++-------
 tools/libxc/xenguest.h                             |   16 +-
 tools/libxl/libxl_dom.c                            |    6 +-
 tools/libxl/libxl_internal.h                       |    1 +
 tools/misc/xen-hptool.c                            |   19 ++-
 tools/python/xen/lowlevel/checkpoint/checkpoint.h  |    2 +-
 .../python/xen/lowlevel/checkpoint/libcheckpoint.c |    7 +-
 tools/xcutils/xc_save.c                            |    8 +-
 8 files changed, 149 insertions(+), 69 deletions(-)
tools/libxc/xc_suspend.c +107 -52
index 7968a44..f0f70c1 100644
@@ -15,66 +15,115 @@
  */
 
 #include <assert.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "xc_private.h"
 #include "xenguest.h"
 
-#define SUSPEND_LOCK_FILE "/var/lib/xen/suspend_evtchn"
-static int lock_suspend_event(xc_interface *xch, int domid)
+#define SUSPEND_LOCK_FILE "/var/run/xen/suspend-evtchn-%d.lock"
+
+/*
+ * locking
+ */
+
+#define ERR(x) do{                                                      \
+    ERROR("Can't " #x " lock file for suspend event channel %s: %s\n",  \
+          suspend_file, strerror(errno));                               \
+    goto err;                                                           \
+}while(0)
+
+#define SUSPEND_FILE_BUFLEN (sizeof(SUSPEND_LOCK_FILE) + 10)
+
+static void get_suspend_file(char buf[SUSPEND_FILE_BUFLEN], int domid)
 {
-    int fd, rc;
-    mode_t mask;
-    char buf[128];
-    char suspend_file[256];
-
-    snprintf(suspend_file, sizeof(suspend_file), "%s_%d_lock.d",
-	    SUSPEND_LOCK_FILE, domid);
-    mask = umask(022);
-    fd = open(suspend_file, O_CREAT | O_EXCL | O_RDWR, 0666);
-    if (fd < 0)
-    {
-        ERROR("Can't create lock file for suspend event channel %s\n",
-		suspend_file);
-        return -EINVAL;
+    snprintf(buf, sizeof(buf), SUSPEND_LOCK_FILE, domid);
+}
+
+static int lock_suspend_event(xc_interface *xch, int domid, int *lockfd)
+{
+    int fd = -1, r;
+    char suspend_file[SUSPEND_FILE_BUFLEN];
+    struct stat ours, theirs;
+    struct flock fl;
+
+    get_suspend_file(suspend_file, domid);
+
+    *lockfd = -1;
+
+    for (;;) {
+        if (fd >= 0)
+            close (fd);
+
+        fd = open(suspend_file, O_CREAT | O_RDWR, 0600);
+        if (fd < 0)
+            ERR("create");
+
+        r = fcntl(fd, F_SETFD, FD_CLOEXEC);
+        if (r)
+            ERR("fcntl F_SETFD FD_CLOEXEC");
+
+        memset(&fl, 0, sizeof(fl));
+        fl.l_type = F_WRLCK;
+        fl.l_whence = SEEK_SET;
+        fl.l_len = 1;
+        r = fcntl(fd, F_SETLK, &fl);
+        if (r)
+            ERR("fcntl F_SETLK");
+
+        r = fstat(fd, &ours);
+        if (r)
+            ERR("fstat");
+
+        r = stat(suspend_file, &theirs);
+        if (r) {
+            if (errno == ENOENT)
+                /* try again */
+                continue;
+            ERR("stat");
+        }
+
+        if (ours.st_ino != theirs.st_ino)
+            /* someone else must have removed it while we were locking it */
+            continue;
+
+        break;
     }
-    umask(mask);
-    snprintf(buf, sizeof(buf), "%10ld", (long)getpid());
 
-    rc = write_exact(fd, buf, strlen(buf));
-    close(fd);
+    *lockfd = fd;
+    return 0;
 
-    return rc;
+ err:
+    if (fd >= 0)
+        close(fd);
+
+    return -1;
 }
 
-static int unlock_suspend_event(xc_interface *xch, int domid)
+static int unlock_suspend_event(xc_interface *xch, int domid, int *lockfd)
 {
-    int fd, pid, n;
-    char buf[128];
-    char suspend_file[256];
+    int r;
+    char suspend_file[SUSPEND_FILE_BUFLEN];
 
-    snprintf(suspend_file, sizeof(suspend_file), "%s_%d_lock.d",
-	    SUSPEND_LOCK_FILE, domid);
-    fd = open(suspend_file, O_RDWR);
+    if (*lockfd < 0)
+        return 0;
 
-    if (fd < 0)
-        return -EINVAL;
+    get_suspend_file(suspend_file, domid);
 
-    n = read(fd, buf, 127);
+    r = unlink(suspend_file);
+    if (r)
+        ERR("unlink");
 
-    close(fd);
+    r = close(*lockfd);
+    *lockfd = -1;
+    if (r)
+        ERR("close");
 
-    if (n > 0)
-    {
-        sscanf(buf, "%d", &pid);
-        /* We are the owner, so we can simply delete the file */
-        if (pid == getpid())
-        {
-            unlink(suspend_file);
-            return 0;
-        }
-    }
+ err:
+    if (*lockfd >= 0)
+        close(*lockfd);
 
-    return -EPERM;
+    return -1;
 }
 
 int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn)
@@ -96,20 +145,26 @@ int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn)
     return 0;
 }
 
-int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int suspend_evtchn)
+/* Internal callers are allowed to call this with suspend_evtchn<0
+ * but *lockfd>0. */
+int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce,
+                              int domid, int suspend_evtchn, int *lockfd)
 {
     if (suspend_evtchn >= 0)
         xc_evtchn_unbind(xce, suspend_evtchn);
 
-    return unlock_suspend_event(xch, domid);
+    return unlock_suspend_event(xch, domid, lockfd);
 }
 
-int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce,
+                                int domid, int port, int *lockfd)
 {
     int rc, suspend_evtchn = -1;
 
-    if (lock_suspend_event(xch, domid))
-        return -EINVAL;
+    if (lock_suspend_event(xch, domid, lockfd)) {
+        errno = EINVAL;
+        goto cleanup;
+    }
 
     suspend_evtchn = xc_evtchn_bind_interdomain(xce, domid, port);
     if (suspend_evtchn < 0) {
@@ -126,17 +181,17 @@ int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, in
     return suspend_evtchn;
 
 cleanup:
-    if (suspend_evtchn != -1)
-        xc_suspend_evtchn_release(xch, xce, domid, suspend_evtchn);
+    xc_suspend_evtchn_release(xch, xce, domid, suspend_evtchn, lockfd);
 
     return -1;
 }
 
-int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port)
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce,
+                                     int domid, int port, int *lockfd)
 {
     int suspend_evtchn;
 
-    suspend_evtchn = xc_suspend_evtchn_init_sane(xch, xce, domid, port);
+    suspend_evtchn = xc_suspend_evtchn_init_sane(xch, xce, domid, port, lockfd);
     if (suspend_evtchn < 0)
         return suspend_evtchn;
 
tools/libxc/xenguest.h +13 -3
index ce5456c..1f216cd 100644
@@ -254,13 +254,19 @@ int xc_hvm_build_target_mem(xc_interface *xch,
                             int target,
                             const char *image_name);
 
-int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int suspend_evtchn);
+/*
+ * Sets *lockfd to -1.
+ * Has deallocated everything even on error.
+ */
+int xc_suspend_evtchn_release(xc_interface *xch, xc_evtchn *xce, int domid, int suspend_evtchn, int *lockfd);
 
 /**
  * This function eats the initial notification.
  * xce must not be used for anything else
+ * See xc_suspend_evtchn_init_sane re lockfd.
  */
-int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+int xc_suspend_evtchn_init_exclusive(xc_interface *xch, xc_evtchn *xce,
+                                     int domid, int port, int *lockfd);
 
 /* xce must not be used for anything else */
 int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn);
@@ -268,8 +274,12 @@ int xc_await_suspend(xc_interface *xch, xc_evtchn *xce, int suspend_evtchn);
 /**
  * The port will be signaled immediately after this call
  * The caller should check the domain status and look for the next event
+ * On success, *lockfd will be set to >=0 and *lockfd must be preserved
+ * and fed to xc_suspend_evtchn_release.  (On error *lockfd is
+ * undefined and xc_suspend_evtchn_release is not allowed.)
  */
-int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce, int domid, int port);
+int xc_suspend_evtchn_init_sane(xc_interface *xch, xc_evtchn *xce,
+                                int domid, int port, int *lockfd);
 
 int xc_get_bit_size(xc_interface *xch,
                     const char *image_name, const char *cmdline,
tools/libxl/libxl_dom.c +4 -2
index 4b42856..48a4b8e 100644
@@ -1340,6 +1340,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
           | (dss->hvm ? XCFLAGS_HVM : 0);
 
     dss->suspend_eventchn = -1;
+    dss->guest_evtchn_lockfd = -1;
     dss->guest_responded = 0;
     dss->dm_savefile = libxl__device_model_savefile(gc, domid);
 
@@ -1358,7 +1359,8 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
 
         if (port >= 0) {
             dss->suspend_eventchn =
-                xc_suspend_evtchn_init_exclusive(CTX->xch, dss->xce, dss->domid, port);
+                xc_suspend_evtchn_init_exclusive(CTX->xch, dss->xce,
+                                  dss->domid, port, &dss->guest_evtchn_lockfd);
 
             if (dss->suspend_eventchn < 0)
                 LOG(WARN, "Suspend event channel initialization failed");
@@ -1522,7 +1524,7 @@ static void domain_suspend_done(libxl__egc *egc,
 
     if (dss->suspend_eventchn > 0)
         xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
-                                  dss->suspend_eventchn);
+                           dss->suspend_eventchn, &dss->guest_evtchn_lockfd);
     if (dss->xce != NULL)
         xc_evtchn_close(dss->xce);
 
tools/libxl/libxl_internal.h +1 -0
index c519abc..01873c0 100644
@@ -2405,6 +2405,7 @@ struct libxl__domain_suspend_state {
     /* private */
     xc_evtchn *xce; /* event channel handle */
     int suspend_eventchn;
+    int guest_evtchn_lockfd;
     int hvm;
     int xcflags;
     int guest_responded;
tools/misc/xen-hptool.c +13 -6
index db76f79..bd9d936 100644
@@ -99,10 +99,13 @@ static int hp_mem_query_func(int argc, char *argv[])
 
 extern int xs_suspend_evtchn_port(int domid);
 
-static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid, int *evtchn)
+static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid,
+                         int *evtchn, int *lockfd)
 {
     int port, rc, suspend_evtchn = -1;
 
+    *lockfd = -1;
+
     if (!evtchn)
         return -1;
 
@@ -112,7 +115,8 @@ static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid, int *evtc
         fprintf(stderr, "DOM%d: No suspend port, try live migration\n", domid);
         goto failed;
     }
-    suspend_evtchn = xc_suspend_evtchn_init_exclusive(xch, xce, domid, port);
+    suspend_evtchn = xc_suspend_evtchn_init_exclusive(xch, xce, domid,
+                                                      port, lockfd);
     if (suspend_evtchn < 0)
     {
         fprintf(stderr, "Suspend evtchn initialization failed\n");
@@ -135,7 +139,8 @@ static int suspend_guest(xc_interface *xch, xc_evtchn *xce, int domid, int *evtc
 
 failed:
     if (suspend_evtchn != -1)
-        xc_suspend_evtchn_release(xch, xce, domid, suspend_evtchn);
+        xc_suspend_evtchn_release(xch, xce, domid,
+                                  suspend_evtchn, lockfd);
 
     return -1;
 }
@@ -193,7 +198,7 @@ static int hp_mem_offline_func(int argc, char *argv[])
                 }
                 else if (status & PG_OFFLINE_OWNED)
                 {
-                    int result, suspend_evtchn = -1;
+                    int result, suspend_evtchn = -1, suspend_lockfd = -1;
                     xc_evtchn *xce;
                     xce = xc_evtchn_open(NULL, 0);
 
@@ -205,7 +210,8 @@ static int hp_mem_offline_func(int argc, char *argv[])
                     }
 
                     domid = status >> PG_OFFLINE_OWNER_SHIFT;
-                    if (suspend_guest(xch, xce, domid, &suspend_evtchn))
+                    if (suspend_guest(xch, xce, domid,
+                                      &suspend_evtchn, &suspend_lockfd))
                     {
                         fprintf(stderr, "Failed to suspend guest %d for"
                                 " mfn %lx\n", domid, mfn);
@@ -231,7 +237,8 @@ static int hp_mem_offline_func(int argc, char *argv[])
                                 mfn, domid);
                     }
                     xc_domain_resume(xch, domid, 1);
-                    xc_suspend_evtchn_release(xch, xce, domid, suspend_evtchn);
+                    xc_suspend_evtchn_release(xch, xce, domid,
+                                              suspend_evtchn, &suspend_lockfd);
                     xc_evtchn_close(xce);
                 }
                 break;
tools/python/xen/lowlevel/checkpoint/checkpoint.h +1 -1
index 187d9d7..2414956 100644
@@ -27,7 +27,7 @@ typedef struct {
     checkpoint_domtype domtype;
     int fd;
 
-    int suspend_evtchn;
+    int suspend_evtchn, suspend_lockfd;
 
     char* errstr;
 
tools/python/xen/lowlevel/checkpoint/libcheckpoint.c +5 -2
index 817d272..74ca062 100644
@@ -57,6 +57,7 @@ void checkpoint_init(checkpoint_state* s)
     s->fd = -1;
 
     s->suspend_evtchn = -1;
+    s->suspend_lockfd = -1;
 
     s->errstr = NULL;
 
@@ -360,7 +361,8 @@ static int setup_suspend_evtchn(checkpoint_state* s)
     return -1;
   }
 
-  s->suspend_evtchn = xc_suspend_evtchn_init_exclusive(s->xch, s->xce, s->domid, port);
+  s->suspend_evtchn = xc_suspend_evtchn_init_exclusive(s->xch, s->xce,
+                                    s->domid, port, &s->suspend_lockfd);
   if (s->suspend_evtchn < 0) {
       s->errstr = "failed to bind suspend event channel";
       return -1;
@@ -377,7 +379,8 @@ static void release_suspend_evtchn(checkpoint_state *s)
 {
   /* TODO: teach xen to clean up if port is unbound */
   if (s->xce != NULL && s->suspend_evtchn >= 0) {
-    xc_suspend_evtchn_release(s->xch, s->xce, s->domid, s->suspend_evtchn);
+    xc_suspend_evtchn_release(s->xch, s->xce, s->domid,
+                              s->suspend_evtchn, &s->suspend_lockfd);
     s->suspend_evtchn = -1;
   }
 }
tools/xcutils/xc_save.c +5 -3
index 974f706..bf74e46 100644
@@ -167,7 +167,7 @@ int
 main(int argc, char **argv)
 {
     unsigned int maxit, max_f, lflags;
-    int io_fd, ret, port;
+    int io_fd, ret, port, suspend_lockfd = -1;
     struct save_callbacks callbacks;
     xentoollog_level lvl;
     xentoollog_logger *l;
@@ -202,7 +202,8 @@ main(int argc, char **argv)
         else
         {
             si.suspend_evtchn =
-              xc_suspend_evtchn_init_exclusive(si.xch, si.xce, si.domid, port);
+              xc_suspend_evtchn_init_exclusive(si.xch, si.xce, si.domid,
+                                               port, &suspend_lockfd);
 
             if (si.suspend_evtchn < 0)
                 warnx("suspend event channel initialization failed, "
@@ -216,7 +217,8 @@ main(int argc, char **argv)
                          &callbacks, !!(si.flags & XCFLAGS_HVM), 0);
 
     if (si.suspend_evtchn > 0)
-	 xc_suspend_evtchn_release(si.xch, si.xce, si.domid, si.suspend_evtchn);
+	 xc_suspend_evtchn_release(si.xch, si.xce, si.domid,
+                                   si.suspend_evtchn, &suspend_lockfd);
 
     if (si.xce > 0)
         xc_evtchn_close(si.xce);
Signature · 1 line
1.7.10.4
#26 lore raw

PATCH 13/23 libxl: suspend: Async libxl__domain_suspend_callback

In reply to Ian Jackson

Mark the suspend callback libxl__domain_suspend_callback as asynchronous in the helper stub generator (libxl_save_msgs_gen.pl). We are going to want to provide an asynchronous version of this function to get rid of the usleeps and waiting loops in the suspend code.

libxl__domain_suspend_common_callback, the common synchronous core, which used to be provided directly as the callback function for the helper machinery, becomes libxl__domain_suspend_callback_common. It can now take a typesafe parameter.

For now, provide two very similar asynchronous wrappers for it. Each is a simple function which contains only boilerplate, calls the common synchronous core, and returns the asynchronous response.

Essentially, we have just moved (in the case of suspend callbacks) the call site of libxl__srm_callout_callback_complete. It was in the switch statement in the autogenerated _libxl_save_msgs_callout.c, and is now in the handwritten libxl_dom.c.

No functional change.

Cc Shriram Rajagopalan <[email protected]>
Cc Ian Campbell <[email protected]>

---
v2: Commit message mentions usleeps, not Remus, as motivation.
CC: Stefano Stabellini <[email protected]>

---
 tools/libxl/libxl_dom.c            |   25 +++++++++++++++++++------
 tools/libxl/libxl_internal.h       |    2 +-
 tools/libxl/libxl_save_msgs_gen.pl |    2 +-
 3 files changed, 21 insertions(+), 8 deletions(-)
tools/libxl/libxl_dom.c +19 -6
index 48a4b8e..1c0f04f 100644
@@ -1020,10 +1020,8 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)
     return 0;
 }
 
-int libxl__domain_suspend_common_callback(void *user)
+int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
 {
-    libxl__save_helper_state *shs = user;
-    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
     int ret;
@@ -1242,12 +1240,27 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
     return 0;
 }
 
+static void libxl__domain_suspend_callback(void *data)
+{
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
+}
+
 /*----- remus callbacks -----*/
 
-static int libxl__remus_domain_suspend_callback(void *data)
+static void libxl__remus_domain_suspend_callback(void *data)
 {
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
     /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
 }
 
 static int libxl__remus_domain_resume_callback(void *data)
@@ -1373,7 +1386,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
     } else
-        callbacks->suspend = libxl__domain_suspend_common_callback;
+        callbacks->suspend = libxl__domain_suspend_callback;
 
     callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
     dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
tools/libxl/libxl_internal.h +1 -1
index 01873c0..9b9264e 100644
@@ -2705,7 +2705,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
 void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
                            libxl__save_helper_state *shs, int return_value);
 
-_hidden int libxl__domain_suspend_common_callback(void *data);
+_hidden int libxl__domain_suspend_callback_common(libxl__domain_suspend_state*);
 _hidden void libxl__domain_suspend_common_switch_qemu_logdirty
                                (int domid, unsigned int enable, void *data);
 _hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
tools/libxl/libxl_save_msgs_gen.pl +1 -1
index ee126c7..3c6bd57 100755
@@ -23,7 +23,7 @@ our @msgs = (
                                                  STRING doing_what),
                                                 'unsigned long', 'done',
                                                 'unsigned long', 'total'] ],
-    [  3, 'scxW',   "suspend", [] ],         
+    [  3, 'scxA',   "suspend", [] ],         
     [  4, 'scxW',   "postcopy", [] ],        
     [  5, 'scxA',   "checkpoint", [] ],      
     [  6, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
Signature · 1 line
1.7.10.4
#27 lore raw

PATCH 14/23 libxl: suspend: Async domain_suspend_callback_common

In reply to Ian Jackson

Make domain_suspend_callback_common do its work and then call dss->callback_common_done, rather than simply returning its answer.

This is preparatory to abolishing the usleeps in this function and replacing them with use of the event machinery.

No functional change.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c      |   46 ++++++++++++++++++++++++++++++------------
 tools/libxl/libxl_internal.h |    4 +++-
 2 files changed, 36 insertions(+), 14 deletions(-)
tools/libxl/libxl_dom.c +33 -13
index 1c0f04f..39f06da 100644
@@ -755,6 +755,10 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
 
 static void domain_suspend_done(libxl__egc *egc,
                         libxl__domain_suspend_state *dss, int rc);
+static void domain_suspend_callback_common_done(libxl__egc *egc,
+                                libxl__domain_suspend_state *dss, int ok);
+static void remus_domain_suspend_callback_common_done(libxl__egc *egc,
+                                libxl__domain_suspend_state *dss, int ok);
 
 /*----- complicated callback, called by xc_domain_save -----*/
 
@@ -1020,7 +1024,9 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)
     return 0;
 }
 
-int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
+/* calls dss->callback_common_done when done */
+static void domain_suspend_callback_common(libxl__egc *egc,
+                                           libxl__domain_suspend_state *dss)
 {
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
@@ -1043,12 +1049,12 @@ int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
         ret = xc_evtchn_notify(dss->xce, dss->suspend_eventchn);
         if (ret < 0) {
             LOG(ERROR, "xc_evtchn_notify failed ret=%d", ret);
-            return 0;
+            goto err;
         }
         ret = xc_await_suspend(CTX->xch, dss->xce, dss->suspend_eventchn);
         if (ret < 0) {
             LOG(ERROR, "xc_await_suspend failed ret=%d", ret);
-            return 0;
+            goto err;
         }
         dss->guest_responded = 1;
         goto guest_suspended;
@@ -1059,7 +1065,7 @@ int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
         ret = xc_domain_shutdown(CTX->xch, domid, SHUTDOWN_suspend);
         if (ret < 0) {
             LOGE(ERROR, "xc_domain_shutdown failed");
-            return 0;
+            goto err;
         }
         /* The guest does not (need to) respond to this sort of request. */
         dss->guest_responded = 1;
@@ -1113,7 +1119,7 @@ int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
          */
         if (!strcmp(state, "suspend")) {
             LOG(ERROR, "guest didn't acknowledge suspend, request cancelled");
-            return 0;
+            goto err;
         }
 
         LOG(DEBUG, "guest acknowledged suspend request");
@@ -1143,17 +1149,19 @@ int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
     }
 
     LOG(ERROR, "guest did not suspend");
-    return 0;
+ err:
+    dss->callback_common_done(egc, dss, 0);
+    return;
 
  guest_suspended:
     if (dss->hvm) {
         ret = libxl__domain_suspend_device_model(gc, dss);
         if (ret) {
             LOG(ERROR, "libxl__domain_suspend_device_model failed ret=%d", ret);
-            return 0;
+            goto err;
         }
     }
-    return 1;
+    dss->callback_common_done(egc, dss, 1);
 }
 
 static inline char *physmap_path(libxl__gc *gc, uint32_t domid,
@@ -1246,10 +1254,16 @@ static void libxl__domain_suspend_callback(void *data)
     libxl__egc *egc = shs->egc;
     libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
 
-    int ok = libxl__domain_suspend_callback_common(dss);
-    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
+    dss->callback_common_done = domain_suspend_callback_common_done;
+    domain_suspend_callback_common(egc, dss);
 }
 
+static void domain_suspend_callback_common_done(libxl__egc *egc,
+                                libxl__domain_suspend_state *dss, int ok)
+{
+    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, ok);
+}    
+
 /*----- remus callbacks -----*/
 
 static void libxl__remus_domain_suspend_callback(void *data)
@@ -1258,11 +1272,17 @@ static void libxl__remus_domain_suspend_callback(void *data)
     libxl__egc *egc = shs->egc;
     libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
 
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    int ok = libxl__domain_suspend_callback_common(dss);
-    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
+    dss->callback_common_done = remus_domain_suspend_callback_common_done;
+    domain_suspend_callback_common(egc, dss);
 }
 
+static void remus_domain_suspend_callback_common_done(libxl__egc *egc,
+                                libxl__domain_suspend_state *dss, int ok)
+{
+    /* REMUS TODO: Issue disk and network checkpoint reqs. */
+    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, ok);
+}    
+
 static int libxl__remus_domain_resume_callback(void *data)
 {
     libxl__save_helper_state *shs = data;
tools/libxl/libxl_internal.h +3 -1
index 9b9264e..55293f1 100644
@@ -2413,6 +2413,8 @@ struct libxl__domain_suspend_state {
     int interval; /* checkpoint interval (for Remus) */
     libxl__save_helper_state shs;
     libxl__logdirty_switch logdirty;
+    void (*callback_common_done)(libxl__egc*,
+                                 struct libxl__domain_suspend_state*, int ok);
     /* private for libxl__domain_save_device_model */
     libxl__save_device_model_cb *save_dm_callback;
     libxl__datacopier_state save_dm_datacopier;
@@ -2705,7 +2707,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
 void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
                            libxl__save_helper_state *shs, int return_value);
 
-_hidden int libxl__domain_suspend_callback_common(libxl__domain_suspend_state*);
+
 _hidden void libxl__domain_suspend_common_switch_qemu_logdirty
                                (int domid, unsigned int enable, void *data);
 _hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
Signature · 1 line
1.7.10.4
#28 lore raw

PATCH 15/23 libxl: suspend: Reorg domain_suspend_callback_common

In reply to Ian Jackson

Make domain_suspend_callback_common more callback-oriented:

* Turn the functionality behind the goto labels "err" and "guest_suspended" into functions which can be called just before "return".

* Deindent the "issuing %s suspend request via XenBus control node" branch; it is going to be split up into various functions as the xenstore work becomes callback-based.

No functional change.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c |  158 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 103 insertions(+), 55 deletions(-)
tools/libxl/libxl_dom.c +103 -55
index 39f06da..59f6ce3 100644
@@ -1024,6 +1024,16 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)
     return 0;
 }
 
+static void domain_suspend_common_wait_guest(libxl__egc *egc,
+                                             libxl__domain_suspend_state *dss);
+static void domain_suspend_common_guest_suspended(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss);
+static void domain_suspend_common_failed(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss);
+static void domain_suspend_common_done(libxl__egc *egc,
+                                       libxl__domain_suspend_state *dss,
+                                       bool ok);
+
 /* calls dss->callback_common_done when done */
 static void domain_suspend_callback_common(libxl__egc *egc,
                                            libxl__domain_suspend_state *dss)
@@ -1057,7 +1067,8 @@ static void domain_suspend_callback_common(libxl__egc *egc,
             goto err;
         }
         dss->guest_responded = 1;
-        goto guest_suspended;
+        domain_suspend_common_guest_suspended(egc, dss);
+        return;
     }
 
     if (dss->hvm && (!hvm_pvdrv || hvm_s_state)) {
@@ -1069,63 +1080,81 @@ static void domain_suspend_callback_common(libxl__egc *egc,
         }
         /* The guest does not (need to) respond to this sort of request. */
         dss->guest_responded = 1;
-    } else {
-        LOG(DEBUG, "issuing %s suspend request via XenBus control node",
-            dss->hvm ? "PVHVM" : "PV");
+        domain_suspend_common_wait_guest(egc, dss);
+        return;
+    }
 
-        libxl__domain_pvcontrol_write(gc, XBT_NULL, domid, "suspend");
+    LOG(DEBUG, "issuing %s suspend request via XenBus control node",
+        dss->hvm ? "PVHVM" : "PV");
 
-        LOG(DEBUG, "wait for the guest to acknowledge suspend request");
-        watchdog = 60;
-        while (!strcmp(state, "suspend") && watchdog > 0) {
-            usleep(100000);
+    libxl__domain_pvcontrol_write(gc, XBT_NULL, domid, "suspend");
 
-            state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
-            if (!state) state = "";
+    LOG(DEBUG, "wait for the guest to acknowledge suspend request");
+    watchdog = 60;
+    while (!strcmp(state, "suspend") && watchdog > 0) {
+        usleep(100000);
 
-            watchdog--;
-        }
+        state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
+        if (!state) state = "";
 
-        /*
-         * Guest appears to not be responding. Cancel the suspend
-         * request.
-         *
-         * We re-read the suspend node and clear it within a
-         * transaction in order to handle the case where we race
-         * against the guest catching up and acknowledging the request
-         * at the last minute.
-         */
-        if (!strcmp(state, "suspend")) {
-            LOG(ERROR, "guest didn't acknowledge suspend, cancelling request");
-        retry_transaction:
-            t = xs_transaction_start(CTX->xsh);
-
-            state = libxl__domain_pvcontrol_read(gc, t, domid);
-            if (!state) state = "";
-
-            if (!strcmp(state, "suspend"))
-                libxl__domain_pvcontrol_write(gc, t, domid, "");
-
-            if (!xs_transaction_end(CTX->xsh, t, 0))
-                if (errno == EAGAIN)
-                    goto retry_transaction;
+        watchdog--;
+    }
 
-        }
+    /*
+     * Guest appears to not be responding. Cancel the suspend
+     * request.
+     *
+     * We re-read the suspend node and clear it within a
+     * transaction in order to handle the case where we race
+     * against the guest catching up and acknowledging the request
+     * at the last minute.
+     */
+    if (!strcmp(state, "suspend")) {
+        LOG(ERROR, "guest didn't acknowledge suspend, cancelling request");
+    retry_transaction:
+        t = xs_transaction_start(CTX->xsh);
 
-        /*
-         * Final check for guest acknowledgement. The guest may have
-         * acknowledged while we were cancelling the request in which
-         * case we lost the race while cancelling and should continue.
-         */
-        if (!strcmp(state, "suspend")) {
-            LOG(ERROR, "guest didn't acknowledge suspend, request cancelled");
-            goto err;
-        }
+        state = libxl__domain_pvcontrol_read(gc, t, domid);
+        if (!state) state = "";
+
+        if (!strcmp(state, "suspend"))
+            libxl__domain_pvcontrol_write(gc, t, domid, "");
+
+        if (!xs_transaction_end(CTX->xsh, t, 0))
+            if (errno == EAGAIN)
+                goto retry_transaction;
 
-        LOG(DEBUG, "guest acknowledged suspend request");
-        dss->guest_responded = 1;
     }
 
+    /*
+     * Final check for guest acknowledgement. The guest may have
+     * acknowledged while we were cancelling the request in which
+     * case we lost the race while cancelling and should continue.
+     */
+    if (!strcmp(state, "suspend")) {
+        LOG(ERROR, "guest didn't acknowledge suspend, request cancelled");
+        goto err;
+    }
+
+    LOG(DEBUG, "guest acknowledged suspend request");
+    dss->guest_responded = 1;
+    domain_suspend_common_wait_guest(egc,dss);
+    return;
+
+ err:
+    domain_suspend_common_failed(egc, dss);
+}
+
+static void domain_suspend_common_wait_guest(libxl__egc *egc,
+                                             libxl__domain_suspend_state *dss)
+{
+    STATE_AO_GC(dss->ao);
+    int ret;
+    int watchdog;
+
+    /* Convenience aliases */
+    const uint32_t domid = dss->domid;
+
     LOG(DEBUG, "wait for the guest to suspend");
     watchdog = 60;
     while (watchdog > 0) {
@@ -1141,7 +1170,8 @@ static void domain_suspend_callback_common(libxl__egc *egc,
                 & XEN_DOMINF_shutdownmask;
             if (shutdown_reason == SHUTDOWN_suspend) {
                 LOG(DEBUG, "guest has suspended");
-                goto guest_suspended;
+                domain_suspend_common_guest_suspended(egc, dss);
+                return;
             }
         }
 
@@ -1149,19 +1179,37 @@ static void domain_suspend_callback_common(libxl__egc *egc,
     }
 
     LOG(ERROR, "guest did not suspend");
- err:
-    dss->callback_common_done(egc, dss, 0);
-    return;
+    domain_suspend_common_failed(egc, dss);
+}
+
+static void domain_suspend_common_guest_suspended(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss)
+{
+    STATE_AO_GC(dss->ao);
+    int ret;
 
- guest_suspended:
     if (dss->hvm) {
         ret = libxl__domain_suspend_device_model(gc, dss);
         if (ret) {
             LOG(ERROR, "libxl__domain_suspend_device_model failed ret=%d", ret);
-            goto err;
+            domain_suspend_common_failed(egc, dss);
+            return;
         }
     }
-    dss->callback_common_done(egc, dss, 1);
+    domain_suspend_common_done(egc, dss, 1);
+}
+
+static void domain_suspend_common_failed(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss)
+{
+    domain_suspend_common_done(egc, dss, 0);
+}
+
+static void domain_suspend_common_done(libxl__egc *egc,
+                                       libxl__domain_suspend_state *dss,
+                                       bool ok)
+{
+    dss->callback_common_done(egc, dss, ok);
 }
 
 static inline char *physmap_path(libxl__gc *gc, uint32_t domid,
Signature · 1 line
1.7.10.4
#29 lore raw

PATCH 16/23 libxl: suspend: New libxl__domain_pvcontrol_xspath

In reply to Ian Jackson

Factor out the pv control node xenstore path calculation into libxl__domain_pvcontrol_xspath.

This xs path calculation was open coded in libxl__domain_pvcontrol_read and _write. This is undesirable because it duplicates the code and because it makes the path inaccessible to other parts of libxl (which are soon going to want it).

No functional change.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl.c          |   21 +++++++++++----------
 tools/libxl/libxl_internal.h |    1 +
 2 files changed, 12 insertions(+), 10 deletions(-)
tools/libxl/libxl.c +11 -10
index 51158aa..8d70a45 100644
@@ -898,17 +898,23 @@ int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid)
     return !!pvdriver;
 }
 
-char * libxl__domain_pvcontrol_read(libxl__gc *gc, xs_transaction_t t,
-                                    uint32_t domid)
+const char *libxl__domain_pvcontrol_xspath(libxl__gc *gc, uint32_t domid)
 {
-    const char *shutdown_path;
     const char *dom_path;
 
     dom_path = libxl__xs_get_dompath(gc, domid);
     if (!dom_path)
         return NULL;
 
-    shutdown_path = libxl__sprintf(gc, "%s/control/shutdown", dom_path);
+    return GCSPRINTF("%s/control/shutdown", dom_path);
+}
+
+char * libxl__domain_pvcontrol_read(libxl__gc *gc, xs_transaction_t t,
+                                    uint32_t domid)
+{
+    const char *shutdown_path;
+
+    shutdown_path = libxl__domain_pvcontrol_xspath(gc, domid);
     if (!shutdown_path)
         return NULL;
 
@@ -919,13 +925,8 @@ int libxl__domain_pvcontrol_write(libxl__gc *gc, xs_transaction_t t,
                                   uint32_t domid, const char *cmd)
 {
     const char *shutdown_path;
-    const char *dom_path;
-
-    dom_path = libxl__xs_get_dompath(gc, domid);
-    if (!dom_path)
-        return ERROR_FAIL;
 
-    shutdown_path = libxl__sprintf(gc, "%s/control/shutdown", dom_path);
+    shutdown_path = libxl__domain_pvcontrol_xspath(gc, domid);
     if (!shutdown_path)
         return ERROR_FAIL;
 
tools/libxl/libxl_internal.h +1 -0
index 55293f1..93e0d4e 100644
@@ -967,6 +967,7 @@ _hidden int libxl__domain_resume(libxl__gc *gc, uint32_t domid,
 /* returns 0 or 1, or a libxl error code */
 _hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t domid);
 
+_hidden const char *libxl__domain_pvcontrol_xspath(libxl__gc*, uint32_t domid);
 _hidden char * libxl__domain_pvcontrol_read(libxl__gc *gc,
                                             xs_transaction_t t, uint32_t domid);
 _hidden int libxl__domain_pvcontrol_write(libxl__gc *gc, xs_transaction_t t,
Signature · 1 line
1.7.10.4
#30 lore raw

PATCH 17/23 libxl: suspend: New domain_suspend_pvcontrol_acked

In reply to Ian Jackson

Factor out domain_suspend_pvcontrol_acked.

This replaces a bunch of open-coded strcmp()s and makes the code clearer. It also eliminates the need to check for state==NULL each time it's read, because we can check for NULL once before the strcmp.

No functional change.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c |   17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
tools/libxl/libxl_dom.c +10 -7
index 59f6ce3..a0b3a57 100644
@@ -1034,6 +1034,12 @@ static void domain_suspend_common_done(libxl__egc *egc,
                                        libxl__domain_suspend_state *dss,
                                        bool ok);
 
+static bool domain_suspend_pvcontrol_acked(const char *state) {
+    /* any value other than "suspend", including ENOENT, is OK */
+    if (!state) return 1;
+    return strcmp(state,"suspend");
+}
+
 /* calls dss->callback_common_done when done */
 static void domain_suspend_callback_common(libxl__egc *egc,
                                            libxl__domain_suspend_state *dss)
@@ -1091,11 +1097,10 @@ static void domain_suspend_callback_common(libxl__egc *egc,
 
     LOG(DEBUG, "wait for the guest to acknowledge suspend request");
     watchdog = 60;
-    while (!strcmp(state, "suspend") && watchdog > 0) {
+    while (!domain_suspend_pvcontrol_acked(state) && watchdog > 0) {
         usleep(100000);
 
         state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
-        if (!state) state = "";
 
         watchdog--;
     }
@@ -1109,21 +1114,19 @@ static void domain_suspend_callback_common(libxl__egc *egc,
      * against the guest catching up and acknowledging the request
      * at the last minute.
      */
-    if (!strcmp(state, "suspend")) {
+    if (!domain_suspend_pvcontrol_acked(state)) {
         LOG(ERROR, "guest didn't acknowledge suspend, cancelling request");
     retry_transaction:
         t = xs_transaction_start(CTX->xsh);
 
         state = libxl__domain_pvcontrol_read(gc, t, domid);
-        if (!state) state = "";
 
-        if (!strcmp(state, "suspend"))
+        if (!domain_suspend_pvcontrol_acked(state))
             libxl__domain_pvcontrol_write(gc, t, domid, "");
 
         if (!xs_transaction_end(CTX->xsh, t, 0))
             if (errno == EAGAIN)
                 goto retry_transaction;
-
     }
 
     /*
@@ -1131,7 +1134,7 @@ static void domain_suspend_callback_common(libxl__egc *egc,
      * acknowledged while we were cancelling the request in which
      * case we lost the race while cancelling and should continue.
      */
-    if (!strcmp(state, "suspend")) {
+    if (!domain_suspend_pvcontrol_acked(state)) {
         LOG(ERROR, "guest didn't acknowledge suspend, request cancelled");
         goto err;
     }
Signature · 1 line
1.7.10.4
#31 lore raw

PATCH 18/23 libxl: suspend: domain_suspend_callback_common xs errs

In reply to Ian Jackson

In domain_suspend_callback_common, use libxl__xs_transaction_start in a loop, rather than xs_transaction_start and a goto label.

This will improve the error handling, but have no other semantic effect.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c |   19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
tools/libxl/libxl_dom.c +11 -8
index a0b3a57..8aceba9 100644
@@ -1050,6 +1050,7 @@ static void domain_suspend_callback_common(libxl__egc *egc,
     char *state = "suspend";
     int watchdog;
     xs_transaction_t t;
+    int rc;
 
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
@@ -1116,17 +1117,19 @@ static void domain_suspend_callback_common(libxl__egc *egc,
      */
     if (!domain_suspend_pvcontrol_acked(state)) {
         LOG(ERROR, "guest didn't acknowledge suspend, cancelling request");
-    retry_transaction:
-        t = xs_transaction_start(CTX->xsh);
+        for (;;) {
+            rc = libxl__xs_transaction_start(gc, &t);
+            if (rc) goto err;
 
-        state = libxl__domain_pvcontrol_read(gc, t, domid);
+            state = libxl__domain_pvcontrol_read(gc, t, domid);
 
-        if (!domain_suspend_pvcontrol_acked(state))
-            libxl__domain_pvcontrol_write(gc, t, domid, "");
+            if (!domain_suspend_pvcontrol_acked(state))
+                libxl__domain_pvcontrol_write(gc, t, domid, "");
 
-        if (!xs_transaction_end(CTX->xsh, t, 0))
-            if (errno == EAGAIN)
-                goto retry_transaction;
+            rc = libxl__xs_transaction_commit(gc, &t);
+            if (!rc) break;
+            if (rc<0) goto err;
+        }
     }
 
     /*
Signature · 1 line
1.7.10.4
#32 lore raw

PATCH 19/23 libxl: suspend: Async xenstore pvcontrol wait

In reply to Ian Jackson

When negotiating guest suspend via the xenstore pvcontrol protocol (ie when the guest does NOT support the evtchn fast suspend protocol):

Replace the use of loops and usleep with a call to libxl__xswait.

Also, replace the xenstore transaction loop with one using libxl__xs_transaction_start et al.

There is not intended to be any semantic change, other than to make the algorithm properly asynchronous.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c      |   93 ++++++++++++++++++++++++++----------------
 tools/libxl/libxl_internal.h |    1 +
 2 files changed, 59 insertions(+), 35 deletions(-)
tools/libxl/libxl_dom.c +58 -35
index 8aceba9..dde7e33 100644
@@ -1028,6 +1028,8 @@ static void domain_suspend_common_wait_guest(libxl__egc *egc,
                                              libxl__domain_suspend_state *dss);
 static void domain_suspend_common_guest_suspended(libxl__egc *egc,
                                          libxl__domain_suspend_state *dss);
+static void domain_suspend_common_pvcontrol_suspending(libxl__egc *egc,
+      libxl__xswait_state *xswa, int rc, const char *state);
 static void domain_suspend_common_failed(libxl__egc *egc,
                                          libxl__domain_suspend_state *dss);
 static void domain_suspend_common_done(libxl__egc *egc,
@@ -1047,10 +1049,6 @@ static void domain_suspend_callback_common(libxl__egc *egc,
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
     int ret;
-    char *state = "suspend";
-    int watchdog;
-    xs_transaction_t t;
-    int rc;
 
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
@@ -1096,59 +1094,81 @@ static void domain_suspend_callback_common(libxl__egc *egc,
 
     libxl__domain_pvcontrol_write(gc, XBT_NULL, domid, "suspend");
 
-    LOG(DEBUG, "wait for the guest to acknowledge suspend request");
-    watchdog = 60;
-    while (!domain_suspend_pvcontrol_acked(state) && watchdog > 0) {
-        usleep(100000);
+    dss->pvcontrol.path = libxl__domain_pvcontrol_xspath(gc, domid);
+    if (!dss->pvcontrol.path) goto err;
 
-        state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
+    dss->pvcontrol.ao = ao;
+    dss->pvcontrol.what = "guest acknowledgement of suspend request";
+    dss->pvcontrol.timeout_ms = 60 * 1000;
+    dss->pvcontrol.callback = domain_suspend_common_pvcontrol_suspending;
+    libxl__xswait_start(gc, &dss->pvcontrol);
+    return;
 
-        watchdog--;
-    }
+ err:
+    domain_suspend_common_failed(egc, dss);
+}
 
-    /*
-     * Guest appears to not be responding. Cancel the suspend
-     * request.
-     *
-     * We re-read the suspend node and clear it within a
-     * transaction in order to handle the case where we race
-     * against the guest catching up and acknowledging the request
-     * at the last minute.
-     */
-    if (!domain_suspend_pvcontrol_acked(state)) {
-        LOG(ERROR, "guest didn't acknowledge suspend, cancelling request");
+static void domain_suspend_common_pvcontrol_suspending(libxl__egc *egc,
+      libxl__xswait_state *xswa, int rc, const char *state)
+{
+    libxl__domain_suspend_state *dss = CONTAINER_OF(xswa, *dss, pvcontrol);
+    STATE_AO_GC(dss->ao);
+    xs_transaction_t t = 0;
+
+    if (!rc && !domain_suspend_pvcontrol_acked(state))
+        /* keep waiting */
+        return;
+
+    libxl__xswait_stop(gc, &dss->pvcontrol);
+
+    if (rc == ERROR_TIMEDOUT) {
+        /*
+         * Guest appears to not be responding. Cancel the suspend
+         * request.
+         *
+         * We re-read the suspend node and clear it within a
+         * transaction in order to handle the case where we race
+         * against the guest catching up and acknowledging the request
+         * at the last minute.
+         */
         for (;;) {
             rc = libxl__xs_transaction_start(gc, &t);
             if (rc) goto err;
 
-            state = libxl__domain_pvcontrol_read(gc, t, domid);
+            rc = libxl__xs_read_checked(gc, t, xswa->path, &state);
+            if (rc) goto err;
+
+            if (domain_suspend_pvcontrol_acked(state))
+                break;
 
-            if (!domain_suspend_pvcontrol_acked(state))
-                libxl__domain_pvcontrol_write(gc, t, domid, "");
+            rc = libxl__xs_write_checked(gc, t, xswa->path, "");
+            if (rc) goto err;
 
             rc = libxl__xs_transaction_commit(gc, &t);
-            if (!rc) break;
+            if (!rc) {
+                LOG(ERROR,
+                    "guest didn't acknowledge suspend, cancelling request");
+                goto err;
+            }
             if (rc<0) goto err;
         }
-    }
-
-    /*
-     * Final check for guest acknowledgement. The guest may have
-     * acknowledged while we were cancelling the request in which
-     * case we lost the race while cancelling and should continue.
-     */
-    if (!domain_suspend_pvcontrol_acked(state)) {
-        LOG(ERROR, "guest didn't acknowledge suspend, request cancelled");
+    } else if (rc) {
+        /* some error in xswait's read of xenstore, already logged */
         goto err;
     }
 
+    assert(domain_suspend_pvcontrol_acked(state));
     LOG(DEBUG, "guest acknowledged suspend request");
+
+    libxl__xs_transaction_abort(gc, &t);
     dss->guest_responded = 1;
     domain_suspend_common_wait_guest(egc,dss);
     return;
 
  err:
+    libxl__xs_transaction_abort(gc, &t);
     domain_suspend_common_failed(egc, dss);
+    return;
 }
 
 static void domain_suspend_common_wait_guest(libxl__egc *egc,
@@ -1215,6 +1235,8 @@ static void domain_suspend_common_done(libxl__egc *egc,
                                        libxl__domain_suspend_state *dss,
                                        bool ok)
 {
+    EGC_GC;
+    assert(!libxl__xswait_inuse(&dss->pvcontrol));
     dss->callback_common_done(egc, dss, ok);
 }
 
@@ -1400,6 +1422,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
         &dss->shs.callbacks.save.a;
 
     logdirty_init(&dss->logdirty);
+    libxl__xswait_init(&dss->pvcontrol);
 
     switch (type) {
     case LIBXL_DOMAIN_TYPE_HVM: {
tools/libxl/libxl_internal.h +1 -0
index 93e0d4e..c4a509b 100644
@@ -2410,6 +2410,7 @@ struct libxl__domain_suspend_state {
     int hvm;
     int xcflags;
     int guest_responded;
+    libxl__xswait_state pvcontrol;
     const char *dm_savefile;
     int interval; /* checkpoint interval (for Remus) */
     libxl__save_helper_state shs;
Signature · 1 line
1.7.10.4
#33 lore raw

PATCH 20/23 libxl: suspend: Abolish usleeps in domain suspend wait

In reply to Ian Jackson

Replace the use of a loop with usleep().

Instead, use a xenstore watch and an event system timeout. xenstore fires watches on @releaseDomain when a domain shuts down.

The logic which checks for the state of the domain is unchanged, and not ideal, but we will leave that for the next patch.

There is not intended to be any semantic change, other than to make the algorithm properly asynchronous and the consequential waiting on xenstore, rather than polling.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c      |   80 ++++++++++++++++++++++++++++++------------
 tools/libxl/libxl_internal.h |    2 ++
 2 files changed, 60 insertions(+), 22 deletions(-)
tools/libxl/libxl_dom.c +58 -22
index dde7e33..6291115 100644
@@ -1028,8 +1028,14 @@ static void domain_suspend_common_wait_guest(libxl__egc *egc,
                                              libxl__domain_suspend_state *dss);
 static void domain_suspend_common_guest_suspended(libxl__egc *egc,
                                          libxl__domain_suspend_state *dss);
+
 static void domain_suspend_common_pvcontrol_suspending(libxl__egc *egc,
       libxl__xswait_state *xswa, int rc, const char *state);
+static void suspend_common_wait_guest_watch(libxl__egc *egc,
+      libxl__ev_xswatch *xsw, const char *watch_path, const char *event_path);
+static void suspend_common_wait_guest_timeout(libxl__egc *egc,
+      libxl__ev_time *ev, const struct timeval *requested_abs);
+
 static void domain_suspend_common_failed(libxl__egc *egc,
                                          libxl__domain_suspend_state *dss);
 static void domain_suspend_common_done(libxl__egc *egc,
@@ -1175,36 +1181,59 @@ static void domain_suspend_common_wait_guest(libxl__egc *egc,
                                              libxl__domain_suspend_state *dss)
 {
     STATE_AO_GC(dss->ao);
+    int rc;
+
+    LOG(DEBUG, "wait for the guest to suspend");
+
+    rc = libxl__ev_xswatch_register(gc, &dss->guest_watch,
+                                    suspend_common_wait_guest_watch,
+                                    "@releaseDomain");
+    if (rc) goto err;
+
+    rc = libxl__ev_time_register_rel(gc, &dss->guest_timeout, 
+                                     suspend_common_wait_guest_timeout,
+                                     60*1000);
+    if (rc) goto err;
+    return;
+
+ err:
+    domain_suspend_common_failed(egc, dss);
+}
+
+static void suspend_common_wait_guest_watch(libxl__egc *egc,
+      libxl__ev_xswatch *xsw, const char *watch_path, const char *event_path)
+{
+    libxl__domain_suspend_state *dss =
+        CONTAINER_OF(xsw, *dss, guest_watch);
+    STATE_AO_GC(dss->ao);
+    xc_domaininfo_t info;
     int ret;
-    int watchdog;
 
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
 
-    LOG(DEBUG, "wait for the guest to suspend");
-    watchdog = 60;
-    while (watchdog > 0) {
-        xc_domaininfo_t info;
-
-        usleep(100000);
-        ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
-        if (ret == 1 && info.domain == domid &&
-            (info.flags & XEN_DOMINF_shutdown)) {
-            int shutdown_reason;
-
-            shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
-                & XEN_DOMINF_shutdownmask;
-            if (shutdown_reason == SHUTDOWN_suspend) {
-                LOG(DEBUG, "guest has suspended");
-                domain_suspend_common_guest_suspended(egc, dss);
-                return;
-            }
+    ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
+    if (ret == 1 && info.domain == domid &&
+        (info.flags & XEN_DOMINF_shutdown)) {
+        int shutdown_reason;
+
+        shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
+            & XEN_DOMINF_shutdownmask;
+        if (shutdown_reason == SHUTDOWN_suspend) {
+            LOG(DEBUG, "guest has suspended");
+            domain_suspend_common_guest_suspended(egc, dss);
+            return;
         }
-
-        watchdog--;
     }
+    /* otherwise, keep waiting */
+}
 
-    LOG(ERROR, "guest did not suspend");
+static void suspend_common_wait_guest_timeout(libxl__egc *egc,
+      libxl__ev_time *ev, const struct timeval *requested_abs)
+{
+    libxl__domain_suspend_state *dss = CONTAINER_OF(ev, *dss, guest_timeout);
+    STATE_AO_GC(dss->ao);
+    LOG(ERROR, "guest did not suspend, timed out");
     domain_suspend_common_failed(egc, dss);
 }
 
@@ -1214,6 +1243,9 @@ static void domain_suspend_common_guest_suspended(libxl__egc *egc,
     STATE_AO_GC(dss->ao);
     int ret;
 
+    libxl__ev_xswatch_deregister(gc, &dss->guest_watch);
+    libxl__ev_time_deregister(gc, &dss->guest_timeout);
+
     if (dss->hvm) {
         ret = libxl__domain_suspend_device_model(gc, dss);
         if (ret) {
@@ -1237,6 +1269,8 @@ static void domain_suspend_common_done(libxl__egc *egc,
 {
     EGC_GC;
     assert(!libxl__xswait_inuse(&dss->pvcontrol));
+    libxl__ev_xswatch_deregister(gc, &dss->guest_watch);
+    libxl__ev_time_deregister(gc, &dss->guest_timeout);
     dss->callback_common_done(egc, dss, ok);
 }
 
@@ -1423,6 +1457,8 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
 
     logdirty_init(&dss->logdirty);
     libxl__xswait_init(&dss->pvcontrol);
+    libxl__ev_xswatch_init(&dss->guest_watch);
+    libxl__ev_time_init(&dss->guest_timeout);
 
     switch (type) {
     case LIBXL_DOMAIN_TYPE_HVM: {
tools/libxl/libxl_internal.h +2 -0
index c4a509b..d428e77 100644
@@ -2411,6 +2411,8 @@ struct libxl__domain_suspend_state {
     int xcflags;
     int guest_responded;
     libxl__xswait_state pvcontrol;
+    libxl__ev_xswatch guest_watch;
+    libxl__ev_time guest_timeout;
     const char *dm_savefile;
     int interval; /* checkpoint interval (for Remus) */
     libxl__save_helper_state shs;
Signature · 1 line
1.7.10.4
#34 lore raw

PATCH 21/23 libxl: suspend: Fix suspend wait corner cases

In reply to Ian Jackson

When we are waiting for a guest to suspend, this suspend operation would continue to wait (until the timeout) if the guest was destroyed or shut down for another reason, or if xc_domain_getinfolist failed.

Handle these cases correctly, as errors.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c |   42 ++++++++++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 12 deletions(-)
tools/libxl/libxl_dom.c +30 -12
index 6291115..88700ee 100644
@@ -1208,24 +1208,42 @@ static void suspend_common_wait_guest_watch(libxl__egc *egc,
     STATE_AO_GC(dss->ao);
     xc_domaininfo_t info;
     int ret;
+    int shutdown_reason;
 
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
 
     ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
-    if (ret == 1 && info.domain == domid &&
-        (info.flags & XEN_DOMINF_shutdown)) {
-        int shutdown_reason;
-
-        shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
-            & XEN_DOMINF_shutdownmask;
-        if (shutdown_reason == SHUTDOWN_suspend) {
-            LOG(DEBUG, "guest has suspended");
-            domain_suspend_common_guest_suspended(egc, dss);
-            return;
-        }
+    if (ret < 0) {
+        LOGE(ERROR, "unable to check for status of guest %"PRId32"", domid);
+        goto err;
+        domain_suspend_common_failed(egc, dss);
+    }
+
+    if (!(ret == 1 && info.domain == domid)) {
+        LOGE(ERROR, "guest %"PRId32" we were suspending has been destroyed",
+             domid);
+        goto err;
+    }
+
+    if (!(info.flags & XEN_DOMINF_shutdown))
+        /* keep waiting */
+        return;
+
+    shutdown_reason = (info.flags >> XEN_DOMINF_shutdownshift)
+        & XEN_DOMINF_shutdownmask;
+    if (shutdown_reason != SHUTDOWN_suspend) {
+        LOG(DEBUG, "guest %"PRId32" we were suspending has shut down"
+            " with unexpected reason code %d", domid, shutdown_reason);
+        goto err;
     }
-    /* otherwise, keep waiting */
+
+    LOG(DEBUG, "guest has suspended");
+    domain_suspend_common_guest_suspended(egc, dss);
+    return;
+
+ err:
+    domain_suspend_common_failed(egc, dss);
 }
 
 static void suspend_common_wait_guest_timeout(libxl__egc *egc,
Signature · 1 line
1.7.10.4
#35 lore raw

PATCH 22/23 libxl: suspend: Async evtchn wait

In reply to Ian Jackson

When negotiating guest suspend via the evtchn ("fast") protocol, abolish synchronous wait for domain suspend.

If the guest supports the event channel suspend protocol, we used to sit in a loop waiting for it to suspend.

Instead, use the new libxl event channel event facility. When we see that the event is signaled, we look at the domain to see if it has suspended.

So the suspend operation no longer blocks with the libxl ctx lock held, and instead returns to the event loop. Additionally, domains which signal the event channel themselves, or undergo other state changes, will be handled more correctly.

We end up making a few more hypercalls.

Also, if we encounter errors setting up the suspend event channel (which should not happen), abort the operation rather than falling back to the xenstore protocol.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c      |   78 ++++++++++++++++++++++++++----------------
 tools/libxl/libxl_internal.h |    3 +-
 2 files changed, 50 insertions(+), 31 deletions(-)
tools/libxl/libxl_dom.c +49 -29
index 88700ee..c431a2d 100644
@@ -1031,8 +1031,12 @@ static void domain_suspend_common_guest_suspended(libxl__egc *egc,
 
 static void domain_suspend_common_pvcontrol_suspending(libxl__egc *egc,
       libxl__xswait_state *xswa, int rc, const char *state);
+static void domain_suspend_common_wait_guest_evtchn(libxl__egc *egc,
+        libxl__ev_evtchn *evev);
 static void suspend_common_wait_guest_watch(libxl__egc *egc,
       libxl__ev_xswatch *xsw, const char *watch_path, const char *event_path);
+static void suspend_common_wait_guest_check(libxl__egc *egc,
+        libxl__domain_suspend_state *dss);
 static void suspend_common_wait_guest_timeout(libxl__egc *egc,
       libxl__ev_time *ev, const struct timeval *requested_abs);
 
@@ -1054,7 +1058,7 @@ static void domain_suspend_callback_common(libxl__egc *egc,
 {
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
-    int ret;
+    int ret, rc;
 
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
@@ -1064,21 +1068,19 @@ static void domain_suspend_callback_common(libxl__egc *egc,
         xc_get_hvm_param(CTX->xch, domid, HVM_PARAM_ACPI_S_STATE, &hvm_s_state);
     }
 
-    if ((hvm_s_state == 0) && (dss->suspend_eventchn >= 0)) {
+    if ((hvm_s_state == 0) && (dss->guest_evtchn.port >= 0)) {
         LOG(DEBUG, "issuing %s suspend request via event channel",
             dss->hvm ? "PVHVM" : "PV");
-        ret = xc_evtchn_notify(dss->xce, dss->suspend_eventchn);
+        ret = xc_evtchn_notify(CTX->xce, dss->guest_evtchn.port);
         if (ret < 0) {
             LOG(ERROR, "xc_evtchn_notify failed ret=%d", ret);
             goto err;
         }
-        ret = xc_await_suspend(CTX->xch, dss->xce, dss->suspend_eventchn);
-        if (ret < 0) {
-            LOG(ERROR, "xc_await_suspend failed ret=%d", ret);
-            goto err;
-        }
-        dss->guest_responded = 1;
-        domain_suspend_common_guest_suspended(egc, dss);
+
+        dss->guest_evtchn.callback = domain_suspend_common_wait_guest_evtchn;
+        rc = libxl__ev_evtchn_wait(gc, &dss->guest_evtchn);
+        if (rc) goto err;
+
         return;
     }
 
@@ -1114,6 +1116,19 @@ static void domain_suspend_callback_common(libxl__egc *egc,
     domain_suspend_common_failed(egc, dss);
 }
 
+static void domain_suspend_common_wait_guest_evtchn(libxl__egc *egc,
+        libxl__ev_evtchn *evev)
+{
+    libxl__domain_suspend_state *dss = CONTAINER_OF(evev, *dss, guest_evtchn);
+    STATE_AO_GC(dss->ao);
+    /* If we should be done waiting, suspend_common_wait_guest_check
+     * will end up calling domain_suspend_common_guest_suspended or
+     * domain_suspend_common_failed, both of which cancel the evtchn
+     * wait.  So re-enable it now. */
+    libxl__ev_evtchn_wait(gc, &dss->guest_evtchn);
+    suspend_common_wait_guest_check(egc, dss);
+}
+
 static void domain_suspend_common_pvcontrol_suspending(libxl__egc *egc,
       libxl__xswait_state *xswa, int rc, const char *state)
 {
@@ -1203,8 +1218,13 @@ static void domain_suspend_common_wait_guest(libxl__egc *egc,
 static void suspend_common_wait_guest_watch(libxl__egc *egc,
       libxl__ev_xswatch *xsw, const char *watch_path, const char *event_path)
 {
-    libxl__domain_suspend_state *dss =
-        CONTAINER_OF(xsw, *dss, guest_watch);
+    libxl__domain_suspend_state *dss = CONTAINER_OF(xsw, *dss, guest_watch);
+    suspend_common_wait_guest_check(egc, dss);
+}
+
+static void suspend_common_wait_guest_check(libxl__egc *egc,
+        libxl__domain_suspend_state *dss)
+{
     STATE_AO_GC(dss->ao);
     xc_domaininfo_t info;
     int ret;
@@ -1261,6 +1281,7 @@ static void domain_suspend_common_guest_suspended(libxl__egc *egc,
     STATE_AO_GC(dss->ao);
     int ret;
 
+    libxl__ev_evtchn_cancel(gc, &dss->guest_evtchn);
     libxl__ev_xswatch_deregister(gc, &dss->guest_watch);
     libxl__ev_time_deregister(gc, &dss->guest_timeout);
 
@@ -1287,6 +1308,7 @@ static void domain_suspend_common_done(libxl__egc *egc,
 {
     EGC_GC;
     assert(!libxl__xswait_inuse(&dss->pvcontrol));
+    libxl__ev_evtchn_cancel(gc, &dss->guest_evtchn);
     libxl__ev_xswatch_deregister(gc, &dss->guest_watch);
     libxl__ev_time_deregister(gc, &dss->guest_timeout);
     dss->callback_common_done(egc, dss, ok);
@@ -1475,6 +1497,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
 
     logdirty_init(&dss->logdirty);
     libxl__xswait_init(&dss->pvcontrol);
+    libxl__ev_evtchn_init(&dss->guest_evtchn);
     libxl__ev_xswatch_init(&dss->guest_watch);
     libxl__ev_time_init(&dss->guest_timeout);
 
@@ -1503,7 +1526,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
           | (debug ? XCFLAGS_DEBUG : 0)
           | (dss->hvm ? XCFLAGS_HVM : 0);
 
-    dss->suspend_eventchn = -1;
+    dss->guest_evtchn.port = -1;
     dss->guest_evtchn_lockfd = -1;
     dss->guest_responded = 0;
     dss->dm_savefile = libxl__device_model_savefile(gc, domid);
@@ -1514,20 +1537,17 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
             dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
     }
 
-    dss->xce = xc_evtchn_open(NULL, 0);
-    if (dss->xce == NULL)
-        goto out;
-    else
-    {
-        port = xs_suspend_evtchn_port(dss->domid);
+    port = xs_suspend_evtchn_port(dss->domid);
 
-        if (port >= 0) {
-            dss->suspend_eventchn =
-                xc_suspend_evtchn_init_exclusive(CTX->xch, dss->xce,
+    if (port >= 0) {
+        dss->guest_evtchn.port =
+            xc_suspend_evtchn_init_exclusive(CTX->xch, CTX->xce,
                                   dss->domid, port, &dss->guest_evtchn_lockfd);
 
-            if (dss->suspend_eventchn < 0)
-                LOG(WARN, "Suspend event channel initialization failed");
+        if (dss->guest_evtchn.port < 0) {
+            LOG(WARN, "Suspend event channel initialization failed");
+            rc = ERROR_FAIL;
+            goto out;
         }
     }
 
@@ -1686,11 +1706,11 @@ static void domain_suspend_done(libxl__egc *egc,
     /* Convenience aliases */
     const uint32_t domid = dss->domid;
 
-    if (dss->suspend_eventchn > 0)
-        xc_suspend_evtchn_release(CTX->xch, dss->xce, domid,
-                           dss->suspend_eventchn, &dss->guest_evtchn_lockfd);
-    if (dss->xce != NULL)
-        xc_evtchn_close(dss->xce);
+    libxl__ev_evtchn_cancel(gc, &dss->guest_evtchn);
+
+    if (dss->guest_evtchn.port > 0)
+        xc_suspend_evtchn_release(CTX->xch, CTX->xce, domid,
+                           dss->guest_evtchn.port, &dss->guest_evtchn_lockfd);
 
     dss->callback(egc, dss, rc);
 }
tools/libxl/libxl_internal.h +1 -2
index d428e77..126e5bf 100644
@@ -2404,8 +2404,7 @@ struct libxl__domain_suspend_state {
     int debug;
     const libxl_domain_remus_info *remus;
     /* private */
-    xc_evtchn *xce; /* event channel handle */
-    int suspend_eventchn;
+    libxl__ev_evtchn guest_evtchn;
     int guest_evtchn_lockfd;
     int hvm;
     int xcflags;
Signature · 1 line
1.7.10.4
#36 lore raw

PATCH 23/23 libxl: suspend: Apply guest timeout in evtchn case

In reply to Ian Jackson

When negotiating guest suspend via the evtchn ("fast") protocol, the guest may still fail to respond.

So set the timeout. The existing error path will already properly tear down our (event channel) wait.

CC Stefano Stabellini <[email protected]>
CC Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c |    5 +++++
 1 file changed, 5 insertions(+)
tools/libxl/libxl_dom.c +5 -0
index c431a2d..820ec3a 100644
@@ -1081,6 +1081,11 @@ static void domain_suspend_callback_common(libxl__egc *egc,
         rc = libxl__ev_evtchn_wait(gc, &dss->guest_evtchn);
         if (rc) goto err;
 
+        rc = libxl__ev_time_register_rel(gc, &dss->guest_timeout,
+                                         suspend_common_wait_guest_timeout,
+                                         60*1000);
+        if (rc) goto err;
+
         return;
     }
 
Signature · 1 line
1.7.10.4
#37 lore raw
In reply to Ian Jackson
Quoted 9 lines · Ian Jackson

This series removes the usleeps and waiting loops in libxl_dom.c and replaces them with event-callback code.

Firstly, some documentation of things I had to reverse-engineer:
01/23 xen: Document XEN_DOMCTL_subscribe
02/23 xen: Document that EVTCHNOP_bind_interdomain signals 03/23 docs: Document event-channel-based suspend protocol 04/23 libxc: Document xenctrl.h event channel calls
Arguably these might be 4.4 material (hence the CC to George).

These document pretty important aspects of behavior (fixing what might be considered documentation bugs), and obviously can have no functional impact. I guess the only risk is that they might be wrong and mislead someone, but I think that's pretty low. :-)

These four:

Acked-by
#38 lore raw
Quoted 11 lines · George Dunlap
Quoted 9 lines · Ian Jackson

This series removes the usleeps and waiting loops in libxl_dom.c and replaces them with event-callback code.

Firstly, some documentation of things I had to reverse-engineer:
01/23 xen: Document XEN_DOMCTL_subscribe
02/23 xen: Document that EVTCHNOP_bind_interdomain signals 03/23 docs: Document event-channel-based suspend protocol 04/23 libxc: Document xenctrl.h event channel calls
Arguably these might be 4.4 material (hence the CC to George).

Quoted 8 lines

These document pretty important aspects of behavior (fixing what might be considered documentation bugs), and obviously can have no functional impact. I guess the only risk is that they might be wrong and mislead someone, but I think that's pretty low. :-)

These four:

They look good to me too:

#39 lore raw
Quoted 17 lines · Ian Campbell
Quoted 16 lines · George Dunlap
Quoted 9 lines · Ian Jackson

This series removes the usleeps and waiting loops in libxl_dom.c and replaces them with event-callback code.

Firstly, some documentation of things I had to reverse-engineer:
01/23 xen: Document XEN_DOMCTL_subscribe
02/23 xen: Document that EVTCHNOP_bind_interdomain signals 03/23 docs: Document event-channel-based suspend protocol 04/23 libxc: Document xenctrl.h event channel calls
Arguably these might be 4.4 material (hence the CC to George).

These document pretty important aspects of behavior (fixing what might be considered documentation bugs), and obviously can have no functional impact. I guess the only risk is that they might be wrong and mislead someone, but I think that's pretty low. :-)

Quoted 6 lines
Quoted 3 lines

These four:

They look good to me too:

And I've now applied just those 4 docs changes.

Ian.