Skip to contentSkip to conversation

xl: fix guest reboot failures

PATCH In review Ian Campbell 3 participants 13 messages Jun 22, 2012 – Jun 29, 2012

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xl: fix guest reboot failures

The following fixes recent test failures reboot a guest (e.g. flight 13302).

The main fix is the first patch which stops us from trying to validate a domains scheduler paramters before it has been created.

The others are miscellaneous fixes I noticed while tracking that down.

#1 lore raw

PATCH 1/1 libxl: validate scheduler parameters

# HG changeset patch
# User Ian Campbell <[email protected]> # Date 1340361703 -3600
# Node ID 998d48ccb8905907cb2f104b475e5ab6ad445348 # Parent 5fb3c536b5a8810fb8be4149df609d272beff959
libxl: validate scheduler parameters

This was previously done by xl itself however the domain was not created at that point so there was no domid to check. This happened to work on first boot because xl's global domid was initialised to zero so we would (incorrectly) validate the new domain to be against domain0. On reboot though we would try to use the old domain's id and fail.

sched_params_valid is moved and gains a gc+domid parameters and s/ctx/CTX/. The call is placed after libxl__domain_build_info_setdefault in the create path, because set_defaults doesn't have access to the domid and there are other callers which don't even have a domid to give it.

tools/libxl/libxl_create.c +49 -0
diff -r 5fb3c536b5a8 -r 998d48ccb890 tools/libxl/libxl_create.c
@@ -72,6 +72,49 @@ int libxl__domain_create_info_setdefault
     return 0;
 }
 
+static int sched_params_valid(libxl__gc *gc,
+                              uint32_t domid, libxl_domain_sched_params *scp)
+{
+    int has_weight = scp->weight != LIBXL_DOMAIN_SCHED_PARAM_WEIGHT_DEFAULT;
+    int has_period = scp->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT;
+    int has_slice = scp->slice != LIBXL_DOMAIN_SCHED_PARAM_SLICE_DEFAULT;
+    int has_extratime =
+                scp->extratime != LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT;
+    libxl_domain_sched_params sci;
+
+    libxl_domain_sched_params_get(CTX, domid, &sci);
+
+    /* The sedf scheduler needs some more consistency checking */
+    if (sci.sched == LIBXL_SCHEDULER_SEDF) {
+        if (has_weight && (has_period || has_slice))
+            return 0;
+        if (has_period != has_slice)
+            return 0;
+
+        /*
+         * Idea is, if we specify a weight, then both period and
+         * slice has to be zero. OTOH, if we do not specify a weight,
+         * that means we want a pure best effort domain or an actual
+         * real-time one. In the former case, it is period that needs
+         * to be zero, in the latter, weight should be.
+         */
+        if (has_weight) {
+            scp->slice = 0;
+            scp->period = 0;
+        }
+        else if (!has_period) {
+            /* We can setup a proper best effort domain (extra time only)
+             * iff we either already have or are asking for some extra time. */
+            scp->weight = has_extratime ? scp->extratime : sci.extratime;
+            scp->period = 0;
+        }
+        if (has_period && has_slice)
+            scp->weight = 0;
+    }
+
+    return 1;
+}
+
 int libxl__domain_build_info_setdefault(libxl__gc *gc,
                                         libxl_domain_build_info *b_info)
 {
@@ -622,6 +665,12 @@ static void initiate_domain_create(libxl
     ret = libxl__domain_build_info_setdefault(gc, &d_config->b_info);
     if (ret) goto error_out;
 
+    if (!sched_params_valid(gc, domid, &d_config->b_info.sched_params)) {
+        LOG(ERROR, "Invalid scheduling parameters\n");
+        ret = ERROR_INVAL;
+        goto error_out;
+    }
+
     for (i = 0; i < d_config->num_disks; i++) {
         ret = libxl__device_disk_setdefault(gc, &d_config->disks[i]);
         if (ret) goto error_out;
tools/libxl/xl_cmdimpl.c +0 -46

58 lines not on this page · raw message

#2 lore raw

Re: [PATCH 1 of 5] libxl: validate scheduler parameters

Quoted 14 lines · Ian Campbell

# HG changeset patch
# User Ian Campbell <[email protected]> # Date 1340361703 -3600
# Node ID 998d48ccb8905907cb2f104b475e5ab6ad445348 # Parent 5fb3c536b5a8810fb8be4149df609d272beff959
libxl: validate scheduler parameters

This was previously done by xl itself however the domain was not created at that point so there was no domid to check. This happened to work on first boot because xl's global domid was initialised to zero so we would (incorrectly) validate the new domain to be against domain0. On reboot though we would try to use the old domain's id and fail.

Quoted 7 lines

sched_params_valid is moved and gains a gc+domid parameters and s/ctx/CTX/. The call is placed after libxl__domain_build_info_setdefault in the create path, because set_defaults doesn't have access to the domid and there are other callers which don't even have a domid to give it.

After consultation with Ian J and Dario I have committed this one in order to get a test pass ASAP. I'll leave the remainder of this series to get properly reviewed.

Ian.

Quoted 135 lines

diff -r 5fb3c536b5a8 -r 998d48ccb890 tools/libxl/libxl_create.c

tools/libxl/libxl_create.c +49 -0
@@ -72,6 +72,49 @@ int libxl__domain_create_info_setdefault
     return 0;
 }
 
+static int sched_params_valid(libxl__gc *gc,
+                              uint32_t domid, libxl_domain_sched_params *scp)
+{
+    int has_weight = scp->weight != LIBXL_DOMAIN_SCHED_PARAM_WEIGHT_DEFAULT;
+    int has_period = scp->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT;
+    int has_slice = scp->slice != LIBXL_DOMAIN_SCHED_PARAM_SLICE_DEFAULT;
+    int has_extratime =
+                scp->extratime != LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT;
+    libxl_domain_sched_params sci;
+
+    libxl_domain_sched_params_get(CTX, domid, &sci);
+
+    /* The sedf scheduler needs some more consistency checking */
+    if (sci.sched == LIBXL_SCHEDULER_SEDF) {
+        if (has_weight && (has_period || has_slice))
+            return 0;
+        if (has_period != has_slice)
+            return 0;
+
+        /*
+         * Idea is, if we specify a weight, then both period and
+         * slice has to be zero. OTOH, if we do not specify a weight,
+         * that means we want a pure best effort domain or an actual
+         * real-time one. In the former case, it is period that needs
+         * to be zero, in the latter, weight should be.
+         */
+        if (has_weight) {
+            scp->slice = 0;
+            scp->period = 0;
+        }
+        else if (!has_period) {
+            /* We can setup a proper best effort domain (extra time only)
+             * iff we either already have or are asking for some extra time. */
+            scp->weight = has_extratime ? scp->extratime : sci.extratime;
+            scp->period = 0;
+        }
+        if (has_period && has_slice)
+            scp->weight = 0;
+    }
+
+    return 1;
+}
+
 int libxl__domain_build_info_setdefault(libxl__gc *gc,
                                         libxl_domain_build_info *b_info)
 {
@@ -622,6 +665,12 @@ static void initiate_domain_create(libxl
     ret = libxl__domain_build_info_setdefault(gc, &d_config->b_info);
     if (ret) goto error_out;
 
+    if (!sched_params_valid(gc, domid, &d_config->b_info.sched_params)) {
+        LOG(ERROR, "Invalid scheduling parameters\n");
+        ret = ERROR_INVAL;
+        goto error_out;
+    }
+
     for (i = 0; i < d_config->num_disks; i++) {
         ret = libxl__device_disk_setdefault(gc, &d_config->disks[i]);
         if (ret) goto error_out;

diff -r 5fb3c536b5a8 -r 998d48ccb890 tools/libxl/xl_cmdimpl.c

tools/libxl/xl_cmdimpl.c +0 -46
@@ -550,48 +550,6 @@ vcpp_out:
     return rc;
 }
 
-static int sched_params_valid(libxl_domain_sched_params *scp)
-{
-    int has_weight = scp->weight != LIBXL_DOMAIN_SCHED_PARAM_WEIGHT_DEFAULT;
-    int has_period = scp->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT;
-    int has_slice = scp->slice != LIBXL_DOMAIN_SCHED_PARAM_SLICE_DEFAULT;
-    int has_extratime =
-                scp->extratime != LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT;
-    libxl_domain_sched_params sci;
-
-    libxl_domain_sched_params_get(ctx, domid, &sci);
-
-    /* The sedf scheduler needs some more consistency checking */
-    if (sci.sched == LIBXL_SCHEDULER_SEDF) {
-        if (has_weight && (has_period || has_slice))
-            return 0;
-        if (has_period != has_slice)
-            return 0;
-
-        /*
-         * Idea is, if we specify a weight, then both period and
-         * slice has to be zero. OTOH, if we do not specify a weight,
-         * that means we want a pure best effort domain or an actual
-         * real-time one. In the former case, it is period that needs
-         * to be zero, in the latter, weight should be.
-         */
-        if (has_weight) {
-            scp->slice = 0;
-            scp->period = 0;
-        }
-        else if (!has_period) {
-            /* We can setup a proper best effort domain (extra time only)
-             * iff we either already have or are asking for some extra time. */
-            scp->weight = has_extratime ? scp->extratime : sci.extratime;
-            scp->period = 0;
-        }
-        if (has_period && has_slice)
-            scp->weight = 0;
-    }
-
-    return 1;
-}
-
 static void parse_config_data(const char *config_source,
                               const char *config_data,
                               int config_len,
@@ -686,10 +644,6 @@ static void parse_config_data(const char
         b_info->sched_params.latency = l;
     if (!xlu_cfg_get_long (config, "extratime", &l, 0))
         b_info->sched_params.extratime = l;
-    if (!sched_params_valid(&b_info->sched_params)) {
-        fprintf(stderr, "Invalid scheduling parameters\n");
-        exit(1);
-    }
 
     if (!xlu_cfg_get_long (config, "vcpus", &l, 0)) {
         b_info->max_vcpus = l;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#3 lore raw

Re: [PATCH 1 of 5] libxl: validate scheduler parameters

Quoted 6 lines · Ian Campbell
Quoted 6 lines

sched_params_valid is moved and gains a gc+domid parameters and s/ctx/CTX/. The call is placed after libxl__domain_build_info_setdefault in the create path, because set_defaults doesn't have access to the domid and there are other callers which don't even have a domid to give it.

Quoted 6 lines
Quoted 1 line

After consultation with Ian J and Dario I have committed this one in order to get a test pass ASAP. I'll leave the remainder of this series to get properly reviewed.

This looked good at inspection, and it indeed works for the credit scheduler. Unfortunately, sedf seems to require the domain's vcpus to be properly setup (in the hypervisor), which is not true at this point (it is still too early!).

In fact, what happens is it fails right here:

Quoted 7 lines
Quoted 7 lines

diff -r 5fb3c536b5a8 -r 998d48ccb890 tools/libxl/libxl_create.c

tools/libxl/libxl_create.c +0 -0
@@ -72,6 +72,49 @@ int libxl__domain_create_info_setdefault
     return 0;
 }
 
Quoted 12 lines

tools/libxl/libxl_create.c int libxl__domain_create_info_setdefault · line 86

Quoted 12 lines
tools/libxl/libxl_create.c +12 -0
+static int sched_params_valid(libxl__gc *gc,
+                              uint32_t domid, libxl_domain_sched_params *scp)
+{
+    int has_weight = scp->weight != LIBXL_DOMAIN_SCHED_PARAM_WEIGHT_DEFAULT;
+    int has_period = scp->period != LIBXL_DOMAIN_SCHED_PARAM_PERIOD_DEFAULT;
+    int has_slice = scp->slice != LIBXL_DOMAIN_SCHED_PARAM_SLICE_DEFAULT;
+    int has_extratime =
+                scp->extratime != LIBXL_DOMAIN_SCHED_PARAM_EXTRATIME_DEFAULT;
+    libxl_domain_sched_params sci;
+
+    libxl_domain_sched_params_get(CTX, domid, &sci);
+

With a splat like this:
(XEN) ----[ Xen-4.2-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82c480120d57>] sedf_adjust+0x8bd/0x9ad
(XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor

(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000001
(XEN) rdx: ffff82c4802b7e48   rsi: ffff83023e401000   rdi: ffff83031b3e34e4

Because of this code in xen/common/sched_sedf.c:

        if ( p->vcpu[0] == NULL )
        {
            rc = -EINVAL;
            goto out;
        }

What I expect to happen, then, is the win7 HVM tests to restart being successful, but the sedf-s ones restart failing. :-(

New patch against this patch coming. This time I tested it on both sedf and credit and tested rebooting a domain as well... Let's hope this is going to be the end of this!

Thanks and Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#4 lore raw

Re: [PATCH 1 of 5] libxl: validate scheduler parameters

Quoted 4 lines · Dario Faggioli

New patch against this patch coming. This time I tested it on both sedf and credit and tested rebooting a domain as well... Let's hope this is going to be the end of this!

Which just for the records is <f5f29d8e2ccf642cad89.1340381818@Solace>, which I think Ian is checking in right now. Let's wait for some verdict from OSStest during the weekend. :-)

Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#5 lore raw

PATCH 1/1 libxl: initialise cpupoolinfo in libxl__domain_scheduler

In reply to Ian Campbell

# HG changeset patch
# User Ian Campbell <[email protected]>
# Date 1340362527 -3600
# Node ID b6a78743e13fb5b7f652f25a541eb425a21f1396 # Parent 998d48ccb8905907cb2f104b475e5ab6ad445348
libxl: initialise cpupoolinfo in libxl__domain_scheduler

If libxl_cpupool_info fails then we would call libxl_cpupoolinfo_dispose on an uninitialised struct, and possibly free an invalid pointer.

tools/libxl/libxl_dom.c +1 -0
diff -r 998d48ccb890 -r b6a78743e13f tools/libxl/libxl_dom.c
@@ -84,6 +84,7 @@ libxl_scheduler libxl__domain_scheduler(
     if (cpupool < 0)
         return sched;
 
+    libxl_cpupoolinfo_init(&poolinfo);
     rc = libxl_cpupool_info(CTX, &poolinfo, cpupool);
     if (rc < 0)
         goto out;
#6 lore raw

PATCH 1/1 libxl: correct type of cpupool variable

In reply to Ian Campbell

# HG changeset patch
# User Ian Campbell <[email protected]> # Date 1340362529 -3600
# Node ID b63faf77bff33d7c9ebfe7260ecaffc26f3773a8 # Parent b6a78743e13fb5b7f652f25a541eb425a21f1396
libxl: correct type of cpupool variable.

libxl__domain_cpupool returns int and can return ERROR_* so we need to use a signed type.

tools/libxl/libxl_dom.c +1 -1
diff -r b6a78743e13f -r b63faf77bff3 tools/libxl/libxl_dom.c
@@ -76,7 +76,7 @@ int libxl__domain_cpupool(libxl__gc *gc,
 
 libxl_scheduler libxl__domain_scheduler(libxl__gc *gc, uint32_t domid)
 {
-    uint32_t cpupool = libxl__domain_cpupool(gc, domid);
+    int cpupool = libxl__domain_cpupool(gc, domid);
     libxl_cpupoolinfo poolinfo;
     libxl_scheduler sched = LIBXL_SCHEDULER_UNKNOWN;
     int rc;
#7 lore raw

PATCH 1/1 libxl: log on failure in cpupool_info and libxl__domain_cpupool

In reply to Ian Campbell

# HG changeset patch
# User Ian Campbell <[email protected]>
# Date 1340362601 -3600
# Node ID 2829b66dfe19afd4537ea4b2cfcef0cf28b49472
# Parent b63faf77bff33d7c9ebfe7260ecaffc26f3773a8
libxl: log on failure in cpupool_info and libxl__domain_cpupool

Also in cpupool_info propagate the failure value from libxl_cpumap_alloc.

tools/libxl/libxl.c +12 -1
diff -r b63faf77bff3 -r 2829b66dfe19 tools/libxl/libxl.c
@@ -562,16 +562,27 @@ static int cpupool_info(libxl__gc *gc,
 
     xcinfo = xc_cpupool_getinfo(CTX->xch, poolid);
     if (xcinfo == NULL)
+    {
+        LOGE(ERROR, "failed to get info for cpupool%d\n", poolid);
         return ERROR_FAIL;
+    }
 
     if (exact && xcinfo->cpupool_id != poolid)
+    {
+        LOG(ERROR, "got info for cpupool%d, wanted cpupool%d\n",
+            xcinfo->cpupool_id, poolid);
         goto out;
+    }
 
     info->poolid = xcinfo->cpupool_id;
     info->sched = xcinfo->sched_id;
     info->n_dom = xcinfo->n_dom;
-    if (libxl_cpumap_alloc(CTX, &info->cpumap))
+    rc = libxl_cpumap_alloc(CTX, &info->cpumap);
+    if (rc)
+    {
+        LOG(ERROR, "unable to allocate cpumap %d\n", rc);
         goto out;
+    }
     memcpy(info->cpumap.map, xcinfo->cpumap, info->cpumap.size);
 
     rc = 0;
tools/libxl/libxl_dom.c +6 -1
diff -r b63faf77bff3 -r 2829b66dfe19 tools/libxl/libxl_dom.c
@@ -67,10 +67,15 @@ int libxl__domain_cpupool(libxl__gc *gc,
 
     ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);
     if (ret != 1)
+    {
+        LOGE(ERROR, "getinfolist failed %d\n", ret);
         return ERROR_FAIL;
+    }
     if (info.domain != domid)
+    {
+        LOGE(ERROR, "got info for dom%d, wanted dom%d\n", info.domain, domid);
         return ERROR_FAIL;
-
+    }
     return info.cpupool;
 }
#8 lore raw

PATCH 1/1 xl: initialise domid to an explicitly invalid value

In reply to Ian Campbell

# HG changeset patch
# User Ian Campbell <[email protected]> # Date 1340362601 -3600
# Node ID 9190b57657f4b6def855fa22de905f45d22ef7de # Parent 2829b66dfe19afd4537ea4b2cfcef0cf28b49472
xl: initialise domid to an explicitly invalid value

also ensure it is invalid whenever we destroy the domain.

tools/libxl/xl_cmdimpl.c +13 -3
diff -r 2829b66dfe19 -r 9190b57657f4 tools/libxl/xl_cmdimpl.c
@@ -68,7 +68,8 @@ libxl_ctx *ctx;
 xlchild children[child_max];
 
 /* when we operate on a domain, it is this one: */
-static uint32_t domid;
+#define INVALID_DOMID ~0
+static uint32_t domid = INVALID_DOMID;
 static const char *common_domname;
 static int fd_lock = -1;
 
@@ -1389,6 +1390,7 @@ static int handle_domain_death(uint32_t 
     case LIBXL_ACTION_ON_SHUTDOWN_DESTROY:
         LOG("Domain %d needs to be cleaned up: destroying the domain", domid);
         libxl_domain_destroy(ctx, domid);
+        domid = INVALID_DOMID;
         break;
 
     case LIBXL_ACTION_ON_SHUTDOWN_COREDUMP_DESTROY:
@@ -1451,6 +1453,12 @@ static int preserve_domain(uint32_t domi
     LOG("Preserving domain %d %s with suffix%s", domid, d_config->c_info.name, stime);
     rc = libxl_domain_preserve(ctx, domid, &d_config->c_info, stime, new_uuid);
 
+    /*
+     * Although domid still exists it is no longer the one we are concerned
+     * with.
+     */
+    domid = INVALID_DOMID;
+
     return rc == 0 ? 1 : 0;
 }
 
@@ -1738,7 +1746,7 @@ static int create_domain(struct domain_c
         goto out;
 
 start:
-    domid = -1;
+    assert(domid == INVALID_DOMID);
 
     rc = acquire_lock();
     if (rc < 0)
@@ -1985,8 +1993,10 @@ start:
 
 error_out:
     release_lock();
-    if (libxl_domid_valid_guest(domid))
+    if (libxl_domid_valid_guest(domid)) {
         libxl_domain_destroy(ctx, domid);
+        domid = INVALID_DOMID;
+    }
 
 out:
     if (logfile != 2)
#10 lore raw
Quoted 5 lines · Ian Jackson

Ian Campbell writes ("[Xen-devel] [PATCH 0 of 5] xl: fix guest reboot failures"):

Quoted 2 lines

The following fixes recent test failures reboot a guest (e.g. flight 13302).

Patches 2..5:

I sent out a V2 of this series with the (trivial) reject fixed. I guess you fixed that yourself?

Quoted 7 lines

1 was applied earlier.

Thanks,
Ian.

#11 lore raw

Ian Campbell writes ("Re: [Xen-devel] [PATCH 0 of 5] xl: fix guest reboot failures"):

Quoted 9 lines · Ian Jackson
Quoted 5 lines

Ian Campbell writes ("[Xen-devel] [PATCH 0 of 5] xl: fix guest reboot failures"):

Quoted 2 lines

The following fixes recent test failures reboot a guest (e.g. flight 13302).

Patches 2..5:

I sent out a V2 of this series with the (trivial) reject fixed. I guess you fixed that yourself?

I just replied to the wrong 0 of 5. There wasn't any reject and I applied v2 of the series.

Sorry for the confusion.

Ian.

#12 lore raw
Quoted 6 lines · Ian Jackson

Ian Campbell writes ("Re: [Xen-devel] [PATCH 0 of 5] xl: fix guest reboot failures"):

Quoted 5 lines · Ian Jackson
Quoted 4 lines

Ian Campbell writes ("[Xen-devel] [PATCH 0 of 5] xl: fix guest reboot failures"):

Quoted 2 lines

The following fixes recent test failures reboot a guest (e.g. flight 13302).

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Patches 2..5:

I sent out a V2 of this series with the (trivial) reject fixed. I guess you fixed that yourself?

I just replied to the wrong 0 of 5. There wasn't any reject and I applied v2 of the series.

Grand!

Quoted 1 line

Sorry for the confusion.

No worries.

Ian.

Quoted 2 lines

Ian.