Skip to contentSkip to conversation

libs/light: fix a race when domain are destroied and created concurrently

PATCH In review Dario Faggioli 2 participants 9 messages Jan 12, 2022 – Jan 19, 2022

lore
Share & tools
lore
#0 lore raw

cover libs/light: fix a race when domain are destroied and created concurrently

It is possible to encounter a segfault in libxl during concurrent domain create and destroy operations.

This is because Placement of existing domains on the host's CPUs is examined when creating a new domain, but the existing logic does not tolerate well a domain disappearing during the examination.

The race is quite difficult to trigger. Whan that happens, this is an example of a backtrace:

#0  libxl_bitmap_dispose (map=map@entry=0x50) at libxl_utils.c:626
#1  0x00007fe72c993a32 in libxl_vcpuinfo_dispose (p=p@entry=0x38) at _libxl_types.c:692
#2  0x00007fe72c94e3c4 in libxl_vcpuinfo_list_free (list=0x0, nr=<optimized out>) at libxl_utils.c:1059
#3  0x00007fe72c9528bf in nr_vcpus_on_nodes (vcpus_on_node=0x7fe71000eb60, suitable_cpumap=0x7fe721df0d38, tinfo_elements=48, tinfo=0x7fe7101b3900, gc=0x7fe7101bbfa0) at libxl_numa.c:258
#4  libxl__get_numa_candidate (gc=gc@entry=0x7fe7100033a0, min_free_memkb=4233216, min_cpus=4, min_nodes=min_nodes@entry=0, max_nodes=max_nodes@entry=0, suitable_cpumap=suitable_cpumap@entry=0x7fe721df0d38, numa_cmpf=0x7fe72c940110 <numa_cmpf>, cndt_out=0x7fe721df0cf0, cndt_found=0x7fe721df0cb4) at libxl_numa.c:394
#5  0x00007fe72c94152b in numa_place_domain (d_config=0x7fe721df11b0, domid=975, gc=0x7fe7100033a0) at libxl_dom.c:209
#6  libxl__build_pre (gc=gc@entry=0x7fe7100033a0, domid=domid@entry=975, d_config=d_config@entry=0x7fe721df11b0, state=state@entry=0x7fe710077700) at libxl_dom.c:436
#7  0x00007fe72c92c4a5 in libxl__domain_build (gc=0x7fe7100033a0, d_config=d_config@entry=0x7fe721df11b0, domid=975, state=0x7fe710077700) at libxl_create.c:444
#8  0x00007fe72c92de8b in domcreate_bootloader_done (egc=0x7fe721df0f60, bl=0x7fe7100778c0, rc=<optimized out>) at libxl_create.c:1222
#9  0x00007fe72c980425 in libxl__bootloader_run (egc=egc@entry=0x7fe721df0f60, bl=bl@entry=0x7fe7100778c0) at libxl_bootloader.c:403

#10 0x00007fe72c92f281 in initiate_domain_create (egc=egc@entry=0x7fe721df0f60, dcs=dcs@entry=0x7fe7100771b0) at libxl_create.c:1159
#11 0x00007fe72c92f456 in do_domain_create (ctx=ctx@entry=0x7fe71001c840, d_config=d_config@entry=0x7fe721df11b0, domid=domid@entry=0x7fe721df10a8, restore_fd=restore_fd@entry=-1, send_back_fd=send_back_fd@entry=-1, params=params@entry=0x0, ao_how=0x0, aop_console_how=0x7fe721df10f0) at libxl_create.c:1856 #12 0x00007fe72c92f776 in libxl_domain_create_new (ctx=0x7fe71001c840, d_config=d_config@entry=0x7fe721df11b0, domid=domid@entry=0x7fe721df10a8, ao_how=ao_how@entry=0x0, aop_console_how=aop_console_how@entry=0x7fe721df10f0) at libxl_create.c:2075

Luckily, it is easy to close the race, by just making sure that libvxl_list_vcpu() returns 0 as the number of vCPUs of the domain, when it also returns NULL as the list of them.

Regards
---
Dario Faggioli (2):
tools/libs/light: numa placement: don't try to free a NULL list of vcpus tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL

 tools/libs/light/libxl_domain.c | 14 ++++++++------
 tools/libs/light/libxl_numa.c   |  4 +++-
 2 files changed, 11 insertions(+), 7 deletions(-)

--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

#1 lore raw

PATCH 1/2 tools/libs/light: numa placement: don't try to free a NULL list of vcpus

If libxl_vcpu_list() returned NULL, we should not call libxl_numainfo_list_free() as:
1) it'll fail trying to (double) free() *list
2) there should be nothing to free anyway

---
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Juergen Gross <[email protected]>

---
 tools/libs/light/libxl_numa.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libs/light/libxl_numa.c +1 -1
index a8a75f89e9..3679028c79 100644
@@ -253,9 +253,9 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo,
             }
         }
 
+        libxl_vcpuinfo_list_free(vinfo, nr_dom_vcpus);
  next:
         libxl_cpupoolinfo_dispose(&cpupool_info);
-        libxl_vcpuinfo_list_free(vinfo, nr_dom_vcpus);
     }
 
     libxl_bitmap_dispose(&dom_nodemap);
#2 lore raw

Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus

Quoted 2 lines · Dario Faggioli

If libxl_vcpu_list() returned NULL, we should not call libxl_numainfo_list_free() as:

You mean libxl_vcpuinfo_list_free() ?

Quoted 1 line

1) it'll fail trying to (double) free() *list

This isn't really an issue. free(NULL) is legit, can be call as many time as you want.

Quoted 1 line

2) there should be nothing to free anyway

The issue here is that it doesn't appear to be true. Even if "info" is NULL, "nr" have an other value than 0, so libxl_vcpuinfo_list_free() will try to access random addresses.

Quoted 2 lines

Can I suggest to make libxl_vcpuinfo_list_free() work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL. Even if that seems wrong, and the caller should use the correct value.

Also I think it is better to keep the free in the exit path at the end of the loop.

Thanks,

Signature · 1 line
Anthony PERARD
#3 lore raw

Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus

Quoted 9 lines · Anthony PERARD
Quoted 1 line · Dario Faggioli

2) there should be nothing to free anyway

The issue here is that it doesn't appear to be true. Even if "info" is
NULL, "nr" have an other value than 0, so libxl_vcpuinfo_list_free() will try to access random addresses.

My point here was that if vinfo is NULL (because libxl_list_vcpu() returned NULL), there aren't any list element or any list to free, so we can avoid calling libxl_vcpuinfo_list_free().

Then, sure, if we call it with a NULL vinfo and a non-zero nr_dom_vcpus, things explode, but this was being addressed in patch 2.

This first one was really only about not trying to free an empty list. And not to workaround the fact that it currently can make things explode, just because it feels pointless.

Quoted 7 lines
Quoted 2 lines

Can I suggest to make libxl_vcpuinfo_list_free() work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL.

I thought about that, and we certainly can do it.

However, I think it's unrelated to this patch so, if I do it, I'll do it in its own one.

Also, if we go that way, I guess we want to change libxl_cputopology_list_free(), libxl_pcitopology_list_free(), libxl_numainfo_list_free(), libxl_dominfo_list_free(), libxl_vminfo_list_free() and libxl_cpupoolinfo_list_free(), don't we?

Quoted 4 lines

Also I think it is better to keep the free in the exit path at the end
of the loop.

Can I ask why as, as I was trying to say above, if we are sent directly to next by one of the goto, we do know that vinfo is NULL and libxl_vcpuinfo_list_free() will be basically a NOP, however it is implemented.

Of course, you're the maintainer of this code, so I'll do like that if you ask... I'm just curious. :-)

Actually, if you really think that the call to libxl_vcpuinfo_list_free() should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the crash.

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)
#4 lore raw

Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus

Quoted 30 lines · Dario Faggioli
Quoted 9 lines · Anthony PERARD
Quoted 1 line · Dario Faggioli

2) there should be nothing to free anyway

The issue here is that it doesn't appear to be true. Even if "info" is
NULL, "nr" have an other value than 0, so libxl_vcpuinfo_list_free() will try to access random addresses.

My point here was that if vinfo is NULL (because libxl_list_vcpu() returned NULL), there aren't any list element or any list to free, so we can avoid calling libxl_vcpuinfo_list_free().

Then, sure, if we call it with a NULL vinfo and a non-zero nr_dom_vcpus, things explode, but this was being addressed in patch 2.

This first one was really only about not trying to free an empty list. And not to workaround the fact that it currently can make things explode, just because it feels pointless.

Quoted 7 lines
Quoted 2 lines

Can I suggest to make libxl_vcpuinfo_list_free() work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL.

I thought about that, and we certainly can do it.

Quoted 7 lines

However, I think it's unrelated to this patch so, if I do it, I'll do it in its own one.

Also, if we go that way, I guess we want to change libxl_cputopology_list_free(), libxl_pcitopology_list_free(), libxl_numainfo_list_free(), libxl_dominfo_list_free(), libxl_vminfo_list_free() and libxl_cpupoolinfo_list_free(), don't we?

I actually don't know if that would be useful. Those functions do work as expected (I think) with the right parameters: they do nothing when called with (NULL, 0). free(NULL) does nothing.

So checking for NULL before using nr would probably just be a workaround for programming mistake in the function allocating the object or some missing initialisation in the caller. So I don't think it is important anymore.

Quoted 11 lines
Quoted 4 lines

Also I think it is better to keep the free in the exit path at the end
of the loop.

Can I ask why as, as I was trying to say above, if we are sent directly to next by one of the goto, we do know that vinfo is NULL and libxl_vcpuinfo_list_free() will be basically a NOP, however it is implemented.

Of course, you're the maintainer of this code, so I'll do like that if you ask... I'm just curious. :-)

Freeing resources should always been attempted, even if that mean to check whether there's something to free before calling the associated free function. Imagine someone adding some code that could fail after the libxl_list_vcpu(), then when that new code fails it would goto next; and the allocated data from libxl_list_vcpu() would never be freed.

Quoted 4 lines

Actually, if you really think that the call to libxl_vcpuinfo_list_free() should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the crash.

This patch is just a workaround a bug in libxl_list_vcpu(), so I think it can be dropped.

Cheers,

Signature · 1 line
Anthony PERARD
#5 lore raw

Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus

Quoted 8 lines · Anthony PERARD
Quoted 6 lines · Dario Faggioli

Also, if we go that way, I guess we want to change libxl_cputopology_list_free(), libxl_pcitopology_list_free(), libxl_numainfo_list_free(), libxl_dominfo_list_free(), libxl_vminfo_list_free() and libxl_cpupoolinfo_list_free(), don't we?

Quoted 5 lines

I actually don't know if that would be useful. Those functions do work
as expected (I think) with the right parameters: they do nothing when called with (NULL, 0). free(NULL) does nothing.

Right.

Quoted 6 lines

So checking for NULL before using nr would probably just be a workaround for programming mistake in the function allocating the object
or some missing initialisation in the caller. So I don't think it is important anymore.

Agreed. That's why, as I said, I though about doing something like that, but ended up deciding not to.

Quoted 15 lines
Quoted 11 lines
Quoted 5 lines

Also I think it is better to keep the free in the exit path at the
end
of the loop.

Can I ask why as, as I was trying to say above, if we are sent directly
to next by one of the goto, we do know that vinfo is NULL and libxl_vcpuinfo_list_free() will be basically a NOP, however it is implemented.

Quoted 3 lines

Of course, you're the maintainer of this code, so I'll do like that if
you ask... I'm just curious. :-)

Quoted 7 lines

Freeing resources should always been attempted, even if that mean to check whether there's something to free before calling the associated free function. Imagine someone adding some code that could fail after the libxl_list_vcpu(), then when that new code fails it would goto next; and the allocated data from libxl_list_vcpu() would never be freed.

Yeah, sure, whoever adds code that does a 'goto next' with an allocated list, should also realize that libxl_vcpuinfo_list_free() needs to be moved after 'next' itself, as a consequence of the very change being done, and this seems fair to me.

But, at the end of the day, it's not a big deal at all. Thanks for satisfying my curiosity and, since you also agree on...

Quoted 10 lines
Quoted 5 lines

Actually, if you really think that the call to libxl_vcpuinfo_list_free() should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the
crash.

This patch is just a workaround a bug in libxl_list_vcpu(), so I think
it can be dropped.

...this, I'll just drop this patch and proceed with just what, in this series, was patch 2.

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)
#6 lore raw

PATCH 2/2 tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL

In reply to Dario Faggioli

If we are in libvxl_list_vcpu() and we are returning NULL, let's avoid touching the output parameter *nr_vcpus_out (which should contain the number of vcpus in the list). Ideally, the caller initialized it to 0, which is therefore consistent with us returning NULL (or, as an alternative, we can explicitly set it to 0 if we're returning null... But just not touching it seems the best behavior).

In fact, the current behavior is especially problematic if, for instance, a domain is destroyed after we have done some steps of the for() loop. In which case, calls like xc_vcpu_getinfo() or xc_vcpu_getaffinity() will start to fail, and we return back to the caller inconsistent information, such as a NULL list of vcpus, but a modified and not 0 any longer, number of vcpus in the list.

---
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Juergen Gross <[email protected]>

---
 tools/libs/light/libxl_domain.c |   14 ++++++++------
 tools/libs/light/libxl_numa.c   |    4 +++-
 2 files changed, 11 insertions(+), 7 deletions(-)
tools/libs/light/libxl_domain.c +8 -6
index 544a9bf59d..aabc264e16 100644
@@ -1661,6 +1661,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid,
     libxl_vcpuinfo *ptr, *ret;
     xc_domaininfo_t domaininfo;
     xc_vcpuinfo_t vcpuinfo;
+    int nr_vcpus;
 
     if (xc_domain_getinfolist(ctx->xch, domid, 1, &domaininfo) != 1) {
         LOGED(ERROR, domid, "Getting infolist");
@@ -1677,27 +1678,27 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid,
     ret = ptr = libxl__calloc(NOGC, domaininfo.max_vcpu_id + 1,
                               sizeof(libxl_vcpuinfo));
 
-    for (*nr_vcpus_out = 0;
-         *nr_vcpus_out <= domaininfo.max_vcpu_id;
-         ++*nr_vcpus_out, ++ptr) {
+    for (nr_vcpus = 0;
+         nr_vcpus <= domaininfo.max_vcpu_id;
+         ++nr_vcpus, ++ptr) {
         libxl_bitmap_init(&ptr->cpumap);
         if (libxl_cpu_bitmap_alloc(ctx, &ptr->cpumap, 0))
             goto err;
         libxl_bitmap_init(&ptr->cpumap_soft);
         if (libxl_cpu_bitmap_alloc(ctx, &ptr->cpumap_soft, 0))
             goto err;
-        if (xc_vcpu_getinfo(ctx->xch, domid, *nr_vcpus_out, &vcpuinfo) == -1) {
+        if (xc_vcpu_getinfo(ctx->xch, domid, nr_vcpus, &vcpuinfo) == -1) {
             LOGED(ERROR, domid, "Getting vcpu info");
             goto err;
         }
 
-        if (xc_vcpu_getaffinity(ctx->xch, domid, *nr_vcpus_out,
+        if (xc_vcpu_getaffinity(ctx->xch, domid, nr_vcpus,
                                 ptr->cpumap.map, ptr->cpumap_soft.map,
                                 XEN_VCPUAFFINITY_SOFT|XEN_VCPUAFFINITY_HARD) == -1) {
             LOGED(ERROR, domid, "Getting vcpu affinity");
             goto err;
         }
-        ptr->vcpuid = *nr_vcpus_out;
+        ptr->vcpuid = nr_vcpus;
         ptr->cpu = vcpuinfo.cpu;
         ptr->online = !!vcpuinfo.online;
         ptr->blocked = !!vcpuinfo.blocked;
@@ -1705,6 +1706,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid,
         ptr->vcpu_time = vcpuinfo.cpu_time;
     }
     GC_FREE;
+    *nr_vcpus_out = nr_vcpus;
     return ret;
 
 err:
tools/libs/light/libxl_numa.c +3 -1
index 3679028c79..b04e3917a0 100644
@@ -219,8 +219,10 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo,
             goto next;
 
         vinfo = libxl_list_vcpu(CTX, dinfo[i].domid, &nr_dom_vcpus, &nr_cpus);
-        if (vinfo == NULL)
+        if (vinfo == NULL) {
+            assert(nr_dom_vcpus == 0);
             goto next;
+        }
 
         /* Retrieve the domain's node-affinity map */
         libxl_domain_get_nodeaffinity(CTX, dinfo[i].domid, &dom_nodemap);
#7 lore raw

Re: [PATCH 2/2] tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL

Quoted 1 line · Dario Faggioli

If we are in libvxl_list_vcpu() and we are returning NULL, let's avoid

extra 'v' ^ here.

Quoted 21 lines

touching the output parameter *nr_vcpus_out (which should contain the number of vcpus in the list). Ideally, the caller initialized it to 0, which is therefore consistent with us returning NULL (or, as an alternative, we can explicitly set it to 0 if we're returning null... But just not touching it seems the best behavior).

In fact, the current behavior is especially problematic if, for instance, a domain is destroyed after we have done some steps of the for() loop. In which case, calls like xc_vcpu_getinfo() or xc_vcpu_getaffinity() will start to fail, and we return back to the caller inconsistent information, such as a NULL list of vcpus, but a modified and not 0 any longer, number of vcpus in the list.

---

tools/libs/light/libxl_domain.c +0 -0
index 544a9bf59d..aabc264e16 100644

tools/libs/light/libxl_domain.c libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid, · line 1709

Quoted 4 lines
tools/libs/light/libxl_domain.c +1 -0
@@ -1705,6 +1706,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid,
         ptr->vcpu_time = vcpuinfo.cpu_time;
     }
     GC_FREE;
+    *nr_vcpus_out = nr_vcpus;

Can you swap those two lines, to keep GC_FREE just before return?

Quoted 8 lines
tools/libs/light/libxl_domain.c +0 -0
     return ret;
 
 err:
tools/libs/light/libxl_numa.c +0 -0
index 3679028c79..b04e3917a0 100644

tools/libs/light/libxl_numa.c static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, · line 223

Quoted 6 lines
tools/libs/light/libxl_numa.c +2 -1
@@ -219,8 +219,10 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo,
             goto next;
 
         vinfo = libxl_list_vcpu(CTX, dinfo[i].domid, &nr_dom_vcpus, &nr_cpus);
-        if (vinfo == NULL)
+        if (vinfo == NULL) {
+            assert(nr_dom_vcpus == 0);

I don't think this assert is necessary.

tools/libs/light/libxl_numa.c static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, · line 225

Quoted 2 lines
tools/libs/light/libxl_numa.c +1 -0
             goto next;
+        }

Otherwise, this patch looks fine.

Thanks,

Signature · 1 line
Anthony PERARD
#8 lore raw
In reply to Dario Faggioli
Quoted 10 lines · Dario Faggioli

It is possible to encounter a segfault in libxl during concurrent domain
create and destroy operations.

This is because Placement of existing domains on the host's CPUs is examined
when creating a new domain, but the existing logic does not tolerate well a
domain disappearing during the examination.

Oh, one more thing. This has been first encountered and reproduced on our SLE15.2, which ships Xen 4.13 so, if the fixes are accepted upstream, I think they should also be backported (to all supported branches, I would say).

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)