libs/light: fix a race when domain are destroied and created concurrently
cover libs/light: fix a race when domain are destroied and created concurrently
It is possible to encounter a segfault in libxl during concurrent domain create and destroy operations.
This is because Placement of existing domains on the host's CPUs is examined when creating a new domain, but the existing logic does not tolerate well a domain disappearing during the examination.
The race is quite difficult to trigger. Whan that happens, this is an example of a backtrace:
#0 libxl_bitmap_dispose (map=map@entry=0x50) at libxl_utils.c:626 #1 0x00007fe72c993a32 in libxl_vcpuinfo_dispose (p=p@entry=0x38) at _libxl_types.c:692 #2 0x00007fe72c94e3c4 in libxl_vcpuinfo_list_free (list=0x0, nr=<optimized out>) at libxl_utils.c:1059 #3 0x00007fe72c9528bf in nr_vcpus_on_nodes (vcpus_on_node=0x7fe71000eb60, suitable_cpumap=0x7fe721df0d38, tinfo_elements=48, tinfo=0x7fe7101b3900, gc=0x7fe7101bbfa0) at libxl_numa.c:258 #4 libxl__get_numa_candidate (gc=gc@entry=0x7fe7100033a0, min_free_memkb=4233216, min_cpus=4, min_nodes=min_nodes@entry=0, max_nodes=max_nodes@entry=0, suitable_cpumap=suitable_cpumap@entry=0x7fe721df0d38, numa_cmpf=0x7fe72c940110 <numa_cmpf>, cndt_out=0x7fe721df0cf0, cndt_found=0x7fe721df0cb4) at libxl_numa.c:394 #5 0x00007fe72c94152b in numa_place_domain (d_config=0x7fe721df11b0, domid=975, gc=0x7fe7100033a0) at libxl_dom.c:209 #6 libxl__build_pre (gc=gc@entry=0x7fe7100033a0, domid=domid@entry=975, d_config=d_config@entry=0x7fe721df11b0, state=state@entry=0x7fe710077700) at libxl_dom.c:436 #7 0x00007fe72c92c4a5 in libxl__domain_build (gc=0x7fe7100033a0, d_config=d_config@entry=0x7fe721df11b0, domid=975, state=0x7fe710077700) at libxl_create.c:444 #8 0x00007fe72c92de8b in domcreate_bootloader_done (egc=0x7fe721df0f60, bl=0x7fe7100778c0, rc=<optimized out>) at libxl_create.c:1222 #9 0x00007fe72c980425 in libxl__bootloader_run (egc=egc@entry=0x7fe721df0f60, bl=bl@entry=0x7fe7100778c0) at libxl_bootloader.c:403
#10 0x00007fe72c92f281 in initiate_domain_create (egc=egc@entry=0x7fe721df0f60, dcs=dcs@entry=0x7fe7100771b0) at libxl_create.c:1159
#11 0x00007fe72c92f456 in do_domain_create (ctx=ctx@entry=0x7fe71001c840, d_config=d_config@entry=0x7fe721df11b0, domid=domid@entry=0x7fe721df10a8, restore_fd=restore_fd@entry=-1, send_back_fd=send_back_fd@entry=-1, params=params@entry=0x0, ao_how=0x0, aop_console_how=0x7fe721df10f0) at libxl_create.c:1856 #12 0x00007fe72c92f776 in libxl_domain_create_new (ctx=0x7fe71001c840, d_config=d_config@entry=0x7fe721df11b0, domid=domid@entry=0x7fe721df10a8, ao_how=ao_how@entry=0x0, aop_console_how=aop_console_how@entry=0x7fe721df10f0) at libxl_create.c:2075
Luckily, it is easy to close the race, by just making sure that libvxl_list_vcpu() returns 0 as the number of vCPUs of the domain, when it also returns NULL as the list of them.
Regards
---
Dario Faggioli (2):
tools/libs/light: numa placement: don't try to free a NULL list of vcpus tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL
tools/libs/light/libxl_domain.c | 14 ++++++++------ tools/libs/light/libxl_numa.c | 4 +++- 2 files changed, 11 insertions(+), 7 deletions(-)
--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)
PATCH 1/2 tools/libs/light: numa placement: don't try to free a NULL list of vcpus
If libxl_vcpu_list() returned NULL, we should not call libxl_numainfo_list_free() as:
1) it'll fail trying to (double) free() *list
2) there should be nothing to free anyway
---
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Juergen Gross <[email protected]>
--- tools/libs/light/libxl_numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libs/light/libxl_numa.c +1 -1
@@ -253,9 +253,9 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, } } +libxl_vcpuinfo_list_free (vinfo, nr_dom_vcpus); next:libxl_cpupoolinfo_dispose (&cpupool_info); -libxl_vcpuinfo_list_free (vinfo, nr_dom_vcpus); }libxl_bitmap_dispose (&dom_nodemap);
Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus
Quoted 2 lines · Dario Faggioli
If
libxl_vcpu_list()returned NULL, we should not calllibxl_numainfo_list_free()as:
You mean libxl_vcpuinfo_list_free() ?
Quoted 1 line
1) it'll fail trying to (double)
free()*list
This isn't really an issue. free(NULL) is legit, can be call as many time as you want.
Quoted 1 line
2) there should be nothing to free anyway
The issue here is that it doesn't appear to be true. Even if "info" is NULL, "nr" have an other value than 0, so libxl_vcpuinfo_list_free() will try to access random addresses.
Quoted 2 lines
Signed-off-by Dario Faggioli <[email protected]>Tested-by James Fehlig <[email protected]>
Can I suggest to make libxl_vcpuinfo_list_free() work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL. Even if that seems wrong, and the caller should use the correct value.
Also I think it is better to keep the free in the exit path at the end of the loop.
Thanks,
Signature · 1 line
Anthony PERARD
Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus
Quoted 9 lines · Anthony PERARD
Quoted 1 line · Dario Faggioli
2) there should be nothing to free anyway
The issue here is that it doesn't appear to be true. Even if "info" is
NULL, "nr" have an other value than 0, solibxl_vcpuinfo_list_free()will try to access random addresses.
My point here was that if vinfo is NULL (because libxl_list_vcpu() returned NULL), there aren't any list element or any list to free, so we can avoid calling libxl_vcpuinfo_list_free().
Then, sure, if we call it with a NULL vinfo and a non-zero nr_dom_vcpus, things explode, but this was being addressed in patch 2.
This first one was really only about not trying to free an empty list. And not to workaround the fact that it currently can make things explode, just because it feels pointless.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Dario Faggioli <[email protected]>Tested-by James Fehlig <[email protected]>Can I suggest to make
libxl_vcpuinfo_list_free()work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL.
I thought about that, and we certainly can do it.
However, I think it's unrelated to this patch so, if I do it, I'll do it in its own one.
Also, if we go that way, I guess we want to change libxl_cputopology_list_free(), libxl_pcitopology_list_free(), libxl_numainfo_list_free(), libxl_dominfo_list_free(), libxl_vminfo_list_free() and libxl_cpupoolinfo_list_free(), don't we?
Quoted 4 lines
Also I think it is better to keep the free in the exit path at the end
of the loop.
Can I ask why as, as I was trying to say above, if we are sent directly to next by one of the goto, we do know that vinfo is NULL and libxl_vcpuinfo_list_free() will be basically a NOP, however it is implemented.
Of course, you're the maintainer of this code, so I'll do like that if you ask... I'm just curious. :-)
Actually, if you really think that the call to libxl_vcpuinfo_list_free() should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the crash.
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus
Quoted 30 lines · Dario Faggioli
Quoted 9 lines · Anthony PERARD
Quoted 1 line · Dario Faggioli
2) there should be nothing to free anyway
The issue here is that it doesn't appear to be true. Even if "info" is
NULL, "nr" have an other value than 0, solibxl_vcpuinfo_list_free()will try to access random addresses.My point here was that if vinfo is NULL (because
libxl_list_vcpu()returned NULL), there aren't any list element or any list to free, so we can avoid callinglibxl_vcpuinfo_list_free().Then, sure, if we call it with a NULL vinfo and a non-zero
nr_dom_vcpus, things explode, but this was being addressed in patch 2.This first one was really only about not trying to free an empty list. And not to workaround the fact that it currently can make things explode, just because it feels pointless.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Dario Faggioli <[email protected]>Tested-by James Fehlig <[email protected]>Can I suggest to make
libxl_vcpuinfo_list_free()work a bit better in case it's "nr" parameter is wrong? It will do nothing if "list" is NULL.I thought about that, and we certainly can do it.
Quoted 7 lines
However, I think it's unrelated to this patch so, if I do it, I'll do it in its own one.
Also, if we go that way, I guess we want to change
libxl_cputopology_list_free(),libxl_pcitopology_list_free(),libxl_numainfo_list_free(),libxl_dominfo_list_free(),libxl_vminfo_list_free()andlibxl_cpupoolinfo_list_free(), don't we?
I actually don't know if that would be useful. Those functions do work as expected (I think) with the right parameters: they do nothing when called with (NULL, 0). free(NULL) does nothing.
So checking for NULL before using nr would probably just be a workaround for programming mistake in the function allocating the object or some missing initialisation in the caller. So I don't think it is important anymore.
Quoted 11 lines
Quoted 4 lines
Also I think it is better to keep the free in the exit path at the end
of the loop.Can I ask why as, as I was trying to say above, if we are sent directly to next by one of the goto, we do know that vinfo is NULL and
libxl_vcpuinfo_list_free()will be basically a NOP, however it is implemented.Of course, you're the maintainer of this code, so I'll do like that if you ask... I'm just curious. :-)
Freeing resources should always been attempted, even if that mean to check whether there's something to free before calling the associated free function. Imagine someone adding some code that could fail after the libxl_list_vcpu(), then when that new code fails it would goto next; and the allocated data from libxl_list_vcpu() would never be freed.
Quoted 4 lines
Actually, if you really think that the call to
libxl_vcpuinfo_list_free()should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the crash.
This patch is just a workaround a bug in libxl_list_vcpu(), so I think it can be dropped.
Cheers,
Signature · 1 line
Anthony PERARD
Re: [PATCH 1/2] tools/libs/light: numa placement: don't try to free a NULL list of vcpus
Quoted 8 lines · Anthony PERARD
Quoted 6 lines · Dario Faggioli
Also, if we go that way, I guess we want to change
libxl_cputopology_list_free(),libxl_pcitopology_list_free(),libxl_numainfo_list_free(),libxl_dominfo_list_free(),libxl_vminfo_list_free()andlibxl_cpupoolinfo_list_free(), don't we?
Quoted 5 lines
I actually don't know if that would be useful. Those functions do work
as expected (I think) with the right parameters: they do nothing when called with (NULL, 0). free(NULL) does nothing.
Right.
Quoted 6 lines
So checking for NULL before using
nrwould probably just be a workaround for programming mistake in the function allocating the object
or some missing initialisation in the caller. So I don't think it is important anymore.
Agreed. That's why, as I said, I though about doing something like that, but ended up deciding not to.
Quoted 15 lines
Quoted 11 lines
Quoted 5 lines
Also I think it is better to keep the free in the exit path at the
end
of the loop.Can I ask why as, as I was trying to say above, if we are sent directly
to next by one of the goto, we do know that vinfo is NULL andlibxl_vcpuinfo_list_free()will be basically a NOP, however it is implemented.Quoted 3 lines
Of course, you're the maintainer of this code, so I'll do like that if
you ask... I'm just curious. :-)
Quoted 7 lines
Freeing resources should always been attempted, even if that mean to check whether there's something to free before calling the associated free function. Imagine someone adding some code that could fail after the
libxl_list_vcpu(), then when that new code fails it wouldgoto next;and the allocated data fromlibxl_list_vcpu()would never be freed.
Yeah, sure, whoever adds code that does a 'goto next' with an allocated list, should also realize that libxl_vcpuinfo_list_free() needs to be moved after 'next' itself, as a consequence of the very change being done, and this seems fair to me.
But, at the end of the day, it's not a big deal at all. Thanks for satisfying my curiosity and, since you also agree on...
Quoted 10 lines
Quoted 5 lines
Actually, if you really think that the call to
libxl_vcpuinfo_list_free()should stay where it is, I can just drop this patch and go on with patch 2 only, which is enough for fixing the
crash.This patch is just a workaround a bug in
libxl_list_vcpu(), so I think
it can be dropped.
...this, I'll just drop this patch and proceed with just what, in this series, was patch 2.
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
PATCH 2/2 tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL
In reply to Dario FaggioliIf we are in libvxl_list_vcpu() and we are returning NULL, let's avoid touching the output parameter *nr_vcpus_out (which should contain the number of vcpus in the list). Ideally, the caller initialized it to 0, which is therefore consistent with us returning NULL (or, as an alternative, we can explicitly set it to 0 if we're returning null... But just not touching it seems the best behavior).
In fact, the current behavior is especially problematic if, for instance, a domain is destroyed after we have done some steps of the for() loop. In which case, calls like xc_vcpu_getinfo() or xc_vcpu_getaffinity() will start to fail, and we return back to the caller inconsistent information, such as a NULL list of vcpus, but a modified and not 0 any longer, number of vcpus in the list.
---
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Juergen Gross <[email protected]>
--- tools/libs/light/libxl_domain.c | 14 ++++++++------ tools/libs/light/libxl_numa.c | 4 +++- 2 files changed, 11 insertions(+), 7 deletions(-)
tools/libs/light/libxl_domain.c +8 -6
@@ -1661,6 +1661,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid,libxl_vcpuinfo *ptr, *ret;xc_domaininfo_t domaininfo;xc_vcpuinfo_t vcpuinfo; +int nr_vcpus;if (xc_domain_getinfolist (ctx->xch, domid,1 , &domaininfo) !=1 ) {LOGED (ERROR , domid,"Getting infolist" );
@@ -1677,27 +1678,27 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid, ret = ptr =libxl__calloc (NOGC , domaininfo.max_vcpu_id +1 ,sizeof (libxl_vcpuinfo)); -for (*nr_vcpus_out =0 ; - *nr_vcpus_out <= domaininfo.max_vcpu_id; - ++*nr_vcpus_out, ++ptr) { +for (nr_vcpus =0 ; + nr_vcpus <= domaininfo.max_vcpu_id; + ++nr_vcpus, ++ptr) {libxl_bitmap_init (&ptr->cpumap);if (libxl_cpu_bitmap_alloc (ctx, &ptr->cpumap,0 )) goto err;libxl_bitmap_init (&ptr->cpumap_soft);if (libxl_cpu_bitmap_alloc (ctx, &ptr->cpumap_soft,0 )) goto err; -if (xc_vcpu_getinfo (ctx->xch, domid, *nr_vcpus_out, &vcpuinfo) ==-1 ) { +if (xc_vcpu_getinfo (ctx->xch, domid, nr_vcpus, &vcpuinfo) ==-1 ) {LOGED (ERROR , domid,"Getting vcpu info" ); goto err; } -if (xc_vcpu_getaffinity (ctx->xch, domid, *nr_vcpus_out, +if (xc_vcpu_getaffinity (ctx->xch, domid, nr_vcpus, ptr->cpumap.map, ptr->cpumap_soft.map,XEN_VCPUAFFINITY_SOFT |XEN_VCPUAFFINITY_HARD ) ==-1 ) {LOGED (ERROR , domid,"Getting vcpu affinity" ); goto err; } - ptr->vcpuid = *nr_vcpus_out; + ptr->vcpuid = nr_vcpus; ptr->cpu = vcpuinfo.cpu; ptr->online = !!vcpuinfo.online; ptr->blocked = !!vcpuinfo.blocked;
@@ -1705,6 +1706,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid, ptr->vcpu_time = vcpuinfo.cpu_time; }GC_FREE ; + *nr_vcpus_out = nr_vcpus;return ret; err:
tools/libs/light/libxl_numa.c +3 -1
@@ -219,8 +219,10 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, goto next; vinfo =libxl_list_vcpu (CTX , dinfo[i].domid, &nr_dom_vcpus, &nr_cpus); -if (vinfo ==NULL ) +if (vinfo ==NULL ) { +assert (nr_dom_vcpus ==0 ); goto next; + }/* Retrieve the domain's node-affinity map */ libxl_domain_get_nodeaffinity (CTX , dinfo[i].domid, &dom_nodemap);
Re: [PATCH 2/2] tools/libs/light: don't touch nr_vcpus_out if listing vcpus and returning NULL
Quoted 1 line · Dario Faggioli
If we are in
libvxl_list_vcpu()and we are returning NULL, let's avoid
extra 'v' ^ here.
Quoted 21 lines
touching the output parameter
*nr_vcpus_out(which should contain the number of vcpus in the list). Ideally, the caller initialized it to 0, which is therefore consistent with us returning NULL (or, as an alternative, we can explicitly set it to 0 if we're returning null... But just not touching it seems the best behavior).In fact, the current behavior is especially problematic if, for instance, a domain is destroyed after we have done some steps of the
for()loop. In which case, calls likexc_vcpu_getinfo()orxc_vcpu_getaffinity()will start to fail, and we return back to the caller inconsistent information, such as a NULL list of vcpus, but a modified and not 0 any longer, number of vcpus in the list.Signed-off-by Dario Faggioli <[email protected]>Tested-by James Fehlig <[email protected]>---
tools/libs/light/libxl_domain.c +0
-0
tools/libs/light/libxl_domain.c libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid, · line 1709
Quoted 4 lines
tools/libs/light/libxl_domain.c +1
-0@@ -1705,6 +1706,7 @@ libxl_vcpuinfo *libxl_list_vcpu(libxl_ctx *ctx, uint32_t domid, ptr->vcpu_time = vcpuinfo.cpu_time; }GC_FREE ; + *nr_vcpus_out = nr_vcpus;
Can you swap those two lines, to keep GC_FREE just before return?
Quoted 8 lines
tools/libs/light/libxl_domain.c +0
-0return ret; err:tools/libs/light/libxl_numa.c +0
-0
tools/libs/light/libxl_numa.c static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, · line 223
Quoted 6 lines
tools/libs/light/libxl_numa.c +2
-1@@ -219,8 +219,10 @@ static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, goto next; vinfo =libxl_list_vcpu (CTX , dinfo[i].domid, &nr_dom_vcpus, &nr_cpus); -if (vinfo ==NULL ) +if (vinfo ==NULL ) { +assert (nr_dom_vcpus ==0 );
I don't think this assert is necessary.
tools/libs/light/libxl_numa.c static int nr_vcpus_on_nodes(libxl__gc *gc, libxl_cputopology *tinfo, · line 225
Quoted 2 lines
tools/libs/light/libxl_numa.c +1
-0goto next; + }
Otherwise, this patch looks fine.
Thanks,
Signature · 1 line
Anthony PERARD
Quoted 10 lines · Dario Faggioli
It is possible to encounter a segfault in libxl during concurrent domain
create and destroy operations.This is because Placement of existing domains on the host's CPUs is examined
when creating a new domain, but the existing logic does not tolerate well a
domain disappearing during the examination.
Oh, one more thing. This has been first encountered and reproduced on our SLE15.2, which ships Xen 4.13 so, if the fixes are accepted upstream, I think they should also be backported (to all supported branches, I would say).
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)