libxl: fix crash in helper_done due to uninitialized data
PATCH 1/1 libxl: fix crash in helper_done due to uninitialized data
A crash in helper_done, called from libxl_domain_suspend, was reported, triggered by 'virsh migrate --live xen+ssh://host':
#1 helper_done (...) at libxl_save_callout.c:371
helper_failed
helper_stop
libxl__save_helper_abort
#2 check_all_finished (..., rc=-3) at libxl_stream_write.c:671
stream_done
stream_complete
write_done
dc->callback == write_done
efd->func == datacopier_writable
#3 afterpoll_internal (...) at libxl_event.c:1269
This is triggered by a failed poll, the actual error was:
libxl_aoutils.c:328:datacopier_writable: unexpected poll event 0x1c on fd 37 (should be POLLOUT) writing libxc header during copy of save v2 stream
In this case revents in datacopier_writable is POLLHUP|POLLERR|POLLOUT, which triggers datacopier_callback. In helper_done, shs->completion_callback is still zero. libxl__xc_domain_save fills dss.sws.shs. But that function is only called after stream_header_done. Any error before that will leave dss partly uninitialized.
Fix this crash by checking if ->completion_callback is valid.
--- tools/libxl/libxl_save_callout.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/libxl/libxl_save_callout.c +3 -2
@@ -366,8 +366,9 @@ static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs)assert (!libxl__save_helper_inuse (shs)); shs->egc = egc; - shs->completion_callback (egc, shs->caller_state, - shs->rc, shs->retval, shs->errnoval); +if (shs->completion_callback) + shs->completion_callback (egc, shs->caller_state, + shs->rc, shs->retval, shs->errnoval); shs->egc =0 ; }
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Olaf Hering writes ("[PATCH v1] libxl: fix crash in helper_done due to uninitialized data"):
Quoted 2 lines
A crash in
helper_done, called fromlibxl_domain_suspend, was reported, triggered by 'virsh migrate --live xen+ssh://host':
...
Quoted 9 lines
This is triggered by a failed poll, the actual error was:
libxl_aoutils.c:328:datacopier_writable: unexpected poll event
0x1con fd 37 (should be POLLOUT) writing libxc header during copy of save v2 streamIn this case revents in
datacopier_writableis POLLHUP|POLLERR|POLLOUT, which triggersdatacopier_callback. Inhelper_done,shs->completion_callbackis still zero.libxl__xc_domain_savefills dss.sws.shs. But that function is only called afterstream_header_done. Any error before that will leave dss partly uninitialized.
Thanks for the diagnosis. And sorry for the inconvenience of this bug.
Quoted 1 line
Fix this crash by checking if ->completion_callback is valid.
But I don't think this fix is right. The bug is that libxl__save_helper_abort is called on a blank shs. (You say "uninitialised" but actually this is all zeroed at some point, so it it's not reading uninitialised memory.)
I think it is in fact a bug that this error path
if (!stream->rc && rc) {
/* First reported failure. Tear everything down. */
stream->rc = rc;
stream->sync_teardown = true;
libxl__stream_read_abort(egc, stream, rc);
libxl__save_helper_abort(egc, &stream->shs);
libxl__conversion_helper_abort(egc, &stream->chs, rc);
stream->sync_teardown = false;
}calls libxl__save_helper_abort when it hasn't ever called anything other than libxl__save_helper_init. Because _abort naturally wants to call the failure callback.
I suggest that we add a check for _inuse to this bit of check_all_finished.
Ross and Andrew, you wrote much of this stream read stuff, what do you think ?
Part of the difficulty is that the possible states and transitions of a libxl__save_helper_state are not formatlly documented. That's my fault - sorry.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Am Fri, 27 Sep 2019 18:17:12 +0100
schrieb Ian Jackson <[email protected]>:
Quoted 4 lines
Olaf Hering writes ("[PATCH v1] libxl: fix crash in
helper_donedue to uninitialized data"):Quoted 2 lines
A crash in
helper_done, called fromlibxl_domain_suspend, was reported,
libxl_aoutils.c:328:datacopier_writable: unexpected poll event0x1con fd 37 (should be POLLOUT) writing libxc header during copy of save v2 streamRoss and Andrew, you wrote much of this stream read stuff, what do you think ?
Did you have a chance to look at this issue?
Olaf
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel