xen: Fix SEGV on domain disconnect
PATCH RESEND v2 1/1 xen: Fix SEGV on domain disconnect
From: Mark Syms <[email protected]>
Ensure the PV ring is drained on disconnect. Also ensure all pending AIO is complete, otherwise AIO tries to complete into a mapping of the ring which has been torn down.
---
CC: Stefano Stabellini <[email protected]>
CC: Anthony Perard <[email protected]>
CC: Paul Durrant <[email protected]>
CC: [email protected]
v2:
* Ensure all inflight requests are completed before teardown
* RESEND to fix formatting
--- hw/block/dataplane/xen-block.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)
hw/block/dataplane/xen-block.c +25 -6
@@ -523,6 +523,10 @@ static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) dataplane->more_work =0 ; +if (dataplane->sring ==0 ) { +return done_something; + } + rc = dataplane->rings.common.req_cons; rp = dataplane->rings.common.sring->req_prod;xen_rmb ();/* Ensure we see queued requests up to 'rp'. */
@@ -666,14 +670,35 @@ void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane)void xen_block_dataplane_stop (XenBlockDataPlane *dataplane) {XenDevice *xendev; +XenBlockRequest *request, *next;if (!dataplane) {return ; } +/* We're about to drain the ring. We can cancel the scheduling of any +* bottom half now */ +qemu_bh_cancel (dataplane->bh); + +/* Ensure we have drained the ring */ +aio_context_acquire (dataplane->ctx); +do { +xen_block_handle_requests (dataplane); + }while (dataplane->more_work); +aio_context_release (dataplane->ctx); + +/* Now ensure that all inflight requests are complete */ +while (!QLIST_EMPTY (&dataplane->inflight)) { + QLIST_FOREACH_SAFE(request, &dataplane->inflight, list, next) { +blk_aio_flush (request->dataplane->blk, xen_block_complete_aio, + request); + } + } + xendev = dataplane->xendev;aio_context_acquire (dataplane->ctx); +if (dataplane->event_channel) {/* Only reason for failure is a NULL channel */ xen_device_set_event_channel_context(xendev, dataplane->event_channel,
@@ -684,12 +709,6 @@ void xen_block_dataplane_stop(XenBlockDataPlane *dataplane)blk_set_aio_context (dataplane->blk,qemu_get_aio_context (), &error_abort);aio_context_release (dataplane->ctx); -/* -* Now that the context has been moved onto the main thread, cancel -* further processing. -*/ -qemu_bh_cancel (dataplane->bh); -if (dataplane->event_channel) {Error *local_err =NULL ;
Signature · 1 line
2.40.0
Quoted 25 lines · On 20/04/2023 12:02, [email protected] wrote:
From: Mark Syms <[email protected]>
Ensure the PV ring is drained on disconnect. Also ensure all pending AIO is complete, otherwise AIO tries to complete into a mapping of the ring which has been torn down.
Signed-off-by Mark Syms <[email protected]>---
CC: Stefano Stabellini <[email protected]>
CC: Anthony Perard <[email protected]>
CC: Paul Durrant <[email protected]>
CC: [email protected]v2:
* Ensure all inflight requests are completed before teardown
* RESEND to fix formatting--- hw/block/dataplane/xen-block.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)hw/block/dataplane/xen-block.c +0
-0
hw/block/dataplane/xen-block.c static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) · line 529
Quoted 7 lines
hw/block/dataplane/xen-block.c +4
-0@@ -523,6 +523,10 @@ static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) dataplane->more_work =0 ; +if (dataplane->sring ==0 ) { +return done_something; + } +
I think you could just return false here... Nothing is ever going to be done if there's no ring :-)
Quoted 13 lines
hw/block/dataplane/xen-block.c +2
-0rc = dataplane->rings.common.req_cons; rp = dataplane->rings.common.sring->req_prod;xen_rmb ();/* Ensure we see queued requests up to 'rp'. */ @@ -666,14 +670,35 @@ void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) {XenDevice *xendev; +XenBlockRequest *request, *next;if (!dataplane) {return ; } +/* We're about to drain the ring. We can cancel the scheduling of any
hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) · line 688
Quoted 10 lines
hw/block/dataplane/xen-block.c +10
-0+* bottom half now */ +qemu_bh_cancel (dataplane->bh); + +/* Ensure we have drained the ring */ +aio_context_acquire (dataplane->ctx); +do { +xen_block_handle_requests (dataplane); + }while (dataplane->more_work); +aio_context_release (dataplane->ctx); +
I don't think we want to be taking new requests, do we?
hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane) · line 697
Quoted 8 lines
hw/block/dataplane/xen-block.c +8
-0+/* Now ensure that all inflight requests are complete */ +while (!QLIST_EMPTY (&dataplane->inflight)) { + QLIST_FOREACH_SAFE(request, &dataplane->inflight, list, next) { +blk_aio_flush (request->dataplane->blk, xen_block_complete_aio, + request); + } + } +
I think this could possibly be simplified by doing the drain after the call to blk_set_aio_context(), as long as we set dataplane->ctx to qemu_get_aio_context(). Alos, as long as more_work is not set then it should still be safe to cancel the bh before the drain AFAICT.
Paul
Quoted 20 lines
hw/block/dataplane/xen-block.c +1
-6xendev = dataplane->xendev;aio_context_acquire (dataplane->ctx); +if (dataplane->event_channel) {/* Only reason for failure is a NULL channel */ xen_device_set_event_channel_context(xendev, dataplane->event_channel,@@ -684,12 +709,6 @@ void xen_block_dataplane_stop(XenBlockDataPlane *dataplane)blk_set_aio_context (dataplane->blk,qemu_get_aio_context (), &error_abort);aio_context_release (dataplane->ctx); -/* -* Now that the context has been moved onto the main thread, cancel -* further processing. -*/ -qemu_bh_cancel (dataplane->bh); -if (dataplane->event_channel) {Error *local_err =NULL ;
Copying in Tim who did the final phase of the changes.
Quoted 15 lines · Paul Durrant
Quoted 13 lines · On 20/04/2023 12:02, [email protected] wrote:
From: Mark Syms <[email protected]>
Ensure the PV ring is drained on disconnect. Also ensure all pending AIO is complete, otherwise AIO tries to complete into a mapping of the ring which has been torn down.
Signed-off-by Mark Syms <[email protected]>---
CC: Stefano Stabellini <[email protected]>
CC: Anthony Perard <[email protected]>
CC: Paul Durrant <[email protected]>
CC: [email protected]
Quoted 8 lines
Review
Quoted 8 lines
v2:
* Ensure all inflight requests are completed before teardown
* RESEND to fix formatting--- hw/block/dataplane/xen-block.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)(quoted diff) +0
-0
b/hw/block/dataplane/xen-block.c
Quoted 4 lines
hw/block/dataplane/xen-block.c static bool
Quoted 4 lines
index
734da42ea7..d9da4090bf100644hw/block/dataplane/xen-block.c +0
-0@@ -523,6 +523,10 @@ static bool
xen_block_handle_requests(XenBlockDataPlane *dataplane)
Quoted 8 lines
hw/block/dataplane/xen-block.c static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) · line 529
Quoted 7 lines
hw/block/dataplane/xen-block.c +4
-0dataplane->more_work =0 ; +if (dataplane->sring ==0 ) { +return done_something; + } +
Quoted 7 lines
I think you could just return false here... Nothing is ever going to be done if there's no ring :-)
hw/block/dataplane/xen-block.c void
Quoted 4 lines
hw/block/dataplane/xen-block.c +0
-0rc = dataplane->rings.common.req_cons; rp = dataplane->rings.common.sring->req_prod;xen_rmb ();/* Ensure we see queued requests up to 'rp'. */ @@ -666,14 +670,35 @@ void
xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > voidxen_block_dataplane_stop(XenBlockDataPlane *dataplane)
Quoted 9 lines
hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane)
Quoted 9 lines
hw/block/dataplane/xen-block.c +2
-0{XenDevice *xendev; +XenBlockRequest *request, *next;if (!dataplane) {return ; } +/* We're about to drain the ring. We can cancel the scheduling of
any
Quoted 4 lines
Quoted 4 lines
hw/block/dataplane/xen-block.c +4
-0+* bottom half now */ +qemu_bh_cancel (dataplane->bh); + +/* Ensure we have drained the ring */
Quoted 13 lines
hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane) · line 689
Quoted 6 lines
hw/block/dataplane/xen-block.c +6
-0+aio_context_acquire (dataplane->ctx); +do { +xen_block_handle_requests (dataplane); + }while (dataplane->more_work); +aio_context_release (dataplane->ctx); +I don't think we want to be taking new requests, do we?
Quoted 4 lines
hw/block/dataplane/xen-block.c +4
-0+/* Now ensure that all inflight requests are complete */ +while (!QLIST_EMPTY (&dataplane->inflight)) { + QLIST_FOREACH_SAFE(request, &dataplane->inflight, list, next) { + blk_aio_flush(request->dataplane->blk,
xen_block_complete_aio,
Quoted 5 lines
hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane) · line 697
Quoted 4 lines
hw/block/dataplane/xen-block.c +4
-0+ request); + } + } +
Quoted 14 lines
I think this could possibly be simplified by doing the drain after the call to
blk_set_aio_context(), as long as we setdataplane->ctxtoqemu_get_aio_context(). Alos, as long asmore_workis not set then it should still be safe to cancel the bh before the drain AFAICT.Paul
Quoted 7 lines
hw/block/dataplane/xen-block.c +1
-0xendev = dataplane->xendev;aio_context_acquire (dataplane->ctx); +if (dataplane->event_channel) {/* Only reason for failure is a NULL channel */ xen_device_set_event_channel_context(xendev,
dataplane->event_channel,
Quoted 1 line
hw/block/dataplane/xen-block.c void xen_block_dataplane_stop(XenBlockDataPlane
Quoted 1 line
hw/block/dataplane/xen-block.c +0
-0@@ -684,12 +709,6 @@ void xen_block_dataplane_stop(XenBlockDataPlane
*dataplane)
Quoted 1 line
hw/block/dataplane/xen-block.c void xen_block_dataplane_stop(XenBlockDataPlane · line 709
Quoted 1 line
hw/block/dataplane/xen-block.c +0
-0blk_set_aio_context(dataplane->blk, qemu_get_aio_context(),
&error_abort);
Quoted 12 lines
Quoted 11 lines
hw/block/dataplane/xen-block.c +0
-6aio_context_release (dataplane->ctx); -/* -* Now that the context has been moved onto the main thread, cancel -* further processing. -*/ -qemu_bh_cancel (dataplane->bh); -if (dataplane->event_channel) {Error *local_err =NULL ;
Quoted 68 lines · Mark Syms
Copying in Tim who did the final phase of the changes.
Quoted 64 lines · Paul Durrant
Quoted 25 lines · On 20/04/2023 12:02, [email protected] wrote:
From: Mark Syms <[email protected]>
Ensure the PV ring is drained on disconnect. Also ensure all pending AIO is complete, otherwise AIO tries to complete into a mapping of the ring which has been torn down.
Signed-off-by Mark Syms <[email protected]>---
CC: Stefano Stabellini <[email protected]>
CC: Anthony Perard <[email protected]>
CC: Paul Durrant <[email protected]>
CC: [email protected]v2:
* Ensure all inflight requests are completed before teardown
* RESEND to fix formatting--- hw/block/dataplane/xen-block.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)hw/block/dataplane/xen-block.c +0
-0hw/block/dataplane/xen-block.c static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) · line 529
Quoted 7 lines
hw/block/dataplane/xen-block.c +4
-0@@ -523,6 +523,10 @@ static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) dataplane->more_work =0 ; +if (dataplane->sring ==0 ) { +return done_something; + } +I think you could just return false here... Nothing is ever going to be done if there's no ring :-)
Quoted 13 lines
hw/block/dataplane/xen-block.c +2
-0rc = dataplane->rings.common.req_cons; rp = dataplane->rings.common.sring->req_prod;xen_rmb ();/* Ensure we see queued requests up to 'rp'. */ @@ -666,14 +670,35 @@ void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) {XenDevice *xendev; +XenBlockRequest *request, *next;if (!dataplane) {return ; } +/* We're about to drain the ring. We can cancel the scheduling of any hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) · line 688
Quoted 10 lines
hw/block/dataplane/xen-block.c +10
-0+* bottom half now */ +qemu_bh_cancel (dataplane->bh); + +/* Ensure we have drained the ring */ +aio_context_acquire (dataplane->ctx); +do { +xen_block_handle_requests (dataplane); + }while (dataplane->more_work); +aio_context_release (dataplane->ctx); +I don't think we want to be taking new requests, do we?
If we're in this situation and the guest has put something on the ring, I think we should do our best with it.
We cannot just rely on the guest to be well-behaved, because they're not :-( We're about to throw the
ring away, so whatever is there would otherwise be lost. This bit is here to try to handle guests which are
less than diligent about their shutdown. We *should* always be past this fast enough when the disconnect()/connect()
of XenbusStateConnected happens that all remains well (if not, we were in a worse situation before).
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
hw/block/dataplane/xen-block.c +8
-0+/* Now ensure that all inflight requests are complete */ +while (!QLIST_EMPTY (&dataplane->inflight)) { + QLIST_FOREACH_SAFE(request, &dataplane->inflight, list, next) { +blk_aio_flush (request->dataplane->blk, xen_block_complete_aio, + request); + } + } +
Quoted 4 lines
Quoted 4 lines
I think this could possibly be simplified by doing the drain after the call to
blk_set_aio_context(), as long as we setdataplane->ctxtoqemu_get_aio_context(). Alos, as long asmore_workis not set then it should still be safe to cancel the bh before the drain AFAICT.
I'm not sure what you mean by simpler? Possibly I'm not getting something.
We have to make sure that any "aio_bh_schedule_oneshot_full()" which happens as a result of
"blk_aio_flush()" has finished before any change of AIO context, because it tries to use the one which
was current at the time of being called (I have the SEGVs to prove it :-)). Whether that happens before or after "blk_set_aio_context(qemu_get_aio_context())" doesn't seem to be a change in complexity to me.
Motivation was to get as much as possible to happen in the way it "normally" would, so that future changes
are less likely to regress, but as mentioned maybe I'm missing something.
The BH needs to be prevented from firing ASAP, otherwise the disconnect()/connect() which happens when XenbusStateConnected can have the bh fire from what the guest does next right in the middle of juggling
contexts for the disconnect() (I have the SEGVs from that too...).
Quoted 13 lines
Quoted 13 lines
Paul
Quoted 11 lines
hw/block/dataplane/xen-block.c +1
-0xendev = dataplane->xendev;aio_context_acquire (dataplane->ctx); +if (dataplane->event_channel) {/* Only reason for failure is a NULL channel */ xen_device_set_event_channel_context(xendev, dataplane->event_channel,@@ -684,12 +709,6 @@ void xen_block_dataplane_stop(XenBlockDataPlane *dataplane)blk_set_aio_context (dataplane->blk,qemu_get_aio_context (), &error_abort);aio_context_release (dataplane->ctx);
Quoted 10 lines
Quoted 10 lines
hw/block/dataplane/xen-block.c void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) · line 714
Quoted 9 lines
hw/block/dataplane/xen-block.c +0
-6-/* -* Now that the context has been moved onto the main thread, cancel -* further processing. -*/ -qemu_bh_cancel (dataplane->bh); -if (dataplane->event_channel) {Error *local_err =NULL ;
Tim (hoping GMail behaves itself with this message...)
Quoted 70 lines · Tim Smith
Quoted 68 lines · Mark Syms
Copying in Tim who did the final phase of the changes.
Quoted 64 lines · Paul Durrant
Quoted 25 lines · On 20/04/2023 12:02, [email protected] wrote:
From: Mark Syms <[email protected]>
Ensure the PV ring is drained on disconnect. Also ensure all pending AIO is complete, otherwise AIO tries to complete into a mapping of the ring which has been torn down.
Signed-off-by Mark Syms <[email protected]>---
CC: Stefano Stabellini <[email protected]>
CC: Anthony Perard <[email protected]>
CC: Paul Durrant <[email protected]>
CC: [email protected]v2:
* Ensure all inflight requests are completed before teardown
* RESEND to fix formatting--- hw/block/dataplane/xen-block.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)hw/block/dataplane/xen-block.c +0
-0hw/block/dataplane/xen-block.c static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) · line 529
Quoted 7 lines
hw/block/dataplane/xen-block.c +4
-0@@ -523,6 +523,10 @@ static bool xen_block_handle_requests(XenBlockDataPlane *dataplane) dataplane->more_work =0 ; +if (dataplane->sring ==0 ) { +return done_something; + } +I think you could just return false here... Nothing is ever going to be done if there's no ring :-)
Quoted 13 lines
hw/block/dataplane/xen-block.c +2
-0rc = dataplane->rings.common.req_cons; rp = dataplane->rings.common.sring->req_prod;xen_rmb ();/* Ensure we see queued requests up to 'rp'. */ @@ -666,14 +670,35 @@ void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) {XenDevice *xendev; +XenBlockRequest *request, *next;if (!dataplane) {return ; } +/* We're about to drain the ring. We can cancel the scheduling of any hw/block/dataplane/xen-block.c void xen_block_dataplane_destroy(XenBlockDataPlane *dataplane > void xen_block_dataplane_stop(XenBlockDataPlane *dataplane) · line 688
Quoted 10 lines
hw/block/dataplane/xen-block.c +10
-0+* bottom half now */ +qemu_bh_cancel (dataplane->bh); + +/* Ensure we have drained the ring */ +aio_context_acquire (dataplane->ctx); +do { +xen_block_handle_requests (dataplane); + }while (dataplane->more_work); +aio_context_release (dataplane->ctx); +I don't think we want to be taking new requests, do we?
Quoted 5 lines
If we're in this situation and the guest has put something on the ring, I think we should do our best with it.
We cannot just rely on the guest to be well-behaved, because they're not :-( We're about to throw the
ring away, so whatever is there would otherwise be lost.
We only throw away our mapping. The memory belongs to the guest and it should ensure it does not submit requests after the state has left 'connected'
Quoted 7 lines
This bit is
here to try to handle guests which are
less than diligent about their shutdown. We *should* always be past this fast enough when the disconnect()/connect()
of XenbusStateConnected happens that all remains well (if not, we were in a worse situation before).
What about a malicious guest that is piling requests into the ring. It could keep us in the loop forever, couldn't it?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
hw/block/dataplane/xen-block.c +8
-0+/* Now ensure that all inflight requests are complete */ +while (!QLIST_EMPTY (&dataplane->inflight)) { + QLIST_FOREACH_SAFE(request, &dataplane->inflight, list, next) { +blk_aio_flush (request->dataplane->blk, xen_block_complete_aio, + request); + } + } +
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
I think this could possibly be simplified by doing the drain after the call to
blk_set_aio_context(), as long as we setdataplane->ctxtoqemu_get_aio_context(). Alos, as long asmore_workis not set then it should still be safe to cancel the bh before the drain AFAICT.I'm not sure what you mean by simpler? Possibly I'm not getting something.
Sorry, I was referring to the need to do aio_context_acquire() calls but they are only around the disputed xen_block_handle_requests() call anyway, so there's no simplification in this bit.
Quoted 6 lines
We have to make sure that any "
aio_bh_schedule_oneshot_full()" which happens as a result of
"blk_aio_flush()" has finished before any change of AIO context, because it tries to use the one which
was current at the time of being called (I have the SEGVs to prove it :-)).
Ok, I had assumed that the issue was the context being picked up inside the xen_block_complete_aio() call.
Quoted 8 lines
Whether that happens before or after "blk_set_aio_context(qemu_get_aio_context())" doesn't seem to be a change in complexity to me.
Motivation was to get as much as possible to happen in the way it "normally" would, so that future changes
are less likely to regress, but as mentioned maybe I'm missing something.
Quoted 6 lines
The BH needs to be prevented from firing ASAP, otherwise the disconnect()/connect() which happens when XenbusStateConnected can have the bh fire from what the guest does next right in the middle of juggling
contexts for thedisconnect()(I have the SEGVs from that too...).
So if you drop the ring drain then this patch should still stop the SEGVs, right?
Paul
Quoted 4 lines · Paul Durrant
So if you drop the ring drain then this patch should still stop the SEGVs, right?
I think that's worth a few test runs. I recall some coredumps in that condition when I was investigating early on, but I don't have them in my collection so maybe I'm misremembering.
Tim