Skip to contentSkip to conversation

vl: pause vcpus before stopping iothreads

PATCH Superseded Stefan Hajnoczi 2 participants 5 messages Jan 30, 2018 – Feb 1, 2018

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vl: pause vcpus before stopping iothreads

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") introduced iothread_stop_all() to avoid the following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made d->conf.blk NULL, blk_get_aio_context() returned the global AioContext instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O requests after iothread_stop_all() has moved the BDS to the global AioContext.

This patch hardens the iothread_stop_all() approach by pausing vcpus before calling iothread_stop_all().

Note that the assertion failure is a race condition. It is not possible to reproduce it reliably.

---
 vl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
vl.c +2 -2
index e517a8d995..011f22ae20 100644
@@ -4765,10 +4765,10 @@ int main(int argc, char **argv, char **envp)
     os_setup_post();
 
     main_loop();
+
     replay_disable_events();
+    pause_all_vcpus();
     iothread_stop_all();
-
-    pause_all_vcpus();
     bdrv_close_all();
     res_free();
 
Signature · 1 line
2.14.3
#1 lore raw

Am 30.01.2018 um 16:38 hat Stefan Hajnoczi geschrieben:

Quoted 14 lines

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") introduced iothread_stop_all() to avoid the following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made d->conf.blk NULL, blk_get_aio_context() returned the global AioContext instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O requests after iothread_stop_all() has moved the BDS to the global AioContext.

Quoted 7 lines

This patch hardens the iothread_stop_all() approach by pausing vcpus before calling iothread_stop_all().

Note that the assertion failure is a race condition. It is not possible to reproduce it reliably.

Does pausing the vcpus actually make sure that the iothread isn't active any more, or do we still have a small window where the vcpu is already stopped, but the iothread is still processing requests?

Essentially, I think the bdrv_set_aio_context() in iothread_stop_all() does either not have any effect, or if it does have an effect, it's wrong. You can't just force an in-use BDS into a different AioContext when the user that set the AioContext is still there.

At the very least, do we need a blk_drain_all() before stopping the iothreads? It would still just be a hack, the proper way seens to be getting the virtio device out of dataplane mode so that the iothread is actually unused and doesn't just happen to not process something at the moment.

Kevin

Quoted 23 lines
 vl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
vl.c +2 -2
index e517a8d995..011f22ae20 100644
@@ -4765,10 +4765,10 @@ int main(int argc, char **argv, char **envp)
     os_setup_post();
 
     main_loop();
+
     replay_disable_events();
+    pause_all_vcpus();
     iothread_stop_all();
-
-    pause_all_vcpus();
     bdrv_close_all();
     res_free();
 
Signature · 1 line
2.14.3
#2 lore raw
Quoted 27 lines · Kevin Wolf

Am 30.01.2018 um 16:38 hat Stefan Hajnoczi geschrieben:

Quoted 14 lines

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") introduced iothread_stop_all() to avoid the following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made d->conf.blk NULL, blk_get_aio_context() returned the global AioContext instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O requests after iothread_stop_all() has moved the BDS to the global AioContext.

Quoted 7 lines

This patch hardens the iothread_stop_all() approach by pausing vcpus before calling iothread_stop_all().

Note that the assertion failure is a race condition. It is not possible to reproduce it reliably.

Does pausing the vcpus actually make sure that the iothread isn't active any more, or do we still have a small window where the vcpu is already stopped, but the iothread is still processing requests?

Quoted 7 lines

Essentially, I think the bdrv_set_aio_context() in iothread_stop_all() does either not have any effect, or if it does have an effect, it's wrong. You can't just force an in-use BDS into a different AioContext when the user that set the AioContext is still there.

At the very least, do we need a blk_drain_all() before stopping the iothreads?

bdrv_set_aio_context() contains aio_disable_external() + bdrv_parent_drained_begin() + bdrv_drain(bs). This should complete all requests, even those sitting in a descriptor ring that hasn't been processed yet.

Quoted 4 lines

It would still just be a hack, the proper way seens to be getting the virtio device out of dataplane mode so that the iothread is actually unused and doesn't just happen to not process something at the moment.

Agreed, the existing approach is a hack. I'm not keen on implementing a proper device<->IOThread detach operation because vl.c:main() seems to be the only place that needs it - and it can get away with just quiescing requests and the IOThread instead.

Stefan

Reviewed-by
#3 lore raw

Am 31.01.2018 um 14:56 hat Stefan Hajnoczi geschrieben:

Quoted 33 lines · Kevin Wolf
Quoted 32 lines

Am 30.01.2018 um 16:38 hat Stefan Hajnoczi geschrieben:

Quoted 14 lines

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") introduced iothread_stop_all() to avoid the following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made d->conf.blk NULL, blk_get_aio_context() returned the global AioContext instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O requests after iothread_stop_all() has moved the BDS to the global AioContext.

Quoted 7 lines

This patch hardens the iothread_stop_all() approach by pausing vcpus before calling iothread_stop_all().

Note that the assertion failure is a race condition. It is not possible to reproduce it reliably.

Does pausing the vcpus actually make sure that the iothread isn't active any more, or do we still have a small window where the vcpu is already stopped, but the iothread is still processing requests?

Essentially, I think the bdrv_set_aio_context() in iothread_stop_all() does either not have any effect, or if it does have an effect, it's wrong. You can't just force an in-use BDS into a different AioContext when the user that set the AioContext is still there.

Quoted 7 lines
Quoted 2 lines

At the very least, do we need a blk_drain_all() before stopping the iothreads?

bdrv_set_aio_context() contains aio_disable_external() + bdrv_parent_drained_begin() + bdrv_drain(bs). This should complete all requests, even those sitting in a descriptor ring that hasn't been processed yet.

Ah, yes. Not very obvious, so I wouldn't mind a comment, but you can have my R-b either way then:

Quoted 9 lines
Quoted 4 lines

It would still just be a hack, the proper way seens to be getting the virtio device out of dataplane mode so that the iothread is actually unused and doesn't just happen to not process something at the moment.

Agreed, the existing approach is a hack. I'm not keen on implementing a proper device<->IOThread detach operation because vl.c:main() seems to be the only place that needs it - and it can get away with just quiescing requests and the IOThread instead.

As long as we don't want to switch devices between iothreads at runtime (and create/delete iothreads over QMP), we probably won't really need it, yes.

Kevin

#4 lore raw
Quoted 42 lines · Kevin Wolf

Am 31.01.2018 um 14:56 hat Stefan Hajnoczi geschrieben:

Quoted 33 lines · Kevin Wolf
Quoted 32 lines

Am 30.01.2018 um 16:38 hat Stefan Hajnoczi geschrieben:

Quoted 14 lines

Commit dce8921b2baaf95974af8176406881872067adfa ("iothread: Stop threads before main() quits") introduced iothread_stop_all() to avoid the following virtio-scsi assertion failure:

  assert(blk_get_aio_context(d->conf.blk) == s->ctx);

Back then the assertion failed because when bdrv_close_all() made d->conf.blk NULL, blk_get_aio_context() returned the global AioContext instead of s->ctx.

The same assertion can still fail today when vcpus submit new I/O requests after iothread_stop_all() has moved the BDS to the global AioContext.

Quoted 7 lines

This patch hardens the iothread_stop_all() approach by pausing vcpus before calling iothread_stop_all().

Note that the assertion failure is a race condition. It is not possible to reproduce it reliably.

Does pausing the vcpus actually make sure that the iothread isn't active any more, or do we still have a small window where the vcpu is already stopped, but the iothread is still processing requests?

Essentially, I think the bdrv_set_aio_context() in iothread_stop_all() does either not have any effect, or if it does have an effect, it's wrong. You can't just force an in-use BDS into a different AioContext when the user that set the AioContext is still there.

Quoted 7 lines
Quoted 2 lines

At the very least, do we need a blk_drain_all() before stopping the iothreads?

bdrv_set_aio_context() contains aio_disable_external() + bdrv_parent_drained_begin() + bdrv_drain(bs). This should complete all requests, even those sitting in a descriptor ring that hasn't been processed yet.

Quoted 4 lines

Ah, yes. Not very obvious, so I wouldn't mind a comment, but you can have my R-b either way then:

Thanks for the review! I have sent v2 with a comment for you to review.

Stefan