virtiofsd: cancel all queue threads on exit in virtio_loop()
PATCH v2 1/1 virtiofsd: cancel all queue threads on exit in virtio_loop()
On guest graceful shutdown, virtiofsd receives VHOST_USER_GET_VRING_BASE request from VMM and shuts down virtqueues by calling fv_set_started(), which joins fv_queue_thread() threads. So when virtio_loop() returns, there should be no thread is still accessing data in fuse session and/or virtio dev.
But on abnormal exit, e.g. guest got killed for whatever reason, vhost-user socket is closed and virtio_loop() breaks out the main loop and returns to main(). But it's possible fv_queue_worker()s are still working and accessing fuse session and virtio dev, which results in crash or use-after-free.
Fix it by cancelling fv_queue_thread()s before virtio_loop() returns, to make sure there's no-one could access fuse session and virtio dev.
---
v1: virtiofsd: sync FUSE_DESTROY with session destroy
https://www.redhat.com/archives/virtio-fs/2019-December/msg00051.html
tools/virtiofsd/fuse_virtio.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
tools/virtiofsd/fuse_virtio.c +35 -1
@@ -671,6 +671,11 @@ out:free (req); } +static void worker_cleanup (void *data) +{ +g_thread_pool_free ((GThreadPool *)data, FALSE, TRUE); +} +/* Thread function for individual queues, created when a queue is 'started' */ static void *fv_queue_thread (void *opaque) {
@@ -687,6 +692,8 @@ static void *fv_queue_thread(void *opaque)return NULL ; } +pthread_cleanup_push (worker_cleanup, pool); +fuse_log (FUSE_LOG_INFO ,"%s: Start for queue %d kick_fd %d\n" , __func__, qi->qidx, qi->kick_fd);while (1 ) {
@@ -770,7 +777,7 @@ static void *fv_queue_thread(void *opaque)pthread_rwlock_unlock (&qi->virtio_dev->vu_dispatch_rwlock); } -g_thread_pool_free (pool, FALSE, TRUE); +pthread_cleanup_pop (1 );return NULL ; }
@@ -916,6 +923,33 @@ int virtio_loop(struct fuse_session *se) } } +/* +* Make sure all fv_queue_thread()s quit on exit, as we're about to +* free virtio dev and fuse session, no one should access them anymore. +*/ +for (int i =0 ; i < se->virtio_dev->nqueues; i++) { +int ret; +pthread_t tid; + +if (!se->virtio_dev->qi[i]) +continue ; + + tid = se->virtio_dev->qi[i]->thread; +fuse_log (FUSE_LOG_INFO ,"%s: Canceling queue %d thread (%lu)\n" , + __func__, i, tid); + ret =pthread_cancel (tid); +if (ret) { +fuse_log (FUSE_LOG_WARNING ,"%s: Cancel queue %d thread: %s\n" , + __func__, i,strerror (ret)); + }else { + ret =pthread_join (tid,NULL ); +if (ret) { +fuse_log (FUSE_LOG_WARNING ,"%s: Join queue %d thread: %s\n" , + __func__, i,strerror (ret)); + } + } + } +fuse_log (FUSE_LOG_INFO ,"%s: Exit\n" , __func__);return 0 ;
Signature · 1 line
2.14.4.44.g2045bb6
Quoted 6 lines · Eryu Guan
On guest graceful shutdown, virtiofsd receives
VHOST_USER_GET_VRING_BASErequest from VMM and shuts down virtqueues by callingfv_set_started(), which joinsfv_queue_thread()threads. So whenvirtio_loop()returns, there should be no thread is still accessing data in fuse session and/or virtio dev.
Quoted 8 lines
But on abnormal exit, e.g. guest got killed for whatever reason, vhost-user socket is closed and
virtio_loop()breaks out the main loop and returns tomain(). But it's possible fv_queue_worker()s are still working and accessing fuse session and virtio dev, which results in crash or use-after-free.Fix it by cancelling fv_queue_thread()s before
virtio_loop()returns, to make sure there's no-one could access fuse session and virtio dev.
Why use pthread_cancel(3) instead of writing 1 to the thread's kill_fd?
I'm concerned about pthread_cancel(3) because the threads access rwlocks, which may be leaked when pthread_cancel(3) terminates the thread.
It would be safer to reuse the following code to get full thread cleanup:
static void fv_queue_set_started(VuDev *dev, int qidx, bool started)
{
...
int ret;
assert(qidx < vud->nqueues);
ourqi = vud->qi[qidx]; /* Kill the thread */
if (eventfd_write(ourqi->kill_fd, 1)) {
fuse_log(FUSE_LOG_ERR, "Eventfd_read for queue: %m\n");
}
ret = pthread_join(ourqi->thread, NULL);
if (ret) {
fuse_log(FUSE_LOG_ERR, "%s: Failed to join thread idx %d err %d\n",
__func__, qidx, ret);
}
pthread_mutex_destroy(&ourqi->vq_lock);
close(ourqi->kill_fd);
ourqi->kick_fd = -1;
free(vud->qi[qidx]);
vud->qi[qidx] = NULL;
...}
Quoted 16 lines · Stefan Hajnoczi
Quoted 6 lines · Eryu Guan
On guest graceful shutdown, virtiofsd receives
VHOST_USER_GET_VRING_BASErequest from VMM and shuts down virtqueues by callingfv_set_started(), which joinsfv_queue_thread()threads. So whenvirtio_loop()returns, there should be no thread is still accessing data in fuse session and/or virtio dev.Quoted 8 lines
But on abnormal exit, e.g. guest got killed for whatever reason, vhost-user socket is closed and
virtio_loop()breaks out the main loop and returns tomain(). But it's possible fv_queue_worker()s are still working and accessing fuse session and virtio dev, which results in crash or use-after-free.Fix it by cancelling fv_queue_thread()s before
virtio_loop()returns, to make sure there's no-one could access fuse session and virtio dev.
Quoted 8 lines
Why use
pthread_cancel(3)instead of writing 1 to the thread'skill_fd?I'm concerned about
pthread_cancel(3)because the threads access rwlocks, which may be leaked whenpthread_cancel(3)terminates the thread.It would be safer to reuse the following code to get full thread cleanup:
Makes sense, fixed in v3.
Thanks,
Eryu
Quoted 24 lines
static void fv_queue_set_started(VuDev *dev, int qidx, bool started) { ... int ret; assert(qidx < vud->nqueues); ourqi = vud->qi[qidx];/* Kill the thread */ if (eventfd_write(ourqi->kill_fd, 1)) { fuse_log(FUSE_LOG_ERR, "Eventfd_read for queue: %m\n"); } ret = pthread_join(ourqi->thread, NULL); if (ret) { fuse_log(FUSE_LOG_ERR, "%s: Failed to join thread idx %d err %d\n", __func__, qidx, ret); } pthread_mutex_destroy(&ourqi->vq_lock); close(ourqi->kill_fd); ourqi->kick_fd = -1; free(vud->qi[qidx]); vud->qi[qidx] = NULL; ...}