Skip to content

virtiofsd: multithreading preparation part 3

Stefan Hajnoczi qemu-devel virtio-fs 1 version Applied

Status from this message: Thanks, applied; some comments below.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 25 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/4virtiofsd: process requests in a thread poolcontrib/virtiofsd/fuse_virtio.c contrib/virtiofsd/seccomp.c+273 -2192 replies
2/4virtiofsd: prevent FUSE_INIT/FUSE_DESTROY racescontrib/virtiofsd/fuse_i.h contrib/virtiofsd/fuse_lowlevel.c+18 -01 reply
3/4virtiofsd: fix lo_destroy() resource leakscontrib/virtiofsd/passthrough_ll.c+20 -234 replies
4/4virtiofsd: add --thread-pool-size=NUM optioncontrib/virtiofsd/fuse_i.h contrib/virtiofsd/fuse_lowlevel.c contrib/virtiofsd/fuse_virtio.c+9 -40 replies

Reviews trailers per patch

2 of 4 patches have a Reviewed-by or Acked-by.

PatchDr. David Alan Gilbert
1/4
2/4R-b
3/4R-b
4/4

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [Qemu-devel] [PATCH 0/4] virtiofsd: multithreading preparation part 3

This patch series introduces the virtiofsd --thread-pool-size=NUM and sets the default value to 64. Each virtqueue has its own thread pool for processing requests. Blocking requests no longer pause virtqueue processing and I/O performance should be greatly improved when the queue depth is greater than 1.

Linux boot and pjdfstest have been tested with these patches and the default thread pool size of 64.

I have now concluded the thread-safety code audit. Please let me know if you have concerns about things I missed.

Performance
-----------
Please try these patches out and share your results.

Scalability
-----------
There are several synchronization primitives that are taken by the virtqueue processing thread or the thread pool worker. Unfortunately this is necessary to protect shared state. It means that thread pool workers contend on or at least access thread synchronization primitives. If anyone has suggestions for improving this situation, please discuss.

1. vu_dispatch_rwlock - libvhost-user from races between the vhost-user protocol thread and the virtqueue processing and thread pool worker threads.

2. vq_lock - protects the virtqueue from races between the virtqueue processing thread and thread pool workers.

3. init_rwlock - protects FUSE_INIT/FUSE_DESTROY from races with other requests.

4. se->lock - protects se->list and the FUSE_INTERRUPT shared state.

Ideally we could avoid hitting all of these locks on each request. That would make the code scale better.

Future work
-----------
This series does not complete the multithreading effort yet. Two items are still missing:
1. Multiqueue support
2. CPU affinity for virtqueue processing threads and thread pools

Stefan Hajnoczi (4):
virtiofsd: process requests in a thread pool
virtiofsd: prevent FUSE_INIT/FUSE_DESTROY races
virtiofsd: fix lo_destroy() resource leaks
virtiofsd: add --thread-pool-size=NUM option

 contrib/virtiofsd/fuse_i.h         |   2 +
 contrib/virtiofsd/fuse_lowlevel.c  |  25 +-
 contrib/virtiofsd/fuse_virtio.c    | 491 ++++++++++++++++-------------
 contrib/virtiofsd/passthrough_ll.c |  43 ++-
 contrib/virtiofsd/seccomp.c        |   1 +
 5 files changed, 318 insertions(+), 244 deletions(-)
Signature · 1 line
2.21.0