virtiofsd: Used glib "shared" thread pool
PATCH 1/1 virtiofsd: Used glib "shared" thread pool
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
--- tools/virtiofsd/fuse_virtio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/virtiofsd/fuse_virtio.c +1 -1
@@ -695,7 +695,7 @@ static void *fv_queue_thread(void *opaqustruct fuse_session *se = qi->virtio_dev->se;GThreadPool *pool; - pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, TRUE, + pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, FALSE,NULL );if (!pool) {fuse_log (FUSE_LOG_ERR ,"%s: g_thread_pool_new failed\n" , __func__);
Quoted 10 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
Quoted 7 lines
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
Needs this as well:
--- qemu.orig/tools/virtiofsd/passthrough_seccomp.c 2020-09-16 20:21:13.168686176 +0200
+++ qemu/tools/virtiofsd/passthrough_seccomp.c 2020-09-22 14:01:38.499164501 +0200
@@ -94,6 +94,8 @@ static const int syscall_whitelist[] = {
SCMP_SYS(rt_sigaction), SCMP_SYS(rt_sigprocmask), SCMP_SYS(rt_sigreturn),
+ SCMP_SYS(sched_getattr),
+ SCMP_SYS(sched_setattr),
SCMP_SYS(sendmsg),
SCMP_SYS(setresgid),
SCMP_SYS(setresuid),
Thanks,
Miklos
Quoted 14 lines · Miklos Szeredi
Quoted 12 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
Quoted 7 lines
Quoted 5 lines
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
Needs this as well:
Thanks Miklos. I am wondering why I did not face this issue. May be I don't have seccomp enabled. Will check.
David, can you please pick this patch as well. May be merge both the patches.
Do let me know if you want to me post a merged patch instead.
Thanks
Vivek
Quoted 17 lines
---
qemu.orig/tools/virtiofsd/passthrough_seccomp.c2020-09-16 20:21:13.168686176 +0200
+++qemu/tools/virtiofsd/passthrough_seccomp.c2020-09-22 14:01:38.499164501 +0200(quoted diff) +2
-0@@ -94,6 +94,8 @@ static const int syscall_whitelist[] = { SCMP_SYS(rt_sigaction), SCMP_SYS(rt_sigprocmask), SCMP_SYS(rt_sigreturn), + SCMP_SYS(sched_getattr), + SCMP_SYS(sched_setattr), SCMP_SYS(sendmsg), SCMP_SYS(setresgid), SCMP_SYS(setresuid),Thanks,
Miklos
Quoted 14 lines · Miklos Szeredi
Quoted 12 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
Quoted 7 lines
Quoted 5 lines
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
Needs this as well:
I was wondering why I did not face this issue. I think my glib is old (glib2, 2.58.3) from fedora 29 and this change about sched_getattr seems relatively recent.
commit 8aeca4fa647bfd0f35c4a86b1e6ca6e955519ca5
Author: Sebastian Dröge <[email protected]> Date: Tue Dec 24 15:33:30 2019 +0200
GThreadPool - Don't inherit thread priorities when creating new threads
I think my glib2 does not have this change and probably that's why seccomp did not trigger.
Vivek
Quoted 17 lines
---
qemu.orig/tools/virtiofsd/passthrough_seccomp.c2020-09-16 20:21:13.168686176 +0200
+++qemu/tools/virtiofsd/passthrough_seccomp.c2020-09-22 14:01:38.499164501 +0200(quoted diff) +2
-0@@ -94,6 +94,8 @@ static const int syscall_whitelist[] = { SCMP_SYS(rt_sigaction), SCMP_SYS(rt_sigprocmask), SCMP_SYS(rt_sigreturn), + SCMP_SYS(sched_getattr), + SCMP_SYS(sched_setattr), SCMP_SYS(sendmsg), SCMP_SYS(setresgid), SCMP_SYS(setresuid),Thanks,
Miklos
Quoted 9 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
Quoted 6 lines
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
I'm really curious why there's any perf difference between shared and exclusive thread pools in the GLib impl.
Looking at the code the main difference between the two is appears to be around the way threads are spawned, specifically around the scheduler attributes assigned.
In the shared case, the threads in the pool will have their scheduler attributes copied from the very first thread that calls g_thread_pool_new.
In the exclusive case, the threads in the pool will inherit their scheduler attributes from the thread which pushs the job that causes the worker thread to be created.
By schedular attributes, I mean all the items in the 'struct schedattr' filled by sched_getattr()
IOW, if threads in virtiofsd have varying schedular attributes this could possibly explain the difference in performance you see between the two setups.
Quoted 13 lines
Signed-off-by Vivek Goyal <[email protected]>--- tools/virtiofsd/fuse_virtio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Index:
qemu/tools/virtiofsd/fuse_virtio.c
===================================================================
tools/virtiofsd/fuse_virtio.c static void *fv_queue_thread(void *opaqu · line 701
Quoted 8 lines
tools/virtiofsd/fuse_virtio.c +1
-1@@ -695,7 +695,7 @@ static void *fv_queue_thread(void *opaqustruct fuse_session *se = qi->virtio_dev->se;GThreadPool *pool; - pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, TRUE, + pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, FALSE,NULL );if (!pool) {fuse_log (FUSE_LOG_ERR ,"%s: g_thread_pool_new failed\n" , __func__);
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 31 lines · Daniel P. Berrangé
Quoted 9 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
Quoted 6 lines
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
I'm really curious why there's any perf difference between shared and exclusive thread pools in the GLib impl.
Looking at the code the main difference between the two is appears to be around the way threads are spawned, specifically around the scheduler attributes assigned.
In the shared case, the threads in the pool will have their scheduler attributes copied from the very first thread that calls
g_thread_pool_new.In the exclusive case, the threads in the pool will inherit their scheduler attributes from the thread which pushs the job that causes the worker thread to be created.
Quoted 6 lines
By schedular attributes, I mean all the items in the 'struct schedattr' filled by
sched_getattr()IOW, if threads in virtiofsd have varying schedular attributes this could possibly explain the difference in performance you see between the two setups.
Hi Daniel,
Few things.
- I think scheduler attributes are same for the thread creating pool as well as for thread pushing the job for virtiofsd.
- My glib2 is old (2.58.3) and I think that did not have sched_getattr() stuff.
- One difference I noticed is that in case of shared pool, it does not create extra threads if client is doing one request at a time. While exclusive pool seemed to push every request to a new thread in pool in sort of round robin fashion. It feels keeping requests being served from same thread helps in this particilar workload case.
Thanks
Vivek
Quoted 29 lines
Quoted 14 lines
Signed-off-by Vivek Goyal <[email protected]>--- tools/virtiofsd/fuse_virtio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Index:
qemu/tools/virtiofsd/fuse_virtio.c
===================================================================tools/virtiofsd/fuse_virtio.c static void *fv_queue_thread(void *opaqu · line 701
Quoted 8 lines
tools/virtiofsd/fuse_virtio.c +1
-1@@ -695,7 +695,7 @@ static void *fv_queue_thread(void *opaqustruct fuse_session *se = qi->virtio_dev->se;GThreadPool *pool; - pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, TRUE, + pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, FALSE,NULL );if (!pool) {fuse_log (FUSE_LOG_ERR ,"%s: g_thread_pool_new failed\n" , __func__);Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 46 lines · Vivek Goyal
Quoted 31 lines · Daniel P. Berrangé
Quoted 9 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
Quoted 6 lines
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
I'm really curious why there's any perf difference between shared and exclusive thread pools in the GLib impl.
Looking at the code the main difference between the two is appears to be around the way threads are spawned, specifically around the scheduler attributes assigned.
In the shared case, the threads in the pool will have their scheduler attributes copied from the very first thread that calls
g_thread_pool_new.In the exclusive case, the threads in the pool will inherit their scheduler attributes from the thread which pushs the job that causes the worker thread to be created.
Quoted 6 lines
By schedular attributes, I mean all the items in the 'struct schedattr' filled by
sched_getattr()IOW, if threads in virtiofsd have varying schedular attributes this could possibly explain the difference in performance you see between the two setups.
Hi Daniel,
Few things.
- I think scheduler attributes are same for the thread creating pool as well as for thread pushing the job for virtiofsd.
Quoted 8 lines
- My glib2 is old (2.58.3) and I think that did not have
sched_getattr()stuff.- One difference I noticed is that in case of shared pool, it does not create extra threads if client is doing one request at a time. While exclusive pool seemed to push every request to a new thread in pool in sort of round robin fashion. It feels keeping requests being served from same thread helps in this particilar workload case.
Yeah, that does sound like a candidate for the cause. I wonder if that was intentional in the GLib design or just an accidental impl they didn't realize had performance implications. Might be worth filing a bug against GLib if someone has free time & motivation to figure out a standalone reproducer to demonstrate the performance difference in the GLib APIs.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 26 lines · Vivek Goyal
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
Signed-off-by Vivek Goyal <[email protected]>--- tools/virtiofsd/fuse_virtio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Index:
qemu/tools/virtiofsd/fuse_virtio.c
===================================================================
tools/virtiofsd/fuse_virtio.c static void *fv_queue_thread(void *opaqu · line 701
Quoted 8 lines
tools/virtiofsd/fuse_virtio.c +1
-1@@ -695,7 +695,7 @@ static void *fv_queue_thread(void *opaqustruct fuse_session *se = qi->virtio_dev->se;GThreadPool *pool; - pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, TRUE, + pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, FALSE,NULL );if (!pool) {fuse_log (FUSE_LOG_ERR ,"%s: g_thread_pool_new failed\n" , __func__);
Quoted 9 lines · * Vivek Goyal ([email protected]) wrote:
glib offers thread pools and it seems to support "exclusive" and "shared" thread pools.
https://developer.gnome.org/glib/stable/glib-Thread-Pools.html#g-thread-pool-new
Currently we use "exlusive" thread pools but its performance seems to be poor. I tried using "shared" thread pools and performance seems much better. I posted performance results here.
Quoted 8 lines
https://www.redhat.com/archives/virtio-fs/2020-September/msg00080.html
So lets switch to shared thread pools. We can think of making it optional once somebody can show in what cases exclusive thread pools offer better results. For now, my simple performance tests across the board see better results with shared thread pools.
Signed-off-by Vivek Goyal <[email protected]>
Queued (with Miklos's seccomp fix); although my gut feeling is we'll be coming back to this again to understand how the threading should work.
Dave
Quoted 17 lines
--- tools/virtiofsd/fuse_virtio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Index:
qemu/tools/virtiofsd/fuse_virtio.c
===================================================================tools/virtiofsd/fuse_virtio.c +1
-1@@ -695,7 +695,7 @@ static void *fv_queue_thread(void *opaqustruct fuse_session *se = qi->virtio_dev->se;GThreadPool *pool; - pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, TRUE, + pool =g_thread_pool_new (fv_queue_worker, qi, se->thread_pool_size, FALSE,NULL );if (!pool) {fuse_log (FUSE_LOG_ERR ,"%s: g_thread_pool_new failed\n" , __func__);
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK