epoll topics
Hi,
I think it would be interesting to discuss recent topics that have come up around epoll.
I recently proposed a new locking scheme that removes the global locking using 'connected components', see: https://lkml.org/lkml/2015/1/15/667.
There is a new syscall to batch epoll_ctl() operations and improve the timer precision that Fam Zheng is proposing: https://lkml.org/lkml/2015/1/8/70.
In addition, Eric Wong has proposed a scheme using 'wait free queues' to improve lock contention between wakeups and event gathering:
https://lkml.org/lkml/2013/3/14/14.
In addition, we've observed some short comings from our production systems. Because wait queues are woken up in order, we can get unfair loading across threads, and we can also receive unecessary wakeups. It would be great to discuss these topics further.
Thanks,
-Jason
Quoted 3 lines · Jason Baron
In addition, Eric Wong has proposed a scheme using 'wait free queues' to improve lock contention between wakeups and event gathering:
https://lkml.org/lkml/2013/3/14/14.
Thanks for bringing this up again. wfcqueue for epoll is still on hold until I take care of other problems in userspace. I'd welcome other interested folks to help get it into shape again.
Quoted 3 lines
In addition, we've observed some short comings from our production systems. Because wait queues are woken up in order, we can get unfair loading across threads,
Are you referring to LIFO scheduling of epoll_wait calling __add_wait_queue_exclusive? I always thought LIFO was preferable since it kept the same threads active to reduce cache misses.
Quoted 2 lines
and we can also receive unecessary wakeups. It would be great to discuss these topics further.
I'm happy to discuss via email; but I don't travel well.
Quoted 7 lines · Eric Wong
Quoted 3 lines · Jason Baron
In addition, we've observed some short comings from our production systems. Because wait queues are woken up in order, we can get unfair loading across threads,
Are you referring to LIFO scheduling of
epoll_waitcalling__add_wait_queue_exclusive? I always thought LIFO was preferable since it kept the same threads active to reduce cache misses.
So the specific case I had in mind was where you have an epfd per-thread that is attached to a single listen socket. When a POLLIN occurs on the listen socket, all threads in epoll_wait will be woken up in the order they were added. Then, network traffic ends up being processed on the thread which does the accept(). This tends to result in an unbalanced load across the threads.
I was thinking that if the wait queues rotated the wake up list after each wake up it might help things in this case. Another option is to employ SO_REUSEPORT, so I was wondering if this was an issue pretty specific to network sockets or perhaps more general...
Thanks,
-Jason
Quoted 9 lines · Jason Baron
Quoted 7 lines · Eric Wong
Quoted 3 lines · Jason Baron
In addition, we've observed some short comings from our production systems. Because wait queues are woken up in order, we can get unfair loading across threads,
Are you referring to LIFO scheduling of
epoll_waitcalling__add_wait_queue_exclusive? I always thought LIFO was preferable since it kept the same threads active to reduce cache misses.
Quoted 7 lines
So the specific case I had in mind was where you have an epfd per-thread that is attached to a single listen socket. When a POLLIN occurs on the listen socket, all threads in
epoll_waitwill be woken up in the order they were added. Then, network traffic ends up being processed on the thread which does theaccept(). This tends to result in an unbalanced load across the threads.
Ah, yes. Non-blocking listen sockets is a problem with epoll.
Have you tried using a dedicated thread for blocking accept() and EPOLL_CTL_ADD?
Other threads will work off entries via epoll_wait and do non-blocking I/O like a normal epoll server. I do this for cmogstored:
git clone git://bogomips.org/cmogstored
(see accept_loop.c, works best w/o TCP_DEFER_ACCEPT)
One extra side effect of blocking accept() is it ends up balancing fairly across multiple processes, too, so cmogstored's undocumented multi-process option can distribute FDs more evenly.
Quoted 16 lines · Eric Wong
Quoted 8 lines · Jason Baron
Quoted 7 lines · Eric Wong
Quoted 3 lines · Jason Baron
In addition, we've observed some short comings from our production systems. Because wait queues are woken up in order, we can get unfair loading across threads,
Are you referring to LIFO scheduling of
epoll_waitcalling__add_wait_queue_exclusive? I always thought LIFO was preferable since it kept the same threads active to reduce cache misses.Quoted 7 lines
So the specific case I had in mind was where you have an epfd per-thread that is attached to a single listen socket. When a POLLIN occurs on the listen socket, all threads in
epoll_waitwill be woken up in the order they were added. Then, network traffic ends up being processed on the thread which does theaccept(). This tends to result in an unbalanced load across the threads.
Quoted 4 lines
Ah, yes. Non-blocking listen sockets is a problem with epoll.
Have you tried using a dedicated thread for blocking
accept()andEPOLL_CTL_ADD?
That's a possibility for us - I am also wondering if the idea of rotating the wait queue will work well in practice, since epoll will naturally tend to wake up the thread that is least busy at the moment (since any threads sitting in epoll_wait will tend to process the wakeup first).
Thanks,
-Jason