The thundering herd like problem when multi epolls on one fd
Hi,
My buddy reported a thundering herd problem about using epoll
on TCP listen sockets. He said their usage like below:
1. sk = new tcp_listen_socket();
2. create many child processes or threads.
3. in new created processes (threads), use epoll API on listen
sk to provide HTTP service.
Such using pattern means we have multi wait queues when
accepting one socket, and it is not exclusive waking up, so we get a thundering herd like problem. And, so I heard many popular applications can use such pattern, which includes nginx, lighttpd, haproxy at least. So should we change this waking up behavior to exclusive too ?
Below is a simple patch (tested and works) for epoll() to do it,
of course, we also should fix select() and poll() syscalls if it is right.
Thanks.
Yu
fs/eventpoll.c +1 -1
@@ -898,7 +899,7 @@ static void ep_ptable_queue_proc(struct file *file, wait_queue_head_t *whead,init_waitqueue_func_entry (&pwq->wait, ep_poll_callback); pwq->whead = whead; pwq->base = epi; -add_wait_queue (whead, &pwq->wait); +add_wait_queue_exclusive (whead, &pwq->wait);list_add_tail (&pwq->llink, &epi->pwqlist); epi->nwait++; }else {
Quoted 10 lines · Li Yu
Hi,
My buddy reported a thundering herd problem about using epoll
on TCP listen sockets. He said their usage like below:1. sk = new
tcp_listen_socket();
2. create many child processes or threads.
3. in new created processes (threads), use epoll API on listen
sk to provide HTTP service.
Quoted 4 lines
Such using pattern means we have multi wait queues when
accepting one socket, and it is not exclusive waking up, so we get a thundering herd like problem. And, so I heard many popular applications can use such pattern, which includes nginx, lighttpd, haproxy at least.
It is not very scalable. But we really lack a fanout mechanism to allow better paralelism on accept(), its not a poll() vs select() vs epoll() problem per se, but a generic problem.
Quoted 2 lines
So should we change this waking up behavior to exclusive too ?
Certainly not.
Quoted 12 lines
Below is a simple patch (tested and works) for
epoll()to do it,
of course, we also should fixselect()andpoll()syscalls if it is right.Thanks.
Yu
fs/eventpoll.c +0
-0
fs/eventpoll.c static void ep_ptable_queue_proc(struct file *file, wait_queue_head_t *whead,
Quoted 9 lines
fs/eventpoll.c +1
-2@@ -898,7 +899,7 @@ static void ep_ptable_queue_proc(struct file *file, wait_queue_head_t *whead,init_waitqueue_func_entry (&pwq->wait, ep_poll_callback); pwq->whead = whead; pwq->base = epi; -add_wait_queue (whead, &pwq->wait); +add_wait_queue_exclusive (whead, &pwq->wait);list_add_tail (&pwq->llink, &epi->pwqlist); epi->nwait++; }else { --
What happens if the awaken thread does not consume the event, and prefer to exit ?
If several threads are doing select()/poll()/epoll() on a shared fd, they _all_ must be notified the fd is ready, as manpages claim.
Doing otherwise would require the prior consent of the user, using a special flag for example, and documentation.
Quoted 6 lines · * Eric Dumazet | 2012-01-14 14:20:47 [+0100]:
What happens if the awaken thread does not consume the event, and prefer to exit ?
If several threads are doing select()/poll()/epoll() on a shared fd, they _all_ must be notified the fd is ready, as manpages claim.
And if it is intended? Signaled via a new epoll_ctl() epoll_event flag? E.g. EPOLLEXCLUSIVE
Quoted 2 lines
Doing otherwise would require the prior consent of the user, using a special flag for example, and documentation.
Oops, yes. A fanout like mechanism _may be_ superior, but I think tests will show some improvements anyway.
Hagen