The thundering herd like problem when multi epolls on one fd

Quoted 10 lines

Hi,

My buddy reported a thundering herd problem about using epoll
on TCP listen sockets. He said their usage like below:

1. sk = new tcp_listen_socket();
2. create many child processes or threads.
3. in new created processes (threads), use epoll API on listen
sk to provide HTTP service.

Quoted 4 lines

Such using pattern means we have multi wait queues when
accepting one socket, and it is not exclusive waking up, so we get a thundering herd like problem. And, so I heard many popular applications can use such pattern, which includes nginx, lighttpd, haproxy at least.

It is not very scalable. But we really lack a fanout mechanism to allow better paralelism on accept(), its not a poll() vs select() vs epoll() problem per se, but a generic problem.

Quoted 2 lines

So should we change this waking up behavior to exclusive too ?

Certainly not.

Quoted 2 lines

Below is a simple patch (tested and works) for epoll() to do it,
of course, we also should fix select() and poll() syscalls if it is

fs/eventpoll.c static void ep_ptable_queue_proc(struct file

Quoted 10 lines

Thanks.

Yu

fs/eventpoll.c +0 -0

index 828e750..a3d6ab4 100644
@@ -898,7 +899,7 @@ static void ep_ptable_queue_proc(struct file

fs/eventpoll.c static void ep_ptable_queue_proc(struct file

Quoted 9 lines

fs/eventpoll.c +1 -2

                init_waitqueue_func_entry(&pwq->wait, ep_poll_callback);
                pwq->whead = whead;
                pwq->base = epi;
-               add_wait_queue(whead, &pwq->wait);
+               add_wait_queue_exclusive(whead, &pwq->wait);
                list_add_tail(&pwq->llink, &epi->pwqlist);
                epi->nwait++;
        } else {
--

What happens if the awaken thread does not consume the event, and prefer to exit ?

If several threads are doing select()/poll()/epoll() on a shared fd, they _all_ must be notified the fd is ready, as manpages claim.

Doing otherwise would require the prior consent of the user, using a special flag for example, and documentation.