From: Eric Wong Subject: Re: [LSF/MM TOPIC] epoll topics Date: Tue, 20 Jan 2015 19:58:12 +0000 Message-ID: <20150120195812.GA5549@dcvr.yhbt.net> References: <54B88EC5.9020904@akamai.com> <20150119035048.GA4533@dcvr.yhbt.net> <54BEA4A9.3020104@akamai.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "lsf-pc@lists.linux-foundation.org" , "linux-fsdevel@vger.kernel.org" , "famz@redhat.com" To: Jason Baron Return-path: Received: from dcvr.yhbt.net ([64.71.152.64]:51756 "EHLO dcvr.yhbt.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751844AbbATT6M (ORCPT ); Tue, 20 Jan 2015 14:58:12 -0500 Content-Disposition: inline In-Reply-To: <54BEA4A9.3020104@akamai.com> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: Jason Baron wrote: > On 01/18/2015 10:50 PM, Eric Wong wrote: > > Jason Baron wrote: > >> In addition, we've observed some short comings from our production systems. > >> Because wait queues are woken up in order, we can get unfair loading > >> across threads, > > Are you referring to LIFO scheduling of epoll_wait calling > > __add_wait_queue_exclusive? I always thought LIFO was preferable > > since it kept the same threads active to reduce cache misses. > > So the specific case I had in mind was where you have an epfd > per-thread that is attached to a single listen socket. When a > POLLIN occurs on the listen socket, all threads in epoll_wait will > be woken up in the order they were added. Then, network > traffic ends up being processed on the thread which does the > accept(). This tends to result in an unbalanced load across the > threads. Ah, yes. Non-blocking listen sockets is a problem with epoll. Have you tried using a dedicated thread for blocking accept() and EPOLL_CTL_ADD? Other threads will work off entries via epoll_wait and do non-blocking I/O like a normal epoll server. I do this for cmogstored: git clone git://bogomips.org/cmogstored (see accept_loop.c, works best w/o TCP_DEFER_ACCEPT) One extra side effect of blocking accept() is it ends up balancing fairly across multiple processes, too, so cmogstored's undocumented multi-process option can distribute FDs more evenly.