accept()ing socket connections with level triggered epoll
Hello,
I think I may have found a bug in Linux's implementation of epoll. My program creates a server socket that listens for incoming SOCK_STREAM connections. It uses epoll to wait for notification of a new connection (and also to handle the client sockets). While the client sockets use edge triggered epoll, for performance reasons, the server socket uses level triggered epoll.
I have found that when I open connections to my program very quickly, it is sometimes possible to call accept more than once before reaching the point where no more connections are available and EAGAIN is returned. If I return to epoll_wait without accepting all of the available connections, I should immediately be notified that a read is still available on the server socket, since I am using level triggered epoll for that descriptor (at least that is my understanding of how all of this is supposed to work ;). However, epoll does not make this notification. Even if the program accepts further incoming connections, the missed connection is never accepted, and eventually times out on the client side.
Kernel version is 2.6.9. I can provide test code if needed.
Thanks,
Kyle Cronan
<[email protected]>
Quoted 1 line · On Wed, Apr 26, 2006 at 02:55:57PM -0600, [email protected] wrote:
Kernel version is 2.6.9. I can provide test code if needed.
Does it happen with the latest kernel? And if so, please provide a test code.
Signature · 3 lines
Heikki Orsila Barbie's law: [email protected] "Math is hard, let's go shopping!" http://www.iki.fi/shd
Quoted 18 lines · On Wed, 26 Apr 2006, [email protected] wrote:
I think I may have found a bug in Linux's implementation of epoll. My program creates a server socket that listens for incoming
SOCK_STREAMconnections. It uses epoll to wait for notification of a new connection (and also to handle the client sockets). While the client sockets use edge triggered epoll, for performance reasons, the server socket uses level triggered epoll.I have found that when I open connections to my program very quickly, it is sometimes possible to call accept more than once before reaching the point where no more connections are available and EAGAIN is returned. If I return to
epoll_waitwithout accepting all of the available connections, I should immediately be notified that a read is still available on the server socket, since I am using level triggered epoll for that descriptor (at least that is my understanding of how all of this is supposed to work ;). However, epoll does not make this notification. Even if the program accepts further incoming connections, the missed connection is never accepted, and eventually times out on the client side.
Quoted 1 line
Kernel version is 2.6.9. I can provide test code if needed.
Correct, if it's LT you have to get the event because before returning from epoll_wait(), the event is automatically re-armed if f_op->poll() returns it. Can you post the *minimal* test code for this case?
- Davide
Quoted 7 lines · Davide Libenzi
Correct, if it's LT you have to get the event because before returning from
epoll_wait(), the event is automatically re-armed iff_op->poll()returns it. Can you post the *minimal* test code for this case?- Davide
I tried reducing the code I have to the minimum necessary to demonstrate the problem. It went away, I'm afraid. Since I'm already aware of a workaround (call accept in a loop until you get EAGAIN), I guess I'll just forget about it. Unfortunately I can't post the full code, not that you'd want to dig through all of it anyway.
Thanks to everyone that responded.
Kyle
<[email protected]>
Quoted 6 lines · On Wed, Apr 26, 2006 at 06:05:20PM -0600, [email protected] wrote:
Quoted 5 lines · Davide Libenzi
Correct, if it's LT you have to get the event because before returning from
epoll_wait(), the event is automatically re-armed iff_op->poll()returns it. Can you post the *minimal* test code for this case?
Quoted 8 lines
Quoted 2 lines
- Davide
I tried reducing the code I have to the minimum necessary to demonstrate the problem. It went away, I'm afraid. Since I'm already aware of a workaround (call accept in a loop until you get EAGAIN), I guess I'll just forget about it. Unfortunately I can't post the full code, not that you'd want to dig through all of it anyway.
Sorry, I misunderstood you the first time. I thought that it was *when* your accept looped that you encountered the problem. If you need performance, I *really* encourage you to loop on accept() as much as you can. Missing an accept() and reading EAGAIN is cheap, while looping through all your event loop is usually more expensive. In haproxy, I had performance problems 5 years ago, I could not get above 1500-2000 sessions/s because I was doing one accept at a time. After putting a small "while" loop around, it immediately jumped over 10000.
Regards,
Willy
Hello,
Quoted 9 lines · On Wed, Apr 26, 2006 at 02:55:57PM -0600, [email protected] wrote:
Hello,
I think I may have found a bug in Linux's implementation of epoll. My program creates a server socket that listens for incoming
SOCK_STREAMconnections. It uses epoll to wait for notification of a new connection (and also to handle the client sockets). While the client sockets use edge triggered epoll, for performance reasons, the server socket uses level triggered epoll.
Quoted 10 lines
I have found that when I open connections to my program very quickly, it is sometimes possible to call accept more than once before reaching the point where no more connections are available and EAGAIN is returned. If I return to
epoll_waitwithout accepting all of the available connections, I should immediately be notified that a read is still available on the server socket, since I am using level triggered epoll for that descriptor (at least that is my understanding of how all of this is supposed to work ;). However, epoll does not make this notification. Even if the program accepts further incoming connections, the missed connection is never accepted, and eventually times out on the client side.
I find this very strange because if your program accepts other connections, I don't see how it could "select" some connections and ignore others. The accept() call returns the next connection(s) in the listen queue. Stupid question : are you sure that you don't miss anything in the loop around accept() ? eg: reinitialise one error code or anything which could prevent accept() from being further called after you have successfully done several accept() at once ? I'm personnally using epoll in level triggered mode in haproxy, which often does multiple accept() per call on very high loads (>10k sessions/s), and although I've encountered difficult beginnings with epoll, it's rock solid now.
Quoted 1 line
Kernel version is 2.6.9. I can provide test code if needed.
I would suggest trying 2.6.16 first to see if it may be related to a bug which has been fixed since then, and otherwise, some test code would help.
Quoted 3 lines
Thanks,
Kyle Cronan
<[email protected]>
Cheers,
Willy