select() efficiency
Please forgive and redirect me if this is not the right place to ask this question:
I'm looking to write a sort of messaging system that would take input from any number of entities that "register" with it.. it would then route the messages to outputs and so forth..
I'm guessing that the messaging system would be a single process on the machine..
So, I'm considering making the means of input to the system be a unix socket. An entity would connect to the socket as it's means of inputting messages into the system.
However, lets suppose that 1000+ entities connect to that socket.. this would require the message system's loop to be adding 1000+ file descriptures to an fd_set and call select() every time it loops around to check for any messages.
So, my question is: how efficient would things be, doing selects() very often on 1000+ file descriptors? I'm not aware of max size for an fd_set.. (I do know that NT is limited to 64 handles.. but that's really beside the point unless I look at porting someday)
Should I go another route?
The system is meant to rapidly route messages ASAP.. so it would be a bad idea to say write them to a file and poll the file or something like that...
Another thought was to use a system-wide mutex and write to a named pipe, but the socket method seems more appealing to me in design... and I didn't know if it was pretty much equivalent either way since either I will do the work of dealing with 1000+ things or the kernel will.
Thanks,
Davy
Quoted 6 lines · Davy Durham
Please forgive and redirect me if this is not the right place to ask this question:
I'm looking to write a sort of messaging system that would take input from any number of entities that "register" with it.. it would then route the messages to outputs and so forth..
Look at epoll, or libevent, which uses epoll to be quick in this scenario.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Re: select() efficiency / epoll
So, I've been trying to use epoll.. on linux-2.6.11-6mdk
However, I'm getting segfaults because some pointers in places are getting set to low integer values (which didn't used to have those values).
The deal is that my application is multi-threaded, and I was wondering if epoll had issues if you use epoll_ctl while an epoll_wait is waiting or something like that. I'm also compiling with -D_MULTI_THREADED. I'm not new to threading, but am stumped at this point.
I'm not ruling out it being my code, but wanted to ask about epoll since it's so new.
Any ideas?
Thanks,
Davy
Quoted 14 lines · bert hubert
Quoted 5 lines · Davy Durham
Please forgive and redirect me if this is not the right place to ask this question:
I'm looking to write a sort of messaging system that would take input from any number of entities that "register" with it.. it would then route the messages to outputs and so forth..
Look at epoll, or libevent, which uses epoll to be quick in this scenario.
Re: select() efficiency / epoll
Quoted 2 lines · Davy Durham
However, I'm getting segfaults because some pointers in places are getting set to low integer values (which didn't used to have those values).
epoll is pretty heavily benchmarked and hence tested. I don't entirely understand the remark above and suggest looking at the generated core dumps.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Re: select() efficiency / epoll
Quoted 4 lines · bert hubert
On Tue, Aug 23, 2005 at 04:49:14AM -0500, Davy Durham wrote:
Quoted 9 lines
Quoted 3 lines
However, I'm getting segfaults because some pointers in places are getting set to low integer values (which didn't used to have those values).
epoll is pretty heavily benchmarked and hence tested. I don't entirely understand the remark above and suggest looking at the generated core dumps.
I just mean that when I debug and catch the segv, it's dies because some pointers now have corrupted values. (usually because something is overwriting some memory some where)
I'm currently re-writing some code to make it use select() instead of epoll_wait() and see if everything is suddently fixed. If so, then I will suspect that epoll has a problem. But it's still not ruled out being my fault since it could be a timing issue that makes the crash show up.
Re: select() efficiency / epoll
Hi,
Quoted 9 lines · Davy Durham
I just mean that when I debug and catch the segv, it's dies because some pointers now have corrupted values. (usually because something is overwriting some memory some where)
I'm currently re-writing some code to make it use
select()instead ofepoll_wait()and see if everything is suddently fixed. If so, then I will suspect that epoll has a problem. But it's still not ruled out being my fault since it could be a timing issue that makes the crash show up.
Just out of curiosity, have you had the opportunity to read some other code which uses epoll ? Maybe reading others code could enlighten you on potential bugs in your code, potential races, etc...
Regards,
Willy
Re: select() efficiency / epoll
That's probably a good idea. Where would I find out what other projects use it?
Quoted 19 lines · Willy Tarreau
Hi,
Quoted 5 lines · Davy Durham
I just mean that when I debug and catch the segv, it's dies because some pointers now have corrupted values. (usually because something is overwriting some memory some where)
I'm currently re-writing some code to make it use
select()instead ofepoll_wait()and see if everything is suddently fixed. If so, then I will suspect that epoll has a problem. But it's still not ruled out being my fault since it could be a timing issue that makes the crash show up.Just out of curiosity, have you had the opportunity to read some other code which uses epoll ? Maybe reading others code could enlighten you on potential bugs in your code, potential races, etc...
Regards,
Willy
Re: select() efficiency / epoll
Quoted 2 lines · Davy Durham
That's probably a good idea. Where would I find out what other projects use it?
I use it in my load-balancer (haproxy), and it could somewhat match your needs, because I ported the select()-based earlier version to epoll() with the smallest possible changes. Indeed, the new epoll() loop still uses the FD_ISSET() to determine what to do with epoll_ctl(). If you have changed your code to use select(), you may find similarities. But I want to tell you from now that my code is NOT multi-threaded. It could be a bug in the epoll implementation, because I don't think that there are so many applications using epoll on MT models. Bert says that the epoll implementation is heavily benchmarked, which is true, but which does not guarantee that it is tested under every condition.
You can download it from there :
http://w.ods.org/tools/haproxy/src/devel/
Use version 1.2.6. I added epoll in 1.2.5, so the diff between 1.2.4 and 1.2.5 could help you too.
Good luck !
Willy
Re: select() efficiency / epoll
Thanks for the info.. I did find this thread and was wondering if this patch ever got put in
http://www.ussg.iu.edu/hypermail/linux/kernel/0303.3/1139.html
Quoted 35 lines · Willy Tarreau
Quoted 3 lines · Davy Durham
That's probably a good idea. Where would I find out what other projects use it?
I use it in my load-balancer (haproxy), and it could somewhat match your needs, because I ported the select()-based earlier version to
epoll()with the smallest possible changes. Indeed, the newepoll()loop still uses theFD_ISSET()to determine what to do withepoll_ctl(). If you have changed your code to useselect(), you may find similarities. But I want to tell you from now that my code is NOT multi-threaded. It could be a bug in the epoll implementation, because I don't think that there are so many applications using epoll on MT models. Bert says that the epoll implementation is heavily benchmarked, which is true, but which does not guarantee that it is tested under every condition.You can download it from there :
http://w.ods.org/tools/haproxy/src/devel/
Use version 1.2.6. I added epoll in 1.2.5, so the diff between 1.2.4 and 1.2.5 could help you too.
Good luck !
Willy-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: select() efficiency / epoll
Quoted 5 lines · Davy Durham
Thanks for the info.. I did find this thread and was wondering if this patch ever got put in
http://www.ussg.iu.edu/hypermail/linux/kernel/0303.3/1139.html
Interesting ! At least it does not seem to be present in the epoll-2.4.24-0.20 I have right here, and although the code changed significantly in 2.6, it does not seem to contain it either. But I don't even see how to merge this into 2.6. You should ask Davide, he knows this code better than anyone else, and could tell us if this patch was simply lost or is unneeded.
Regards,
Willy
Re: select() efficiency / epoll
Quoted 7 lines · Willy Tarreau
Quoted 5 lines · Davy Durham
Thanks for the info.. I did find this thread and was wondering if this patch ever got put in
http://www.ussg.iu.edu/hypermail/linux/kernel/0303.3/1139.html
Quoted 6 lines
Interesting ! At least it does not seem to be present in the epoll-2.4.24-0.20 I have right here, and although the code changed significantly in 2.6, it does not seem to contain it either. But I don't even see how to merge this into 2.6. You should ask Davide, he knows this code better than anyone else, and could tell us if this patch was simply lost or is unneeded.
I should mention that the 2.4 patch is old WRT mainline epoll in 2.6 (I stopped maintaining it when 2.6 went "stable"). I'd definitely suggest to use 2.6 if you are looking at epoll.
- Davide
Re: select() efficiency / epoll
Quoted 3 lines · Davide Libenzi
I should mention that the 2.4 patch is old WRT mainline epoll in 2.6 (I stopped maintaining it when 2.6 went "stable"). I'd definitely suggest to use 2.6 if you are looking at epoll.
I am using linux-2.6.11 and glibc-2.3.4 .. and using select() in it's place seems to work fine. Are there any known issues with say, one thread does epoll_wait()s while other threads may be doing epoll_ctl()s?
Is there someone else I should be asking this question?
Thanks,
Davy
Re: select() efficiency / epoll
Quoted 6 lines · Davy Durham
Quoted 3 lines · Davide Libenzi
I should mention that the 2.4 patch is old WRT mainline epoll in 2.6 (I stopped maintaining it when 2.6 went "stable"). I'd definitely suggest to use 2.6 if you are looking at epoll.
I am using linux-2.6.11 and glibc-2.3.4 .. and using
select()in it's place seems to work fine. Are there any known issues with say, one thread does epoll_wait()s while other threads may be doing epoll_ctl()s?
There is no known problem in using epoll_ctl() in one thread while another does epoll_wait().
I suggest you to ask Valgrind to take a look at you binary. Since I have no clue of what your software does, please create the *minimal* code snippet that exploit the eventual problem, and post it.
- Davide
Re: select() efficiency / epoll
Quoted 4 lines · Davide Libenzi
There is no known problem in using
epoll_ctl()in one thread while another doesepoll_wait().
I suggest you to ask Valgrind to take a look at you binary. Since I have no clue of what your software does, please create the *minimal* code snippet that exploit the eventual problem, and post it.
Yes, I have pretty much confirmed this. And unfortunately I tried to make a minimal code snippet which demonstrates the problem, but wasn't able to do that before I figured out a work-around. I may still try to create something for you to test against so you can fix it. But I'm going to have to continue to work with the existing implementation since I'm going to be running this code on some production servers where updating the kernel might not be an option.
The work-around is as follows:
1) I create a queue that can hold operations to perform on the epoll structure and I protect it with a mutex.
2) Other threads (when needing to modify the epoll) lock the mutex and enque the operation into the operation queue instead of calling epoll_ctl itself (i.e. add this socket for reading.. add this socket for writing, remove this socket.. etc) *and* then cancel the epoll_wait() I implemented the cancel by having a pipe() always being watched for
read, and write a byte to it when I want to cancel (is there a better way?)
There are several operations that could be supported
(add/remove/modify/change userdata/etc), but I only need two myself.
3) There's only one thread that actually does the epoll_wait(). When epoll_wait() returns, (I first drain the cancel pipe so it never fills up) I handle what events need handling, and then lock the operations queue mutex, perform all the operations in the queue then clear the queue
So, this works for me now.
Thanks for all your guys' info.
-- Davy
P.S. Davide, I still might get you that snipped, but it's not a trivial snippet as you can imagine... and timing is everything to the problem :( .. and also the question of WHERE it corrupts memory.. it seemed to be unpredictable so far.
Re: select() efficiency / epoll
In reply to Davy DurhamQuoted 3 lines · Davy Durham
I'm currently re-writing some code to make it use
select()instead ofepoll_wait()and see if everything is suddently fixed. If so, then I will suspect that epoll has a problem. But it's still not ruled out being my fault since it could be a timing issue that makes the crash show up.
Well, the select() replacement works fine... so hrmm..
Re: select() efficiency / epoll
In reply to Davy DurhamQuoted 3 lines · Davy Durham
However, I'm getting segfaults because some pointers in places are getting set to low integer values (which didn't used to have those values).
Is it possible that you are overwritting the pointers with file descriptors, as those would have low integer values?
Signature · 3 lines
Rakshasa Nyaa?
Re: select() efficiency / epoll
Quoted 13 lines · Jari Sundell
Quoted 4 lines · Davy Durham
However, I'm getting segfaults because some pointers in places are getting set to low integer values (which didn't used to have those values).
Is it possible that you are overwritting the pointers with file descriptors, as those would have low integer values?
Yes, that is what I was thinking and is why I mentioned that. But I'm apparently not overwriting the pointers with FDs.. it seems that epoll is the cause at this point (unless I'm misusing the epoll API). I've made some changes to now use select() instead of epoll and things work flawlessly (although it obviously won't work as efficiently when I really connect a lot of clients to this server)
Re: select() efficiency / epoll
Quoted 6 lines · Davy Durham
Yes, that is what I was thinking and is why I mentioned that. But I'm apparently not overwriting the pointers with FDs.. it seems that epoll is the cause at this point (unless I'm misusing the epoll API). I've made some changes to now use
select()instead of epoll and things work flawlessly (although it obviously won't work as efficiently when I really connect a lot of clients to this server)
I was hoping you would mention in your reply that you knew epoll_data_t was an union and you didn't touch epoll_data::fd, so i wouldn't have to say it explicitly. ;)
Signature · 3 lines
Rakshasa Nyaa?
Re: select() efficiency / epoll
Quoted 9 lines · Jari Sundell
On 8/23/05, Davy Durham <[email protected]> wrote:
I was hoping you would mention in your reply that you knew
epoll_data_twas an union and you didn't touchepoll_data::fd, so i wouldn't have to say it explicitly. ;)
No, I saw that epoll_data_t was a union (although, it kind of makes the ptr useless as a user data pointer.. but I'm not using it for that)
When I mean that pointers are getting corrupted, I just mean in other parts of the code (actually it's some C++ STL container's data and is completely unrelated to the epoll specific code) Something, somewhere seems to be writing to memory that it's not supposed to be writing to. And as far as I can tell, it happens when I use epoll and doesn't when I use select :-/
-- Davy
Re: select() efficiency / epoll
In reply to Jari SundellQuoted 10 lines · Jari Sundell
On 8/23/05, Davy Durham <[email protected]> wrote:
I was hoping you would mention in your reply that you knew
epoll_data_twas an union and you didn't touchepoll_data::fd, so i wouldn't have to say it explicitly. ;)
Oh!.. unless the epoll_data_t is a union just for convenience in that it already has an 'int fd' if you want to use that, but don't have to.. that at least makes the void *ptr, useful.. The example in 'man epoll' sorta made it look necessary to set the 'fd' of the union.
But that still doesn't fix the issue of course.. but good to know.