epoll design problems with common fork/exec patterns
Hi!
I ran into what I see as unsolvable problems that make epoll useless as a generic event mechanism.
I recently switched to libevent as event loop, and found that my programs work fine when it is using select or poll, but work eratically or halt when using epoll.
The reason as I found out is the peculiar behaviour of epoll over fork. It doesn't work as documented, and even if, it would make the use of third-party libraries using fork usually impossible.
Here are two scenarios where it screws up:
- some library forks, explicitly closes all fd's it doesn't need, and execs another program (which is common behvaiour).
In this case, the parent process works fine until the child closes fds, after which the fds become unarmed in the parent too. This works as documented, but since libraries expect this to work without affecting the parent, this puts a new and incompatible strain on what libraries can do, which in turn makes epoll unsuitable in cases where you don't control all your code.
- I have a library that emulates asynchronous I/O with a thread pool, and uses a pipe for event notification. That library registers a fork handler that closes the pipe in the child and recreates it, so the child could continue doing AIO (as could the parent).
This, too, screws up notifications for the parent,
Now, the epoll manpage says that closing a fd will remove it from all fd sets. This would explain the behaviour above. Unfortunately (or fortunately?) this is not what happens: when the fds are being closed by exec or exit, the fds do not get removed from the epoll set.
This behaviour strikes me as extremely illogical. On the one hand, one cannot share the epoll fd between processes normally, but on fork, you can, even though it makes no sense (the child has a different fd "namespace" than the parent) and actually works on (then( unrelated fds in the other process.
It also strikes as weird that the order of closing fds should make so much of a difference: if the epoll fd is closed first in the child, the other fds will survive in the parent, if its closed last, they don't. Makes no sense to me.
Now, the problem I see is not that it makes no sense to me - thats clearly my problem. The problem I see is that there is no way to avoid the associated problems except by patching all code that would ever use fork, even if it never has heard anything about epoll yet. This is extremely nonlocal action at a distance, as this affects a lot of code not even the author might be aware of (fork is rather common).
To illustrate, here are some workarounds I thought about:
- rearming all fds after fork: doesn't work, as the fds get removed asynchronously so I would have to wait for the child to do it.
- closing the epoll fd after fork: doesn't work unless I control the fork. I can install a handler to be called using pthreads, but that won't help as other handlers might be called first (as in the case of the aio library above), screwing me.
- closing and recreating the epoll fd before the fork: isn't support event remotely by libevent or similar event loops, and would not help either as I cnanot control the calls to fork.
Is epoll really designed to be so incompatible with the most commno fork patterns? Shouldn't epoll do refcounting, as is commonly done under Unix? As the fd space is not shared between rpocesses, why does epoll try? Shouldn't the epoll information be copied just like the fd table itself, memory, and other resources?
As it looks now, epoll looks useless except in the most controlled environments, as it doesn't duplicate state on fork as is done with the other fd-related resources (as opposed to the underlying files, which are properly shared).
Signature · 6 lines
The choice of a
-----==- _GNU_ Deliantra, the free in data+content MORPG
----==-- _ generation
---==---(_)__ __ ____ __ http://www.deliantra.net/
--==---/ / _ \/ // /\ \/ /
-=====/_/_//_/\_,_/ /_/\_\Quoted 13 lines · Marc Lehmann
Hi!
I ran into what I see as unsolvable problems that make epoll useless as a generic event mechanism.
I recently switched to libevent as event loop, and found that my programs work fine when it is using select or poll, but work eratically or halt when using epoll.
The reason as I found out is the peculiar behaviour of epoll over fork. It doesn't work as documented, and even if, it would make the use of third-party libraries using fork usually impossible.
Quoted 7 lines
Here are two scenarios where it screws up:
- some library forks, explicitly closes all fd's it doesn't need, and execs another program (which is common behvaiour).
In this case, the parent process works fine until the child closes fds, after which the fds become unarmed in the parent too. This works as
I have no idea what exact problem you have. But if the child closes some file descriptor that were 'cloned' at fork() time, this only decrements a refcount, and definitely should not close it for the 'parent'. epoll in this regard uses a generic kernel service (file descriptor sharing between tasks).
I have some apps that are happily using epoll() and fork()/exec() and have no problem at all. I usually use O_CLOEXEC so that all close() are done at exec() time without having to do it in a loop. epoll continues to work as expected in the parent process.
Quoted 10 lines
documented, but since libraries expect this to work without affecting the parent, this puts a new and incompatible strain on what libraries can do, which in turn makes epoll unsuitable in cases where you don't control all your code.
- I have a library that emulates asynchronous I/O with a thread pool, and uses a pipe for event notification. That library registers a fork handler that closes the pipe in the child and recreates it, so the child could continue doing AIO (as could the parent).
Quoted 6 lines
This, too, screws up notifications for the parent,
Now, the epoll manpage says that closing a fd will remove it from all fd sets. This would explain the behaviour above. Unfortunately (or fortunately?) this is not what happens: when the fds are being closed by exec or exit, the fds do not get removed from the epoll set.
at exec() (granted CLOEXEC is asserted) or exit() time, only the refcount of each file is decremented. Only if their refcount becomes NULL, files are then removed from epoll set.
Quoted 31 lines
This behaviour strikes me as extremely illogical. On the one hand, one cannot share the epoll fd between processes normally, but on fork, you can, even though it makes no sense (the child has a different fd "namespace" than the parent) and actually works on (then( unrelated fds in the other process.
It also strikes as weird that the order of closing fds should make so much of a difference: if the epoll fd is closed first in the child, the other fds will survive in the parent, if its closed last, they don't. Makes no sense to me.
Now, the problem I see is not that it makes no sense to me - thats clearly my problem. The problem I see is that there is no way to avoid the associated problems except by patching all code that would ever use fork, even if it never has heard anything about epoll yet. This is extremely nonlocal action at a distance, as this affects a lot of code not even the author might be aware of (fork is rather common).
To illustrate, here are some workarounds I thought about:
- rearming all fds after fork: doesn't work, as the fds get removed asynchronously so I would have to wait for the child to do it.
- closing the epoll fd after fork: doesn't work unless I control the fork. I can install a handler to be called using pthreads, but that won't help as other handlers might be called first (as in the case of the aio library above), screwing me.
- closing and recreating the epoll fd before the fork: isn't support event remotely by libevent or similar event loops, and would not help either as I cnanot control the calls to fork.
Quoted 5 lines
Is epoll really designed to be so incompatible with the most commno fork patterns? Shouldn't epoll do refcounting, as is commonly done under Unix? As the fd space is not shared between rpocesses, why does epoll try? Shouldn't the epoll information be copied just like the fd table itself, memory, and other resources?
Too many questions here, showing lack of understanding.
Quoted 6 lines
As it looks now, epoll looks useless except in the most controlled environments, as it doesn't duplicate state on fork as is done with the other fd-related resources (as opposed to the underlying files, which are properly shared).
epoll definitly is not useless. It is used on major and critical apps. You certainly missed something.
Please provide some code to illustrate one exact problem you have.
Quoted 4 lines · Eric Dumazet
Quoted 2 lines
In this case, the parent process works fine until the child closes fds, after which the fds become unarmed in the parent too. This works as
I have no idea what exact problem you have.
Well, I explained it rather succinctly, I think. If you tell me whats unclear I can explain...
Quoted 3 lines
But if the child closes some
file descriptor that were 'cloned' atfork()time, this only decrements a refcount, and definitely should not close it for the 'parent'.
It doesn't. It removes it from the epoll set, though, so the parent will not receive events for that fd anymore.
Quoted 1 line
I have some apps that are happily using
epoll()and fork()/exec() and have
The problem I described is fork/close/exec. close being the explicit syscall.
Quoted 3 lines
no problem at all. I usually use
O_CLOEXECso that allclose()are done atexec()time without having to do it in a loop. epoll continues to work as expected in the parent process.
This is because epoll doesn't behave like documented: It removes the fd from the parents epoll set only on an explicit close() syscall, not on an implicit close from exec.
Quoted 7 lines
Quoted 3 lines
fd sets. This would explain the behaviour above. Unfortunately (or fortunately?) this is not what happens: when the fds are being closed by exec or exit, the fds do not get removed from the epoll set.
at
exec()(granted CLOEXEC is asserted) orexit()time, only the refcount of each file is decremented. Only if their refcount becomes NULL, files are then removed from epoll set.
Yes. But thats obviously not the only way to close fds.
Quoted 7 lines
Quoted 5 lines
Is epoll really designed to be so incompatible with the most commno fork patterns? Shouldn't epoll do refcounting, as is commonly done under Unix? As the fd space is not shared between rpocesses, why does epoll try? Shouldn't the epoll information be copied just like the fd table itself, memory, and other resources?
Too many questions here, showing lack of understanding.
You already said you don't the problem. No need to get insulting :(
Quoted 2 lines
epoll definitly is not useless. It is used on major and critical apps. You certainly missed something.
Well, it behaves like documented, which is the problem. You admit you don't understand the problem or the documentation, so again, no need to insult me.
Quoted 1 line
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5
if (fork () = 0)
{
close (5);
// fd 5 is now removed from the epoll set of the parent.
_exit (0);
}Signature · 6 lines
The choice of a
-----==- _GNU_
----==-- _ generation Marc Lehmann
---==---(_)__ __ ____ __ [email protected]
--==---/ / _ \/ // /\ \/ / http://schmorp.de/
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPEQuoted 38 lines · Marc Lehmann
Quoted 3 lines · Eric Dumazet
Quoted 2 lines
In this case, the parent process works fine until the child closes fds, after which the fds become unarmed in the parent too. This works as
I have no idea what exact problem you have.
Well, I explained it rather succinctly, I think. If you tell me whats unclear I can explain...
Quoted 1 line
But if the child closes some file descriptor that were 'cloned' at
fork()time, this only decrements a refcount, and definitely should not close it for the 'parent'.It doesn't. It removes it from the epoll set, though, so the parent will not receive events for that fd anymore.
Quoted 1 line
I have some apps that are happily using
epoll()and fork()/exec() and haveThe problem I described is fork/close/exec. close being the explicit syscall.
Quoted 1 line
no problem at all. I usually use
O_CLOEXECso that allclose()are done atexec()time without having to do it in a loop. epoll continues to work as expected in the parent process.This is because epoll doesn't behave like documented: It removes the fd from the parents epoll set only on an explicit
close()syscall, not on an implicit close from exec.Quoted 4 lines
Quoted 3 lines
fd sets. This would explain the behaviour above. Unfortunately (or fortunately?) this is not what happens: when the fds are being closed by exec or exit, the fds do not get removed from the epoll set.
at
exec()(granted CLOEXEC is asserted) orexit()time, only the refcount of each file is decremented. Only if their refcount becomes NULL, files are then removed from epoll set.Yes. But thats obviously not the only way to close fds.
Quoted 6 lines
Quoted 5 lines
Is epoll really designed to be so incompatible with the most commno fork patterns? Shouldn't epoll do refcounting, as is commonly done under Unix? As the fd space is not shared between rpocesses, why does epoll try? Shouldn't the epoll information be copied just like the fd table itself, memory, and other resources?
Too many questions here, showing lack of understanding.
Quoted 8 lines
You already said you don't the problem. No need to get insulting :(
Quoted 2 lines
epoll definitly is not useless. It is used on major and critical apps. You certainly missed something.
Well, it behaves like documented, which is the problem. You admit you don't understand the problem or the documentation, so again, no need to insult me.
Hum... I will update my english vocabulary and mark "missed" as an insult.
I have no problem with epoll nor its documentation.
Quoted 11 lines
Quoted 2 lines
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5 if (fork () = 0) { close (5); // fd 5 is now removed from the epoll set of the parent. _exit (0); }
It doesnt on every kernels I had played with. And I played with *lot* of kernels you know.
If such a bug exists on your kernel, please fill a complete bug report, giving details.
Thank you
Quoted 5 lines · Eric Dumazet
Quoted 3 lines
Well, it behaves like documented, which is the problem. You admit you don't understand the problem or the documentation, so again, no need to insult me.
Hum... I will update my english vocabulary and mark "missed" as an insult.
Well, ignoring my arguments by claiming I lack understanding is an insult, as you didn't take my arguments at face value but declassified them by attacking my person.
Quoted 1 line
I have no problem with epoll nor its documentation.
Thats fine for you. But I have, at least, with epoll, as the documented and observed behaviour makes epoll unusable as a general event loop replacement.
Quoted 2 lines
It doesnt on every kernels I had played with. And I played with *lot* of kernels you know.
No, I don't know that. And so far you only said you used fork+exec, not close in between, so maybe the playing you did was not related to this problem?
I also played with a lot of kernels, but for epoll specifically, I played with 2.6.21-2-amd64 and 2.6.22-1-amd64, both from debian unstable with no customisations.
Quoted 2 lines
If such a bug exists on your kernel, please fill a complete bug report, giving details.
As this behaviour is clearly documented in the epoll manpage, why do you think it is a bug? I think its fairly bad, but at least tis documented as the behaviour it should be:
Q6 Will the close of an fd cause it to be removed from all epoll sets automatically?
A6 Yes.
As such filing, a bug report for behaviour which isn't in fact a bug would be counterproductive. My goal in my mail was to find out if there are work arounds for this peculiar behaviour (Or inspire discussion on this behaviour).
Of course, one can create big programs using epoll to their advantage. I never claimed otherwise. But as a general event loop replacement (i.e. outside of controleld environments), epoll does not currently qualify, as I would have to control an awful lot of code (think of an perl module interfacing to epoll: you would not have to control all third-party modules that might interfere with fork+close+exec. This is very common in scripting languages).
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 10 lines · Marc Lehmann
Quoted 1 line · Eric Dumazet
If such a bug exists on your kernel, please fill a complete bug report, giving details.
As this behaviour is clearly documented in the epoll manpage, why do you think it is a bug? I think its fairly bad, but at least tis documented as the behaviour it should be:
Q6 Will the close of an fd cause it to be removed from all epoll sets automatically?
A6 Yes.
Answer : epoll documentation cannot explain the full semantic of file descriptors, or difference between user side (file descriptors) and kernel side (files and fds) Or should, since you had problems. But then, if the epoll documentation has to document the full Unix/Linux files semantic, nobody will read it.
The 'close' of a file is not close(fd) :)
But : the last close() so that underlying file refcount is 0
example 1)
fd = open("somefile", ...)
fd1 = dup(fd);
epoll_add_in_my_set(fd1);/* setup epoll work on fd1 */ {do_something;}
close(fd1); /* this is not the last close and will NOT close 'somefile' */
/* It wont be removed from epoll sets NOW */
close(fd); /* oh yes, this one is the real 'file close', now we perform epoll cleanups */
epoll has to deal with files, but documentation is a User side documentation, so has to use 'file descriptors'. So everything that plays with the file descriptor table can make the thing complex to understand/document. (fork()/dup()/close()/exit()/exec()....)
example 2)
int pfd[2];
pipe(pfd);
epoll_add_in_my_set(pfd[0]);/* setup epoll work on pfd[0] for example */ pid = fork();
if (pid == 0) {
close(pfd[0]); /* this is not the last close and will NOT close pipe */
/* epoll has NO WAY to perform some cleanup at this stage */
close(pfd[1]); /* this not the last close and will NOT close the pipe*/ _exit(0);
}close(pfd[1]);
wait(NULL);
{do_something_epoll_related;}close(pfd[0]); /* finally we close the pipe, and epoll can do its cleanup */
fork() is acting sort of dup() , as it increases all file refcounts.
You have problems about close()/dup()/fork()/... file descriptors semantic, which is handled by a layer independent from epoll stuff.
Quoted 5 lines · Eric Dumazet
Quoted 3 lines
Q6 Will the close of an fd cause it to be removed from all epoll
sets automatically?
A6 Yes.Answer : epoll documentation cannot explain the full semantic of file
epoll documentation easily can. there is nothig keeping it from it. don't make silly arguments like that.
Quoted 1 line
Or should, since you had problems
You are again implying I lakc understanding. That is, however, not true. I don't see the point in being insulted by you, so I won'T continue talking to you :(
Quoted 1 line
The 'close' of a file is not close(fd) :)
Good that you understand that.
That is one of my problems, as the manpage talks about closing of the fd, but there are multiple ways to do that, and some are not handled the same way.
Quoted 2 lines
epoll has to deal with files, but documentation is a User side documentation, so has to use 'file descriptors'.
There is obviously no need for documentation to do that, contrary to your claim. The manpages for e.g. dup or the official sus manpages manage to document it (mostly) correctly, so your claim that documentation must use file descriptors when the underlying file structure is meant is disproven.
Quoted 4 lines
fork()is acting sort ofdup(), as it increases all file refcounts.You have problems about close()/dup()/fork()/... file descriptors semantic, which is handled by a layer independent from epoll stuff.
No, I have no problem with dup at all.
I have a problem with explicitlx closing file descriptors in the child will stop events for those files to be reported in the parent.
I am sorry, but I epxlained this very clearly a number of times, but for some reason, apart from accusing me to not understanding files and file descritpors or (clear enough) documentation, you ignore that and instead hammer on other problems.
To me, it seems you are not the one who understands.
Signature · 6 lines
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ \/ // /\ \/ / [email protected]
-=====/_/_//_/\_,_/ /_/\_\Quoted 9 lines · Marc Lehmann
Quoted 1 line
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5 if (fork () = 0) { close (5); // fd 5 is now removed from the epoll set of the parent. _exit (0); }
Hmmm ... what? I assume you know that:
1) A file descriptor is a userspace view/handle of a kernel object
2) The kernel object has a use-count for as many file descriptors that have been handed out to userspace
3) A close() decreases the internal counter by one
4) The kernel object gets effectively closed when the internal counter goes to zero
5) A fork() acts as a dup() on the file descriptors by hence bumping up its internal counter
6) Epoll removes the file from the set, when the *kernel* object gets closed (internal use-count goes to zero)
With that in mind, how can the code snippet above trigger a removal from the epoll set?
- Davide
Quoted 24 lines · Davide Libenzi
Quoted 9 lines · Marc Lehmann
Quoted 1 line
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5 if (fork () = 0) { close (5); // fd 5 is now removed from the epoll set of the parent. _exit (0); }Hmmm ... what? I assume you know that:
1) A file descriptor is a userspace view/handle of a kernel object
2) The kernel object has a use-count for as many file descriptors that have been handed out to userspace
3) A
close()decreases the internal counter by one4) The kernel object gets effectively closed when the internal counter goes to zero
Quoted 8 lines
5) A
fork()acts as adup()on the file descriptors by hence bumping up its internal counter6) Epoll removes the file from the set, when the *kernel* object gets closed (internal use-count goes to zero)
With that in mind, how can the code snippet above trigger a removal from the epoll set?
Davide,
from what I understand, Marc is not asking for the code above to remove the fd from the epoll set, but he's in fact complaining that he *observed* that the fd was removed from the epoll set in the *parent* process when the child closes it, which is of course not expected at all. As strange as it looks like, this might need investigation. It is possible that there is some strange bug somewhere in some kernel versions.
Marc, I think that if you indicate the last kernel version on which you observed this and provide a very short and easy reproducer, it would help everyone investigating this. Basically something which reports "OK" or "KO".
Regards,
Willy
Quoted 34 lines · Willy Tarreau
Quoted 24 lines · Davide Libenzi
Quoted 9 lines · Marc Lehmann
Quoted 1 line
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5 if (fork () = 0) { close (5); // fd 5 is now removed from the epoll set of the parent. _exit (0); }Hmmm ... what? I assume you know that:
1) A file descriptor is a userspace view/handle of a kernel object
2) The kernel object has a use-count for as many file descriptors that have been handed out to userspace
3) A
close()decreases the internal counter by one4) The kernel object gets effectively closed when the internal counter goes to zero
Quoted 8 lines
5) A
fork()acts as adup()on the file descriptors by hence bumping up its internal counter6) Epoll removes the file from the set, when the *kernel* object gets closed (internal use-count goes to zero)
With that in mind, how can the code snippet above trigger a removal from the epoll set?
Quoted 8 lines
Davide,
from what I understand, Marc is not asking for the code above to remove the fd from the epoll set, but he's in fact complaining that he *observed* that the fd was removed from the epoll set in the *parent* process when the child closes it, which is of course not expected at all. As strange as it looks like, this might need investigation. It is possible that there is some strange bug somewhere in some kernel versions.
That would be *really* strange, since epoll hooks in __fput() in order to perform proper cleanup. This means that, in the case above, the file will be really closed in the parent too. That, I think, would trigger way more serious problems in userspace.
Quoted 4 lines
Marc, I think that if you indicate the last kernel version on which you observed this and provide a very short and easy reproducer, it would help everyone investigating this. Basically something which reports "OK" or "KO".
Of course. That'd be great.
- Davide
Quoted 10 lines · Willy Tarreau
Quoted 10 lines · Marc Lehmann
Quoted 9 lines
Quoted 2 lines
Please provide some code to illustrate one exact problem you have.
// assume there is an open epoll set that listens for events on fd 5 if (fork () = 0)
{ close (5); // fd 5 is now removed from the epoll set of the parent. _exit (0); }
..
Quoted 11 lines
from what I understand, Marc is not asking for the code above to remove the fd from the epoll set, but he's in fact complaining that he *observed* that the fd was removed from the epoll set in the *parent* process when the child closes it, which is of course not expected at all. As strange as it looks like, this might need investigation. It is possible that there is some strange bug somewhere in some kernel versions.
Marc, I think that if you indicate the last kernel version on which you observed this and provide a very short and easy reproducer, it would help everyone investigating this. Basically something which reports "OK" or "KO".
That's how I read it, too.
So basically, a program like this, perhaps.
Except that, here running 2.6.23.1, it works just fine (no removal bug).
#include <sys/epoll.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/socket.h>
static int del_from_epoll_set (int efd, int fd, const char *msg)
{
struct epoll_event e;
memset(&e, 0, sizeof(e));
e.data.fd = fd;
if (epoll_ctl(efd, EPOLL_CTL_DEL, fd, &e) == -1) {
int err = errno;
fprintf(stderr, "epoll_ctl(DEL) failed (%s): %s\n", msg, strerror(err));
return -1;
}
return 0;
}static int add_to_epoll_set (int efd, int fd, __uint32_t events, const char *msg)
{
struct epoll_event e;
memset(&e, 0, sizeof(e));
e.events = events;
e.data.fd = fd;
if (epoll_ctl(efd, EPOLL_CTL_ADD, fd, &e) == -1) {
int err = errno;
fprintf(stderr, "epoll_ctl(ADD) failed (%s): %s\n", msg, strerror(err));
return -1;
}
return 0;
}int main (int argc, char **argv)
{
int efd, sd, fds[2];
pid_t cpid;
if (pipe(fds) == -1) {
perror("pipe()");
exit(1);
}
sd = socket(PF_INET, SOCK_STREAM, 0);
if (sd == -1) {
perror("socket");
exit(1);
}
efd = epoll_create(5);
if (efd == -1) {
perror("epoll_create");
exit(1);
}
if (add_to_epoll_set(efd, fileno(stdin), EPOLLIN, "stdin"))
exit(1);
if (add_to_epoll_set(efd, fileno(stdout), EPOLLOUT, "stdout"))
exit(1);
if (add_to_epoll_set(efd, fds[0], EPOLLIN, "pipe_read"))
exit(1);
if (add_to_epoll_set(efd, fds[1], EPOLLOUT, "pipe_write"))
exit(1);
if (add_to_epoll_set(efd, sd, EPOLLIN|EPOLLOUT, "socket"))
exit(1);
// assume there is an open epoll set that listens for events on fd 5
cpid = fork();
if (cpid == 0) {
close(fileno(stdin));
close(fileno(stdout));
close(fds[0]);
close(fds[1]);
close(sd);
exit(0);
}
waitpid(cpid, NULL, 0);
// now test whether the fd's are still in the epoll set:
add_to_epoll_set(efd, sd, EPOLLIN|EPOLLOUT, "sd");
add_to_epoll_set(efd, fds[0], EPOLLIN, "fds[0]");
add_to_epoll_set(efd, fds[1], EPOLLOUT, "fds[1]");
add_to_epoll_set(efd, fileno(stdin), EPOLLIN, "fileno(stdin)");
add_to_epoll_set(efd, fileno(stdout), EPOLLOUT, "fileno(stdout)");
del_from_epoll_set(efd, sd, "sd");
del_from_epoll_set(efd, fds[0], "fds[0]");
del_from_epoll_set(efd, fds[1], "fds[1]");
del_from_epoll_set(efd, fileno(stdin), "fileno(stdin)");
del_from_epoll_set(efd, fileno(stdout), "fileno(stdout)");
printf("Done.\n");
exit(0);
}Quoted 6 lines
6) Epoll removes the file from the set, when the *kernel* object gets closed (internal use-count goes to zero)
With that in mind, how can the code snippet above trigger a removal from the epoll set?
I don't see how that can be. Suppose I add fd 8 to an epoll set. Suppose fd
5 is a dup of fd 8. Now, I close fd 8. How can fd 8 remain in my epoll set, since there no longer is an fd 8? Events on files registered for epoll notification are reported by descriptor, so the set membership has to be associated (as reflected into userspace) with the descriptor, not the file.
For example, consider:
1) Process creates an epoll set, the set gets fd 4.
2) Process creates a socket, it gets fd 5.
3) The process adds fd 5 to set 4.
4) The process forks.
5) The child inherits the epoll set but not the socket.
Here the kernel cannot quite do the right thing. Ideally, the parent would
still have fd 5 in its version of the epoll set. After all, it has not closed fd 5. However, the child *cannot* see fd 5 in its version of the epoll set since it has no fd 5. An event reported for fd 5 would be nonsense.
So it seems the kernel either has to break one of these "would/cannot"
requirements, or it has to split the epoll set in two. However, splitting the set into two sets is clearly wrong since the processes should share it.
Q6 Will the close of an fd cause it to be removed from
all
epoll sets automatically?
A6 Yes.
Note that this talks of the close of an "fd", not a file. The 'close'
function in fact closes an fd, as that fd is then reusable. So it sounds like the problem above is solved by removing the fd from the set, but in practice this doesn't happen. I have programs that call 'close' between 'fork' and 'exec' and do not see the socket removed from the poll set.
DS
Quoted 11 lines · David Schwartz
Quoted 5 lines
6) Epoll removes the file from the set, when the *kernel* object gets closed (internal use-count goes to zero)
With that in mind, how can the code snippet above trigger a removal from the epoll set?
I don't see how that can be. Suppose I add fd 8 to an epoll set. Suppose fd
5 is a dup of fd 8. Now, I close fd 8. How can fd 8 remain in my epoll set, since there no longer is an fd 8? Events on files registered for epoll notification are reported by descriptor, so the set membership has to be associated (as reflected into userspace) with the descriptor, not the file.
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user app is playing dup()/close() games for example.
typedef union epoll_data
{
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
} epoll_data_t;
It's true some applications are using 'fd' field from epoll_data_t, but in this case they should not play dup()/close() games that could change the meaning of their 'epoll tags'. They would better use 'ptr/u64' for example to map the event to an application object. In this object they might find the correct handle (fd) to communicate with the kernel for a given 'file'. This handle could then be remapped to another handle using dup()/fcntl()/close()...
Quoted 11 lines
For example, consider:
1) Process creates an epoll set, the set gets fd 4.
2) Process creates a socket, it gets fd 5.
3) The process adds fd 5 to set 4.
4) The process forks.
Quoted 7 lines
5) The child inherits the epoll set but not the socket.
Here the kernel cannot quite do the right thing. Ideally, the parent would
still have fd 5 in its version of the epoll set. After all, it has not closed fd 5. However, the child *cannot* see fd 5 in its version of the epoll set since it has no fd 5. An event reported for fd 5 would be nonsense.
Yes, it would be nonsense that the child still tries to get events from the epoll set while he cannot possibly use the socket. If you use 'ptr' field to retrieve an object, this object probably would have no meaning in the child anyway, especially after an exec() syscall.
That kind of user error can also happens with select()/poll(), if you do for example :
FD_ZERO(&fdset);FD_SET(fd, &fdset);select(fd+1,&fdset, NULL, NULL, NULL);
newfd = dup(fd);
close(fd);
for (i = 0 ; i < maxfd ; i++)
if (FD_ISSET(i, &fdset))
read(i, ...)
Quoted 6 lines · Eric Dumazet
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user
app is playing dup()/close() games for example.
Great. So the only issue then is that the documentation is confusing. It frequently uses the term "fd" where it means file. For example, it says:
Q1 What happens if you add the same fd to anepoll_set
twice?
A1 You will probably get EEXIST. However, it is
possible
that two threads may add the same fd twice. This is
a
harmless condition.
This gives no reason to think there's anything wrong with adding the same file twice so long as you do so through different descriptors. (One can imagine an application that does this to segregate read and write operations to avoid a race where the descriptor is closed from under a writer due to handling a fatal read error.) Obviously, that won't work.
And this part:
Q6 Will the close of an fd cause it to be removed from
all
epoll sets automatically?
A6 Yes.
This is incorrect. Closing an fd will not cause it to be removed from all epoll sets automatically. Only closing a file will. This is what caused the OP's confusion, and it is at best imprecise and, at worst, flat out wrong.
DS
PS: It is customary to trim individuals off of CC lists when replying to a list when the subject matter of the post is squarely inside the subject of the list. If the person CC'd was interested in the list's subject, he or she would presumably subscribe to the list. Not everyone wants two copies of every post. Not everyone wants a personal copy of every sub-thread that results from a post they make. In the past few years, I've received approximately an equal number of complaints about trimming CC's on posts to LKML and not trimming CC's on such posts.
Quoted 23 lines · David Schwartz
Quoted 6 lines · Eric Dumazet
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user
app is playing dup()/close() games for example.Great. So the only issue then is that the documentation is confusing. It frequently uses the term "fd" where it means file. For example, it says:
Q1 What happens if you add the same fd to an
epoll_set
twice?A1 You will probably get EEXIST. However, it is
possible
that two threads may add the same fd twice. This is
a
harmless condition.
Quoted 5 lines
This gives no reason to think there's anything wrong with adding the same file twice so long as you do so through different descriptors. (One can imagine an application that does this to segregate read and write operations to avoid a race where the descriptor is closed from under a writer due to handling a fatal read error.) Obviously, that won't work.
I agree, that is confusing. However, you can safely add two different file descriptors pointing to the same file*, with different event masks, and that will work as expected.
Quoted 11 lines
And this part:
Q6 Will the close of an fd cause it to be removed from
all
epoll sets automatically?A6 Yes.
This is incorrect. Closing an fd will not cause it to be removed from all epoll sets automatically. Only closing a file will. This is what caused the OP's confusion, and it is at best imprecise and, at worst, flat out wrong.
OTOH you cannot list *every* possible scenario in a man page, otherwise you end up writing a book instead of a man page. I will try to find some time with Michael to refine the man page.
Quoted 8 lines
PS: It is customary to trim individuals off of CC lists when replying to a list when the subject matter of the post is squarely inside the subject of the list. If the person CC'd was interested in the list's subject, he or she would presumably subscribe to the list. Not everyone wants two copies of every post. Not everyone wants a personal copy of every sub-thread that results from a post they make. In the past few years, I've received approximately an equal number of complaints about trimming CC's on posts to LKML and not trimming CC's on such posts.
Does anyone that in 2007 still did not manage to find a way to avoid dups in hitting his mailbox, deserve any consideration at all? OTOH many ppl, like myself, uses To and Cc header to direct email to proper folders, where they are treated with a different level of attention. And your stripp-all-headers mania screws that up badly.
- Davide
Quoted 29 lines · Davide Libenzi
Quoted 21 lines · David Schwartz
Quoted 6 lines · Eric Dumazet
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user
app is playing dup()/close() games for example.Great. So the only issue then is that the documentation is confusing. It frequently uses the term "fd" where it means file. For example, it says:
Q1 What happens if you add the same fd to an
epoll_set
twice?A1 You will probably get EEXIST. However, it is
possible
that two threads may add the same fd twice. This is
a
harmless condition.Quoted 5 lines
This gives no reason to think there's anything wrong with adding the same file twice so long as you do so through different descriptors. (One can imagine an application that does this to segregate read and write operations to avoid a race where the descriptor is closed from under a writer due to handling a fatal read error.) Obviously, that won't work.
Quoted 3 lines
I agree, that is confusing. However, you can safely add two different file descriptors pointing to the same file*, with different event masks, and that will work as expected.
So can I summarize what I understand:
a) Adding the same file descriptor twice to an epoll set will cause an error (EEXIST).
b) In a separate message to linux-man, Chris Heath says that two threads *can't* add the same fd twice to an epoll set, despite what the existing man page text says. I haven't tested that, but it sounds to me as though it is likely to be true. Can you comment please Davide?
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?
A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2) F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.
Seem okay Davide?
Cheers,
Michael
PS I've trimmed the part of this thread about Q6/A6, since I dealt with that in another thread ("epoll and shared fd's").
Signature · 5 lines
Michael Kerrisk Maintainer of the Linux man-pages project http://www.kernel.org/doc/man-pages/ Want to report a man-pages bug? Look here: http://www.kernel.org/doc/man-pages/reporting_bugs.html
Quoted 30 lines · Michael Kerrisk
Quoted 29 lines · Davide Libenzi
Quoted 26 lines · David Schwartz
Quoted 6 lines · Eric Dumazet
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user
app is playing dup()/close() games for example.Great. So the only issue then is that the documentation is confusing. It frequently uses the term "fd" where it means file. For example, it says:
Q1 What happens if you add the same fd to an
epoll_set
twice?A1 You will probably get EEXIST. However, it is
possible
that two threads may add the same fd twice. This is
a
harmless condition.This gives no reason to think there's anything wrong with adding the same file twice so long as you do so through different descriptors. (One can imagine an application that does this to segregate read and write operations to avoid a race where the descriptor is closed from under a writer due to handling a fatal read error.) Obviously, that won't work.
Quoted 8 lines
Quoted 3 lines
I agree, that is confusing. However, you can safely add two different file descriptors pointing to the same file*, with different event masks, and that will work as expected.
So can I summarize what I understand:
a) Adding the same file descriptor twice to an epoll set will cause an error (EEXIST).
Yes.
Quoted 4 lines
b) In a separate message to linux-man, Chris Heath says that two threads *can't* add the same fd twice to an epoll set, despite what the existing man page text says. I haven't tested that, but it sounds to me as though it is likely to be true. Can you comment please Davide?
Yes, you can't add the same fd twice. Think about a DB where "file*,fd" is the key.
Quoted 11 lines
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?
Quoted 8 lines
A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2)F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.Seem okay Davide?
Looks sane to me.
- Davide
Quoted 45 lines · Davide Libenzi
Quoted 30 lines · Michael Kerrisk
Quoted 29 lines · Davide Libenzi
Quoted 26 lines · David Schwartz
Quoted 6 lines · Eric Dumazet
Events are not necessarly reported "by descriptors". epoll uses an opaque field provided by the user.
It's up to the user to properly chose a tag that will makes sense if the user
app is playing dup()/close() games for example.Great. So the only issue then is that the documentation is confusing. It frequently uses the term "fd" where it means file. For example, it says:
Q1 What happens if you add the same fd to an
epoll_set
twice?A1 You will probably get EEXIST. However, it is
possible
that two threads may add the same fd twice. This is
a
harmless condition.This gives no reason to think there's anything wrong with adding the same file twice so long as you do so through different descriptors. (One can imagine an application that does this to segregate read and write operations to avoid a race where the descriptor is closed from under a writer due to handling a fatal read error.) Obviously, that won't work.
Quoted 8 lines
Quoted 3 lines
I agree, that is confusing. However, you can safely add two different file descriptors pointing to the same file*, with different event masks, and that will work as expected.
So can I summarize what I understand:
a) Adding the same file descriptor twice to an epoll set will cause an error (EEXIST).
Yes.
Quoted 7 lines
Quoted 4 lines
b) In a separate message to linux-man, Chris Heath says that two threads *can't* add the same fd twice to an epoll set, despite what the existing man page text says. I haven't tested that, but it sounds to me as though it is likely to be true. Can you comment please Davide?
Yes, you can't add the same fd twice. Think about a DB where "file*,fd" is the key.
To clarify, the key appears to be file* plus the user-space integer that represents the fd.
Quoted 18 lines
Quoted 18 lines
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2)F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.
Quoted 3 lines
Quoted 1 line
Seem okay Davide?
Looks sane to me.
I think fork(2) should not be in the above list. fork(2) duplicates the kernel's fd, but the user-space integer that represents the fd remains the same, so you will get EEXIST if you try to add the fd that was duplicated by fork.
Chris
Quoted 7 lines · Chris "�~B�" Heath
Quoted 3 lines · Davide Libenzi
Yes, you can't add the same fd twice. Think about a DB where "file*,fd" is the key.
To clarify, the key appears to be file* plus the user-space integer that represents the fd.
Yes, that's what I said.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2)F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Seem okay Davide?
Looks sane to me.
I think fork(2) should not be in the above list. fork(2) duplicates the kernel's fd, but the user-space integer that represents the fd remains the same, so you will get EEXIST if you try to add the fd that was duplicated by fork.
Good catch, fork(2) should not be there.
- Davide
Quoted 33 lines · Davide Libenzi
Quoted 8 lines · Chris "ã~B¯" Heath
Quoted 4 lines · Davide Libenzi
Yes, you can't add the same fd twice. Think about a DB where "file*,fd" is the key.
To clarify, the key appears to be file* plus the user-space integer that represents the fd.
Yes, that's what I said.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2)F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.Seem okay Davide?
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Looks sane to me.
I think fork(2) should not be in the above list. fork(2) duplicates the kernel's fd, but the user-space integer that represents the fd remains the same, so you will get EEXIST if you try to add the fd that was duplicated by fork.
Good catch, fork(2) should not be there.
Okay -- removed.
But it is an ugly inconsistency. On the one hand, a child process cannot add the duplicate file descriptor to the epoll set. (In every other case that I can think of , descriptors duplicated by fork have similar semantics to descriptors duplicated by dup() and friends.) On the other hand, the very fact that the child has a duplicate of the descriptor means that even if the parent closes its descriptor, then epoll_wait() in the parent will continue to receive notifications for that descriptor because of the duplicated descriptor in the child.
The choice of [file *, fd] as the key for epoll sets really does seem unfortunate. Keying on [pid, fd] would have given saner semantics, it seems to me. Obviously it can't be changed now though.
Cheers,
Michael
Signature · 5 lines
Michael Kerrisk Maintainer of the Linux man-pages project http://www.kernel.org/doc/man-pages/ Want to report a man-pages bug? Look here: http://www.kernel.org/doc/man-pages/reporting_bugs.html
Quoted 55 lines · Michael Kerrisk
Quoted 33 lines · Davide Libenzi
Quoted 8 lines · Chris "ã~B¯" Heath
Quoted 4 lines · Davide Libenzi
Yes, you can't add the same fd twice. Think about a DB where "file*,fd" is the key.
To clarify, the key appears to be file* plus the user-space integer that represents the fd.
Yes, that's what I said.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
c) It is possible to add duplicated file descriptors referring to the same underlying open file description ("file *"). As you note, this can be a useful filtering technique, if the two file descriptors specify different masks.
Assuming that is all correct, for man-pages-2.79, I've reworked the text for Q1/A1 as follows:
Q1 What happens if you add the same file descriptor
to an epoll set twice?A1 You will probably get EEXIST. However, it is pos-
sible to add a duplicate (dup(2), dup2(2),
fcntl(2)F_DUPFD, fork(2)) descriptor to the same
epoll set. This can be a useful technique for
filtering events, if the duplicate file descrip- tors are registered with different events masks.Seem okay Davide?
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Looks sane to me.
I think fork(2) should not be in the above list. fork(2) duplicates the kernel's fd, but the user-space integer that represents the fd remains the same, so you will get EEXIST if you try to add the fd that was duplicated by fork.
Good catch, fork(2) should not be there.
Okay -- removed.
But it is an ugly inconsistency. On the one hand, a child process cannot add the duplicate file descriptor to the epoll set. (In every other case that I can think of , descriptors duplicated by fork have similar semantics to descriptors duplicated by
dup()and friends.) On the other hand, the very fact that the child has a duplicate of the descriptor means that even if the parent closes its descriptor, thenepoll_wait()in the parent will continue to receive notifications for that descriptor because of the duplicated descriptor in the child.
Quoted 3 lines
The choice of [file *, fd] as the key for epoll sets really does seem unfortunate. Keying on [pid, fd] would have given saner semantics, it seems to me. Obviously it can't be changed now though.
Davide,
with the earlier discussion in this thread in mind, I added a Q0/A0 to epoll.7, just make the point about keys clear:
Q0 What is the key used to distinguish the file descrip-
tors in an epoll set?
A0 The key is the combination of the file descriptor
number and the open file description (also known as "open file handle", the kernel's internal representa- tion of an open file).
Does that seem okay?
Cheers,
Michael
Signature · 9 lines
Michael Kerrisk Maintainer of the Linux man-pages project http://www.kernel.org/doc/man-pages/ Want to report a man-pages bug? Look here: http://www.kernel.org/doc/man-pages/reporting_bugs.html -- To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Michael Kerrisk
Davide,
with the earlier discussion in this thread in mind, I added a Q0/A0 to epoll.7, just make the point about keys clear:
Q0 What is the key used to distinguish the file descrip-
tors in an epoll set?
Quoted 6 lines
A0 The key is the combination of the file descriptor
number and the open file description (also known as "open file handle", the kernel's internal representa- tion of an open file).Does that seem okay?
Looks fine to me! We need to clarify better Q6 WRT fork().
- Davide
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Michael Kerrisk
But it is an ugly inconsistency. On the one hand, a child process cannot add the duplicate file descriptor to the epoll set. (In every other case that I can think of , descriptors duplicated by fork have similar semantics to descriptors duplicated by
dup()and friends.) On the other hand, the very fact that the child has a duplicate of the descriptor means that even if the parent closes its descriptor, thenepoll_wait()in the parent will continue to receive notifications for that descriptor because of the duplicated descriptor in the child.
Have you ever tried to think what it means for different *processes* sharing a single epoll fd and doing epoll_wait() over it? Most common case is a single event fetch thread plus dispatch. Going to epoll_wait() over a single epoll fd from many *threads* is very much possible, but requires care (news at 11, system software development requires care too).
Sharing a single epoll fd (by the means of any process sharing it doing add/wait) from different *processes* makes almost no sense at all. "a child process cannot add the duplicate file descriptor to the epoll set" ... how do you expect the parent (that doesn't even have the new fd mapped) to react to such events?
If the next question is "But then why we made the epoll fd inheritable?", the answer is, because it makes sense in many cases for a parent to hand over an fd set to a child.
Quoted 3 lines
The choice of [file *, fd] as the key for epoll sets really does seem unfortunate. Keying on [pid, fd] would have given saner semantics, it seems to me. Obviously it can't be changed now though.
I think we already went over this, and I think I clearly explained you the reasons of not hooking into sys_close.
- Davide
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Davide,
Quoted 11 lines · Davide Libenzi
Quoted 8 lines · Michael Kerrisk
But it is an ugly inconsistency. On the one hand, a child process cannot add the duplicate file descriptor to the epoll set. (In every other case that I can think of , descriptors duplicated by fork have similar semantics to descriptors duplicated by
dup()and friends.) On the other hand, the very fact that the child has a duplicate of the descriptor means that even if the parent closes its descriptor, thenepoll_wait()in the parent will continue to receive notifications for that descriptor because of the duplicated descriptor in the child.
Quoted 2 lines
Have you ever tried to think what it means for different *processes* sharing a single epoll fd and doing
epoll_wait()over it?
As I think is clear, I've only given it very limited thought ;-).
The point is that the existing implementation actually supports "different *processes* sharing a single epoll fd and doing epoll_wait() over it", but the semantics are unintuitive. It may be that the existing implementation was the best way of doing things. But when I see the strange corner cases in the semantics, I can't help but wonder (way too late), whether there might have been some other way of implementing things that led to more intuitive semantics.
Quoted 10 lines
Most common case is a single event fetch thread plus dispatch. Going to
epoll_wait()over a single epoll fd from many *threads* is very much possible, but requires care (news at 11, system software development requires care too).
Sharing a single epoll fd (by the means of any process sharing it doing add/wait) from different *processes* makes almost no sense at all."a child process cannot add the duplicate file descriptor to the epoll
set" ... how do you expect the parent (that doesn't even have the new fd mapped) to react to such events?
(Not sure if you missed my meaning here. Of course the parent already has the fd mapped; it's the fd that the child inherited. Anyway, my real point was that while descriptors duplicated by fork() are normally semantically similar to other duplicated descriptors, when it comes to epoll they are not -- and that has the potential to surprise users.)
Quoted 3 lines
If the next question is "But then why we made the epoll fd inheritable?", the answer is, because it makes sense in many cases for a parent to hand over an fd set to a child.
Fair enough.
So here's an idea about how things might alternatively have been done:
a) The key for epoll entries could have been [file *, fd, PID]
b) an epoll_wait() only returns events for fds where the PID maps that of the caller.
c) a close of a file descriptor removes the corresponding [file *, fd, PID] from the epoll set.
d) when a fork() is done, then the epoll set has a new set of keys added. These are duplicates of the [file *, fd, PID] entries for the parent, but with the PID of the child substituted into the new keys. Say the parent had PID 1000, and the child has PID 2000. If the epoll set initially contained:
[X, 3, 1000]
[Y, 4, 1000]
then after fork() we'd have:
[X, 3, 1000]
[Y, 4, 1000]
[X, 3, 2000]
[Y, 4, 2000]
There is of course room for debate about the efficiency of this approach, I suppose. But it seems to me (and perhaps I've missed a number of things) that that could have given sane semantics with respect to fork(), duplicated descriptors, and close(). Furthermore, it would have allowed us to sanely support "different *processes* sharing a single epoll fd and doing epoll_wait() over it".
Of course, this is all academic now: we can't change the ABI.
Quoted 6 lines
Quoted 3 lines
The choice of [file *, fd] as the key for epoll sets really does seem unfortunate. Keying on [pid, fd] would have given saner semantics, it seems to me. Obviously it can't be changed now though.
I think we already went over this, and I think I clearly explained you the reasons of not hooking into
sys_close.
You said elsewhere:
[[
That'd mean placing an eventpoll custom hook into sys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing the f_op pointer), but a design/style issue. ]]
But that wasn't very clear to me actually. I note that filp_close() already has special case handling for dnotify (R.I.P.) and fcntl() )aka POSIX) file locks, so there was already precedent for a custom hook, AFAICS, and epoll is at least as worthy of special treatment as either of those cases.
Cheers,
Michael
Signature · 9 lines
Michael Kerrisk Maintainer of the Linux man-pages project http://www.kernel.org/doc/man-pages/ Want to report a man-pages bug? Look here: http://www.kernel.org/doc/man-pages/reporting_bugs.html -- To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Michael Kerrisk
As I think is clear, I've only given it very limited thought ;-).
The point is that the existing implementation actually supports "different *processes* sharing a single epoll fd and doing
epoll_wait()over it", but the semantics are unintuitive. It may be that the existing implementation was the best way of doing things. But when I see the strange corner cases in the semantics, I can't help but wonder (way too late), whether there might have been some other way of implementing things that led to more intuitive semantics.
Oh boy. The fact that you can have an epoll fd cross the fork boundary, does not mean that any indiscriminate use of it leads to sane results:
efd = epoll_create(); fork(); pipe(fds); epoll_ctl(efd, ADD, fds[0]); epoll_wait(); ???? ... pipe(fds); epoll_ctl(efd, ADD, fds[0]); epoll_wait(); ????
It is *NOT* a matter of semantics.
Quoted 27 lines
Quoted 3 lines
If the next question is "But then why we made the epoll fd inheritable?", the answer is, because it makes sense in many cases for a parent to hand over an fd set to a child.
Fair enough.
So here's an idea about how things might alternatively have been done:
a) The key for epoll entries could have been [file *, fd, PID]
b) an
epoll_wait()only returns events for fds where the PID maps that of the caller.c) a close of a file descriptor removes the corresponding [file *, fd, PID] from the epoll set.
d) when a
fork()is done, then the epoll set has a new set of keys added. These are duplicates of the [file *, fd, PID] entries for the parent, but with the PID of the child substituted into the new keys. Say the parent had PID 1000, and the child has PID 2000. If the epoll set initially contained:[X, 3, 1000]
[Y, 4, 1000]then after
fork()we'd have:
Quoted 7 lines
[X, 3, 1000]
[Y, 4, 1000]
[X, 3, 2000]
[Y, 4, 2000]There is of course room for debate about the efficiency of this approach, I suppose.
There sure is :)
Quoted 9 lines
You said elsewhere:
[[
That'd mean placing an eventpoll custom hook intosys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing thef_oppointer), but a design/style issue. ]]
Quoted 5 lines
But that wasn't very clear to me actually. I note that
filp_close()already has special case handling for dnotify (R.I.P.) andfcntl())aka POSIX) file locks, so there was already precedent for a custom hook, AFAICS, and epoll is at least as worthy of special treatment as either of those cases.
I guess that over the time, Al became software WRT junk going there :)
- Davide
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · Davide Libenzi
Quoted 9 lines · Michael Kerrisk
As I think is clear, I've only given it very limited thought ;-).
The point is that the existing implementation actually supports "different *processes* sharing a single epoll fd and doing
epoll_wait()over it", but the semantics are unintuitive. It may be that the existing implementation was the best way of doing things. But when I see the strange corner cases in the semantics, I can't help but wonder (way too late), whether there might have been some other way of implementing things that led to more intuitive semantics.
Quoted 2 lines
Oh boy. The fact that you can have an epoll fd cross the fork boundary, does not mean that any indiscriminate use of it leads to sane results:
I ddidn't mean that it did. Certainly in the current implementation it there will insane situations ;-).
Quoted 11 lines
efd = epoll_create();
fork();
pipe(fds);
epoll_ctl(efd, ADD, fds[0]);
epoll_wait(); ????
...
pipe(fds);
epoll_ctl(efd, ADD, fds[0]);
epoll_wait(); ????Quoted 1 line
It is *NOT* a matter of semantics.
Of course -- but I don't think I suggested that I disagree on this.
Quoted 32 lines
Quoted 32 lines
Quoted 3 lines
If the next question is "But then why we made the epoll fd inheritable?", the answer is, because it makes sense in many cases for a parent to hand over an fd set to a child.
Fair enough.
So here's an idea about how things might alternatively have been done:
a) The key for epoll entries could have been [file *, fd, PID]
b) an
epoll_wait()only returns events for fds where the PID maps that of the caller.c) a close of a file descriptor removes the corresponding [file *, fd, PID] from the epoll set.
d) when a
fork()is done, then the epoll set has a new set of keys added. These are duplicates of the [file *, fd, PID] entries for the parent, but with the PID of the child substituted into the new keys. Say the parent had PID 1000, and the child has PID 2000. If the epoll set initially contained:[X, 3, 1000]
[Y, 4, 1000]then after
fork()we'd have:[X, 3, 1000]
[Y, 4, 1000]
[X, 3, 2000]
[Y, 4, 2000]
Quoted 4 lines
Quoted 2 lines
There is of course room for debate about the efficiency of this approach, I suppose.
There sure is :)
Okay -- but I suspect it could have been made fairly efficient.
Quoted 9 lines
Quoted 9 lines
You said elsewhere:
[[
That'd mean placing an eventpoll custom hook intosys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing thef_oppointer), but a design/style issue. ]]
Quoted 7 lines
Quoted 5 lines
But that wasn't very clear to me actually. I note that
filp_close()already has special case handling for dnotify (R.I.P.) andfcntl())aka POSIX) file locks, so there was already precedent for a custom hook, AFAICS, and epoll is at least as worthy of special treatment as either of those cases.I guess that over the time, Al became software WRT junk going there :)
Sorry -- I don't understand that last sentence?
Hijacking this epoll thread, the following related question occurs to me:
#Q8
# Does an operation on a file descriptor affect the already collected but #not yet reported events?
#
#A8
# You can do two operations on an existing file descriptor. Remove would #be meaningless for this case. Modify will re-read available I/O.
Why is EPOLL_CTL_DEL considered meaningless? A process is wrapping up its business and is preparing to remove the fd from the epoll set, and then close the file descriptor itself. In the meantime, the fd became readable, and a POLLIN event gets collected. So, what happens to the collected event, when the EPOLL_CTL_DEL operation is made?
Quoted 10 lines · Sam Varshavchik
Hijacking this epoll thread, the following related question occurs to me:
#Q8
# Does an operation on a file descriptor affect the already collected but
#not yet reported events?
#
#A8
# You can do two operations on an existing file descriptor. Remove would
#be meaningless for this case. Modify will re-read available I/O.
Quoted 5 lines
Why is
EPOLL_CTL_DELconsidered meaningless? A process is wrapping up its business and is preparing to remove the fd from the epoll set, and then close the file descriptor itself. In the meantime, the fd became readable, and a POLLIN event gets collected. So, what happens to the collected event, when theEPOLL_CTL_DELoperation is made?
Any epoll_wait() done after the POLLIN and before the EPOLL_CTL_DEL, will show up. After the EPOLL_CTL_DEL, of course, no events will be reported.
- Davide
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · David Schwartz
I don't see how that can be. Suppose I add fd 8 to an epoll set. Suppose fd
5 is a dup of fd 8. Now, I close fd 8. How can fd 8 remain in my epoll set, since there no longer is an fd 8? Events on files registered for epoll notification are reported by descriptor, so the set membership has to be associated (as reflected into userspace) with the descriptor, not the file.
Eric already answered to your question (epoll deals with internal kernel objects - aka file*).
I just want to answer this one for another reason. WTF is wrong with all of you Cc-list-trimmers?
Could you *please* stop trimming Cc-lists?
- Davide