Skip to contentSkip to conversation

epoll and shared fd's

DISCUSSION Pierre Habouzit 4 participants 9 messages Jan 18, 2008 – Feb 26, 2008 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

I _believe_ this is if not a bug, at least a misfeature, hence I'm
reporting the issue :)

PS: please Cc: on answers me I'm not subscribed.

[0] and despite the bug in our software that leaked the socket, P1 is supposed to only close the socket when P2 acks the fact that it received a valid fd (else P1 tries to send it to a P2'), and there may be uncontrollable races that could trigger the issue again (with P2 closing the socket before P1 had time to process the ACK and close the socket on its end).

Signature · 3 lines
·O·  Pierre Habouzit
··O                                                [email protected]
OOO                                                http://www.madism.org
#1 lore raw
Quoted 17 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Quoted 8 lines

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

- Davide

#2 lore raw
Quoted 24 lines · Davide Libenzi
Quoted 22 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Quoted 8 lines
Quoted 3 lines

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Signature · 3 lines
·O·  Pierre Habouzit
··O                                                [email protected]
OOO                                                http://www.madism.org
#3 lore raw
Quoted 29 lines · Pierre Habouzit
Quoted 28 lines · Davide Libenzi
Quoted 25 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

Quoted 7 lines
Quoted 4 lines

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Sure. I'll send Michael Kerrisk and updated statement for the A6 answer in the epoll man page.

- Davide

#4 lore raw
Quoted 37 lines · Davide Libenzi
Quoted 34 lines · Pierre Habouzit
Quoted 32 lines · Davide Libenzi
Quoted 17 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Quoted 8 lines

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Quoted 5 lines
Quoted 2 lines

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Sure. I'll send Michael Kerrisk and updated statement for the A6 answer in the epoll man page.

Thanks Davide -- yes please send me a patch.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Following up after quite some time:

Quoted 69 lines · Davide Libenzi
Quoted 37 lines · Michael Kerrisk
Quoted 36 lines · Davide Libenzi
Quoted 34 lines · Pierre Habouzit
Quoted 24 lines · Davide Libenzi
Quoted 22 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Quoted 7 lines
Quoted 3 lines

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Quoted 9 lines
Quoted 2 lines

Sure. I'll send Michael Kerrisk and updated statement for the A6 answer in the epoll man page.

Thanks Davide -- yes please send me a patch.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Something like the one below ...

- Davide

epoll.4.new renamed from epoll.4 +6 -1
@@ -285,7 +285,19 @@
 sets automatically?
 .TP
 .B A6
-Yes.
+A file descriptor is the userspace counterpart of an internal kernel handle.
+Every time a process calls functions liks
+.BR dup (2),
+.BR dup2 (2)
+or
+.BR fork (2),

epoll.4.new line 303

Quoted 10 lines
epoll.4.new renamed from epoll.4 +7 -0
+a new file descriptor referring to the same internal kernel handle is
+created. The internal kernel handle remains alive until all the userspace
+file descriptors have been closed.
+The
+.BR epoll (4)
+interface automatically removes the internal kernel handle from the set,
+once all the file descriptor instances have been closed.
 .TP
 .B Q7
 If more than one event occurs between

Davide,

Two points.

a) I did a

s/internal kernel handle/open file description/

since that is the POSIX term for the internal handle.

b) It seems to me that you text doesn't quite make the point explicit enough. I've tried to rewrite it; could you please check:

A6 Yes, but be aware of the following point. A file
descriptor is a reference to an open file descrip-
tion (see open(2)). Whenever a descriptor is
duplicated via dup(2), dup2(2), fcntl(2) F_DUPFD, or fork(2), a new file descriptor referring to the same open file description is created. An open file description continues to exist until all file descriptors referring to it have been closed. The
epoll interface automatically removes a file
descriptor from an epoll set only after all the file descriptors referring to the underlying open file handle have been closed. This means that even after a file descriptor that is part of an epoll set has been closed, events may be reported for that file descriptor if other file descriptors referring to the same underlying file description remain open.

Does that seem okay? I plan to include the text in man-pages-2.79.

Was there some reason why removing a file descriptor couldn't have been made to do the "expected" thing (i.e., remove notifications for that file descriptor, regardless of whether the underlying file description remains open)?

Cheers,

Michael

Signature · 5 lines
Michael Kerrisk
Maintainer of the Linux man-pages project
http://www.kernel.org/doc/man-pages/
Want to report a man-pages bug?  Look here:
http://www.kernel.org/doc/man-pages/reporting_bugs.html
#7 lore raw
Quoted 114 lines · Michael Kerrisk

Following up after quite some time:

Quoted 69 lines · Davide Libenzi
Quoted 37 lines · Michael Kerrisk
Quoted 36 lines · Davide Libenzi
Quoted 34 lines · Pierre Habouzit
Quoted 24 lines · Davide Libenzi
Quoted 22 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Quoted 7 lines
Quoted 3 lines

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Quoted 9 lines
Quoted 2 lines

Sure. I'll send Michael Kerrisk and updated statement for the A6 answer in the epoll man page.

Thanks Davide -- yes please send me a patch.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Something like the one below ...

- Davide

epoll.4.new renamed from epoll.4 +6 -1
@@ -285,7 +285,19 @@
 sets automatically?
 .TP
 .B A6
-Yes.
+A file descriptor is the userspace counterpart of an internal kernel handle.
+Every time a process calls functions liks
+.BR dup (2),
+.BR dup2 (2)
+or
+.BR fork (2),

epoll.4.new line 303

Quoted 10 lines
epoll.4.new renamed from epoll.4 +7 -0
+a new file descriptor referring to the same internal kernel handle is
+created. The internal kernel handle remains alive until all the userspace
+file descriptors have been closed.
+The
+.BR epoll (4)
+interface automatically removes the internal kernel handle from the set,
+once all the file descriptor instances have been closed.
 .TP
 .B Q7
 If more than one event occurs between

Davide,

Two points.

a) I did a

s/internal kernel handle/open file description/

since that is the POSIX term for the internal handle.

b) It seems to me that you text doesn't quite make the point explicit enough. I've tried to rewrite it; could you please check:

A6 Yes, but be aware of the following point. A file
descriptor is a reference to an open file descrip-
tion (see open(2)). Whenever a descriptor is
duplicated via dup(2), dup2(2), fcntl(2) F_DUPFD, or fork(2), a new file descriptor referring to the same open file description is created. An open file description continues to exist until all file descriptors referring to it have been closed. The
epoll interface automatically removes a file
descriptor from an epoll set only after all the file descriptors referring to the underlying open file handle have been closed. This means that even after a file descriptor that is part of an epoll set has been closed, events may be reported for that file descriptor if other file descriptors referring to the same underlying file description remain open.

Quoted 1 line

Does that seem okay? I plan to include the text in man-pages-2.79.

I agree with Bodo, it is kinda confusing. The name "open file description", even though POSIX, looks very similar to "file descriptor".
I honestly don't know how more easily such concept could be expressed. IMHO at least "internal kernel handle" does not play look-alike games with "file descriptor".

Quoted 4 lines

Was there some reason why removing a file descriptor couldn't have been made to do the "expected" thing (i.e., remove notifications for that file descriptor, regardless of whether the underlying file description remains open)?

That'd mean placing an eventpoll custom hook into sys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing the f_op pointer), but a design/style issue. On top of that, the interface is already out by many years, so changing it will like going to cause problems.

- Davide

#8 lore raw
Quoted 117 lines · Davide Libenzi
Quoted 114 lines · Michael Kerrisk

Following up after quite some time:

Quoted 69 lines · Davide Libenzi
Quoted 37 lines · Michael Kerrisk
Quoted 36 lines · Davide Libenzi
Quoted 34 lines · Pierre Habouzit
Quoted 24 lines · Davide Libenzi
Quoted 22 lines · Pierre Habouzit

Hi,

I just came across a strange behavior of epoll that seems to
contradict the documentation. Here is what happens:

* I have two processes P1 and P2, P1 accept()s connections, and send the resulting file descriptors to P2 through a unix socket.

* P2 registers the received socket in his epollfd.

[time passes]

* P2 is done with the socket and closes it

* P2 gets events for the socket again !

Though the documentation says that if a process closes a file
descriptor, it gets unregistered. And yes I'm sure that P2 doens't dup() the file descriptor. Though (because of a bug) it was still open in P1[0], hence the referenced socket still live at the kernel level.

Quoted 7 lines
Quoted 3 lines

Of course the userland workaround is to force the EPOLL_CTL_DEL before
the close, which I now do, but costs me a syscall where I wanted to spare one :|

For epoll, a close is when the kernel file* is released (that is, when all its instances are gone).
We could put a special handling in filp_close(), but I don't think is a good idea, and we're better live with the current behaviour.

Okay, maybe updating the linux manpages to be more clear about that is
the way to go then. Thanks

Quoted 9 lines
Quoted 2 lines

Sure. I'll send Michael Kerrisk and updated statement for the A6 answer in the epoll man page.

Thanks Davide -- yes please send me a patch.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Something like the one below ...

- Davide

epoll.4.new 2008-01-26 13:06:36.000000000 -0800 renamed from epoll.4 2008-01-26 12:58:21.000000000 -0800 +6 -1
@@ -285,7 +285,19 @@
 sets automatically?
 .TP
 .B A6
-Yes.
+A file descriptor is the userspace counterpart of an internal kernel handle.
+Every time a process calls functions liks
+.BR dup (2),
+.BR dup2 (2)
+or
+.BR fork (2),

epoll.4.new 2008-01-26 13:06:36.000000000 -0800 line 303

Quoted 10 lines
epoll.4.new 2008-01-26 13:06:36.000000000 -0800 renamed from epoll.4 2008-01-26 12:58:21.000000000 -0800 +7 -0
+a new file descriptor referring to the same internal kernel handle is
+created. The internal kernel handle remains alive until all the userspace
+file descriptors have been closed.
+The
+.BR epoll (4)
+interface automatically removes the internal kernel handle from the set,
+once all the file descriptor instances have been closed.
 .TP
 .B Q7
 If more than one event occurs between

Davide,

Two points.

a) I did a

s/internal kernel handle/open file description/

since that is the POSIX term for the internal handle.

b) It seems to me that you text doesn't quite make the point explicit enough. I've tried to rewrite it; could you please check:

A6 Yes, but be aware of the following point. A file
descriptor is a reference to an open file descrip-
tion (see open(2)). Whenever a descriptor is
duplicated via dup(2), dup2(2), fcntl(2) F_DUPFD, or fork(2), a new file descriptor referring to the same open file description is created. An open file description continues to exist until all file descriptors referring to it have been closed. The
epoll interface automatically removes a file
descriptor from an epoll set only after all the file descriptors referring to the underlying open file handle have been closed. This means that even after a file descriptor that is part of an epoll set has been closed, events may be reported for that file descriptor if other file descriptors referring to the same underlying file description remain open.

Quoted 7 lines
Quoted 1 line

Does that seem okay? I plan to include the text in man-pages-2.79.

I agree with Bodo, it is kinda confusing. The name "open file description", even though POSIX, looks very similar to "file descriptor".
I honestly don't know how more easily such concept could be expressed. IMHO at least "internal kernel handle" does not play look-alike games with "file descriptor".

Okay -- I'll look at it some more. I am however loathe to drop the term open file description, because POSIX uses, as well as a number of other Linux man pages by now.

Quoted 11 lines
Quoted 4 lines

Was there some reason why removing a file descriptor couldn't have been made to do the "expected" thing (i.e., remove notifications for that file descriptor, regardless of whether the underlying file description remains open)?

That'd mean placing an eventpoll custom hook into sys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing the f_op pointer), but a design/style issue. On top of that, the interface is already out by many years, so changing it will like going to cause problems.

Oh -- I wasn't suggesting we could make the change now -- it would break the ABI and all that. I was just wondering why the decision wasn't made to do it the other way to begin with. The existing semantics are somewhat couterintuitive, and potentially interact libraries that do private manipulations with file descriptors.

Cheers,

Michael

Signature · 5 lines
Michael Kerrisk
Maintainer of the Linux man-pages project
http://www.kernel.org/doc/man-pages/
Want to report a man-pages bug?  Look here:
http://www.kernel.org/doc/man-pages/reporting_bugs.html
#9 lore raw
Quoted 3 lines · Michael Kerrisk

Okay -- I'll look at it some more. I am however loathe to drop the term open file description, because POSIX uses, as well as a number of other Linux man pages by now.

Heh, POSIX. Now doesn't take a genius to see that "file description" and "file descriptor" looks amazingly similar, does it? :)

Quoted 11 lines
Quoted 5 lines

That'd mean placing an eventpoll custom hook into sys_close(). Looks very bad to me, and probably will look even worse to other kernel folks. Is not much a performance issue (a check to see if a file* is an eventpoll file is as easy as comparing the f_op pointer), but a design/style issue.

Oh -- I wasn't suggesting we could make the change now -- it would break the ABI and all that. I was just wondering why the decision wasn't made to do it the other way to begin with. The existing semantics are somewhat couterintuitive, and potentially interact libraries that do private manipulations with file descriptors.

For the same reason that a custom hook in sys_close wouldn't have passed the radar ;)
As far as problems with libraries doing tricks with fds, that's an issue that goes beyond epoll.

- Davide