epoll and closed file descriptors
I am running repeatedly into a scenario where epoll notifies userland of events on a closed file descriptor.
I am running a single thread application, on a single CPU machine so multiple threads isn't the issue.
A sample set of events that I have seen
- File descriptor (13) for a socket is closed
- epoll_wait returns with no events.
- Several epoll related calls happen
- More than 20 seconds after the "close", epoll_wait finds an event on fd 13 with EPOLLIN|EPOLLERR|EPOLLHUP.
- epoll_wait continues to report this event
Running kernel 2.6.24. Some technical problems are preventing me from trying a newer kernel at the moment.
One more thing worth mentioning: the application uses libcurl, leading to a situation where the file is closed before the descriptor was removed from the epoll descriptor. The code should be able to handle that AFAIK.
Any ideas would be appreciated.
Gilad
Quoted 5 lines · Gilad Benjamini
I am running repeatedly into a scenario where epoll notifies userland of events on a closed file descriptor.
I am running a single thread application, on a single CPU machine so multiple threads isn't the issue.
Quoted 7 lines
A sample set of events that I have seen
- File descriptor (13) for a socket is closed
-epoll_waitreturns with no events.
- Several epoll related calls happen
- More than 20 seconds after the "close",epoll_waitfinds an event on fd 13 with EPOLLIN|EPOLLERR|EPOLLHUP.
-epoll_waitcontinues to report this event
Epoll removes the fd from its container, when the last instance of the underlying kernel file pointer is released (or when you explicitly remove it with epoll_ctl(EPOLL_CTL_DEL)).
If you continue to get the event, it means that someone else has an instance of the socket (that, looking at the events, saw a shutdown) open, by hence keeping the kernel object alive.
If you don't want to see the events, just remove the socket from the epoll set before closing.
Or, you remove the socket the first time you see an EPOLLHUP.
- Davide
Quoted 31 lines · Davide
Quoted 1 line · Gilad Benjamini
I am running repeatedly into a scenario where epoll notifies
userland of
Quoted 9 lines
events on a closed file descriptor.
I am running a single thread application, on a single CPU machine so multiple threads isn't the issue.A sample set of events that I have seen
- File descriptor (13) for a socket is closed
-epoll_waitreturns with no events.
- Several epoll related calls happen
- More than 20 seconds after the "close",epoll_waitfinds an eventon fd 13
Quoted 2 lines
with EPOLLIN|EPOLLERR|EPOLLHUP.
-epoll_waitcontinues to report this eventEpoll removes the fd from its container, when the last instance of the underlying kernel file pointer is released (or when you explicitly remove it with
epoll_ctl(EPOLL_CTL_DEL)).
If you continue to get the event, it means that someone else has an instance of the socket (that, looking at the events, saw a shutdown) open,
by hence keeping the kernel object alive.
If you don't want to see the events, just remove the socket from the epoll
set before closing.
Or, you remove the socket the first time you see an EPOLLHUP.
Quoted 1 line
- Davide
I would, but epoll is preventing me from doing so. Early in sys_epoll_ctl there are these lines
file = fget(epfd);
if (!file)
goto error_return;Leaving me in a kind of dead lock
Gilad
Quoted 34 lines · Gilad Benjamini
Quoted 31 lines · Davide
Quoted 1 line · Gilad Benjamini
I am running repeatedly into a scenario where epoll notifies
userland of
Quoted 9 lines
events on a closed file descriptor.
I am running a single thread application, on a single CPU machine so multiple threads isn't the issue.A sample set of events that I have seen
- File descriptor (13) for a socket is closed
-epoll_waitreturns with no events.
- Several epoll related calls happen
- More than 20 seconds after the "close",epoll_waitfinds an eventon fd 13
Quoted 2 lines
with EPOLLIN|EPOLLERR|EPOLLHUP.
-epoll_waitcontinues to report this eventEpoll removes the fd from its container, when the last instance of the underlying kernel file pointer is released (or when you explicitly remove it with
epoll_ctl(EPOLL_CTL_DEL)).
If you continue to get the event, it means that someone else has an instance of the socket (that, looking at the events, saw a shutdown) open,
by hence keeping the kernel object alive.
If you don't want to see the events, just remove the socket from the epoll
set before closing.
Or, you remove the socket the first time you see an EPOLLHUP.Quoted 1 line
- Davide
Quoted 8 lines
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesfile = fget(epfd);
if (!file)
gotoerror_return;Leaving me in a kind of dead lock
You could dup() the file descriptor before handing it to curl, and use your own copy for epoll operations.
Quoted 8 lines · Gilad Benjamini
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesfile = fget(epfd); if (!file) goto error_return;Leaving me in a kind of dead lock
The 'epfd' in there, is the _epoll fd_, which, if fget() fails, means you close it.
You see likely failing the 'tfile = fget(fd)' (of course, you closed it), so if someone else keeps the socket open and you have no chance in telling it to drop it (really?), you need to remove the socket from the set before closing it.
- Davide
Quoted 24 lines · Davide
Quoted 8 lines · Gilad Benjamini
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesfile = fget(epfd); if (!file) goto error_return;Leaving me in a kind of dead lock
The 'epfd' in there, is the
_epollfd_, which, iffget()fails, means you
close it.
You see likely failing the 'tfile = fget(fd)' (of course, you closed it),
so if someone else keeps the socket open and you have no chance in telling
it to drop it (really?), you need to remove the socket from the set before
closing it.
Quoted 1 line
- Davide
My bad. I meant to quote the line that you mentioned.
I agree that the right thing to do is to remove the fd from epoll before closing it.
However, due to the way curl works, I cannot do that. Changing the curl code doesn't seem trivial.
Regardless, I still don't see how the kernel got into this situation, and if this situation is valid, why it doesn't bail out of it.
Quoted 27 lines · Gilad Benjamini
Quoted 24 lines · Davide
Quoted 8 lines · Gilad Benjamini
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesfile = fget(epfd);
if (!file)
gotoerror_return;Leaving me in a kind of dead lock
The 'epfd' in there, is the
_epollfd_, which, iffget()fails, means you
close it.
You see likely failing the 'tfile = fget(fd)' (of course, you closed it),
so if someone else keeps the socket open and you have no chance in telling
it to drop it (really?), you need to remove the socket from the set before
closing it.Quoted 1 line
- Davide
Quoted 8 lines
My bad. I meant to quote the line that you mentioned.
I agree that the right thing to do is to remove the fd from epoll before closing it.
However, due to the way curl works, I cannot do that. Changing the curl code doesn't seem trivial.Regardless, I still don't see how the kernel got into this situation, and if this situation is valid, why it doesn't bail out of it.
epoll references the underlying file object; the fd is used _only_ to obtain this file object, and then never used again. Determining when the fd goes away then requires iterating over all fds, and since epoll was designed to avoid doing exactly that, it isn't an acceptable solution.
As I mentioned in the other email, by dup()ing the file descriptor, you can get control over when the copy is closed without changing curl. Then just make sure to remove it from the epoll set before closing your copy.
Quoted 41 lines
Subject: Re: epoll and closed file descriptors
Quoted 6 lines · Gilad Benjamini
Quoted 5 lines · Davide
Quoted 3 lines · Gilad Benjamini
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
file = fget(epfd);
if (!file)
gotoerror_return;Leaving me in a kind of dead lock
The 'epfd' in there, is the
_epollfd_, which, iffget()fails,means
Quoted 12 lines
Quoted 12 lines
you
close it.
You see likely failing the 'tfile = fget(fd)' (of course, you closed it),
so if someone else keeps the socket open and you have no chance in telling
it to drop it (really?), you need to remove the socket from the set before
closing it.Quoted 4 lines
Quoted 1 line
- Davide
My bad. I meant to quote the line that you mentioned.
I agree that the right thing to do is to remove the fd from epollbefore
Quoted 2 lines
closing it.
However, due to the way curl works, I cannot do that. Changing thecurl code
Quoted 3 lines
doesn't seem trivial.
Regardless, I still don't see how the kernel got into this situation,
Quoted 8 lines
and if
Quoted 1 line
this situation is valid, why it doesn't bail out of it.
epoll references the underlying file object; the fd is used _only_ to obtain this file object, and then never used again. Determining when the fd goes away then requires iterating over all fds, and since epoll was designed to avoid doing exactly that, it isn't an acceptable solution.
Regarding bailing out of the situation, I see the logic in your answer. What about the first part ? Any ideas how the kernel actually got into that tight spot ?
Looking into the code I can't find a path that can lead into this situation.
Quoted 5 lines · Gilad Benjamini
My bad. I meant to quote the line that you mentioned.
I agree that the right thing to do is to remove the fd from epoll before closing it.
However, due to the way curl works, I cannot do that. Changing the curl code doesn't seem trivial.
You probably need to explain how you use libcurl together with epoll, even though this is not a help-userspace mailing list.
Quoted 2 lines
Regardless, I still don't see how the kernel got into this situation, and if this situation is valid, why it doesn't bail out of it.
I think this was already explained.
- Davide
2009/9/16 Gilad Benjamini <[email protected]>:
Quoted 50 lines
Quoted 41 lines
Subject: Re: epoll and closed file descriptors
Quoted 6 lines · Gilad Benjamini
Quoted 5 lines · Davide
Quoted 3 lines · Gilad Benjamini
I would, but epoll is preventing me from doing so. Early in
sys_epoll_ctlthere are these linesQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
file = fget(epfd);
if (!file)
gotoerror_return;Leaving me in a kind of dead lock
The 'epfd' in there, is the
_epollfd_, which, iffget()fails,means
Quoted 12 lines
Quoted 12 lines
you
close it.
You see likely failing the 'tfile = fget(fd)' (of course, you closed it),
so if someone else keeps the socket open and you have no chance in telling
it to drop it (really?), you need to remove the socket from the set before
closing it.Quoted 4 lines
Quoted 1 line
- Davide
My bad. I meant to quote the line that you mentioned.
I agree that the right thing to do is to remove the fd from epollbefore
Quoted 2 lines
closing it.
However, due to the way curl works, I cannot do that. Changing thecurl code
Quoted 3 lines
doesn't seem trivial.
Regardless, I still don't see how the kernel got into this situation,
Quoted 8 lines
and if
Quoted 1 line
this situation is valid, why it doesn't bail out of it.
epoll references the underlying file object; the fd is used _only_ to obtain this file object, and then never used again. Determining when the fd goes away then requires iterating over all fds, and since epoll was designed to avoid doing exactly that, it isn't an acceptable solution.
Quoted 4 lines
Regarding bailing out of the situation, I see the logic in your answer. What about the first part ? Any ideas how the kernel actually got into that tight spot ?
Looking into the code I can't find a path that can lead into this situation.
Userspace passes in a fd that is unused (closed). Kernel can't find the file object corresponding to the fd (because the fd went away when it wasn't looking), so it says to userspace, "Sorry, no such file!". And it's completely correct.
So, the basic problem is userspace becomes unable to refer to the file object. The kernel's doing just fine; it's not in a "tight spot" at all. It's perfectly happy to refer to the file object by a direct pointer to it - it's just userspace is unable to tell the kernel to remove it from the epoll, because the only name userspace had for it has been removed.