Skip to contentSkip to conversation

unexpected extra pollout events from epoll

DISCUSSION Paul P 2 participants 4 messages Oct 26, 2008

lore
Share & tools
lore

Paul P

author
#0 lore raw

I am programming a server using the epoll interface and have the receive portion of the server working fine, but for some reason as I implement the send portion, I noticed a few things that seem like strange behaviors in the implementation of epoll in the kernel.

I'm running Opensuse 11 and it has a 2.6.25 kernel.

The behavior that I can seeing is when I do a full read on an edge triggered fd, for some reason, it seems to be triggering an epollout event after each loop of the read events on a socket. (before I've done any writes at all to the socket)

This is very strange behavior as I would expect that the epollout event would only be triggered if I did a write and the socket recieved an ack which cleared out the send buffer.

The documentation on epollout is really sparse, so any help at all from the list would be very much appreciated. Do I need to manually arm the epollout flag after a write? I thought this was only necessary for level triggered epoll.

I was hoping someone more knowledgeable on the subject here might be able to help explain the epollout behavior and whether or not the extra events are normal and if so, what is the traditional way to handle these extra events in an edge triggered scenario.

Thanks!

Paul

#1 lore raw
Quoted 9 lines · Paul P

I am programming a server using the epoll interface and have the receive portion of the server working fine, but for some reason as I implement the send portion, I noticed a few things that seem like strange behaviors in the implementation of epoll in the kernel.

I'm running Opensuse 11 and it has a 2.6.25 kernel.

The behavior that I can seeing is when I do a full read on an edge triggered fd, for some reason, it seems to be triggering an epollout event after each loop of the read events on a socket. (before I've done any writes at all to the socket)

Quoted 8 lines

This is very strange behavior as I would expect that the epollout event would only be triggered if I did a write and the socket recieved an ack which cleared out the send buffer.

The documentation on epollout is really sparse, so any help at all from the list would be very much appreciated. Do I need to manually arm the epollout flag after a write? I thought this was only necessary for level triggered epoll.

The way epoll works, is by hooking into the existing kernel poll subsystem. It hooks into the poll wakeups, via callback, and it that way it knows that "something" is changed. Then it reads the status of a file via f_op->poll() to know the status.
What happens is that, if you listen for EPOLLIN|EPOLLOUT, when a packet arrives the callback hook is hit, and the file is put into a maybe-ready list. Maybe-ready because at the time of the callback, epoll has no clue of what happened.
After that, via epoll_wait(), f_op->poll() is called to get the status of the file, and since POLLIN|POLLOUT is returned (and since you're listening for EPOLLIN|EPOLLOUT), that gets reported back to you.
The POLLOUT event, by meaning a buffer-full->buffer-avail transition, did not really happen, but since POLLOUT is true, that gets reported back too. This, again, since epoll has no clue of what happened at callback hit time. I'm working on changes that will make epoll aware (by using the existing support for the "key" parameter of wakeups) of events at callback time, but this is something that is still up for discussion and definitely won't be in .28.
The best way to do it ATM, is to wait for POLLOUT only when really needed.

- Davide

author
#2 lore raw
Quoted 1 line

After that, via epoll_wait(), f_op->poll() is called to get the status of the file, and since POLLIN|POLLOUT is returned (and since you're listening for EPOLLIN|EPOLLOUT), that gets reported back to you. The POLLOUT event, by meaning a buffer-full->buffer-avail transition, did not really happen, but since POLLOUT is true, that gets reported back too.

Ok, so make sure I understand you correctly, you're saying that currently the kernel doesn't have awareness of the difference between EPOLLIN and EPOLLOUT events because at the time of the event, both EPOLLIN/EPOLLOUT are returned from the kernel and that at least for the near term that's not going to change. At some point, we can expect the EPOLLOUT to give the correct event, but not till later than .28.

Quoted 2 lines

The best way to do it ATM, is to wait for POLLOUT only when really needed.

I'm a little unclear how to do this. If I set the epoll_wait call to wait for just epollin events, that's fine. But when I send a large buffer of data and use epoll_ctl to look for epollin|epollout events, don't I have the same problem?

Let's say I'm sending a large buffer of data and I arm the fd to epollin|epollout (I'm adding an epollin flag because a message could come in while I'm sending)

If an event gets triggered on an fd, then I have no way of knowing if the event is from the socket being available to send data or if there is data waiting to be received since the epollin|epollout flag could be either one. So what am I to do when I get an event?

Are you saying that I can't do sending and receiving simultaneously with epoll? If that's the case, then is everyone simply setting the epollout flag when sending and ignoring the possibility of data coming in while data is being sent?

I didn't want to have to manually set fd's with epoll_ctl, but now I guess the epoll_one_shot flag makes more sense.

Paul

#3 lore raw

[Can you try to trim lines at 80 chars or so?]

Quoted 7 lines · Paul P
Quoted 6 lines

After that, via epoll_wait(), f_op->poll() is called to get the status of the file, and since POLLIN|POLLOUT is returned (and since you're listening for EPOLLIN|EPOLLOUT), that gets reported back to you. The POLLOUT event, by meaning a buffer-full->buffer-avail transition, did not really happen, but since POLLOUT is true, that gets reported back too.

Quoted 6 lines

Ok, so make sure I understand you correctly, you're saying that currently the kernel doesn't have awareness of the difference between EPOLLIN and EPOLLOUT events because at the time of the event, both EPOLLIN/EPOLLOUT are returned from the kernel and that at least for the near term that's not going to change. At some point, we can expect the EPOLLOUT to give the correct event, but not till later than .28.

The kernel knows the difference between EPOLLIN and EPOLLOUT, of course. At the moment though, such condition is not reported during wakeups, and this is what is going to be changing.

Quoted 7 lines
Quoted 2 lines

The best way to do it ATM, is to wait for POLLOUT only when really needed.

I'm a little unclear how to do this. If I set the epoll_wait call to wait for just epollin events, that's fine. But when I send a large buffer of data and use epoll_ctl to look for epollin|epollout events, don't I have the same problem?

You do that by writing data until it's finished, or you get EAGAIN. If you get EAGAIN, you listen for EPOLLOUT.
Reading is same, but you'd wait for EPOLLIN.

- Davide