epoll: add exclusive wakeups flag

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Thanks,

-Jason

Sample epoll_clt text:

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

Hi,

Quoted 47 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

Quoted 3 lines

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the exclusive adds will as well.

Quoted 5 lines

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect on epoll sets connected to fd that do not specify it.

Quoted 6 lines

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one would need to either re-arm the thread that set the 'events' field to 0 (by setting back to non-zero), or re-arm in at least one other thread via EPOLL_CTL_MOD (or delete and add).

Quoted 3 lines

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

So I was just trying to say that we return -EINVAL if you try to do and EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd returned via epoll_create().

Hello Jason,

Quoted 50 lines · Jason Baron

Hi,

Quoted 47 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

Quoted 6 lines

Quoted 3 lines

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the exclusive adds will as well.

So is it fair to say that the expected use case is that all epoll sets would use EPOLLEXCLUSIVE?

Quoted 18 lines

Quoted 5 lines

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect on epoll sets connected to fd that do not specify it.

Quoted 6 lines

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

Quoted 8 lines

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one would need to either re-arm the thread that set the 'events' field to 0 (by setting back to non-zero), or re-arm in at least one other thread via EPOLL_CTL_MOD (or delete and add).

Okay -- so when an EPOLLEXCLUSIVE FD becomes disarmed it is possible to re-enable rith EPOLL_CTL_MOD; one doesn't need to delete and re-add the FD.

Quoted 4 lines

Quoted 3 lines

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

Quoted 3 lines

So I was just trying to say that we return -EINVAL if you try to do and EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd returned via epoll_create().

Okay -- that definitely belongs in the man page.

I'll work up a text, but would like to get input about the "use case" question above.

Hello Jason,

Quoted 50 lines · Jason Baron

Hi,

Quoted 47 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

Quoted 6 lines

Quoted 3 lines

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the exclusive adds will as well.

So is it fair to say that the expected use case is that all epoll sets would use EPOLLEXCLUSIVE?

Quoted 18 lines

Quoted 5 lines

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect on epoll sets connected to fd that do not specify it.

Quoted 6 lines

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

Quoted 8 lines

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one would need to either re-arm the thread that set the 'events' field to 0 (by setting back to non-zero), or re-arm in at least one other thread via EPOLL_CTL_MOD (or delete and add).

Okay -- so when an EPOLLEXCLUSIVE FD becomes disarmed it is possible to re-enable rith EPOLL_CTL_MOD; one doesn't need to delete and re-add the FD.

Quoted 7 lines

Quoted 3 lines

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

So I was just trying to say that we return -EINVAL if you try to do and EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd returned via epoll_create().

Okay -- that definitely belongs in the man page.

I'll work up a text, but would like to get input about the "use case" question above.

Cheers,

Michael

Ok, here's some updated text:

EPOLLEXCLUSIVE

Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. When a wakeup event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event.

The events supported by EPOLLEXCLUSIVE are: EPOLLIN, EPOLLOUT, EPOLLERR, EPOLLHUP, EPOLLWAKEUP, and EPOLLET. epoll_wait(2) will always wait for EPOLLERR and EPOLLHUP; it is not necessary to set it in events. If EPOLLEXCLUSIVE is set using epoll_ctl(2), then a subsequent EPOLL_CTL_MOD on the same epfd, fd pair will retrun -EINVAL. An epoll_ctl(2) that specifies EPOLLEXCLUSIVE in events and specifies the target file descriptor fd as an epoll instance will return -EINVAL as well.

Hi Michael,

Quoted 101 lines · Michael Kerrisk (man-pages)

Hello Jason,

Quoted 50 lines · Jason Baron

Hi,

Quoted 47 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

Quoted 6 lines

Quoted 3 lines

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the exclusive adds will as well.

So is it fair to say that the expected use case is that all epoll sets would use EPOLLEXCLUSIVE?

Quoted 18 lines

Quoted 5 lines

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect on epoll sets connected to fd that do not specify it.

Quoted 6 lines

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

Quoted 8 lines

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one would need to either re-arm the thread that set the 'events' field to 0 (by setting back to non-zero), or re-arm in at least one other thread via EPOLL_CTL_MOD (or delete and add).

Okay -- so when an EPOLLEXCLUSIVE FD becomes disarmed it is possible to re-enable rith EPOLL_CTL_MOD; one doesn't need to delete and re-add the FD.

Quoted 7 lines

Quoted 3 lines

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

So I was just trying to say that we return -EINVAL if you try to do and EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd returned via epoll_create().

Quoted 11 lines

Okay -- that definitely belongs in the man page.

I'll work up a text, but would like to get input about the "use case" question above.

Cheers,

Michael

Ok, here's some updated text:

EPOLLEXCLUSIVE

Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. When a wakeup event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event.

The events supported by EPOLLEXCLUSIVE are: EPOLLIN, EPOLLOUT, EPOLLERR, EPOLLHUP, EPOLLWAKEUP, and EPOLLET. epoll_wait(2) will always wait for EPOLLERR and EPOLLHUP; it is not necessary to set it in events. If EPOLLEXCLUSIVE is set using epoll_ctl(2), then a subsequent EPOLL_CTL_MOD on the same epfd, fd pair will retrun -EINVAL. An epoll_ctl(2) that specifies EPOLLEXCLUSIVE in events and specifies the target file descriptor fd as an epoll instance will return -EINVAL as well.

Quoted 127 lines · Jason Baron

Hi Michael,

Quoted 101 lines · Michael Kerrisk (man-pages)

Hello Jason,

Quoted 50 lines · Jason Baron

Hi,

Quoted 47 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 13 lines · Jason Baron

Hi,

Re-post of an old series addressing thundering herd issues when sharing an event source fd amongst multiple epoll fds. Last posting was here for reference: https://lkml.org/lkml/2015/2/25/56

The patch herein drops the core scheduler 'rotate' changes I had previously proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Quoted 5 lines

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points that are not quite clear to me.

Quoted 8 lines

EPOLLEXCLUSIVE
Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. Thus, when an event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event. EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6) epoll FDs, and some (say 3) of those attachments were done using EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE will receive an event.

Quoted 6 lines

Quoted 3 lines

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the exclusive adds will as well.

So is it fair to say that the expected use case is that all epoll sets would use EPOLLEXCLUSIVE?

Quoted 18 lines

Quoted 5 lines

I suppose one point I'm trying to uncover in the above is: what is the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's FD, or is it setting an attribute in the epoll "interest list" record for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect on epoll sets connected to fd that do not specify it.

Quoted 6 lines

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with the 'events' field set to 0)?

Quoted 8 lines

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one would need to either re-arm the thread that set the 'events' field to 0 (by setting back to non-zero), or re-arm in at least one other thread via EPOLL_CTL_MOD (or delete and add).

Okay -- so when an EPOLLEXCLUSIVE FD becomes disarmed it is possible to re-enable rith EPOLL_CTL_MOD; one doesn't need to delete and re-add the FD.

Quoted 7 lines

Quoted 3 lines

(2) The source code contains a comment "we do not currently supported nested exclusive wakeups". Could you elaborate on this point? It sounds like something that should be documented.

So I was just trying to say that we return -EINVAL if you try to do and EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd returned via epoll_create().

Quoted 11 lines

Okay -- that definitely belongs in the man page.

I'll work up a text, but would like to get input about the "use case" question above.

Cheers,

Michael

Ok, here's some updated text:

EPOLLEXCLUSIVE

Sets an exclusive wakeup mode for the epfd file descriptor that is being attached to the target file descriptor, fd. When a wakeup event occurs and multiple epfd file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more epfds will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epfds to receive an event.

Quoted 8 lines

The events supported by EPOLLEXCLUSIVE are: EPOLLIN, EPOLLOUT, EPOLLERR, EPOLLHUP, EPOLLWAKEUP, and EPOLLET. epoll_wait(2) will always wait for EPOLLERR and EPOLLHUP; it is not necessary to set it in events. If EPOLLEXCLUSIVE is set using epoll_ctl(2), then a subsequent EPOLL_CTL_MOD on the same epfd, fd pair will retrun -EINVAL. An epoll_ctl(2) that specifies EPOLLEXCLUSIVE in events and specifies the target file descriptor fd as an epoll instance will return -EINVAL as well.

By the way, in the code you have

        case EPOLL_CTL_MOD:
                if (epi) { 
                        if (!(epi->event.events & EPOLLEXCLUSIVE)) {
                                epds.events |= POLLERR | POLLHUP;
                                error = ep_modify(ep, epi, &epds);
                        }

I think the "if" here is redundant. IIUC, earlier in the code you disallow EPOLL_CTL_MOD with EPOLLEXCLUSIVE.

Cheers,

Michael

Quoted 9 lines

By the way, in the code you have

        case EPOLL_CTL_MOD:
                if (epi) {
                        if (!(epi->event.events & EPOLLEXCLUSIVE)) {
                                epds.events |= POLLERR | POLLHUP;
                                error = ep_modify(ep, epi, &epds);
                        }

Quoted 8 lines

I think the "if" here is redundant. IIUC, earlier in the code you disallow EPOLL_CTL_MOD with EPOLLEXCLUSIVE.

Cheers,

Michael

Hi Michael,

So the previous check ensures that you can not add the EPOLLEXCLUSIVE flag to the events via an EPOLL_CTL_MOD operation, where EPOLLEXCLUSIVE may not be the existing events set. While this check here ensure you can't modify an existing set that already has the EPOLLEXCLUSIVE flag.

Quoted 2 lines · Michael Kerrisk (man-pages)

Quoted 1 line · Jason Baron

On 03/11/2016 03:30 PM, Michael Kerrisk (man-pages) wrote:

Hi Michael,

Looks good. One comment below.

Thanks,

Quoted 19 lines

EPOLLEXCLUSIVE (since Linux 4.5)
Sets an exclusive wakeup mode for the epoll file descriptor that is being attached to the target file descriptor, fd. When a wakeup event occurs and multiple epoll file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epoll file descriptors to receive an event. EPOLLEXCLUSIVE is thus useful for avoiding thun‐ dering herd problems in certain scenarios.

If the same file descriptor is in multiple epoll instances, some with the EPOLLEXCLUSIVE flag, and others
without, then events will provided to all epoll
instances that did not specify EPOLLEXCLUSIVE, and at least one of the epoll instances that did specify EPOLLEXCLUSIVE.

Quoted 4 lines

The following values may be specified in conjunction with EPOLLEXCLUSIVE: EPOLLIN, EPOLLOUT, EPOLLWAKEUP, and EPOLLET. EPOLLHUP and EPOLLERR can also be specified, but are ignored (as usual). Attempts to specify other

I'm not sure 'ignored' is the right wording here. 'EPOLLHUP' and 'EPOLERR' are always included in the set of events when something is added as EPOLLEXCLUSIVE. This is consistent with the non-EPOLLEXCLUSIVE add case.

So 'EPOLLHUP' and 'EPOLERR' may be specified but will be included in the set of events on an add, whether they are specified or not.

Quoted 5 lines

values in events yield an error. EPOLLEXCLUSIVE may be used only in an EPOLL_CTL_ADD operation; attempts to
employ it with EPOLL_CTL_MOD yield an error. If
EPOLLEXCLUSIVE has set using epoll_ctl(2), then a subse‐ quent EPOLL_CTL_MOD on the same epfd, fd pair yields an

Quoted 8 lines

events and specifies the target file descriptor fd as an epoll instance will likewise fail. The error in all of these cases is EINVAL.

ERRORS
EINVAL An invalid event type was specified along with EPOLLEX‐
CLUSIVE in events.

Quoted 7 lines

EINVAL op was EPOLL_CTL_MOD and events included EPOLLEXCLUSIVE.

EINVAL op was EPOLL_CTL_MOD and the EPOLLEXCLUSIVE flag has
previously been applied to this epfd, fd pair.

EINVAL EPOLLEXCLUSIVE was specified in event and fd is refers
to an epoll instance.

[Restoring CC, which I see I accidentally dropped, one iteration back.]

Hi Jason,

Thanks for the review. I've tweaked one piece to respond to your feedback. But I also have another new question below.

Quoted 3 lines · Jason Baron

Quoted 2 lines · Michael Kerrisk (man-pages)

Quoted 1 line · Jason Baron

On 03/11/2016 03:30 PM, Michael Kerrisk (man-pages) wrote:

[...]

Quoted 30 lines

Hi Michael,

Looks good. One comment below.

Thanks,

Quoted 19 lines

EPOLLEXCLUSIVE (since Linux 4.5)
Sets an exclusive wakeup mode for the epoll file descriptor that is being attached to the target file descriptor, fd. When a wakeup event occurs and multiple epoll file descriptors are attached to the same target file using EPOLLEXCLUSIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2). The default in this scenario (when EPOLLEXCLUSIVE is not set) is for all epoll file descriptors to receive an event. EPOLLEXCLUSIVE is thus useful for avoiding thun‐ dering herd problems in certain scenarios.

If the same file descriptor is in multiple epoll instances, some with the EPOLLEXCLUSIVE flag, and others
without, then events will provided to all epoll
instances that did not specify EPOLLEXCLUSIVE, and at least one of the epoll instances that did specify EPOLLEXCLUSIVE.

Quoted 4 lines

The following values may be specified in conjunction with EPOLLEXCLUSIVE: EPOLLIN, EPOLLOUT, EPOLLWAKEUP, and EPOLLET. EPOLLHUP and EPOLLERR can also be specified, but are ignored (as usual). Attempts to specify other

Quoted 4 lines

I'm not sure 'ignored' is the right wording here. 'EPOLLHUP' and 'EPOLERR' are always included in the set of events when something is added as EPOLLEXCLUSIVE. This is consistent with the non-EPOLLEXCLUSIVE add case.

Yes.

Quoted 2 lines

So 'EPOLLHUP' and 'EPOLERR' may be specified but will be included in the set of events on an add, whether they are specified or not.

Yes. I understand your discomfort with the work "ignored", but the problem was that, because it made special mention of EPOLLHUP and EPOLLERR, your proposed text made it sound as though EPOLLEXCLUSIVE somehow was special with respect to these two flags. I wanted to clarify that it is not. How about this:

The following values may be specified in conjunction with EPOLLEXCLUSIVE: EPOLLIN, EPOLLOUT, EPOLLWAKEUP, and EPOLLET. EPOLLHUP and EPOLLERR can also be specified, but this is not required: as usual, these events are always reported if they occur, regardless of whether they are specified in events.
?

Quoted 6 lines

Quoted 6 lines

values in events yield an error. EPOLLEXCLUSIVE may be used only in an EPOLL_CTL_ADD operation; attempts to
employ it with EPOLL_CTL_MOD yield an error. If
EPOLLEXCLUSIVE has set using epoll_ctl(2), then a subse‐ quent EPOLL_CTL_MOD on the same epfd, fd pair yields an

b>> error. An epoll_ctl(2) that specifies EPOLLEXCLUSIVE in

Quoted 8 lines

Quoted 8 lines

events and specifies the target file descriptor fd as an epoll instance will likewise fail. The error in all of these cases is EINVAL.

ERRORS
EINVAL An invalid event type was specified along with EPOLLEX‐
CLUSIVE in events.

Quoted 7 lines

Quoted 7 lines

EINVAL op was EPOLL_CTL_MOD and events included EPOLLEXCLUSIVE.

EINVAL op was EPOLL_CTL_MOD and the EPOLLEXCLUSIVE flag has
previously been applied to this epfd, fd pair.

EINVAL EPOLLEXCLUSIVE was specified in event and fd is refers
to an epoll instance.

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

===

Cheers,

Michael

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

Quoted 24 lines

Quoted 5 lines

Quoted 5 lines

values in events yield an error. EPOLLEXCLUSIVE may be used only in an EPOLL_CTL_ADD operation; attempts to
employ it with EPOLL_CTL_MOD yield an error. If
EPOLLEXCLUSIVE has set using epoll_ctl(2), then a subse‐ quent EPOLL_CTL_MOD on the same epfd, fd pair yields an

b>> error. An epoll_ctl(2) that specifies EPOLLEXCLUSIVE in

Quoted 8 lines

Quoted 8 lines

events and specifies the target file descriptor fd as an epoll instance will likewise fail. The error in all of these cases is EINVAL.

ERRORS
EINVAL An invalid event type was specified along with EPOLLEX‐
CLUSIVE in events.

Quoted 7 lines

Quoted 7 lines

EINVAL op was EPOLL_CTL_MOD and events included EPOLLEXCLUSIVE.

EINVAL op was EPOLL_CTL_MOD and the EPOLLEXCLUSIVE flag has
previously been applied to this epfd, fd pair.

EINVAL EPOLLEXCLUSIVE was specified in event and fd is refers
to an epoll instance.

Returning to the second sentence in this description:

Quoted 9 lines

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

Quoted 36 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

===

Cheers,

Michael

Hi Jason,

Quoted 4 lines · Jason Baron

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

Quoted 17 lines

Quoted 11 lines

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

Quoted 5 lines

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

So, in both scenarios, *one or more* processes will get a wakeup? (I'll try to add something to the text to clarify the detail we're discussing.)

Quoted 2 lines

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

I'm not sure of the point you are trying to make here?

Cheers,

Michael

Quoted 15 lines

Quoted 15 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

Quoted 7 lines

Quoted 7 lines

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

Quoted 6 lines

Quoted 6 lines

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

Hi Jason,

Quoted 7 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 4 lines · Jason Baron

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

[...]

Quoted 17 lines

Quoted 17 lines

Quoted 11 lines

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

Quoted 5 lines

Quoted 5 lines

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

So, I was actually a little surprised by this, and went away and tested this point. It appears to me that that the two scenarios described below do NOT have the same behavior with respect to EPOLLEXCLUSIVE. See below.

Quoted 4 lines

So, in both scenarios, *one or more* processes will get a wakeup? (I'll try to add something to the text to clarify the detail we're discussing.)

Quoted 25 lines

Quoted 2 lines

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

I'm not sure of the point you are trying to make here?

Cheers,

Michael

Quoted 15 lines

Quoted 15 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, all three processes get a wakeup.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, one process gets a wakeup.

In other words, "target file" appears to mean open file description (aka open file table entry), not inode.

This is actually what I suspected might be the case, but now I am puzzled. Given what I've discovered and what you suggest are the semantics, is the implementation correct? (I suspect that it is, but it is at odds with your statement above. My test programs are inline below.

Cheers,

Michael

============

/* t_EPOLLEXCLUSIVE_multipen.c

Licensed under GNU GPLv2 or later.
*/
#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;

    if (argc != 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO>n", argv[0]);

    epfd = epoll_create(2);
    if (epfd == -1)
        errExit("epoll_create");

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    ev.events = EPOLLIN | EPOLLEXCLUSIVE;
    if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
        errExit("epoll_ctl");

    nready = epoll_wait(epfd, &rev, 1, -1);
    if (nready == -1)
        errExit("epoll-wait");
    printf("epoll_wait() returned %d\n", nready);

    exit(EXIT_SUCCESS);
}

===============

/* t_EPOLLEXCLUSIVE_fork.c

Licensed under GNU GPLv2 or later.
*/

#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;
    int cnum;

    if (argc != 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO>n", argv[0]);

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    for (cnum = 0; cnum < 3; cnum++) {
        switch (fork()) {
        case -1:
            errExit("fork");

        case 0: /* Child */
            epfd = epoll_create(2);
            if (epfd == -1)
                errExit("epoll_create");

            ev.events = EPOLLIN | EPOLLEXCLUSIVE;
            if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
                errExit("epoll_ctl");

            nready = epoll_wait(epfd, &rev, 1, -1);
            if (nready == -1)
                errExit("epoll-wait");
            printf("Child %d: epoll_wait() returned %d\n", cnum, nready);
            exit(EXIT_SUCCESS);

        default:
            break;
        }
    }

    wait(NULL);
    wait(NULL);
    wait(NULL);

    exit(EXIT_SUCCESS);
}

Hi Michael,

Quoted 100 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 7 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 4 lines · Jason Baron

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

[...]

Quoted 20 lines

Quoted 20 lines

Quoted 10 lines

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file
descrip‐
tors are attached to the same target file using
EPOLLEXCLU‐
SIVE, one or more of the epoll file descriptors
will
receive an event with epoll_wait(2).

Quoted 4 lines

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

Quoted 5 lines

Quoted 5 lines

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

So, I was actually a little surprised by this, and went away and tested this point. It appears to me that that the two scenarios described below do NOT have the same behavior with respect to EPOLLEXCLUSIVE. See below.

Quoted 4 lines

So, in both scenarios, *one or more* processes will get a wakeup? (I'll try to add something to the text to clarify the detail we're discussing.)

Quoted 25 lines

Quoted 2 lines

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

I'm not sure of the point you are trying to make here?

Cheers,

Michael

Quoted 15 lines

Quoted 15 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, all three processes get a wakeup.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, one process gets a wakeup.

In other words, "target file" appears to mean open file description (aka open file table entry), not inode.

This is actually what I suspected might be the case, but now I am puzzled. Given what I've discovered and what you suggest are the semantics, is the implementation correct? (I suspect that it is, but it is at odds with your statement above. My test programs are inline below.

Quoted 4 lines

Cheers,

Michael

Thanks for the test cases. So in your first test case, you are exiting immediately after the epoll_wait() returns. So this is actually causing the next wakeup. And then the 2nd thread returns from epoll_wait() and this causes the 3rd wakeup.

So the wakeups are actually not happening from the write directly, but instead from the readers doing a close(). If you do some sort of sleep after the epoll_wait() you can confirm the behavior. So I believe this is working as expected.

Thanks,

-Jason

Quoted 134 lines

============

/* t_EPOLLEXCLUSIVE_multipen.c

Licensed under GNU GPLv2 or later.
*/
#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;

    if (argc != 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO>n", argv[0]);

    epfd = epoll_create(2);
    if (epfd == -1)
        errExit("epoll_create");

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    ev.events = EPOLLIN | EPOLLEXCLUSIVE;
    if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
        errExit("epoll_ctl");

    nready = epoll_wait(epfd, &rev, 1, -1);
    if (nready == -1)
        errExit("epoll-wait");
    printf("epoll_wait() returned %d\n", nready);

    exit(EXIT_SUCCESS);
}

===============

/* t_EPOLLEXCLUSIVE_fork.c

Licensed under GNU GPLv2 or later.
*/

#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;
    int cnum;

    if (argc != 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO>n", argv[0]);

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    for (cnum = 0; cnum < 3; cnum++) {
        switch (fork()) {
        case -1:
            errExit("fork");

        case 0: /* Child */
            epfd = epoll_create(2);
            if (epfd == -1)
                errExit("epoll_create");

            ev.events = EPOLLIN | EPOLLEXCLUSIVE;
            if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
                errExit("epoll_ctl");

            nready = epoll_wait(epfd, &rev, 1, -1);
            if (nready == -1)
                errExit("epoll-wait");
            printf("Child %d: epoll_wait() returned %d\n", cnum, 
nready);
            exit(EXIT_SUCCESS);

        default:
            break;
        }
    }

    wait(NULL);
    wait(NULL);
    wait(NULL);

    exit(EXIT_SUCCESS);
}

Hi Michael,

Quoted 97 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 7 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 4 lines · Jason Baron

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

[...]

Quoted 17 lines

Quoted 17 lines

Quoted 11 lines

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

Quoted 5 lines

Quoted 5 lines

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

So, I was actually a little surprised by this, and went away and tested this point. It appears to me that that the two scenarios described below do NOT have the same behavior with respect to EPOLLEXCLUSIVE. See below.

Quoted 4 lines

So, in both scenarios, *one or more* processes will get a wakeup? (I'll try to add something to the text to clarify the detail we're discussing.)

Quoted 25 lines

Quoted 2 lines

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

I'm not sure of the point you are trying to make here?

Cheers,

Michael

Quoted 15 lines

Quoted 15 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, all three processes get a wakeup.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, one process gets a wakeup.

In other words, "target file" appears to mean open file description (aka open file table entry), not inode.

This is actually what I suspected might be the case, but now I am puzzled. Given what I've discovered and what you suggest are the semantics, is the implementation correct? (I suspect that it is, but it is at odds with your statement above. My test programs are inline below.

Quoted 4 lines

Cheers,

Michael

Thanks for the test cases. So in your first test case, you are exiting immediately after the epoll_wait() returns. So this is actually causing the next wakeup.

And then the 2nd thread returns from epoll_wait() and this causes the 3rd wakeup.

So the wakeups are actually not happening from the write directly, but instead from the readers doing a close(). If you do some sort of sleep after the epoll_wait() you can confirm the behavior. So I believe this is working as expected.

Hi Jason,

Quoted 100 lines · Jason Baron

Hi Michael,

Quoted 97 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 7 lines · Michael Kerrisk (man-pages)

Hi Jason,

Quoted 4 lines · Jason Baron

Quoted 1 line · Michael Kerrisk (man-pages)

[Restoring CC, which I see I accidentally dropped, one iteration back.]

[...]

Quoted 17 lines

Quoted 17 lines

Quoted 11 lines

Returning to the second sentence in this description:

When a wakeup event occurs and multiple epoll file descrip‐ tors are attached to the same target file using EPOLLEXCLU‐ SIVE, one or more of the epoll file descriptors will receive an event with epoll_wait(2).

There is a point that is unclear to me: what does "target file" refer to? Is it an open file description (aka open file table entry) or an inode? I suspect the former, but it was not clear in your original text.

So from epoll's perspective, the wakeups are associated with a 'wait queue'. So if the open() and subsequent EPOLL_CTL_ADD (which is done via file->poll()) results in adding to the same 'wait queue' then we will get 'exclusive' wakeup behavior.

Quoted 5 lines

Quoted 5 lines

So in general, I think the answer here is that its associated with the inode (I coudn't say with 100% certainty without really looking at all file->poll() implementations). Certainly, with the 'FIFO' example below, the two scenarios will have the same behavior with respect to EPOLLEXCLUSIVE.

So, I was actually a little surprised by this, and went away and tested this point. It appears to me that that the two scenarios described below do NOT have the same behavior with respect to EPOLLEXCLUSIVE. See below.

Quoted 4 lines

So, in both scenarios, *one or more* processes will get a wakeup? (I'll try to add something to the text to clarify the detail we're discussing.)

Quoted 25 lines

Quoted 2 lines

Also, the 'non-exclusive' mode would be subject to the same question of which wait queue is the epfd is associated with...

I'm not sure of the point you are trying to make here?

Cheers,

Michael

Quoted 15 lines

Quoted 15 lines

To make this point even clearer, here are two scenarios I'm thinking of. In each case, we're talking of monitoring the read end of a FIFO.

===

Scenario 1:

We have three processes each of which
1. Creates an epoll instance
2. Opens the read end of the FIFO
3. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, all three processes get a wakeup.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

===

Scenario 3

A parent process opens the read end of a FIFO and then calls fork() three times to create three children. Each child then:

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

1. Creates an epoll instance
2. Adds the read end of the FIFO to the epoll instance, specifying EPOLLEXCLUSIVE

When input becomes available on the FIFO, how many processes get a wakeup?

When I test this scenario, one process gets a wakeup.

In other words, "target file" appears to mean open file description (aka open file table entry), not inode.

This is actually what I suspected might be the case, but now I am puzzled. Given what I've discovered and what you suggest are the semantics, is the implementation correct? (I suspect that it is, but it is at odds with your statement above. My test programs are inline below.

Quoted 8 lines

Quoted 4 lines

Cheers,

Michael

Thanks for the test cases. So in your first test case, you are exiting immediately after the epoll_wait() returns. So this is actually causing the next wakeup.

Can I just check my understanding of the rationale for the preceding point. The next process is getting woken up, because the previous process did not "consume" the event (that is, the input is still available on the FIFO). Right?

Quoted 2 lines

And then the 2nd thread returns from epoll_wait() and this causes the 3rd wakeup.

I added the sleep() calls, but still things don't seem to happen quite as you suggest. In the first scenario, after the first process terminates, *all* of the remaining processes wake from epoll_wait(). What's happening in this case? (This smells like a possible bug.)

In the second scenario (fork()), after the first process terminates (without consuming the FIFO input), all of the other processes remain blocked in epoll-wait(). (Note, I extended the test program here to allow the number of child processes to be specified as a command-line argument.) I think I can make sense of that: it's because the open file descriptor for the read end of the FIFO has been duplicated in all of the child processes, and closing the FD in one child does not cause the corresponding open file description in other processes to be torn down because there are other FDs that still refer to it.

Quoted 4 lines

So the wakeups are actually not happening from the write directly, but instead from the readers doing a close(). If you do some sort of sleep after the epoll_wait() you can confirm the behavior. So I believe this is working as expected.

As note above, I'm still slightly puzzled.
Revised test programs pasted below.

Cheers,

Michael

==========

/* t_EPOLLEXCLUSIVE_multiopen.c

Licensed under GNU GPLv2 or later.
*/

#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;

    if (argc != 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO>\n", argv[0]);

    epfd = epoll_create(2);
    if (epfd == -1)
        errExit("epoll_create");

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    ev.events = EPOLLIN | EPOLLEXCLUSIVE;
    if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
        errExit("epoll_ctl");

    nready = epoll_wait(epfd, &rev, 1, -1);
    if (nready == -1)
        errExit("epoll-wait");
    printf("epoll_wait() returned %d\n", nready);

    printf("sleeping\n");
    sleep(3);
    printf("Terminating\n");
    exit(EXIT_SUCCESS);
}

===================

/* t_EPOLLEXCLUSIVE_fork.c

Licensed under GNU GPLv2 or later.
*/

#include <sys/epoll.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

#define usageErr(msg, progName) \
                        do { fprintf(stderr, "Usage: "); \
                             fprintf(stderr, msg, progName); \
                             exit(EXIT_FAILURE); } while (0)

#ifndef EPOLLEXCLUSIVE
#define EPOLLEXCLUSIVE (1 << 28)
#endif

int
main(int argc, char *argv[])

{
    int fd, epfd, nready;
    struct epoll_event ev, rev;
    int cnum, cmax;

    if (argc < 2 || strcmp(argv[1], "--help") == 0)
        usageErr("%s <FIFO> [num-children]\n", argv[0]);

    fd = open(argv[1], O_RDONLY);
    if (fd == -1)
        errExit("open");
    printf("Opened %s\n", argv[1]);

    cmax = (argc > 2) ? atoi(argv[2]) : 3;

    for (cnum = 0; cnum < cmax; cnum++) {
        switch (fork()) {
        case -1:
            errExit("fork");

        case 0: /* Child */
            epfd = epoll_create(2);
            if (epfd == -1)
                errExit("epoll_create");

            ev.events = EPOLLIN | EPOLLEXCLUSIVE;
            if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
                errExit("epoll_ctl");

            nready = epoll_wait(epfd, &rev, 1, -1);
            if (nready == -1)
                errExit("epoll-wait");
            printf("Child %d: epoll_wait() returned %d\n", cnum, nready);
            printf("sleeping\n");
            sleep(3);
            printf("Child %d terminating\n", cnum);
            exit(EXIT_SUCCESS);

        default:
            break;
        }
    }

    for (cnum = 0; cnum < cmax; cnum++)
        wait(NULL);

    exit(EXIT_SUCCESS);
}