pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers
PATCH v4 1/1 pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers
pipe_poll() unconditionally sets ->poll_usage on the first call, forcing anon_pipe_write() to wake up readers on every write even if the pipe was not empty.
The reason is that some legacy epoll(EPOLLET) users depend on historical per-write wakeups, see commit 3a34b13a88ca ("pipe: make pipe writes always wake up readers").
Test-case:
#include <unistd.h>
#include <sys/epoll.h>
#include <assert.h>
int main(void)
{
int pfd[2], efd;
struct epoll_event evt = { .events = EPOLLIN | EPOLLET };
pipe(pfd);
efd = epoll_create1(0);
epoll_ctl(efd, EPOLL_CTL_ADD, pfd[0], &evt);
for (int i = 0; i < 2; ++i) {
write(pfd[1], "", 1);
assert(epoll_wait(efd, &evt, 1, 0) == 1);
}
return 0;
}it fails if WRITE_ONCE(poll_usage, true) is removed from pipe_poll(). However, without EPOLLET in .events, it does not need the extra wakeup and succeeds even if write() is called only once before the main loop.
Currently io_uring without (unsupported) IORING_POLL_ADD_LEVEL always sets EPOLLET, and in IORING_POLL_ADD_MULTI mode it depends on per-write wakeups the same way:
#include <unistd.h>
#include <sys/mman.h>
#include <sys/epoll.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>
#include <assert.h>
int main(void)
{
struct io_uring_params p = {};
int fd, pfd[2];
pipe(pfd);
fd = syscall(SYS_io_uring_setup, 2, &p);
assert(fd >= 0);
void *ring = mmap(0, p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe),
PROT_READ | PROT_WRITE, MAP_SHARED, fd, IORING_OFF_SQ_RING);
assert(ring != MAP_FAILED);
*(unsigned *)(ring + p.sq_off.tail) = 1;
struct io_uring_sqe *sqes = mmap(0, p.sq_entries * sizeof(*sqes),
PROT_READ | PROT_WRITE, MAP_SHARED, fd, IORING_OFF_SQES);
assert(sqes != MAP_FAILED);
sqes[0].opcode = IORING_OP_POLL_ADD;
sqes[0].fd = pfd[0];
sqes[0].len = IORING_POLL_ADD_MULTI;
sqes[0].poll32_events = EPOLLIN;
syscall(SYS_io_uring_enter, fd, 1, 0, 0, 0, 0);
unsigned *cq_head = ring + p.cq_off.head;
unsigned *cq_tail = ring + p.cq_off.tail;
for (int i = 0; i < 2; ++i) {
write(pfd[1], "", 1);
syscall(SYS_io_uring_enter, fd, 0, 0, IORING_ENTER_GETEVENTS, 0, 0);
assert(*cq_tail == ++*cq_head);
}
return 0;
}the 2nd assert() in the main loop fails without ->poll_usage == true.
Rename ->poll_usage to ->pseudo_edgetrigger to make the purpose clearer, update the comments, and change pipe_poll() to set ->pseudo_edgetrigger only if wait->_key & EPOLLET is true. This check should catch both users, and this way poll/select and epoll without EPOLLET users will not pay for the extra wakeup.
--- fs/pipe.c | 20 +++++++++++++------- include/linux/pipe_fs_i.h | 4 ++-- 2 files changed, 15 insertions(+), 9 deletions(-)
fs/pipe.c +13 -7
@@ -686,10 +686,9 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)* how (for example) the GNU make jobserver uses small writes to * wake up pending jobs * -* Epoll nonsensically wants a wakeup whether the pipe -* was already empty or not. +* ->pseudo_edgetrigger enables per-write wakeups, see pipe_poll() */ -if (was_empty || pipe->poll_usage) +if (was_empty ||READ_ONCE (pipe->pseudo_edgetrigger))wake_up_interruptible_sync_poll (&pipe->rd_wait,EPOLLIN |EPOLLRDNORM );kill_fasync (&pipe->fasync_readers,SIGIO ,POLL_IN );if (wake_next_writer)
@@ -752,7 +751,6 @@ static long pipe_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) } } -/* No kernel lock held - fine */ static __poll_t pipe_poll (struct file *filp,poll_table *wait) {
@@ -760,9 +758,17 @@ pipe_poll(struct file *filp, poll_table *wait)struct pipe_inode_info *pipe = filp->private_data;union pipe_index idx; -/* Epoll has some historical nasty semantics, this enables them */ -if (unlikely (!READ_ONCE (pipe->poll_usage))) -WRITE_ONCE (pipe->poll_usage, true); +/* +* Legacy epoll(EPOLLET) users depend on historical per-write wakeups, +* see 3a34b13a88ca ("pipe: make pipe writes always wake up readers") +* and the ->pseudo_edgetrigger check in anon_pipe_write(). +* Currently io_uring sets EPOLLET for multishot polls, so it gets the +* same behaviour. +*/ +if ((filp->f_mode &FMODE_READ ) && + wait && (wait->_key &EPOLLET ) && +unlikely (!READ_ONCE (pipe->pseudo_edgetrigger))) +WRITE_ONCE (pipe->pseudo_edgetrigger, true);/* * Reading pipe state only -- no need for acquiring the semaphore.
include/linux/pipe_fs_i.h +2 -2
@@ -74,7 +74,7 @@ union pipe_index {* @files: number of struct file referring this pipe (protected by ->i_lock) * @r_counter: reader counter * @w_counter: writer counter -* @poll_usage: is this pipe used for epoll, which has crazy wakeups? +* @pseudo_edgetrigger: has an EPOLLET consumer, enable per-write wakeups * @fasync_readers: reader side fasync * @fasync_writers: writer side fasync * @bufs: the circular array of pipe buffers
@@ -95,7 +95,7 @@ struct pipe_inode_info {unsigned int files;unsigned int r_counter;unsigned int w_counter; -bool poll_usage; +bool pseudo_edgetrigger;#ifdef CONFIG_WATCH_QUEUE bool note_loss;#endif
Signature · 1 line
2.52.0
fs/pipe.c pipe_poll(struct file *filp, poll_table *wait) · line 766
Quoted 2 lines · Oleg Nesterov
fs/pipe.c +2
-0+* Currently io_uring sets EPOLLET for multishot polls, so it gets the +* same behaviour.
I got the impression that Linus wanted the comment epoll-only, and wanted the io_uring case treated as something to remove, not to bless.
Quoted 7 lines · Breno Leitao
fs/pipe.c pipe_poll(struct file *filp, poll_table *wait) · line 766
Quoted 2 lines · Oleg Nesterov
fs/pipe.c +2
-0+* Currently io_uring sets EPOLLET for multishot polls, so it gets the +* same behaviour. I got the impression that Linus wanted the comment epoll-only, and wanted the
io_uringcase treated as something to remove, not to bless.
Well, perhaps I misunderstood Linus... but it seems he was fine with this comment which I showed in
https://lore.kernel.org/all/[email protected]/
And. This comment (and the test-case in the changelog) doesn't try to bless the current behaviour, it only tries to document the "status quo".
Let me also quote my other reply to Linus:
> The only reason that ugly hack exists is because we did have that > break user space. If we can get rid of th eugly hack for io_uring, > that would only be a good thing.
Perhaps, I can't really comment. But it seems to me that in this case it would make more sense to change the logic in io_uring/ rather than in pipe_poll(). And this needs a separate change.
Oleg.
Quoted 1 line · Oleg Nesterov
pipe: only enable the extra
wake_up(rd_wait)for EPOLLET consumers
Applied to the vfs-7.3.misc branch of the vfs/vfs.git tree. Patches in the vfs-7.3.misc branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs-7.3.misc
[1/1] pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers
https://git.kernel.org/vfs/vfs/c/07c3efe20e24