io_uring/poll: fix multishot recv missing EOF on wakeup race
Jens Axboe
maintainerPATCH v3 1/1 io_uring/poll: fix multishot recv missing EOF on wakeup race
When a socket send and shutdown() happen back-to-back, both fire wake-ups before the receiver's task_work has a chance to run. The first wake gets poll ownership (poll_refs=1), and the second bumps it to 2. When io_poll_check_events() runs, it calls io_poll_issue() which does a recv that reads the data and returns IOU_RETRY. The loop then drains all accumulated refs (atomic_sub_return(2) -> 0) and exits, even though only the first event was consumed. Since the shutdown is a persistent state change, no further wakeups will happen, and the multishot recv can hang forever.
Check specifically for HUP in the poll loop, and ensure that another loop is done to check for status if more than a single poll activation is pending. This ensures we don't lose the shutdown event.
---
V3: split mshot and !mshot cases, and simply use the number of refs
gotten in the beginning for gating retry. if one is dropped when we want to retry, we'll loop again as we'd still have remaining refs.
io_uring/poll.c +15 -6
@@ -228,6 +228,19 @@ static inline void io_poll_execute(struct io_kiocb *req, int res)__io_poll_execute (req, res); } +static inline void io_mshot_check_retry (struct io_kiocb *req,int *v) +{ +/* +* Release all references, retry if someone tried to restart +* task_work while we were executing it. +*/ + *v &=IO_POLL_REF_MASK ; + +/* multiple refs and HUP, ensure we loop once more */ +if ((req->cqe.res & (POLLHUP |POLLRDHUP )) && *v !=1 ) + (*v)--; +} +/* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc.
@@ -303,6 +316,7 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)io_req_set_res (req, mask,0 );return IOU_POLL_REMOVE_POLL_USE_RES ; } + v &=IO_POLL_REF_MASK ; }else {int ret =io_poll_issue (req, tw);
@@ -312,16 +326,11 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)return IOU_POLL_REQUEUE ;if (ret !=IOU_RETRY && ret <0 )return ret; +io_mshot_check_retry (req, &v); }/* force the next iteration to vfs_poll() */ req->cqe.res =0 ; - -/* -* Release all references, retry if someone tried to restart -* task_work while we were executing it. -*/ - v &=IO_POLL_REF_MASK ; }while (atomic_sub_return (v, &req->poll_refs) &IO_POLL_REF_MASK );io_napi_add (req);
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
When a socket send and
shutdown()happen back-to-back, both fire wake-ups before the receiver'stask_workhas a chance to run. The first wake gets poll ownership (poll_refs=1), and the second bumps it to 2. Whenio_poll_check_events()runs, it callsio_poll_issue()which does a recv that reads the data and returnsIOU_RETRY. The loop then drains all accumulated refs (atomic_sub_return(2)-> 0) and exits, even though only the first event was consumed. Since the shutdown is a persistent state change, no further wakeups will happen, and the multishot recv can hang forever.
Quoted 3 lines
Check specifically for HUP in the poll loop, and ensure that another loop is done to check for status if more than a single poll activation is pending. This ensures we don't lose the shutdown event.
Sounds fine with comments below.
Btw, did you look into whether it's a INQ issue? Polling expects multishots to handle all those conditions, which usually goes in a form of:
while (1) {
ret = do_IO();
if (ret == -EAGAIN)
goto continue_poll;
if (ret < 0)
goto fail;
if (ret == 0)
goto terminate_req;
...
// partial progress, try again
}and recv was following this pattern before, but maybe it's sth like recv() returning some bytes, inq rightfully saying that there are no more bytes left but forgets to check for terminators like shutdown.
Quoted 18 lines
Fixes dbc2564cfe0f ("io_uring: let fast poll support multishot")Reported-by Francis Brosseau <[email protected]>Signed-off-by Jens Axboe <[email protected]>---
V3: split mshot and !mshot cases, and simply use the number of refs
gotten in the beginning for gating retry. if one is dropped when we want to retry, we'll loop again as we'd still have remaining refs.io_uring/poll.c +0
-0
io_uring/poll.c static inline void io_poll_execute(struct io_kiocb *req, int res) · line 236
Quoted 9 lines
io_uring/poll.c +6
-0@@ -228,6 +228,19 @@ static inline void io_poll_execute(struct io_kiocb *req, int res)__io_poll_execute (req, res); } +static inline void io_mshot_check_retry (struct io_kiocb *req,int *v) +{ +/* +* Release all references, retry if someone tried to restart +* task_work while we were executing it. +*/
This comment belongs to the atomic sub, not masking.
io_uring/poll.c static inline void io_poll_execute(struct io_kiocb *req, int res) · line 237
Quoted 1 line
io_uring/poll.c +1
-0+ *v &=IO_POLL_REF_MASK ;
nit: seems like you can just do that inside the
"if (unlikely(v != 1)) { ... }" block.
Quoted 18 lines
io_uring/poll.c +7
-0+ +/* multiple refs and HUP, ensure we loop once more */ +if ((req->cqe.res & (POLLHUP |POLLRDHUP )) && *v !=1 ) + (*v)--; +} +/* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. @@ -303,6 +316,7 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)io_req_set_res (req, mask,0 );return IOU_POLL_REMOVE_POLL_USE_RES ; } + v &=IO_POLL_REF_MASK ; }else {int ret =io_poll_issue (req, tw);
io_uring/poll.c static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw) · line 329
Quoted 4 lines
io_uring/poll.c +1
-0@@ -312,16 +326,11 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)return IOU_POLL_REQUEUE ;if (ret !=IOU_RETRY && ret <0 )return ret; +io_mshot_check_retry (req, &v);
Should go before io_poll_issue(), req->cqe.res might already be invalid.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 11 lines · Pavel Begunkov
Quoted 10 lines · Jens Axboe
When a socket send and
shutdown()happen back-to-back, both fire wake-ups before the receiver'stask_workhas a chance to run. The first wake gets poll ownership (poll_refs=1), and the second bumps it to 2. Whenio_poll_check_events()runs, it callsio_poll_issue()which does a recv that reads the data and returnsIOU_RETRY. The loop then drains all accumulated refs (atomic_sub_return(2)-> 0) and exits, even though only the first event was consumed. Since the shutdown is a persistent state change, no further wakeups will happen, and the multishot recv can hang forever.
Quoted 5 lines
Quoted 3 lines
Check specifically for HUP in the poll loop, and ensure that another loop is done to check for status if more than a single poll activation is pending. This ensures we don't lose the shutdown event.
Sounds fine with comments below.
Thanks
Quoted 16 lines
Btw, did you look into whether it's a INQ issue? Polling expects multishots to handle all those conditions, which usually goes in a form of:
while (1) { ret = do_IO(); if (ret == -EAGAIN) goto continue_poll; if (ret < 0) goto fail; if (ret == 0) goto terminate_req; ... // partial progress, try again }
Quoted 4 lines
and recv was following this pattern before, but maybe it's sth like
recv()returning some bytes, inq rightfully saying that there are no more bytes left but forgets to check for terminators like shutdown.
Right, as per my earlier emails, this is what introduced the issue for AF_UNIX, when the INQ support was added. We read the whole thing, and INQ is correctly returned as having 0 bytes left. Hence no retry happens, and the EOF is missed. We could do something ala the below, entirely untested, which would ensure we retry for that condition.
I don't love the poll HUP hack, but I also don't really like how the poll event handling will coalesce the events effectively. Since this particular issue will need to go back to 6.17+ stable, I'm also open to doing the HUP hack and just doing something cleaner on top.
io_uring/net.c +4 -1
@@ -903,10 +903,13 @@ static inline bool io_recv_finish(struct io_kiocb *req,*/ if ((req->flags &REQ_F_APOLL_MULTISHOT ) && !mshot_finished &&io_req_post_cqe (req, sel->val, cflags |IORING_CQE_F_MORE )) { +struct socket *sock =sock_from_file (req->file); + sel->val =IOU_RETRY ;io_mshot_prep_retry (req, kmsg);/* Known not-empty or unknown state, retry */ -if (cflags &IORING_CQE_F_SOCK_NONEMPTY || kmsg->msg.msg_inq <0 ) { +if (cflags &IORING_CQE_F_SOCK_NONEMPTY || kmsg->msg.msg_inq <0 || +READ_ONCE (sock->sk->sk_shutdown) &SHUTDOWN_MASK ) {if (sr->nr_multishot_loops++ <MULTISHOT_MAX_RETRY && !(sr->flags &IORING_RECV_MSHOT_CAP )) {return false;
Quoted 7 lines
Quoted 7 lines
io_uring/poll.c +0
-0@@ -228,6 +228,19 @@ static inline void io_poll_execute(struct io_kiocb *req, int res)__io_poll_execute (req, res); }
Quoted 8 lines
io_uring/poll.c static inline void io_poll_execute(struct io_kiocb *req, int res) · line 235
Quoted 6 lines
io_uring/poll.c +5
-0+static inline void io_mshot_check_retry(struct io_kiocb *req, int *v) +{ +/* +* Release all references, retry if someone tried to restart +* task_work while we were executing it. +*/ This comment belongs to the atomic sub, not masking.
True, should've left that there.
Quoted 4 lines
io_uring/poll.c static inline void io_poll_execute(struct io_kiocb *req, int res) · line 236
Quoted 1 line
io_uring/poll.c +1
-0+ *v &=IO_POLL_REF_MASK ;nit: seems like you can just do that inside the
"if (unlikely(v != 1)) { ... }" block.
That could work, then we don't need it in both the other branches.
Quoted 15 lines
Quoted 15 lines
io_uring/poll.c +6
-0+/* multiple refs and HUP, ensure we loop once more */ +if ((req->cqe.res & (POLLHUP |POLLRDHUP )) && *v !=1 ) + (*v)--; +} +/* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. @@ -303,6 +316,7 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)io_req_set_res (req, mask,0 );return IOU_POLL_REMOVE_POLL_USE_RES ; } + v &=IO_POLL_REF_MASK ; }else {int ret =io_poll_issue (req, tw);
Quoted 8 lines
Quoted 5 lines
io_uring/poll.c +0
-0@@ -312,16 +326,11 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw)return IOU_POLL_REQUEUE; if (ret != IOU_RETRY && ret < 0) return ret;+
io_mshot_check_retry(req, &v);Should go before
io_poll_issue(),req->cqe.resmight already be invalid.
Yeah good point, it was above it before. Too much late night consolidation...
Signature · 1 line
Jens Axboe
On 3/17/26 13:07, Jens Axboe wrote:
...
Quoted 5 lines
Right, as per my earlier emails, this is what introduced the issue for
AF_UNIX, when the INQ support was added. We read the whole thing, and INQ is correctly returned as having 0 bytes left. Hence no retry happens, and the EOF is missed. We could do something ala the below, entirely untested, which would ensure we retry for that condition.
static int tcp_inq_hint(struct sock *sk)
{
...
if (inq == 0 && sock_flag(sk, SOCK_DONE))
inq = 1;
return inq;
}Assuming TCP doesn't work either, I guess I was curious whether it gets shutdown but the sock is !SOCK_DONE, or whether inq=1 is correct. Just thinking out loud, maybe I will check later.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 16 lines · Pavel Begunkov
On 3/17/26 13:07, Jens Axboe wrote:
...Quoted 5 lines
Right, as per my earlier emails, this is what introduced the issue for
AF_UNIX, when the INQ support was added. We read the whole thing, and INQ is correctly returned as having 0 bytes left. Hence no retry happens, and the EOF is missed. We could do something ala the below, entirely untested, which would ensure we retry for that condition.static int
tcp_inq_hint(struct sock *sk)
{
...
if (inq == 0 &&sock_flag(sk, SOCK_DONE))
inq = 1;
return inq;
}
Quoted 3 lines
Assuming TCP doesn't work either, I guess I was curious whether it gets shutdown but the sock is !SOCK_DONE, or whether inq=1 is correct. Just thinking out loud, maybe I will check later.
Ah indeed, yes that's a good find. Let me test that real quick... I feel like the AF_UNIX inq addition was somewhat half baked.
Signature · 1 line
Jens Axboe