tcp: fix listener wakeup after reuseport migration
Subsystems (7)
v2 cover tcp: fix listener wakeup after reuseport migration
This series fixes a missing wakeup when inet_csk_listen_stop() migrates an established child socket from a closing listener to another socket in the same SO_REUSEPORT group after the child has already been queued for accept.
The target listener receives the migrated accept-queue entry via inet_csk_reqsk_queue_add(), but its waiters are not notified. Nonblocking accept() still succeeds because it checks the accept queue directly, but readiness-based waiters can remain asleep until another connection generates a wakeup.
Patch 1 notifies the target listener after a successful migration in inet_csk_listen_stop() and protects the post-queue_add() nsk accesses with rcu_read_lock()/rcu_read_unlock().
Patch 2 extends the existing migrate_reuseport BPF selftest with an epoll readiness check for the TCP_ESTABLISHED migration case.
Testing:
- On a patched kernel booted under QEMU, the full migrate_reuseport selftest passes with SELFTEST_RC=0.
---
v2:
- wrap the post-queue_add() nsk dereferences with rcu_read_lock()/ rcu_read_unlock() to prevent a potential UAF (Eric Dumazet)
- extend tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c instead of adding standalone net selftests (Kuniyuki Iwashima)
- limit the epoll readiness check to TCP_ESTABLISHED cases
v1:
https://lore.kernel.org/netdev/[email protected]/
Zhenzhong Wu (2):
tcp: call sk_data_ready() after listener migration selftests/bpf: check epoll readiness after reuseport migration
net/ipv4/inet_connection_sock.c | 3 ++ .../bpf/prog_tests/migrate_reuseport.c | 32 ++++++++++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-)
base-commit: 52bcb57a4e8a0865a76c587c2451906342ae1b2d
Signature · 1 line
2.43.0
PATCH v2 1/2 tcp: call sk_data_ready() after listener migration
When inet_csk_listen_stop() migrates an established child socket from a closing listener to another socket in the same SO_REUSEPORT group, the target listener gets a new accept-queue entry via inet_csk_reqsk_queue_add(), but that path never notifies the target listener's waiters. A nonblocking accept() still works because it checks the queue directly, but poll()/epoll_wait() waiters and blocking accept() callers can also remain asleep indefinitely.
Call READ_ONCE(nsk->sk_data_ready)(nsk) after a successful migration in inet_csk_listen_stop().
However, after inet_csk_reqsk_queue_add() succeeds, the ref acquired in reuseport_migrate_sock() is effectively transferred to nreq->rsk_listener. Another CPU can then dequeue nreq via accept() or listener shutdown, hit reqsk_put(), and drop that listener ref. Since listeners are SOCK_RCU_FREE, wrap the post-queue_add() dereferences of nsk in rcu_read_lock()/rcu_read_unlock(), which also covers the existing sock_net(nsk) access in that path.
The reqsk_timer_handler() path does not need the same changes for two
reasons: half-open requests become readable only after the final ACK, where tcp_child_process() already wakes the listener; and once nreq is visible via inet_ehash_insert(), the success path no longer touches nsk directly.
--- net/ipv4/inet_connection_sock.c | 3 +++ 1 file changed, 3 insertions(+)
net/ipv4/inet_connection_sock.c +3 -0
@@ -1479,16 +1479,19 @@ void inet_csk_listen_stop(struct sock *sk)if (nreq) {refcount_set (&nreq->rsk_refcnt,1 ); +rcu_read_lock ();if (inet_csk_reqsk_queue_add (nsk, nreq, child)) {__NET_INC_STATS (sock_net (nsk),LINUX_MIB_TCPMIGRATEREQSUCCESS );reqsk_migrate_reset (req); +READ_ONCE (nsk->sk_data_ready)(nsk); }else {__NET_INC_STATS (sock_net (nsk),LINUX_MIB_TCPMIGRATEREQFAILURE );reqsk_migrate_reset (nreq);__reqsk_free (nreq); } +rcu_read_unlock ();/* inet_csk_reqsk_queue_add() has already * called inet_child_forget() on failure case.
Signature · 1 line
2.43.0
PATCH v2 2/2 selftests/bpf: check epoll readiness after reuseport migration
In reply to Zhenzhong WuAfter migrate_dance() moves established children to the target listener, add it to an epoll set and verify that epoll_wait(..., 0) reports it ready before accept().
This adds epoll coverage for the TCP_ESTABLISHED reuseport migration case in migrate_reuseport.
Keep the check limited to TCP_ESTABLISHED cases. TCP_SYN_RECV and TCP_NEW_SYN_RECV still depend on asynchronous handshake completion, so a zero-timeout epoll_wait() would race there.
--- .../bpf/prog_tests/migrate_reuseport.c | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +31 -1
@@ -18,13 +18,16 @@* 9. call shutdown() for the second server * and migrate the requests in the accept queue * to the last server socket. -* 10. call accept() for the last server socket. +* 10. for TCP_ESTABLISHED cases, call epoll_wait(..., 0) +* for the last server socket. +* 11. call accept() for the last server socket. * * Author: Kuniyuki Iwashima <[email protected]> */ #include <bpf/bpf.h> #include <bpf/libbpf.h> +#include <sys/epoll.h> #include "test_progs.h" #include "test_migrate_reuseport.skel.h"
@@ -522,6 +525,33 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_clients; } +/* Only TCP_ESTABLISHED has already-migrated accept-queue entries +* here. Later states still depend on follow-up handshake work. +*/ +if (test_case->state ==BPF_TCP_ESTABLISHED ) { +struct epoll_event ev = { + .events =EPOLLIN , + }; +int epfd; +int nfds; + + epfd =epoll_create1 (EPOLL_CLOEXEC ); +if (!ASSERT_NEQ (epfd,-1 ,"epoll_create1" )) + goto close_clients; + + ev.data.fd = test_case->servers[MIGRATED_TO ]; +if (!ASSERT_OK (epoll_ctl (epfd,EPOLL_CTL_ADD , + test_case->servers[MIGRATED_TO ], &ev), +"epoll_ctl" )) + goto close_epfd; + + nfds =epoll_wait (epfd, &ev,1 ,0 ); +ASSERT_EQ (nfds,1 ,"epoll_wait" ); + +close_epfd: +close (epfd); + } +count_requests (test_case, skel); close_clients:
Signature · 1 line
2.43.0
Kuniyuki Iwashima
reviewerRe: [PATCH net v2 2/2] selftests/bpf: check epoll readiness after reuseport migration
From: Zhenzhong Wu <[email protected]>
Date: Sun, 19 Apr 2026 02:13:33 +0800
Quoted 54 lines
After
migrate_dance()moves established children to the target listener, add it to an epoll set and verify thatepoll_wait(..., 0)reports it ready beforeaccept().This adds epoll coverage for the
TCP_ESTABLISHEDreuseport migration case inmigrate_reuseport.Keep the check limited to
TCP_ESTABLISHEDcases.TCP_SYN_RECVandTCP_NEW_SYN_RECVstill depend on asynchronous handshake completion, so a zero-timeoutepoll_wait()would race there.Signed-off-by Zhenzhong Wu <[email protected]>--- .../bpf/prog_tests/migrate_reuseport.c | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +16
-1@@ -18,13 +18,16 @@* 9. call shutdown() for the second server * and migrate the requests in the accept queue * to the last server socket. -* 10. call accept() for the last server socket. +* 10. for TCP_ESTABLISHED cases, call epoll_wait(..., 0) +* for the last server socket. +* 11. call accept() for the last server socket. * * Author: Kuniyuki Iwashima <[email protected]> */ #include <bpf/bpf.h> #include <bpf/libbpf.h> +#include <sys/epoll.h> #include "test_progs.h" #include "test_migrate_reuseport.skel.h" @@ -522,6 +525,33 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_clients; } +/* Only TCP_ESTABLISHED has already-migrated accept-queue entries +* here. Later states still depend on follow-up handshake work. +*/ +if (test_case->state ==BPF_TCP_ESTABLISHED ) { +struct epoll_event ev = { + .events =EPOLLIN , + }; +int epfd; +int nfds; + + epfd =epoll_create1 (EPOLL_CLOEXEC ); +if (!ASSERT_NEQ (epfd,-1 ,"epoll_create1" ))
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c static void run_test(struct migrate_reuseport_test_case *test_case, · line 549
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +10
-0+ goto close_clients; + + ev.data.fd = test_case->servers[MIGRATED_TO ]; +if (!ASSERT_OK (epoll_ctl (epfd,EPOLL_CTL_ADD , + test_case->servers[MIGRATED_TO ], &ev), +"epoll_ctl" )) + goto close_epfd; + + nfds =epoll_wait (epfd, &ev,1 ,0 ); +ASSERT_EQ (nfds,1 ,"epoll_wait" );
Thanks for the update, but the test passes without patch 1.
I think it would be best to test just after shutdown() where migration happens.
Also, TCP_SYN_RECV should be covered in the same way.
---8<---
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +30 -27
@@ -353,8 +353,29 @@ static int update_maps(struct migrate_reuseport_test_case *test_case,static int migrate_dance (struct migrate_reuseport_test_case *test_case) { +struct epoll_event ev = { + .events =EPOLLIN , + }; +int epoll, nfds;int i, err; +if (test_case->state !=BPF_TCP_NEW_SYN_RECV ) { + epoll =epoll_create1 (0 ); +if (!ASSERT_NEQ (epoll,-1 ,"epoll_create1" )) +return -1 ; + + ev.data.fd = test_case->servers[MIGRATED_TO ]; +if (!ASSERT_OK (epoll_ctl (epoll,EPOLL_CTL_ADD , + test_case->servers[MIGRATED_TO ], &ev), +"epoll_ctl" )) { + goto close_epoll; + } + + nfds =epoll_wait (epoll, &ev,1 ,0 ); +if (!ASSERT_EQ (nfds,0 ,"epoll_wait 1" )) + goto close_epoll; + } +/* Migrate TCP_ESTABLISHED and TCP_SYN_RECV requests * to the last listener based on eBPF. */
@@ -368,6 +389,15 @@ static int migrate_dance(struct migrate_reuseport_test_case *test_case)if (test_case->state ==BPF_TCP_NEW_SYN_RECV )return 0 ; + nfds =epoll_wait (epoll, &ev,1 ,0 ); +if (!ASSERT_EQ (nfds,1 ,"epoll_wait 2" )) { +close_epoll: +close (epoll); +return -1 ; + } + +close (epoll); +/* Note that we use the second listener instead of the * first one here. *
@@ -525,33 +555,6 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_clients; } -/* Only TCP_ESTABLISHED has already-migrated accept-queue entries -* here. Later states still depend on follow-up handshake work. -*/ -if (test_case->state ==BPF_TCP_ESTABLISHED ) { -struct epoll_event ev = { - .events =EPOLLIN , - }; -int epfd; -int nfds; - - epfd =epoll_create1 (EPOLL_CLOEXEC ); -if (!ASSERT_NEQ (epfd,-1 ,"epoll_create1" )) - goto close_clients; - - ev.data.fd = test_case->servers[MIGRATED_TO ]; -if (!ASSERT_OK (epoll_ctl (epfd,EPOLL_CTL_ADD , - test_case->servers[MIGRATED_TO ], &ev), -"epoll_ctl" )) - goto close_epfd; - - nfds =epoll_wait (epfd, &ev,1 ,0 ); -ASSERT_EQ (nfds,1 ,"epoll_wait" ); - -close_epfd: -close (epfd); - } -count_requests (test_case, skel); close_clients:
---8<---
Re: [PATCH net v2 2/2] selftests/bpf: check epoll readiness after reuseport migration
Thanks Kuniyuki, will fold this into v3.
Your approach of checking epoll right around shutdown() is much better. I didn't think to put the check inside migrate_dance(). With my original placement, the test still passed without patch 1 — I should have called this out in the cover letter instead of settling for what was effectively just a smoke test.
Quoted 161 lines · Kuniyuki Iwashima
From: Zhenzhong Wu <[email protected]>
Date: Sun, 19 Apr 2026 02:13:33 +0800Quoted 54 lines
After
migrate_dance()moves established children to the target listener, add it to an epoll set and verify thatepoll_wait(..., 0)reports it ready beforeaccept().This adds epoll coverage for the
TCP_ESTABLISHEDreuseport migration case inmigrate_reuseport.Keep the check limited to
TCP_ESTABLISHEDcases.TCP_SYN_RECVandTCP_NEW_SYN_RECVstill depend on asynchronous handshake completion, so a zero-timeoutepoll_wait()would race there.Signed-off-by Zhenzhong Wu <[email protected]>--- .../bpf/prog_tests/migrate_reuseport.c | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +16
-1@@ -18,13 +18,16 @@* 9. call shutdown() for the second server * and migrate the requests in the accept queue * to the last server socket. -* 10. call accept() for the last server socket. +* 10. for TCP_ESTABLISHED cases, call epoll_wait(..., 0) +* for the last server socket. +* 11. call accept() for the last server socket. * * Author: Kuniyuki Iwashima <[email protected]> */ #include <bpf/bpf.h> #include <bpf/libbpf.h> +#include <sys/epoll.h> #include "test_progs.h" #include "test_migrate_reuseport.skel.h" @@ -522,6 +525,33 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_clients; } +/* Only TCP_ESTABLISHED has already-migrated accept-queue entries +* here. Later states still depend on follow-up handshake work. +*/ +if (test_case->state ==BPF_TCP_ESTABLISHED ) { +struct epoll_event ev = { + .events =EPOLLIN , + }; +int epfd; +int nfds; + + epfd =epoll_create1 (EPOLL_CLOEXEC ); +if (!ASSERT_NEQ (epfd,-1 ,"epoll_create1" ))tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c static void run_test(struct migrate_reuseport_test_case *test_case, · line 549
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +10
-0+ goto close_clients; + + ev.data.fd = test_case->servers[MIGRATED_TO ]; +if (!ASSERT_OK (epoll_ctl (epfd,EPOLL_CTL_ADD , + test_case->servers[MIGRATED_TO ], &ev), +"epoll_ctl" )) + goto close_epfd; + + nfds =epoll_wait (epfd, &ev,1 ,0 ); +ASSERT_EQ (nfds,1 ,"epoll_wait" );Thanks for the update, but the test passes without patch 1.
I think it would be best to test just after
shutdown()where migration happens.Also,
TCP_SYN_RECVshould be covered in the same way.---8<---
tools/testing/selftests/bpf/prog_tests/migrate_reuseport.c +30
-28@@ -353,8 +353,29 @@ static int update_maps(struct migrate_reuseport_test_case *test_case,static int migrate_dance (struct migrate_reuseport_test_case *test_case) { +struct epoll_event ev = { + .events =EPOLLIN , + }; +int epoll, nfds;int i, err; +if (test_case->state !=BPF_TCP_NEW_SYN_RECV ) { + epoll =epoll_create1 (0 ); +if (!ASSERT_NEQ (epoll,-1 ,"epoll_create1" )) +return -1 ; + + ev.data.fd = test_case->servers[MIGRATED_TO ]; +if (!ASSERT_OK (epoll_ctl (epoll,EPOLL_CTL_ADD , + test_case->servers[MIGRATED_TO ], &ev), +"epoll_ctl" )) { + goto close_epoll; + } + + nfds =epoll_wait (epoll, &ev,1 ,0 ); +if (!ASSERT_EQ (nfds,0 ,"epoll_wait 1" )) + goto close_epoll; + } +/* Migrate TCP_ESTABLISHED and TCP_SYN_RECV requests * to the last listener based on eBPF. */ @@ -368,6 +389,15 @@ static int migrate_dance(struct migrate_reuseport_test_case *test_case)if (test_case->state ==BPF_TCP_NEW_SYN_RECV )return 0 ; + nfds =epoll_wait (epoll, &ev,1 ,0 ); +if (!ASSERT_EQ (nfds,1 ,"epoll_wait 2" )) { +close_epoll: +close (epoll); +return -1 ; + } + +close (epoll); +/* Note that we use the second listener instead of the * first one here. * @@ -525,33 +555,6 @@ static void run_test(struct migrate_reuseport_test_case *test_case, goto close_clients; } -/* Only TCP_ESTABLISHED has already-migrated accept-queue entries -* here. Later states still depend on follow-up handshake work. -*/ -if (test_case->state ==BPF_TCP_ESTABLISHED ) { -struct epoll_event ev = { - .events =EPOLLIN , - }; -int epfd; -int nfds; - - epfd =epoll_create1 (EPOLL_CLOEXEC ); -if (!ASSERT_NEQ (epfd,-1 ,"epoll_create1" )) - goto close_clients; - - ev.data.fd = test_case->servers[MIGRATED_TO ]; -if (!ASSERT_OK (epoll_ctl (epfd,EPOLL_CTL_ADD , - test_case->servers[MIGRATED_TO ], &ev), -"epoll_ctl" )) - goto close_epfd; - - nfds =epoll_wait (epfd, &ev,1 ,0 ); -ASSERT_EQ (nfds,1 ,"epoll_wait" ); - -close_epfd: -close (epfd); - } -count_requests (test_case, skel); close_clients: ---8<---