soreuseport: Fix broken SO_INCOMING_CPU.
Subsystems (4)
v3 cover soreuseport: Fix broken SO_INCOMING_CPU.
setsockopt(SO_INCOMING_CPU) for UDP/TCP is broken since 4.5/4.6 due to these commits:
* e32ea7e74727 ("soreuseport: fast reuseport UDP socket selection")
* c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
These commits introduced the O(1) socket selection algorithm and removed O(n) iteration over the list, but it ignores the score calculated by compute_score(). As a result, it caused two misbehaviours:
* Unconnected sockets receive packets sent to connected sockets
* SO_INCOMING_CPU does not work
The former is fixed by commit acdcecc61285 ("udp: correct reuseport selection with connected sockets"). This series fixes the latter and adds some tests for SO_INCOMING_CPU.
Changes:
v3:
* Rebase and resolve conflicts
v2: https://lore.kernel.org/netdev/[email protected]/
* patch 1
* Rename helper functions
* Remove unnecessary arg sk from '__' helpers
* Fix reuseport_update_incoming_cpu() logic
* patch 2
* Add test cases
* Change when to set SO_INCOMING_CPU
* Add/Remove non-SO_INCOMING_CPU socket
v1: https://lore.kernel.org/netdev/[email protected]/
Kuniyuki Iwashima (2):
soreuseport: Fix socket selection for SO_INCOMING_CPU.
selftest: Add test for SO_INCOMING_CPU.
include/net/sock_reuseport.h | 2 + net/core/sock.c | 2 +- net/core/sock_reuseport.c | 94 ++++++- tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/so_incoming_cpu.c | 242 ++++++++++++++++++ 6 files changed, 336 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/net/so_incoming_cpu.c
Signature · 1 line
2.30.2
PATCH v3 1/2 soreuseport: Fix socket selection for SO_INCOMING_CPU.
Kazuho Oku reported that setsockopt(SO_INCOMING_CPU) does not work with setsockopt(SO_REUSEPORT) since v4.6.
With the combination of SO_REUSEPORT and SO_INCOMING_CPU, we could build a highly efficient server application.
setsockopt(SO_INCOMING_CPU) associates a CPU with a TCP listener or UDP socket, and then incoming packets processed on the CPU will likely be distributed to the socket. Technically, a socket could even receive packets handled on another CPU if no sockets in the reuseport group have the same CPU receiving the flow.
The logic exists in compute_score() so that a socket will get a higher score if it has the same CPU with the flow. However, the score gets ignored after the blamed two commits, which introduced a faster socket selection algorithm for SO_REUSEPORT.
This patch introduces a counter of sockets with SO_INCOMING_CPU in a reuseport group to check if we should iterate all sockets to find a proper one. We increment the counter when
* calling listen() if the socket has SO_INCOMING_CPU and SO_REUSEPORT
* enabling SO_INCOMING_CPU if the socket is in a reuseport group
Also, we decrement it when
* detaching a socket out of the group to apply SO_INCOMING_CPU to migrated TCP requests
* disabling SO_INCOMING_CPU if the socket is in a reuseport group
When the counter reaches 0, we can get back to the O(1) selection algorithm.
The overall changes are negligible for the non-SO_INCOMING_CPU case, and the only notable thing is that we have to update sk_incomnig_cpu under reuseport_lock. Otherwise, the race prevents transitioning to the O(n) algorithm and results in the wrong socket selection.
cpu1 (setsockopt) cpu2 (listen)
+-----------------+ +-------------+
lock_sock(sk1) lock_sock(sk2)
reuseport_update_incoming_cpu(sk1, val)
.
| /* set CPU as 0 */ |- WRITE_ONCE(sk1->incoming_cpu, val) | | spin_lock_bh(&reuseport_lock) | reuseport_grow(sk2, reuse) | . | |- more_socks_size = reuse->max_socks * 2U; | |- if (more_socks_size > U16_MAX && | | reuse->num_closed_socks) | | . | | |- RCU_INIT_POINTER(sk1->sk_reuseport_cb, NULL); | | `- __reuseport_detach_closed_sock(sk1, reuse) | | . | | `- reuseport_put_incoming_cpu(sk1, reuse) | | . | | | /* Read shutdown()ed sk1's sk_incoming_cpu | | | * without lock_sock(). | | | */ | | `- if (sk1->sk_incoming_cpu >= 0) | | . | | | /* decrement not-yet-incremented | | | * count, which is never incremented. | | | */ | | `- __reuseport_put_incoming_cpu(reuse); | | | `- spin_lock_bh(&reuseport_lock) | |- spin_lock_bh(&reuseport_lock) | |- reuse = rcu_dereference_protected(sk1->sk_reuseport_cb, ...) |- if (!reuse) | . | | /* Cannot increment reuse->incoming_cpu. */ | `- goto out; |
`- spin_unlock_bh(&reuseport_lock)
--- include/net/sock_reuseport.h | 2 + net/core/sock.c | 2 +- net/core/sock_reuseport.c | 94 ++++++++++++++++++++++++++++++++++-- 3 files changed, 92 insertions(+), 6 deletions(-)
include/net/sock_reuseport.h +2 -0
@@ -16,6 +16,7 @@ struct sock_reuseport {u16 max_socks;/* length of socks */ u16 num_socks;/* elements in socks */ u16 num_closed_socks;/* closed elements in socks */ +u16 incoming_cpu;/* The last synq overflow event timestamp of this * reuse->socks[] group. */
@@ -58,5 +59,6 @@ static inline bool reuseport_has_conns(struct sock *sk) }void reuseport_has_conns_set (struct sock *sk); +void reuseport_update_incoming_cpu (struct sock *sk,int val);#endif /* _SOCK_REUSEPORT_H */
net/core/sock.c +1 -1
@@ -1436,7 +1436,7 @@ int sk_setsockopt(struct sock *sk, int level, int optname,break ; }case SO_INCOMING_CPU : -WRITE_ONCE (sk->sk_incoming_cpu, val); +reuseport_update_incoming_cpu (sk, val);break ;case SO_CNX_ADVICE :
net/core/sock_reuseport.c +89 -5
149 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v3 2/2 selftest: Add test for SO_INCOMING_CPU.
In reply to Kuniyuki IwashimaSome highly optimised applications use SO_INCOMING_CPU to make them efficient, but they didn't test if it's working correctly by getsockopt() to avoid slowing down. As a result, no one noticed it had been broken for years, so it's a good time to add a test to catch future regression.
The test does
1) Create $(nproc) TCP listeners associated with each CPU.
2) Create 32 child sockets for each listener by calling sched_setaffinity() for each CPU.
3) Check if accept()ed sockets' sk_incoming_cpu matches listener's one.
If we see -EAGAIN, SO_INCOMING_CPU is broken. However, we might not see any error even if broken; the kernel could miraculously distribute all SYN to correct listeners. Not to let that happen, we must increase the number of clients and CPUs to some extent, so the test requires $(nproc) >= 2 and creates 64 sockets at least.
Test:
$ nproc 96 $ ./so_incoming_cpu
Before the previous patch:
# Starting 12 tests from 5 test cases.
# RUN so_incoming_cpu.before_reuseport.test1 ...
# so_incoming_cpu.c:191:test1:Expected cpu (5) == i (0) # test1: Test terminated by assertion
# FAIL so_incoming_cpu.before_reuseport.test1
not ok 1 so_incoming_cpu.before_reuseport.test1
...
# FAILED: 0 / 12 tests passed.
# Totals: pass:0 fail:12 xfail:0 xpass:0 skip:0 error:0
After:
# Starting 12 tests from 5 test cases.
# RUN so_incoming_cpu.before_reuseport.test1 ...
# so_incoming_cpu.c:199:test1:SO_INCOMING_CPU is very likely to be working correctly with 3072 sockets.
# OK so_incoming_cpu.before_reuseport.test1
ok 1 so_incoming_cpu.before_reuseport.test1
...
# PASSED: 12 / 12 tests passed.
# Totals: pass:12 fail:0 xfail:0 xpass:0 skip:0 error:0
--- tools/testing/selftests/net/.gitignore | 1 + tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/so_incoming_cpu.c | 242 ++++++++++++++++++ 3 files changed, 244 insertions(+) create mode 100644 tools/testing/selftests/net/so_incoming_cpu.c
tools/testing/selftests/net/.gitignore +1 -0
@@ -25,6 +25,7 @@ rxtimestamp sk_bind_sendto_listen sk_connect_zero_addr socket +so_incoming_cpu so_netns_cookie so_txtime stress_reuseport_listen
tools/testing/selftests/net/Makefile +1 -0
@@ -71,6 +71,7 @@ TEST_GEN_FILES += bind_bhash TEST_GEN_PROGS += sk_bind_sendto_listen TEST_GEN_PROGS += sk_connect_zero_addr TEST_PROGS += test_ingress_egress_chaining.sh +TEST_GEN_PROGS += so_incoming_cpu TEST_FILES := settings
tools/testing/selftests/net/so_incoming_cpu.c new file +242 -0
242 lines not on this page · raw message
Signature · 1 line
2.30.2
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:
Quoted 11 lines · On Fri, 21 Oct 2022 13:44:33 -0700 you wrote:
setsockopt(SO_INCOMING_CPU)for UDP/TCP is broken since 4.5/4.6 due to these commits:*
e32ea7e74727("soreuseport: fast reuseport UDP socket selection")
*c125e80b8868("soreuseport: fast reuseport TCP socket selection")These commits introduced the O(1) socket selection algorithm and removed O(n) iteration over the list, but it ignores the score calculated by
compute_score(). As a result, it caused two misbehaviours:[...]
Here is the summary with links:
- [v3,net-next,1/2] soreuseport: Fix socket selection for SO_INCOMING_CPU.
https://git.kernel.org/netdev/net-next/c/b261eda84ec1 - [v3,net-next,2/2] selftest: Add test for SO_INCOMING_CPU. https://git.kernel.org/netdev/net-next/c/6df96146b202
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html