tcp: sk_forced_mem_schedule() optimization
Subsystems (3)
PATCH 1/1 tcp: sk_forced_mem_schedule() optimization
From: Eric Dumazet <[email protected]>
sk_memory_allocated_add() has three callers, and returns to them @memory_allocated.
sk_forced_mem_schedule() is one of them, and ignores the returned value.
Change sk_memory_allocated_add() to return void.
Change sock_reserve_memory() and __sk_mem_raise_allocated() to call sk_memory_allocated().
This removes one cache line miss [1] for RPC workloads, as first skbs in TCP write queue and receive queue go through sk_forced_mem_schedule().
[1] Cache line holding tcp_memory_allocated.
--- include/net/sock.h | 3 +-- net/core/sock.c | 9 ++++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
include/net/sock.h +1 -2
@@ -1412,7 +1412,7 @@ sk_memory_allocated(const struct sock *sk)/* 1 MB per cpu, in page units */ #define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT)) -static inline long +static inline void sk_memory_allocated_add (struct sock *sk,int amt) {int local_reserve;
@@ -1424,7 +1424,6 @@ sk_memory_allocated_add(struct sock *sk, int amt)atomic_long_add (local_reserve, sk->sk_prot->memory_allocated); }preempt_enable (); -return sk_memory_allocated (sk); } static inline void
net/core/sock.c +6 -3
@@ -1019,7 +1019,8 @@ static int sock_reserve_memory(struct sock *sk, int bytes)return -ENOMEM ;/* pre-charge to forward_alloc */ - allocated =sk_memory_allocated_add (sk, pages); +sk_memory_allocated_add (sk, pages); + allocated =sk_memory_allocated (sk);/* If the system goes into memory pressure with this * precharge, give up and return error. */
@@ -2906,11 +2907,13 @@ EXPORT_SYMBOL(sk_wait_data);*/ int __sk_mem_raise_allocated (struct sock *sk,int size,int amt,int kind) { -struct proto *prot = sk->sk_prot; -long allocated =sk_memory_allocated_add (sk, amt);bool memcg_charge = mem_cgroup_sockets_enabled && sk->sk_memcg; +struct proto *prot = sk->sk_prot;bool charged = true; +long allocated; +sk_memory_allocated_add (sk, amt); + allocated =sk_memory_allocated (sk);if (memcg_charge && !(charged =mem_cgroup_charge_skmem (sk->sk_memcg, amt,gfp_memcg_charge ())))
Signature · 1 line
2.36.1.476.g0c4daa206d-goog
Quoted 14 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
sk_memory_allocated_add()has three callers, and returns to them @memory_allocated.
sk_forced_mem_schedule()is one of them, and ignores the returned value.Change
sk_memory_allocated_add()to return void.Change
sock_reserve_memory()and__sk_mem_raise_allocated()to callsk_memory_allocated().
Quoted 7 lines
This removes one cache line miss [1] for RPC workloads, as first skbs in TCP write queue and receive queue go through
sk_forced_mem_schedule().[1] Cache line holding
tcp_memory_allocated.Signed-off-by Eric Dumazet <[email protected]>
Nice find!
Quoted 59 lines
--- include/net/sock.h | 3 +-- net/core/sock.c | 9 ++++++--- 2 files changed, 7 insertions(+), 5 deletions(-)include/net/sock.h +1
-2@@ -1412,7 +1412,7 @@ sk_memory_allocated(const struct sock *sk)/* 1 MB per cpu, in page units */ #define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT)) -static inline long +static inline void sk_memory_allocated_add (struct sock *sk,int amt) {int local_reserve;@@ -1424,7 +1424,6 @@ sk_memory_allocated_add(struct sock *sk, int amt)atomic_long_add (local_reserve, sk->sk_prot->memory_allocated); }preempt_enable (); -return sk_memory_allocated (sk); } static inline voidnet/core/sock.c +6
-4@@ -1019,7 +1019,8 @@ static int sock_reserve_memory(struct sock *sk, int bytes)return -ENOMEM ;/* pre-charge to forward_alloc */ - allocated =sk_memory_allocated_add (sk, pages); +sk_memory_allocated_add (sk, pages); + allocated =sk_memory_allocated (sk);/* If the system goes into memory pressure with this * precharge, give up and return error. */ @@ -2906,11 +2907,13 @@ EXPORT_SYMBOL(sk_wait_data);*/ int __sk_mem_raise_allocated (struct sock *sk,int size,int amt,int kind) { -struct proto *prot = sk->sk_prot; -long allocated =sk_memory_allocated_add (sk, amt);bool memcg_charge = mem_cgroup_sockets_enabled && sk->sk_memcg; +struct proto *prot = sk->sk_prot;bool charged = true; +long allocated; +sk_memory_allocated_add (sk, amt); + allocated =sk_memory_allocated (sk);if (memcg_charge && !(charged =mem_cgroup_charge_skmem (sk->sk_memcg, amt,gfp_memcg_charge ()))) --2.36.1.476.g0c4daa206d-goog
Quoted 14 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
sk_memory_allocated_add()has three callers, and returns to them @memory_allocated.
sk_forced_mem_schedule()is one of them, and ignores the returned value.Change
sk_memory_allocated_add()to return void.Change
sock_reserve_memory()and__sk_mem_raise_allocated()to callsk_memory_allocated().
Quoted 7 lines
This removes one cache line miss [1] for RPC workloads, as first skbs in TCP write queue and receive queue go through
sk_forced_mem_schedule().[1] Cache line holding
tcp_memory_allocated.Signed-off-by Eric Dumazet <[email protected]>
This patch was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 9 lines · On Fri, 10 Jun 2022 20:30:16 -0700 you wrote:
From: Eric Dumazet <[email protected]>
sk_memory_allocated_add()has three callers, and returns to them @memory_allocated.
sk_forced_mem_schedule()is one of them, and ignores the returned value.[...]
Here is the summary with links:
- [net-next] tcp: sk_forced_mem_schedule() optimization
https://git.kernel.org/netdev/net-next/c/219160be496f
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html