bpf: lru: adjust free target to avoid global table starvation
Subsystems (6)
PATCH v2 1/1 bpf: lru: adjust free target to avoid global table starvation
From: Willem de Bruijn <[email protected]>
BPF_MAP_TYPE_LRU_HASH can recycle most recent elements well before the map is full, due to percpu reservations and force shrink before neighbor stealing. Once a CPU is unable to borrow from the global map, it will once steal one elem from a neighbor and after that each time flush this one element to the global list and immediately recycle it.
Batch value LOCAL_FREE_TARGET (128) will exhaust a 10K element map with 79 CPUs. CPU 79 will observe this behavior even while its neighbors hold 78 * 127 + 1 * 15 == 9921 free elements (99%).
CPUs need not be active concurrently. The issue can appear with affinity migration, e.g., irqbalance. Each CPU can reserve and then hold onto its 128 elements indefinitely.
Avoid global list exhaustion by limiting aggregate percpu caches to half of map size, by adjusting LOCAL_FREE_TARGET based on cpu count. This change has no effect on sufficiently large tables.
Similar to LOCAL_NR_SCANS and lru->nr_scans, introduce a map variable lru->free_target. The extra field fits in a hole in struct bpf_lru. The cacheline is already warm where read in the hot path. The field is only accessed with the lru lock held.
---
Changes
v1 -> v2
- Update documentation (Stanislav)
- Add Tested-by (Anton)
- Update test comments (As mentioned in v1)
v1: https://lore.kernel.org/netdev/[email protected]/
--- Documentation/bpf/map_hash.rst | 8 ++- Documentation/bpf/map_lru_hash_update.dot | 6 +- kernel/bpf/bpf_lru_list.c | 9 ++- kernel/bpf/bpf_lru_list.h | 1 + tools/testing/selftests/bpf/test_lru_map.c | 72 +++++++++++----------- 5 files changed, 52 insertions(+), 44 deletions(-)
Documentation/bpf/map_hash.rst +7 -1
@@ -233,10 +233,16 @@ attempts in order to enforce the LRU property which have increasing impacts on other CPUs involved in the following operation attempts: - Attempt to use CPU-local state to batch operations -- Attempt to fetch free nodes from global lists +- Attempt to fetch ``target_free`` free nodes from global lists - Attempt to pull any node from a global list and remove it from the hashmap - Attempt to pull any node from any CPU's list and remove it from the hashmap +The number of nodes to borrow from the global list in a batch, ``target_free``, +depends on the size of the map. Larger batch size reduces lock contention, but +may also exhaust the global structure. The value is computed at map init to +avoid exhaustion, by limiting aggregate reservation by all CPUs to half the map +size. With a minimum of a single element and maximum budget of 128 at a time. + This algorithm is described visually in the following diagram. See the description in commit 3a08c2fd7634 ("bpf: LRU List") for a full explanation of the corresponding operations:
Documentation/bpf/map_lru_hash_update.dot +3 -3
@@ -35,18 +35,18 @@ digraph { fn_bpf_lru_list_pop_free_to_local [shape=rectangle,fillcolor=2, label="Flush local pending, Rotate Global list, move - LOCAL_FREE_TARGET + target_free from global -> local"] // Also corresponds to: // fn__local_list_flush() // fn_bpf_lru_list_rotate() fn___bpf_lru_node_move_to_free[shape=diamond,fillcolor=2, - label="Able to free\nLOCAL_FREE_TARGET\nnodes?"] + label="Able to free\ntarget_free\nnodes?"] fn___bpf_lru_list_shrink_inactive [shape=rectangle,fillcolor=3, label="Shrink inactive list up to remaining - LOCAL_FREE_TARGET + target_free (global LRU -> local)"] fn___bpf_lru_list_shrink [shape=diamond,fillcolor=2, label="> 0 entries in\nlocal free list?"]
kernel/bpf/bpf_lru_list.c +6 -3
@@ -337,12 +337,12 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, list) {__bpf_lru_node_move_to_free (l, node,local_free_list (loc_l),BPF_LRU_LOCAL_LIST_T_FREE ); -if (++nfree ==LOCAL_FREE_TARGET ) +if (++nfree == lru->target_free)break ; } -if (nfree <LOCAL_FREE_TARGET ) -__bpf_lru_list_shrink (lru, l,LOCAL_FREE_TARGET - nfree, +if (nfree < lru->target_free) +__bpf_lru_list_shrink (lru, l, lru->target_free - nfree,local_free_list (loc_l),BPF_LRU_LOCAL_LIST_T_FREE );
@@ -577,6 +577,9 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf,list_add (&node->list, &l->lists[BPF_LRU_LIST_T_FREE ]); buf += elem_size; } + + lru->target_free =clamp ((nr_elems /num_possible_cpus ()) /2 , +1 ,LOCAL_FREE_TARGET ); } static void bpf_percpu_lru_populate(struct bpf_lru *lru, void *buf,
kernel/bpf/bpf_lru_list.h +1 -0
@@ -58,6 +58,7 @@ struct bpf_lru {del_from_htab_func del_from_htab;void *del_arg;unsigned int hash_offset; +unsigned int target_free;unsigned int nr_scans;bool percpu; };
tools/testing/selftests/bpf/test_lru_map.c +35 -37
156 lines not on this page · raw message
Signature · 1 line
2.50.0.rc2.701.gf1e915cc24-goog
Quoted 20 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
BPF_MAP_TYPE_LRU_HASHcan recycle most recent elements well before the map is full, due to percpu reservations and force shrink before neighbor stealing. Once a CPU is unable to borrow from the global map, it will once steal one elem from a neighbor and after that each time flush this one element to the global list and immediately recycle it.Batch value
LOCAL_FREE_TARGET(128) will exhaust a 10K element map with 79 CPUs. CPU 79 will observe this behavior even while its neighbors hold 78 * 127 + 1 * 15 == 9921 free elements (99%).CPUs need not be active concurrently. The issue can appear with affinity migration, e.g., irqbalance. Each CPU can reserve and then hold onto its 128 elements indefinitely.
Avoid global list exhaustion by limiting aggregate percpu caches to half of map size, by adjusting
LOCAL_FREE_TARGETbased on cpu count. This change has no effect on sufficiently large tables.
Quoted 7 lines
Similar to
LOCAL_NR_SCANSandlru->nr_scans, introduce a map variablelru->free_target. The extra field fits in a hole in structbpf_lru. The cacheline is already warm where read in the hot path. The field is only accessed with the lru lock held.Tested-by Anton Protopopov <[email protected]>Signed-off-by Willem de Bruijn <[email protected]>
This patch was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This patch was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Wed, 18 Jun 2025 17:57:40 -0400 you wrote:
From: Willem de Bruijn <[email protected]>
BPF_MAP_TYPE_LRU_HASHcan recycle most recent elements well before the map is full, due to percpu reservations and force shrink before neighbor stealing. Once a CPU is unable to borrow from the global map, it will once steal one elem from a neighbor and after that each time flush this one element to the global list and immediately recycle it.[...]
Here is the summary with links:
- [bpf,v2] bpf: lru: adjust free target to avoid global table starvation
https://git.kernel.org/bpf/bpf/c/d4adf1c9ee77
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html