bpf: Try harder when allocating memory for large maps
Subsystems (4)
PATCH 1/1 bpf: Try harder when allocating memory for large maps
It has been observed that sometimes a higher order memory allocation for BPF maps fails when there is no obvious memory pressure in a system.
E.g. the map (BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) could not be created due to vmalloc unable to allocate 75497472B, when the system's memory consumption (in MB) was the following:
Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727
Later analysis [1] by Michal Hocko showed that the vmalloc was not trying to reclaim memory from the page cache and was failing prematurely due to __GFP_NORETRY.
Considering dcda9b0471 ("mm, tree wide: replace __GFP_REPEAT by __GFP_RETRY_MAYFAIL with more useful semantic") and [1], we can replace __GFP_NORETRY with __GFP_RETRY_MAYFAIL, as it won't invoke OOM killer and will try harder to fulfil allocation requests.
The change has been tested with the workloads mentioned above and by observing oom_kill value from /proc/vmstat.
[1]: https://lore.kernel.org/bpf/[email protected]/
--- kernel/bpf/syscall.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/bpf/syscall.c +12 -6
@@ -136,20 +136,26 @@ static struct bpf_map *find_and_alloc_map(union bpf_attr *attr)void *bpf_map_area_alloc (size_t size,int numa_node) { -/* We definitely need __GFP_NORETRY, so OOM killer doesn't -* trigger under memory pressure as we really just want to -* fail instead. +/* We definitely need __GFP_NORETRY or __GFP_RETRY_MAYFAIL, so +* OOM killer doesn't trigger under memory pressure as we really +* just want to fail instead. */ -const gfp_t flags = __GFP_NOWARN | __GFP_NORETRY | __GFP_ZERO; +const gfp_t flags = __GFP_NOWARN | __GFP_ZERO;void *area;if (size <= (PAGE_SIZE <<PAGE_ALLOC_COSTLY_ORDER )) { - area =kmalloc_node (size,GFP_USER | flags, numa_node); +/* To avoid bypassing slab alloc for lower order allocs, +* __GFP_NORETRY is used instead of __GFP_RETRY_MAYFAIL. +*/ + area =kmalloc_node (size,GFP_USER | __GFP_NORETRY | flags, + numa_node);if (area !=NULL )return area; } -return __vmalloc_node_flags_caller (size, numa_node,GFP_KERNEL | flags, +return __vmalloc_node_flags_caller (size, numa_node, +GFP_KERNEL | __GFP_RETRY_MAYFAIL | + flags,__builtin_return_address (0 )); }
Signature · 1 line
2.21.0
Quoted 13 lines · Martynas Pumputis
It has been observed that sometimes a higher order memory allocation for BPF maps fails when there is no obvious memory pressure in a system.
E.g. the map (
BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) could not be created due to vmalloc unable to allocate 75497472B, when the system's memory consumption (in MB) was the following:Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727Later analysis [1] by Michal Hocko showed that the vmalloc was not trying to reclaim memory from the page cache and was failing prematurely due to
__GFP_NORETRY.
Quoted 7 lines
Considering
dcda9b0471("mm, tree wide: replace__GFP_REPEATby__GFP_RETRY_MAYFAILwith more useful semantic") and [1], we can replace__GFP_NORETRYwith__GFP_RETRY_MAYFAIL, as it won't invoke OOM killer and will try harder to fulfil allocation requests.The change has been tested with the workloads mentioned above and by observing
oom_killvalue from/proc/vmstat.
Please document why kvmalloc_node is not used.
Quoted 47 lines
[1]: https://lore.kernel.org/bpf/[email protected]/
Signed-off-by Martynas Pumputis <[email protected]>--- kernel/bpf/syscall.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)kernel/bpf/syscall.c +12
-6@@ -136,20 +136,26 @@ static struct bpf_map *find_and_alloc_map(union bpf_attr *attr)void *bpf_map_area_alloc (size_t size,int numa_node) { -/* We definitely need __GFP_NORETRY, so OOM killer doesn't -* trigger under memory pressure as we really just want to -* fail instead. +/* We definitely need __GFP_NORETRY or __GFP_RETRY_MAYFAIL, so +* OOM killer doesn't trigger under memory pressure as we really +* just want to fail instead. */ -const gfp_t flags = __GFP_NOWARN | __GFP_NORETRY | __GFP_ZERO; +const gfp_t flags = __GFP_NOWARN | __GFP_ZERO;void *area;if (size <= (PAGE_SIZE <<PAGE_ALLOC_COSTLY_ORDER )) { - area =kmalloc_node (size,GFP_USER | flags, numa_node); +/* To avoid bypassing slab alloc for lower order allocs, +* __GFP_NORETRY is used instead of __GFP_RETRY_MAYFAIL. +*/ + area =kmalloc_node (size,GFP_USER | __GFP_NORETRY | flags, + numa_node);if (area !=NULL )return area; } -return __vmalloc_node_flags_caller (size, numa_node,GFP_KERNEL | flags, +return __vmalloc_node_flags_caller (size, numa_node, +GFP_KERNEL | __GFP_RETRY_MAYFAIL | + flags,__builtin_return_address (0 )); }Signature · 1 line
2.21.0
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 19 lines · Martynas Pumputis
It has been observed that sometimes a higher order memory allocation for BPF maps fails when there is no obvious memory pressure in a system.
E.g. the map (
BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) could not be created due to vmalloc unable to allocate 75497472B, when the system's memory consumption (in MB) was the following:Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727Later analysis [1] by Michal Hocko showed that the vmalloc was not trying to reclaim memory from the page cache and was failing prematurely due to
__GFP_NORETRY.Considering
dcda9b0471("mm, tree wide: replace__GFP_REPEATby__GFP_RETRY_MAYFAILwith more useful semantic") and [1], we can replace__GFP_NORETRYwith__GFP_RETRY_MAYFAIL, as it won't invoke OOM killer and will try harder to fulfil allocation requests.
Quoted 6 lines
The change has been tested with the workloads mentioned above and by observing
oom_killvalue from/proc/vmstat.[1]: https://lore.kernel.org/bpf/[email protected]/
Signed-off-by Martynas Pumputis <[email protected]>