bpf: Don't hold pack_mutex over arm64 IPIs
PATCH 1/1 bpf: Don't hold pack_mutex over arm64 IPIs
From: Alexei Starovoitov <[email protected]>
bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex held. On arm64 it interrupts all other CPUs and waits for them:
bpf_prog_free_deferred
bpf_jit_free
bpf_prog_pack_free
bpf_arch_text_invalidate
aarch64_insn_set
flush_icache_range
kick_all_cpus_sync
When many progs are freed at once, every prog load waits in bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with many CPUs we saw hung task reports with sk_attach_filter() blocked on pack_mutex for minutes.
The bits of the range are still set in the bitmap during the invalidation, so nobody can allocate it and the pack cannot be freed. Drop pack_mutex around bpf_arch_text_invalidate() and take it again to clear the bits.
Do it on arm64 only. x86, riscv, powerpc and loongarch take text_mutex in bpf_arch_text_invalidate(). The frees would wait there instead, and the load would wait for them in bpf_arch_text_copy().
Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2 processes attach a cBPF filter to 256 sockets and close them in a loop for 20 sec. One more process attaches a filter every msec. Median of 5 runs.
The loops get more done. Sockets attached and closed in 20 sec, in millions (higher is better):
before after
16 cpus 1.6 4.7
96 cpus 0.38 4.2
setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec takes less time, in msec (lower is better):
16 cpus 96 cpus
before after before after
avg 0.97 0.18 11 0.54
p99 20 2.8 421 4.0
max 204 90 1738 255
--- kernel/bpf/core.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/bpf/core.c +9 -0
@@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size) nbits =BPF_PROG_SIZE_TO_NBITS (size); pos = ((unsigned long )ptr - (unsigned long )pack->ptr) >>BPF_PROG_CHUNK_SHIFT ; +/* +* arm64 sends IPI to all CPUs in bpf_arch_text_invalidate(). +* Don't hold pack_mutex across it. The bits are still set, +* so the range cannot be reused and the pack cannot be freed. +*/ +if (IS_ENABLED (CONFIG_ARM64 )) +mutex_unlock (&pack_mutex);WARN_ONCE (bpf_arch_text_invalidate (ptr, size),"bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n" ); +if (IS_ENABLED (CONFIG_ARM64 )) +mutex_lock (&pack_mutex);bitmap_clear (pack->bitmap, pos, nbits);
base-commit: b116ac6087541d3d98031e786adbe9c733cd6df3
Signature · 1 line
2.55.0
Quoted 62 lines · Alexei Starovoitov
From: Alexei Starovoitov <[email protected]>
bpf_prog_pack_free()callsbpf_arch_text_invalidate()withpack_mutexheld. On arm64 it interrupts all other CPUs and waits for them:
bpf_prog_free_deferred
bpf_jit_free
bpf_prog_pack_free
bpf_arch_text_invalidate
aarch64_insn_set
flush_icache_range
kick_all_cpus_syncWhen many progs are freed at once, every prog load waits in
bpf_prog_pack_alloc()for all of these broadcasts. On arm64 hosts with many CPUs we saw hung task reports withsk_attach_filter()blocked onpack_mutexfor minutes.The bits of the range are still set in the bitmap during the invalidation, so nobody can allocate it and the pack cannot be freed. Drop
pack_mutexaroundbpf_arch_text_invalidate()and take it again to clear the bits.Do it on arm64 only. x86, riscv, powerpc and loongarch take
text_mutexinbpf_arch_text_invalidate(). The frees would wait there instead, and the load would wait for them inbpf_arch_text_copy().Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2 processes attach a cBPF filter to 256 sockets and close them in a loop for 20 sec. One more process attaches a filter every msec. Median of 5 runs.
The loops get more done. Sockets attached and closed in 20 sec, in millions (higher is better):
before after
16 cpus 1.6 4.7
96 cpus 0.38 4.2
setsockopt(SO_ATTACH_FILTER)of the process that attaches every msec takes less time, in msec (lower is better):16 cpus 96 cpus
before after before after
avg 0.97 0.18 11 0.54
p99 20 2.8 421 4.0
max 204 90 1738 255Fixes 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")Signed-off-by Alexei Starovoitov <[email protected]>--- kernel/bpf/core.c | 9 +++++++++ 1 file changed, 9 insertions(+)kernel/bpf/core.c +0
-0@@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size) nbits =BPF_PROG_SIZE_TO_NBITS (size); pos = ((unsigned long )ptr - (unsigned long )pack->ptr) >>BPF_PROG_CHUNK_SHIFT ;
kernel/bpf/core.c void bpf_prog_pack_free(void *ptr, u32 size) · line 1081
Quoted 10 lines
kernel/bpf/core.c +8
-0+/* +* arm64 sends IPI to all CPUs in bpf_arch_text_invalidate(). +* Don't hold pack_mutex across it. The bits are still set, +* so the range cannot be reused and the pack cannot be freed. +*/ +if (IS_ENABLED (CONFIG_ARM64 )) +mutex_unlock (&pack_mutex);WARN_ONCE (bpf_arch_text_invalidate (ptr, size),"bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n" ); +if (IS_ENABLED (CONFIG_ARM64 ))
Can't we do the bpf_arch_text_invalidate() before taking the pack_mutex? then it works on all architectures and there is no lock nesting.
Because the caller owns [ptr, ptr+size) until bitmap_clear() so it should work.
I can commandeer it if you want.
Thanks,
Puranjay
Quoted 5 lines · Puranjay Mohan
Can't we do the
bpf_arch_text_invalidate()before taking thepack_mutex? then it works on all architectures and there is no lock nesting.Because the caller owns [ptr, ptr+size) until
bitmap_clear()so it should work.
Interesting idea. I guess that's simpler. bpf_arch_text_copy() already writes into a pack without pack_mutex.
but other archs gain nothing. fine.
Quoted 1 line
I can commandeer it if you want.
Sure. pls go ahead.
pw-bot:cr
first time for my own patches. next level unlocked :)