Skip to contentSkip to conversation

bpf: Don't hold pack_mutex over arm64 IPIs

PATCH In review Alexei Starovoitov 2 participants 3 messages Oct 24 – Oct 28

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpf: Don't hold pack_mutex over arm64 IPIs

From: Alexei Starovoitov <[email protected]>

bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex held. On arm64 it interrupts all other CPUs and waits for them:

bpf_prog_free_deferred
bpf_jit_free
bpf_prog_pack_free
bpf_arch_text_invalidate
aarch64_insn_set
flush_icache_range
kick_all_cpus_sync

When many progs are freed at once, every prog load waits in bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with many CPUs we saw hung task reports with sk_attach_filter() blocked on pack_mutex for minutes.

The bits of the range are still set in the bitmap during the invalidation, so nobody can allocate it and the pack cannot be freed. Drop pack_mutex around bpf_arch_text_invalidate() and take it again to clear the bits.

Do it on arm64 only. x86, riscv, powerpc and loongarch take text_mutex in bpf_arch_text_invalidate(). The frees would wait there instead, and the load would wait for them in bpf_arch_text_copy().

Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2 processes attach a cBPF filter to 256 sockets and close them in a loop for 20 sec. One more process attaches a filter every msec. Median of 5 runs.

The loops get more done. Sockets attached and closed in 20 sec, in millions (higher is better):

before after
16 cpus 1.6 4.7
96 cpus 0.38 4.2

setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec takes less time, in msec (lower is better):

16 cpus 96 cpus
before after before after
avg 0.97 0.18 11 0.54
p99 20 2.8 421 4.0
max 204 90 1738 255

Fixes 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")
---
 kernel/bpf/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)
kernel/bpf/core.c +9 -0
index 3030beb531d9..0fef87cfa152 100644
@@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size)
 	nbits = BPF_PROG_SIZE_TO_NBITS(size);
 	pos = ((unsigned long)ptr - (unsigned long)pack->ptr) >> BPF_PROG_CHUNK_SHIFT;
 
+	/*
+	 * arm64 sends IPI to all CPUs in bpf_arch_text_invalidate().
+	 * Don't hold pack_mutex across it. The bits are still set,
+	 * so the range cannot be reused and the pack cannot be freed.
+	 */
+	if (IS_ENABLED(CONFIG_ARM64))
+		mutex_unlock(&pack_mutex);
 	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
 		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
+	if (IS_ENABLED(CONFIG_ARM64))
+		mutex_lock(&pack_mutex);
 
 	bitmap_clear(pack->bitmap, pos, nbits);
 

base-commit: b116ac6087541d3d98031e786adbe9c733cd6df3

Signature · 1 line
2.55.0
#1 lore raw
Quoted 62 lines · Alexei Starovoitov

From: Alexei Starovoitov <[email protected]>

bpf_prog_pack_free() calls bpf_arch_text_invalidate() with pack_mutex held. On arm64 it interrupts all other CPUs and waits for them:

bpf_prog_free_deferred
bpf_jit_free
bpf_prog_pack_free
bpf_arch_text_invalidate
aarch64_insn_set
flush_icache_range
kick_all_cpus_sync

When many progs are freed at once, every prog load waits in bpf_prog_pack_alloc() for all of these broadcasts. On arm64 hosts with many CPUs we saw hung task reports with sk_attach_filter() blocked on pack_mutex for minutes.

The bits of the range are still set in the bitmap during the invalidation, so nobody can allocate it and the pack cannot be freed. Drop pack_mutex around bpf_arch_text_invalidate() and take it again to clear the bits.

Do it on arm64 only. x86, riscv, powerpc and loongarch take text_mutex in bpf_arch_text_invalidate(). The frees would wait there instead, and the load would wait for them in bpf_arch_text_copy().

Tested in arm64 KVM guests with 16 and 96 cpus on a 96 cpu host. cpus/2 processes attach a cBPF filter to 256 sockets and close them in a loop for 20 sec. One more process attaches a filter every msec. Median of 5 runs.

The loops get more done. Sockets attached and closed in 20 sec, in millions (higher is better):

before after
16 cpus 1.6 4.7
96 cpus 0.38 4.2

setsockopt(SO_ATTACH_FILTER) of the process that attaches every msec takes less time, in msec (lower is better):

16 cpus 96 cpus
before after before after
avg 0.97 0.18 11 0.54
p99 20 2.8 421 4.0
max 204 90 1738 255

Fixes 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")
---
 kernel/bpf/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)
kernel/bpf/core.c +0 -0
index 3030beb531d9..0fef87cfa152 100644
@@ -1069,8 +1069,17 @@ void bpf_prog_pack_free(void *ptr, u32 size)
 	nbits = BPF_PROG_SIZE_TO_NBITS(size);
 	pos = ((unsigned long)ptr - (unsigned long)pack->ptr) >> BPF_PROG_CHUNK_SHIFT;
 

kernel/bpf/core.c void bpf_prog_pack_free(void *ptr, u32 size) · line 1081

Quoted 10 lines
kernel/bpf/core.c +8 -0
+	/*
+	 * arm64 sends IPI to all CPUs in bpf_arch_text_invalidate().
+	 * Don't hold pack_mutex across it. The bits are still set,
+	 * so the range cannot be reused and the pack cannot be freed.
+	 */
+	if (IS_ENABLED(CONFIG_ARM64))
+		mutex_unlock(&pack_mutex);
 	WARN_ONCE(bpf_arch_text_invalidate(ptr, size),
 		  "bpf_prog_pack bug: missing bpf_arch_text_invalidate?\n");
+	if (IS_ENABLED(CONFIG_ARM64))

Can't we do the bpf_arch_text_invalidate() before taking the pack_mutex? then it works on all architectures and there is no lock nesting.

Because the caller owns [ptr, ptr+size) until bitmap_clear() so it should work.

I can commandeer it if you want.

Thanks,
Puranjay

#2 lore raw
Quoted 5 lines · Puranjay Mohan

Can't we do the bpf_arch_text_invalidate() before taking the pack_mutex? then it works on all architectures and there is no lock nesting.

Because the caller owns [ptr, ptr+size) until bitmap_clear() so it should work.

Interesting idea. I guess that's simpler. bpf_arch_text_copy() already writes into a pack without pack_mutex.

but other archs gain nothing. fine.

Quoted 1 line

I can commandeer it if you want.

Sure. pls go ahead.

pw-bot:cr

first time for my own patches. next level unlocked :)