bpf, arm64/riscv: Remove redundant icache flush after pack allocator finalize

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management"), this flush is redundant.

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> aarch64_insn_copy() -> __text_poke(), and __text_poke() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range, including an unnecessary second synchronous IPI to all CPUs via kick_all_cpus_sync().

Remove the redundant bpf_flush_icache() call and its now-unused definition.

Fixes 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")

Acked-by Song Liu <[email protected]>

Signed-off-by Puranjay Mohan <[email protected]>

---
  arch/arm64/net/bpf_jit_comp.c | 12 ------------
  1 file changed, 12 deletions(-)

arch/arm64/net/bpf_jit_comp.c +0 -12

index adf84962d579..7417d24a5b6f 100644
@@ -18,7 +18,6 @@
 
  #include <asm/asm-extable.h>
  #include <asm/byteorder.h>
-#include <asm/cacheflush.h>
  #include <asm/cpufeature.h>
  #include <asm/debug-monitors.h>
  #include <asm/insn.h>

@@ -1961,11 +1960,6 @@ static int validate_ctx(struct jit_ctx *ctx)
  	return 0;
  }
 
-static inline void bpf_flush_icache(void *start, void *end)
-{
-	flush_icache_range((unsigned long)start, (unsigned long)end);
-}
-
  static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)
  {
  	int cpu, underflow_idx = (alloc_size - PRIV_STACK_GUARD_SZ) >> 3;

@@ -2204,12 +2198,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  			prog = orig_prog;
  			goto out_off;
  		}
-		/*
-		 * The instructions have now been copied to the ROX region from
-		 * where they will execute. Now the data cache has to be cleaned to
-		 * the PoU and the I-cache has to be invalidated for the VAs.
-		 */
-		bpf_flush_icache(ro_header, ctx.ro_image + ctx.idx);
  	} else {
  		jit_data->ctx = ctx;
  		jit_data->ro_image = ro_image_ptr;

Quoted 5 lines · Puranjay Mohan

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management"), this flush is redundant.

Quoted 7 lines

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> aarch64_insn_copy() -> __text_poke(), and __text_poke() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range, including an unnecessary second synchronous IPI to all CPUs via kick_all_cpus_sync().

So icache is flushed twice: once per instruction and again after all instructions are copied. I think it's better to remove the per-instruction flush and retain the single final flush to avoid repeating flush overhead for each instruction.

Quoted 15 lines · Xu Kuohai

Quoted 5 lines · Puranjay Mohan

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management"), this flush is redundant.

Quoted 7 lines

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> aarch64_insn_copy() -> __text_poke(), and __text_poke() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range, including an unnecessary second synchronous IPI to all CPUs via kick_all_cpus_sync().

Quoted 4 lines

So icache is flushed twice: once per instruction and again after all instructions are copied. I think it's better to remove the per-instruction flush and retain the single final flush to avoid repeating flush overhead for each instruction.

No, bpf_jit_binary_pack_finalize() is called at the end after the whole program is jited, and it calls: bpf_arch_text_copy(ro_header, rw_header, rw_header->size); which does aarch64_insn_copy(dst, src, len), this calls __text_poke() which copies the whole program and then does flush_icache_range((uintptr_t)addr, (uintptr_t)addr + len); once. This is correct, after this we don't need to call flush_icache_range() on the same range again.

If we had been calling flush_icache_range() for each instruction, the system would hang due to the storm of IPIs.

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management"), this flush is redundant.

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> aarch64_insn_copy() -> __text_poke(), and __text_poke() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range, including an unnecessary second synchronous IPI to all CPUs via kick_all_cpus_sync().

Remove the redundant bpf_flush_icache() call and its now-unused definition.

Fixes 1dad391daef1 ("bpf, arm64: use bpf_prog_pack for memory management")

Acked-by Song Liu <[email protected]>

Signed-off-by Puranjay Mohan <[email protected]>

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 48a8f78c50bd ("bpf, riscv: use prog pack allocator in the BPF JIT"), this flush is redundant.

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> patch_text_nosync(), and patch_text_nosync() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range.

Remove the redundant bpf_flush_icache() call and its now-unused definition.

Fixes 48a8f78c50bd ("bpf, riscv: use prog pack allocator in the BPF JIT")

Acked-by Song Liu <[email protected]>

Signed-off-by Puranjay Mohan <[email protected]>

---
  arch/riscv/net/bpf_jit.h      | 6 ------
  arch/riscv/net/bpf_jit_core.c | 7 -------
  2 files changed, 13 deletions(-)

arch/riscv/net/bpf_jit.h +0 -6

index 632ced07bca4..da0271790244 100644
@@ -11,7 +11,6 @@
 
  #include <linux/bpf.h>
  #include <linux/filter.h>
-#include <asm/cacheflush.h>
 
  /* verify runtime detection extension status */
  #define rv_ext_enabled(ext) \

@@ -105,11 +104,6 @@ static inline void bpf_fill_ill_insns(void *area, unsigned int size)
  	memset(area, 0, size);
  }
 
-static inline void bpf_flush_icache(void *start, void *end)
-{
-	flush_icache_range((unsigned long)start, (unsigned long)end);
-}
-
  /* Emit a 4-byte riscv instruction. */
  static inline void emit(const u32 insn, struct rv_jit_context *ctx)
  {

arch/riscv/net/bpf_jit_core.c +0 -0

index b3581e926436..f7fd4afc3ca3 100644
@@ -183,13 +183,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  			prog = orig_prog;
  			goto out_offset;
  		}

bpf_flush_icache() calls flush_icache_range() to clean the data cache and invalidate the instruction cache for the JITed code region. However, since commit 48a8f78c50bd ("bpf, riscv: use prog pack allocator in the BPF JIT"), this flush is redundant.

bpf_jit_binary_pack_finalize() copies the JITed instructions to the ROX region via bpf_arch_text_copy() -> patch_text_nosync(), and patch_text_nosync() already calls flush_icache_range() on the written range. The subsequent bpf_flush_icache() repeats the same cache maintenance on an overlapping range.

Remove the redundant bpf_flush_icache() call and its now-unused definition.

Fixes 48a8f78c50bd ("bpf, riscv: use prog pack allocator in the BPF JIT")

Acked-by Song Liu <[email protected]>

Signed-off-by Puranjay Mohan <[email protected]>