Skip to contentSkip to conversation

bpf, riscv: use BPF prog pack allocator in BPF JIT

PATCH Applied Puranjay Mohan 4 participants 8 messages Aug 31, 2023 – Sep 6, 2023

Subsystems (5)

v4 of 9 Applied 4/4 Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover bpf, riscv: use BPF prog pack allocator in BPF JIT

Changes in v3 -> v4:
1. Add Acked-by:, Tested-by:, etc.
2. Add the core BPF patch[3] which was earlier sent with ARM64 series to this series so it can go with this.

Changes in v2 -> v3:
1. Fix maximum width of code in patches from 80 to 100. [All patches]
2. Add checks for ctx->ro_insns == NULL. [Patch 3]
3. Fix check for edge condition where amount of text to set > 2 * pagesize [Patch 1 and 2]
4. Add reviewed-by in patches.
5. Adding results of selftest here:
Using the command: ./test_progs on qemu
Without the series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED With this series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED

Changes in v1 -> v2:
1. Implement a new function patch_text_set_nosync() to be used in bpf_arch_text_invalidate(). The implementation in v1 called patch_text_nosync() in a loop and it was bad as it would call flush_icache_range() for every word making it really slow. This was found by running the test_tag selftest which would take forever to complete.

Here is some data to prove the V2 fixes the problem:

Without this series:

root@rv-selftester:~/src/kselftest/bpf# time ./test_tag
test_tag: OK (40945 tests)
real    7m47.562s
user    0m24.145s
sys     6m37.064s

With this series applied:

root@rv-selftester:~/src/selftest/bpf# time ./test_tag
test_tag: OK (40945 tests)
real    7m29.472s
user    0m25.865s
sys     6m18.401s

BPF programs currently consume a page each on RISCV. For systems with many BPF programs, this adds significant pressure to instruction TLB. High iTLB pressure usually causes slow down for the whole system.

Song Liu introduced the BPF prog pack allocator[1] to mitigate the above issue. It packs multiple BPF programs into a single huge page. It is currently only enabled for the x86_64 BPF JIT.

I enabled this allocator on the ARM64 BPF JIT[2]. It is being reviewed now.

This patch series enables the BPF prog pack allocator for the RISCV BPF JIT.

======================================================
Performance Analysis of prog pack allocator on RISCV64
======================================================

Test setup:
===========

Host machine: Debian GNU/Linux 11 (bullseye)
Qemu Version: QEMU emulator version 8.0.3 (Debian 1:8.0.3+dfsg-1) u-boot-qemu Version: 2023.07+dfsg-1
opensbi Version: 1.3-1

To test the performance of the BPF prog pack allocator on RV, a stresser tool[4] linked below was built. This tool loads 8 BPF programs on the system and triggers 5 of them in an infinite loop by doing system calls.

The runner script starts 20 instances of the above which loads 8*20=160 BPF programs on the system, 5*20=100 of which are being constantly triggered. The script is passed a command which would be run in the above environment.

The script was run with following perf command:

./run.sh "perf stat -a \
        -e iTLB-load-misses \
        -e dTLB-load-misses  \
        -e dTLB-store-misses \
        -e instructions \
        --timeout 60000"

The output of the above command is discussed below before and after enabling the BPF prog pack allocator.

The tests were run on qemu-system-riscv64 with 8 cpus, 16G memory. The rootfs was created using Bjorn's riscv-cross-builder[5] docker container linked below.

Results
=======

Before enabling prog pack allocator:
------------------------------------

Performance counter stats for 'system wide':

           4939048      iTLB-load-misses
           5468689      dTLB-load-misses
            465234      dTLB-store-misses
     1441082097998      instructions
      60.045791200 seconds time elapsed

After enabling prog pack allocator:
-----------------------------------

Performance counter stats for 'system wide':

           3430035      iTLB-load-misses
           5008745      dTLB-load-misses
            409944      dTLB-store-misses
     1441535637988      instructions
      60.046296600 seconds time elapsed

Improvements in metrics
=======================

It was expected that the iTLB-load-misses would decrease as now a single huge page is used to keep all the BPF programs compared to a single page for each program earlier.

--------------------------------------------
The improvement in iTLB-load-misses: -30.5 %
--------------------------------------------

I repeated this expriment more than 100 times in different setups and the improvement was always greater than 30%.

This patch series is boot tested on the Starfive VisionFive 2 board[6]. The performance analysis was not done on the board because it doesn't expose iTLB-load-misses, etc. The stresser program was run on the board to test the loading and unloading of BPF programs

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/puranjaymohan/BPF-Allocator-Bench
[5] https://github.com/bjoto/riscv-cross-builder
[6] https://www.starfivetech.com/en/site/boards

Puranjay Mohan (4):
bpf: make bpf_prog_pack allocator portable
riscv: extend patch_text_nosync() for multiple pages
riscv: implement a memset like function for text bpf, riscv: use prog pack allocator in the BPF JIT

 arch/riscv/include/asm/patch.h  |   1 +
 arch/riscv/kernel/patch.c       | 114 ++++++++++++++++++++++++++++++--
 arch/riscv/net/bpf_jit.h        |   3 +
 arch/riscv/net/bpf_jit_comp64.c |  60 +++++++++++++----
 arch/riscv/net/bpf_jit_core.c   | 106 ++++++++++++++++++++++++-----
 kernel/bpf/core.c               |   8 +--
 6 files changed, 255 insertions(+), 37 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v4 1/4 bpf: make bpf_prog_pack allocator portable

The bpf_prog_pack allocator currently uses module_alloc() and module_memfree() to allocate and free memory. This is not portable because different architectures use different methods for allocating memory for BPF programs. Like ARM64 and riscv use vmalloc()/vfree().

Use bpf_jit_alloc_exec() and bpf_jit_free_exec() for memory management in bpf_prog_pack allocator. Other architectures can override these with their implementation and will be able to use bpf_prog_pack directly.

On architectures that don't override bpf_jit_alloc/free_exec() this is basically a NOP.

---
 kernel/bpf/core.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/bpf/core.c +4 -4
index 0f8f036d8bd1..4e3ce0542e31 100644
@@ -870,7 +870,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
 		       GFP_KERNEL);
 	if (!pack)
 		return NULL;
-	pack->ptr = module_alloc(BPF_PROG_PACK_SIZE);
+	pack->ptr = bpf_jit_alloc_exec(BPF_PROG_PACK_SIZE);
 	if (!pack->ptr) {
 		kfree(pack);
 		return NULL;
@@ -894,7 +894,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
 	mutex_lock(&pack_mutex);
 	if (size > BPF_PROG_PACK_SIZE) {
 		size = round_up(size, PAGE_SIZE);
-		ptr = module_alloc(size);
+		ptr = bpf_jit_alloc_exec(size);
 		if (ptr) {
 			bpf_fill_ill_insns(ptr, size);
 			set_vm_flush_reset_perms(ptr);
@@ -932,7 +932,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr)
 
 	mutex_lock(&pack_mutex);
 	if (hdr->size > BPF_PROG_PACK_SIZE) {
-		module_memfree(hdr);
+		bpf_jit_free_exec(hdr);
 		goto out;
 	}
 
@@ -956,7 +956,7 @@ void bpf_prog_pack_free(struct bpf_binary_header *hdr)
 	if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0,
 				       BPF_PROG_CHUNK_COUNT, 0) == 0) {
 		list_del(&pack->list);
-		module_memfree(pack->ptr);
+		bpf_jit_free_exec(pack->ptr);
 		kfree(pack);
 	}
 out:
Signature · 1 line
2.39.2
#2 lore raw

PATCH v4 2/4 riscv: extend patch_text_nosync() for multiple pages

In reply to Puranjay Mohan

The patch_insn_write() function currently doesn't work for multiple pages of instructions, therefore patch_text_nosync() will fail with a page fault if called with lengths spanning multiple pages.

This commit extends the patch_insn_write() function to support multiple pages by copying at max 2 pages at a time in a loop. This implementation is similar to text_poke_copy() function of x86.

---
 arch/riscv/kernel/patch.c | 37 ++++++++++++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 5 deletions(-)
arch/riscv/kernel/patch.c +32 -5
index 575e71d6c8ae..2c97e246f4dc 100644
@@ -53,12 +53,18 @@ static void patch_unmap(int fixmap)
 }
 NOKPROBE_SYMBOL(patch_unmap);
 
-static int patch_insn_write(void *addr, const void *insn, size_t len)
+static int __patch_insn_write(void *addr, const void *insn, size_t len)
 {
 	void *waddr = addr;
 	bool across_pages = (((uintptr_t) addr & ~PAGE_MASK) + len) > PAGE_SIZE;
 	int ret;
 
+	/*
+	 * Only two pages can be mapped at a time for writing.
+	 */
+	if (len + offset_in_page(addr) > 2 * PAGE_SIZE)
+		return -EINVAL;
+
 	/*
 	 * Before reaching here, it was expected to lock the text_mutex
 	 * already, so we don't need to give another lock here and could
@@ -74,7 +80,7 @@ static int patch_insn_write(void *addr, const void *insn, size_t len)
 		lockdep_assert_held(&text_mutex);
 
 	if (across_pages)
-		patch_map(addr + len, FIX_TEXT_POKE1);
+		patch_map(addr + PAGE_SIZE, FIX_TEXT_POKE1);
 
 	waddr = patch_map(addr, FIX_TEXT_POKE0);
 
@@ -87,15 +93,36 @@ static int patch_insn_write(void *addr, const void *insn, size_t len)
 
 	return ret;
 }
-NOKPROBE_SYMBOL(patch_insn_write);
+NOKPROBE_SYMBOL(__patch_insn_write);
 #else
-static int patch_insn_write(void *addr, const void *insn, size_t len)
+static int __patch_insn_write(void *addr, const void *insn, size_t len)
 {
 	return copy_to_kernel_nofault(addr, insn, len);
 }
-NOKPROBE_SYMBOL(patch_insn_write);
+NOKPROBE_SYMBOL(__patch_insn_write);
 #endif /* CONFIG_MMU */
 
+static int patch_insn_write(void *addr, const void *insn, size_t len)
+{
+	size_t patched = 0;
+	size_t size;
+	int ret = 0;
+
+	/*
+	 * Copy the instructions to the destination address, two pages at a time
+	 * because __patch_insn_write() can only handle len <= 2 * PAGE_SIZE.
+	 */
+	while (patched < len && !ret) {
+		size = min_t(size_t, PAGE_SIZE * 2 - offset_in_page(addr + patched), len - patched);
+		ret = __patch_insn_write(addr + patched, insn + patched, size);
+
+		patched += size;
+	}
+
+	return ret;
+}
+NOKPROBE_SYMBOL(patch_insn_write);
+
 int patch_text_nosync(void *addr, const void *insns, size_t len)
 {
 	u32 *tp = addr;
Signature · 1 line
2.39.2
#3 lore raw

PATCH v4 3/4 riscv: implement a memset like function for text

In reply to Puranjay Mohan

The BPF JIT needs to write invalid instructions to RX regions of memory to invalidate removed BPF programs. This needs a function like memset() that can work with RX memory.

Implement patch_text_set_nosync() which is similar to text_poke_set() of x86.

---
 arch/riscv/include/asm/patch.h |  1 +
 arch/riscv/kernel/patch.c      | 77 ++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)
arch/riscv/include/asm/patch.h +1 -0
index 63c98833d510..e88b52d39eac 100644
@@ -7,6 +7,7 @@
 #define _ASM_RISCV_PATCH_H
 
 int patch_text_nosync(void *addr, const void *insns, size_t len);
+int patch_text_set_nosync(void *addr, u8 c, size_t len);
 int patch_text(void *addr, u32 *insns, int ninsns);
 
 extern int riscv_patch_in_stop_machine;
arch/riscv/kernel/patch.c +77 -0

101 lines not on this page · raw message

Signature · 1 line
2.39.2
#4 lore raw

PATCH v4 4/4 bpf, riscv: use prog pack allocator in the BPF JIT

In reply to Puranjay Mohan

Use bpf_jit_binary_pack_alloc() for memory management of JIT binaries in RISCV BPF JIT. The bpf_jit_binary_pack_alloc creates a pair of RW and RX buffers. The JIT writes the program into the RW buffer. When the JIT is done, the program is copied to the final RX buffer with bpf_jit_binary_pack_finalize.

Implement bpf_arch_text_copy() and bpf_arch_text_invalidate() for RISCV JIT as these functions are required by bpf_jit_binary_pack allocator.

---
 arch/riscv/net/bpf_jit.h        |   3 +
 arch/riscv/net/bpf_jit_comp64.c |  60 ++++++++++++++----
 arch/riscv/net/bpf_jit_core.c   | 106 +++++++++++++++++++++++++++-----
 3 files changed, 141 insertions(+), 28 deletions(-)
arch/riscv/net/bpf_jit.h +3 -0
index d21c6c92a683..a5ce1ab76ece 100644
@@ -68,6 +68,7 @@ static inline bool is_creg(u8 reg)
 struct rv_jit_context {
 	struct bpf_prog *prog;
 	u16 *insns;		/* RV insns */
+	u16 *ro_insns;
 	int ninsns;
 	int prologue_len;
 	int epilogue_offset;
@@ -85,7 +86,9 @@ static inline int ninsns_rvoff(int ninsns)
 
 struct rv_jit_data {
 	struct bpf_binary_header *header;
+	struct bpf_binary_header *ro_header;
 	u8 *image;
+	u8 *ro_image;
 	struct rv_jit_context ctx;
 };
 
arch/riscv/net/bpf_jit_comp64.c +48 -12

109 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +90 -16

158 lines not on this page · raw message

Signature · 1 line
2.39.2
maintainer Acked-by on the v4 series
#5 lore raw
In reply to Puranjay Mohan
Quoted 124 lines · Puranjay Mohan

Changes in v3 -> v4:
1. Add Acked-by:, Tested-by:, etc.
2. Add the core BPF patch[3] which was earlier sent with ARM64 series to this series so it can go with this.

Changes in v2 -> v3:
1. Fix maximum width of code in patches from 80 to 100. [All patches]
2. Add checks for ctx->ro_insns == NULL. [Patch 3]
3. Fix check for edge condition where amount of text to set > 2 * pagesize

    [Patch 1 and 2]

4. Add reviewed-by in patches.
5. Adding results of selftest here:
Using the command: ./test_progs on qemu
Without the series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED With this series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED

Changes in v1 -> v2:
1. Implement a new function patch_text_set_nosync() to be used in bpf_arch_text_invalidate(). The implementation in v1 called patch_text_nosync() in a loop and it was bad as it would call flush_icache_range() for every word making it really slow. This was found by running the test_tag selftest which would take forever to complete.

Here is some data to prove the V2 fixes the problem:

Without this series:

root@rv-selftester:~/src/kselftest/bpf# time ./test_tag
test_tag: OK (40945 tests)
real    7m47.562s
user    0m24.145s
sys     6m37.064s

With this series applied:

root@rv-selftester:~/src/selftest/bpf# time ./test_tag
test_tag: OK (40945 tests)
real    7m29.472s
user    0m25.865s
sys     6m18.401s

BPF programs currently consume a page each on RISCV. For systems with many BPF programs, this adds significant pressure to instruction TLB. High iTLB pressure usually causes slow down for the whole system.

Song Liu introduced the BPF prog pack allocator[1] to mitigate the above issue. It packs multiple BPF programs into a single huge page. It is currently only enabled for the x86_64 BPF JIT.

I enabled this allocator on the ARM64 BPF JIT[2]. It is being reviewed now.

This patch series enables the BPF prog pack allocator for the RISCV BPF JIT.

======================================================
Performance Analysis of prog pack allocator on RISCV64
======================================================

Test setup:
===========

Host machine: Debian GNU/Linux 11 (bullseye)
Qemu Version: QEMU emulator version 8.0.3 (Debian 1:8.0.3+dfsg-1) u-boot-qemu Version: 2023.07+dfsg-1
opensbi Version: 1.3-1

To test the performance of the BPF prog pack allocator on RV, a stresser tool[4] linked below was built. This tool loads 8 BPF programs on the system and triggers 5 of them in an infinite loop by doing system calls.

The runner script starts 20 instances of the above which loads 8*20=160 BPF programs on the system, 5*20=100 of which are being constantly triggered. The script is passed a command which would be run in the above environment.

The script was run with following perf command:

./run.sh "perf stat -a \
         -e iTLB-load-misses \
         -e dTLB-load-misses  \
         -e dTLB-store-misses \
         -e instructions \
         --timeout 60000"

The output of the above command is discussed below before and after enabling the BPF prog pack allocator.

The tests were run on qemu-system-riscv64 with 8 cpus, 16G memory. The rootfs was created using Bjorn's riscv-cross-builder[5] docker container linked below.

Results
=======

Before enabling prog pack allocator:
------------------------------------

Performance counter stats for 'system wide':

            4939048      iTLB-load-misses
            5468689      dTLB-load-misses
             465234      dTLB-store-misses
      1441082097998      instructions
       60.045791200 seconds time elapsed

After enabling prog pack allocator:
-----------------------------------

Performance counter stats for 'system wide':

            3430035      iTLB-load-misses
            5008745      dTLB-load-misses
             409944      dTLB-store-misses
      1441535637988      instructions
       60.046296600 seconds time elapsed

Improvements in metrics
=======================

It was expected that the iTLB-load-misses would decrease as now a single huge page is used to keep all the BPF programs compared to a single page for each program earlier.

--------------------------------------------
The improvement in iTLB-load-misses: -30.5 %
--------------------------------------------

Quoted 7 lines

I repeated this expriment more than 100 times in different setups and the improvement was always greater than 30%.

This patch series is boot tested on the Starfive VisionFive 2 board[6]. The performance analysis was not done on the board because it doesn't expose iTLB-load-misses, etc. The stresser program was run on the board to test the loading and unloading of BPF programs

@Palmer, as discussed with regards to routing via riscv PR in this merge win - for the series:

Thanks,
Daniel

Quoted 21 lines

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/puranjaymohan/BPF-Allocator-Bench
[5] https://github.com/bjoto/riscv-cross-builder
[6] https://www.starfivetech.com/en/site/boards

Puranjay Mohan (4):
bpf: make bpf_prog_pack allocator portable
riscv: extend patch_text_nosync() for multiple pages
riscv: implement a memset like function for text bpf, riscv: use prog pack allocator in the BPF JIT

  arch/riscv/include/asm/patch.h  |   1 +
  arch/riscv/kernel/patch.c       | 114 ++++++++++++++++++++++++++++++--
  arch/riscv/net/bpf_jit.h        |   3 +
  arch/riscv/net/bpf_jit_comp64.c |  60 +++++++++++++----
  arch/riscv/net/bpf_jit_core.c   | 106 ++++++++++++++++++++++++-----
  kernel/bpf/core.c               |   8 +--
  6 files changed, 255 insertions(+), 37 deletions(-)
#6 lore raw
In reply to Puranjay Mohan
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Hello:

This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Quoted 16 lines · On Thu, 31 Aug 2023 13:12:25 +0000 you wrote:

Changes in v3 -> v4:
1. Add Acked-by:, Tested-by:, etc.
2. Add the core BPF patch[3] which was earlier sent with ARM64 series to this series so it can go with this.

Changes in v2 -> v3:
1. Fix maximum width of code in patches from 80 to 100. [All patches]
2. Add checks for ctx->ro_insns == NULL. [Patch 3]
3. Fix check for edge condition where amount of text to set > 2 * pagesize [Patch 1 and 2]
4. Add reviewed-by in patches.
5. Adding results of selftest here:
Using the command: ./test_progs on qemu
Without the series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED With this series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED

Quoted 1 line

[...]

Here is the summary with links:
- [bpf-next,v4,1/4] bpf: make bpf_prog_pack allocator portable

    https://git.kernel.org/riscv/c/063119d90a06
  - [bpf-next,v4,2/4] riscv: extend patch_text_nosync() for multiple pages
    https://git.kernel.org/riscv/c/fb81d562ed1f
  - [bpf-next,v4,3/4] riscv: implement a memset like function for text
    https://git.kernel.org/riscv/c/f071fe652d73
  - [bpf-next,v4,4/4] bpf, riscv: use prog pack allocator in the BPF JIT
    https://git.kernel.org/riscv/c/19ea9d201008

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
maintainer
#7 lore raw
Quoted 34 lines · On Tue, 05 Sep 2023 19:50:27 PDT (-0700), [email protected]…

Hello:

This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Quoted 16 lines · On Thu, 31 Aug 2023 13:12:25 +0000 you wrote:

Changes in v3 -> v4:
1. Add Acked-by:, Tested-by:, etc.
2. Add the core BPF patch[3] which was earlier sent with ARM64 series to this series so it can go with this.

Changes in v2 -> v3:
1. Fix maximum width of code in patches from 80 to 100. [All patches]
2. Add checks for ctx->ro_insns == NULL. [Patch 3]
3. Fix check for edge condition where amount of text to set > 2 * pagesize [Patch 1 and 2]
4. Add reviewed-by in patches.
5. Adding results of selftest here:
Using the command: ./test_progs on qemu
Without the series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED With this series: Summary: 336/3162 PASSED, 56 SKIPPED, 90 FAILED

Quoted 1 line

[...]

Here is the summary with links:
- [bpf-next,v4,1/4] bpf: make bpf_prog_pack allocator portable

    https://git.kernel.org/riscv/c/063119d90a06
  - [bpf-next,v4,2/4] riscv: extend patch_text_nosync() for multiple pages
    https://git.kernel.org/riscv/c/fb81d562ed1f
  - [bpf-next,v4,3/4] riscv: implement a memset like function for text
    https://git.kernel.org/riscv/c/f071fe652d73
  - [bpf-next,v4,4/4] bpf, riscv: use prog pack allocator in the BPF JIT
    https://git.kernel.org/riscv/c/19ea9d201008
Quoted 1 line

You are awesome, thank you!

Looks like I screwed up the merge a bit here, so these hashes have changed a bit.