Skip to contentSkip to conversation

eBPF arm64 JIT support

PATCH Applied [email protected] 8 participants 32 messages Sep 3, 2019 – Apr 6, 2020 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover eBPF arm64 JIT support

From: Jerin Jacob <[email protected]>

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

dpdk.org/examples/bpf/t1.c application shows around 50% improvement on OCTEON TX2 platform in JIT vs interpreter mode.

Verified the implementation using existing bpf_autotest application.

# echo "bpf_autotest" | sudo ./build/app/test -c 0x3

RTE>>bpf_autotest
run_test(test_store1) start run_test(test_store2) start run_test(test_load1) start run_test(test_ldimm1) start run_test(test_mul1) start run_test(test_shift1) start run_test(test_jump1) start run_test(test_alu1) start run_test(test_bele1) start run_test(test_xadd1) start run_test(test_div1) start bpf_exec(0xffffa37c0000): division by 0 at pc: 0x68; run_test(test_call1) start run_test(test_call2) start run_test(test_call3) start
Test OK

Jerin Jacob (8):
bpf/arm64: add build infrastructure bpf/arm64: add prologue and epilogue bpf/arm64: add basic arithmetic operations bpf/arm64: add logical operations bpf/arm64: add byte swap operations bpf/arm64: add load and store operations bpf/arm64: add atomic-exchange-and-add operation bpf/arm64: add branch operation

 MAINTAINERS                            |    1 +
 doc/guides/prog_guide/bpf_lib.rst      |    2 +-
 doc/guides/rel_notes/release_19_11.rst |    5 +
 lib/librte_bpf/Makefile                |    2 +
 lib/librte_bpf/bpf.c                   |    4 +-
 lib/librte_bpf/bpf_impl.h              |    3 +-
 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
 lib/librte_bpf/meson.build             |    2 +
 8 files changed, 1466 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_bpf/bpf_jit_arm64.c
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/8 bpf/arm64: add build infrastructure

From: Jerin Jacob <[email protected]>

Add build infrastructure and documentation
update for arm64 JIT support.

---
 MAINTAINERS                            |  1 +
 doc/guides/prog_guide/bpf_lib.rst      |  2 +-
 doc/guides/rel_notes/release_19_11.rst |  5 +++++
 lib/librte_bpf/Makefile                |  2 ++
 lib/librte_bpf/bpf.c                   |  4 +++-
 lib/librte_bpf/bpf_impl.h              |  3 +--
 lib/librte_bpf/bpf_jit_arm64.c         | 19 +++++++++++++++++++
 lib/librte_bpf/meson.build             |  2 ++
 8 files changed, 34 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_bpf/bpf_jit_arm64.c
MAINTAINERS +1 -0
index 410026086..c2e91343c 100644
@@ -252,6 +252,7 @@ M: Gavin Hu <[email protected]>
 F: lib/librte_eal/common/include/arch/arm/*_64.h
 F: lib/librte_net/net_crc_neon.h
 F: lib/librte_acl/acl_run_neon.*
+F: lib/librte_bpf/bpf_jit_arm64.c
 F: lib/librte_lpm/rte_lpm_neon.h
 F: lib/librte_hash/rte*_arm64.h
 F: lib/librte_efd/rte*_arm64.h
doc/guides/prog_guide/bpf_lib.rst +1 -1
index 7c08e6b2d..9c728da7b 100644
@@ -30,7 +30,7 @@ The library API provides the following basic operations:
 Not currently supported eBPF features
 -------------------------------------
 
- - JIT for non X86_64 platforms
+ - JIT support only available for X86_64 and arm64 platforms
  - cBPF
  - tail-pointer call
  - eBPF MAP
doc/guides/rel_notes/release_19_11.rst +5 -0
index 8490d897c..a0a92b8ae 100644
@@ -56,6 +56,11 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added eBPF JIT support for arm64.**
+
+  Added eBPF JIT support for arm64 architecture to improve the eBPF program
+  performance.
+
 
 Removed Items
 -------------
lib/librte_bpf/Makefile +2 -0
index c0e8aaa68..419a5162e 100644
@@ -31,6 +31,8 @@ SRCS-$(CONFIG_RTE_LIBRTE_BPF) += bpf_load_elf.c
 endif
 ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
 SRCS-$(CONFIG_RTE_LIBRTE_BPF) += bpf_jit_x86.c
+else ifeq ($(CONFIG_RTE_ARCH_ARM64),y)
+SRCS-$(CONFIG_RTE_LIBRTE_BPF) += bpf_jit_arm64.c
 endif
 
 # install header files
lib/librte_bpf/bpf.c +3 -1
index cc963d52e..7e1879ffa 100644
@@ -41,8 +41,10 @@ bpf_jit(struct rte_bpf *bpf)
 {
 	int32_t rc;
 
-#ifdef RTE_ARCH_X86_64
+#if defined(RTE_ARCH_X86_64)
 	rc = bpf_jit_x86(bpf);
+#elif defined(RTE_ARCH_ARM64)
+	rc = bpf_jit_arm64(bpf);
 #else
 	rc = -ENOTSUP;
 #endif
lib/librte_bpf/bpf_impl.h +1 -2
index b577e2cbe..03ba0ae11 100644
@@ -25,9 +25,8 @@ extern int bpf_validate(struct rte_bpf *bpf);
 
 extern int bpf_jit(struct rte_bpf *bpf);
 
-#ifdef RTE_ARCH_X86_64
 extern int bpf_jit_x86(struct rte_bpf *);
-#endif
+extern int bpf_jit_arm64(struct rte_bpf *);
 
 extern int rte_bpf_logtype;
 
lib/librte_bpf/bpf_jit_arm64.c new file +19 -0
new file mode 100644
index 000000000..621bb7f46
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2019 Marvell International Ltd.
+ */
+
+#include <errno.h>
+
+#include <rte_common.h>
+
+#include "bpf_impl.h"
+/*
+ * Produce a native ISA version of the given BPF code.
+ */
+int
+bpf_jit_arm64(struct rte_bpf *bpf)
+{
+	RTE_SET_USED(bpf);
+
+	return -ENOTSUP;
+}
lib/librte_bpf/meson.build +2 -0
index 11c1fb558..13fc02db3 100644
@@ -10,6 +10,8 @@ sources = files('bpf.c',
 
 if arch_subdir == 'x86' and dpdk_conf.get('RTE_ARCH_64')
 	sources += files('bpf_jit_x86.c')
+elif dpdk_conf.has('RTE_ARCH_ARM64')
+	sources += files('bpf_jit_arm64.c')
 endif
 
 install_headers = files('bpf_def.h',
Signature · 1 line
2.23.0
#2 lore raw

PATCH 2/8 bpf/arm64: add prologue and epilogue

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Add prologue and epilogue as per arm64 procedure call standard.

As an optimization the generated instructions are the function of whether eBPF program has stack and/or CALL class.

---
 lib/librte_bpf/bpf_jit_arm64.c | 466 ++++++++++++++++++++++++++++++++-
 1 file changed, 464 insertions(+), 2 deletions(-)
lib/librte_bpf/bpf_jit_arm64.c +464 -2
index 621bb7f46..548408a61 100644
@@ -3,17 +3,479 @@
  */
 
 #include <errno.h>
+#include <stdbool.h>
 
 #include <rte_common.h>
 
 #include "bpf_impl.h"
+
+#define A64_REG_MASK(r)		((r) & 0x1f)
+#define A64_INVALID_OP_CODE	(0xffffffff)
+
+#define TMP_REG_1		(EBPF_REG_10 + 1)
+#define TMP_REG_2		(EBPF_REG_10 + 2)
+#define TMP_REG_3		(EBPF_REG_10 + 3)
+
+#define EBPF_FP			(EBPF_REG_10)
+#define EBPF_OP_GET(op)		(BPF_OP(op) >> 4)
+
+#define A64_R(x)		x
+#define A64_FP			29
+#define A64_LR			30
+#define A64_SP			31
+#define A64_ZR			31
+
+#define check_imm(n, val) (((val) >= 0) ? !!((val) >> (n)) : !!((~val) >> (n)))
+#define mask_imm(n, val) ((val) & ((1 << (n)) - 1))
+
+struct ebpf_a64_map {
+	uint32_t off; /* eBPF to arm64 insn offset mapping for jump */
+	uint8_t off_to_b; /* Offset to branch instruction delta */
+};
+
+struct a64_jit_ctx {
+	size_t stack_sz;          /* Stack size */
+	uint32_t *ins;            /* ARM64 instructions. NULL if first pass */
+	struct ebpf_a64_map *map; /* eBPF to arm64 insn mapping for jump */
+	uint32_t idx;             /* Current instruction index */
+	uint32_t program_start;   /* Program index, Just after prologue */
+	uint32_t program_sz;      /* Program size. Found in first pass */
+	uint8_t foundcall;        /* Found EBPF_CALL class code in eBPF pgm */
+};
+
+static int
+check_reg(uint8_t r)
+{
+	return (r > 31) ? 1 : 0;
+}
+
+static int
+is_first_pass(struct a64_jit_ctx *ctx)
+{
+	return (ctx->ins == NULL);
+}
+
+static int
+check_invalid_args(struct a64_jit_ctx *ctx, uint32_t limit)
+{
+	uint32_t idx;
+
+	if (is_first_pass(ctx))
+		return 0;
+
+	for (idx = 0; idx < limit; idx++) {
+		if (rte_le_to_cpu_32(ctx->ins[idx]) == A64_INVALID_OP_CODE) {
+			RTE_BPF_LOG(ERR,
+				"%s: invalid opcode at %u;\n", __func__, idx);
+			return -EINVAL;
+		}
+	}
+	return 0;
+}
+
+/* Emit an instruction */
+static inline void
+emit_insn(struct a64_jit_ctx *ctx, uint32_t insn, int error)
+{
+	if (error)
+		insn = A64_INVALID_OP_CODE;
+
+	if (ctx->ins)
+		ctx->ins[ctx->idx] = rte_cpu_to_le_32(insn);
+
+	ctx->idx++;
+}
+
+static void
+emit_ret(struct a64_jit_ctx *ctx)
+{
+	emit_insn(ctx, 0xd65f03c0, 0);
+}
+
+static void
+emit_add_sub_imm(struct a64_jit_ctx *ctx, bool is64, bool sub, uint8_t rd,
+		 uint8_t rn, int16_t imm12)
+{
+	uint32_t insn, imm;
+
+	imm = mask_imm(12, imm12);
+	insn = (!!is64) << 31;
+	insn |= (!!sub) << 30;
+	insn |= 0x11000000;
+	insn |= rd;
+	insn |= rn << 5;
+	insn |= imm << 10;
+
+	emit_insn(ctx, insn,
+		  check_reg(rd) || check_reg(rn) || check_imm(12, imm12));
+}
+
+static void
+emit_add_imm_64(struct a64_jit_ctx *ctx, uint8_t rd, uint8_t rn, uint16_t imm12)
+{
+	emit_add_sub_imm(ctx, 1, 0, rd, rn, imm12);
+}
+
+static void
+emit_sub_imm_64(struct a64_jit_ctx *ctx, uint8_t rd, uint8_t rn, uint16_t imm12)
+{
+	emit_add_sub_imm(ctx, 1, 1, rd, rn, imm12);
+}
+
+static void
+emit_mov(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rn)
+{
+	emit_add_sub_imm(ctx, is64, 0, rd, rn, 0);
+}
+
+static void
+emit_mov_64(struct a64_jit_ctx *ctx, uint8_t rd, uint8_t rn)
+{
+	emit_mov(ctx, 1, rd, rn);
+}
+
+static void
+emit_ls_pair_64(struct a64_jit_ctx *ctx, uint8_t rt, uint8_t rt2, uint8_t rn,
+		bool push, bool load, bool pre_index)
+{
+	uint32_t insn;
+
+	insn = (!!load) << 22;
+	insn |= (!!pre_index) << 24;
+	insn |= 0xa8800000;
+	insn |= rt;
+	insn |= rn << 5;
+	insn |= rt2 << 10;
+	if (push)
+		insn |= 0x7e << 15; /* 0x7e means -2 with imm7 */
+	else
+		insn |= 0x2 << 15;
+
+	emit_insn(ctx, insn, check_reg(rn) || check_reg(rt) || check_reg(rt2));
+
+}
+
+/* Emit stp rt, rt2, [sp, #-16]! */
+static void
+emit_stack_push(struct a64_jit_ctx *ctx, uint8_t rt, uint8_t rt2)
+{
+	emit_ls_pair_64(ctx, rt, rt2, A64_SP, 1, 0, 1);
+}
+
+/* Emit ldp rt, rt2, [sp, #16] */
+static void
+emit_stack_pop(struct a64_jit_ctx *ctx, uint8_t rt, uint8_t rt2)
+{
+	emit_ls_pair_64(ctx, rt, rt2, A64_SP, 0, 1, 0);
+}
+
+static uint8_t
+ebpf_to_a64_reg(struct a64_jit_ctx *ctx, uint8_t reg)
+{
+	const uint32_t ebpf2a64_has_call[] = {
+		/* Map A64 R7 register as EBPF return register */
+		[EBPF_REG_0] = A64_R(7),
+		/* Map A64 arguments register as EBPF arguments register */
+		[EBPF_REG_1] = A64_R(0),
+		[EBPF_REG_2] = A64_R(1),
+		[EBPF_REG_3] = A64_R(2),
+		[EBPF_REG_4] = A64_R(3),
+		[EBPF_REG_5] = A64_R(4),
+		/* Map A64 callee save register as EBPF callee save register */
+		[EBPF_REG_6] = A64_R(19),
+		[EBPF_REG_7] = A64_R(20),
+		[EBPF_REG_8] = A64_R(21),
+		[EBPF_REG_9] = A64_R(22),
+		[EBPF_FP]    = A64_R(25),
+		/* Map A64 scratch registers as temporary storage */
+		[TMP_REG_1] = A64_R(9),
+		[TMP_REG_2] = A64_R(10),
+		[TMP_REG_3] = A64_R(11),
+	};
+
+	const uint32_t ebpf2a64_no_call[] = {
+		/* Map A64 R7 register as EBPF return register */
+		[EBPF_REG_0] = A64_R(7),
+		/* Map A64 arguments register as EBPF arguments register */
+		[EBPF_REG_1] = A64_R(0),
+		[EBPF_REG_2] = A64_R(1),
+		[EBPF_REG_3] = A64_R(2),
+		[EBPF_REG_4] = A64_R(3),
+		[EBPF_REG_5] = A64_R(4),
+		/*
+		 * EBPF program does not have EBPF_CALL op code,
+		 * Map A64 scratch registers as EBPF callee save registers.
+		 */
+		[EBPF_REG_6] = A64_R(9),
+		[EBPF_REG_7] = A64_R(10),
+		[EBPF_REG_8] = A64_R(11),
+		[EBPF_REG_9] = A64_R(12),
+		/* Map A64 FP register as EBPF FP register */
+		[EBPF_FP]    = A64_FP,
+		/* Map remaining A64 scratch registers as temporary storage */
+		[TMP_REG_1] = A64_R(13),
+		[TMP_REG_2] = A64_R(14),
+		[TMP_REG_3] = A64_R(15),
+	};
+
+	if (ctx->foundcall)
+		return ebpf2a64_has_call[reg];
+	else
+		return ebpf2a64_no_call[reg];
+}
+
+/*
+ * Procedure call standard for the arm64
+ * -------------------------------------
+ * R0..R7  - Parameter/result registers
+ * R8      - Indirect result location register
+ * R9..R15 - Scratch registers
+ * R15     - Platform Register
+ * R16     - First intra-procedure-call scratch register
+ * R17     - Second intra-procedure-call temporary register
+ * R19-R28 - Callee saved registers
+ * R29     - Frame pointer
+ * R30     - Link register
+ * R31     - Stack pointer
+ */
+static void
+emit_prologue_has_call(struct a64_jit_ctx *ctx)
+{
+	uint8_t r6, r7, r8, r9, fp;
+
+	r6 = ebpf_to_a64_reg(ctx, EBPF_REG_6);
+	r7 = ebpf_to_a64_reg(ctx, EBPF_REG_7);
+	r8 = ebpf_to_a64_reg(ctx, EBPF_REG_8);
+	r9 = ebpf_to_a64_reg(ctx, EBPF_REG_9);
+	fp = ebpf_to_a64_reg(ctx, EBPF_FP);
+
+	/*
+	 * eBPF prog stack layout
+	 *
+	 *                               high
+	 *       eBPF prologue       0:+-----+ <= original A64_SP
+	 *                             |FP/LR|
+	 *                         -16:+-----+ <= current A64_FP
+	 *    Callee saved registers   | ... |
+	 *             EBPF_FP =>  -64:+-----+
+	 *                             |     |
+	 *       eBPF prog stack       | ... |
+	 *                             |     |
+	 * (EBPF_FP - bpf->stack_sz)=> +-----+
+	 * Pad for A64_SP 16B alignment| PAD |
+	 * (EBPF_FP - ctx->stack_sz)=> +-----+ <= current A64_SP
+	 *                             |     |
+	 *                             | ... | Function call stack
+	 *                             |     |
+	 *                             +-----+
+	 *                              low
+	 */
+	emit_stack_push(ctx, A64_FP, A64_LR);
+	emit_mov_64(ctx, A64_FP, A64_SP);
+	emit_stack_push(ctx, r6, r7);
+	emit_stack_push(ctx, r8, r9);
+	/*
+	 * There is no requirement to save A64_R(28) in stack. Doing it here,
+	 * because, A64_SP needs be to 16B aligned and STR vs STP
+	 * takes same number of cycles(typically).
+	 */
+	emit_stack_push(ctx, fp, A64_R(28));
+	emit_mov_64(ctx, fp, A64_SP);
+	if (ctx->stack_sz)
+		emit_sub_imm_64(ctx, A64_SP, A64_SP, ctx->stack_sz);
+}
+
+static void
+emit_epilogue_has_call(struct a64_jit_ctx *ctx)
+{
+	uint8_t r6, r7, r8, r9, fp, r0;
+
+	r6 = ebpf_to_a64_reg(ctx, EBPF_REG_6);
+	r7 = ebpf_to_a64_reg(ctx, EBPF_REG_7);
+	r8 = ebpf_to_a64_reg(ctx, EBPF_REG_8);
+	r9 = ebpf_to_a64_reg(ctx, EBPF_REG_9);
+	fp = ebpf_to_a64_reg(ctx, EBPF_FP);
+	r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0);
+
+	if (ctx->stack_sz)
+		emit_add_imm_64(ctx, A64_SP, A64_SP, ctx->stack_sz);
+	emit_stack_pop(ctx, fp, A64_R(28));
+	emit_stack_pop(ctx, r8, r9);
+	emit_stack_pop(ctx, r6, r7);
+	emit_stack_pop(ctx, A64_FP, A64_LR);
+	emit_mov_64(ctx, A64_R(0), r0);
+	emit_ret(ctx);
+}
+
+static void
+emit_prologue_no_call(struct a64_jit_ctx *ctx)
+{
+	/*
+	 * eBPF prog stack layout without EBPF_CALL opcode
+	 *
+	 *                               high
+	 *    eBPF prologue(EBPF_FP) 0:+-----+ <= original A64_SP/current A64_FP
+	 *                             |     |
+	 *                             | ... |
+	 *            eBPF prog stack  |     |
+	 *                             |     |
+	 * (EBPF_FP - bpf->stack_sz)=> +-----+
+	 * Pad for A64_SP 16B alignment| PAD |
+	 * (EBPF_FP - ctx->stack_sz)=> +-----+ <= current A64_SP
+	 *                             |     |
+	 *                             | ... | Function call stack
+	 *                             |     |
+	 *                             +-----+
+	 *                              low
+	 */
+	if (ctx->stack_sz) {
+		emit_mov_64(ctx, A64_FP, A64_SP);
+		emit_sub_imm_64(ctx, A64_SP, A64_SP, ctx->stack_sz);
+	}
+}
+
+static void
+emit_epilogue_no_call(struct a64_jit_ctx *ctx)
+{
+	if (ctx->stack_sz)
+		emit_add_imm_64(ctx, A64_SP, A64_SP, ctx->stack_sz);
+	emit_mov_64(ctx, A64_R(0), ebpf_to_a64_reg(ctx, EBPF_REG_0));
+	emit_ret(ctx);
+}
+
+static void
+emit_prologue(struct a64_jit_ctx *ctx)
+{
+	if (ctx->foundcall)
+		emit_prologue_has_call(ctx);
+	else
+		emit_prologue_no_call(ctx);
+
+	ctx->program_start = ctx->idx;
+}
+
+static void
+emit_epilogue(struct a64_jit_ctx *ctx)
+{
+	ctx->program_sz = ctx->idx - ctx->program_start;
+
+	if (ctx->foundcall)
+		emit_epilogue_has_call(ctx);
+	else
+		emit_epilogue_no_call(ctx);
+}
+
+static void
+check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
+{
+	const struct ebpf_insn *ins;
+	uint8_t op;
+	uint32_t i;
+
+	for (i = 0; i != bpf->prm.nb_ins; i++) {
+		ins = bpf->prm.ins + i;
+		op = ins->code;
+
+		switch (op) {
+		/* Call imm */
+		case (BPF_JMP | EBPF_CALL):
+			ctx->foundcall = 1;
+			return;
+		}
+	}
+}
+
+/*
+ * Walk through eBPF code and translate them to arm64 one.
+ */
+static int
+emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
+{
+	uint8_t op;
+	const struct ebpf_insn *ins;
+	uint32_t i;
+	int rc;
+
+	/* Reset context fields */
+	ctx->idx = 0;
+	/* arm64 SP must be aligned to 16 */
+	ctx->stack_sz = RTE_ALIGN_MUL_CEIL(bpf->stack_sz, 16);
+
+	emit_prologue(ctx);
+
+	for (i = 0; i != bpf->prm.nb_ins; i++) {
+
+		ins = bpf->prm.ins + i;
+		op = ins->code;
+
+		switch (op) {
+		/* Return r0 */
+		case (BPF_JMP | EBPF_EXIT):
+			emit_epilogue(ctx);
+			break;
+		default:
+			RTE_BPF_LOG(ERR,
+				"%s(%p): invalid opcode %#x at pc: %u;\n",
+				__func__, bpf, ins->code, i);
+			return -EINVAL;
+		}
+	}
+	rc = check_invalid_args(ctx, ctx->idx);
+
+	return rc;
+}
+
 /*
  * Produce a native ISA version of the given BPF code.
  */
 int
 bpf_jit_arm64(struct rte_bpf *bpf)
 {
-	RTE_SET_USED(bpf);
+	struct a64_jit_ctx ctx;
+	size_t size;
+	int rc;
+
+	/* Init JIT context */
+	memset(&ctx, 0, sizeof(ctx));
+
+	/* Find eBPF program has call class or not */
+	check_program_has_call(&ctx, bpf);
+
+	/* First pass to calculate total code size and valid jump offsets */
+	rc = emit(&ctx, bpf);
+	if (rc)
+		goto finish;
+
+	size = ctx.idx * sizeof(uint32_t);
+	/* Allocate JIT program memory */
+	ctx.ins = mmap(NULL, size, PROT_READ | PROT_WRITE,
+			       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (ctx.ins == MAP_FAILED) {
+		rc = -ENOMEM;
+		goto finish;
+	}
+
+	/* Second pass to generate code */
+	rc = emit(&ctx, bpf);
+	if (rc)
+		goto munmap;
+
+	rc = mprotect(ctx.ins, size, PROT_READ | PROT_EXEC) != 0;
+	if (rc) {
+		rc = -errno;
+		goto munmap;
+	}
+
+	/* Flush the icache */
+	__builtin___clear_cache(ctx.ins, ctx.ins + ctx.idx);
+
+	bpf->jit.func = (void *)ctx.ins;
+	bpf->jit.sz = size;
+
+	goto finish;
 
-	return -ENOTSUP;
+munmap:
+	munmap(ctx.ins, size);
+finish:
+	return rc;
 }
Signature · 1 line
2.23.0
#3 lore raw

PATCH 3/8 bpf/arm64: add basic arithmetic operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Add mov, add, sub, mul, div and mod arithmetic operations for immediate and source register variants.

---
 lib/librte_bpf/bpf_jit_arm64.c | 300 ++++++++++++++++++++++++++++++++-
 1 file changed, 299 insertions(+), 1 deletion(-)
lib/librte_bpf/bpf_jit_arm64.c +299 -1
index 548408a61..5d2ce378c 100644
@@ -43,6 +43,17 @@ struct a64_jit_ctx {
 	uint8_t foundcall;        /* Found EBPF_CALL class code in eBPF pgm */
 };
 
+static int
+check_mov_hw(bool is64, const uint8_t val)
+{
+	if (val == 16 || val == 0)
+		return 0;
+	else if (is64 && val != 64 && val != 48 && val != 32)
+		return 1;
+
+	return 0;
+}
+
 static int
 check_reg(uint8_t r)
 {
@@ -169,6 +180,179 @@ emit_stack_pop(struct a64_jit_ctx *ctx, uint8_t rt, uint8_t rt2)
 	emit_ls_pair_64(ctx, rt, rt2, A64_SP, 0, 1, 0);
 }
 
+#define A64_MOVN 0
+#define A64_MOVZ 2
+#define A64_MOVK 3
+static void
+mov_imm(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t type,
+	uint16_t imm16, uint8_t shift)
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= type << 29;
+	insn |= 0x25 << 23;
+	insn |= (shift/16) << 21;
+	insn |= imm16 << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_mov_hw(is64, shift));
+}
+
+static void
+emit_mov_imm32(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint32_t val)
+{
+	uint16_t upper = val >> 16;
+	uint16_t lower = val & 0xffff;
+
+	/* Positive number */
+	if ((val & 1UL << 31) == 0) {
+		mov_imm(ctx, is64, rd, A64_MOVZ, lower, 0);
+		if (upper)
+			mov_imm(ctx, is64, rd, A64_MOVK, upper, 16);
+	} else { /* Negative number */
+		if (upper == 0xffff) {
+			mov_imm(ctx, is64, rd, A64_MOVN, ~lower, 0);
+		} else {
+			mov_imm(ctx, is64, rd, A64_MOVN, ~upper, 16);
+			if (lower != 0xffff)
+				mov_imm(ctx, is64, rd, A64_MOVK, lower, 0);
+		}
+	}
+}
+
+static int
+u16_blocks_weight(const uint64_t val, bool one)
+{
+	return (((val >>  0) & 0xffff) == (one ? 0xffff : 0x0000)) +
+	       (((val >> 16) & 0xffff) == (one ? 0xffff : 0x0000)) +
+	       (((val >> 32) & 0xffff) == (one ? 0xffff : 0x0000)) +
+	       (((val >> 48) & 0xffff) == (one ? 0xffff : 0x0000));
+}
+
+static void
+emit_mov_imm(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint64_t val)
+{
+	uint64_t nval = ~val;
+	int movn, sr;
+
+	if (is64 == 0)
+		return emit_mov_imm32(ctx, 0, rd, (uint32_t)(val & 0xffffffff));
+
+	/* Find MOVN or MOVZ first */
+	movn = u16_blocks_weight(val, true) > u16_blocks_weight(val, false);
+	/* Find shift right value */
+	sr = movn ? rte_fls_u64(nval) - 1 : rte_fls_u64(val) - 1;
+	sr = RTE_ALIGN_FLOOR(sr, 16);
+	sr = RTE_MAX(sr, 0);
+
+	if (movn)
+		mov_imm(ctx, 1, rd, A64_MOVN, (nval >> sr) & 0xffff, sr);
+	else
+		mov_imm(ctx, 1, rd, A64_MOVZ, (val >> sr) & 0xffff, sr);
+
+	sr -= 16;
+	while (sr >= 0) {
+		if (((val >> sr) & 0xffff) != (movn ? 0xffff : 0x0000))
+			mov_imm(ctx, 1, rd, A64_MOVK, (val >> sr) & 0xffff, sr);
+		sr -= 16;
+	}
+}
+
+#define A64_ADD 0x58
+#define A64_SUB 0x258
+static void
+emit_add_sub(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rn,
+	     uint8_t rm, uint16_t op)
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= op << 21; /* shift == 0 */
+	insn |= rm << 16;
+	insn |= rn << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rm));
+}
+
+static void
+emit_add(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_add_sub(ctx, is64, rd, rd, rm, A64_ADD);
+}
+
+static void
+emit_sub(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_add_sub(ctx, is64, rd, rd, rm, A64_SUB);
+}
+
+static void
+emit_mul(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= 0xd8 << 21;
+	insn |= rm << 16;
+	insn |= A64_ZR << 10;
+	insn |= rd << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rm));
+}
+
+#define A64_UDIV 0x2
+static void
+emit_data_process_two_src(struct a64_jit_ctx *ctx, bool is64, uint8_t rd,
+			  uint8_t rn, uint8_t rm, uint16_t op)
+
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= 0xd6 << 21;
+	insn |= rm << 16;
+	insn |= op << 10;
+	insn |= rn << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rm));
+}
+
+static void
+emit_div(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_data_process_two_src(ctx, is64, rd, rd, rm, A64_UDIV);
+}
+
+static void
+emit_msub(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rn,
+	  uint8_t rm, uint8_t ra)
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= 0xd8 << 21;
+	insn |= rm << 16;
+	insn |= 0x1 << 15;
+	insn |= ra << 10;
+	insn |= rn << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rn) || check_reg(rm) ||
+		  check_reg(ra));
+}
+
+static void
+emit_mod(struct a64_jit_ctx *ctx, bool is64, uint8_t tmp, uint8_t rd,
+	 uint8_t rm)
+{
+	emit_data_process_two_src(ctx, is64, tmp, rd, rm, A64_UDIV);
+	emit_msub(ctx, is64, rd, tmp, rm, rd);
+}
+
 static uint8_t
 ebpf_to_a64_reg(struct a64_jit_ctx *ctx, uint8_t reg)
 {
@@ -365,6 +549,44 @@ emit_epilogue(struct a64_jit_ctx *ctx)
 		emit_epilogue_no_call(ctx);
 }
 
+static void
+emit_cbnz(struct a64_jit_ctx *ctx, bool is64, uint8_t rt, int32_t imm19)
+{
+	uint32_t insn, imm;
+
+	imm = mask_imm(19, imm19);
+	insn = (!!is64) << 31;
+	insn |= 0x35 << 24;
+	insn |= imm << 5;
+	insn |= rt;
+
+	emit_insn(ctx, insn, check_reg(rt) || check_imm(19, imm19));
+}
+
+static void
+emit_b(struct a64_jit_ctx *ctx, int32_t imm26)
+{
+	uint32_t insn, imm;
+
+	imm = mask_imm(26, imm26);
+	insn = 0x5 << 26;
+	insn |= imm;
+
+	emit_insn(ctx, insn, check_imm(26, imm26));
+}
+
+static void
+emit_return_zero_if_src_zero(struct a64_jit_ctx *ctx, bool is64, uint8_t src)
+{
+	uint8_t r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0);
+	uint16_t jump_to_epilogue;
+
+	emit_cbnz(ctx, is64, src, 3);
+	emit_mov_imm(ctx, is64, r0, 0);
+	jump_to_epilogue = (ctx->program_start + ctx->program_sz) - ctx->idx;
+	emit_b(ctx, jump_to_epilogue);
+}
+
 static void
 check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
@@ -391,15 +613,19 @@ check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 static int
 emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
-	uint8_t op;
+	uint8_t op, dst, src, tmp1, tmp2;
 	const struct ebpf_insn *ins;
+	int32_t imm;
 	uint32_t i;
+	bool is64;
 	int rc;
 
 	/* Reset context fields */
 	ctx->idx = 0;
 	/* arm64 SP must be aligned to 16 */
 	ctx->stack_sz = RTE_ALIGN_MUL_CEIL(bpf->stack_sz, 16);
+	tmp1 = ebpf_to_a64_reg(ctx, TMP_REG_1);
+	tmp2 = ebpf_to_a64_reg(ctx, TMP_REG_2);
 
 	emit_prologue(ctx);
 
@@ -407,8 +633,80 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 
 		ins = bpf->prm.ins + i;
 		op = ins->code;
+		imm = ins->imm;
+
+		dst = ebpf_to_a64_reg(ctx, ins->dst_reg);
+		src = ebpf_to_a64_reg(ctx, ins->src_reg);
+		is64 = (BPF_CLASS(op) == EBPF_ALU64);
 
 		switch (op) {
+		/* dst = src */
+		case (BPF_ALU | EBPF_MOV | BPF_X):
+		case (EBPF_ALU64 | EBPF_MOV | BPF_X):
+			emit_mov(ctx, is64, dst, src);
+			break;
+		/* dst = imm */
+		case (BPF_ALU | EBPF_MOV | BPF_K):
+		case (EBPF_ALU64 | EBPF_MOV | BPF_K):
+			emit_mov_imm(ctx, is64, dst, imm);
+			break;
+		/* dst += src */
+		case (BPF_ALU | BPF_ADD | BPF_X):
+		case (EBPF_ALU64 | BPF_ADD | BPF_X):
+			emit_add(ctx, is64, dst, src);
+			break;
+		/* dst += imm */
+		case (BPF_ALU | BPF_ADD | BPF_K):
+		case (EBPF_ALU64 | BPF_ADD | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_add(ctx, is64, dst, tmp1);
+			break;
+		/* dst -= src */
+		case (BPF_ALU | BPF_SUB | BPF_X):
+		case (EBPF_ALU64 | BPF_SUB | BPF_X):
+			emit_sub(ctx, is64, dst, src);
+			break;
+		/* dst -= imm */
+		case (BPF_ALU | BPF_SUB | BPF_K):
+		case (EBPF_ALU64 | BPF_SUB | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_sub(ctx, is64, dst, tmp1);
+			break;
+		/* dst *= src */
+		case (BPF_ALU | BPF_MUL | BPF_X):
+		case (EBPF_ALU64 | BPF_MUL | BPF_X):
+			emit_mul(ctx, is64, dst, src);
+			break;
+		/* dst *= imm */
+		case (BPF_ALU | BPF_MUL | BPF_K):
+		case (EBPF_ALU64 | BPF_MUL | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_mul(ctx, is64, dst, tmp1);
+			break;
+		/* dst /= src */
+		case (BPF_ALU | BPF_DIV | BPF_X):
+		case (EBPF_ALU64 | BPF_DIV | BPF_X):
+			emit_return_zero_if_src_zero(ctx, is64, src);
+			emit_div(ctx, is64, dst, src);
+			break;
+		/* dst /= imm */
+		case (BPF_ALU | BPF_DIV | BPF_K):
+		case (EBPF_ALU64 | BPF_DIV | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_div(ctx, is64, dst, tmp1);
+			break;
+		/* dst %= src */
+		case (BPF_ALU | BPF_MOD | BPF_X):
+		case (EBPF_ALU64 | BPF_MOD | BPF_X):
+			emit_return_zero_if_src_zero(ctx, is64, src);
+			emit_mod(ctx, is64, tmp1, dst, src);
+			break;
+		/* dst %= imm */
+		case (BPF_ALU | BPF_MOD | BPF_K):
+		case (EBPF_ALU64 | BPF_MOD | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_mod(ctx, is64, tmp2, dst, tmp1);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
Signature · 1 line
2.23.0
#4 lore raw

PATCH 4/8 bpf/arm64: add logical operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Add OR, AND, NEG, XOR, shift operations for immediate and source register variants.

---
 lib/librte_bpf/bpf_jit_arm64.c | 189 +++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
lib/librte_bpf/bpf_jit_arm64.c +189 -0
index 5d2ce378c..fcde3583b 100644
@@ -43,6 +43,17 @@ struct a64_jit_ctx {
 	uint8_t foundcall;        /* Found EBPF_CALL class code in eBPF pgm */
 };
 
+static int
+check_immr_imms(bool is64, uint8_t immr, uint8_t imms)
+{
+	const unsigned int width = is64 ? 64 : 32;
+
+	if (immr >= width || imms >= width)
+		return 1;
+
+	return 0;
+}
+
 static int
 check_mov_hw(bool is64, const uint8_t val)
 {
@@ -288,6 +299,12 @@ emit_sub(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
 	emit_add_sub(ctx, is64, rd, rd, rm, A64_SUB);
 }
 
+static void
+emit_neg(struct a64_jit_ctx *ctx, bool is64, uint8_t rd)
+{
+	emit_add_sub(ctx, is64, rd, A64_ZR, rd, A64_SUB);
+}
+
 static void
 emit_mul(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
 {
@@ -304,6 +321,9 @@ emit_mul(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
 }
 
 #define A64_UDIV 0x2
+#define A64_LSLV 0x8
+#define A64_LSRV 0x9
+#define A64_ASRV 0xA
 static void
 emit_data_process_two_src(struct a64_jit_ctx *ctx, bool is64, uint8_t rd,
 			  uint8_t rn, uint8_t rm, uint16_t op)
@@ -327,6 +347,107 @@ emit_div(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
 	emit_data_process_two_src(ctx, is64, rd, rd, rm, A64_UDIV);
 }
 
+static void
+emit_lslv(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_data_process_two_src(ctx, is64, rd, rd, rm, A64_LSLV);
+}
+
+static void
+emit_lsrv(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_data_process_two_src(ctx, is64, rd, rd, rm, A64_LSRV);
+}
+
+static void
+emit_asrv(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_data_process_two_src(ctx, is64, rd, rd, rm, A64_ASRV);
+}
+
+#define A64_UBFM 0x2
+#define A64_SBFM 0x0
+static void
+emit_bitfield(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rn,
+	      uint8_t immr, uint8_t imms, uint16_t op)
+
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	if (insn)
+		insn |= 1 << 22; /* Set N bit when is64 is set */
+	insn |= op << 29;
+	insn |= 0x26 << 23;
+	insn |= immr << 16;
+	insn |= imms << 10;
+	insn |= rn << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rn) ||
+		  check_immr_imms(is64, immr, imms));
+}
+static void
+emit_lsl(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t imm)
+{
+	const unsigned int width = is64 ? 64 : 32;
+	uint8_t imms, immr;
+
+	immr = (width - imm) & (width - 1);
+	imms = width - 1 - imm;
+
+	emit_bitfield(ctx, is64, rd, rd, immr, imms, A64_UBFM);
+}
+
+static void
+emit_lsr(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t imm)
+{
+	emit_bitfield(ctx, is64, rd, rd, imm, is64 ? 63 : 31, A64_UBFM);
+}
+
+static void
+emit_asr(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t imm)
+{
+	emit_bitfield(ctx, is64, rd, rd, imm, is64 ? 63 : 31, A64_SBFM);
+}
+
+#define A64_AND 0
+#define A64_OR 1
+#define A64_XOR 2
+static void
+emit_logical(struct a64_jit_ctx *ctx, bool is64, uint8_t rd,
+	     uint8_t rm, uint16_t op)
+{
+	uint32_t insn;
+
+	insn = (!!is64) << 31;
+	insn |= op << 29;
+	insn |= 0x50 << 21;
+	insn |= rm << 16;
+	insn |= rd << 5;
+	insn |= rd;
+
+	emit_insn(ctx, insn, check_reg(rd) || check_reg(rm));
+}
+
+static void
+emit_or(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_logical(ctx, is64, rd, rm, A64_OR);
+}
+
+static void
+emit_and(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_logical(ctx, is64, rd, rm, A64_AND);
+}
+
+static void
+emit_xor(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rm)
+{
+	emit_logical(ctx, is64, rd, rm, A64_XOR);
+}
+
 static void
 emit_msub(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint8_t rn,
 	  uint8_t rm, uint8_t ra)
@@ -707,6 +828,74 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 			emit_mov_imm(ctx, is64, tmp1, imm);
 			emit_mod(ctx, is64, tmp2, dst, tmp1);
 			break;
+		/* dst |= src */
+		case (BPF_ALU | BPF_OR | BPF_X):
+		case (EBPF_ALU64 | BPF_OR | BPF_X):
+			emit_or(ctx, is64, dst, src);
+			break;
+		/* dst |= imm */
+		case (BPF_ALU | BPF_OR | BPF_K):
+		case (EBPF_ALU64 | BPF_OR | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_or(ctx, is64, dst, tmp1);
+			break;
+		/* dst &= src */
+		case (BPF_ALU | BPF_AND | BPF_X):
+		case (EBPF_ALU64 | BPF_AND | BPF_X):
+			emit_and(ctx, is64, dst, src);
+			break;
+		/* dst &= imm */
+		case (BPF_ALU | BPF_AND | BPF_K):
+		case (EBPF_ALU64 | BPF_AND | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_and(ctx, is64, dst, tmp1);
+			break;
+		/* dst ^= src */
+		case (BPF_ALU | BPF_XOR | BPF_X):
+		case (EBPF_ALU64 | BPF_XOR | BPF_X):
+			emit_xor(ctx, is64, dst, src);
+			break;
+		/* dst ^= imm */
+		case (BPF_ALU | BPF_XOR | BPF_K):
+		case (EBPF_ALU64 | BPF_XOR | BPF_K):
+			emit_mov_imm(ctx, is64, tmp1, imm);
+			emit_xor(ctx, is64, dst, tmp1);
+			break;
+		/* dst = -dst */
+		case (BPF_ALU | BPF_NEG):
+		case (EBPF_ALU64 | BPF_NEG):
+			emit_neg(ctx, is64, dst);
+			break;
+		/* dst <<= src */
+		case BPF_ALU | BPF_LSH | BPF_X:
+		case EBPF_ALU64 | BPF_LSH | BPF_X:
+			emit_lslv(ctx, is64, dst, src);
+			break;
+		/* dst <<= imm */
+		case BPF_ALU | BPF_LSH | BPF_K:
+		case EBPF_ALU64 | BPF_LSH | BPF_K:
+			emit_lsl(ctx, is64, dst, imm);
+			break;
+		/* dst >>= src */
+		case BPF_ALU | BPF_RSH | BPF_X:
+		case EBPF_ALU64 | BPF_RSH | BPF_X:
+			emit_lsrv(ctx, is64, dst, src);
+			break;
+		/* dst >>= imm */
+		case BPF_ALU | BPF_RSH | BPF_K:
+		case EBPF_ALU64 | BPF_RSH | BPF_K:
+			emit_lsr(ctx, is64, dst, imm);
+			break;
+		/* dst >>= src (arithmetic) */
+		case BPF_ALU | EBPF_ARSH | BPF_X:
+		case EBPF_ALU64 | EBPF_ARSH | BPF_X:
+			emit_asrv(ctx, is64, dst, src);
+			break;
+		/* dst >>= imm (arithmetic) */
+		case BPF_ALU | EBPF_ARSH | BPF_K:
+		case EBPF_ALU64 | EBPF_ARSH | BPF_K:
+			emit_asr(ctx, is64, dst, imm);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
Signature · 1 line
2.23.0
#5 lore raw

PATCH 5/8 bpf/arm64: add byte swap operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

add le16, le32, le64, be16, be32 and be64 operations.

---
 lib/librte_bpf/bpf_jit_arm64.c | 87 ++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
lib/librte_bpf/bpf_jit_arm64.c +87 -0
index fcde3583b..ec165d627 100644
@@ -6,6 +6,7 @@
 #include <stdbool.h>
 
 #include <rte_common.h>
+#include <rte_byteorder.h>
 
 #include "bpf_impl.h"
 
@@ -474,6 +475,84 @@ emit_mod(struct a64_jit_ctx *ctx, bool is64, uint8_t tmp, uint8_t rd,
 	emit_msub(ctx, is64, rd, tmp, rm, rd);
 }
 
+static void
+emit_zero_extend(struct a64_jit_ctx *ctx, uint8_t rd, int32_t imm)
+{
+	switch (imm) {
+	case 16:
+		/* Zero-extend 16 bits into 64 bits */
+		emit_bitfield(ctx, 1, rd, rd, 0, 15, A64_UBFM);
+		break;
+	case 32:
+		/* Zero-extend 32 bits into 64 bits */
+		emit_bitfield(ctx, 1, rd, rd, 0, 31, A64_UBFM);
+		break;
+	case 64:
+		break;
+	default:
+		/* Generate error */
+		emit_insn(ctx, 0, 1);
+	}
+}
+
+static void
+emit_rev(struct a64_jit_ctx *ctx, uint8_t rd, int32_t imm)
+{
+	uint32_t insn;
+
+	insn = 0xdac00000;
+	insn |= rd << 5;
+	insn |= rd;
+
+	switch (imm) {
+	case 16:
+		insn |= 1 << 10;
+		emit_insn(ctx, insn, check_reg(rd));
+		emit_zero_extend(ctx, rd, 16);
+		break;
+	case 32:
+		insn |= 2 << 10;
+		emit_insn(ctx, insn, check_reg(rd));
+		/* Upper 32 bits already cleared */
+		break;
+	case 64:
+		insn |= 3 << 10;
+		emit_insn(ctx, insn, check_reg(rd));
+		break;
+	default:
+		/* Generate error */
+		emit_insn(ctx, insn, 1);
+	}
+}
+
+static int
+is_be(void)
+{
+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
+	return 1;
+#else
+	return 0;
+#endif
+}
+
+static void
+emit_be(struct a64_jit_ctx *ctx, uint8_t rd, int32_t imm)
+{
+	if (is_be())
+		emit_zero_extend(ctx, rd, imm);
+	else
+		emit_rev(ctx, rd, imm);
+}
+
+static void
+emit_le(struct a64_jit_ctx *ctx, uint8_t rd, int32_t imm)
+{
+	if (is_be())
+		emit_rev(ctx, rd, imm);
+	else
+		emit_zero_extend(ctx, rd, imm);
+}
+
 static uint8_t
 ebpf_to_a64_reg(struct a64_jit_ctx *ctx, uint8_t reg)
 {
@@ -896,6 +975,14 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 		case EBPF_ALU64 | EBPF_ARSH | BPF_K:
 			emit_asr(ctx, is64, dst, imm);
 			break;
+		/* dst = be##imm(dst) */
+		case (BPF_ALU | EBPF_END | EBPF_TO_BE):
+			emit_be(ctx, dst, imm);
+			break;
+		/* dst = le##imm(dst) */
+		case (BPF_ALU | EBPF_END | EBPF_TO_LE):
+			emit_le(ctx, dst, imm);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
Signature · 1 line
2.23.0
#6 lore raw

PATCH 6/8 bpf/arm64: add load and store operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Add load and store operations.

---
 lib/librte_bpf/bpf_jit_arm64.c | 84 ++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)
lib/librte_bpf/bpf_jit_arm64.c +84 -0
index ec165d627..c797c9c62 100644
@@ -66,6 +66,15 @@ check_mov_hw(bool is64, const uint8_t val)
 	return 0;
 }
 
+static int
+check_ls_sz(uint8_t sz)
+{
+	if (sz == BPF_B || sz == BPF_H || sz == BPF_W || sz == EBPF_DW)
+		return 0;
+
+	return 1;
+}
+
 static int
 check_reg(uint8_t r)
 {
@@ -271,6 +280,47 @@ emit_mov_imm(struct a64_jit_ctx *ctx, bool is64, uint8_t rd, uint64_t val)
 	}
 }
 
+static void
+emit_ls(struct a64_jit_ctx *ctx, uint8_t sz, uint8_t rt, uint8_t rn, uint8_t rm,
+	bool load)
+{
+	uint32_t insn;
+
+	insn = 0x1c1 << 21;
+	if (load)
+		insn |= 1 << 22;
+	if (sz == BPF_B)
+		insn |= 0 << 30;
+	else if (sz == BPF_H)
+		insn |= 1 << 30;
+	else if (sz == BPF_W)
+		insn |= 2 << 30;
+	else if (sz == EBPF_DW)
+		insn |= 3 << 30;
+
+	insn |= rm << 16;
+	insn |= 0x1a << 10; /* LSL and S = 0 */
+	insn |= rn << 5;
+	insn |= rt;
+
+	emit_insn(ctx, insn, check_reg(rt) || check_reg(rn) || check_reg(rm) ||
+		  check_ls_sz(sz));
+}
+
+static void
+emit_str(struct a64_jit_ctx *ctx, uint8_t sz, uint8_t rt, uint8_t rn,
+	 uint8_t rm)
+{
+	emit_ls(ctx, sz, rt, rn, rm, 0);
+}
+
+static void
+emit_ldr(struct a64_jit_ctx *ctx, uint8_t sz, uint8_t rt, uint8_t rn,
+	 uint8_t rm)
+{
+	emit_ls(ctx, sz, rt, rn, rm, 1);
+}
+
 #define A64_ADD 0x58
 #define A64_SUB 0x258
 static void
@@ -815,6 +865,8 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
 	uint8_t op, dst, src, tmp1, tmp2;
 	const struct ebpf_insn *ins;
+	uint64_t u64;
+	int16_t off;
 	int32_t imm;
 	uint32_t i;
 	bool is64;
@@ -833,6 +885,7 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 
 		ins = bpf->prm.ins + i;
 		op = ins->code;
+		off = ins->off;
 		imm = ins->imm;
 
 		dst = ebpf_to_a64_reg(ctx, ins->dst_reg);
@@ -983,6 +1036,37 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 		case (BPF_ALU | EBPF_END | EBPF_TO_LE):
 			emit_le(ctx, dst, imm);
 			break;
+		/* dst = *(size *) (src + off) */
+		case (BPF_LDX | BPF_MEM | BPF_B):
+		case (BPF_LDX | BPF_MEM | BPF_H):
+		case (BPF_LDX | BPF_MEM | BPF_W):
+		case (BPF_LDX | BPF_MEM | EBPF_DW):
+			emit_mov_imm(ctx, 1, tmp1, off);
+			emit_ldr(ctx, BPF_SIZE(op), dst, src, tmp1);
+			break;
+		/* dst = imm64 */
+		case (BPF_LD | BPF_IMM | EBPF_DW):
+			u64 = ((uint64_t)ins[1].imm << 32) | (uint32_t)imm;
+			emit_mov_imm(ctx, 1, dst, u64);
+			i++;
+			break;
+		/* *(size *)(dst + off) = src */
+		case (BPF_STX | BPF_MEM | BPF_B):
+		case (BPF_STX | BPF_MEM | BPF_H):
+		case (BPF_STX | BPF_MEM | BPF_W):
+		case (BPF_STX | BPF_MEM | EBPF_DW):
+			emit_mov_imm(ctx, 1, tmp1, off);
+			emit_str(ctx, BPF_SIZE(op), src, dst, tmp1);
+			break;
+		/* *(size *)(dst + off) = imm */
+		case (BPF_ST | BPF_MEM | BPF_B):
+		case (BPF_ST | BPF_MEM | BPF_H):
+		case (BPF_ST | BPF_MEM | BPF_W):
+		case (BPF_ST | BPF_MEM | EBPF_DW):
+			emit_mov_imm(ctx, 1, tmp1, imm);
+			emit_mov_imm(ctx, 1, tmp2, off);
+			emit_str(ctx, BPF_SIZE(op), tmp1, dst, tmp2);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
Signature · 1 line
2.23.0
#7 lore raw

PATCH 7/8 bpf/arm64: add atomic-exchange-and-add operation

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Implement XADD eBPF instruction using STADD arm64 instruction. If the given platform does not have atomics support, use LDXR and STXR pair for critical section instead of STADD.

---
 lib/librte_bpf/bpf_jit_arm64.c | 85 +++++++++++++++++++++++++++++++++-
 1 file changed, 84 insertions(+), 1 deletion(-)
lib/librte_bpf/bpf_jit_arm64.c +84 -1
index c797c9c62..62fa6a505 100644
@@ -837,6 +837,83 @@ emit_return_zero_if_src_zero(struct a64_jit_ctx *ctx, bool is64, uint8_t src)
 	emit_b(ctx, jump_to_epilogue);
 }
 
+static void
+emit_stadd(struct a64_jit_ctx *ctx, bool is64, uint8_t rs, uint8_t rn)
+{
+	uint32_t insn;
+
+	insn = 0xb820001f;
+	insn |= (!!is64) << 30;
+	insn |= rs << 16;
+	insn |= rn << 5;
+
+	emit_insn(ctx, insn, check_reg(rs) || check_reg(rn));
+}
+
+static void
+emit_ldxr(struct a64_jit_ctx *ctx, bool is64, uint8_t rt, uint8_t rn)
+{
+	uint32_t insn;
+
+	insn = 0x885f7c00;
+	insn |= (!!is64) << 30;
+	insn |= rn << 5;
+	insn |= rt;
+
+	emit_insn(ctx, insn, check_reg(rt) || check_reg(rn));
+}
+
+static void
+emit_stxr(struct a64_jit_ctx *ctx, bool is64, uint8_t rs, uint8_t rt,
+	  uint8_t rn)
+{
+	uint32_t insn;
+
+	insn = 0x88007c00;
+	insn |= (!!is64) << 30;
+	insn |= rs << 16;
+	insn |= rn << 5;
+	insn |= rt;
+
+	emit_insn(ctx, insn, check_reg(rs) || check_reg(rt) || check_reg(rn));
+}
+
+static int
+has_atomics(void)
+{
+	int rc = 0;
+
+#if defined(__ARM_FEATURE_ATOMICS) || defined(RTE_ARM_FEATURE_ATOMICS)
+	rc = 1;
+#endif
+	return rc;
+}
+
+static void
+emit_xadd(struct a64_jit_ctx *ctx, uint8_t op, uint8_t tmp1, uint8_t tmp2,
+	  uint8_t tmp3, uint8_t dst, int16_t off, uint8_t src)
+{
+	bool is64 = (BPF_SIZE(op) == EBPF_DW);
+	uint8_t rn;
+
+	if (off) {
+		emit_mov_imm(ctx, 1, tmp1, off);
+		emit_add(ctx, 1, tmp1, dst);
+		rn = tmp1;
+	} else {
+		rn = dst;
+	}
+
+	if (has_atomics()) {
+		emit_stadd(ctx, is64, src, rn);
+	} else {
+		emit_ldxr(ctx, is64, tmp2, rn);
+		emit_add(ctx, is64, tmp2, src);
+		emit_stxr(ctx, is64, tmp3, tmp2, rn);
+		emit_cbnz(ctx, is64, tmp3, -3);
+	}
+}
+
 static void
 check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
@@ -863,7 +940,7 @@ check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 static int
 emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
-	uint8_t op, dst, src, tmp1, tmp2;
+	uint8_t op, dst, src, tmp1, tmp2, tmp3;
 	const struct ebpf_insn *ins;
 	uint64_t u64;
 	int16_t off;
@@ -878,6 +955,7 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 	ctx->stack_sz = RTE_ALIGN_MUL_CEIL(bpf->stack_sz, 16);
 	tmp1 = ebpf_to_a64_reg(ctx, TMP_REG_1);
 	tmp2 = ebpf_to_a64_reg(ctx, TMP_REG_2);
+	tmp3 = ebpf_to_a64_reg(ctx, TMP_REG_3);
 
 	emit_prologue(ctx);
 
@@ -1067,6 +1145,11 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 			emit_mov_imm(ctx, 1, tmp2, off);
 			emit_str(ctx, BPF_SIZE(op), tmp1, dst, tmp2);
 			break;
+		/* STX XADD: lock *(size *)(dst + off) += src */
+		case (BPF_STX | EBPF_XADD | BPF_W):
+		case (BPF_STX | EBPF_XADD | EBPF_DW):
+			emit_xadd(ctx, op, tmp1, tmp2, tmp3, dst, off, src);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
Signature · 1 line
2.23.0
#8 lore raw

Re: [dpdk-dev] [PATCH 7/8] bpf/arm64: add atomic-exchange-and-add operation

Quoted 6 lines · On Tue, Sep 3, 2019 at 1:00 PM <[email protected]> wrote:

From: Jerin Jacob <[email protected]>

Implement XADD eBPF instruction using STADD arm64 instruction. If the given platform does not have atomics support, use LDXR and STXR pair for critical section instead of STADD.

For the record, this patch had a missed dependency on the 128-bits atomic for arm64 patch because of RTE_ARM_FEATURE_ATOMICS. This will be resolved once the latter is merged.

Quoted 8 lines
---
 lib/librte_bpf/bpf_jit_arm64.c | 85 +++++++++++++++++++++++++++++++++-
 1 file changed, 84 insertions(+), 1 deletion(-)
lib/librte_bpf/bpf_jit_arm64.c +0 -0
index c797c9c62..62fa6a505 100644

[snip]

Quoted 10 lines
lib/librte_bpf/bpf_jit_arm64.c +10 -0
+static int
+has_atomics(void)
+{
+       int rc = 0;
+
+#if defined(__ARM_FEATURE_ATOMICS) || defined(RTE_ARM_FEATURE_ATOMICS)
+       rc = 1;
+#endif
+       return rc;
+}

[snip]

Signature · 1 line
David Marchand
#9 lore raw

PATCH 8/8 bpf/arm64: add branch operation

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Add branch and call operations.

jump_offset_* APIs used for finding the relative offset to jump w.r.t current eBPF program PC.

---
 lib/librte_bpf/bpf_jit_arm64.c | 229 +++++++++++++++++++++++++++++++++
 1 file changed, 229 insertions(+)
lib/librte_bpf/bpf_jit_arm64.c +229 -0
index 62fa6a505..8882fee67 100644
@@ -105,6 +105,112 @@ check_invalid_args(struct a64_jit_ctx *ctx, uint32_t limit)
 	return 0;
 }
 
+static int
+jump_offset_init(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
+{
+	uint32_t i;
+
+	ctx->map = malloc(bpf->prm.nb_ins * sizeof(ctx->map[0]));
+	if (ctx->map == NULL)
+		return -ENOMEM;
+
+	/* Fill with fake offsets */
+	for (i = 0; i != bpf->prm.nb_ins; i++) {
+		ctx->map[i].off = INT32_MAX;
+		ctx->map[i].off_to_b = 0;
+	}
+	return 0;
+}
+
+static void
+jump_offset_fini(struct a64_jit_ctx *ctx)
+{
+	free(ctx->map);
+}
+
+static void
+jump_offset_update(struct a64_jit_ctx *ctx, uint32_t ebpf_idx)
+{
+	if (is_first_pass(ctx))
+		ctx->map[ebpf_idx].off = ctx->idx;
+}
+
+static void
+jump_offset_to_branch_update(struct a64_jit_ctx *ctx, uint32_t ebpf_idx)
+{
+	if (is_first_pass(ctx))
+		ctx->map[ebpf_idx].off_to_b = ctx->idx - ctx->map[ebpf_idx].off;
+
+}
+
+static int32_t
+jump_offset_get(struct a64_jit_ctx *ctx, uint32_t from, int16_t offset)
+{
+	int32_t a64_from, a64_to;
+
+	a64_from = ctx->map[from].off +  ctx->map[from].off_to_b;
+	a64_to = ctx->map[from + offset + 1].off;
+
+	if (a64_to == INT32_MAX)
+		return a64_to;
+
+	return a64_to - a64_from;
+}
+
+enum a64_cond_e {
+	A64_EQ = 0x0, /* == */
+	A64_NE = 0x1, /* != */
+	A64_CS = 0x2, /* Unsigned >= */
+	A64_CC = 0x3, /* Unsigned < */
+	A64_MI = 0x4, /* < 0 */
+	A64_PL = 0x5, /* >= 0 */
+	A64_VS = 0x6, /* Overflow */
+	A64_VC = 0x7, /* No overflow */
+	A64_HI = 0x8, /* Unsigned > */
+	A64_LS = 0x9, /* Unsigned <= */
+	A64_GE = 0xa, /* Signed >= */
+	A64_LT = 0xb, /* Signed < */
+	A64_GT = 0xc, /* Signed > */
+	A64_LE = 0xd, /* Signed <= */
+	A64_AL = 0xe, /* Always */
+};
+
+static int
+check_cond(uint8_t cond)
+{
+	return (cond >= A64_AL) ? 1 : 0;
+}
+
+static uint8_t
+ebpf_to_a64_cond(uint8_t op)
+{
+	switch (BPF_OP(op)) {
+	case BPF_JEQ:
+		return A64_EQ;
+	case BPF_JGT:
+		return A64_HI;
+	case EBPF_JLT:
+		return A64_CC;
+	case BPF_JGE:
+		return A64_CS;
+	case EBPF_JLE:
+		return A64_LS;
+	case BPF_JSET:
+	case EBPF_JNE:
+		return A64_NE;
+	case EBPF_JSGT:
+		return A64_GT;
+	case EBPF_JSLT:
+		return A64_LT;
+	case EBPF_JSGE:
+		return A64_GE;
+	case EBPF_JSLE:
+		return A64_LE;
+	default:
+		return UINT8_MAX;
+	}
+}
+
 /* Emit an instruction */
 static inline void
 emit_insn(struct a64_jit_ctx *ctx, uint32_t insn, int error)
@@ -525,6 +631,17 @@ emit_mod(struct a64_jit_ctx *ctx, bool is64, uint8_t tmp, uint8_t rd,
 	emit_msub(ctx, is64, rd, tmp, rm, rd);
 }
 
+static void
+emit_blr(struct a64_jit_ctx *ctx, uint8_t rn)
+{
+	uint32_t insn;
+
+	insn = 0xd63f0000;
+	insn |= rn << 5;
+
+	emit_insn(ctx, insn, check_reg(rn));
+}
+
 static void
 emit_zero_extend(struct a64_jit_ctx *ctx, uint8_t rd, int32_t imm)
 {
@@ -799,6 +916,16 @@ emit_epilogue(struct a64_jit_ctx *ctx)
 		emit_epilogue_no_call(ctx);
 }
 
+static void
+emit_call(struct a64_jit_ctx *ctx, uint8_t tmp, void *func)
+{
+	uint8_t r0 = ebpf_to_a64_reg(ctx, EBPF_REG_0);
+
+	emit_mov_imm(ctx, 1, tmp, (uint64_t)func);
+	emit_blr(ctx, tmp);
+	emit_mov_64(ctx, r0, A64_R(0));
+}
+
 static void
 emit_cbnz(struct a64_jit_ctx *ctx, bool is64, uint8_t rt, int32_t imm19)
 {
@@ -914,6 +1041,54 @@ emit_xadd(struct a64_jit_ctx *ctx, uint8_t op, uint8_t tmp1, uint8_t tmp2,
 	}
 }
 
+#define A64_CMP 0x6b00000f
+#define A64_TST 0x6a00000f
+static void
+emit_cmp_tst(struct a64_jit_ctx *ctx, bool is64, uint8_t rn, uint8_t rm,
+	     uint32_t opc)
+{
+	uint32_t insn;
+
+	insn = opc;
+	insn |= (!!is64) << 31;
+	insn |= rm << 16;
+	insn |= rn << 5;
+
+	emit_insn(ctx, insn, check_reg(rn) || check_reg(rm));
+}
+
+static void
+emit_cmp(struct a64_jit_ctx *ctx, bool is64, uint8_t rn, uint8_t rm)
+{
+	emit_cmp_tst(ctx, is64, rn, rm, A64_CMP);
+}
+
+static void
+emit_tst(struct a64_jit_ctx *ctx, bool is64, uint8_t rn, uint8_t rm)
+{
+	emit_cmp_tst(ctx, is64, rn, rm, A64_TST);
+}
+
+static void
+emit_b_cond(struct a64_jit_ctx *ctx, uint8_t cond, int32_t imm19)
+{
+	uint32_t insn, imm;
+
+	imm = mask_imm(19, imm19);
+	insn = 0x15 << 26;
+	insn |= imm << 5;
+	insn |= cond;
+
+	emit_insn(ctx, insn, check_cond(cond) || check_imm(19, imm19));
+}
+
+static void
+emit_branch(struct a64_jit_ctx *ctx, uint8_t op, uint32_t i, int16_t off)
+{
+	jump_offset_to_branch_update(ctx, i);
+	emit_b_cond(ctx, ebpf_to_a64_cond(op), jump_offset_get(ctx, i, off));
+}
+
 static void
 check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 {
@@ -961,6 +1136,7 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 
 	for (i = 0; i != bpf->prm.nb_ins; i++) {
 
+		jump_offset_update(ctx, i);
 		ins = bpf->prm.ins + i;
 		op = ins->code;
 		off = ins->off;
@@ -1150,6 +1326,52 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf)
 		case (BPF_STX | EBPF_XADD | EBPF_DW):
 			emit_xadd(ctx, op, tmp1, tmp2, tmp3, dst, off, src);
 			break;
+		/* PC += off */
+		case (BPF_JMP | BPF_JA):
+			emit_b(ctx, jump_offset_get(ctx, i, off));
+			break;
+		/* PC += off if dst COND imm */
+		case (BPF_JMP | BPF_JEQ | BPF_K):
+		case (BPF_JMP | EBPF_JNE | BPF_K):
+		case (BPF_JMP | BPF_JGT | BPF_K):
+		case (BPF_JMP | EBPF_JLT | BPF_K):
+		case (BPF_JMP | BPF_JGE | BPF_K):
+		case (BPF_JMP | EBPF_JLE | BPF_K):
+		case (BPF_JMP | EBPF_JSGT | BPF_K):
+		case (BPF_JMP | EBPF_JSLT | BPF_K):
+		case (BPF_JMP | EBPF_JSGE | BPF_K):
+		case (BPF_JMP | EBPF_JSLE | BPF_K):
+			emit_mov_imm(ctx, 1, tmp1, imm);
+			emit_cmp(ctx, 1, dst, tmp1);
+			emit_branch(ctx, op, i, off);
+			break;
+		case (BPF_JMP | BPF_JSET | BPF_K):
+			emit_mov_imm(ctx, 1, tmp1, imm);
+			emit_tst(ctx, 1, dst, tmp1);
+			emit_branch(ctx, op, i, off);
+			break;
+		/* PC += off if dst COND src */
+		case (BPF_JMP | BPF_JEQ | BPF_X):
+		case (BPF_JMP | EBPF_JNE | BPF_X):
+		case (BPF_JMP | BPF_JGT | BPF_X):
+		case (BPF_JMP | EBPF_JLT | BPF_X):
+		case (BPF_JMP | BPF_JGE | BPF_X):
+		case (BPF_JMP | EBPF_JLE | BPF_X):
+		case (BPF_JMP | EBPF_JSGT | BPF_X):
+		case (BPF_JMP | EBPF_JSLT | BPF_X):
+		case (BPF_JMP | EBPF_JSGE | BPF_X):
+		case (BPF_JMP | EBPF_JSLE | BPF_X):
+			emit_cmp(ctx, 1, dst, src);
+			emit_branch(ctx, op, i, off);
+			break;
+		case (BPF_JMP | BPF_JSET | BPF_X):
+			emit_tst(ctx, 1, dst, src);
+			emit_branch(ctx, op, i, off);
+			break;
+		/* Call imm */
+		case (BPF_JMP | EBPF_CALL):
+			emit_call(ctx, tmp1, bpf->prm.xsym[ins->imm].func.val);
+			break;
 		/* Return r0 */
 		case (BPF_JMP | EBPF_EXIT):
 			emit_epilogue(ctx);
@@ -1179,6 +1401,11 @@ bpf_jit_arm64(struct rte_bpf *bpf)
 	/* Init JIT context */
 	memset(&ctx, 0, sizeof(ctx));
 
+	/* Initialize the memory for eBPF to a64 insn offset map for jump */
+	rc = jump_offset_init(&ctx, bpf);
+	if (rc)
+		goto error;
+
 	/* Find eBPF program has call class or not */
 	check_program_has_call(&ctx, bpf);
 
@@ -1218,5 +1445,7 @@ bpf_jit_arm64(struct rte_bpf *bpf)
 munmap:
 	munmap(ctx.ins, size);
 finish:
+	jump_offset_fini(&ctx);
+error:
 	return rc;
 }
Signature · 1 line
2.23.0
#10 lore raw
In reply to [email protected]
Quoted 52 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

dpdk.org/examples/bpf/t1.c application shows around 50% improvement on OCTEON TX2 platform in JIT vs interpreter mode.

Verified the implementation using existing bpf_autotest application.

# echo "bpf_autotest" | sudo ./build/app/test -c 0x3

RTE>>bpf_autotest
run_test(test_store1) start run_test(test_store2) start run_test(test_load1) start run_test(test_ldimm1) start run_test(test_mul1) start run_test(test_shift1) start run_test(test_jump1) start run_test(test_alu1) start run_test(test_bele1) start run_test(test_xadd1) start run_test(test_div1) start bpf_exec(0xffffa37c0000): division by 0 at pc: 0x68; run_test(test_call1) start run_test(test_call2) start run_test(test_call3) start
Test OK

Jerin Jacob (8):
bpf/arm64: add build infrastructure bpf/arm64: add prologue and epilogue bpf/arm64: add basic arithmetic operations bpf/arm64: add logical operations bpf/arm64: add byte swap operations bpf/arm64: add load and store operations bpf/arm64: add atomic-exchange-and-add operation bpf/arm64: add branch operation

 MAINTAINERS                            |    1 +
 doc/guides/prog_guide/bpf_lib.rst      |    2 +-
 doc/guides/rel_notes/release_19_11.rst |    5 +
 lib/librte_bpf/Makefile                |    2 +
 lib/librte_bpf/bpf.c                   |    4 +-
 lib/librte_bpf/bpf_impl.h              |    3 +-
 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
 lib/librte_bpf/meson.build             |    2 +
 8 files changed, 1466 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_bpf/bpf_jit_arm64.c
Quoted 1 line

--

Series Acked-by: Konstantin Ananyev <[email protected]>

Quoted 1 line

2.23.0

#11 lore raw

24/09/2019 19:03, Ananyev, Konstantin:

Quoted 11 lines
Quoted 10 lines

Jerin Jacob (8):
bpf/arm64: add build infrastructure bpf/arm64: add prologue and epilogue bpf/arm64: add basic arithmetic operations bpf/arm64: add logical operations bpf/arm64: add byte swap operations bpf/arm64: add load and store operations bpf/arm64: add atomic-exchange-and-add operation bpf/arm64: add branch operation

Quoted 1 line

Series Acked-by: Konstantin Ananyev <[email protected]>

Applied

Please continue the discussions started with the Linux maintainers in order to try reducing the code duplication between the two projects. Thanks

#12 lore raw
In reply to [email protected]

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

#13 lore raw
Quoted 10 lines · Thomas Monjalon

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 3 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code
from Linux as is.

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

Quoted 2 lines
#14 lore raw

Hi everyone,

Quoted 30 lines · Thomas Monjalon
Quoted 10 lines

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 3 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code
from Linux as is.

Quoted 8 lines

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

My thought would be that if we have JIT eBPF compiler already in DPDK
for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm). About having a common code-base with Linux eBPF JITs implementation -
I think it is a very good idea,
but I don’t' think it could be achieved without significant effort.
DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
meanwhile start talking with linux guys how common JIT code-base could be achieved. Konstantin

#15 lore raw

Adding Arm JIT and Kernel experts

Quoted 67 lines

-----Original Message-----
From: Ananyev, Konstantin <[email protected]>
Sent: Thursday, October 3, 2019 10:06 AM
To: Jerin Jacob <[email protected]>; [email protected]
Cc: [email protected]; dpdk-dev <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: RE: [dpdk-dev] [PATCH 0/8] eBPF arm64 JIT support

Hi everyone,

Quoted 2 lines

On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <[email protected]>

Quoted 14 lines · wrote:
Quoted 10 lines

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 1 line
Quoted 9 lines
Quoted 2 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e
4a

Quoted 5 lines

2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code from Linux as is.

Quoted 8 lines

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

My thought would be that if we have JIT eBPF compiler already in DPDK for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
About having a common code-base with Linux eBPF JITs implementation - I think it is a very good idea, but I don’t' think it could be achieved without significant effort.
DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11, meanwhile start talking with linux guys how common JIT code-base could be achieved.
Konstantin

#16 lore raw
Quoted 1 line · Honnappa Nagarahalli

Adding Arm JIT and Kernel experts

Hi Honnappa,
I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64
M: Daniel Borkmann <[email protected]>
M: Alexei Starovoitov <[email protected]>
M: Zi Shen Lim <[email protected]>
L: [email protected]
L: [email protected]
S: Supported
F: arch/arm64/net/

As they will have much better knowledge of the state of play and will be better able to advise.

Cheers,

Signature · 1 line
Steve
#17 lore raw

04/10/2019 11:54, Steve Capper:

Quoted 10 lines

I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64
M: Daniel Borkmann <[email protected]>
M: Alexei Starovoitov <[email protected]>
M: Zi Shen Lim <[email protected]>
L: [email protected]
L: [email protected]
S: Supported
F: arch/arm64/net/

Quoted 2 lines

As they will have much better knowledge of the state of play and will be better able to advise.

As far as I know Alexei and Daniel are OK with the idea. But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT for userspace usage (not only DPDK) as a library.
I don't understand why the DPDK JIT should be different or optimized differently.
The only real issue I see is the need for a dual licensing BSD-GPL.

#18 lore raw
Quoted 14 lines · Thomas Monjalon

04/10/2019 11:54, Steve Capper:

Quoted 10 lines

I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64
M: Daniel Borkmann <[email protected]>
M: Alexei Starovoitov <[email protected]>
M: Zi Shen Lim <[email protected]>
L: [email protected]
L: [email protected]
S: Supported
F: arch/arm64/net/

Quoted 2 lines

As they will have much better knowledge of the state of play and will be better able to advise.

Quoted 7 lines

As far as I know Alexei and Daniel are OK with the idea. But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT for userspace usage (not only DPDK) as a library. I don't understand why the DPDK JIT should be different or optimized differently.

That would be great indeed as both projects would benefit from a shared JIT instead of reimplementing everything twice. I never looked into DPDK too much, but I presume the idea would be as well to take the LLVM (or bpf-gcc) generated object file and load it into a BPF 'engine' that sits in user space on top of DPDK? Presumably loader could be libbpf here as well since it already knows how to parse the ELF, perform the relocations etc. The only difference would be that you have a different context and different helpers? Is that the goal eventually?

Quoted 1 line

The only real issue I see is the need for a dual licensing BSD-GPL.

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

# bpftool prog
2: cgroup_skb  tag 7be49e3934a125ba  gpl
        loaded_at 2019-10-03T12:53:11+0200  uid 0
        xlated 296B  jited 229B  memlock 4096B  map_ids 2,3
[...]
# bpftool prog dump xlated id 2
   0: (bf) r6 = r1
   1: (69) r7 = *(u16 *)(r6 +192)
   2: (b4) w8 = 0
   3: (55) if r7 != 0x8 goto pc+14
   4: (bf) r1 = r6
   5: (b4) w2 = 16
   6: (bf) r3 = r10
   7: (07) r3 += -4
   8: (b4) w4 = 4
   9: (85) call bpf_skb_load_bytes#7484768
  10: (18) r1 = map[id:2]
  12: (bf) r2 = r10
  13: (07) r2 += -8
  14: (62) *(u32 *)(r2 +0) = 32
  15: (85) call trie_lookup_elem#90800
  16: (15) if r0 == 0x0 goto pc+1
  17: (44) w8 |= 2
  18: (55) if r7 != 0xdd86 goto pc+14
  19: (bf) r1 = r6
  20: (b4) w2 = 24
  21: (bf) r3 = r10
  22: (07) r3 += -16
  23: (b4) w4 = 16
  24: (85) call bpf_skb_load_bytes#7484768
  25: (18) r1 = map[id:3]
  27: (bf) r2 = r10
  28: (07) r2 += -20
  29: (62) *(u32 *)(r2 +0) = 128
  30: (85) call trie_lookup_elem#90800
  31: (15) if r0 == 0x0 goto pc+1
  32: (44) w8 |= 2
  33: (b7) r0 = 1
  34: (55) if r8 != 0x2 goto pc+1
  35: (b7) r0 = 0
  36: (95) exit
# bpftool prog dump jited id 2 opcodes
   0:   push   %rbp
        55
   1:   mov    %rsp,%rbp
        48 89 e5
   4:   sub    $0x40,%rsp
        48 81 ec 40 00 00 00
   b:   sub    $0x28,%rbp
        48 83 ed 28
   f:   mov    %rbx,0x0(%rbp)
        48 89 5d 00
  13:   mov    %r13,0x8(%rbp)
        4c 89 6d 08
  17:   mov    %r14,0x10(%rbp)
        4c 89 75 10
  1b:   mov    %r15,0x18(%rbp)
        4c 89 7d 18
  1f:   xor    %eax,%eax
        31 c0
  21:   mov    %rax,0x20(%rbp)
        48 89 45 20
  25:   mov    %rdi,%rbx
        48 89 fb
  28:   movzwq 0xc0(%rbx),%r13
        4c 0f b7 ab c0 00 00 00
  30:   xor    %r14d,%r14d
        45 31 f6
  33:   cmp    $0x8,%r13
        49 83 fd 08
  37:   jne    0x0000000000000079
        75 40
  39:   mov    %rbx,%rdi
        48 89 df
  3c:   mov    $0x10,%esi
        be 10 00 00 00
[...]
  cb:   jne    0x00000000000000cf
        75 02
  cd:   xor    %eax,%eax
        31 c0
  cf:   mov    0x0(%rbp),%rbx
        48 8b 5d 00
  d3:   mov    0x8(%rbp),%r13
        4c 8b 6d 08
  d7:   mov    0x10(%rbp),%r14
        4c 8b 75 10
  db:   mov    0x18(%rbp),%r15
        4c 8b 7d 18
  df:   add    $0x28,%rbp
        48 83 c5 28
  e3:   leaveq
        c9
  e4:   retq
        c3

Thanks,
Daniel

#19 lore raw
Quoted 35 lines · Daniel Borkmann
Quoted 14 lines · Thomas Monjalon

04/10/2019 11:54, Steve Capper:

Quoted 10 lines

I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64

M:   Daniel Borkmann <[email protected]>
M:   Alexei Starovoitov <[email protected]>
M:   Zi Shen Lim <[email protected]>
L:   [email protected]
L:   [email protected]
S:   Supported
F:   arch/arm64/net/
Quoted 2 lines

As they will have much better knowledge of the state of play and will be better able to advise.

Quoted 7 lines

As far as I know Alexei and Daniel are OK with the idea. But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT for userspace usage (not only DPDK) as a library. I don't understand why the DPDK JIT should be different or optimized differently.

That would be great indeed as both projects would benefit from a shared JIT instead of reimplementing everything twice. I never looked into DPDK too much, but I presume the idea would be as well to take the LLVM (or bpf-gcc) generated object file and load it into a BPF 'engine' that sits in user space on top of DPDK? Presumably loader could be libbpf here as well since it already knows how to parse the ELF, perform the relocations etc. The only difference would be that you have a different context and different helpers? Is that the goal eventually?

Quoted 1 line

The only real issue I see is the need for a dual licensing BSD-GPL.

Quoted 12 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

This idea looks good. This can remove the verifier code also from DPDK.
A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.
# DPDK supports FreeBSD and Windows OS as well
# Need a different treatment for old Linux kernels.

Quoted 2 lines
#20 lore raw
Quoted 48 lines · Jerin Jacob
Quoted 34 lines · Daniel Borkmann
Quoted 14 lines · Thomas Monjalon

04/10/2019 11:54, Steve Capper:

Quoted 10 lines

I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64

M:   Daniel Borkmann <[email protected]>
M:   Alexei Starovoitov <[email protected]>
M:   Zi Shen Lim <[email protected]>
L:   [email protected]
L:   [email protected]
S:   Supported
F:   arch/arm64/net/
Quoted 2 lines

As they will have much better knowledge of the state of play and will be better able to advise.

Quoted 7 lines

As far as I know Alexei and Daniel are OK with the idea. But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT for userspace usage (not only DPDK) as a library. I don't understand why the DPDK JIT should be different or optimized differently.

That would be great indeed as both projects would benefit from a shared JIT instead of reimplementing everything twice. I never looked into DPDK too much, but I presume the idea would be as well to take the LLVM (or bpf-gcc) generated object file and load it into a BPF 'engine' that sits in user space on top of DPDK? Presumably loader could be libbpf here as well since it already knows how to parse the ELF, perform the relocations etc. The only difference would be that you have a different context and different helpers? Is that the goal eventually?

Quoted 1 line

The only real issue I see is the need for a dual licensing BSD-GPL.

Quoted 12 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

Quoted 1 line

This idea looks good. This can remove the verifier code also from DPDK.

Right, definitely makes sense to have consolidation also on this one as well aside from the JITs, and pushing to the kernel and receiving back the JITed image seems quite nice and would be generic to open up many other use cases outside of networking. From app pov, it's just an implementation detail where to get that BPF opcode image from anyway.

Quoted 4 lines

A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.

Yep, that part would be needed, potentially BTF could be of help here as well for the description of the user space runtime environment like context, helpers etc, so that JIT knows how to handle this.

Quoted 1 line

# DPDK supports FreeBSD and Windows OS as well

I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD at least had some effort to implement a BPF runtime into their kernel as well, so similar interface could be provided, but presumably as starting point vast majority of DPDK users are running Linux underneath anyway?

Quoted 1 line

# Need a different treatment for old Linux kernels.

Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from what we see major cloud providers usually offer quite recent kernels as well as most mainstream distros that are run there, but again, environments where DPDK is typically deployed may differ (?), so cannot really comment.

Thanks,
Daniel

#21 lore raw
Quoted 22 lines
Quoted 14 lines
Quoted 13 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

Quoted 1 line

This idea looks good. This can remove the verifier code also from DPDK.

Right, definitely makes sense to have consolidation also on this one as well aside from the JITs, and pushing to the kernel and receiving back the JITed image seems quite nice and would be generic to open up many other use cases outside of networking. From app pov, it's just an implementation detail where to get that BPF opcode image from anyway.

Quoted 8 lines
Quoted 4 lines

A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.

Yep, that part would be needed, potentially BTF could be of help here as well for the description of the user space runtime environment like context, helpers etc, so that JIT knows how to handle this.

Though, We can not conclude the following non-technical aspects in this forum like
# Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
# DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS? # Need a different treatment for old Linux kernels.
This would call for immediate DPDK release to follow the existing semantics.

Yes, For the long term, Using the Kernel JITed EBPF program for userspace will be helpful. At least DPDK can use it in the future. A couple of other things to consider when someone does this # https://github.com/iovisor/ubpf can also benefit from this. # We need to think about how to support tail call in userspace # It is possible to have burst mode support in userspace as an improvement (dealing with 4 packets at the time with SIMD). Dealing SIMD in kernel space will be an issue or dealing with such improvement in general.
# Kernel verifier and dealing with address has a lot of security requirements that may not apply for userspace, and therefore some of the
optimization specific to userspace needs to consider some way # Based on my understanding the Linux and DPDK JITed code, following optimizations may need/have a different path

a) Userspace JIT has to deal with a 64bit address space. Kernel BPF code can assume the Kernel virtual address space range and optimize.
b) I see, Kernel JIT always makes stack size as non zero even though BPF applications are not using the stack. I am not sure why it is that(Could be some security issue). There could be some optimization not to push the stack pointer in the prologue if EBPF program does
not use stack
c) In DPDK JIT compiler, In this first pass, we are checking the actual registers really in use and based on that we are crafting prologue and
epilogue at runtime to improve the performance. It could be just implementation detail, but not sure why Linux kernel is not doing such optimization.

Konstantin is the author DPDK EBPF support, I just added arm64 JIT support. Maybe he has more data for this direction.

Quoted 24 lines
Quoted 2 lines

# DPDK supports FreeBSD and Windows OS as well

I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD at least had some effort to implement a BPF runtime into their kernel as well, so similar interface could be provided, but presumably as starting point vast majority of DPDK users are running Linux underneath anyway?

Quoted 1 line

# Need a different treatment for old Linux kernels.

Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from what we see major cloud providers usually offer quite recent kernels as well as most mainstream distros that are run there, but again, environments where DPDK is typically deployed may differ (?), so cannot really comment.

Thanks,
Daniel

#22 lore raw

Hi everyone,

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

Quoted 1 line
Quoted 1 line
Quoted 1 line

This idea looks good. This can remove the verifier code also from DPDK.

Yes, from one side that idea looks very tempting,
As I understand in that case we wouldn't need to worry about licensing, plus we getting JIT, verifier and might be even cBPF support for free... As the downside, as Jerin also outlined below - no support for other OSes, plus in future to get new feature users might need to upgrade to latest kernel.

Just exploring the alternate approach - if we put away for now this licensing hussle, how difficult you think it would be to make current bpf kernel code sort of platform independent entity that could be build both as part of kernel and as standalone user-space lib?
Again would it be useful for current eBPF users to have an ability to build/run verifier/jit in user-space too (might be easier to debug, faster prototyping, etc.),
or just extra pain for maintainers?

Konstantin

Quoted 81 lines
Quoted 7 lines

Right, definitely makes sense to have consolidation also on this one as well aside from the JITs, and pushing to the kernel and receiving back the JITed image seems quite nice and would be generic to open up many other use cases outside of networking. From app pov, it's just an implementation detail where to get that BPF opcode image from anyway.

Quoted 8 lines
Quoted 4 lines

A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.

Yep, that part would be needed, potentially BTF could be of help here as well for the description of the user space runtime environment like context, helpers etc, so that JIT knows how to handle this.

Though, We can not conclude the following non-technical aspects in this forum like
# Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
# DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS? # Need a different treatment for old Linux kernels.
This would call for immediate DPDK release to follow the existing semantics.

Yes, For the long term, Using the Kernel JITed EBPF program for userspace will be helpful. At least DPDK can use it in the future. A couple of other things to consider when someone does this # https://github.com/iovisor/ubpf can also benefit from this. # We need to think about how to support tail call in userspace # It is possible to have burst mode support in userspace as an improvement (dealing with 4 packets at the time with SIMD). Dealing SIMD in kernel space will be an issue or dealing with such improvement in general.
# Kernel verifier and dealing with address has a lot of security requirements that may not apply for userspace, and therefore some of the
optimization specific to userspace needs to consider some way # Based on my understanding the Linux and DPDK JITed code, following optimizations may need/have a different path

a) Userspace JIT has to deal with a 64bit address space. Kernel BPF code can assume the Kernel virtual address space range and optimize.
b) I see, Kernel JIT always makes stack size as non zero even though BPF applications are not using the stack. I am not sure why it is that(Could be some security issue). There could be some optimization not to push the stack pointer in the prologue if EBPF program does
not use stack
c) In DPDK JIT compiler, In this first pass, we are checking the actual registers really in use and based on that we are crafting prologue and
epilogue at runtime to improve the performance. It could be just implementation detail, but not sure why Linux kernel is not doing such optimization.

Konstantin is the author DPDK EBPF support, I just added arm64 JIT support. Maybe he has more data for this direction.

Quoted 24 lines
Quoted 2 lines

# DPDK supports FreeBSD and Windows OS as well

I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD at least had some effort to implement a BPF runtime into their kernel as well, so similar interface could be provided, but presumably as starting point vast majority of DPDK users are running Linux underneath anyway?

Quoted 1 line

# Need a different treatment for old Linux kernels.

Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from what we see major cloud providers usually offer quite recent kernels as well as most mainstream distros that are run there, but again, environments where DPDK is typically deployed may differ (?), so cannot really comment.

Thanks,
Daniel

#23 lore raw
In reply to Jerin Jacob
Quoted 55 lines · Jerin Jacob
Quoted 23 lines
Quoted 15 lines
Quoted 14 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

Quoted 1 line

This idea looks good. This can remove the verifier code also from DPDK.

Right, definitely makes sense to have consolidation also on this one as well aside from the JITs, and pushing to the kernel and receiving back the JITed image seems quite nice and would be generic to open up many other use cases outside of networking. From app pov, it's just an implementation detail where to get that BPF opcode image from anyway.

Quoted 8 lines
Quoted 4 lines

A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.

Yep, that part would be needed, potentially BTF could be of help here as well for the description of the user space runtime environment like context, helpers etc, so that JIT knows how to handle this.

Though, We can not conclude the following non-technical aspects in this forum like
# Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
# DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS? # Need a different treatment for old Linux kernels.
This would call for immediate DPDK release to follow the existing semantics.

Yes, For the long term, Using the Kernel JITed EBPF program for userspace will be helpful. At least DPDK can use it in the future. A couple of other things to consider when someone does this # https://github.com/iovisor/ubpf can also benefit from this. # We need to think about how to support tail call in userspace # It is possible to have burst mode support in userspace as an improvement (dealing with 4 packets at the time with SIMD). Dealing SIMD in kernel space will be an issue or dealing with such improvement in general.
# Kernel verifier and dealing with address has a lot of security requirements that may not apply for userspace, and therefore some of the
optimization specific to userspace needs to consider some way # Based on my understanding the Linux and DPDK JITed code, following optimizations may need/have a different path

Quoted 14 lines

a) Userspace JIT has to deal with a 64bit address space. Kernel BPF code can assume the Kernel virtual address space range and optimize.
b) I see, Kernel JIT always makes stack size as non zero even though BPF applications are not using the stack. I am not sure why it is that(Could be some security issue). There could be some optimization not to push the stack pointer in the prologue if EBPF program does
not use stack
c) In DPDK JIT compiler, In this first pass, we are checking the actual registers really in use and based on that we are crafting prologue and
epilogue at runtime to improve the performance. It could be just implementation detail, but not sure why Linux kernel is not doing such optimization.

Just pinging back to see if anyone interested in the common library. Answers to the above questions and mainly this below thead will make us forward progress for common library if there is still interest in collaboration.

http://mails.dpdk.org/archives/dev/2019-October/146063.html

#24 lore raw
In reply to Jerin Jacob

Hi guys,

Quoted 49 lines · Daniel Borkmann
Quoted 35 lines · Thomas Monjalon
Quoted 14 lines

04/10/2019 11:54, Steve Capper:

Quoted 10 lines

I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64

M:   Daniel Borkmann <[email protected]>
M:   Alexei Starovoitov <[email protected]>
M:   Zi Shen Lim <[email protected]>
L:   [email protected]
L:   [email protected]
S:   Supported
F:   arch/arm64/net/
Quoted 2 lines

As they will have much better knowledge of the state of play and will be better able to advise.

Quoted 7 lines

As far as I know Alexei and Daniel are OK with the idea. But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT for userspace usage (not only DPDK) as a library. I don't understand why the DPDK JIT should be different or optimized differently.

That would be great indeed as both projects would benefit from a shared JIT instead of reimplementing everything twice. I never looked into DPDK too much, but I presume the idea would be as well to take the LLVM (or bpf-gcc) generated object file and load it into a BPF 'engine' that sits in user space on top of DPDK? Presumably loader could be libbpf here as well since it already knows how to parse the ELF, perform the relocations etc. The only difference would be that you have a different context and different helpers? Is that the goal eventually?

Quoted 1 line

The only real issue I see is the need for a dual licensing BSD-GPL.

Quoted 12 lines

This might be one avenue if all kernel JIT contributors would be on board. Another option I'm wondering could be to extend the bpf() syscall in order to pass down a description of context and helper mappings e.g. via BTF and let everything go through the verifier in the kernel the usual way (I presume one goal might be that you want to assure that the generated BPF code passes the safety checks before running the prog), then have it JITed and extract the generated image in order to use it from user space. Kernel would have to make sure it never actually allows attaching this program in the kernel. Generated opcodes can already be retrieved today (see below). Such infra could potentially help bpf-gcc folks as well as they expressed desire to have some sort of a simulator for their gcc BPF test suite.. and it would allow for consistent behavior of the BPF runtime. Just a thought.

Quoted 7 lines

This idea looks good. This can remove the verifier code also from DPDK.
A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address and its symbols for CALL and MEM access operations.
# DPDK supports FreeBSD and Windows OS as well
# Need a different treatment for old Linux kernels.

Seems like discussion died out eventually.
Pinging to check is there still any interest on that subject from kernel community.
Thanks
Konstantin

#25 lore raw
In reply to Ananyev, Konstantin
Quoted 42 lines · Konstantin

Hi everyone,

Quoted 30 lines · Thomas Monjalon
Quoted 10 lines

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 3 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code
from Linux as is.

Quoted 8 lines

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

Quoted 8 lines

My thought would be that if we have JIT eBPF compiler already in DPDK
for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm). About having a common code-base with Linux eBPF JITs implementation -
I think it is a very good idea,
but I don’t' think it could be achieved without significant effort.
DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
meanwhile start talking with linux guys how common JIT code-base could be achieved.

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code duplication etc and it will take a couple of months to converge(if at all it happens)

Just to be clear, I assume, you are OK to merge this code for 19.11(If no more technical comment on the patch).

I am only afraid of, our typical last-minute surprise pattern and followed by back and forth open ended discussions.

i.e

# Code submitted before the proposal window # Gets ACK from Maintainer
# New non-technical concerns start just before RC1

Quoted 5 lines

Konstantin

#26 lore raw

04/10/2019 17:39, Jerin Jacob:

Quoted 65 lines
Quoted 42 lines · Konstantin

Hi everyone,

Quoted 30 lines · Thomas Monjalon
Quoted 10 lines

03/09/2019 12:59, [email protected]:

Quoted 4 lines

Added eBPF arm64 JIT support to improve the eBPF program performance on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 3 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code
from Linux as is.

Quoted 8 lines

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

Quoted 8 lines

My thought would be that if we have JIT eBPF compiler already in DPDK
for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm). About having a common code-base with Linux eBPF JITs implementation -
I think it is a very good idea,
but I don’t' think it could be achieved without significant effort.
DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
meanwhile start talking with linux guys how common JIT code-base could be achieved.

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code duplication etc and it will take a couple of months to converge(if at all it happens)

Just to be clear, I assume, you are OK to merge this code for 19.11(If no more technical comment on the patch).

Quoted 8 lines

I am only afraid of, our typical last-minute surprise pattern and followed by back and forth open ended discussions.

i.e

# Code submitted before the proposal window # Gets ACK from Maintainer
# New non-technical concerns start just before RC1

I hope you are not against discussing the real good questions, even if they come a month after the first submission.

I don't care merging such patch in 19.11, but I would have preferred such questions were open when introducing this new library (for x86).

About your urge of having this code merged,
please can you explain what is your usage?

#27 lore raw
Quoted 80 lines · Thomas Monjalon,

04/10/2019 17:39, Jerin Jacob:

Quoted 7 lines
Quoted 5 lines · Konstantin

Hi everyone,

Quoted 2 lines

On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

03/09/2019 12:59, [email protected]:

Quoted 1 line

Added eBPF arm64 JIT support to improve the eBPF program

performance

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

on arm64.

 lib/librte_bpf/bpf_jit_arm64.c         | 1451

++++++++++++++++++++++++

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I am concerned about duplicating the BPF JIT effort in DPDK and

Linux.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 2 lines

After a quick discussion, it seems the Linux authors are OK to

arrange

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also

to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code
from Linux as is.

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in

DPDK).

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

My thought would be that if we have JIT eBPF compiler already in DPDK for one arch (x86) there is absolutely no reason why we shouldn't

allow it for different arch (arm).

Quoted 6 lines
Quoted 6 lines

About having a common code-base with Linux eBPF JITs implementation - I think it is a very good idea,
but I don’t' think it could be achieved without significant effort. DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11, meanwhile start talking with linux guys how common JIT code-base could

be achieved.

Quoted 8 lines

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code

duplication etc

Quoted 8 lines

and it will take a couple of months to converge(if at all it happens)

Just to be clear, I assume, you are OK to merge this code for 19.11(If no more technical comment on the patch).

I am only afraid of, our typical last-minute surprise pattern and followed by back and forth open ended discussions.

Quoted 9 lines
Quoted 5 lines

i.e

# Code submitted before the proposal window # Gets ACK from Maintainer
# New non-technical concerns start just before RC1

I hope you are not against discussing the real good questions, even if they come a month after the first submission.

I am not against discussing the technical data about the 'patch' and review it. If there is a review with respect to content of the patch it is very good, I am happy to address it. Stuff like I don't have any control ( changing the licence) etc, I have am not comfortable to take in last minute. I have already shared the eBPF ARM64 JIT support in roadmap a month ago before implementing it. No question asked that time. Spend a almost month to add support for it and It is not a simple C code. Now I am not comfortable in asking the fundamental questions like why this eBPF it self is required and code duplication ( code was duplicated when x86 support has been added) and therefore stall the patch at this point of time, where this library and x86 support added a year back.

Quoted 7 lines

I don't care merging such patch in 19.11, but I would have preferred such questions were open when introducing this new library (for x86).

Konstantin added enough data on ml this when this library gets added on reply to different users.

Quoted 3 lines

About your urge of having this code merged,
please can you explain what is your usage?

As an ARM64 maintainter, I would like to fix any disparity in terms of the features with respect to x86 and I have been doing for last 3 years. If some one using eBPF on x86, I want to make sure it run in similar "performance" on arm64 on architecture perspective.

Quoted 2 lines
#28 lore raw

07/10/2019 15:00, Jerin Jacob:

Quoted 73 lines · Thomas Monjalon
Quoted 63 lines

04/10/2019 17:39, Jerin Jacob:

Quoted 62 lines · Konstantin
Quoted 35 lines
Quoted 26 lines · Thomas Monjalon
Quoted 10 lines

03/09/2019 12:59, [email protected]:

Quoted 5 lines

Added eBPF arm64 JIT support to improve the eBPF
program performance on arm64.

lib/librte_bpf/bpf_jit_arm64.c
| 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux. Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 2 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch also to Linux as well.(Some more pending as well)
https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging etc. So We can not copy and paste the code from Linux as is.

Quoted 8 lines

My view to keep a different code base optimize for DPDK use cases and library requirements(for example, tail call is not supported in DPDK). For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

Quoted 10 lines

My thought would be that if we have JIT eBPF compiler already in DPDK for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
About having a common code-base with Linux eBPF JITs implementation - I think it is a very good idea,
but I don’t' think it could be achieved without significant effort. DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11, meanwhile start talking with linux guys how common JIT code-base could be achieved.

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code duplication etc and it will take a couple of months to converge (if at all it happens)
Just to be clear, I assume, you are OK to merge this code for 19.11 (If no more technical comment on the patch).

I am only afraid of, our typical last-minute surprise pattern and followed by back and forth open ended discussions.

Quoted 8 lines
Quoted 5 lines

i.e

# Code submitted before the proposal window # Gets ACK from Maintainer
# New non-technical concerns start just before RC1

I hope you are not against discussing the real good questions, even if they come a month after the first submission.

Quoted 11 lines

I am not against discussing the technical data about the 'patch' and review it. If there is a review with respect to content of the patch it is very good, I am happy to address it. Stuff like I don't have any control ( changing the licence) etc, I have am not comfortable to take in last minute. I have already shared the eBPF ARM64 JIT support in roadmap a month ago before implementing it. No question asked that time. Spend a almost month to add support for it and It is not a simple C code. Now I am not comfortable in asking the fundamental questions like why this eBPF it self is required and code duplication ( code was duplicated when x86 support has been added) and therefore stall the patch at this point of time, where this library and x86 support added a year back.

I really don't like this reaction.
First, I never said this discussion was blocking the patch. Second, why am I the only one asking such obvious questions as not duplicating work?

Quoted 3 lines
Quoted 3 lines

I don't care merging such patch in 19.11, but I would have preferred such questions were open when introducing this new library (for x86).

You Jerin and Konstantin should have answered these questions a long time ago before starting such development.
Is it so hard to require a bit of thoughts before starting something new?

Quoted 2 lines

Konstantin added enough data on ml this when this library gets added on reply to different users.

Really? which data?

Quoted 7 lines
Quoted 2 lines

About your urge of having this code merged,
please can you explain what is your usage?

As an ARM64 maintainter, I would like to fix any disparity in terms of the features with respect to x86 and I have been doing for last 3 years. If some one using eBPF on x86, I want to make sure it run in similar "performance" on arm64 on architecture perspective.

So we are debating about a library which is probably not used by anybody. That's not how I plan to spend my time on DPDK.

Sorry Jerin, I really like working with you,
but I think you forward too much pressure here,
instead of quietly discussing the future of DPDK.

Please forget the deadline (we will agree on merging anyway) and let's restart from the beginning by answering simple questions:
- what are the use cases of BPF in DPDK?
- how much we'll benefit from sharing code with Linux?
- what can we lose in a single JIT implementation?

#29 lore raw
Quoted 95 lines · Thomas Monjalon,

07/10/2019 15:00, Jerin Jacob:

Quoted 12 lines · Thomas Monjalon
Quoted 11 lines

04/10/2019 17:39, Jerin Jacob:

Quoted 10 lines · Konstantin
Quoted 9 lines
Quoted 9 lines · Thomas Monjalon
Quoted 8 lines

03/09/2019 12:59, [email protected]:

Quoted 5 lines

Added eBPF arm64 JIT support to improve the eBPF
program performance on arm64.

lib/librte_bpf/bpf_jit_arm64.c
| 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK

and Linux.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

Could we try to pull the Linux JIT?
Is the license the only issue?

That's one issue.

Quoted 2 lines

After a quick discussion, it seems the Linux authors are OK to arrange their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization

for

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

DPDK etc(Like if stack is not used then don't push stack etc) and wherever Linux can be improved, I have submitted the patch

also

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

And Linux has a framework for instruction generation for

debugging

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

etc. So We can not copy and paste the code from Linux as is.

My view to keep a different code base optimize for DPDK use

cases and

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

library requirements(for example, tail call is not supported in

DPDK).

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

For arm64/x86 case the code is done so it is not worth sync with Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

My thought would be that if we have JIT eBPF compiler already in

DPDK

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
About having a common code-base with Linux eBPF JITs

implementation -

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I think it is a very good idea,
but I don’t' think it could be achieved without significant effort. DPDK and Linux JIT code-generators differ quite a bit. So my suggestion - let's go ahead and integrate Jerin patch into

19.11,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

meanwhile start talking with linux guys how common JIT code-base

could

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 1 line

be achieved.

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code duplication etc and it will take a couple of months to converge (if at all it happens)
Just to be clear, I assume, you are OK to merge this code for 19.11 (If no more technical comment on the patch).

I am only afraid of, our typical last-minute surprise pattern and followed by back and forth open ended discussions.

i.e

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

# Code submitted before the proposal window # Gets ACK from Maintainer
# New non-technical concerns start just before RC1

I hope you are not against discussing the real good questions, even if they come a month after the first submission.

I am not against discussing the technical data about the 'patch' and

review

Quoted 4 lines

it. If there is a review with respect to content of the patch it is very good, I am happy to address it. Stuff like I don't have any control ( changing the licence) etc, I have am not comfortable to take in last minute. I have already shared the eBPF ARM64 JIT support in roadmap a

month

Quoted 3 lines

ago before implementing it. No question asked that time. Spend a almost month to add support for it and It is not a simple C code. Now I am not comfortable in asking the fundamental questions like why this eBPF it

Quoted 8 lines

self

Quoted 2 lines

is required and code duplication ( code was duplicated when x86 support has been added) and therefore stall the patch at this point of time,

where

Quoted 1 line

this library and x86 support added a year back.

I really don't like this reaction.

If it hurts you in some way then I am sorry about that.

First, I never said this discussion was blocking the patch.

Quoted 1 line

You said you have concern with this patch. Sorry,
I am not sure how to interpret that and if I don't jump in it will be stalled for sure. That's my experience. Sorry if you dis agree.

Second, why am I the only one asking such obvious questions

Quoted 2 lines

as not duplicating work?

Some things it does not converge at all. Especially relicecing some code from linux. There are a lot developers(even me) are involved in that code base. Why would everyone agree? The list would include a recent RISCV JIT contributer from gmail.com as example.

Duplication the semantics some times gives the morecontrol. We already did that for rte_flow, rcu etc. I have mentioned the performance reason as well for JIT in the other thread.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

I don't care merging such patch in 19.11, but I would have preferred such questions were open when introducing this new library (for x86).

You Jerin and Konstantin should have answered these questions a long time ago before starting such development.
Is it so hard to require a bit of thoughts before starting something new?

For me, I don't see any better approach to have user space eBPF to support all OS in DPDK.

Quoted 5 lines
Quoted 2 lines

Konstantin added enough data on ml this when this library gets added on reply to different users.

Really? which data?

I am talking about the discussion with
niterome developer.I don't have exact email thread, probably Konstantin may have

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

About your urge of having this code merged,
please can you explain what is your usage?

Quoted 9 lines
Quoted 1 line

As an ARM64 maintainter, I would like to fix any disparity in terms of

the

Quoted 3 lines

features with respect to x86 and I have been doing for last 3 years. If some one using eBPF on x86, I want to make sure it run in similar "performance" on arm64 on architecture perspective.

So we are debating about a library which is probably not used by anybody. That's not how I plan to spend my time on DPDK.

How do anyone know that the library is not used by anyone in community if it is part of dpdk.org and a customer asked does arm64 has JIT support too.

If something needs to be dynamically controlled then eBPF can be used, couple of use cases

# packet filtering
# debugging
# function call tracing
# There are some Lua JIT based dataplane implementations. Which can be
replaced with eBPF with JIT.
Quoted 4 lines

Sorry Jerin, I really like working with you,

Mee too.

but I think you forward too much pressure here,

Quoted 4 lines

instead of quietly discussing the future of DPDK.

Please forget the deadline (we will agree on merging anyway)

Ok.

and let's restart from the beginning by answering simple questions:

Quoted 2 lines

- what are the use cases of BPF in DPDK?

I meantioned what I know,

- how much we'll benefit from sharing code with Linux?

Quoted 1 line

I have mentioned some of the performance constraint in the other thread. Moreover I don't believe it is not easy task for Linux eBPF to run as userspace and I not sure who is going to do that

- what can we lose in a single JIT implementation?

Quoted 1 line

Sorry, I didn't understood this question?

Quoted 2 lines
#30 lore raw

07/10/2019 21:29, Jerin Jacob:

Quoted 1 line · Thomas Monjalon,

[...]

Quoted 12 lines

let's restart from the beginning by answering simple questions:

Quoted 1 line

- what are the use cases of BPF in DPDK?

If something needs to be dynamically controlled then eBPF can be used, couple of use cases

# packet filtering
# debugging
# function call tracing
# There are some Lua JIT based dataplane implementations. Which can be
replaced with eBPF with JIT.
Quoted 5 lines

- how much we'll benefit from sharing code with Linux?

I have mentioned some of the performance constraint in the other thread. Moreover I don't believe it is not easy task for Linux eBPF to run as userspace and I not sure who is going to do that

I was asking the benefits here:
- sharing optimizations in both projects
- get verifier support
What else?

Quoted 3 lines

- what can we lose in a single JIT implementation?

Sorry, I didn't understood this question?

I mean what are the drawbacks of using a Linux implementation? How performance constraints are differents, etc?

Note: as a lot of people, I don't really know BPF,
so these are real questions to help understanding the challenge.

#31 lore raw
Quoted 16 lines · Thomas Monjalon,

07/10/2019 21:29, Jerin Jacob:

Quoted 1 line

On Mon, 7 Oct, 2019, 11:35 PM Thomas Monjalon, <[email protected]>

wrote:
[...]

Quoted 12 lines

let's restart from the beginning by answering simple questions:

Quoted 1 line

- what are the use cases of BPF in DPDK?

If something needs to be dynamically controlled then eBPF can be used, couple of use cases

# packet filtering
# debugging
# function call tracing
# There are some Lua JIT based dataplane implementations. Which can be
replaced with eBPF with JIT.
Quoted 9 lines
Quoted 5 lines

- how much we'll benefit from sharing code with Linux?

I have mentioned some of the performance constraint in the other thread. Moreover I don't believe it is not easy task for Linux eBPF to run as userspace and I not sure who is going to do that

I was asking the benefits here:
- sharing optimizations in both projects

Yes. But even if it is different code base it is possible to share the optimization.

- get verifier support

Quoted 1 line

Verifier support already available in the library.

What else?

Quoted 1 line

I see only avoiding code duplication and getting new feature like cBPF.

Quoted 7 lines
Quoted 3 lines

- what can we lose in a single JIT implementation?

Sorry, I didn't understood this question?

I mean what are the drawbacks of using a Linux implementation? How performance constraints are differents, etc?

Mention the details in the below thread. Waiting for feedback from Kernel maintainer.

http://mails.dpdk.org/archives/dev/2019-October/146004.html

http://mails.dpdk.org/archives/dev/2019-October/146063.html

Quoted 8 lines

Note: as a lot of people, I don't really know BPF,
so these are real questions to help understanding the challenge.