Skip to contentSkip to conversation

LSF/MM session: eBPF standardization

DISCUSSION Christoph Hellwig 3 participants 5 messages May 3, 2022 – May 17, 2022 BPF

lore
Share & tools
lore
#0 lore raw

Hi Alexei and Daniel,

if you have some time left I'd like to kick off a short discussion on what to do about documenting and standardizing eBPF.

#1 lore raw

Thanks everyone who participated.

Here is my rough memory an action items from the meeting. As I was on stage and did not take notes these might be a bit off and may need correction.

The separate instruction set document wasn't known by everyone but seens as a good idea.

The content needs a little more work:

- document the version levels, based on the clang cpu levels (I plan to do this ASAP)
- we need to decide to do about the legacy BPF packet access instrutions. Alexei mentioned that the modern JIT doesn't even use those internally any more.
- we need to document behavior for underflows / overflows and other behavior not mentioned. The example in the session was divive by zero behavior. Are there any notes on what the consensus for a lot of this behavior is, or do we need to reverse engineer it from the implementation? I'd happily write the documentation, but I'd be really grateful for any input into what needs to go into it

Discussion on where to host a definitive version of the document:

- I think the rough consensus is to just host regular (hopefully low cadence) documents and maybe the latest gratest at a eBPF foundation website. Whom do we need to work with at the fundation to make this happen?
- On a technical side we need to figure out a way how to build a standalone document from the kerneldoc tree of documents. I volunteers to look into that as well.

The verifier is not very well documented, and mixes up generic behavior with that of specific implementations and program types.

- as idea it was brought up to write a doument with the minimal verification requirements required for any eBPF implementation independent of the program type. Again I can volunteer to draft a documentation, but I need input on what such a consensus would be. In this case input from the non-Linux verifier implementors (I only know the Microsoft research one) would be very helpful as well.

#2 lore raw
Quoted 7 lines · Christoph Hellwig

Thanks everyone who participated.

Here is my rough memory an action items from the meeting. As I was on stage and did not take notes these might be a bit off and may need correction.

Quoted 7 lines

The separate instruction set document wasn't known by everyone but seens as a good idea.

The content needs a little more work:

- document the version levels, based on the clang cpu levels (I plan to do this ASAP)

Turns out that clang -mcpu=v1,v2,v3 are not exactly correct. We've extended ISA more than three times.
For example when we added more atomics insns in
https://lore.kernel.org/bpf/[email protected]/

The corresponding llvm diff didn't add a new -mcpu flavor. There was no need to do it.

Also llvm flags can turn a subset of insns on and off. Like llvm can turn on alu32, but without <,<= insns. -mcpu=v3 includes both. -mcpu=v2 are only <,<=.

So we need a plan B.

How about using the commit sha where support was added to the verifier as a 'version' of the ISA ?

We can try to use a kernel version, but backports will ruin that picture.
Looks like upstream 'commit sha' is the only stable number.

Another approach would be to declare the current ISA as 1.0 (or bpf-isa-may-2022) and
in the future bump it with every new insn.

Quoted 3 lines

- we need to decide to do about the legacy BPF packet access instrutions. Alexei mentioned that the modern JIT doesn't even use those internally any more.

I think we need to document them as supported in the linux kernel, but deprecated in general.
The standard might say "implementation defined" meaning that different run-times don't have to support them.

Quoted 7 lines

- we need to document behavior for underflows / overflows and other behavior not mentioned. The example in the session was divive by zero behavior. Are there any notes on what the consensus for a lot of this behavior is, or do we need to reverse engineer it from the implementation? I'd happily write the documentation, but I'd be really grateful for any input into what needs to go into it

For div by zero see do_misc_fixups().
We patch it with: 'if src_reg == 0 -> xor dst_reg, dst_reg'. Interpreter and JITs will execute div/0 as-is, but in practice it cannot happen because the verifier patched it.

Other undefined/underflows/overflows are implementation defined. Meaning after JIT-ing they may behave differently on different architectures.
For example the interpreter for shifts does
DST = DST OP (SRC & 63);
where OP is <<, >>
to avoid undefined behavior in C.

The JITs won't be adding the masking insns, since CPU HW will do a mask implicitly. Which could potentially change from one CPU to another.
I don't think it's worth documenting all that.
I would group all undefined/underflow/overflow as implementation defined and document only things that matter.

Quoted 7 lines

Discussion on where to host a definitive version of the document:

- I think the rough consensus is to just host regular (hopefully low cadence) documents and maybe the latest gratest at a eBPF foundation website. Whom do we need to work with at the fundation to make this happen?

foundation folks cc-ed.

Quoted 3 lines

- On a technical side we need to figure out a way how to build a standalone document from the kerneldoc tree of documents. I volunteers to look into that as well.

+1

Quoted 10 lines

The verifier is not very well documented, and mixes up generic behavior with that of specific implementations and program types.

- as idea it was brought up to write a doument with the minimal verification requirements required for any eBPF implementation independent of the program type. Again I can volunteer to draft a documentation, but I need input on what such a consensus would be. In this case input from the non-Linux verifier implementors (I only know the Microsoft research one) would be very helpful as well.

The verifier is a moving target.
I'd say minimal verification is the one that checks that:
- instructions are formed correctly
- opcode is valid
- no reserved bits are used
- registers are within range (r11+ are not used)
- combination of opcode+regs+off+imm is valid
- simple things like that

#3 lore raw
Quoted 7 lines · Alexei Starovoitov

Turns out that clang -mcpu=v1,v2,v3 are not exactly correct. We've extended ISA more than three times.
For example when we added more atomics insns in
https://lore.kernel.org/bpf/[email protected]/

The corresponding llvm diff didn't add a new -mcpu flavor. There was no need to do it.

.. because all uses of these new instructions are through builtins that wouldn't other be available, yes.

Quoted 5 lines

Also llvm flags can turn a subset of insns on and off. Like llvm can turn on alu32, but without <,<= insns. -mcpu=v3 includes both. -mcpu=v2 are only <,<=.

So we need a plan B.

Yes.

Quoted 6 lines

How about using the commit sha where support was added to the verifier as a 'version' of the ISA ?

We can try to use a kernel version, but backports will ruin that picture.
Looks like upstream 'commit sha' is the only stable number.

Using kernel release hashes is a pretty horrible interface, especially for non-kernel users. I also think the compilers and other tools would really like some vaguely meaninfuly identifiers.

Quoted 3 lines

Another approach would be to declare the current ISA as 1.0 (or bpf-isa-may-2022) and
in the future bump it with every new insn.

I think that is a much more reasonable starting position. However, are we sure the ISA actually evolves linearly? As far as I can tell support for full atomics only exists for a few JITs so far.

So maybe starting with a basedline, and then just have name for each meaningful extension (e.g. the full atomics as a start) might be even better. For the Linux kernel case we can then also have a user interface where userspace programs can query which extensions are supported before loading eBPF programs that rely on them instead of doing a roundtrip through the verifier.

Quoted 8 lines
Quoted 3 lines

- we need to decide to do about the legacy BPF packet access instrutions. Alexei mentioned that the modern JIT doesn't even use those internally any more.

I think we need to document them as supported in the linux kernel, but deprecated in general.
The standard might say "implementation defined" meaning that different run-times don't have to support them.

Yeah. If we do the extensions proposal above we could make these a specific extension as well.

Quoted 4 lines

[...]
I don't think it's worth documenting all that.
I would group all undefined/underflow/overflow as implementation defined and document only things that matter.

Makese sense.

Quoted 8 lines
Quoted 6 lines

Discussion on where to host a definitive version of the document:

- I think the rough consensus is to just host regular (hopefully low cadence) documents and maybe the latest gratest at a eBPF foundation website. Whom do we need to work with at the fundation to make this happen?

foundation folks cc-ed.

I'd be rally glad if we could kick off this ASAP. Feel free to contact me privately if we want to keep it off the list.

Quoted 9 lines
Quoted 7 lines

- as idea it was brought up to write a doument with the minimal verification requirements required for any eBPF implementation independent of the program type. Again I can volunteer to draft a documentation, but I need input on what such a consensus would be. In this case input from the non-Linux verifier implementors (I only know the Microsoft research one) would be very helpful as well.

The verifier is a moving target.

Absolutely.

Quoted 7 lines

I'd say minimal verification is the one that checks that:
- instructions are formed correctly
- opcode is valid
- no reserved bits are used
- registers are within range (r11+ are not used)
- combination of opcode+regs+off+imm is valid
- simple things like that

Sounds good. One useful thing for this would be an opcode table with all the optional field usage in machine readable format.

Jim who is on CC has already built a nice table off all opcodes based on existing material that might be a good starting point.

#4 lore raw
Quoted 101 lines · Christoph Hellwig
Quoted 7 lines · Alexei Starovoitov

Turns out that clang -mcpu=v1,v2,v3 are not exactly correct. We've extended ISA more than three times.
For example when we added more atomics insns in
https://lore.kernel.org/bpf/[email protected]/

The corresponding llvm diff didn't add a new -mcpu flavor. There was no need to do it.

.. because all uses of these new instructions are through builtins that wouldn't other be available, yes.

Quoted 5 lines

Also llvm flags can turn a subset of insns on and off. Like llvm can turn on alu32, but without <,<= insns. -mcpu=v3 includes both. -mcpu=v2 are only <,<=.

So we need a plan B.

Yes.

Quoted 6 lines

How about using the commit sha where support was added to the verifier as a 'version' of the ISA ?

We can try to use a kernel version, but backports will ruin that picture.
Looks like upstream 'commit sha' is the only stable number.

Using kernel release hashes is a pretty horrible interface, especially for non-kernel users. I also think the compilers and other tools would really like some vaguely meaninfuly identifiers.

Quoted 3 lines

Another approach would be to declare the current ISA as 1.0 (or bpf-isa-may-2022) and
in the future bump it with every new insn.

I think that is a much more reasonable starting position. However, are we sure the ISA actually evolves linearly? As far as I can tell support for full atomics only exists for a few JITs so far.

So maybe starting with a basedline, and then just have name for each meaningful extension (e.g. the full atomics as a start) might be even better. For the Linux kernel case we can then also have a user interface where userspace programs can query which extensions are supported before loading eBPF programs that rely on them instead of doing a roundtrip through the verifier.

Quoted 8 lines
Quoted 3 lines

- we need to decide to do about the legacy BPF packet access instrutions. Alexei mentioned that the modern JIT doesn't even use those internally any more.

I think we need to document them as supported in the linux kernel, but deprecated in general.
The standard might say "implementation defined" meaning that different run-times don't have to support them.

Yeah. If we do the extensions proposal above we could make these a specific extension as well.

Quoted 4 lines

[...]
I don't think it's worth documenting all that.
I would group all undefined/underflow/overflow as implementation defined and document only things that matter.

Makese sense.

Quoted 8 lines
Quoted 6 lines

Discussion on where to host a definitive version of the document:

- I think the rough consensus is to just host regular (hopefully low cadence) documents and maybe the latest gratest at a eBPF foundation website. Whom do we need to work with at the fundation to make this happen?

foundation folks cc-ed.

I'd be rally glad if we could kick off this ASAP. Feel free to contact me privately if we want to keep it off the list.

Quoted 9 lines
Quoted 7 lines

- as idea it was brought up to write a doument with the minimal verification requirements required for any eBPF implementation independent of the program type. Again I can volunteer to draft a documentation, but I need input on what such a consensus would be. In this case input from the non-Linux verifier implementors (I only know the Microsoft research one) would be very helpful as well.

The verifier is a moving target.

Absolutely.

Quoted 7 lines

I'd say minimal verification is the one that checks that:
- instructions are formed correctly
- opcode is valid
- no reserved bits are used
- registers are within range (r11+ are not used)
- combination of opcode+regs+off+imm is valid
- simple things like that

Quoted 5 lines

Sounds good. One useful thing for this would be an opcode table with all the optional field usage in machine readable format.

Jim who is on CC has already built a nice table off all opcodes based on existing material that might be a good starting point.

Table is inline below. I used the tables in the iovisor project documentation as a starting point for this table.

https://github.com/iovisor/bpf-docs/blob/master/eBPF.md.

Feedback welcome. The atomic sections at the bottom could especially use some careful review for correctness.

BPF_ALU64 (opc & 0x07 == 0x07)
==============================

0x07		BPF_ALU64 | BPF_K | BPF_ADD	bpf_add dst, imm	dst += imm
0x0f		BPF_ALU64 | BPF_X | BPF_ADD	bpf_add dst, src	dst += src
0x17		BPF_ALU64 | BPF_K | BPF_SUB	bpf_sub dst, imm	dst -= imm
0x1f		BPF_ALU64 | BPF_X | BPF_SUB	bpf_sub dst, src	dst -= src
0x27		BPF_ALU64 | BPF_K | BPF_MUL	bpf_mul dst, imm	dst *= imm
0x2f		BPF_ALU64 | BPF_X | BPF_MUL	bpf_mul dst, src	dst *= src
0x37		BPF_ALU64 | BPF_K | BPF_DIV	bpf_div dst, imm	dst /= imm
0x3f		BPF_ALU64 | BPF_X | BPF_DIV	bpf_div dst, src	dst /= src
0x47		BPF_ALU64 | BPF_K | BPF_OR	bpf_or dst, imm		dst |= imm
0x4f		BPF_ALU64 | BPF_X | BPF_OR	bpf_or dst, src		dst |= src
0x57		BPF_ALU64 | BPF_K | BPF_AND	bpf_and dst, imm	dst &= imm
0x5f		BPF_ALU64 | BPF_X | BPF_AND	bpf_and dst, src	dst &= src
0x67		BPF_ALU64 | BPF_K | BPF_LSH	bpf_lsh dst, imm	dst <<= imm
0x6f		BPF_ALU64 | BPF_X | BPF_LSH	bpf_lsh dst, src	dst <<= src
0x77		BPF_ALU64 | BPF_K | BPF_RSH	bpf_lsh dst, imm	dst >>= imm (logical)
0x7f		BPF_ALU64 | BPF_X | BPF_RSH	bpf_lsh dst, src	dst >>= src (logical)
0x87		BPF_ALU64 | BPF_K | BPF_NEG	bpf_neg dst		dst = ~dst
0x97		BPF_ALU64 | BPF_K | BPF_MOD	bpf_mod dst, imm	dst %= imm
0x9f		BPF_ALU64 | BPF_X | BPF_MOD	bpf_mod dst, src	dst %= src
0xa7		BPF_ALU64 | BPF_K | BPF_XOR	bpf_xor dst, imm	dst ^= imm
0xaf		BPF_ALU64 | BPF_X | BPF_XOR	bpf_xor dst, src	dst ^= src
0xb7		BPF_ALU64 | BPF_K | BPF_MOV	bpf_mov dst, imm	dst = imm
0xbf		BPF_ALU64 | BPF_X | BPF_MOV	bpf_mov dst, src	dst = src
0xc7		BPF_ALU64 | BPF_K | BPF_ARSH	bpf_arsh dst, imm	dst >>= imm (arithmetic)
0xcf		BPF_ALU64 | BPF_X | BPF_ARSH	bpf_arsh dst, src	dst >>= src (arithmetic)

BPF_ALU (opc & 0x07 == 0x04)
============================

0x04		BPF_ALU | BPF_K | BPF_ADD	bpf_add32 dst, imm	dst32 += imm
0x0c		BPF_ALU | BPF_X | BPF_ADD	bpf_add32 dst, src	dst32 += src32
0x14		BPF_ALU | BPF_K | BPF_SUB	bpf_sub32 dst, imm	dst32 -= imm
0x1c		BPF_ALU | BPF_X | BPF_SUB	bpf_sub32 dst, src	dst32 -= src32
0x24		BPF_ALU | BPF_K | BPF_MUL	bpf_mul32 dst, imm	dst32 *= imm
0x2c		BPF_ALU | BPF_X | BPF_MUL	bpf_mul32 dst, src	dst32 *= src32
0x34		BPF_ALU | BPF_K | BPF_DIV	bpf_div32 dst, imm	dst32 /= imm
0x3c		BPF_ALU | BPF_X | BPF_DIV	bpf_div32 dst, src	dst32 /= src32
0x44		BPF_ALU | BPF_K | BPF_OR	bpf_or32 dst, imm	dst32 |= imm
0x4c		BPF_ALU | BPF_X | BPF_OR	bpf_or32 dst, src	dst32 |= src32
0x54		BPF_ALU | BPF_K | BPF_AND	bpf_and32 dst, imm	dst32 &= imm
0x5c		BPF_ALU | BPF_X | BPF_AND	bpf_and32 dst, src	dst32 &= src32
0x64		BPF_ALU | BPF_K | BPF_LSH	bpf_lsh32 dst, imm	dst32 <<= imm
0x6c		BPF_ALU | BPF_X | BPF_LSH	bpf_lsh32 dst, src	dst32 <<= src32
0x74		BPF_ALU | BPF_K | BPF_RSH	bpf_lsh32 dst, imm	dst32 >>= imm (logical)
0x7c		BPF_ALU | BPF_X | BPF_RSH	bpf_lsh32 dst, src	dst32 >>= src32 (logical)
0x84		BPF_ALU | BPF_K | BPF_NEG	bpf_neg32 dst		dst32 = ~dst32
0x94		BPF_ALU | BPF_K | BPF_MOD	bpf_mod32 dst, imm	dst32 %= imm
0x9c		BPF_ALU | BPF_X | BPF_MOD	bpf_mod32 dst, src	dst32 %= src32
0xa4		BPF_ALU | BPF_K | BPF_XOR	bpf_xor32 dst, imm	dst32 ^= imm
0xac		BPF_ALU | BPF_X | BPF_XOR	bpf_xor32 dst, src	dst32 ^= src32
0xb4		BPF_ALU | BPF_K | BPF_MOV	bpf_mov32 dst, imm	dst32 = imm
0xbc		BPF_ALU | BPF_X | BPF_MOV	bpf_mov32 dst, src	dst32 = src32
0xc4		BPF_ALU | BPF_K | BPF_ARSH	bpf_arsh32 dst, imm	dst32 >>= imm (arithmetic)
0xcc		BPF_ALU | BPF_X | BPF_ARSH	bpf_arsh32 dst, src	dst32 >>= src32 (arithmetic)

For BPF_END instructions, BPF_K == htole conversion, BPF_X == htobe conversion Operation size (16, 32, 64 bit) determined by 'imm' value of instruction (upper 32 bits)

0xd4 (.imm = 16)	BPF_ALU | BPF_K | BPF_END	bpf_le16 dst	dst16 = htole16(dst16)
0xd4 (.imm = 32)	BPF_ALU | BPF_K | BPF_END	bpf_le32 dst	dst32 = htole32(dst32)
0xd4 (.imm = 64)	BPF_ALU | BPF_K | BPF_END	bpf_le64 dst	dst = htole64(dst)
0xdc (.imm = 16)	BPF_ALU | BPF_X | BPF_END	bpf_be16 dst	dst16 = htobe16(dst16)
0xdc (.imm = 32)	BPF_ALU | BPF_X | BPF_END	bpf_be32 dst	dst32 = htobe32(dst32)
0xdc (.imm = 64)	BPF_ALU | BPF_X | BPF_END	bpf_be64 dst	dst = htobe64(dst)

BPF_JMP (opc & 0x07 == 0x05)
============================

0x05		BPF_JMP | BPF_K | BPF_JA	bpf_ja +off			PC += off
0x15		BPF_JMP | BPF_K | BPF_JEQ	bpf_jeq dst, imm, +off		PC += off if dst == imm
0x1d		BPF_JMP | BPF_X | BPF_JEQ	bpf_jeq dst, src, +off		PC += off if dst == src
0x25		BPF_JMP | BPF_K | BPF_JGT	bpf_jgt dst, imm, +off		PC += off if dst > imm
0x2d		BPF_JMP | BPF_X | BPF_JGT	bpf_jgt dst, src, +off		PC += off if dst > src
0x35		BPF_JMP | BPF_K | BPF_JGE	bpf_jge dst, imm, +off		PC += off if dst >= imm
0x3d		BPF_JMP | BPF_X | BPF_JGE	bpf_jge dst, src, +off		PC += off if dst >= src
0x45		BPF_JMP | BPF_K | BPF_JSET	bpf_jset dst, imm, +off		PC += off if dst & imm
0x4d		BPF_JMP | BPF_X | BPF_JSET	bpf_jset dst, src, +off		PC += off if dst & src
0x55		BPF_JMP | BPF_K | BPF_JNE	bpf_jne dst, imm, +off		PC += off if dst != imm
0x5d		BPF_JMP | BPF_X | BPF_JNE	bpf_jne dst, src, +off		PC += off if dst != src
0x65		BPF_JMP | BPF_K | BPF_JSGT	bpf_jsgt dst, imm, +off		PC += off if dst > imm (signed)
0x6d		BPF_JMP | BPF_X | BPF_JSGT	bpf_jsgt dst, src, +off		PC += off if dst > src (signed)
0x75		BPF_JMP | BPF_K | BPF_JSGE	bpf_jsge dst, imm, +off		PC += off if dst >= imm (signed)
0x7d		BPF_JMP | BPF_X | BPF_JSGE	bpf_jsge dst, src, +off		PC += off if dst >= src (signed)
0x85		BPF_JMP | BPF_K | BPF_CALL	bpf_call imm			Function call
0x95		BPF_JMP | BPF_K | BPF_EXIT	bpf_exit			return r0
0xa5		BPF_JMP | BPF_K | BPF_JLT	bpf_jlt dst, imm, +off		PC += off if dst < imm
0xad		BPF_JMP | BPF_X | BPF_JLT	bpf_jlt dst, src, +off		PC += off if dst < src
0xb5		BPF_JMP | BPF_K | BPF_JLE	bpf_jle dst, imm, +off		PC += off if dst <= imm
0xbd		BPF_JMP | BPF_X | BPF_JLE	bpf_jle dst, src, +off		PC += off if dst <= src
0xc5		BPF_JMP | BPF_K | BPF_JSLT	bpf_jslt dst, imm, +off		PC += off if dst < imm (signed)
0xcd		BPF_JMP | BPF_X | BPF_JSLT	bpf_jslt dst, src, +off		PC += off if dst < src (signed)
0xd5		BPF_JMP | BPF_K | BPF_JSLE	bpf_jsle dst, imm, +off		PC += off if dst <= imm (signed)
0xdd		BPF_JMP | BPF_X | BPF_JSLE	bpf_jsle dst, src, +off		PC += off if dst <= src (signed)

BPF_JMP32 (opc & 0x07 == 0x06)
==============================

0x16		BPF_JMP32 | BPF_K | BPF_JEQ	bpf_jeq32 dst, imm, +off	PC += off if dst32 == imm
0x1e		BPF_JMP32 | BPF_X | BPF_JEQ	bpf_jeq32 dst, src, +off	PC += off if dst32 == src32
0x26		BPF_JMP32 | BPF_K | BPF_JGT	bpf_jgt32 dst, imm, +off	PC += off if dst32 > imm
0x2e		BPF_JMP32 | BPF_X | BPF_JGT	bpf_jgt32 dst, src, +off	PC += off if dst32 > src32
0x36		BPF_JMP32 | BPF_K | BPF_JGE	bpf_jge32 dst, imm, +off	PC += off if dst32 >= imm
0x3e		BPF_JMP32 | BPF_X | BPF_JGE	bpf_jge32 dst, src, +off	PC += off if dst32 >= src32
0x46		BPF_JMP32 | BPF_K | BPF_JSET	bpf_jset32 dst, imm, +off	PC += off if dst32 & imm
0x4e		BPF_JMP32 | BPF_X | BPF_JSET	bpf_jset32 dst, src, +off	PC += off if dst32 & src32
0x56		BPF_JMP32 | BPF_K | BPF_JNE	bpf_jne32 dst, imm, +off	PC += off if dst32 != imm
0x5e		BPF_JMP32 | BPF_X | BPF_JNE	bpf_jne32 dst, src, +off	PC += off if dst32 != src32
0x66		BPF_JMP32 | BPF_K | BPF_JSGT	bpf_jsgt32 dst, imm, +off	PC += off if dst32 > imm (signed)
0x6e		BPF_JMP32 | BPF_X | BPF_JSGT	bpf_jsgt32 dst, src, +off	PC += off if dst32 > src32 (signed)
0x76		BPF_JMP32 | BPF_K | BPF_JSGE	bpf_jsge32 dst, imm, +off	PC += off if dst32 >= imm (signed)
0x7e		BPF_JMP32 | BPF_X | BPF_JSGE	bpf_jsge32 dst, src, +off	PC += off if dst32 >= src32 (signed)
0xa6		BPF_JMP32 | BPF_K | BPF_JLT	bpf_jlt32 dst, imm, +off	PC += off if dst32 < imm
0xae		BPF_JMP32 | BPF_X | BPF_JLT	bpf_jlt32 dst, src, +off	PC += off if dst32 < src32
0xb6		BPF_JMP32 | BPF_K | BPF_JLE	bpf_jle32 dst, imm, +off	PC += off if dst32 <= imm
0xbe		BPF_JMP32 | BPF_X | BPF_JLE	bpf_jle32 dst, src, +off	PC += off if dst32 <= src32
0xc6		BPF_JMP32 | BPF_K | BPF_JSLT	bpf_jslt32 dst, imm, +off	PC += off if dst32 < imm (signed)
0xce		BPF_JMP32 | BPF_X | BPF_JSLT	bpf_jslt32 dst, src, +off	PC += off if dst32 < src32 (signed)
0xd6		BPF_JMP32 | BPF_K | BPF_JSLE	bpf_jsle32 dst, imm, +off	PC += off if dst32 <= imm (signed)
0xde		BPF_JMP32 | BPF_X | BPF_JSLE	bpf_jsle32 dst, src, +off	PC += off if dst32 <= src32 (signed)

BPF_LD (opc & 0x07 == 0x00)
===========================
0x18 BPF_LD | BPF_DW | BPF_IMM bpf_lddw dst, imm64 dst = imm64
Note: imm64 expressed in 8 bytes following instruction

BPF_LDX (opc & 0x07 == 0x01)
============================

0x61		BPF_LDX | BPF_W | BPF_MEM	bpf_ldxw dst, [src + off]	dst32 = *(u32 *)(src + off)
0x69		BPF_LDX | BPF_H | BPF_MEM	bpf_ldxh dst, [src + off]	dst16 = *(u16 *)(src + off)
0x71		BPF_LDX | BPF_B | BPF_MEM	bpf_ldxb dst, [src + off]	dst8 = *(u8 *)(src + off)
0x79		BPF_LDX | BPF_DW | BPF_MEM	bpf_ldxdw dst, [src + off]	dst = *(u64 *)(src + off)

BPF_ST (opc & 0x07 == 0x02)
===========================

0x62		BPF_ST | BPF_W | BPF_IMM	bpf_stw [dst + off], imm	*(u32 *)(dst + off) = imm
0x6a		BPF_ST | BPF_H | BPF_IMM	bpf_sth [dst + off], imm	*(u16 *)(dst + off) = imm
0x72		BPF_ST | BPF_B | BPF_IMM	bpf_stb [dst + off], imm	*(u8 *)(dst + off) = imm
0x7a		BPF_ST | BPF_DW | BPF_IMM	bpf_stdw [dst + off], imm	*(u64 *)(dst + off) = imm

BPF_STX (opc & 0x07 == 0x03)
============================

0x63		BPF_STX | BPF_W | BPF_MEM	bpf_stxw [dst + off], src	*(u32 *)(dst + off) = src32
0x6b		BPF_STX | BPF_H | BPF_MEM	bpf_stxh [dst + off], src	*(u16 *)(dst + off) = src16
0x73		BPF_STX | BPF_B | BPF_MEM	bpf_stxb [dst + off], src	*(u8 *)(dst + off) = src8
0x7b		BPF_STX | BPF_DW | BPF_MEM	bpf_stxdw [dst + off], src	*(u64 *)(dst + off) = src
0xdb		BPF_STX | BPF_DW | BPF_ATOMIC	64-bit atomic instructions (see below)
0xc3		BPF_STX | BPF_W | BPF_ATOMIC	32-bit atomic instructions (see below)

Note: mnemonic for atomic instructions? for example, eBPF originally had only XADD atomic instruction which could be represented as bpf_xadd. But with addition of atomic operations for AND, OR, XOR - that same pattern no longer works. So for now, just show pseudocode for each atomic operation.

64-bit atomic instructions (opc == 0xdb)
========================================
The following table applies to opc 0xdb (BPF_STX | BPF_DW | BPF_ATOMIC) which are 64-bit atomic operations. The imm (immediate) value in column 1 specifies the type of atomic operation.

0x00 BPF_ADD lock *(u64 *)(dst + off) += src
0x01 BPF_ADD | BPF_FETCH src = atomic_fetch_add((u64 *)(dst + off), src)
0x40 BPF_OR lock *(u64 *)(dst + off) |= src
0x41 BPF_OR | BPF_FETCH src = atomic_fetch_or((u64 *)(dst + off), src)
0x50 BPF_AND lock *(u64 *)(dst + off) &= src
0x51 BPF_AND | BPF_FETCH src = atomic_fetch_and((u64 *)(dst + off), src)
0xa0 BPF_XOR lock *(u64 *)(dst + off) ^= src
0xa1 BPF_XOR | BPF_FETCH src = atomic_fetch_xor((u64 *)(dst + off), src)
0xe1 BPF_XCHG | BPF_FETCH src = xchg((u64 *)(dst + off), src)
0xf1 BPF_CMPXCHG | BPF_FETCH r0 = cmpxchg((u64 *)(dst + off), r0, src)

32-bit atomic instructions (opc == 0xc3)
========================================
The following table applies to opc 0xc3 (BPF_STX | BPF_W | BPF_ATOMIC) which are 32-bit atomic operations. The imm (immediate) value in column 1 specifies the type of atomic operation.

0x00 BPF_ADD lock *(u32 *)(dst + off) += src32
0x01 BPF_ADD | BPF_FETCH src32 = atomic_fetch_add((u32 *)(dst + off), src32)
0x40 BPF_OR lock *(u32 *)(dst + off) |= src32
0x41 BPF_OR | BPF_FETCH src32 = atomic_fetch_or((u32 *)(dst + off), src32)
0x50 BPF_AND lock *(u32 *)(dst + off) &= src32
0x51 BPF_AND | BPF_FETCH src32 = atomic_fetch_and((u32 *)(dst + off), src32)
0xa0 BPF_XOR lock *(u32 *)(dst + off) ^= src32
0xa1 BPF_XOR | BPF_FETCH src32 = atomic_fetch_xor((u32 *)(dst + off), src32)
0xe1 BPF_XCHG | BPF_FETCH src32 = xchg((u32 *)(dst + off), src32)
0xf1 BPF_CMPXCHG | BPF_FETCH r0 = cmpxchg((u32 *)(dst + off), r0, src32)