Skip to contentSkip to conversation

Standardizing BPF assembly language?

DISCUSSION [email protected] 4 participants 8 messages Jan 23, 2024 – Jan 27, 2024 BPF

lore
Share & tools
lore
#0 lore raw

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

During Jose's talk, I discovered that uBPF didn't quote match the second dialect
and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?
One possible answer would be the ISA document that specifies the instructions,
since that would the IANA registry could list the assembly for each instruction,
and any future documents that add instructions would necessarily need to specify
the assembly for them, preventing variants from springing up for new instructions.

A second question would be, which dialect(s) to standardize. Jose's link above
argues that the second dialect should be the one standardized (tools are free to
support multiple dialects for backwards compat if they want). See the link for
rationale.

Thoughts?

Dave

#1 lore raw
Quoted 28 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?

Quoted 5 lines

One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.

I'm not opposed to this, but would strongly prefer that we do it as an extension if we go this route to avoid scope creep for the first iteration.

Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were going to continue to support both. If we wanted to standardize, I have a hard time seeing any other way other than to standardize both dialects unless there's been a significant change in sentiment since LSFMM.

#2 lore raw
Quoted 7 lines

-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?

Quoted 25 lines
Quoted 23 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's

standardized.

Quoted 5 lines
Quoted 4 lines

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?

Quoted 7 lines
Quoted 5 lines

One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.

I'm not opposed to this, but would strongly prefer that we do it as an

extension

Quoted 1 line

if we go this route to avoid scope creep for the first iteration.

If the first iteration does not have it, then presumably the initial IANA registry
would not have it either, since this iteration creates the registry and the rules for it.

That's doable, but may continue to proliferate more and more variants until it is addressed.

If it's in another document, do you agree it would still fall under the existing
charter bullet about "defining the instructions"

Quoted 2 lines

[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,

?

Quoted 6 lines
Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were

going

Quoted 1 line

to continue to support both. If we wanted to standardize, I have a hard

time

Quoted 1 line

seeing any other way other than to standardize both dialects unless

there's

Quoted 1 line

been a significant change in sentiment since LSFMM.

If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?

Dave

#3 lore raw
Quoted 46 lines · On Tue, Jan 23, 2024 at 01:41:10PM -0800, [email protected] wrote:
Quoted 7 lines

-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?

Quoted 25 lines
Quoted 23 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's

standardized.

Quoted 5 lines
Quoted 4 lines

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?

Quoted 7 lines
Quoted 5 lines

One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.

I'm not opposed to this, but would strongly prefer that we do it as an

extension

Quoted 8 lines
Quoted 1 line

if we go this route to avoid scope creep for the first iteration.

If the first iteration does not have it, then presumably the initial IANA registry would not have it either, since this iteration creates the registry and the rules for it.

That's doable, but may continue to proliferate more and more variants until it is addressed.

The same could be said for any new instructions that are added while we sort out standardizing the assembly language as well, no?

Quoted 5 lines

If it's in another document, do you agree it would still fall under the existing charter bullet about "defining the instructions"

Quoted 2 lines

[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,

?

I wouldn't say it's illogical to group assembly language in this bucket, but I would say that defining the assembly language does not need to be tied at the hip with defining instruction encodings and semantics. So my answer is "yes, I think it belongs here", but I also don't think it's necessary or desirable for the first iteration.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were

going

Quoted 1 line

to continue to support both. If we wanted to standardize, I have a hard

time

Quoted 8 lines
Quoted 1 line

seeing any other way other than to standardize both dialects unless

there's

Quoted 1 line

been a significant change in sentiment since LSFMM.

If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?

Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects. Practically speaking that's already what's happening, no? Both dialects are already pervasive, so it seems unlikely that a tool would succeed without supporting both regardless. To Jose's point (pasted below), there are of course drawbacks:

Quoted 4 lines

- Expensive :: it makes it very difficult to reuse infrastructure.
- Problematic :: dis/assemblers, CGEN, LaTeX, editors, IDEs, etc.
- Ambiguous :: with both GAS and llvm/MCParser: symbol assignments.
- Pervasive :: because of the inline asm.

I think it would be a lot simpler to standardize on only a single dialect, but I also think the standard should reflect how BPF is being used in practice.

#4 lore raw
Quoted 7 lines

-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:52 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?

Quoted 13 lines
Quoted 11 lines · On Tue, Jan 23, 2024 at 01:41:10PM -0800, [email protected] wrote:
Quoted 11 lines

-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?

Quoted 2 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language

(http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Quoted 39 lines
Quoted 39 lines
Quoted 14 lines
Quoted 14 lines

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

Quoted 7 lines
Quoted 7 lines

During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's

standardized.

Quoted 6 lines
Quoted 5 lines

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?

Quoted 8 lines
Quoted 5 lines

One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.

I'm not opposed to this, but would strongly prefer that we do it as an

extension

Quoted 1 line

if we go this route to avoid scope creep for the first iteration.

Quoted 8 lines
Quoted 6 lines

If the first iteration does not have it, then presumably the initial IANA registry would not have it either, since this iteration creates the registry and the rules for it.

That's doable, but may continue to proliferate more and more variants until it is addressed.

The same could be said for any new instructions that are added while we

sort

Quoted 1 line

out standardizing the assembly language as well, no?

Yes, that was my point. If the initial ISA spec at time of publication includes the
assembly language then there's no issue.

Not saying we have to wait, just that this which document to put it in is what the WG should agree on in my view.

Quoted 7 lines
Quoted 5 lines

If it's in another document, do you agree it would still fall under the existing charter bullet about "defining the instructions"

Quoted 2 lines

[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,

?

I wouldn't say it's illogical to group assembly language in this bucket,

but I

Quoted 1 line

would say that defining the assembly language does not need to be tied at

the

Quoted 1 line

hip with defining instruction encodings and semantics. So my answer is

"yes, I

Quoted 1 line

think it belongs here", but I also don't think it's necessary or desirable

for the

Quoted 15 lines

first iteration.

Quoted 13 lines
Quoted 7 lines
Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were

going

Quoted 2 lines

to continue to support both. If we wanted to standardize, I have a hard

time

Quoted 1 line

seeing any other way other than to standardize both dialects unless

there's

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

been a significant change in sentiment since LSFMM.

If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document adding any new instructions to specify the assembly in both dialects?

Well, if we're standardizing on both, then yes I think it would be

mandatory for

Quoted 1 line

a tool to support both, and I think instructions would require assembly

for both

Quoted 1 line

dialects. Practically speaking that's already what's happening, no? Both

dialects

Quoted 1 line

are already pervasive, so it seems unlikely that a tool would succeed

without

Quoted 1 line

supporting both regardless.

There's plenty of counter examples of things that exist (whether they "succeed"
or not depends on the definition of succeed) that support or supported neither.
E.g., uBPF prior to Jose's talk.

Quoted 8 lines

To Jose's point (pasted below), there are of course drawbacks:

Quoted 4 lines

- Expensive :: it makes it very difficult to reuse infrastructure.
- Problematic :: dis/assemblers, CGEN, LaTeX, editors, IDEs, etc.
- Ambiguous :: with both GAS and llvm/MCParser: symbol assignments.
- Pervasive :: because of the inline asm.

I think it would be a lot simpler to standardize on only a single dialect,

but I also

Quoted 1 line

think the standard should reflect how BPF is being used in practice.

Dave

#5 lore raw
In reply to David Vernet
Quoted 13 lines · David Vernet
Quoted 13 lines
Quoted 6 lines
Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were

going

Quoted 1 line

to continue to support both. If we wanted to standardize, I have a hard

time

Quoted 1 line

seeing any other way other than to standardize both dialects unless

there's

Quoted 1 line

been a significant change in sentiment since LSFMM.

Quoted 8 lines
Quoted 4 lines

If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?

Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects.

I think it's obvious that there is no way we will add gcc's flavor of asm to kernel and llvm.

Quoted 2 lines

Practically speaking that's already
what's happening, no? Both dialects are already pervasive,

They are not. There are thousands of lines of asm written in pseudo-c used in production applications and probably only ubpf/tests and gcc/tests in that other asm, since gcc bpf support is not yet in the released gcc version.

There is also this asm flavor:
https://github.com/Xilinx-CNS/ebpf_asm

Which is different from pseudo-c and ubpf asm.

I don't think asm syntax should be an IETF draft.

#7 lore raw
In reply to [email protected]
Quoted 46 lines · On Tue, Jan 23, 2024 at 8:46 AM <[email protected]>…

At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).

Jose wrote in that link:

Quoted 11 lines

There are two dialects of BPF assembler in use today:

- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 + 0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3

- An "assembler-like" dialect
: ldxdw %r1, [%r2 + 0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3

During Jose's talk, I discovered that uBPF didn't quote match the second dialect
and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.

As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.

Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?
One possible answer would be the ISA document that specifies the instructions,
since that would the IANA registry could list the assembly for each instruction,
and any future documents that add instructions would necessarily need to specify
the assembly for them, preventing variants from springing up for new instructions.

A second question would be, which dialect(s) to standardize. Jose's link above
argues that the second dialect should be the one standardized (tools are free to
support multiple dialects for backwards compat if they want). See the link for
rationale.

Quoted 1 line

Thoughts?

Someone from Bell will go off and invent their own variation no matter what we do. Snark aside I think it's useful for documents to be able to contain excepts of code without needing to expect readers to decode BPF from hex in their heads, and we should write down that format somewhere.

Sincerely,
Watson

--
Astra mortemque praestare gradatim

#6 lore raw
In reply to Alexei Starovoitov
Quoted 19 lines · Alexei Starovoitov
Quoted 18 lines · David Vernet
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines

A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.

My recollection was that the outcome of that discussion is that we were

going

Quoted 1 line

to continue to support both. If we wanted to standardize, I have a hard

time

Quoted 1 line

seeing any other way other than to standardize both dialects unless

there's

Quoted 1 line

been a significant change in sentiment since LSFMM.

If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?

Quoted 6 lines
Quoted 3 lines

Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects.

I think it's obvious that there is no way we will add gcc's flavor of asm to kernel and llvm.

Well, it will depend on how widely it's used. Or if it's used widely :-)

Quoted 7 lines
Quoted 2 lines

Practically speaking that's already
what's happening, no? Both dialects are already pervasive,

They are not. There are thousands of lines of asm written in pseudo-c used in production applications and probably only ubpf/tests and gcc/tests in that other asm, since gcc bpf support is not yet in the released gcc version.

Quoted 6 lines

There is also this asm flavor:
https://github.com/Xilinx-CNS/ebpf_asm

Which is different from pseudo-c and ubpf asm.

I don't think asm syntax should be an IETF draft.

Ok, fair enough. Another thought that occurred to me after thinking about this more -- even if we want source code compatibility (which is still an open question), that doesn't necessarily imply or require assembly dialect compatibility. Let's table this for now, and revisit another day, should we ever find that it makes sense to do so.

Thanks,
David