Standardizing BPF assembly language?
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3
During Jose's talk, I discovered that uBPF didn't quote match the second dialect
and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.
As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?
One possible answer would be the ISA document that specifies the instructions,
since that would the IANA registry could list the assembly for each instruction,
and any future documents that add instructions would necessarily need to specify
the assembly for them, preventing variants from springing up for new instructions.
A second question would be, which dialect(s) to standardize. Jose's link above
argues that the second dialect should be the one standardized (tools are free to
support multiple dialects for backwards compat if they want). See the link for
rationale.
Thoughts?
Dave
Quoted 28 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.
As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?
Quoted 5 lines
One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.
I'm not opposed to this, but would strongly prefer that we do it as an extension if we go this route to avoid scope creep for the first iteration.
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were going to continue to support both. If we wanted to standardize, I have a hard time seeing any other way other than to standardize both dialects unless there's been a significant change in sentiment since LSFMM.
Quoted 7 lines
-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?
Quoted 25 lines
Quoted 23 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.
As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's
standardized.
Quoted 5 lines
Quoted 4 lines
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?
Quoted 7 lines
Quoted 5 lines
One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.
I'm not opposed to this, but would strongly prefer that we do it as an
extension
Quoted 1 line
if we go this route to avoid scope creep for the first iteration.
If the first iteration does not have it, then presumably the initial IANA registry
would not have it either, since this iteration creates the registry and the rules for it.
That's doable, but may continue to proliferate more and more variants until it is addressed.
If it's in another document, do you agree it would still fall under the existing
charter bullet about "defining the instructions"
Quoted 2 lines
[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,
?
Quoted 6 lines
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were
going
Quoted 1 line
to continue to support both. If we wanted to standardize, I have a hard
time
Quoted 1 line
seeing any other way other than to standardize both dialects unless
there's
Quoted 1 line
been a significant change in sentiment since LSFMM.
If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?
Dave
Quoted 46 lines · On Tue, Jan 23, 2024 at 01:41:10PM -0800, [email protected] wrote:
Quoted 7 lines
-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?Quoted 25 lines
Quoted 23 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.
As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's
standardized.
Quoted 5 lines
Quoted 4 lines
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?Quoted 7 lines
Quoted 5 lines
One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.
I'm not opposed to this, but would strongly prefer that we do it as an
extension
Quoted 8 lines
Quoted 1 line
if we go this route to avoid scope creep for the first iteration.
If the first iteration does not have it, then presumably the initial IANA registry would not have it either, since this iteration creates the registry and the rules for it.
That's doable, but may continue to proliferate more and more variants until it is addressed.
The same could be said for any new instructions that are added while we sort out standardizing the assembly language as well, no?
Quoted 5 lines
If it's in another document, do you agree it would still fall under the existing charter bullet about "defining the instructions"
Quoted 2 lines
[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,
?
I wouldn't say it's illogical to group assembly language in this bucket, but I would say that defining the assembly language does not need to be tied at the hip with defining instruction encodings and semantics. So my answer is "yes, I think it belongs here", but I also don't think it's necessary or desirable for the first iteration.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were
going
Quoted 1 line
to continue to support both. If we wanted to standardize, I have a hard
time
Quoted 8 lines
Quoted 1 line
seeing any other way other than to standardize both dialects unless
there's
Quoted 1 line
been a significant change in sentiment since LSFMM.
If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?
Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects. Practically speaking that's already what's happening, no? Both dialects are already pervasive, so it seems unlikely that a tool would succeed without supporting both regardless. To Jose's point (pasted below), there are of course drawbacks:
Quoted 4 lines
- Expensive :: it makes it very difficult to reuse infrastructure.
- Problematic :: dis/assemblers, CGEN, LaTeX, editors, IDEs, etc.
- Ambiguous :: with both GAS and llvm/MCParser: symbol assignments.
- Pervasive :: because of the inline asm.
I think it would be a lot simpler to standardize on only a single dialect, but I also think the standard should reflect how BPF is being used in practice.
Quoted 7 lines
-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:52 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?
Quoted 13 lines
Quoted 11 lines · On Tue, Jan 23, 2024 at 01:41:10PM -0800, [email protected] wrote:
Quoted 11 lines
-----Original Message-----
From: David Vernet <[email protected]>
Sent: Tuesday, January 23, 2024 1:31 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Bpf] Standardizing BPF assembly language?Quoted 2 lines · On Tue, Jan 23, 2024 at 08:45:32AM -0800…
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language
(http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Quoted 39 lines
Quoted 39 lines
Quoted 14 lines
Quoted 14 lines
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3Quoted 7 lines
Quoted 7 lines
During Jose's talk, I discovered that uBPF didn't quote match the second dialect and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.
As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's
standardized.
Quoted 6 lines
Quoted 5 lines
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is
then: which document?Quoted 8 lines
Quoted 5 lines
One possible answer would be the ISA document that specifies the instructions, since that would the IANA registry could list the assembly for each instruction, and any future documents that add instructions would necessarily need to specify the assembly for them, preventing variants from springing up for new instructions.
I'm not opposed to this, but would strongly prefer that we do it as an
extension
Quoted 1 line
if we go this route to avoid scope creep for the first iteration.
Quoted 8 lines
Quoted 6 lines
If the first iteration does not have it, then presumably the initial IANA registry would not have it either, since this iteration creates the registry and the rules for it.
That's doable, but may continue to proliferate more and more variants until it is addressed.
The same could be said for any new instructions that are added while we
sort
Quoted 1 line
out standardizing the assembly language as well, no?
Yes, that was my point. If the initial ISA spec at time of publication includes the
assembly language then there's no issue.
Not saying we have to wait, just that this which document to put it in is what the WG should agree on in my view.
Quoted 7 lines
Quoted 5 lines
If it's in another document, do you agree it would still fall under the existing charter bullet about "defining the instructions"
Quoted 2 lines
[PS] the BPF instruction set architecture (ISA) that defines the instructions and low-level virtual machine for BPF programs,
?
I wouldn't say it's illogical to group assembly language in this bucket,
but I
Quoted 1 line
would say that defining the assembly language does not need to be tied at
the
Quoted 1 line
hip with defining instruction encodings and semantics. So my answer is
"yes, I
Quoted 1 line
think it belongs here", but I also don't think it's necessary or desirable
for the
Quoted 15 lines
first iteration.
Quoted 13 lines
Quoted 7 lines
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were
going
Quoted 2 lines
to continue to support both. If we wanted to standardize, I have a hard
time
Quoted 1 line
seeing any other way other than to standardize both dialects unless
there's
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
been a significant change in sentiment since LSFMM.
If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document adding any new instructions to specify the assembly in both dialects?
Well, if we're standardizing on both, then yes I think it would be
mandatory for
Quoted 1 line
a tool to support both, and I think instructions would require assembly
for both
Quoted 1 line
dialects. Practically speaking that's already what's happening, no? Both
dialects
Quoted 1 line
are already pervasive, so it seems unlikely that a tool would succeed
without
Quoted 1 line
supporting both regardless.
There's plenty of counter examples of things that exist (whether they "succeed"
or not depends on the definition of succeed) that support or supported neither.
E.g., uBPF prior to Jose's talk.
Quoted 8 lines
To Jose's point (pasted below), there are of course drawbacks:
Quoted 4 lines
- Expensive :: it makes it very difficult to reuse infrastructure.
- Problematic :: dis/assemblers, CGEN, LaTeX, editors, IDEs, etc.
- Ambiguous :: with both GAS and llvm/MCParser: symbol assignments.
- Pervasive :: because of the inline asm.I think it would be a lot simpler to standardize on only a single dialect,
but I also
Quoted 1 line
think the standard should reflect how BPF is being used in practice.
Dave
Quoted 13 lines · David Vernet
Quoted 13 lines
Quoted 6 lines
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were
going
Quoted 1 line
to continue to support both. If we wanted to standardize, I have a hard
time
Quoted 1 line
seeing any other way other than to standardize both dialects unless
there's
Quoted 1 line
been a significant change in sentiment since LSFMM.
Quoted 8 lines
Quoted 4 lines
If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects.
I think it's obvious that there is no way we will add gcc's flavor of asm to kernel and llvm.
Quoted 2 lines
Practically speaking that's already
what's happening, no? Both dialects are already pervasive,
They are not. There are thousands of lines of asm written in pseudo-c used in production applications and probably only ubpf/tests and gcc/tests in that other asm, since gcc bpf support is not yet in the released gcc version.
There is also this asm flavor:
https://github.com/Xilinx-CNS/ebpf_asm
Which is different from pseudo-c and ubpf asm.
I don't think asm syntax should be an IETF draft.
Quoted 46 lines · On Tue, Jan 23, 2024 at 8:46 AM <[email protected]>…
At LSF/MM/BPF 2023, Jose gave a presentation about BPF assembly language (http://vger.kernel.org/bpfconf2023_material/compiled_bpf.txt).
Jose wrote in that link:
Quoted 11 lines
There are two dialects of BPF assembler in use today:
- A "pseudo-c" dialect (originally "BPF verifier format") : r1 = *(u64 *)(r2 +
0x00f0)
: if r1 > 2 goto label
: lock *(u32 *)(r2 + 10) += r3- An "assembler-like" dialect
: ldxdw %r1, [%r2 +0x00f0]
: jgt %r1, 2, label
: xaddw [%r2 + 2], r3During Jose's talk, I discovered that uBPF didn't quote match the second dialect
and submitted a bug report. By the time the conference was over, uBPF had been updated to match GCC, so that discussion worked to reduce the number of variants.As more instructions get added and supported by more tools and compilers there's the risk of even more variants unless it's standardized.
Hence I'd recommend that BPF assembly language get documented in some WG draft. If folks agree with that premise, the first question is then: which document?
One possible answer would be the ISA document that specifies the instructions,
since that would the IANA registry could list the assembly for each instruction,
and any future documents that add instructions would necessarily need to specify
the assembly for them, preventing variants from springing up for new instructions.A second question would be, which dialect(s) to standardize. Jose's link above
argues that the second dialect should be the one standardized (tools are free to
support multiple dialects for backwards compat if they want). See the link for
rationale.
Quoted 1 line
Thoughts?
Someone from Bell will go off and invent their own variation no matter what we do. Snark aside I think it's useful for documents to be able to contain excepts of code without needing to expect readers to decode BPF from hex in their heads, and we should write down that format somewhere.
Sincerely,
Watson
--
Astra mortemque praestare gradatim
Quoted 19 lines · Alexei Starovoitov
Quoted 18 lines · David Vernet
Quoted 17 lines
Quoted 6 lines
Quoted 4 lines
A second question would be, which dialect(s) to standardize. Jose's link above argues that the second dialect should be the one standardized (tools are free to support multiple dialects for backwards compat if they want). See the link for rationale.
My recollection was that the outcome of that discussion is that we were
going
Quoted 1 line
to continue to support both. If we wanted to standardize, I have a hard
time
Quoted 1 line
seeing any other way other than to standardize both dialects unless
there's
Quoted 1 line
been a significant change in sentiment since LSFMM.
If "standardize both", does that mean neither is mandatory and each tool is free to pick one or the other? And would the IANA registry require a document
adding any new instructions to specify the assembly in both dialects?
Quoted 6 lines
Quoted 3 lines
Well, if we're standardizing on both, then yes I think it would be mandatory for a tool to support both, and I think instructions would require assembly for both dialects.
I think it's obvious that there is no way we will add gcc's flavor of asm to kernel and llvm.
Well, it will depend on how widely it's used. Or if it's used widely :-)
Quoted 7 lines
Quoted 2 lines
Practically speaking that's already
what's happening, no? Both dialects are already pervasive,They are not. There are thousands of lines of asm written in pseudo-c used in production applications and probably only ubpf/tests and gcc/tests in that other asm, since gcc bpf support is not yet in the released gcc version.
Quoted 6 lines
There is also this asm flavor:
https://github.com/Xilinx-CNS/ebpf_asmWhich is different from pseudo-c and ubpf asm.
I don't think asm syntax should be an IETF draft.
Ok, fair enough. Another thought that occurred to me after thinking about this more -- even if we want source code compatibility (which is still an open question), that doesn't necessarily imply or require assembly dialect compatibility. Let's table this for now, and revisit another day, should we ever find that it makes sense to do so.
Thanks,
David