Introducing P4TC (series 1)
Subsystems (6)
Jamal Hadi Salim
maintainerv16 cover Introducing P4TC (series 1)
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
Please, if you want to discuss a slightly tangential subject like offload or even your politics then start another thread with a different subject line. The way you do it is to change the subject line to for example "<Your New Subject Here> (WAS: <original subject line here>)".
In this cover letter i am restoring text i took out in V10 which stated "our requirements".
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
1) P4TC uses the TC model - therefore the design is centred around TC filters, actions etc. It means a unified TC control via netlink for s/w + h/w twins. It means the P4 objects(tables, actions, externs, etc) and associated data are owned by P4TC. None of the other "innovations" that are divorced from TC such as tcx make any sense to solving the engineering problem at stake. And therefore the argument that "tc actions and filters are a mistake or inferior and you have to use what we innovated" is a non-starter and both arrogant and condescending. We use eBPF as an infra tool not as the answer looking for a question. Sorry.
2) We use kfuncs to access the P4 objects for the s/w datapath. AFAIK, kfuncs contributions do not have to be sent to the ebpf mailing list for review or approval. Infact, kfuncs can be implemented in a kernel module and do not need to be upstreamed. But it is "encouraged to upstream for sharing reasons". So somehow picking when you want to move the goal posts for political nack purposes is abuse of power. For our work there are certain features that need to be upstreamed so the community can have full access to say the P4 PNA architecture and not need to install oot kernel modules.
For this reason, we are need to push the kfuncs as part of the series. It does not make sense to make them oot.
3) Just a reminder: This code is entirely in the TC domain and does not make any changes to ebpf code. So for people from the ebpf domain who do not maintain the TC code to step in and nack TC patches is most certainly overstepping.
__Description of these Patches__
These Patches are constrained entirely within the TC domain with very tiny changes made to TC core code in patch 1-5. eBPF is used as an infrastructure component for the software datapath and no changes are made to any eBPF code, only kfuncs are introduced in patch 14.
Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect on the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions needed for P4.
The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.
There are a few more patches not in this patchset that deal with externs, test cases, etc.
What is P4?
-----------
The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
A lot more on why P4 - see small treatise here:[4].
What is P4TC?
-------------
P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure.
IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware acquisition(example [12]).
Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC requires zero kernel or user space changes. We refer to this as:_kernel and user space code change independence_.
Meaning:
A P4 program describes headers, how to parse, etc alongside prescribing the datapath processing logic; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".
Some History
------------
There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
__P4TC Architecture__
The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch descriptions...
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TC Workflow__
The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above
b) The parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 \
p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"
See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TC Runtime Control Path__
The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+) Read(</path/to/object>, [optional filter]) Update(</path/to/object>, DATA>+) Delete(</path/to/object>, [optional filter]) Subscribe(</path/to/object>, [optional filter])
Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted.
"Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1
tc p4ctrl create myprog/table/mytable \
dstAddr 10.0.1.2/32 action send_to_port param port eno1
..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 action send_to_port param port eno1 \
entry dstAddr 10.1.10.2/32 action send_to_port param port eno10 \
entry dstAddr 10.0.2.2/32 action send_to_port param port eno2
..Get an entry (note "read" is interchangeably used as "get" which is a common semantic in tc):
tc p4ctrl read myprog/table/mytable \
dstAddr 10.0.2.2/32
..dump mytable
tc p4ctrl read myprog/table/mytable
..dump mytable for all entries whose key fits within 10.1.0.0/16
tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16
..dump all mytable entries which have an action send_to_port with param "eno1"
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"
The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1" && \
key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filter msecs_since < 10000
Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types).
All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TC Datapath__
The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 (illustrated in section "P4TC Workflow" above will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address labeled "dstAddr" above in section "P4TC Runtime Control Path"). The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__
Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath). Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.
In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__Restating Our Requirements__
Given this code is not intrusive at all because it only touches TC. We would like to emphasize that we see eBPF as _infrastructure tooling available to us and not the end goal_. Please help us with technical input on for example how we can do better kfuncs, etc. If you want to critique, then our requirements should be your guide and please be considerate that this is about P4, not eBPF. IOW:
We would appreciate technical commentary instead of bikeshedding on how _you_ would have implemented this probably with more eBPF or some other clever tricks. It is sad to see there was zero input from anyone in the eBPF world for 7 RFC postings (in a period of 9 months).
If i am ranting here is because we have spent over a year now on this topic - we have taken the initial input and have given you eBPF. So lets make progress please.
The initial release was presented in October 2022[20] and RFC in January 2023 had a "scriptable" datapath (the idea built on the u32 classifier[17] and pedit action[18] approach. Post RFC V1, we made changes to fit the feedback to integrate eBPF to replace the "scriptable" software datapath. On our part, the goal for the change was to meet folks in the middle as a compromise.
No regrets on the journey since after all the effort because we ended getting XDP which was not in the original picture. Some of our efforts are captured at [1][3] and in the patch history.
In this section we review the original scriptable version against the current implementation which uses eBPF and in the process re-enumerate our requirements.
To be very clear: Our intention for P4TC is to target _the TC crowd_. Essentially developers and ops people already familiar and deploying TC based infra.
More importantly the original intent for P4TC was to enable _ops folks_ more than devs (given code is being generated and doesn't need humans to write it).
With TC, we gain the whole "familiar" package of match-action pipeline abstraction++, meaning from the control plane(see discussion above) all the way to the tooling infra, i.e iproute2/tc cli, netlink infra interface (request/response, event subscribe/multicast-publish, congestion control etc), s/w and h/w symbiosis, the autonomous kernel control, etc. The main advantage over vendor specific implementations(which is the current alternative) is: with P4TC we have a singular vendor-neutral interface via the kernel using well understood mechanisms that have gained learnings from deployment experience.
So lets list some of these requirements and compare whether moving to eBPF affected us or gave us an advantage.
0) Understood Control Plane semantics
This requirement is unaffected.
The control plane remains as netlink and therefore we get the classical multi-user CRUD+Publish/subscribe APIs built in.
1) Must support SW/HW equivalence
This requirement is unaffected. The control plane is netlink. Any semantics to select between sw and hw via skip_sw/hw semantics is maintained.
2) Supporting expressibility of the universe set of P4 progs
It is a must to support 100% of all possible P4 programs. In the past the eBPF verifier, for example in [13], had to be worked around and even then there are cases where we couldnt avoid path explosion when branching isi involved and failed to run. So we were skeptical about using eBPF to begin with.
Kfuncs changed our minds. Note, there are still challenges running all potential P4 programs at the XDP level - but the pipeline could be split between XDP and TC in such cases. The compiler can be told to generate pieces that run on XDP and other on TC (see examples).
Summary: This requirement is unaffected.
3) Operational usability
By maintaining the TC control plane (even in presence of eBPF datapath) runtime aspects remain unchanged. So for our target audience of folks who have deployed tc, including offloads, the comfort zone is unchanged.
There is some loss in operational usability because we now have more knobs:
the extra compilation, loading and syncing of ebpf binaries, etc. IOW, I can no longer just ship someone a shell script(ascii) in an email to someone and say "go run this and "myprog" will just work".
4) Operational and development Debuggability
If something goes wrong, the tc craftsperson is now required to have additional knowledge of eBPF code and process.
Our intent is to compensate this challenge with debug tools that ease the craftperson's debugging.
5) Opportunity for rapid prototyping of new ideas
This is not exactly a requirement but something that became a useful feature during the P4TC development phase. When the compiler was lagging behind in features was to often handcode the template scripts. Then you would dump back the template from the kernel and do a diff to ensure the kernel didn't get something wrong. Essentially, this was a nice debug feature. During development, we wrote scripts that covered a range of P4 architectures(PSA, V1, etc) which required no kernel code changes.
Over time the debug feature morphed into: a) start by handcoding scripts then b) read it back and then c) generate the P4 code.
It means one could start with the template scripts outside of the constraints of a P4 architecture spec(PNA/PSA) or even within a P4 architecture then test some ideas and eventually feed back the concepts to the compiler authors or modify or create a new P4 architecture and share with the P4 standards folks.
To summarize in presence of eBPF: The debugging idea is probably still alive. One could dump, with proper tooling(bpftool for example), the loaded eBPF code and be able to check for differences. But this is not the interesting part.
The concept of going back from whats in the kernel to P4 is a lot more difficult to implement mostly due to scoping of DSL vs general purpose. It may be lost. We have been discussing ways to use BTF and embedding annotations in the eBPF code and binary but more thought is required and we welcome suggestions.
6) Supporting per namespace program
In P4TC every program and its associated objects have unique IDs which are generated by the compiler. Multiple or the same P4 program(s) can run independently in different namespaces alongside their appropriate state and object instance parameterization (despite name or ID collission). This requirement is still met (by virtue of keeping P4 program control objects within the TC domain and attaching to a netns).
__References__
[1]https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf [2]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#historical-perspective-for-p4tc [3]https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation [4]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#so-why-p4-and-how-does-p4-help-here [5]https://lore.kernel.org/netdev/[email protected]/T/#mf59be7abc5df3473cff3879c8cc3e2369c0640a6 [6]https://lore.kernel.org/netdev/[email protected]/T/#m783cfd79e9d755cf0e7afc1a7d5404635a5b1919 [7]https://lore.kernel.org/netdev/[email protected]/T/#ma8c84df0f7043d17b98f3d67aab0f4904c600469 [8]https://github.com/p4lang/p4c/tree/main/backends/tc
[9]https://p4.org/ [10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
[12]https://github.com/sonic-net/DASH/tree/main
[13]https://github.com/p4lang/p4c/tree/main/backends/ebpf [14]https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html [15]https://dl.acm.org/doi/10.1145/3630047.3630193 [16]https://lore.kernel.org/netdev/[email protected]/ [17.a]https://netdevconf.info/0x13/session.html?talk-tc-u-classifier
[17.b]man tc-u32
[18]man tc-pedit
[19] https://lore.kernel.org/netdev/[email protected]/T/#m86e71743d1d83b728bb29d5b877797cb4942e835
[20.a] https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html
[20.b] https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html
--------
HISTORY
--------
Changes in Version 16
----------------------
1) Add Daniel's and John's Nack to patch 14
Changes in Version 15
----------------------
1) Add Alexei's Nack to patch 14
Changes in Version 14
----------------------
1) #UNDEF HWRITE/HREAD and remove unnecessary checks (Paolo)
2) Remove const cast added in v13 as a result of changes suggested suggested by Paolo (Marcelo)
3) Introduce type validate for s8 caught as a result of audit from #1
4) S/GFP_KERNEL/GFP_KERNEL_ACCOUNT for types and runtime objects (Paolo)
5) Syzkaller caught an invalid netlink attribute bug that has existed since v5! As noted in patch0 we've been running syzkaller for months.
6) Add Marcelo's reviewed-by for patch 14 and Toke's ACK to the series.
Changes in Version 13
----------------------
1) Remove ops->print() from p4 types (Paolo).
2) Use mutex instead of rwlock for dynamic actions since rwlock is discouraged these days(Paolo).
3) Constify action init_ops() ops parameter (Paolo).
4) Use struct sk_buff in kfunc instead of struct __sk_buff (Martin) Use struct xdp_buff in kfunc instead of struct xdp_md (Martin)
5) Replace BTF_SET8_START with BTF_KFUNCS_START and replace BTF_SET8_END with BTF_KFUNCS_END (Martin)
6) Add params__sz argument to all kfuncs to guard against future change to parameter structures being passed between bpf and tc. For kfunc xdp/bpf_p4tc_entry_create() we already had the max(5) allowed number of of parameters. To work around this we had to merge two structs together in order to maintain the number of params to 5 (Martin).
7) Add more info on commit log to explain the relation between the kfuncs and TC for patch #14 (Martin).
Changes in Version 12
----------------------
0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.
2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.
2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).
2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------
1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to p4tc_xxx for uapi and p4a_xxx for actions to provide consistency(Jiri).
4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------
1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------
0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------
1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------
1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------
1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------
These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------
Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).
Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".
This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.
The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via Kconfig).
Jamal Hadi Salim (15):
net: sched: act_api: Introduce P4 actions list
net/sched: act_api: increase action kind string length net/sched: act_api: Update tc_action_ops to account for P4 actions net/sched: act_api: add struct p4tc_action_ops as a parameter to
lookup callback
net: sched: act_api: Add support for preallocated P4 action instances
p4tc: add P4 data types
p4tc: add template API
p4tc: add template pipeline create, get, update, delete
p4tc: add template action create, update, delete, get, flush and dump
p4tc: add runtime action support
p4tc: add template table create, update, delete, get, flush and dump
p4tc: add runtime table entry create and update
p4tc: add runtime table entry get, delete, flush and dump
p4tc: add set of P4TC table kfuncs
p4tc: add P4 classifier
include/linux/bitops.h | 1 + include/net/act_api.h | 23 +- include/net/p4tc.h | 714 +++++++ include/net/p4tc_types.h | 89 + include/net/tc_act/p4tc.h | 79 + include/uapi/linux/p4tc.h | 465 +++++ include/uapi/linux/pkt_cls.h | 15 + include/uapi/linux/rtnetlink.h | 18 + include/uapi/linux/tc_act/tc_p4.h | 11 + net/sched/Kconfig | 23 + net/sched/Makefile | 3 + net/sched/act_api.c | 192 +- net/sched/cls_api.c | 2 +- net/sched/cls_p4.c | 305 +++ net/sched/p4tc/Makefile | 8 + net/sched/p4tc/p4tc_action.c | 2419 +++++++++++++++++++++++ net/sched/p4tc/p4tc_bpf.c | 360 ++++ net/sched/p4tc/p4tc_filter.c | 1012 ++++++++++ net/sched/p4tc/p4tc_pipeline.c | 700 +++++++ net/sched/p4tc/p4tc_runtime_api.c | 145 ++ net/sched/p4tc/p4tc_table.c | 1820 +++++++++++++++++ net/sched/p4tc/p4tc_tbl_entry.c | 3071 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 440 +++++ net/sched/p4tc/p4tc_types.c | 1213 ++++++++++++ net/sched/p4tc/trace.c | 10 + net/sched/p4tc/trace.h | 44 + security/selinux/nlmsgtab.c | 10 +- 27 files changed, 13156 insertions(+), 36 deletions(-) create mode 100644 include/net/p4tc.h create mode 100644 include/net/p4tc_types.h create mode 100644 include/net/tc_act/p4tc.h create mode 100644 include/uapi/linux/p4tc.h create mode 100644 include/uapi/linux/tc_act/tc_p4.h create mode 100644 net/sched/cls_p4.c create mode 100644 net/sched/p4tc/Makefile create mode 100644 net/sched/p4tc/p4tc_action.c create mode 100644 net/sched/p4tc/p4tc_bpf.c create mode 100644 net/sched/p4tc/p4tc_filter.c create mode 100644 net/sched/p4tc/p4tc_pipeline.c create mode 100644 net/sched/p4tc/p4tc_runtime_api.c create mode 100644 net/sched/p4tc/p4tc_table.c create mode 100644 net/sched/p4tc/p4tc_tbl_entry.c create mode 100644 net/sched/p4tc/p4tc_tmpl_api.c create mode 100644 net/sched/p4tc/p4tc_types.c create mode 100644 net/sched/p4tc/trace.c create mode 100644 net/sched/p4tc/trace.h
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 1/15 net: sched: act_api: Introduce P4 actions list
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)
include/net/act_api.h +6 -2
@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;
@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);
@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,
net/sched/act_api.c +109 -14
215 lines not on this page · raw message
net/sched/cls_api.c +1 -1
@@ -3367,7 +3367,7 @@ int tcf_exts_validate_ex(struct net *net, struct tcf_proto *tp, struct nlattr **struct tc_action_ops *a_o; flags |=TCA_ACT_FLAGS_POLICE |TCA_ACT_FLAGS_BIND ; - a_o =tc_action_load_ops (tb[exts->police], flags, + a_o =tc_action_load_ops (net, tb[exts->police], flags, extack);if (IS_ERR (a_o))return PTR_ERR (a_o);
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 2/15 net/sched: act_api: increase action kind string length
In reply to Jamal Hadi SalimIncrease action kind string length from IFNAMSIZ to 64
The new P4 actions, created via templates, will have longer names of format: "pipeline_name/act_name". IFNAMSIZ is currently 16 and is most of the times undersized for the above format.
So, to conform to this new format, we increase the maximum name length and change its definition from IFNAMSIZ to ACTNAMSIZ to account for this extra string (pipeline name) and the '/' character.
--- include/net/act_api.h | 2 +- include/uapi/linux/pkt_cls.h | 1 + net/sched/act_api.c | 8 ++++---- 3 files changed, 6 insertions(+), 5 deletions(-)
include/net/act_api.h +1 -1
@@ -106,7 +106,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head;struct list_head p4_head; -char kind[IFNAMSIZ ]; +char kind[ACTNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;size_t size;
include/uapi/linux/pkt_cls.h +1 -0
@@ -6,6 +6,7 @@#include <linux/pkt_sched.h> #define TC_COOKIE_MAX_SIZE 16 +#define ACTNAMSIZ 64/* Action attributes */ enum {
net/sched/act_api.c +4 -4
@@ -476,7 +476,7 @@ static size_t tcf_action_shared_attrs_size(const struct tc_action *act)rcu_read_unlock ();return nla_total_size (0 )/* action number nested */ - +nla_total_size (IFNAMSIZ )/* TCA_ACT_KIND */ + +nla_total_size (ACTNAMSIZ )/* TCA_ACT_KIND */ + cookie_len/* TCA_ACT_COOKIE */ +nla_total_size (sizeof (struct nla_bitfield32 ))/* TCA_ACT_HW_STATS */ +nla_total_size (0 )/* TCA_ACT_STATS nested */
@@ -1424,7 +1424,7 @@ tc_action_load_ops(struct net *net, struct nlattr *nla,bool police = flags &TCA_ACT_FLAGS_POLICE ;struct nlattr *tb[TCA_ACT_MAX +1 ];struct tc_action_ops *a_o; -char act_name[IFNAMSIZ ]; +char act_name[ACTNAMSIZ ];struct nlattr *kind;int err;
@@ -1439,12 +1439,12 @@ tc_action_load_ops(struct net *net, struct nlattr *nla,NL_SET_ERR_MSG (extack,"TC action kind must be specified" );return ERR_PTR (err); } -if (nla_strscpy (act_name, kind,IFNAMSIZ ) <0 ) { +if (nla_strscpy (act_name, kind,ACTNAMSIZ ) <0 ) {NL_SET_ERR_MSG (extack,"TC action name too long" );return ERR_PTR (err); } }else { -if (strscpy (act_name,"police" ,IFNAMSIZ ) <0 ) { +if (strscpy (act_name,"police" ,ACTNAMSIZ ) <0 ) {NL_SET_ERR_MSG (extack,"TC action name too long" );return ERR_PTR (-EINVAL ); }
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 3/15 net/sched: act_api: Update tc_action_ops to account for P4 actions
In reply to Jamal Hadi SalimThe initialisation of P4TC action instances require access to a struct p4tc_act (which appears in later patches) to help us to retrieve information like the P4 action parameters etc. In order to retrieve struct p4tc_act we need the pipeline name or id and the action name or id. Also recall that P4TC action IDs are P4 and are net namespace specific and not global like standard tc actions.
The init callback from tc_action_ops parameters had no way of supplying us that information. To solve this issue, we decided to create a new tc_action_ops callback (init_ops), that provies us with the tc_action_ops struct which then provides us with the pipeline and action name. In addition we add a new refcount to struct tc_action_ops called dyn_ref, which accounts for how many action instances we have of a specific action.
--- include/net/act_api.h | 6 ++++++ net/sched/act_api.c | 14 +++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)
include/net/act_api.h +6 -0
@@ -120,6 +120,12 @@ struct tc_action_ops { struct nlattr *est, struct tc_action **act, struct tcf_proto *tp, u32 flags, struct netlink_ext_ack *extack); +/* This should be merged with the original init action */ +int (*init_ops)(struct net *net,struct nlattr *nla, +struct nlattr *est,struct tc_action **act, +struct tcf_proto *tp, +const struct tc_action_ops *ops,u32 flags, +struct netlink_ext_ack *extack); int (*walk)(struct net *, struct sk_buff *, struct netlink_callback *, int, const struct tc_action_ops *,
net/sched/act_api.c +11 -3
@@ -1044,7 +1044,7 @@ int tcf_register_action(struct tc_action_ops *act,struct tc_action_ops *a;int ret; -if (!act->act || !act->dump || !act->init) +if (!act->act || !act->dump || (!act->init && !act->init_ops))return -EINVAL ;/* We have to register pernet ops before making the action ops visible,
@@ -1517,8 +1517,16 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, } } - err = a_o->init (net, tb[TCA_ACT_OPTIONS ], est, &a, tp, - userflags.value | flags, extack); +/* When we arrive here we guarantee that a_o->init or +* a_o->init_ops exist. +*/ +if (a_o->init) + err = a_o->init (net, tb[TCA_ACT_OPTIONS ], est, &a, tp, + userflags.value | flags, extack); +else + err = a_o->init_ops (net, tb[TCA_ACT_OPTIONS ], est, &a, + tp, a_o, userflags.value | flags, + extack); }else { err = a_o->init (net, nla, est, &a, tp, userflags.value | flags, extack);
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 4/15 net/sched: act_api: add struct p4tc_action_ops as a parameter to lookup callback
In reply to Jamal Hadi SalimFor P4 actions, we require information from struct tc_action_ops, specifically the action kind, to find and locate the P4 action information for the lookup operation.
--- include/net/act_api.h | 3 ++- net/sched/act_api.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
include/net/act_api.h +2 -1
@@ -115,7 +115,8 @@ struct tc_action_ops { struct tcf_result *); /* called under RCU BH lock*/int (*dump)(struct sk_buff *,struct tc_action *,int ,int );void (*cleanup)(struct tc_action *); -int (*lookup)(struct net *net,struct tc_action **a,u32 index); +int (*lookup)(struct net *net,const struct tc_action_ops *ops, +struct tc_action **a,u32 index); int (*init)(struct net *net, struct nlattr *nla, struct nlattr *est, struct tc_action **act, struct tcf_proto *tp,
net/sched/act_api.c +1 -1
@@ -726,7 +726,7 @@ static int __tcf_idr_search(struct net *net,struct tc_action_net *tn =net_generic (net, ops->net_id);if (unlikely (ops->lookup)) -return ops->lookup (net, a, index); +return ops->lookup (net, ops, a, index);return tcf_idr_search (tn, a, index); }
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 5/15 net: sched: act_api: Add support for preallocated P4 action instances
In reply to Jamal Hadi SalimIn P4, actions are assumed to pre exist and have an upper bound number of instances. Typically if you a table defined with 1M table entries you want to allocate enough action instances to cover the 1M entries. However, this is a big waste of memory if the action instances are not in use. So for our case, we allow the user to specify a minimal amount of actions in the template and then if more P4 action instances are needed then they will be added on demand as in the current approach with tc filter-action relationship.
Add the necessary code to preallocate actions instances for P4 actions.
We add 2 new actions flags:
- TCA_ACT_FLAGS_PREALLOC: Indicates the action instance is a P4 action and was preallocated for future use the templating phase of P4TC
- TCA_ACT_FLAGS_UNREFERENCED: Indicates the action instance was preallocated and is currently not being referenced by any other object. Which means it won't show up in an action instance dump.
Once an action instance is created we don't free it when the last table entry referring to it is deleted.
Instead we add it to the pool/cache of action instances for that specific action kind i.e it counts as if it is preallocated. Preallocated actions can't be deleted by the tc actions runtime commands and a dump or a get will only show preallocated actions instances which are being used (i.e TCA_ACT_FLAGS_UNREFERENCED == false).
The preallocated actions will be deleted once the pipeline is deleted (which will purge the P4 action kind and its instances).
For example, if we were to create a P4 action that preallocates 128 elements and dumped:
$ tc -j p4template get action/myprog/send_nh | jq .
We'd see the following:
[
{
"obj": "action template",
"pname": "myprog",
"pipeid": 1
},
{
"templates": [
{
"aname": "myprog/send_nh",
"actid": 1,
"params": [
{
"name": "port",
"type": "dev",
"id": 1
}
],
"prealloc": 128
}
] }
]
If we try to dump the P4 action instances, we won't see any:
$ tc -j actions ls action myprog/send_nh | jq .
[]
However, if we create a table entry which references this action kind:
$ tc p4ctrl create myprog/table/cb/FDB \ dstAddr d2:96:91:5d:02:86 action myprog/send_nh \ param port type dev dummy0
Dumping the action instance will now show this one instance which is associated with the table entry:
$ tc -j actions ls action myprog/send_nh | jq .
[
{
"total acts": 1
},
{
"actions": [
{
"order": 0,
"kind": "myprog/send_nh",
"index": 1,
"ref": 1,
"bind": 1,
"params": [
{
"name": "port",
"type": "dev",
"value": "dummy0",
"id": 1
}
],
"not_in_hw": true
}
] }
]
--- include/net/act_api.h | 3 +++ net/sched/act_api.c | 45 +++++++++++++++++++++++++++++++++++-------- 2 files changed, 40 insertions(+), 8 deletions(-)
include/net/act_api.h +3 -0
@@ -68,6 +68,8 @@ struct tc_action {#define TCA_ACT_FLAGS_REPLACE (1U << (TCA_ACT_FLAGS_USER_BITS + 2))#define TCA_ACT_FLAGS_NO_RTNL (1U << (TCA_ACT_FLAGS_USER_BITS + 3))#define TCA_ACT_FLAGS_AT_INGRESS (1U << (TCA_ACT_FLAGS_USER_BITS + 4)) +#define TCA_ACT_FLAGS_PREALLOC (1U << (TCA_ACT_FLAGS_USER_BITS + 5)) +#define TCA_ACT_FLAGS_UNREFERENCED (1U << (TCA_ACT_FLAGS_USER_BITS + 6))/* Update lastuse only if needed, to avoid dirtying a cache line. * We use a temp variable to avoid fetching jiffies twice.
@@ -201,6 +203,7 @@ int tcf_idr_create_from_flags(struct tc_action_net *tn, u32 index, const struct tc_action_ops *ops, int bind, u32 flags);void tcf_idr_insert_many (struct tc_action *actions[],int init_res[]); +void tcf_idr_insert_n (struct tc_action *actions[],const u32 n);void tcf_idr_cleanup (struct tc_action_net *tn,u32 index);int tcf_idr_check_alloc (struct tc_action_net *tn,u32 *index,struct tc_action **a,int bind);
net/sched/act_api.c +37 -8
82 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 6/15 p4tc: add P4 data types
In reply to Jamal Hadi SalimIntroduce abstraction that represents P4 data types.
This also introduces the Kconfig and Makefile which later patches use. Numeric types could be little, host or big endian definitions. The abstraction also supports defining:
a) bitstrings using P4 annotations that look like "bit<X>" where X is the number of bits defined in a type
b) bitslices such that one can define in P4 as bit<8>[0-3] and bit<16>[4-9]. A 4-bit slice from bits 0-3 and a 6-bit slice from bits 4-9 respectively.
c) speacialized types like dev (which stands for a netdev), key, etc
Each type has a bitsize, a name (for debugging purposes), an ID and methods/ops. The P4 types will be used by externs, dynamic actions, packet headers and other parts of P4TC.
Each type has four ops:
- validate_p4t: Which validates if a given value of a specific type meets valid boundary conditions.
- create_bitops: Which, given a bitsize, bitstart and bitend allocates and returns a mask and a shift value. For example, if we have type bit<8>[3-3] meaning bitstart = 3 and bitend = 3, we'll create a mask which would only give us the fourth bit of a bit8 value, that is, 0x08. Since we are interested in the fourth bit, the bit shift value will be 3. This is also useful if an "irregular" bitsize is used, for example, bit24. In that case bitstart = 0 and bitend = 23. Shift will be 0 and the mask will be 0xFFFFFF00 if the machine is big endian.
- host_read : Which reads the value of a given type and transforms it to host order (if needed)
- host_write : Which writes a provided host order value and transforms it to the type's native order (if needed)
--- include/net/p4tc_types.h | 89 +++ include/uapi/linux/p4tc.h | 33 + net/sched/Kconfig | 11 + net/sched/Makefile | 2 + net/sched/p4tc/Makefile | 3 + net/sched/p4tc/p4tc_types.c | 1213 +++++++++++++++++++++++++++++++++++ 6 files changed, 1351 insertions(+) create mode 100644 include/net/p4tc_types.h create mode 100644 include/uapi/linux/p4tc.h create mode 100644 net/sched/p4tc/Makefile create mode 100644 net/sched/p4tc/p4tc_types.c
include/net/p4tc_types.h new file +89 -0
@@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_P4TYPES_H +#define __NET_P4TYPES_H + +#include <linux/netlink.h> +#include <linux/pkt_cls.h> +#include <linux/types.h> + +#include <uapi/linux/p4tc.h> + +#define P4TC_T_MAX_BITSZ 512 + +struct p4tc_type_mask_shift { +void *mask; +u8 shift; +}; + +struct p4tc_type ; +struct p4tc_type_ops { +int (*validate_p4t)(struct p4tc_type *container,void *value, +u16 startbit,u16 endbit, +struct netlink_ext_ack *extack); +struct p4tc_type_mask_shift *(*create_bitops)(u16 bitsz,u16 bitstart, +u16 bitend, +struct netlink_ext_ack *extack); +void (*host_read)(struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +void (*host_write)(struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +}; + +#define P4TC_T_MAX_STR_SZ 32 +struct p4tc_type { +char name[P4TC_T_MAX_STR_SZ ]; +const struct p4tc_type_ops *ops; +size_t container_bitsz; +size_t bitsz; +int typeid; +}; + +struct p4tc_type *p4type_find_byid (int id); +bool p4tc_is_type_unsigned_he (int typeid); +bool p4tc_is_type_numeric (int typeid); + +void p4t_copy (struct p4tc_type_mask_shift *dst_mask_shift, +struct p4tc_type *dst_t,void *dstv, +struct p4tc_type_mask_shift *src_mask_shift, +struct p4tc_type *src_t,void *srcv); +int p4t_cmp (struct p4tc_type_mask_shift *dst_mask_shift, +struct p4tc_type *dst_t,void *dstv, +struct p4tc_type_mask_shift *src_mask_shift, +struct p4tc_type *src_t,void *srcv); +void p4t_release (struct p4tc_type_mask_shift *mask_shift); + +int p4tc_register_types (void ); +void p4tc_unregister_types (void ); + +#ifdef CONFIG_RETPOLINE +void __p4tc_type_host_read (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +void __p4tc_type_host_write (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +#else +static inline void +__p4tc_type_host_read (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift, +void *sval,void *dval) +{ +return ops->host_read (container, mask_shift, sval, dval); +} + +static inline void +__p4tc_type_host_write (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift, +void *sval,void *dval) +{ +return ops->host_write (container, mask_shift, sval, dval); +} +#endif + +#endif
include/uapi/linux/p4tc.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef __LINUX_P4TC_H +#define __LINUX_P4TC_H + +#define P4TC_MAX_KEYSZ 512 + +enum { +P4TC_T_UNSPEC , +P4TC_T_U8 , +P4TC_T_U16 , +P4TC_T_U32 , +P4TC_T_U64 , +P4TC_T_STRING , +P4TC_T_S8 , +P4TC_T_S16 , +P4TC_T_S32 , +P4TC_T_S64 , +P4TC_T_MACADDR , +P4TC_T_IPV4ADDR , +P4TC_T_BE16 , +P4TC_T_BE32 , +P4TC_T_BE64 , +P4TC_T_U128 , +P4TC_T_S128 , +P4TC_T_BOOL , +P4TC_T_DEV , +P4TC_T_KEY , + __P4TC_T_MAX, +}; + +#define P4TC_T_MAX (__P4TC_T_MAX - 1) + +#endif
net/sched/Kconfig +11 -0
@@ -675,6 +675,17 @@ config NET_EMATCH_IPT To compile this code as a module, choose M here: the module will be called em_ipt. +config NET_P4TC + bool "P4TC support" + select NET_CLS_ACT + help + Say Y here if you want to use P4 features on top of TC. + P4 is an open source, domain-specific programming language for + specifying data plane behavior. By enabling P4TC you will be able to + write a P4 program, use a P4 compiler that supports P4TC backend to + generate all needed artificats, which when loaded allow you to + introduce a new kernel datapath that can be controlled via TC. + config NET_CLS_ACT bool "Actions" select NET_CLS
net/sched/Makefile +2 -0
@@ -81,3 +81,5 @@ obj-$(CONFIG_NET_EMATCH_TEXT) += em_text.o obj-$(CONFIG_NET_EMATCH_CANID) += em_canid.o obj-$(CONFIG_NET_EMATCH_IPSET) += em_ipset.o obj-$(CONFIG_NET_EMATCH_IPT) += em_ipt.o + +obj-$(CONFIG_NET_P4TC) += p4tc/
net/sched/p4tc/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-y := p4tc_types.o
net/sched/p4tc/p4tc_types.c new file +1213 -0
1,213 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 7/15 p4tc: add template API
In reply to Jamal Hadi SalimAdd p4tc template API that will serve as infrastructure for all future template objects (in this set pipeline, table, action, and more later)
This commit is not functional by itself. It needs the subsequent patch to be of any use. This commit's purpose is to ease review. If something were to break and you do git bisect to this patch it will not be helpful. In the next release we are planning to merge the two back again.
The template API infrastructure follows the CRUD (Create, Read/get, Update, and Delete) commands.
To issue a p4template create command the user will follow the below grammar:
tc p4template create objtype/[objpath] [objid] objparams
To show a more concrete example, to create a new pipeline (pipelines come in the next commit), the user would issue the following command:
tc p4template create pipeline/aP4proggie pipeid 1 numtables 1 ...
Note that the user may specify an optional ID to the obj ("pipeid 1" above), if none is specified, the kernel will assign one.
The command for update is analogous:
tc p4template update objtype/[objpath] [objid] objparams
Note that for the user may refer to the object by name (in the objpath) or directly by ID.
Delete is also analogous:
tc p4template delete objtype/[objpath] [objid]
As is get:
tc p4template get objtype/[objpath] [objid]
One can also dump or flush template objects. This will be better exposed in the object specific commits in this patchset
--- include/net/p4tc.h | 53 +++++ include/uapi/linux/p4tc.h | 42 ++++ include/uapi/linux/rtnetlink.h | 9 + net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_tmpl_api.c | 361 +++++++++++++++++++++++++++++++++ security/selinux/nlmsgtab.c | 6 +- 6 files changed, 471 insertions(+), 2 deletions(-) create mode 100644 include/net/p4tc.h create mode 100644 net/sched/p4tc/p4tc_tmpl_api.c
include/net/p4tc.h new file +53 -0
@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_P4TC_H +#define __NET_P4TC_H + +#include <uapi/linux/p4tc.h> +#include <linux/workqueue.h> +#include <net/sch_generic.h> +#include <net/net_namespace.h> +#include <linux/refcount.h> +#include <linux/rhashtable.h> +#include <linux/rhashtable-types.h> + +#define P4TC_PATH_MAX 3 + +struct p4tc_dump_ctx { +u32 ids[P4TC_PATH_MAX ]; +}; + +struct p4tc_template_common ; + +struct p4tc_template_ops { +struct p4tc_template_common *(*cu)(struct net *net,struct nlmsghdr *n, +struct nlattr *nla, +struct netlink_ext_ack *extack); +int (*put)(struct p4tc_template_common *tmpl, +struct netlink_ext_ack *extack); +int (*gd)(struct net *net,struct sk_buff *skb,struct nlmsghdr *n, +struct nlattr *nla,struct netlink_ext_ack *extack); +int (*fill_nlmsg)(struct net *net,struct sk_buff *skb, +struct p4tc_template_common *tmpl, +struct netlink_ext_ack *extack); +int (*dump)(struct sk_buff *skb,struct p4tc_dump_ctx *ctx, +struct nlattr *nla,u32 *ids, +struct netlink_ext_ack *extack); +int (*dump_1)(struct sk_buff *skb,struct p4tc_template_common *common); +u32 obj_id; +}; + +struct p4tc_template_common { +char name[P4TC_TMPL_NAMSZ ]; +struct p4tc_template_ops *ops; +}; + +static inline bool p4tc_tmpl_msg_is_update (struct nlmsghdr *n) +{ +return n->nlmsg_type ==RTM_UPDATEP4TEMPLATE ; +} + +int p4tc_tmpl_generic_dump (struct sk_buff *skb,struct p4tc_dump_ctx *ctx, +struct idr *idr,int idx, +struct netlink_ext_ack *extack); + +#endif
include/uapi/linux/p4tc.h +42 -0
54 lines not on this page · raw message
include/uapi/linux/rtnetlink.h +9 -0
@@ -194,6 +194,15 @@ enum { RTM_GETTUNNEL,#define RTM_GETTUNNEL RTM_GETTUNNEL + RTM_CREATEP4TEMPLATE = 124, +#define RTM_CREATEP4TEMPLATE RTM_CREATEP4TEMPLATE + RTM_DELP4TEMPLATE, +#define RTM_DELP4TEMPLATE RTM_DELP4TEMPLATE + RTM_GETP4TEMPLATE, +#define RTM_GETP4TEMPLATE RTM_GETP4TEMPLATE + RTM_UPDATEP4TEMPLATE, +#define RTM_UPDATEP4TEMPLATE RTM_UPDATEP4TEMPLATE + __RTM_MAX,#define RTM_MAX (((__RTM_MAX + 3) & ~3) - 1) };
net/sched/p4tc/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o +obj-y := p4tc_types.o p4tc_tmpl_api.o
net/sched/p4tc/p4tc_tmpl_api.c new file +361 -0
361 lines not on this page · raw message
security/selinux/nlmsgtab.c +5 -1
@@ -94,6 +94,10 @@ static const struct nlmsg_perm nlmsg_route_perms[] = { { RTM_NEWTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_DELTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_GETTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_CREATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_DELP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_GETP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_UPDATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, }; static const struct nlmsg_perm nlmsg_tcpdiag_perms[] = {
@@ -177,7 +181,7 @@ int selinux_nlmsg_lookup(u16 sclass, u16 nlmsg_type, u32 *perm)* structures at the top of this file with the new mappings * before updating the BUILD_BUG_ON() macro! */ -BUILD_BUG_ON (RTM_MAX != (RTM_NEWTUNNEL +3 )); +BUILD_BUG_ON (RTM_MAX != (RTM_CREATEP4TEMPLATE +3 )); err =nlmsg_perm (nlmsg_type, perm, nlmsg_route_perms,sizeof (nlmsg_route_perms));break ;
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 8/15 p4tc: add template pipeline create, get, update, delete
In reply to Jamal Hadi Salim__Introducing P4 TC Pipeline__
This commit introduces P4 TC pipelines, which emulate the semantics of a P4 program/pipeline using the TC infrastructure.
This patch relies on the previous one to be functional. They were split to ease review.
One can refer to P4 programs/pipelines using their names or their specific pipeline ids (pipeid)
P4 template CRUD (Create, Read/get, Update and Delete) commands apply on a pipeline.
As an example, to create a P4 program/pipeline named aP4proggie with a single table in its pipeline, one would use the following command from user space tc (as generated by the compiler):
tc p4template create pipeline/aP4proggie numtables 1 pipeid 1
Note that, in the above command, the numtables is set as 1; the default is 0 because it is feasible to have a P4 program with no tables at all.
Note: if no pipeid is specified, the kernel will issue one. To see what pipeline ID is issued, one would add -echo option and the response back from the kernel will contain the details:
tc -echo p4template create pipeline/aP4proggie numtables 1
To Read pipeline aP4proggie attributes, one would retrieve those details as follows:
tc p4template get pipeline/[aP4proggie] [pipeid 1]
Note that in the above command one may specify pipeline ID, name or both.
To Update aP4proggie pipeline from 1 to 10 tables, one would use the following command:
tc p4template update pipeline/[aP4proggie] [pipeid 1] numtables 10
Note that, in the above command, one could use the P4 program/pipeline name, id or both to specify which P4 program/pipeline to update.
To Delete a P4 program/pipeline named aP4proggie with a pipeid of 1, one would use the following command:
tc p4template del pipeline/[aP4proggie] [pipeid 1]
Note that, in the above command, one could use the P4 program/pipeline name, id or both to specify which P4 program/pipeline to delete
If one wished to dump all the created P4 programs/pipelines, one would use the following command:
tc p4template get pipeline/
__Pipeline Lifetime__
After _Create_ is issued, one can Read/get, Update and Delete pipeline objects; however the pipeline can only be put to use after it is "sealed". To seal a pipeline, one would issue the following command:
tc p4template update pipeline/aP4proggie state ready
After a pipeline is sealed it can be instantiated via the TC P4 classifier. For example:
tc filter add $DEV ingress protocol any prio 6 p4 pname aP4proggie \
action bpf obj $PARSER.o section p4tc/parse
action bpf obj $PROGNAME.o section p4tc/main
Instantiates aP4proggie in the ingress of $DEV. One could also attach it to a block of ports (example tc block 22) as such:
tc filter add block 22 ingress protocol all prio 6 p4 pname aP4proggie \
action bpf obj $PARSER.o section p4tc/parse
action bpf obj $PROGNAME.o section p4tc/main
We can, add a table entry after the pipeline is sealed (even before instantiating). Like, for example:
tc p4ctrl create aP4proggie/table/cb/aP4table \
dstAddr 10.10.10.0/24 srcAddr 192.168.0.0/16 prio 16 \
action dropOnce the pipeline is instantiated on a device or block it cannot be deleted. It becomes Read-only from the control plane/user space. The pipeline can be deleted when there are no longer any users left by destroying all instances (i.e all instantiated filters are deleted).
--- include/net/p4tc.h | 66 +++- include/uapi/linux/p4tc.h | 24 ++ net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_pipeline.c | 634 +++++++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 101 +++++- 5 files changed, 807 insertions(+), 20 deletions(-) create mode 100644 net/sched/p4tc/p4tc_pipeline.c
include/net/p4tc.h +63 -3
@@ -10,27 +10,43 @@#include <linux/rhashtable.h> #include <linux/rhashtable-types.h> +#define P4TC_DEFAULT_NUM_TABLES P4TC_MINTABLES_COUNT +#define P4TC_DEFAULT_MAX_RULES 1#define P4TC_PATH_MAX 3 +#define P4TC_KERNEL_PIPEID 0 + +#define P4TC_PID_IDX 0 +struct p4tc_dump_ctx {u32 ids[P4TC_PATH_MAX ]; };struct p4tc_template_common ; +struct p4tc_path_nlattrs { +char *pname; +u32 *ids; +bool pname_passed; +}; + +struct p4tc_pipeline ;struct p4tc_template_ops {struct p4tc_template_common *(*cu)(struct net *net,struct nlmsghdr *n,struct nlattr *nla, +struct p4tc_path_nlattrs *nl_pname,struct netlink_ext_ack *extack); -int (*put)(struct p4tc_template_common *tmpl, +int (*put)(struct p4tc_pipeline *pipeline, +struct p4tc_template_common *tmpl,struct netlink_ext_ack *extack);int (*gd)(struct net *net,struct sk_buff *skb,struct nlmsghdr *n, -struct nlattr *nla,struct netlink_ext_ack *extack); +struct nlattr *nla,struct p4tc_path_nlattrs *nl_pname, +struct netlink_ext_ack *extack);int (*fill_nlmsg)(struct net *net,struct sk_buff *skb,struct p4tc_template_common *tmpl,struct netlink_ext_ack *extack);int (*dump)(struct sk_buff *skb,struct p4tc_dump_ctx *ctx, -struct nlattr *nla,u32 *ids, +struct nlattr *nla,char **p_name,u32 *ids,struct netlink_ext_ack *extack);int (*dump_1)(struct sk_buff *skb,struct p4tc_template_common *common);u32 obj_id;
@@ -39,6 +55,25 @@ struct p4tc_template_ops {struct p4tc_template_common {char name[P4TC_TMPL_NAMSZ ];struct p4tc_template_ops *ops; +u32 p_id; +u32 __pad0; +}; + +struct p4tc_pipeline { +struct p4tc_template_common common; +struct rcu_head rcu; +struct net *net; +/* Accounts for how many entities are referencing this pipeline. +* As for now only P4 filters can refer to pipelines. +*/ +refcount_t p_ctrl_ref; +u16 num_tables; +u16 curr_tables; +u8 p_state; +}; + +struct p4tc_pipeline_net { +struct idr pipeline_idr; };static inline bool p4tc_tmpl_msg_is_update (struct nlmsghdr *n)
@@ -46,8 +81,33 @@ static inline bool p4tc_tmpl_msg_is_update(struct nlmsghdr *n)return n->nlmsg_type ==RTM_UPDATEP4TEMPLATE ; } +int p4tc_tmpl_register_ops (const struct p4tc_template_ops *tmpl_ops); +int p4tc_tmpl_generic_dump (struct sk_buff *skb,struct p4tc_dump_ctx *ctx,struct idr *idr,int idx,struct netlink_ext_ack *extack); +struct p4tc_pipeline *p4tc_pipeline_find_byany (struct net *net, +const char *p_name, +const u32 pipeid, +struct netlink_ext_ack *extack); +struct p4tc_pipeline *p4tc_pipeline_find_byid (struct net *net, +const u32 pipeid); +struct p4tc_pipeline * +p4tc_pipeline_find_get (struct net *net,const char *p_name, +const u32 pipeid,struct netlink_ext_ack *extack); + +static inline bool p4tc_pipeline_get (struct p4tc_pipeline *pipeline) +{ +return refcount_inc_not_zero (&pipeline->p_ctrl_ref); +} + +void p4tc_pipeline_put (struct p4tc_pipeline *pipeline); +struct p4tc_pipeline * +p4tc_pipeline_find_byany_unsealed (struct net *net,const char *p_name, +const u32 pipeid, +struct netlink_ext_ack *extack); + +#define to_pipeline (t) ((struct p4tc_pipeline *)t) +#endif
include/uapi/linux/p4tc.h +24 -0
@@ -7,19 +7,25 @@/* pipeline header */ struct p4tcmsg { +__u32 pipeid;__u32 obj; }; +#define P4TC_MAXPIPELINE_COUNT 32 +#define P4TC_MAXTABLES_COUNT 32 +#define P4TC_MINTABLES_COUNT 0#define P4TC_MSGBATCH_SIZE 16#define P4TC_MAX_KEYSZ 512#define P4TC_TMPL_NAMSZ 32 +#define P4TC_PIPELINE_NAMSIZ P4TC_TMPL_NAMSZ/* Root attributes */ enum {P4TC_ROOT_UNSPEC ,P4TC_ROOT ,/* nested messages */ +P4TC_ROOT_PNAME ,/* string - mandatory for pipeline create */ __P4TC_ROOT_MAX, };
@@ -28,6 +34,7 @@ enum {/* P4 Object types */ enum {P4TC_OBJ_UNSPEC , +P4TC_OBJ_PIPELINE , __P4TC_OBJ_MAX, };
@@ -43,6 +50,23 @@ enum {#define P4TC_MAX (__P4TC_MAX - 1) +/* PIPELINE attributes */ +enum { +P4TC_PIPELINE_UNSPEC , +P4TC_PIPELINE_NUMTABLES ,/* u16 */ +P4TC_PIPELINE_STATE ,/* u8 */ +P4TC_PIPELINE_NAME ,/* string only used for pipeline dump */ + __P4TC_PIPELINE_MAX +}; + +#define P4TC_PIPELINE_MAX (__P4TC_PIPELINE_MAX - 1) + +/* PIPELINE states */ +enum { +P4TC_STATE_NOT_READY , +P4TC_STATE_READY , +}; +enum {P4TC_T_UNSPEC ,P4TC_T_U8 ,
net/sched/p4tc/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o p4tc_tmpl_api.o +obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o
net/sched/p4tc/p4tc_pipeline.c new file +634 -0
634 lines not on this page · raw message
net/sched/p4tc/p4tc_tmpl_api.c +85 -16
248 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 9/15 p4tc: add template action create, update, delete, get, flush and dump
In reply to Jamal Hadi SalimThis commit allows the control plane to create, update, delete, get, flush and dump action templates based on P4 action definitions.
Visualize the following action in a P4 program named aP4Proggie:
action send_nh(@tc_type("macaddr) bit<48> dstAddr, @tc_type("dev") bit<8> port)
{
hdr.ethernet.dstAddr = dstMac;
send_to_port(port);
}The above is an action called send_nh which receives as parameters a bit<48> dstAddr (a mac address) and a bit<8> port (something close to ifindex).
which is applied on a P4 table match as such:
table mytable {
key = {
hdr.ipv4.dstAddr @tc_type("ipv4"): lpm;
}
actions = {
send_nh;
drop;
NoAction;
}
size = 1024;
}The mechanics of actions follow the CRUD semantics which are illustrated in the text below.
___P4 ACTION KIND CREATION___
In this stage we create the p4 action kind by specifying the action name, its ID, its parameters and the parameter types.
So for the send_nh action, the creation would look something like this:
tc p4template create action/aP4proggie/send_nh \
param dstAddr type macaddr id 1 param port type dev id 2
All the template commands (tc p4template) are generated by the p4c compiler (but of course could be hand coded by humans).
Also note that an action name has to specify the program name since P4 actions are unique to a program. As an example, the above command creates an action template that is bounded to pipeline/program named "aP4proggie".
Note2: In P4, actions are assumed to pre-exist and have an upper bound number of instances. Typically, if you have a max of 1024 "mytable" table entries you want to allocate enough action instances to cover the 1024 entries. However, this is a big waste of memory when we have low table occupancy. We pick a middle ground by providing pre-allocation control via attribute "num_prealloc".
The compiler generated template does not specify it and by default we preallocate 16 entries. The user can override this value by editing the generated text, for example to change the number to 128 as such:
tc p4template create action/aP4proggie/send_nh num_prealloc 128 \
param dstAddr type macaddr id 1 param port type dev id 2
When all preallocated action instances are exhausted (used in table entries) then the behavior switches to the current tc action approach i.e for every table entry created a new action instance is dynamically allocated if no "index" attribute is specified. Once an instance is created it is added to the pool and never freed.
Note, Current tc action behavior is maintained:
a) If the user wishes to preallocate more action instances later at runtime to take advantage of a faster table entry creation (by avoiding dynamic allocation at table entry creation time), they will have to individually create actions via the control plane using the classical "tc actions" command.
For example:
tc actions add action aP4proggie/send_nh \
param dstAddr AA:BB:CC:DD:EE:DD param port eth1
The action is added to the pool of action aP4proggie/send_nh instances and any table entry creation will grab it. The parameters specified above will be replaced when the table entry is created.
b) Sharing of action instances works the same way i.e you could autobind to any action instance in a table entry creation by specifying the action "index".
___ACTION KIND ACTIVATION___
Once we provided all the necessary information for the new p4 action, we can go to the final stage: action activation. In this template stage, we activate the p4 action and make it available for instantiation. To activate the action template, we issue the following command:
tc p4template update action/aP4proggie/send_nh state active
___OTHER CONTROL COMMANDS___
The lifetime of the p4 action is tied to its pipeline
(see earlier patches). As with all pipeline components, write operations to action templates (i.e action kinds), such as create, update and delete, can only be executed if the pipeline is not sealed. Read/get can be issued even after the pipeline is sealed.
If, after we are done with our action template we want to delete it, we could issue the following command:
tc p4template del action/aP4proggie/send_nh
Note: If any instance was created for this action (as illustrated earlier) than this action cannot be deleted, unless you delete all instances first.
If we had created more action templates and wanted to flush all of the action templates from pipeline aP4proggie, one would use the following command:
tc p4template del action/aP4proggie/
After creating or updating a p4 actions, if one wishes to verify that the p4 action was created correctly, one would use the following sample command:
tc p4template get action/aP4proggie/send_nh
The above command will display the relevant data for the action, such as parameter names, types, etc.
If one wanted to check which action templates were associated to a specific pipeline, one could use the following command:
tc p4template get action/aP4proggie/
Note that this command will only display the name of these action templates. To verify their specific details, one should use the get command, which was previously described.
--- include/net/act_api.h | 1 + include/net/p4tc.h | 72 ++ include/net/tc_act/p4tc.h | 27 + include/uapi/linux/p4tc.h | 50 ++ include/uapi/linux/tc_act/tc_p4.h | 11 + net/sched/p4tc/Makefile | 3 +- net/sched/p4tc/p4tc_action.c | 1105 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_pipeline.c | 16 +- net/sched/p4tc/p4tc_tmpl_api.c | 10 + 9 files changed, 1293 insertions(+), 2 deletions(-) create mode 100644 include/net/tc_act/p4tc.h create mode 100644 include/uapi/linux/tc_act/tc_p4.h create mode 100644 net/sched/p4tc/p4tc_action.c
include/net/act_api.h +1 -0
@@ -70,6 +70,7 @@ struct tc_action {#define TCA_ACT_FLAGS_AT_INGRESS (1U << (TCA_ACT_FLAGS_USER_BITS + 4))#define TCA_ACT_FLAGS_PREALLOC (1U << (TCA_ACT_FLAGS_USER_BITS + 5))#define TCA_ACT_FLAGS_UNREFERENCED (1U << (TCA_ACT_FLAGS_USER_BITS + 6)) +#define TCA_ACT_FLAGS_FROM_P4TC (1U << (TCA_ACT_FLAGS_USER_BITS + 7))/* Update lastuse only if needed, to avoid dirtying a cache line. * We use a temp variable to avoid fetching jiffies twice.
include/net/p4tc.h +72 -0
103 lines not on this page · raw message
include/net/tc_act/p4tc.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_TC_ACT_P4_H +#define __NET_TC_ACT_P4_H + +#include <net/pkt_cls.h> +#include <net/act_api.h> + +struct tcf_p4act_params { +struct idr params_idr; +struct p4tc_act_param **params_array; +struct rcu_head rcu; +u32 num_params; +u32 tot_params_sz; +}; + +struct tcf_p4act { +struct tc_action common; +/* Params IDR reference passed during runtime */ +struct tcf_p4act_params __rcu *params; +u32 p_id; +u32 act_id; +struct list_head node; +}; + +#define to_p4act (a) ((struct tcf_p4act *)a) + +#endif /* __NET_TC_ACT_P4_H */
include/uapi/linux/p4tc.h +50 -0
83 lines not on this page · raw message
include/uapi/linux/tc_act/tc_p4.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef __LINUX_TC_P4_H +#define __LINUX_TC_P4_H + +#include <linux/pkt_cls.h> + +struct tc_act_p4 { +tc_gen ; +}; + +#endif
net/sched/p4tc/Makefile +2 -1
@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o +obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ + p4tc_action.o
net/sched/p4tc/p4tc_action.c new file +1105 -0
1,105 lines not on this page · raw message
net/sched/p4tc/p4tc_pipeline.c +15 -1
@@ -75,6 +75,8 @@ static const struct nla_policy tc_pipeline_policy[P4TC_PIPELINE_MAX + 1] = {static void p4tc_pipeline_destroy (struct p4tc_pipeline *pipeline) { +idr_destroy (&pipeline->p_act_idr); +kfree (pipeline); }
@@ -96,6 +98,12 @@ static void p4tc_pipeline_teardown(struct p4tc_pipeline *pipeline,struct net *net = pipeline->net;struct p4tc_pipeline_net *pipe_net =net_generic (net, pipeline_net_id);struct net *pipeline_net =maybe_get_net (net); +unsigned long iter_act_id; +struct p4tc_act *act; +unsigned long tmp; + +idr_for_each_entry_ul (&pipeline->p_act_idr, act, tmp, iter_act_id) + act->common.ops->put (pipeline, &act->common, extack);/* If we are on netns cleanup we can't touch the pipeline_idr. * On pre_exit we will destroy the idr but never call into teardown
@@ -152,6 +160,7 @@ static int pipeline_try_set_state_ready(struct p4tc_pipeline *pipeline, } pipeline->p_state =P4TC_STATE_READY ; +return true; }
@@ -251,6 +260,10 @@ p4tc_pipeline_create(struct net *net, struct nlmsghdr *n,else pipeline->num_tables =P4TC_DEFAULT_NUM_TABLES ; +idr_init (&pipeline->p_act_idr); + + pipeline->num_created_acts =0 ; + pipeline->p_state =P4TC_STATE_NOT_READY ; pipeline->net = net;
@@ -505,7 +518,8 @@ static int p4tc_pipeline_gd(struct net *net, struct sk_buff *skb,return PTR_ERR (pipeline); tmpl = (struct p4tc_template_common *)pipeline; -if (p4tc_pipeline_fill_nlmsg (net, skb, tmpl, extack) <0 ) + ret =p4tc_pipeline_fill_nlmsg (net, skb, tmpl, extack); +if (ret <0 )return -1 ;if (!ids[P4TC_PID_IDX ])
net/sched/p4tc/p4tc_tmpl_api.c +10 -0
@@ -158,6 +158,11 @@ static int tc_ctl_p4_tmpl_1(struct sk_buff *skb, struct nlmsghdr *n, ids[P4TC_PID_IDX ] = t->pipeid; +if (tb[P4TC_PATH ]) { +const u32 *arg_ids =nla_data (tb[P4TC_PATH ]); + +memcpy (&ids[P4TC_PID_IDX +1 ], arg_ids,nla_len (tb[P4TC_PATH ])); + } nl_path_attrs.ids = ids; nskb =alloc_skb (NLMSG_GOODSIZE ,GFP_KERNEL );
@@ -365,6 +370,11 @@ static int tc_ctl_p4_tmpl_dump_1(struct sk_buff *skb, struct nlattr *arg, root =nla_nest_start (skb,P4TC_ROOT ); ids[P4TC_PID_IDX ] = t->pipeid; +if (tb[P4TC_PATH ]) { +const u32 *arg_ids =nla_data (tb[P4TC_PATH ]); + +memcpy (&ids[P4TC_PID_IDX +1 ], arg_ids,nla_len (tb[P4TC_PATH ])); + } obj_op = (struct p4tc_template_ops *)p4tc_ops[t->obj]; ret = obj_op->dump (skb, ctx, tb[P4TC_PARAMS ], &p_name, ids, extack);
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 10/15 p4tc: add runtime action support
In reply to Jamal Hadi SalimThis commit deals with the runtime part of P4 actions i.e instantiation and binding of action kinds which are created via templates (see previous patch).
For illustration we repeat the P4 code snippet from the action template commit:
action send_nh(@tc_type("macaddr) bit<48> dstAddr, @tc_type("dev") bit<8> port)
{
hdr.ethernet.dstAddr = dstMac;
send_to_port(port);
}table mytable {
key = {
hdr.ipv4.dstAddr @tc_type("ipv4"): lpm;
}
actions = {
send_nh;
drop;
NoAction;
}
size = 1024;
}One could create a table entry alongside an action instance as follows:
tc p4ctrl create aP4proggie/table/mycontrol/mytable \
srcAddr 10.10.10.0/24 \
action send_nh param dstAddr AA:BB:CC:DD:EE:FF param port eth0
As previously stated, we refer to the action by it's "full name" (pipeline_name/action_name). In the above the pipeline_name is inherited from the table patch ("aP4proggie"), so it doesn't need to be specified. Above we are creating an instance of the send_nh action specifying as parameter values AA:BB:CC:DD:EE:FF for dstAddr and eth0 for port.
We could also create the action instance outside of the table entry which is then added to the pool of send_nh action kind's instances as follows:
tc actions add action aP4proggie/send_nh \
param dstAddr AA:BB:CC:DD:EE:FF param port eth0
Observe these are _exactly the same semantics_ as what tc today already provides with a caveat that we have a keyword "param" to precede the appropriate parameters.
Note: We can create as many instances for action templates as we wish, as long as we do not exceed the maximum allowed actions - in this specific case 1024 for table "mytable". Any creation of action instances above the max allowed will be rejected by the kernel.
Action sharing still works the same way (as in classical tc). For example if we know the action index of the previous instance is 100 then we can bind it to a table entry, for example
tc p4ctrl create aP4proggie/table/mycontrol/mytable \
srcAddr 11.11.0.0/16 action send_nh index 100
--- include/net/p4tc.h | 115 ++++ include/net/tc_act/p4tc.h | 1 + include/uapi/linux/p4tc.h | 13 + net/sched/p4tc/p4tc_action.c | 1263 +++++++++++++++++++++++++++++++++- 4 files changed, 1382 insertions(+), 10 deletions(-)
include/net/p4tc.h +115 -0
@@ -116,6 +116,56 @@ p4tc_pipeline_find_byany_unsealed(struct net *net, const char *p_name,const u32 pipeid, struct netlink_ext_ack *extack); +struct p4tc_act *p4a_runt_find (struct net *net, +const struct tc_action_ops *a_o, +struct netlink_ext_ack *extack); +void +p4a_runt_prealloc_put (struct p4tc_act *act,struct tcf_p4act *p4_act); + +static inline int p4tc_action_destroy (struct tc_action *acts[]) +{ +struct tc_action *acts_non_prealloc[TCA_ACT_MAX_PRIO ] = {NULL }; +struct tc_action *a; +int ret =0 ; +int j =0 ; +int i; + +tcf_act_for_each_action (i, a, acts) { +if (acts[i]->tcfa_flags &TCA_ACT_FLAGS_PREALLOC ) { +struct tcf_p4act *p4act; +struct p4tc_act *act; +struct net *net; + + p4act = (struct tcf_p4act *)acts[i]; + net =maybe_get_net (acts[i]->idrinfo->net); + +if (net) { +const struct tc_action_ops *ops; + + ops = acts[i]->ops; + act =p4a_runt_find (net, ops,NULL ); +p4a_runt_prealloc_put (act, p4act); +put_net (net); + }else { +/* If net is coming down, template +* action will be deleted, so no need to +* remove from prealloc list, just decr +* refcounts. +*/ + acts_non_prealloc[j] = acts[i]; + j++; + } + }else { + acts_non_prealloc[j] = acts[i]; + j++; + } + } + + ret =tcf_action_destroy (acts_non_prealloc,TCA_ACT_UNBIND ); + +return ret; +} +struct p4tc_act_param {struct list_head head;struct rcu_head rcu;
@@ -164,10 +214,67 @@ struct p4tc_act {* pipeline it belongs to. */ refcount_t a_ref; +atomic_t num_insts;bool active;char fullname[ACTNAMSIZ ]; }; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla, +struct tc_action *acts[],u32 pipeid, +u32 flags,struct netlink_ext_ack *extack) +{ +struct nlattr *tb[TCA_ACT_MAX_PRIO +1 ] = {}; +int init_res[TCA_ACT_MAX_PRIO ]; +struct tc_action *a; +size_t attrs_size; +size_t nacts =0 ; +int ret; +int i; + + ret =nla_parse_nested_deprecated (tb,TCA_ACT_MAX_PRIO , nla,NULL , + extack); +if (ret <0 ) +return ret; + +for (i =1 ; i <TCA_ACT_MAX_PRIO +1 ; i++) + nacts += !!tb[i]; + +if (nacts >1 ) { +NL_SET_ERR_MSG (extack,"Only one action is allowed" ); +return -E2BIG ; + } + +/* If action was already created, just bind to existing one */ + flags |=TCA_ACT_FLAGS_BIND ; + flags |=TCA_ACT_FLAGS_FROM_P4TC ; + ret =tcf_action_init (net,NULL , nla,NULL , acts, init_res, &attrs_size, + flags,0 , extack); + +/* Check if we are trying to bind to dynamic action from different +* pipeline. +*/ +tcf_act_for_each_action (i, a, acts) { +struct tcf_p4act *p; + +if (a->ops->id <=TCA_ID_MAX ) +continue ; + + p =to_p4act (a); +if (p->p_id != pipeid) { +NL_SET_ERR_MSG (extack, +"Unable to bind to dynact from different pipeline" ); + ret = -EPERM ; + goto destroy_acts; + } + } + +return ret; + +destroy_acts: +p4tc_action_destroy (acts); +return ret; +} +struct p4tc_act *p4a_tmpl_get (struct p4tc_pipeline *pipeline,const char *act_name,const u32 a_id,struct netlink_ext_ack *extack);
@@ -179,6 +286,14 @@ static inline bool p4tc_action_put_ref(struct p4tc_act *act)return refcount_dec_not_one (&act->a_ref); } +struct tcf_p4act * +p4a_runt_prealloc_get_next (struct p4tc_act *act); +void p4a_runt_prealloc_reference (struct p4tc_act *act,struct tcf_p4act *p4act); +void p4a_runt_parm_destroy (struct p4tc_act_param *parm); +struct p4tc_act_param * +p4a_runt_parm_init (struct net *net,struct p4tc_act *act, +struct nlattr *nla,struct netlink_ext_ack *extack); +#define to_pipeline (t) ((struct p4tc_pipeline *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t)
include/net/tc_act/p4tc.h +1 -0
@@ -20,6 +20,7 @@ struct tcf_p4act {u32 p_id;u32 act_id;struct list_head node; +u32 num_runt_params; };#define to_p4act (a) ((struct tcf_p4act *)a)
include/uapi/linux/p4tc.h +13 -0
@@ -115,6 +115,17 @@ enum { };#define P4TC_ACT_MAX (__P4TC_ACT_MAX - 1) + +/* Action params value attributes */ + +enum { +P4TC_ACT_PARAMS_VALUE_UNSPEC , +P4TC_ACT_PARAMS_VALUE_RAW ,/* binary */ + __P4TC_ACT_PARAMS_VALUE_MAX +}; + +#define P4TC_ACT_VALUE_PARAMS_MAX (__P4TC_ACT_PARAMS_VALUE_MAX - 1) +enum {P4TC_ACT_PARAMS_TYPE_UNSPEC ,P4TC_ACT_PARAMS_TYPE_BITEND ,/* u16 */
@@ -138,6 +149,8 @@ enum {P4TC_ACT_PARAMS_ID ,/* u32 */ P4TC_ACT_PARAMS_TYPE ,/* nested type - mandatory for params create */ P4TC_ACT_PARAMS_FLAGS ,/* u8 */ +P4TC_ACT_PARAMS_VALUE ,/* bytes - mandatory for runtime params create */ +P4TC_ACT_PARAMS_MASK ,/* bytes */ __P4TC_ACT_PARAMS_MAX };
net/sched/p4tc/p4tc_action.c +1253 -10
1,382 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 11/15 p4tc: add template table create, update, delete, get, flush and dump
In reply to Jamal Hadi SalimThis commit introduces code to creation and maintenance of P4 tables defined in a P4 program.
As with all other P4TC objects, tables' lifetimes conform to extended CRUD operations and are maintained via templates.
It's important to note that write operations, such as create, update and delete can only be made if the pipeline is not sealed.
Per the P4 specification, tables prefix their name with the control block (although this could be overridden by P4 annotations).
As an example, if one were to create a table named table1 in a pipeline named myprog1, on control block "mycontrol", one would use the following command:
tc p4template create table/myprog1/mycontrol/table1 tblid 1 \
keysz 32 nummasks 8 tentries 8192
Above says that we are creating a table (table1) attached to pipeline myprog1 on control block mycontrol. Table1's key size is 32 bits wide and it can have up to 8 associated masks and 8192 entries. The table id for table1 is 1. The table id is typically provided by the compiler.
Parameters such as nummasks (number of masks this table may have) and tentries (maximum number of entries this table may have); either may be omitted in which case it defaults to 8 masks and 256 entries respectfully.
P4 tables have many associated attributes that further refine how these tables operate. Some attributes are per table and others per entry. Attributes include:
- Timer profiles: The timer profiles for the table entries belonging to the table. These profiles contain the supported aging values for the table. There are currently 4 different built-in profiles ID0=30s(default), ID1=60s, ID2=120s and ID3=180s.
The user can override the defaults by specifying the number of profiles per table. In that case, the kernel will generate the specified number of profiles with their aging values abiding by the following rule:
- Profile ID 0 - 30000ms
- Profile ID(n) - Profile ID(n - 1) + 30000ms
So, for example, if the user specified num_timer_profiles as 5, the profile IDs and aging values would be the following:
- Profile ID 0 - 30000ms (default profile)
- Profile ID 1 - 60000ms
- Profile ID 2 - 90000ms
- Profile ID 3 - 120000ms
- Profile ID 4 - 150000ms
The values of the different profiles could be changed at runtime. The default profile is used if the user specifies an out of range value.
- Match key type: The match type of the table key (exact, LPM, ternary, etc).
Note in P4 a key may constitute multiple (sub)key types, eg matching on srcip using a LPM prefix and an exact match for dstip. In such a case it is up to the compiler to generalize the key used (For example in this case the overall key may endup being LPM or ternary).
- Direct Counter: Table counter instances used directly by the table, when specified in the P4 program, there will be one counter per entry.
- Direct Meter: Table meter instances used directly by the table, when specified in the P4 program, there will be one meter per entry.
- CRUDXPS Permissions both for specific entries and tables. The permissions are applicable to both the control plane and the datapath (see "Table Permissions" further below).
- Allowed Actions List. This will be a list of all possible actions that can be added to table entries that are added to the specified table.
- Action profiles. When defined in a P4 program, action profiles provide a mechanism to share action instances.
- Actions Selectors. When defined in a P4 program can be used to select the execution of one or more action instance selected at table lookup time by using a hash computation.
- Default hit action. When a default hit action is defined it is used when a matched table entry did not define an action. Depending on the P4 program the default hit action can be updated at runtime (in addition to being specified in the template).
- Default miss action. When a default miss action is defined it is used when a lookup that table fails.
- Action scope. In addition to actions being annotated as default hit or miss they can also be annotated to be either specific to a table of globally available to multiple tables within the same P4 program.
- Max entries. This is an upper bound for number of entries a specific table allows.
- Num masks. In the case of LPM or ternary matches, this defines the maximum allowed masks for that table.
- Timers. When defined in a P4 program, each entry has an associated timer. Depending on the programmed timer profile (see above), an entry gets a timeout. The timer attribute specifies the behavior of a table entry expiration.
The timer is refreshed every time there's a hit. After an idle period the P4 program can define using this attribute, whether it wants to have an event generated to user space (and have user space delete the entry), or whether it wants the kernel to delete it and send the event to announce the deletion. The default in P4TC is to both delete and generate an event.
- per entry "static" vs "dynamic" entry. By default all entries created from the control plane are "static" unless otherwise specified. All entries added from datapath are "dynamic" unless otherwise specified. "Dynamic" entries are subject to deletion when idle (subject to the rules specified in "Timers" above).
If one were to retrieve the template details of a table named table1 (before or after the pipeline is sealed) one would use the following command:
tc p4template get table/myprog1/mycontrol/table1
If one were to dump all the tables from a pipeline named myprog1, one would use the following command:
tc p4template get table/myprog1
If one were to update table1 (before the pipeline is sealed) one would use the following command:
tc p4template update table/myprog1/mycontrol/table1 ....
If one were to delete table1 (before the pipeline is sealed) one would use the following command:
tc p4template del table/myprog1/mycontrol/table1
If one were to flush all the tables from a pipeline named myprog1, control block "mycontrol" one would use the following command:
tc p4template del table/myprog1/mycontrol/
___Table Permissions___
Tables can have permissions which apply to all the entries in the specified table. Permissions are defined for both what the control plane (user space) as well as the data path are allowed to do on the table.
The permissions field is a 16bit value which will hold CRUDXPS (create, read, update, delete, execute, publish and subscribe) permissions for control and data path. Bits 13-7 will have the CRUDXPS values for control and bits 6-0 will have CRUDXPS values for data path. By default each table has the following permissions:
CRUD-PS-R--X--
Which means the control plane can perform CRUDPS operations whereas the data path (-R--X--) can only Read and execute on the entries. The user can override these permissions when creating the table or when updating.
For example, the following command will create a table which will not allow the datapath to create, update or delete entries but give full CRUDS permissions for the control plane.
$TC p4template create table/aP4proggie/cb/tname tblid 1 keysz 64 type lpm \
permissions 0x3D24 ...
Recall that these permissions come in the form of CRUDXPSCRUDXPS, where the first CRUDXPS block is for control and the last is for data path.
So 0x34A4 is equivalent to CR-D--S-R--X--
If we were to issue a read command on a table (tname):
$TC -j p4template get table/aP4proggie/cb/tname | jq .
The output would be the following:
[
{
"obj": "table",
"pname": "aP4Proggie",
"pipeid": 22
},
{
"templates": [
{
"tblid": 1,
"tname": "cb/tname",
"keysz": 64,
"max_entries": 256,
"masks": 8,
"entries": 0,
"permissions": "CRUD--S-R--X--",
"table_type": "lpm",
"acts_list": []
}
] }
]
Note, the permissions concept is more powerful than classical const definition currently taken by P4 which makes everything in a table read-only.
___Initial Table Entries___
Templating can create initial table entries. For example:
tc p4template create table/myprog/cb/tname \
entry srcAddr 10.10.10.10/32 dstAddr 1.1.1.0/24 prio 17
In this command we are creating table tname with an entry which will be present before even the P4 program starts executing. This entry has as its key IPv4 srcAddr and dstAddr and prio 17.
If one was to read back the entry by issuing the following runtime command:
tc p4ctrl get myprog/table/cb/tname
They would get:
...
entry priority 17[permissions-RUD--S-R--X--]
entry key
srcAddr id:1 size:32b type:ipv4 exact fieldval 10.10.10.10/32
dstAddr id:2 size:32b type:ipv4 exact fieldval 1.1.1.0/24We will explain the p4ctrl commands in more details on the next patches.
Before the pipeline is sealed we can only do a p4ctrl get. The write commands (create, update, delete and flush) and not allowed before sealing.
___Table Actions List___
P4 tables can be programmed to allow only a specified list of actions to be part of match entry on a table. P4 also defines default actions to be executed when no entries match; we have extended this concept to have a default hit, which is executed upon matching an entry which has no action associated with it.
We also allow flags for each of the actions in this list that specify if the action can be added only as a table entry (tableonly), or only as a default action (defaultonly). If no flags are specified, it is assumed that the action can be used in both contexts.
Both default hit and default missactions are optional.
An example of specifying a default miss action is as follows:
tc p4template update table/myprog/cb/mytable \
default_miss_action permissions 0x1124 action dropThe above will drop packets if the entry is not found in mytable. Note the above makes the default action a const. Meaning the control plane can neither replace it nor delete it.
tc p4template update table/myprog/mytable \
default_hit_action permissions 0x3004 action ok
Whereas the above allows a default hit action to accept the packet. The permission 0x3004 (binary 11000000000100) means we have only Create and Read permissions in the control plane and eXecute permissions in the data plane. This means, for example, that now we can only delete the default hit action from the control plane.
--- include/net/p4tc.h | 155 ++++ include/uapi/linux/p4tc.h | 120 +++ net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_action.c | 4 +- net/sched/p4tc/p4tc_pipeline.c | 23 +- net/sched/p4tc/p4tc_table.c | 1596 ++++++++++++++++++++++++++++++++ 6 files changed, 1890 insertions(+), 10 deletions(-) create mode 100644 net/sched/p4tc/p4tc_table.c
include/net/p4tc.h +155 -0
@@ -16,10 +16,23 @@#define P4TC_DEFAULT_MAX_RULES 1#define P4TC_PATH_MAX 3#define P4TC_MAX_TENTRIES 0x2000000 +#define P4TC_DEFAULT_TENTRIES 256 +#define P4TC_MAX_TMASKS 1024 +#define P4TC_DEFAULT_TMASKS 8 +#define P4TC_MAX_T_AGING_MS 864000000 +#define P4TC_DEFAULT_T_AGING_MS 30000 +#define P4TC_DEFAULT_NUM_TIMER_PROFILES 4 +#define P4TC_MAX_NUM_TIMER_PROFILES P4TC_MSGBATCH_SIZE + +#define P4TC_TIMER_PROFILE_ZERO_AGING_MS 30000 +#define P4TC_DEFAULT_TIMER_PROFILE_ID 0 + +#define P4TC_MAX_PERMISSION (GENMASK(P4TC_PERM_MAX_BIT, 0))#define P4TC_KERNEL_PIPEID 0#define P4TC_PID_IDX 0 +#define P4TC_TBLID_IDX 1#define P4TC_AID_IDX 1#define P4TC_PARSEID_IDX 1
@@ -68,6 +81,7 @@ struct p4tc_template_common {struct p4tc_pipeline {struct p4tc_template_common common;struct idr p_act_idr; +struct idr p_tbl_idr;struct rcu_head rcu;struct net *net;u32 num_created_acts;
@@ -122,6 +136,11 @@ struct p4tc_act *p4a_runt_find(struct net *net,void p4a_runt_prealloc_put (struct p4tc_act *act,struct tcf_p4act *p4_act); +static inline bool p4tc_pipeline_sealed (struct p4tc_pipeline *pipeline) +{ +return pipeline->p_state ==P4TC_STATE_READY ; +} +static inline int p4tc_action_destroy (struct tc_action *acts[]) {struct tc_action *acts_non_prealloc[TCA_ACT_MAX_PRIO ] = {NULL };
@@ -166,6 +185,65 @@ static inline int p4tc_action_destroy(struct tc_action *acts[])return ret; } +#define P4TC_CONTROL_PERMISSIONS (GENMASK(13, 7)) +#define P4TC_DATA_PERMISSIONS (GENMASK(6, 0)) + +#define P4TC_TABLE_DEFAULT_PERMISSIONS \ + ((GENMASK(P4TC_CTRL_PERM_C_BIT, P4TC_CTRL_PERM_D_BIT)) | \ + P4TC_CTRL_PERM_P | P4TC_CTRL_PERM_S | P4TC_DATA_PERM_R | \ + P4TC_DATA_PERM_X) + +#define P4TC_PERMISSIONS_UNINIT (1 << P4TC_PERM_MAX_BIT) + +struct p4tc_table_defact { +struct tc_action *acts[2 ]; +/* Will have two 7 bits blocks containing CRUDXPS (Create, read, update, +* delete, execute, publish and subscribe) permissions for control plane +* and data plane. The first 5 bits are for control and the next five +* are for data plane. |crudxpscrudxps| if we were to denote it as UNIX +* permission flags. +*/ +__u16 perm; +struct rcu_head rcu; +}; + +struct p4tc_table_perm { +__u16 permissions; +struct rcu_head rcu; +}; + +struct p4tc_table { +struct p4tc_template_common common; +struct list_head tbl_acts_list; +struct idr tbl_masks_idr; +struct ida tbl_prio_ida; +struct xarray tbl_profiles_xa; +struct rhltable tbl_entries; +/* Mutex that protects tbl_profiles_xa */ +struct mutex tbl_profiles_xa_lock; +struct p4tc_table_defact __rcu *tbl_dflt_hitact; +struct p4tc_table_defact __rcu *tbl_dflt_missact; +struct p4tc_table_perm __rcu *tbl_permissions; +struct p4tc_table_entry_mask __rcu **tbl_masks_array; +unsigned long __rcu *tbl_free_masks_bitmap; +/* Locks the available masks IDR which will be used when adding and +* deleting table entries. +*/ +spinlock_t tbl_masks_idr_lock; +u32 tbl_keysz; +u32 tbl_id; +u32 tbl_max_entries; +u32 tbl_max_masks; +u32 tbl_curr_num_masks; +atomic_t tbl_num_timer_profiles; +/* Accounts for how many entities refer to this table. Usually just the +* pipeline it belongs to. +*/ +refcount_t tbl_ctrl_ref; +u16 tbl_type; +u16 __pad0; +}; +struct p4tc_act_param {struct list_head head;struct rcu_head rcu;
@@ -219,6 +297,18 @@ struct p4tc_act {char fullname[ACTNAMSIZ ]; }; +struct p4tc_table_act { +struct list_head node; +struct p4tc_act *act; +u8 flags; +}; + +struct p4tc_table_timer_profile { +struct rcu_head rcu; +u64 aging_ms; +u32 profile_id; +}; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla,struct tc_action *acts[],u32 pipeid,u32 flags,struct netlink_ext_ack *extack)
@@ -286,6 +376,70 @@ static inline bool p4tc_action_put_ref(struct p4tc_act *act)return refcount_dec_not_one (&act->a_ref); } +struct p4tc_act_param *p4a_parm_find_byid (struct idr *params_idr, +const u32 param_id); +struct p4tc_act_param * +p4a_parm_find_byany (struct p4tc_act *act,const char *param_name, +const u32 param_id,struct netlink_ext_ack *extack); + +struct p4tc_table *p4tc_table_find_byany (struct p4tc_pipeline *pipeline, +const char *tblname,const u32 tbl_id, +struct netlink_ext_ack *extack); +struct p4tc_table *p4tc_table_find_byid (struct p4tc_pipeline *pipeline, +const u32 tbl_id); +int p4tc_table_try_set_state_ready (struct p4tc_pipeline *pipeline, +struct netlink_ext_ack *extack); +void p4tc_table_put_mask_array (struct p4tc_pipeline *pipeline); +struct p4tc_table *p4tc_table_find_get (struct p4tc_pipeline *pipeline, +const char *tblname,const u32 tbl_id, +struct netlink_ext_ack *extack); + +static inline bool p4tc_table_put_ref (struct p4tc_table *table) +{ +return refcount_dec_not_one (&table->tbl_ctrl_ref); +} + +struct p4tc_table_defact_params { +struct p4tc_table_defact *hitact; +struct p4tc_table_defact *missact; +struct nlattr *nla_hit; +struct nlattr *nla_miss; +}; + +int p4tc_table_init_default_acts (struct net *net, +struct p4tc_table_defact_params *dflt, +struct p4tc_table *table, +struct list_head *acts_list, +struct netlink_ext_ack *extack); + +static inline bool p4tc_table_act_is_noaction (struct p4tc_table_act *table_act) +{ +return !table_act->act->common.p_id && !table_act->act->a_id; +} + +static inline bool p4tc_table_defact_is_noaction (struct tcf_p4act *p4_defact) +{ +return !p4_defact->p_id && !p4_defact->act_id; +} + +static inline void p4tc_table_defacts_acts_copy (struct p4tc_table_defact *dst, +struct p4tc_table_defact *src) +{ + dst->acts[0 ] = src->acts[0 ]; + dst->acts[1 ] =NULL ; +} + +void p4tc_table_replace_default_acts (struct p4tc_table *table, +struct p4tc_table_defact_params *dflt, +bool lock_rtnl); + +struct p4tc_table_perm * +p4tc_table_init_permissions (struct p4tc_table *table,u16 permissions, +struct netlink_ext_ack *extack); +void p4tc_table_replace_permissions (struct p4tc_table *table, +struct p4tc_table_perm *tbl_perm, +bool lock_rtnl); +struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);void p4a_runt_prealloc_reference (struct p4tc_act *act,struct tcf_p4act *p4act);
@@ -296,5 +450,6 @@ p4a_runt_parm_init(struct net *net, struct p4tc_act *act,#define to_pipeline (t) ((struct p4tc_pipeline *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t) +#define p4tc_to_table (t) ((struct p4tc_table *)t)#endif
include/uapi/linux/p4tc.h +120 -0
138 lines not on this page · raw message
net/sched/p4tc/Makefile +1 -1
@@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ - p4tc_action.o + p4tc_action.o p4tc_table.o
net/sched/p4tc/p4tc_action.c +2 -2
@@ -678,13 +678,13 @@ p4a_parm_find_byname(struct idr *params_idr, const char *param_name)return NULL ; } -static struct p4tc_act_param * +struct p4tc_act_param *p4a_parm_find_byid (struct idr *params_idr,const u32 param_id) {return idr_find (params_idr, param_id); } -static struct p4tc_act_param * +struct p4tc_act_param *p4a_parm_find_byany (struct p4tc_act *act,const char *param_name,const u32 param_id,struct netlink_ext_ack *extack) {
net/sched/p4tc/p4tc_pipeline.c +16 -7
@@ -76,6 +76,7 @@ static const struct nla_policy tc_pipeline_policy[P4TC_PIPELINE_MAX + 1] = {static void p4tc_pipeline_destroy (struct p4tc_pipeline *pipeline) {idr_destroy (&pipeline->p_act_idr); +idr_destroy (&pipeline->p_tbl_idr);kfree (pipeline); }
@@ -98,9 +99,13 @@ static void p4tc_pipeline_teardown(struct p4tc_pipeline *pipeline,struct net *net = pipeline->net;struct p4tc_pipeline_net *pipe_net =net_generic (net, pipeline_net_id);struct net *pipeline_net =maybe_get_net (net); -unsigned long iter_act_id; +unsigned long iter_act_id, tmp; +struct p4tc_table *table;struct p4tc_act *act; -unsigned long tmp; +unsigned long tbl_id; + +idr_for_each_entry_ul (&pipeline->p_tbl_idr, table, tmp, tbl_id) + table->common.ops->put (pipeline, &table->common, extack);idr_for_each_entry_ul (&pipeline->p_act_idr, act, tmp, iter_act_id) act->common.ops->put (pipeline, &act->common, extack);
@@ -153,22 +158,23 @@ static int __p4tc_pipeline_put(struct p4tc_pipeline *pipeline,static int pipeline_try_set_state_ready (struct p4tc_pipeline *pipeline,struct netlink_ext_ack *extack) { +int ret; +if (pipeline->curr_tables != pipeline->num_tables) {NL_SET_ERR_MSG (extack,"Must have all table defined to update state to ready" );return -EINVAL ; } + ret =p4tc_table_try_set_state_ready (pipeline, extack); +if (ret <0 ) +return ret; + pipeline->p_state =P4TC_STATE_READY ;return true; } -static bool p4tc_pipeline_sealed (struct p4tc_pipeline *pipeline) -{ -return pipeline->p_state ==P4TC_STATE_READY ; -} -struct p4tc_pipeline *p4tc_pipeline_find_byid (struct net *net,const u32 pipeid) {struct p4tc_pipeline_net *pipe_net;
@@ -262,6 +268,9 @@ p4tc_pipeline_create(struct net *net, struct nlmsghdr *n,idr_init (&pipeline->p_act_idr); +idr_init (&pipeline->p_tbl_idr); + pipeline->curr_tables =0 ; + pipeline->num_created_acts =0 ; pipeline->p_state =P4TC_STATE_NOT_READY ;
net/sched/p4tc/p4tc_table.c new file +1596 -0
1,596 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 12/15 p4tc: add runtime table entry create and update
In reply to Jamal Hadi SalimTables are conceptually similar to TCAMs and this implementation could be labelled as an "algorithmic" TCAM. Tables have a key of a specific size, maximum number of entries and masks allowed. The basic P4 key types are supported (exact, LPM, ternary, and ranges) although the kernel side is oblivious of all that and sees only bit blobs which it masks before a lookup is performed.
This commit allows users to create and update table _entries_ (templates were described in earlier patch). We'll add get, delete and dump in the next commit
Note that table entries can only be created once the pipeline template is sealed.
For example, a user issuing the following command:
tc p4ctrl create myprog/table/cb/tname \
dstAddr 10.10.10.0/24 srcAddr 192.168.0.0/16 prio 16 \
action send param port port1
indicates we are creating a table entry in table "cb/tname" (a table residing in control block "cb") on a pipeline named "myprog".
User space tc will create a key which has a value of 0x0a0a0a00c0a00000 (10.10.10.0 concatenated with 192.168.0.0) and a mask value of 0xffffff00ffff0000 (/24 concatenated with /16) that will be sent to the kernel. In addition a priority field of 16 is passed to the kernel as well as the action definition.
The priority field is needed to disambiguate in case two entries match. In that case, the kernel will choose the one with lowest priority number.
If the user wanted to, for example, update our just created entry with an action, they'd issue the following command:
tc p4ctrl update myprog/table/cb/tname srcAddr 10.10.10.0/24 \
dstAddr 192.168.0.0/16 prio 16 action send param port port5
In this case, the user needs to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry.
__Table Attributes__
Just for context, these are the table attributes (presented in the table template commit):
- Timer profiles: The timer profiles for the table entries belonging to the table. These profiles contain the supported aging values for the table. There are currently 4 different built-in profiles ID0=30s(default), ID1=60s, ID2=120s and ID3=180s.
The user can override the defaults by specifying the number of profiles per table. In that case, the kernel will generate the specified number of profiles with their aging values abiding by the following rule:
- Profile ID 0 - 30000ms
- Profile ID(n) - Profile ID(n - 1) + 30000ms
So, for example, if the user specified num_timer_profiles as 5, the profile IDs and aging values would be the following:
- Profile ID 0 - 30000ms (default profile)
- Profile ID 1 - 60000ms
- Profile ID 2 - 90000ms
- Profile ID 3 - 120000ms
- Profile ID 4 - 150000ms
The values of the different profiles could be changed at runtime. The default profile is used if the user specifies an out of range value.
- Match key type: The match type of the table key (exact, LPM, ternary, etc).
Note in P4 a key may constitute multiple (sub)key types, eg matching on srcip using a prefix and an exact match for dstip. In such a case it is up to the compiler to generalize the key used (For example in this case the overall key may endup being LPM or ternary).
- Direct Counter: Table counter instances used directly by the table, when specified in the P4 program, there will be one counter per entry.
- Direct Meter: Table meter instances used directly by the table, when specified in the P4 program, there will be one meter per entry.
- CRUDXPS Permissions both for specific entries and tables. The permissions are applicable to both the control plane and the datapath (see "Table Permissions" further below).
- Allowed Actions List. This will be a list of all possible actions that can be added to table entries that are added to the specified table.
- Action profiles. When defined in a P4 program, action profiles provide a mechanism to share action instances.
- Actions Selectors. When defined in a P4 program can be used to select the execution of one or more action instance selected at table lookup time by using a hash computation.
- Default hit action. When a default hit action is defined it is used when a matched table entry did not define an action. Depending on the P4 program the default hit action can be updated at runtime (in addition to being specified in the template).
- Default miss action. When a default miss action is defined it is used when a lookup that table fails.
- Action scope. In addition to actions being annotated as default hit or miss they can also be annotated to be either specific to a table of globally available to multiple tables within the same P4 program.
- Max entries. This is an upper bound for number of entries a specific table allows.
- Num masks. In the case of LPM or ternary matches, this defines the maximum allowed masks for that table.
- Timers. When defined in a P4 program, each entry has an associated timer. Depending on the programmed timer profile (see above), an entry gets a timeout. The timer attribute specifies the behavior of a table entry expiration.
The timer is refreshed every time there's a hit. After an idle period the P4 program can define using this attribute, whether it wants to have an event generated to user space (and have user space delete the entry), or whether it wants the kernel to delete it and send the event to announce the deletion. The default in P4TC is to both delete and generate an event.
- per entry "static" vs "dynamic" entry. By default all entries created from the control plane are "static" unless otherwise specified. All entries added from datapath are "dynamic" unless otherwise specified. "Dynamic" entries are subject to deletion when idle (subject to the rules specified in "Timers" above).
__Table Entry Permissions__
Whereas templates can define the default permissions for tables, runtime has ability to define table entry permissions as they are added.
To reiterate there are two types of permissions:
- Table permissions which are a property of the table (think directory in file systems). These are set by the template (see earlier patch on table template).
- Table entry permissions which are specific to a table entry (think a file in a directory). This patch describes those permissions.
Furthermore in both cases the permissions are split into datapath vs control path. The template definition can set either one. For example, one could allow for adding table entries by the datapath in case of PNA add-on-miss is needed.
By default tables entries have control plane RUD, meaning the control plane can Read, Update or Delete entries. By default, as well, the control plane can create new entries unless specified otherwise by the template.
Lets see an example of which creates the table "cb/tname" at template time:
tc p4template create table/aP4proggie/cb/tname tblid 1 keysz 64 \
permissions 0x3C24 ...Above is setting the table tname's permission to be 0x3C24 is equivalent to CRUD----R--X-- meaning:
The control plane can Create, Read, Update, Delete The datapath can only Read and Execute table entries. If one was to dump this table with:
tc -j p4template get table/aP4proggie/cb/tname | jq .
The output would be the following:
[
{
"obj": "table",
"pname": "aP4proggie",
"pipeid": 22
},
{
"templates": [
{
"tblid": 1,
"tname": "cb/tname",
"keysz": 64,
"max_entries": 256,
"masks": 8,
"entries": 0,
"permissions": "CRUD----R--X--",
"table_type": "exact",
"acts_list": []
}
] }
]
The expressed permissions above are probably the most practical for most use cases.
__Constant Tables And P4-programmed Defined Entries__
If one wanted to restrict the table to be an equivalent to a "const" then the permissions would be set to be: -R------R--X-- by the template.
In such a case, typically the P4 program will have some entries defined (see the famous P4 calculator example). The "initial entries" specified in the P4 program will have to be added by the template (as generated by the compiler), as such:
tc p4template create table/aP4proggie/cb/tname \
entry srcAddr 10.10.10.10/24 dstAddr 1.1.1.0/24 prio 17
This table cannot be updated at runtime. Any attempt to add an entry of a table which is read-only at runtime will get a "permission denied" response back from the kernel.
Note: If one was to create an equivalent for PNA add-on-miss feature for this table, then the template would issue table permissions as:
-R-----CR--X-- PNA doesn't specify whether the datapath can also delete or update entries, but if it did then more appropriate permissions will be:
-R-----CRUDX--
__Mix And Match of RW vs Constant Entries__
Lets look at other scenarios; lets say the table has CRUD----R--X-- permissions as defined by the template...
At runtime the user could add entries which are "const" - by specifying the entry's permission as -R------R--X-- example:
tc p4ctrl create aP4proggie/table/cb/tname srcAddr 10.10.10.10/24 \
dstAddr 1.1.1.0/24 prio 17 permissions 0x1024 action drop
or not specify permissions at all as such:
tc p4ctrl create aP4proggie/table/cb/tname srcAddr 10.10.10.10/24 \
dstAddr 1.1.1.0/24 prio 17 action drop
in which case the table's permissions defined at template time(CRUD----R--X--) are assumed; meaning the table entry can be deleted or updated by the control plane.
__Entries permissions Allowed On A Table Entry Creation At Runtime__
When entries are created without specifying permissions they inherit the table permissions set by the template.
When an entry is added with expressed permissions it has at most to have what the template table definition expressed but could ask for less permission. For example, assuming a table with templated specified permissions of CR-D----R--X--:
An entry created at runtime with permission of -R------R--X-- is allowed but an entry with -RUD----R--X-- will be rejected.
--- include/net/p4tc.h | 113 ++ include/uapi/linux/p4tc.h | 72 ++ include/uapi/linux/rtnetlink.h | 9 + net/sched/p4tc/Makefile | 3 +- net/sched/p4tc/p4tc_runtime_api.c | 82 ++ net/sched/p4tc/p4tc_table.c | 92 +- net/sched/p4tc/p4tc_tbl_entry.c | 1818 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 4 +- security/selinux/nlmsgtab.c | 6 +- 9 files changed, 2187 insertions(+), 12 deletions(-) create mode 100644 net/sched/p4tc/p4tc_runtime_api.c create mode 100644 net/sched/p4tc/p4tc_tbl_entry.c
include/net/p4tc.h +113 -0
@@ -124,6 +124,11 @@ static inline bool p4tc_pipeline_get(struct p4tc_pipeline *pipeline)return refcount_inc_not_zero (&pipeline->p_ctrl_ref); } +static inline void p4tc_pipeline_put_ref (struct p4tc_pipeline *pipeline) +{ +refcount_dec (&pipeline->p_ctrl_ref); +} +void p4tc_pipeline_put (struct p4tc_pipeline *pipeline);struct p4tc_pipeline * p4tc_pipeline_find_byany_unsealed(struct net *net, const char *p_name,
@@ -221,6 +226,7 @@ struct p4tc_table {struct rhltable tbl_entries;/* Mutex that protects tbl_profiles_xa */ struct mutex tbl_profiles_xa_lock; +struct p4tc_table_entry *tbl_entry;struct p4tc_table_defact __rcu *tbl_dflt_hitact;struct p4tc_table_defact __rcu *tbl_dflt_missact;struct p4tc_table_perm __rcu *tbl_permissions;
@@ -236,6 +242,8 @@ struct p4tc_table {u32 tbl_max_masks;u32 tbl_curr_num_masks;atomic_t tbl_num_timer_profiles; +/* Accounts for how many entries this table has */ +atomic_t tbl_nelems;/* Accounts for how many entities refer to this table. Usually just the * pipeline it belongs to. */
@@ -309,6 +317,76 @@ struct p4tc_table_timer_profile {u32 profile_id; }; +extern const struct rhashtable_params entry_hlt_params; + +struct p4tc_table_entry ; +struct p4tc_table_entry_work { +struct work_struct work; +struct p4tc_pipeline *pipeline; +struct p4tc_table_entry *entry; +struct p4tc_table *table; +u16 who_deleted; +bool send_event; +}; + +struct p4tc_table_entry_key { +u32 keysz; +/* Key start */ +u32 maskid; +unsigned char fa_key[]__aligned(8) ; +}; + +struct p4tc_table_entry_value { +struct tc_action *acts[2 ]; +/* Accounts for how many entities are referencing it +* eg: Data path, one or more control path and timer. +*/ +u32 prio; +refcount_t entries_ref; +u32 permissions; +u32 __pad0; +struct p4tc_table_entry_tm __rcu *tm; +struct p4tc_table_entry_work *entry_work; +u64 aging_ms; +struct hrtimer entry_timer; +bool tmpl_created; +}; + +struct p4tc_table_entry_mask { +struct rcu_head rcu; +u32 sz; +u32 mask_index; +/* Accounts for how many entries are using this mask */ +refcount_t mask_ref; +u32 mask_id; +unsigned char fa_value[]__aligned(8) ; +}; + +struct p4tc_table_entry { +struct rcu_head rcu; +struct rhlist_head ht_node; +struct p4tc_table_entry_key key; +/* fallthrough: key data + value */ +}; + +#define P4TC_KEYSZ_BYTES (bits) (round_up(BITS_TO_BYTES(bits), 8)) + +static inline void *p4tc_table_entry_value (struct p4tc_table_entry *entry) +{ +return entry->key.fa_key +P4TC_KEYSZ_BYTES (entry->key.keysz); +} + +static inline struct p4tc_table_entry_work * +p4tc_table_entry_work (struct p4tc_table_entry *entry) +{ +struct p4tc_table_entry_value *value =p4tc_table_entry_value (entry); + +return value->entry_work; +} + +extern const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ]; +extern const struct nla_policy p4tc_policy[P4TC_MAX +1 ]; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla,struct tc_action *acts[],u32 pipeid,u32 flags,struct netlink_ext_ack *extack)
@@ -422,6 +500,21 @@ static inline bool p4tc_table_defact_is_noaction(struct tcf_p4act *p4_defact)return !p4_defact->p_id && !p4_defact->act_id; } +static inline void p4tc_table_defact_destroy (struct p4tc_table_defact *defact) +{ +if (defact) { +if (defact->acts[0 ]) { +struct tcf_p4act *dflt =to_p4act (defact->acts[0 ]); + +if (p4tc_table_defact_is_noaction (dflt)) +kfree (dflt); +else +p4tc_action_destroy (defact->acts); + } +kfree (defact); + } +} +static inline void p4tc_table_defacts_acts_copy (struct p4tc_table_defact *dst,struct p4tc_table_defact *src) {
@@ -439,6 +532,21 @@ p4tc_table_init_permissions(struct p4tc_table *table, u16 permissions,void p4tc_table_replace_permissions (struct p4tc_table *table,struct p4tc_table_perm *tbl_perm,bool lock_rtnl); +void p4tc_table_entry_destroy_hash (void *ptr,void *arg); + +struct p4tc_table_entry * +p4tc_tmpl_table_entry_cu (struct net *net,struct nlattr *arg, +struct p4tc_pipeline *pipeline, +struct p4tc_table *table, +struct netlink_ext_ack *extack); +int p4tc_tbl_entry_root (struct net *net,struct sk_buff *skb, +struct nlmsghdr *n,int cmd, +struct netlink_ext_ack *extack); +int p4tc_tbl_entry_dumpit (struct net *net,struct sk_buff *skb, +struct netlink_callback *cb, +struct nlattr *arg,char *p_name); +int p4tc_tbl_entry_fill (struct sk_buff *skb,struct p4tc_table *table, +struct p4tc_table_entry *entry,u32 tbl_id);struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);
@@ -448,6 +556,11 @@ struct p4tc_act_param * p4a_runt_parm_init(struct net *net, struct p4tc_act *act, struct nlattr *nla, struct netlink_ext_ack *extack); +static inline bool p4tc_runtime_msg_is_update (struct nlmsghdr *n) +{ +return n->nlmsg_type ==RTM_P4TC_UPDATE ; +} +#define to_pipeline (t) ((struct p4tc_pipeline *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t)#define p4tc_to_table (t) ((struct p4tc_table *)t)
include/uapi/linux/p4tc.h +72 -0
102 lines not on this page · raw message
include/uapi/linux/rtnetlink.h +9 -0
@@ -203,6 +203,15 @@ enum { RTM_UPDATEP4TEMPLATE,#define RTM_UPDATEP4TEMPLATE RTM_UPDATEP4TEMPLATE + RTM_P4TC_CREATE = 128, +#define RTM_P4TC_CREATE RTM_P4TC_CREATE + RTM_P4TC_DEL, +#define RTM_P4TC_DEL RTM_P4TC_DEL + RTM_P4TC_GET, +#define RTM_P4TC_GET RTM_P4TC_GET + RTM_P4TC_UPDATE, +#define RTM_P4TC_UPDATE RTM_P4TC_UPDATE + __RTM_MAX,#define RTM_MAX (((__RTM_MAX + 3) & ~3) - 1) };
net/sched/p4tc/Makefile +2 -1
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ - p4tc_action.o p4tc_table.o + p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ + p4tc_runtime_api.o
net/sched/p4tc/p4tc_runtime_api.c new file +82 -0
82 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +84 -8
157 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c new file +1818 -0
1,818 lines not on this page · raw message
net/sched/p4tc/p4tc_tmpl_api.c +2 -2
@@ -27,12 +27,12 @@#include <net/netlink.h> #include <net/flow_offload.h> -static const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ] = { +const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ] = { [P4TC_ROOT ] = { .type =NLA_NESTED }, [P4TC_ROOT_PNAME ] = { .type =NLA_STRING , .len =P4TC_PIPELINE_NAMSIZ }, }; -static const struct nla_policy p4tc_policy[P4TC_MAX +1 ] = { +const struct nla_policy p4tc_policy[P4TC_MAX +1 ] = { [P4TC_PATH ] = { .type =NLA_BINARY , .len =P4TC_PATH_MAX *sizeof (u32) }, [P4TC_PARAMS ] = { .type =NLA_NESTED },
security/selinux/nlmsgtab.c +5 -1
@@ -98,6 +98,10 @@ static const struct nlmsg_perm nlmsg_route_perms[] = { { RTM_DELP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_GETP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_READ }, { RTM_UPDATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_CREATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_DEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_GET, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_P4TC_UPDATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, }; static const struct nlmsg_perm nlmsg_tcpdiag_perms[] = {
@@ -181,7 +185,7 @@ int selinux_nlmsg_lookup(u16 sclass, u16 nlmsg_type, u32 *perm)* structures at the top of this file with the new mappings * before updating the BUILD_BUG_ON() macro! */ -BUILD_BUG_ON (RTM_MAX != (RTM_CREATEP4TEMPLATE +3 )); +BUILD_BUG_ON (RTM_MAX != (RTM_P4TC_CREATE +3 )); err =nlmsg_perm (nlmsg_type, perm, nlmsg_route_perms,sizeof (nlmsg_route_perms));break ;
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 13/15 p4tc: add runtime table entry get, delete, flush and dump
In reply to Jamal Hadi SalimThis commit allows users to get, delete, flush and dump table _entries_ (templates were described in earlier patch and so were create and update).
If the user wants to, for example, read a table entry from a table (nh_table) which has as key nh_index (bit32), belongs to pipeline (routing) and has as its key an nh_index of 1, they'd issue the following command:
tc p4ctrl get routing/table/Main/nh_table nh_index 1
Which will give us the following output:
$TC p4ctrl get routing/table/Main/nh_table
pipeline: routing(id 1)
table: Main/nh_table(id 1)entry priority 64000[permissions -RUD-PS-R--X--]
entry key
nh_index id:1 size:32b type:bit32 exact fieldval 1
entry actions:
action order 1: routing/Main/set_nh index 2 ref 1 bind 1
params:
dmac type macaddr value: 13:37:13:37:13:37 id 1
port type dev value: port1 id 2
created by: tc (id 2)
dynamic false
created 20338 sec used 39 sec
Note that, as with create and update, we need to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry. Also, in this case, the entry had an action which has a parameter dmac (a mac address) and port1 (a net device).
If the user wanted to delete the same table entry, they'd issue the following command:
tc p4ctrl del routing/table/Main/nh_table nh_index 1
Note that, again, we need to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry.
We can also flush all the table entries from a specific table. To flush the table entries of table tname ane pipeline ptables, the user would issue the following command:
tc p4ctrl del routing/table/Main/nh_table
Likewise, we can also dump all the table entries from a specific table. To dump the table entries of table tname and pipeline myprog, the user would issue the following command:
tc p4ctrl get routing/table/Main/nh_table
Which will give the following output (in case we had one entry with nh_index key value 1):
pipeline: routing(id 1)
table: Main/nh_table(id 1)entry priority 64000[permissions -RUD-PS-R--X--]
entry key
nh_index id:1 size:32b type:bit32 exact fieldval 1
entry actions:
action order 1: routing/Main/set_nh index 2 ref 1 bind 1
params:
dmac type macaddr value: 13:37:13:37:13:37 id 1
port type dev value: port1 id 2
created by: tc (id 2)
dynamic false
created 20338 sec used 39 sec
--- include/net/p4tc.h | 61 +- include/uapi/linux/p4tc.h | 109 ++++ net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_filter.c | 1012 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_runtime_api.c | 63 ++ net/sched/p4tc/p4tc_table.c | 118 +++- net/sched/p4tc/p4tc_tbl_entry.c | 883 ++++++++++++++++++++++++- 7 files changed, 2225 insertions(+), 23 deletions(-) create mode 100644 net/sched/p4tc/p4tc_filter.c
include/net/p4tc.h +60 -1
@@ -36,11 +36,44 @@#define P4TC_AID_IDX 1#define P4TC_PARSEID_IDX 1 +struct p4tc_filter { +struct p4tc_filter_oper *operation; +union { +struct { +u32 tbl_id; + }; + }; +u32 obj_id; +}; + +struct p4tc_filter_context { +struct p4tc_pipeline *pipeline; +union { +struct p4tc_table *table; + }; +int cmd; +u32 obj_id; +}; +struct p4tc_dump_ctx {u32 ids[P4TC_PATH_MAX ];struct rhashtable_iter *iter; +struct p4tc_filter *entry_filter; }; +enum { +P4TC_FILTER_OBJ_UNSPEC , +P4TC_FILTER_OBJ_TMPL_PIPELINE , +P4TC_FILTER_OBJ_TMPL_ACT , +P4TC_FILTER_OBJ_TMPL_TABLE , +P4TC_FILTER_OBJ_TMPL_EXT , +P4TC_FILTER_OBJ_RUNTIME_TABLE , +P4TC_FILTER_OBJ_RUNTIME_EXT , + __P4TC_FILTER_OBJ_MAX, +}; + +#define P4TC_FILTER_OBJ_MAX (__P4TC_FILTER_OBJ_MAX - 1) +struct p4tc_template_common ;struct p4tc_path_nlattrs {
@@ -349,6 +382,7 @@ struct p4tc_table_entry_value {struct p4tc_table_entry_work *entry_work;u64 aging_ms;struct hrtimer entry_timer; +bool is_dyn;bool tmpl_created; };
@@ -468,6 +502,12 @@ struct p4tc_table *p4tc_table_find_byid(struct p4tc_pipeline *pipeline,int p4tc_table_try_set_state_ready (struct p4tc_pipeline *pipeline,struct netlink_ext_ack *extack);void p4tc_table_put_mask_array (struct p4tc_pipeline *pipeline); + +static inline int p4tc_table_get (struct p4tc_table *table) +{ +return refcount_inc_not_zero (&table->tbl_ctrl_ref); +} +struct p4tc_table *p4tc_table_find_get (struct p4tc_pipeline *pipeline,const char *tblname,const u32 tbl_id,struct netlink_ext_ack *extack);
@@ -532,6 +572,15 @@ p4tc_table_init_permissions(struct p4tc_table *table, u16 permissions,void p4tc_table_replace_permissions (struct p4tc_table *table,struct p4tc_table_perm *tbl_perm,bool lock_rtnl); +int p4tc_table_timer_profile_update (struct p4tc_table *table, +struct nlattr *nla, +struct netlink_ext_ack *extack); +struct p4tc_table_timer_profile * +p4tc_table_timer_profile_find_byaging (struct p4tc_table *table, +u64 aging_ms); +struct p4tc_table_timer_profile * +p4tc_table_timer_profile_find (struct p4tc_table *table,u32 profile_id); +void p4tc_table_entry_destroy_hash (void *ptr,void *arg); struct p4tc_table_entry *
@@ -546,7 +595,17 @@ int p4tc_tbl_entry_dumpit(struct net *net, struct sk_buff *skb,struct netlink_callback *cb, struct nlattr *arg, char *p_name);int p4tc_tbl_entry_fill (struct sk_buff *skb,struct p4tc_table *table, -struct p4tc_table_entry *entry,u32 tbl_id); +struct p4tc_table_entry *entry,u32 tbl_id, +u16 who_deleted); +void p4tc_tbl_entry_mask_key (u8 *masked_key,u8 *key,const u8 *mask, +u32 masksz); + +struct p4tc_filter * +p4tc_filter_build (struct p4tc_filter_context *ctx, +struct nlattr *nla,struct netlink_ext_ack *extack); +bool p4tc_filter_exec (struct p4tc_filter *filter, +struct p4tc_table_entry *entry); +void p4tc_filter_destroy (struct p4tc_filter *filter);struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);
include/uapi/linux/p4tc.h +109 -0
126 lines not on this page · raw message
net/sched/p4tc/Makefile +1 -1
@@ -2,4 +2,4 @@ obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ - p4tc_runtime_api.o + p4tc_filter.o p4tc_runtime_api.o
net/sched/p4tc/p4tc_filter.c new file +1012 -0
1,012 lines not on this page · raw message
net/sched/p4tc/p4tc_runtime_api.c +63 -0
81 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +112 -6
142 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c +868 -15
1,039 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v16 14/15 p4tc: add set of P4TC table kfuncs
In reply to Jamal Hadi SalimWe add an initial set of kfuncs to allow interactions from eBPF programs to the P4TC domain.
- bpf_p4tc_tbl_read: Used to lookup a table entry from a BPF program installed in TC. To find the table entry we take in an skb, the pipeline ID, the table ID, a key and a key size.
We use the skb to get the network namespace structure where all the pipelines are stored. After that we use the pipeline ID and the table ID, to find the table. We then use the key to search for the entry. We return an entry on success and NULL on failure.
- xdp_p4tc_tbl_read: Used to lookup a table entry from a BPF program installed in XDP. To find the table entry we take in an xdp_md, the pipeline ID, the table ID, a key and a key size.
We use struct xdp_md to get the network namespace structure where all the pipelines are stored. After that we use the pipeline ID and the table ID, to find the table. We then use the key to search for the entry. We return an entry on success and NULL on failure.
- bpf_p4tc_entry_create: Used to create a table entry from a BPF program installed in TC. To create the table entry we take an skb, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_create: Used to create a table entry from a BPF program installed in XDP. To create the table entry we take an xdp_md, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_create_on_miss: conforms to PNA "add on miss". First does a lookup using the passed key and upon a miss will add the entry to the table.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_create_on_miss: conforms to PNA "add on miss". First does a lookup using the passed key and upon a miss will add the entry to the table.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_update: Used to update a table entry from a BPF program installed in TC. To update the table entry we take an skb, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_update: Used to update a table entry from a BPF program installed in XDP. To update the table entry we take an xdp_md, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_delete: Used to delete a table entry from a BPF program installed in TC. To delete the table entry we take an skb, the pipeline ID, the table ID, a key and a key size.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_delete: Used to delete a table entry from a BPF program installed in XDP. To delete the table entry we take an xdp_md, the pipeline ID, the table ID, a key and a key size.
We return 0 on success and a negative errno on failure
Note:
All P4 objects are owned and reside on the P4TC side. IOW, they are controlled via TC netlink interfaces and their resources are managed (created, updated, freed, etc) by the TC side. As an example, the structure p4tc_table_entry_act is returned to the ebpf side on table lookup. On the TC side that struct is wrapped around p4tc_table_entry_act_bpf_kern. A multitude of these structure p4tc_table_entry_act_bpf_kern are preallocated (to match the P4 architecture, patch #9 describes some of the subtleties involved) by the P4TC control plane and put in a kernel pool. Their purpose is to hold the action parameters for either a table entry, a global per-table "miss" and "hit" action, etc - which are instantiated and updated via netlink per runtime requests. An instance of the p4tc_table_entry_act_bpf_kern.p4tc_table_entry_act is returned to ebpf when there is a un/successful table lookup depending on how the P4 program is written. When the table entry is deleted the instance of the struct p4tc_table_entry_act_bpf_kern is recycled to the pool to be reused for a future table entry. The only time the pool memory is released is when the pipeline is deleted.
--- include/linux/bitops.h | 1 + include/net/p4tc.h | 93 ++++++- include/net/tc_act/p4tc.h | 51 ++++ include/uapi/linux/p4tc.h | 2 + net/sched/p4tc/Makefile | 1 + net/sched/p4tc/p4tc_action.c | 71 +++++ net/sched/p4tc/p4tc_bpf.c | 360 +++++++++++++++++++++++++ net/sched/p4tc/p4tc_pipeline.c | 43 +++ net/sched/p4tc/p4tc_table.c | 50 +++- net/sched/p4tc/p4tc_tbl_entry.c | 458 ++++++++++++++++++++++++++++++-- 10 files changed, 1094 insertions(+), 36 deletions(-) create mode 100644 net/sched/p4tc/p4tc_bpf.c
include/linux/bitops.h +1 -0
@@ -19,6 +19,7 @@#define BITS_TO_LONGS (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))#define BITS_TO_U64 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u64))#define BITS_TO_U32 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u32)) +#define BITS_TO_U16 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u16))#define BITS_TO_BYTES (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(char))#define BYTES_TO_BITS (nb) ((nb) * BITS_PER_BYTE)
include/net/p4tc.h +90 -3
146 lines not on this page · raw message
include/net/tc_act/p4tc.h +51 -0
65 lines not on this page · raw message
include/uapi/linux/p4tc.h +2 -0
@@ -19,6 +19,8 @@ struct p4tcmsg {#define P4TC_MINTABLES_COUNT 0#define P4TC_MSGBATCH_SIZE 16 +#define P4TC_ACT_MAX_NUM_PARAMS P4TC_MSGBATCH_SIZE +#define P4TC_MAX_KEYSZ 512#define P4TC_DEFAULT_NUM_PREALLOC 16
net/sched/p4tc/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ p4tc_filter.o p4tc_runtime_api.o +obj-$(CONFIG_DEBUG_INFO_BTF) += p4tc_bpf.o
net/sched/p4tc/p4tc_action.c +71 -0
133 lines not on this page · raw message
net/sched/p4tc/p4tc_bpf.c new file +360 -0
360 lines not on this page · raw message
net/sched/p4tc/p4tc_pipeline.c +43 -0
55 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +46 -4
129 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c +429 -29
517 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v16 15/15 p4tc: add P4 classifier
In reply to Jamal Hadi SalimIntroduce P4 tc classifier. The main task of this classifier is to manage the lifetime of pipeline instances across one or more netdev ports.
Note: a defined pipeline may be instantiated multiple times across one or more tc chains and different priorities.
Note that part or whole of the P4 pipeline could reside in tc, XDP or even hardware depending on how the P4 program was compiled. This classifier only deals with tc layer.
To use the P4 classifier you must specify a pipeline name that will be associated to the filter instance, a s/w parser (eBPF) and datapath P4 control block program (eBPF) program. Although this patchset does not deal with offloads, it is also possible to load the h/w part using this filter. We will illustrate a few examples further below to clarify. Please treat the illustrated split as an example - there are probably more pragmatic approaches to splitting the pipeline; however, regardless of where the different pieces of the pipeline are placed (tc, XDP, HW) and what each layer will implement (what part of the pipeline) - these examples are merely showing what is possible.
The pipeline is assumed to have already been created via a template.
For example, if we were to add a filter to ingress of a group of netdevs (tc block 22) and associate it to P4 pipeline simple_l3 we could issue the following command:
tc filter add block 22 parent ffff: protocol all prio 6 p4 pname simple_l3 \
action bpf obj $PARSER.o ... \
action bpf obj $PROGNAME.o section p4tc/main
The above uses the classical tc action mechanism in which the first action runs the P4 parser and if that goes well then the P4 control block is executed. Note, although not shown above, one could also append the command line with other traditional tc actions.
Given one of the objectives of this classifier is to manage the lifetime of the p4 program and said program may be split across tc:xdp:hardware we allow specification of where the xdp (and in the future hardware) programs can be found. For this reason when instantiating one could specify where the associated XDP program using they syntax "prog type xdp progname", where progname refers to the XDP ebpf program name. The control plane side (below we show iproute2) will be responsible for loading the XDP program. The kernel is unaware of the XDP side.
There is an ongoing discussion in the P4TC community biweekly meetings which is likely going to have us add another location definition "prog type hw" which will specify the hardware object file name and other related attributes. The current discussion is that this h/w piece will go via the p4 classifier.
An example using xdp and tc:
tc filter add dev $P0 ingress protocol all prio 1 p4 pname simple_l3 \
prog type xdp obj $PARSER.o section p4tc/parse-xdp \
action bpf obj $PROGNAME.o section p4tc/main
In this case, the parser will be executed in the XDP layer and the rest of P4 control block as a tc action.
For illustration sake, the hw one looks as follows (please note there's still a lot of discussions going on in the meetings - the example is here merely to illustrate the tc filter functionality):
tc filter add block 22 ingress protocol all prio 1 p4 pname simple_l3 \
prog type hw filename "mypnameprog.o" ... \
prog type xdp obj $PARSER.o section p4tc/parse-xdp \
action bpf obj $PROGNAME.o section p4tc/main
The theory of operations is as follows:
================================1. PARSING================================
The packet first encounters the parser.
The parser is implemented in ebpf residing either at the TC or XDP level. The parsed header values are stored in a shared per-cpu eBPF map. When the parser runs at XDP level, we load it into XDP using the control plane (tc filter command) and pin it to a file.
=============================2. ACTIONS=============================
In the above example, the P4 program (minus the parser) is encoded in an action($PROGNAME.o). It should be noted that classical tc actions continue to work:
IOW, someone could decide to add a mirred action to mirror all packets after or before the ebpf action.
tc filter add dev $P0 parent ffff: protocol all prio 6 p4 pname simple_l3 \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main \
action mirred egress mirror index 1 dev $P1 \
action bpf obj $ANOTHERPROG.o section mysect/section-1It should also be noted that it is feasible to split some of the ingress datapath into XDP first and more into TC later (as was shown above for example where the parser runs at XDP level). YMMV. Regardless of choice of which scheme to use, none of these will affect UAPI. It will all depend on whether you generate code to load on XDP vs tc, etc. We expect the compiler to evolve over time (but that has nothing to do with the kernel part).
--- include/uapi/linux/pkt_cls.h | 14 ++ net/sched/Kconfig | 12 ++ net/sched/Makefile | 1 + net/sched/cls_p4.c | 305 +++++++++++++++++++++++++++++++++++ net/sched/p4tc/Makefile | 4 +- net/sched/p4tc/trace.c | 10 ++ net/sched/p4tc/trace.h | 44 +++++ 7 files changed, 389 insertions(+), 1 deletion(-) create mode 100644 net/sched/cls_p4.c create mode 100644 net/sched/p4tc/trace.c create mode 100644 net/sched/p4tc/trace.h
include/uapi/linux/pkt_cls.h +14 -0
@@ -706,6 +706,20 @@ enum {#define TCA_MATCHALL_MAX (__TCA_MATCHALL_MAX - 1) +/* P4 classifier */ + +enum { +TCA_P4_UNSPEC , +TCA_P4_CLASSID , +TCA_P4_ACT , +TCA_P4_PNAME , +TCA_P4_PIPEID , +TCA_P4_PAD , + __TCA_P4_MAX, +}; + +#define TCA_P4_MAX (__TCA_P4_MAX - 1) +/* Extended Matches */ struct tcf_ematch_tree_hdr {
net/sched/Kconfig +12 -0
@@ -565,6 +565,18 @@ config NET_CLS_MATCHALL To compile this code as a module, choose M here: the module will be called cls_matchall. +config NET_CLS_P4 + tristate "P4 classifier" + select NET_CLS + select NET_P4TC + help + If you say Y here, you will be able to bind a P4 pipeline + program. You will need to install a P4 template representing the + program successfully to use this feature. + + To compile this code as a module, choose M here: the module will + be called cls_p4. + config NET_EMATCH bool "Extended Matches" select NET_CLS
net/sched/Makefile +1 -0
@@ -72,6 +72,7 @@ obj-$(CONFIG_NET_CLS_CGROUP) += cls_cgroup.o obj-$(CONFIG_NET_CLS_BPF) += cls_bpf.o obj-$(CONFIG_NET_CLS_FLOWER) += cls_flower.o obj-$(CONFIG_NET_CLS_MATCHALL) += cls_matchall.o +obj-$(CONFIG_NET_CLS_P4) += cls_p4.o obj-$(CONFIG_NET_EMATCH) += ematch.o obj-$(CONFIG_NET_EMATCH_CMP) += em_cmp.o obj-$(CONFIG_NET_EMATCH_NBYTE) += em_nbyte.o
net/sched/cls_p4.c new file +305 -0
305 lines not on this page · raw message
net/sched/p4tc/Makefile +3 -1
@@ -1,6 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 +CFLAGS_trace.o := -I$(src) + obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ - p4tc_filter.o p4tc_runtime_api.o + p4tc_filter.o p4tc_runtime_api.o trace.o obj-$(CONFIG_DEBUG_INFO_BTF) += p4tc_bpf.o
net/sched/p4tc/trace.c new file +10 -0
@@ -0,0 +1,10 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) + +#include <net/p4tc.h> + +#ifndef __CHECKER__ + +#define CREATE_TRACE_POINTS +#include "trace.h" +EXPORT_TRACEPOINT_SYMBOL_GPL (p4_classify); +#endif
net/sched/p4tc/trace.h new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.34.1
Paolo Abeni
maintainerQuoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Paolo
Jamal Hadi Salim
maintainerQuoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.
cheers,
jamal
Jamal Hadi Salim
maintainerQuoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.
Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:
1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.png
While the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.
3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.
Now for a little rant:
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
cheers,
jamal
Quoted 4 lines · Jamal Hadi Salim
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
My Nack applies to the whole set. The kernel doesn't need this anti-feature for many reasons already explained.
Jamal Hadi Salim
maintainerQuoted 9 lines · Alexei Starovoitov
Quoted 4 lines · Jamal Hadi Salim
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
My Nack applies to the whole set. The kernel doesn't need this anti-feature for many reasons already explained.
Can you be more explicit? What else would you add to the list i posted above?
cheers,
jamal
Quoted 6 lines · Jamal Hadi Salim
Quoted 3 lines · Alexei Starovoitov
Quoted 1 line · Jamal Hadi Salim
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
My Nack applies to the whole set. The kernel doesn't need this anti-feature for many reasons already explained.
Can you be more explicit? What else would you add to the list i posted above?
Since you're refusing to work with us your only option is to mention my Nack in the cover letter and send it as a PR to Linus during the merge window.
Jamal Hadi Salim
maintainerQuoted 12 lines · Alexei Starovoitov
Quoted 10 lines · Jamal Hadi Salim
Quoted 7 lines · Alexei Starovoitov
Quoted 4 lines · Jamal Hadi Salim
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
My Nack applies to the whole set. The kernel doesn't need this anti-feature for many reasons already explained.
Can you be more explicit? What else would you add to the list i posted above?
Since you're refusing to work with us your only option
Who is "us"? ebpf? I hope you are not speaking on behalf of the net subsystem. You are entitled to your opinion (and aggression) - and there is a lot of that with you, but this should be based on technical merit not your emotions.
I summarized the reasons brought up by you and Cilium. Do you have more to add to that list? If you do, please add to it.
Quoted 2 lines
is to mention my Nack in the cover letter and send it as a PR to Linus during the merge window.
You dont get to decide that - I was talking to the networking people.
cheers,
jamal
Quoted 2 lines · Jamal Hadi Salim
You dont get to decide that - I was talking to the networking people.
You think they want net-next PR to get derailed because of this?
Jamal Hadi Salim
maintainerQuoted 6 lines · Alexei Starovoitov
Quoted 2 lines · Jamal Hadi Salim
You dont get to decide that - I was talking to the networking people.
You think they want net-next PR to get derailed because of this?
Why would it be derailed?
cheers,
jamal
Paolo Abeni
maintainerQuoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).
It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the p4tc_table_entry_act_bpf_kern entries and let p4a_runt_create_bpf() fail if the pool is empty? would that satisfy the bpf requirement?
Otherwise could p4tc force free the p4tc_table_entry_act_bpf_kern at unload time?
Thanks,
Paolo
Jamal Hadi Salim
maintainerQuoted 118 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
Quoted 6 lines
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?
Let me think about it and weigh the consequences.
Quoted 2 lines
Otherwise could p4tc force free the
p4tc_table_entry_act_bpf_kernat unload time?
This one wont work for us unfortunately. If we have entries added by the control plane with skip_sw just because the ebpf program is gone doesnt mean they disappear.
cheers,
jamal
there are use cases where no entry is loaded by the s/w datap
Quoted 5 lines
Thanks,
Paolo
Jamal Hadi Salim
maintainerQuoted 120 lines · Jamal Hadi Salim
Quoted 118 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
Quoted 9 lines
Quoted 6 lines
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?Let me think about it and weigh the consequences.
Sorry, was busy evaluating. Yes, we can enforce the memory allocation constraints such that when the ebpf program is removed any entries added by said ebpf program can be removed from the datapath.
Quoted 6 lines
Quoted 2 lines
Otherwise could p4tc force free the
p4tc_table_entry_act_bpf_kernat unload time?This one wont work for us unfortunately. If we have entries added by the control plane with
skip_swjust because the ebpf program is gone doesnt mean they disappear.
Just to clarify (the figure
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.png
should help) :
For P4 table objects, there are 3 types of entries: 1) created by control path for s/w datapath with skip_hw 2) created by control path for h/w datapath with skip_sw and 3) dynamically created by s/w datapath (ebpf) not far off from conntrack.
The only ones we can remove when the ebpf program goes away are from #3.
cheers,
jamal
Paolo Abeni
maintainerQuoted 128 lines · Jamal Hadi Salim
Quoted 127 lines · Jamal Hadi Salim
Quoted 124 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?
Quoted 6 lines
Quoted 2 lines
Let me think about it and weigh the consequences.
Sorry, was busy evaluating. Yes, we can enforce the memory allocation constraints such that when the ebpf program is removed any entries added by said ebpf program can be removed from the datapath.
I suggested the such changes based on my interpretation of this long and complex discussion, I can have missed some or many relevant points. @Alexei: could you please double check the above and eventually, hopefully, confirm that such change would lift your nacked-by?
Thanks!
Paolo
Quoted 133 lines · Paolo Abeni
Quoted 131 lines · Jamal Hadi Salim
Quoted 129 lines · Jamal Hadi Salim
Quoted 118 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
Quoted 6 lines
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?Let me think about it and weigh the consequences.
Quoted 8 lines
Quoted 3 lines
Sorry, was busy evaluating. Yes, we can enforce the memory allocation constraints such that when the ebpf program is removed any entries added by said ebpf program can be removed from the datapath.
I suggested the such changes based on my interpretation of this long and complex discussion, I can have missed some or many relevant points. @Alexei: could you please double check the above and eventually, hopefully, confirm that such change would lift your nacked-by?
No. The whole design is broken.
Remembering what was allocated by kfunc and freeing it later is not fixing the design at all.
Sorry.
Jamal Hadi Salim
maintainerQuoted 139 lines · Alexei Starovoitov
Quoted 137 lines · Paolo Abeni
Quoted 134 lines · Jamal Hadi Salim
Quoted 120 lines · Jamal Hadi Salim
Quoted 118 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
Quoted 9 lines
Quoted 6 lines
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?Let me think about it and weigh the consequences.
Sorry, was busy evaluating. Yes, we can enforce the memory allocation constraints such that when the ebpf program is removed any entries added by said ebpf program can be removed from the datapath.
Quoted 8 lines
Quoted 4 lines
I suggested the such changes based on my interpretation of this long and complex discussion, I can have missed some or many relevant points. @Alexei: could you please double check the above and eventually, hopefully, confirm that such change would lift your nacked-by?
No. The whole design is broken.
Remembering what was allocated by kfunc and freeing it later is not fixing the design at all.
Can you be a little less vague?
We are dealing with multiple domains here _including hw offloads_ and as mentioned already, a few times now, for that reason these objects belong to the P4TC domain. If it wasnt clear this diagram explains the design:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.png
IOW, P4 objects(to be specific table entries in this discussion) may be shared between s/w and/or h/w.
Note: there is no allocation done by the kfunc - it will just pick from a fixed pool of pre-allocated entries. Where is the "design broken" considering all this?
cheers,
jamal
Jamal Hadi Salim
maintainerQuoted 151 lines · Jamal Hadi Salim
Quoted 139 lines · Alexei Starovoitov
Quoted 137 lines · Paolo Abeni
Quoted 134 lines · Jamal Hadi Salim
Quoted 120 lines · Jamal Hadi Salim
Quoted 118 lines · Paolo Abeni
Quoted 88 lines · Jamal Hadi Salim
Quoted 17 lines · Jamal Hadi Salim
Quoted 7 lines · Paolo Abeni
Quoted 4 lines · Jamal Hadi Salim
The only change that v16 makes is to add a nack to patch 14 on kfuncs from Daniel and John. We strongly disagree with the nack; unfortunately I have to rehash whats already in the cover letter and has been discussed over and over and over again:
Quoted 6 lines
I feel bad asking, but I have to, since all options I have here are IMHO quite sub-optimal.
How bad would be dropping patch 14 and reworking the rest with alternative s/w datapath? (I guess restoring it from oldest revision of this series).
Quoted 17 lines
We want to keep using ebpf for the s/w datapath if that is not clear by now. I do not understand the obstructionism tbh. Are users allowed to use kfuncs as part of infra or not? My understanding is yes.
This community is getting too political and my worry is that we have corporatism creeping in like it is in standards bodies.
We started by not using ebpf. The same people who are objecting now went up in arms and insisted we use ebpf. As a member of this community, my motivation was to meet them in the middle by compromising. We invested another year to move to that middle ground. Now they are insisting we do not use ebpf because they dont like our design or how we are using ebpf or maybe it's not a use case they have any need for or some other politics. I lost track of the moving goal posts. Open source is about solving your itch. This code is entirely on TC, zero code changed in ebpf core. The new goalpost is based on emotional outrage over use of functions. The whole thing is getting extremely toxic.Paolo,
Following up since no movement for a week now;->
I am going to give benefit of doubt that there was miscommunication or misunderstanding for all the back and forth that has happened so far with the nackers. I will provide a summary below on the main points raised and then provide responses:1) "Use maps"
It doesnt make sense for our requirement. The reason we are using TC is because a) P4 has an excellent fit with TC match action paradigm b) we are targeting both s/w and h/w and the TC model caters well for this. The objects belong to TC, shared between s/w, h/w and control plane (and netlink is the API). Maybe this diagram would help:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.pngWhile the s/w part stands on its own accord (as elaborated many times), for TC which has offloads, the s/w twin is introduced before the h/w equivalent. This is what this series is doing.
2) "but ... it is not performant"
This has been brought up in regards to netlink and kfuncs. Performance is a lower priority to P4 correctness and expressibility. Netlink provides us the abstractions we need, it works with TC for both s/w and h/w offload and has a lot of knowledge base for expressing control plane APIs. We dont believe reinventing all that makes sense.
Kfuncs are a means to an end - they provide us the gluing we need to have an ebpf s/w datapath to the TC objects. Getting an extra 10-100Kpps is not a driving factor.3) "but you did it wrong, here's how you do it..."
I gave up on responding to this - but do note this sentiment is a big theme in the exchanges and consumed most of the electrons. We are _never_ going to get any consensus with statements like "tc actions are a mistake" or "use tcx".
4) "... drop the kfunc patch"
kfuncs essentially boil down to function calls. They don't require any special handling by the eBPF verifier nor introduce new semantics to eBPF. They are similar in nature to the already existing kfuncs interacting with other kernel objects such as
nf_conntrack. The precedence (repeated in conferences and email threads multiple times) is: kfuncs dont have to be sent to ebpf list or reviewed by folks in the ebpf world. And We believe that rule applies to us as well. Either kfuncs (and frankly ebpf) is infrastructure glue or it's not.Now for a little rant:
Quoted 11 lines
Open source is not a zero-sum game. Ebpf already coexists with netfilter, tc, etc and various subsystems happily.
I hope our requirement is clear and i dont have to keep justifying why P4 or relitigate over and over again why we need TC. Open source is about scratching your itch and our itch is totally contained within TC. I cant help but feel that this community is getting way too pervasive with politics and obscure agendas. I understand agendas, I just dont understand the zero-sum thinking.
My view is this series should still be applied with the nacks since it sits entirely on its own silo within networking/TC (and has nothing to do with ebpf).It's really hard for me - meaning I'll not do that - applying a series that has been so fiercely nacked, especially given that the other maintainers are not supporting it.
I really understand this is very bad for you.
Let me try to do an extreme attempt to find some middle ground between this series and the bpf folks.
My understanding is that the most disliked item is the lifecycle for the objects allocated via the kfunc(s).
If I understand correctly, the hard requirement on bpf side is that any kernel object allocated by kfunc must be released at program unload time. p4tc postpone such allocation to recycle the structure.
Quoted 9 lines
Quoted 6 lines
While there are other arguments, my reading of the past few iterations is that solving the above node should lift the nack, am I correct?
Could p4tc pre-allocate all the
p4tc_table_entry_act_bpf_kernentries and letp4a_runt_create_bpf()fail if the pool is empty? would that satisfy the bpf requirement?Let me think about it and weigh the consequences.
Sorry, was busy evaluating. Yes, we can enforce the memory allocation constraints such that when the ebpf program is removed any entries added by said ebpf program can be removed from the datapath.
Quoted 8 lines
Quoted 4 lines
I suggested the such changes based on my interpretation of this long and complex discussion, I can have missed some or many relevant points. @Alexei: could you please double check the above and eventually, hopefully, confirm that such change would lift your nacked-by?
No. The whole design is broken.
Remembering what was allocated by kfunc and freeing it later is not fixing the design at all.
Quoted 11 lines
Can you be a little less vague?
We are dealing with multiple domains here_includinghwoffloads_and as mentioned already, a few times now, for that reason these objects belong to the P4TC domain. If it wasnt clear this diagram explains the design:
https://github.com/p4tc-dev/docs/blob/main/images/why-p4tc/p4tc-runtime-pipeline.png
IOW, P4 objects(to be specific table entries in this discussion) may be shared between s/w and/or h/w.
Note: there is no allocation done by the kfunc - it will just pick from a fixed pool of pre-allocated entries. Where is the "design broken" considering all this?
Ok, not that i was expecting an answer and i think i have waited long enough.
Frankly my agreement to make the change and the time spent to validate were just an attempt to make an effort for a compromise (as we have done many many times) - but really that approach works against our requirements to control the aging/deletion/replacement policy. I dont believe there's any good faith from the nackers. For that reason that offer is off the table.
It should be noted our changes that Alexei is objecting to is more tame than for example
https://elixir.bootlin.com/linux/latest/source/net/netfilter/nf_conntrack_bpf.c#L318
We didnt see Alexeis nack on that code.
I am not asking to be given speacial treatment but it is clear we have a hole in the process currently. All i am asking for is fair treatment. At this point, given that Paolo says the patches cant be applied because of 3 cross-subsystem nacks, my suggestion on how we resolve this is to appoint a third person arbitrator. This person cannot be part of the TC or eBPF collective and has to be agreed to by both parties.
Hopefully this will introduce some new set of rules that will help the maintainers resolve such issues should they surface in the future.
I will collect all the other issues raised and my responses and create a web page so things dont get lost in the noise. I will post then and maybe send to a wider audience.
cheers,
jamal
Quoted 2 lines
cheers,
jamal
Jamal Hadi Salim
maintainerOn the NACKs on P4TC patches
As stated a few times, we strongly disagree with the nature of the Nacks from Alexei, Daniel and John. We dont think there is good ground for the Nacks.
A brief history on the P4TC patches:
We posted V1 in January 2023. The main objection then was that we needed to use eBPF. After some discussion and investigation on our part we found that using kfuncs would satisfy our goals as well as the objections raised. We posted 28 RFC patches looking for feedback from eBPF and other folks with V2 in May 2023 - these patches were not ready but we were nevertheless soliciting for feedback. By Version 7 in October/2023 we removed the RFC tag (meaning we are asking for inclusion). In Version 8 we sent the first 15 patches as series 1(following netdev rules that allow only 15 patches); 5 of these patches are trivial tc core patches. Starting with V8 and upto V14 the releases were mostly suggested changes (much thanks to folks who made suggestions for technical changes) and at one point it was a bug fix for an issue caught by our syzkaller instance.
When it seemed like Paolo was heading towards applying series 1 given the feedback, Alexei nacked patch 14 when we released V14, see:
https://lore.kernel.org/bpf/[email protected]/
V15 only change was adding Alexei's nack. V15 was followed by Daniel and then John also nacking the same patch 14. V16's only change was to add these extra Nacks.
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF. Paolo then asked if we could satisfy one of the points Alexei raised in terms of clearing table entries when an eBPF program was unloaded. We spent a week investigating and came to a conclusion that we could do it as a compromise (even though it is not something fitting to our requirements and there is existing code that we copied from doing exactly what Alexei is objecting to). Alexei rejected this offer. This puts Paolo in a difficult position because it is clear there is no compromise to be had. I feel we are in uncharted teritory.
Since we are in a quagmire, I am asking for a third party mediator to review the objections and validate if they have merit.
I have created a web page to capture all the objections raised by the 3 gents over a period of time at:
https://github.com/p4tc-dev/pushback-patches
If any of the 3 people feel i have misrepresented their objections or missed an important detail please let me know and i will fix the page.
cheers,
jamal
Jakub Kicinski
maintainerRe: On the NACKs on P4TC patches
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.
AFAICT there's some but not very strong support for P4TC, and it doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)
So from my perspective, the submission is neither technically strong enough, nor broadly useful enough to consider making questionable precedents for, i.e. to override maintainers on how their subsystems are extended.
Jamal Hadi Salim
maintainerRe: On the NACKs on P4TC patches
Quoted 15 lines · Jakub Kicinski
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.
It's not what I "like" , rather it is a requirement to support both s/w and h/w offload. The TC model is the traditional approach to deploy these models. I addressed the same comment you are making above in #1a and #1b (https://github.com/p4tc-dev/pushback-patches).
OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)
I am not sure where the parser thing comes from - the parser is generated as eBPF.
Quoted 3 lines
So from my perspective, the submission is neither technically strong enough, nor broadly useful enough to consider making questionable precedents for, i.e. to override maintainers on how their subsystems are extended.
I believe as a community nobody should just have the power to nack things just because - as i stated in the page, not even Linus. That code doesnt touch anything to do with eBPF maintainers (meaning things they have to fix when an issue shows up) neither does it "extend" as you state any ebpf code and it is all part of the networking subsystem. Sure, anybody has the right to nack but I contend that nacks should be based on technical reasons. I have listed all the objections in that page and how i have responded to them over time. Someone needs to look at those objectively and say if they are valid. The arguement made so far(By Paolo and now by you) is "we cant override maintainers on how their subsystems are used" then we are in uncharted territory, thats why i am asking for arbitration.
cheers,
jamal
RE: On the NACKs on P4TC patches
Quoted 1 line · Jakub Kicinski
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
Quoted 2 lines · Jamal Hadi Salim
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project >responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. >Sample:
https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/
FWIW, Intel is in full support of P4TC as we have stated several times in the past.
A message that is not in the archive (sent off-list or not yet received): MW4PR12MB71927C9E4B94871B45F845DF97F52@MW4PR12MB7192.namprd12.prod.outlook.com
Re: On the NACKs on P4TC patches
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
From: Jain, Vipin <[email protected]>
Sent: Friday, May 24, 2024 2:28 PM
To: Singhai, Anjali <[email protected]>; Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: Re: On the NACKs on P4TC patches
[AMD Official Use Only - AMD Internal Distribution Only]
I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC.
Happy to elaborate more if needed.
Thank you,
Vipin Jain
Sr Fellow Engineer, AMD
________________________________________
From: Singhai, Anjali <[email protected]>
Sent: Wednesday, May 22, 2024 5:30 PM
To: Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Jain, Vipin <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: RE: On the NACKs on P4TC patches
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 1 line · Jakub Kicinski
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
Quoted 2 lines · Jamal Hadi Salim
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project >responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. >Sample:
https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/
FWIW, Intel is in full support of P4TC as we have stated several times in the past.
VJ> I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC. Happy to elaborate more if needed. VJ> Thanks, Vipin
Re: On the NACKs on P4TC patches
Quoted 36 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
From: Jain, Vipin <[email protected]>
Sent: Friday, May 24, 2024 2:28 PM
To: Singhai, Anjali <[email protected]>; Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: Re: On the NACKs on P4TC patches[AMD Official Use Only - AMD Internal Distribution Only]
I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC.
Happy to elaborate more if needed.
Thank you,
Vipin Jain
Sr Fellow Engineer, AMD
________________________________________
From: Singhai, Anjali <[email protected]>
Sent: Wednesday, May 22, 2024 5:30 PM
To: Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Jain, Vipin <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: RE: On the NACKs on P4TC patchesCaution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 1 line · Jakub Kicinski
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
Quoted 8 lines
Quoted 2 lines · Jamal Hadi Salim
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project >responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. >Sample:
https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/FWIW, Intel is in full support of P4TC as we have stated several times in the past.
VJ> I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC. Happy to elaborate more if needed. VJ> Thanks, Vipin
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions. This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't see the need for P4TC in this context.
If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
RE: On the NACKs on P4TC patches
Quoted 15 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changing kernel/user/driver code every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.
Anjali
Re: On the NACKs on P4TC patches
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.
The risk if we allow this into the kernel is that a vendor might be tempted to point to P4-TC performance as a baseline to justify to customers that they need to buy specialized hardware to get performance, whereas if XDP was used maybe they don't need the performance and cost of hardware. Note, this scenario already happened once before, when the DPDK joined LF they made bogus claims that they got a 100x performance over the kernel-- had they put at least the slightest effort into tuning the kernel that would have dropped the delta by an order of magnitude, and since then we've pretty much closed the gap (actually, this is precisely what motivated the creation of XDP so I guess that story had a happy ending!) . There are circumstances where hardware offload may be warranted, but it needs to be honestly justified by comparing it to an optimized software solution-- so in the case of P4, it should be compared to well written XDP code for instance, not P4-TC.
Tom
Quoted 4 lines
Anjali
RE: On the NACKs on P4TC patches
Quoted 34 lines
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.
Quoted 2 lines
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.
Anjali, thanks for asking. Agreed, I like the flexibility of accommodating a variety of platforms depending upon performance requirements and intended target system. For me, flexibility is important. Some solutions need an inline filter and P4-TC makes it so easy. The fact I will be able to get HW offload means I'm not performance bound. Some other solutions might need DPDK implementation, so P4-DPDK is a choice there as well, and there are acceleration options. Keeping much of the dataplane design in one language (P4) makes it easier for more developers to create products without having to be platform-level experts. As someone who's worked with P4 Tofino, P4-TC, bmv2, etc. I can authoritatively state that all have their proper place.
Quoted 14 lines
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!mHilz4xBMimnfapDG8BEgqOuPw_Mn-KiMHb-aNbl8nB8TwfOfSleeIANiNRFQtTc5zfR0aK1TE2J8lT2Fg$
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.
AFAIK P4-TC can emit XDP or eBPF code depending upon the situation, someone more knowledgeable should chime in. However, I don't agree that comparing the speeds of XDP vs. P4-TC should even be a deciding factor. If P4-TC is good enough for a lot of applications, that is fine by me and over time it'll only get better. If we held back every innovation because it was slower than something else, progress would suffer.
Quoted 6 lines
The risk if we allow this into the kernel is that a vendor might be tempted to point to P4-TC performance as a baseline to justify to customers that they need to buy specialized hardware to get performance, whereas if XDP was used maybe they don't need the performance and cost of hardware.
I really don't buy this argument, it's FUD. Let's judge P4-TC on its merits, not prejudge it as a ploy to sell vendor hardware.
Quoted 11 lines
Note, this scenario already happened
once before, when the DPDK joined LF they made bogus claims that they got a 100x performance over the kernel-- had they put at least the slightest effort into tuning the kernel that would have dropped the delta by an order of magnitude, and since then we've pretty much closed the gap (actually, this is precisely what motivated the creation of XDP so I guess that story had a happy ending!) . There are circumstances where hardware offload may be warranted, but it needs to be honestly justified by comparing it to an optimized software solution-- so in the case of P4, it should be compared to well written XDP code for instance, not P4-TC.
I strongly disagree that it "it needs to be honestly justified by comparing it to an optimized software solution."
Says who? This is no more factual than saying "C or golang need to be judged by comparing it to assembly language."
Today the gap between C and assembly is small, but way back in my career, C was way slower.
Over time optimizing compilers have closed the gap. Who's to say P4 technologies won't do the same?
P4-TC can be judged on its own merits for its utility and productivity. I can't stress enough that P4 is very productive when applied to certain problems.
Note, P4-BMv2 has been used by thousands of developers, researchers and students and it is relatively slow. Yet that doesn't deter users.
There is a Google Summer of Code project to add PNA support, rather ambitious. However, P4-TC already partially supports PNA and the gap is closing.
I feel like P4-TC could replace the use of BMv2 in a lot of applications and if it were upstreamed, it'd eventually be available on all Linux machines. The ability to write custom externs is very compelling. Eventual HW offload using the same code will be game-changing. Bmv2 is a big c++ program and somewhat intimidating to dig into to make enhancements, especially at the architectural level. There is no HW offload path, and it's not really fast, so it remains mainly a researchy-thing and will stay that way. P4-TC could span the needs from research to production in SW, and performant production with HW offload.
Quoted 7 lines
Tom
Quoted 3 lines
Anjali
Jamal Hadi Salim
maintainerRe: On the NACKs on P4TC patches
Not sure why my email was tagged as html and blocked, but here goes again:
Quoted 54 lines · Chris Sommers
Quoted 35 lines
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.Quoted 2 lines
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.Anjali, thanks for asking. Agreed, I like the flexibility of accommodating a variety of platforms depending upon performance requirements and intended target system. For me, flexibility is important. Some solutions need an inline filter and P4-TC makes it so easy. The fact I will be able to get HW offload means I'm not performance bound. Some other solutions might need DPDK implementation, so P4-DPDK is a choice there as well, and there are acceleration options. Keeping much of the dataplane design in one language (P4) makes it easier for more developers to create products without having to be platform-level experts. As someone who's worked with P4 Tofino, P4-TC, bmv2, etc. I can authoritatively state that all have their proper place.
Quoted 14 lines
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!mHilz4xBMimnfapDG8BEgqOuPw_Mn-KiMHb-aNbl8nB8TwfOfSleeIANiNRFQtTc5zfR0aK1TE2J8lT2Fg$
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.
Quoted 4 lines
AFAIK P4-TC can emit XDP or eBPF code depending upon the situation, someone more knowledgeable should chime in. However, I don't agree that comparing the speeds of XDP vs. P4-TC should even be a deciding factor. If P4-TC is good enough for a lot of applications, that is fine by me and over time it'll only get better. If we held back every innovation because it was slower than something else, progress would suffer.
Yes, XDP can be emitted based on compiler options (and was a motivation factor in considering use of eBPF). Tom's comment above seems to confuse the fact that XDP tends to be faster than TC with eBPF as the fault of P4TC.
In any case this statement falls under:
https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#2b-comment-but--it-is-not-performant
On Tom's theory that the vendors are going to push inferior s/w for the sake of selling h/w - I would argues that we are not in the 90s anymore and I dont believe there's any vendor conspiracy theory here ;-> a single port can do 100s of Gbps, and of course if you want to do high speed you need to offload, no general purpose CPU will save you. And really the arguement that "offload=evil" holds no water anymore.
cheers,
jamal
A message that is not in the archive (sent off-list or not yet received): CAM0EoMnyn9Bfufar5rv6cbRRTHKCaZ1q-b93T2EWUKcBv_ibNw@mail.gmail.com
Re: On the NACKs on P4TC patches
Quoted 65 lines · Jamal Hadi Salim
Quoted 54 lines · Chris Sommers
Quoted 35 lines
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.Quoted 2 lines
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.Anjali, thanks for asking. Agreed, I like the flexibility of accommodating a variety of platforms depending upon performance requirements and intended target system. For me, flexibility is important. Some solutions need an inline filter and P4-TC makes it so easy. The fact I will be able to get HW offload means I'm not performance bound. Some other solutions might need DPDK implementation, so P4-DPDK is a choice there as well, and there are acceleration options. Keeping much of the dataplane design in one language (P4) makes it easier for more developers to create products without having to be platform-level experts. As someone who's worked with P4 Tofino, P4-TC, bmv2, etc. I can authoritatively state that all have their proper place.
Quoted 14 lines
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!mHilz4xBMimnfapDG8BEgqOuPw_Mn-KiMHb-aNbl8nB8TwfOfSleeIANiNRFQtTc5zfR0aK1TE2J8lT2Fg$
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.Quoted 5 lines
AFAIK P4-TC can emit XDP or eBPF code depending upon the situation, someone more knowledgeable should chime in. However, I don't agree that comparing the speeds of XDP vs. P4-TC should even be a deciding factor. If P4-TC is good enough for a lot of applications, that is fine by me and over time it'll only get better. If we held back every innovation because it was slower than something else, progress would suffer.
Quoted 1 line
Quoted 2 lines
Yes, XDP can be emitted based on compiler options (and was a motivation factor in considering use of eBPF). Tom's comment above seems to confuse the fact that XDP tends to be faster than TC with eBPF as the fault of P4TC. In any case this statement falls under: https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#2b-comment-but--it-is-not-performant
Jamal,
From that: "My response has always consistently been: performance is a lower priority to P4 correctness and expressibility." That might be true for P4, but not for the kernel. CPU performance is important, and your statement below that justifies offloads on the basis that "no general purpose CPU will save you" confirms that. Please be more upfront about what the performance is like including performance numbers in the cover letter for the next patch set. This is the best way to avoid confusion and rampant speculation, and if performance isn't stellar being open about it in the community is the best way to figure out how to improve it.
Quoted 2 lines
On Tom's theory that the vendors are going to push inferior s/w for the sake of selling h/w: we are not in the 90s anymore and there's no vendor conspiracy theory here: a single port can do 100s of Gbps, and of course if you want to do high speed you need to offload, no general purpose CPU will save you.
Let's not pretend that offloads are a magic bullet that just makes everything better, if that were true then we'd all be using TOE by now! There are a myriad of factors to consider whether offloading is worth it. What is "high speed", is this small packets or big packets, are we terminating TCP, are we doing some sort of fast/slow path split which might work great in the lab but on the Internet can become a DOS vector? What's the application? Are we just trying to offload parts of the datapath, TCP, RDMA, memcached, ML reduce operations? Are we trying to do line rate encryption, compression, trying to do a billion PCB lookups a second? Are we taking into account continuing advancements in the CPU that have in the past made offloads obsolete (for instance, AES instructions pretty much obsoleted initial attempts to obsolete IPsec)? How simple is the programming model, how debuggable is it, what's the TCO?
I do believe offload is part of the solution. And the good news is that programmable devices facilitate that. IMO, our challenge is to create a facility in the kernel to kernel offloads in a much better way (I don't believe there's disagreement with these points).
Tom
Quoted 43 lines
cheers,
jamalQuoted 39 lines
Quoted 6 lines
The risk if we allow this into the kernel is that a vendor might be tempted to point to P4-TC performance as a baseline to justify to customers that they need to buy specialized hardware to get performance, whereas if XDP was used maybe they don't need the performance and cost of hardware.
I really don't buy this argument, it's FUD. Let's judge P4-TC on its merits, not prejudge it as a ploy to sell vendor hardware.
Quoted 11 lines
Note, this scenario already happened
once before, when the DPDK joined LF they made bogus claims that they got a 100x performance over the kernel-- had they put at least the slightest effort into tuning the kernel that would have dropped the delta by an order of magnitude, and since then we've pretty much closed the gap (actually, this is precisely what motivated the creation of XDP so I guess that story had a happy ending!) . There are circumstances where hardware offload may be warranted, but it needs to be honestly justified by comparing it to an optimized software solution-- so in the case of P4, it should be compared to well written XDP code for instance, not P4-TC.I strongly disagree that it "it needs to be honestly justified by comparing it to an optimized software solution."
Says who? This is no more factual than saying "C or golang need to be judged by comparing it to assembly language."
Today the gap between C and assembly is small, but way back in my career, C was way slower.
Over time optimizing compilers have closed the gap. Who's to say P4 technologies won't do the same?
P4-TC can be judged on its own merits for its utility and productivity. I can't stress enough that P4 is very productive when applied to certain problems.Note, P4-BMv2 has been used by thousands of developers, researchers and students and it is relatively slow. Yet that doesn't deter users.
There is a Google Summer of Code project to add PNA support, rather ambitious. However, P4-TC already partially supports PNA and the gap is closing.
I feel like P4-TC could replace the use of BMv2 in a lot of applications and if it were upstreamed, it'd eventually be available on all Linux machines. The ability to write custom externs is very compelling. Eventual HW offload using the same code will be game-changing. Bmv2 is a big c++ program and somewhat intimidating to dig into to make enhancements, especially at the architectural level. There is no HW offload path, and it's not really fast, so it remains mainly a researchy-thing and will stay that way. P4-TC could span the needs from research to production in SW, and performant production with HW offload.Quoted 7 lines
Tom
Quoted 3 lines
Anjali
Jamal Hadi Salim
maintainerRe: On the NACKs on P4TC patches
Quoted 72 lines · Tom Herbert
Quoted 65 lines · Jamal Hadi Salim
Quoted 54 lines · Chris Sommers
Quoted 35 lines
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.Quoted 2 lines
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.Anjali, thanks for asking. Agreed, I like the flexibility of accommodating a variety of platforms depending upon performance requirements and intended target system. For me, flexibility is important. Some solutions need an inline filter and P4-TC makes it so easy. The fact I will be able to get HW offload means I'm not performance bound. Some other solutions might need DPDK implementation, so P4-DPDK is a choice there as well, and there are acceleration options. Keeping much of the dataplane design in one language (P4) makes it easier for more developers to create products without having to be platform-level experts. As someone who's worked with P4 Tofino, P4-TC, bmv2, etc. I can authoritatively state that all have their proper place.
Quoted 14 lines
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!mHilz4xBMimnfapDG8BEgqOuPw_Mn-KiMHb-aNbl8nB8TwfOfSleeIANiNRFQtTc5zfR0aK1TE2J8lT2Fg$
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.Quoted 5 lines
AFAIK P4-TC can emit XDP or eBPF code depending upon the situation, someone more knowledgeable should chime in. However, I don't agree that comparing the speeds of XDP vs. P4-TC should even be a deciding factor. If P4-TC is good enough for a lot of applications, that is fine by me and over time it'll only get better. If we held back every innovation because it was slower than something else, progress would suffer.
Quoted 1 line
Quoted 2 lines
Yes, XDP can be emitted based on compiler options (and was a motivation factor in considering use of eBPF). Tom's comment above seems to confuse the fact that XDP tends to be faster than TC with eBPF as the fault of P4TC. In any case this statement falls under: https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#2b-comment-but--it-is-not-performant
Jamal,
Quoted 10 lines
From that: "My response has always consistently been: performance is a lower priority to P4 correctness and expressibility." That might be true for P4, but not for the kernel. CPU performance is important, and your statement below that justifies offloads on the basis that "no general purpose CPU will save you" confirms that. Please be more upfront about what the performance is like including performance numbers in the cover letter for the next patch set. This is the best way to avoid confusion and rampant speculation, and if performance isn't stellar being open about it in the community is the best way to figure out how to improve it.
I believe you are misreading those graphs or maybe you are mixing it with the original u32/pedit script approach? The tests are run at TC and XDP layers. Pay particular attention to the results of the handcoded/tuned eBPF datapath at TC and at XDP compared to analogous ones generated by the compiler. You will notice +/-5% or so differences. That is with the current compiler generated code. We are looking to improve that - but do note that is generated code, nothing to do with the kernel. As the P4 program becomes more complex (many tables, longer keys, more entries, more complex actions) then we become compute bound, so no difference really.
Now having said that: yes - s/w performance is certainly _not our highest priority feature_ and that is not saying we dont care but as the text said If i am getting 2Mpps using handcoding vs 1.84Mpps using generated code(per those graphs) and i can generate code and execute it in 5 minutes (Chris who is knowledgeable in P4 was able to do it in less time), then _i pick the code generation any day of the week_. Tooling, tooling, tooling.
To re-iterate, the most important requirement is the abstraction, meaning:
I can take the same P4 program I am running in s/w and generate using a different backend for AMD or Intel offload equivalent and get several magnitude improvements in performance because it is now running in h/w. I still get to use the same application controlling either s/w and/or hardware, etc
TBH, I am indifferent and could add some numbers but it is missing the emphasis of what we are trying to achieve, the cover letter is already half a novel - with the short attention span most people have it will be just muddying the waters.
Quoted 18 lines
Quoted 2 lines
On Tom's theory that the vendors are going to push inferior s/w for the sake of selling h/w: we are not in the 90s anymore and there's no vendor conspiracy theory here: a single port can do 100s of Gbps, and of course if you want to do high speed you need to offload, no general purpose CPU will save you.
Let's not pretend that offloads are a magic bullet that just makes everything better, if that were true then we'd all be using TOE by now! There are a myriad of factors to consider whether offloading is worth it. What is "high speed", is this small packets or big packets, are we terminating TCP, are we doing some sort of fast/slow path split which might work great in the lab but on the Internet can become a DOS vector? What's the application? Are we just trying to offload parts of the datapath, TCP, RDMA, memcached, ML reduce operations? Are we trying to do line rate encryption, compression, trying to do a billion PCB lookups a second? Are we taking into account continuing advancements in the CPU that have in the past made offloads obsolete (for instance, AES instructions pretty much obsoleted initial attempts to obsolete IPsec)? How simple is the programming model, how debuggable is it, what's the TCO?
Quoted 5 lines
I do believe offload is part of the solution. And the good news is that programmable devices facilitate that. IMO, our challenge is to create a facility in the kernel to kernel offloads in a much better way (I don't believe there's disagreement with these points).
This is about a MAT(match-action table) model whose offloads are covered via TC and is well understood and is very specific. We are not trying to solve "the world of offloads" which includes TOEs. P4 aware NICs are in the market and afaik those ASICs are not solving TOE. I thought you understand the scope but if not start by reading this: https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
cheers,
jamal
Quoted 49 lines
Tom
Quoted 43 lines
cheers,
jamalQuoted 39 lines
Quoted 6 lines
The risk if we allow this into the kernel is that a vendor might be tempted to point to P4-TC performance as a baseline to justify to customers that they need to buy specialized hardware to get performance, whereas if XDP was used maybe they don't need the performance and cost of hardware.
I really don't buy this argument, it's FUD. Let's judge P4-TC on its merits, not prejudge it as a ploy to sell vendor hardware.
Quoted 11 lines
Note, this scenario already happened
once before, when the DPDK joined LF they made bogus claims that they got a 100x performance over the kernel-- had they put at least the slightest effort into tuning the kernel that would have dropped the delta by an order of magnitude, and since then we've pretty much closed the gap (actually, this is precisely what motivated the creation of XDP so I guess that story had a happy ending!) . There are circumstances where hardware offload may be warranted, but it needs to be honestly justified by comparing it to an optimized software solution-- so in the case of P4, it should be compared to well written XDP code for instance, not P4-TC.I strongly disagree that it "it needs to be honestly justified by comparing it to an optimized software solution."
Says who? This is no more factual than saying "C or golang need to be judged by comparing it to assembly language."
Today the gap between C and assembly is small, but way back in my career, C was way slower.
Over time optimizing compilers have closed the gap. Who's to say P4 technologies won't do the same?
P4-TC can be judged on its own merits for its utility and productivity. I can't stress enough that P4 is very productive when applied to certain problems.Note, P4-BMv2 has been used by thousands of developers, researchers and students and it is relatively slow. Yet that doesn't deter users.
There is a Google Summer of Code project to add PNA support, rather ambitious. However, P4-TC already partially supports PNA and the gap is closing.
I feel like P4-TC could replace the use of BMv2 in a lot of applications and if it were upstreamed, it'd eventually be available on all Linux machines. The ability to write custom externs is very compelling. Eventual HW offload using the same code will be game-changing. Bmv2 is a big c++ program and somewhat intimidating to dig into to make enhancements, especially at the architectural level. There is no HW offload path, and it's not really fast, so it remains mainly a researchy-thing and will stay that way. P4-TC could span the needs from research to production in SW, and performant production with HW offload.Quoted 7 lines
Tom
Quoted 3 lines
Anjali
Re: On the NACKs on P4TC patches
Quoted 96 lines · Jamal Hadi Salim
Quoted 72 lines · Tom Herbert
Quoted 65 lines · Jamal Hadi Salim
Quoted 54 lines · Chris Sommers
Quoted 35 lines
Quoted 32 lines · Anjali
Quoted 16 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
Quoted 7 lines
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
We feel P4TC approach is the path to add Linux kernel support.Quoted 2 lines
Quoted 2 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.Anjali, thanks for asking. Agreed, I like the flexibility of accommodating a variety of platforms depending upon performance requirements and intended target system. For me, flexibility is important. Some solutions need an inline filter and P4-TC makes it so easy. The fact I will be able to get HW offload means I'm not performance bound. Some other solutions might need DPDK implementation, so P4-DPDK is a choice there as well, and there are acceleration options. Keeping much of the dataplane design in one language (P4) makes it easier for more developers to create products without having to be platform-level experts. As someone who's worked with P4 Tofino, P4-TC, bmv2, etc. I can authoritatively state that all have their proper place.
Quoted 14 lines
Hi Anjali,
Are there any use cases of P4-TC that don't involve P4 hardware? If someone wanted to write one off datapath code for their deployment and they didn't have P4 hardware would you suggest that they write they're code in P4-TC? The reason I ask is because I'm concerned about the performance of P4-TC. Like John said, this is mapping code that is intended to run in specialized hardware into a CPU, and it's also interpreted execution in TC. The performance numbers in
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!mHilz4xBMimnfapDG8BEgqOuPw_Mn-KiMHb-aNbl8nB8TwfOfSleeIANiNRFQtTc5zfR0aK1TE2J8lT2Fg$
seem to show that P4-TC has about half the performance of XDP. Even with a lot of work, it's going to be difficult to substantially close that gap.Quoted 5 lines
AFAIK P4-TC can emit XDP or eBPF code depending upon the situation, someone more knowledgeable should chime in. However, I don't agree that comparing the speeds of XDP vs. P4-TC should even be a deciding factor. If P4-TC is good enough for a lot of applications, that is fine by me and over time it'll only get better. If we held back every innovation because it was slower than something else, progress would suffer.
Quoted 1 line
Quoted 2 lines
Yes, XDP can be emitted based on compiler options (and was a motivation factor in considering use of eBPF). Tom's comment above seems to confuse the fact that XDP tends to be faster than TC with eBPF as the fault of P4TC. In any case this statement falls under: https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#2b-comment-but--it-is-not-performant
Jamal,
Quoted 10 lines
From that: "My response has always consistently been: performance is a lower priority to P4 correctness and expressibility." That might be true for P4, but not for the kernel. CPU performance is important, and your statement below that justifies offloads on the basis that "no general purpose CPU will save you" confirms that. Please be more upfront about what the performance is like including performance numbers in the cover letter for the next patch set. This is the best way to avoid confusion and rampant speculation, and if performance isn't stellar being open about it in the community is the best way to figure out how to improve it.
I believe you are misreading those graphs or maybe you are mixing it with the original u32/pedit script approach? The tests are run at TC and XDP layers. Pay particular attention to the results of the handcoded/tuned eBPF datapath at TC and at XDP compared to analogous ones generated by the compiler. You will notice +/-5% or so differences. That is with the current compiler generated code. We are looking to improve that - but do note that is generated code, nothing to do with the kernel. As the P4 program becomes more complex (many tables, longer keys, more entries, more complex actions) then we become compute bound, so no difference really.
Quoted 13 lines
Now having said that: yes - s/w performance is certainly
_notour highest priorityfeature_and that is not saying we dont care but as the text said If i am getting 2Mpps using handcoding vs 1.84Mpps using generated code(per those graphs) and i can generate code and execute it in 5 minutes (Chris who is knowledgeable in P4 was able to do it in less time), then _i pick the code generation any day of theweek_. Tooling, tooling, tooling.
To re-iterate, the most important requirement is the abstraction, meaning:
I can take the same P4 program I am running in s/w and generate using a different backend for AMD or Intel offload equivalent and get several magnitude improvements in performance because it is now running in h/w. I still get to use the same application controlling either s/w and/or hardware, etc
Jamal,
I believe you're making contradictory points here. On one hand you're saying that performance isn't a high priority and that it's enough to get the abstraction right. On the other hand you seem to be making the argument that we need hardware offload because performance of software in a CPU is so bad. I can't rectify these statements.
Also, when you claim that hardware is going to deliver "several magnitude improvements in performance" over an implementation that has not been optimized for performance in a CPU, then you are heading down the path of justifying hardware offload on the basis that it performs better than baseline software which has not been at all optimized. IMO, that is not valid justification and I believe it would be a disservice to our users if they buy into hardware where a software solution would have been sufficient had someone put in the effort to optimize it.
Quoted 5 lines
TBH, I am indifferent and could add some numbers but it is missing the emphasis of what we are trying to achieve, the cover letter is already half a novel - with the short attention span most people have it will be just muddying the waters.
This is putting code in the kernel that runs in the Linux networking data path. It shouldn't be any surprise that we're asking for some quantification and analysis of performance in the patch description.
Tom
Quoted 84 lines
Quoted 19 lines
Quoted 3 lines
On Tom's theory that the vendors are going to push inferior s/w for the sake of selling h/w: we are not in the 90s anymore and there's no vendor conspiracy theory here: a single port can do 100s of Gbps, and of course if you want to do high speed you need to offload, no general purpose CPU will save you.
Let's not pretend that offloads are a magic bullet that just makes everything better, if that were true then we'd all be using TOE by now! There are a myriad of factors to consider whether offloading is worth it. What is "high speed", is this small packets or big packets, are we terminating TCP, are we doing some sort of fast/slow path split which might work great in the lab but on the Internet can become a DOS vector? What's the application? Are we just trying to offload parts of the datapath, TCP, RDMA, memcached, ML reduce operations? Are we trying to do line rate encryption, compression, trying to do a billion PCB lookups a second? Are we taking into account continuing advancements in the CPU that have in the past made offloads obsolete (for instance, AES instructions pretty much obsoleted initial attempts to obsolete IPsec)? How simple is the programming model, how debuggable is it, what's the TCO?
Quoted 5 lines
I do believe offload is part of the solution. And the good news is that programmable devices facilitate that. IMO, our challenge is to create a facility in the kernel to kernel offloads in a much better way (I don't believe there's disagreement with these points).
This is about a MAT(match-action table) model whose offloads are covered via TC and is well understood and is very specific. We are not trying to solve "the world of offloads" which includes TOEs. P4 aware NICs are in the market and afaik those ASICs are not solving TOE. I thought you understand the scope but if not start by reading this: https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
cheers,
jamalQuoted 49 lines
Tom
Quoted 43 lines
cheers,
jamalQuoted 39 lines
Quoted 6 lines
The risk if we allow this into the kernel is that a vendor might be tempted to point to P4-TC performance as a baseline to justify to customers that they need to buy specialized hardware to get performance, whereas if XDP was used maybe they don't need the performance and cost of hardware.
I really don't buy this argument, it's FUD. Let's judge P4-TC on its merits, not prejudge it as a ploy to sell vendor hardware.
Quoted 11 lines
Note, this scenario already happened
once before, when the DPDK joined LF they made bogus claims that they got a 100x performance over the kernel-- had they put at least the slightest effort into tuning the kernel that would have dropped the delta by an order of magnitude, and since then we've pretty much closed the gap (actually, this is precisely what motivated the creation of XDP so I guess that story had a happy ending!) . There are circumstances where hardware offload may be warranted, but it needs to be honestly justified by comparing it to an optimized software solution-- so in the case of P4, it should be compared to well written XDP code for instance, not P4-TC.I strongly disagree that it "it needs to be honestly justified by comparing it to an optimized software solution."
Says who? This is no more factual than saying "C or golang need to be judged by comparing it to assembly language."
Today the gap between C and assembly is small, but way back in my career, C was way slower.
Over time optimizing compilers have closed the gap. Who's to say P4 technologies won't do the same?
P4-TC can be judged on its own merits for its utility and productivity. I can't stress enough that P4 is very productive when applied to certain problems.Note, P4-BMv2 has been used by thousands of developers, researchers and students and it is relatively slow. Yet that doesn't deter users.
There is a Google Summer of Code project to add PNA support, rather ambitious. However, P4-TC already partially supports PNA and the gap is closing.
I feel like P4-TC could replace the use of BMv2 in a lot of applications and if it were upstreamed, it'd eventually be available on all Linux machines. The ability to write custom externs is very compelling. Eventual HW offload using the same code will be game-changing. Bmv2 is a big c++ program and somewhat intimidating to dig into to make enhancements, especially at the architectural level. There is no HW offload path, and it's not really fast, so it remains mainly a researchy-thing and will stay that way. P4-TC could span the needs from research to production in SW, and performant production with HW offload.Quoted 7 lines
Tom
Quoted 3 lines
Anjali
RE: On the NACKs on P4TC patches
In reply to Singhai, AnjaliQuoted 19 lines · Anjali
Quoted 18 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.
Quoted 7 lines
Quoted 3 lines
If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.
Maybe more direct what Linux drivers support this? That would be a good first place to start IMO. Similarly what AMD hardware driver supports this. If I have two drivers from two vendors with P4 support this is great.
For Intel I assume this is idpf?
To be concrete can we start with Linux driver A and P4 program P. Modprobe driver A and push P4 program P so that it does something very simple, and drop a CIDR/Port range into a table. Perhaps this is so obvious in your community the trouble is in the context of a Linux driver its not immediately obvious to me and I would suspect its not obvious to many others.
I really think walking through the key steps here would really help?
1. $ p4IntelCompiler p4-dos.p4 -o myp4
2. $ modprobe idpf
3. $ ping -i eth0 10.0.0.1 // good
4. $ p4Load p4-dos.p4
5. -- load cidr into the hardware somehow -- p4rt-ctrl?
6. $ ping -i eth0 10.0.0.1 // dropped
This is an honest attempt to help fwiw. Questions would be.
For compilation do we need an artifact from Intel it seems so from docs. But maybe a typo not sure. I'm not overly stuck on it but worth mentioning if folks try to follow your docs.
For 2 I assume this is just normal every day module load nothing to see. Does it pop something up in /proc or in firmware or...? How do I know its P4 ready?
For 4. How does this actually work? Is it a file in a directory the driver pushes into firmware? How does the firmware know I've done this? Does the Linux driver already support this?
For 5 (most interesting) how does this work today. How are you currently talking to the driver/firmware to insert rules and discover the tables? And does the idpf driver do this already? Some side channel I guess? This is p4rt-ctrl?
I've seen docs for above in ipdk, but they are a bit hard to follow if I'm honest.
I assume IPDK is the source folks talk to when we mention there is hardware somewhere. Also it seems there is an IPDK BPF support as well which is interesting.
And do you know how the DPDK implementation works? Can we learn from them is it just on top of Flow API which we could easily use in devlink or some other *link I suspect.
Quoted 3 lines
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.
I think many 1st order and important points have been skipped. How do you program the device is it a firmware blob, a set of firmware commands, something that comes to you on device so only vendor sees this? Maybe I can infer this from some docs and some examples (by the way I ran through some of your DPU docs and such) but its unclear how these map onto Linux networking. Jiri started into this earlier and was cut off because p4tc was not for hardware offload. Now it is apparently.
P4 is a good DSL for this sure and it has a runtime already specified which is great.
This is not a qdisc/tc its an entire hardware pipeline I don't see the reason to put it in TC at all.
Quoted 1 line
We feel P4TC approach is the path to add Linux kernel support.
I disagree with your implementation not your goals to support flexible hardware.
Quoted 3 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.
None of above requires P4TC. For different architectures you build optimal backend compilers. You have a Xilenx backend, an Intel backend, and a Linux CPU based backend. I see no reason to constrain the software case to map to a pipeline model for example. Software running on a CPU has very different characteristics from something running on a TOR, or FPGA. Trying to push all these into one backend "model" will result in suboptimal result for every target. At the end of the day my .02$, P4 is a DSL it needs a target dependent compiler in front of it. I want to optimize my software pipeline the compiler should compress tables as much as possible and search for a O(1) lookup even if getting that key is somewhat expensive. Conversely a TCAM changes the game. An FPGA is going to be flexible and make lots of tradeoffs here of which I'm not an expert. Also by avoiding loading the DSL into the kernel you leave room for others to build new/better/worse DSLs as they please.
The P4 community writes control applicatoins on top of the runtime spec right? p4rt-ctl being the thing I found. This should abstract the endpoint away to work with hardware or software or FPGA or anything else.
.John
IR for Programmable Datapaths [WAS Re: On the NACKs on P4TC patches]
Quoted 9 lines
None of above requires P4TC. For different architectures you build optimal backend compilers. You have a Xilenx backend, an Intel backend, and a Linux CPU based backend. I see no reason to constrain the software case to map to a pipeline model for example. Software running on a CPU has very different characteristics from something running on a TOR, or FPGA. Trying to push all these into one backend "model" will result in suboptimal result for every target. At the end of the day my .02$, P4 is a DSL it needs a target dependent compiler
Quoted 9 lines
in front of it. I want to optimize my software pipeline the compiler should compress tables as much as possible and search for a O(1) lookup even if getting that key is somewhat expensive. Conversely a TCAM changes the game. An FPGA is going to be flexible and make lots of tradeoffs here of which I'm not an expert. Also by avoiding loading the DSL into the kernel you leave room for others to build new/better/worse DSLs as they please.
I think the general ask here is to define an Intermediate Representation that describes a programmed data path where it's a combination of declarative and imperative elements (parsers and table descriptions are better in declarative representation, functional logic seems more imperative). We also want references to accelerators with dynamic runtime binding to hardware (there are some interesting tricks we can do in the loader for a CPU target-- will talk about at Netdev). With a good IR we can decouple the frontend from the backend target which enables mixing and matching programming languages with arbitrary HW or SW targets. So a good IR potentially enables a lot of flexibility and freedom on both sides of the equation.
An IR also facilitates reasonable kernel offload via signing images with a hash of the IR. So for instance, a frontend compiler could compile a P4 program into the IR. That code could then be compiled into a SW target, say eBPF, and maybe P4 hardware. Each image has the hash of the IR. At runtime, the eBPF code could be loaded into the kernel. The hardware image can be loaded into the device using a side band mechanism. To offload, we would query the device-- if the hash reported by the device matches the hash in the eBPF then we know that the offload is viable. No jits, no pushing firmware bits through the kernel, no need for device capabilities flags, and avoids the pitfalls of TC flower.
There is one challenge here in how to deal with offloads that are already integrated into the kernel. I think GRO is a great example. GRO has been especially elusive as an offload since it requires a device to autonomously parse packets on input. We really want a GRO offload that parses the same exact protocols the kernel does (including encapsulations), but also implements the exact same logic in timers and pushing reassembled segments. So this needs to be programmable. The problem with the technique I described is that GRO is integrated into the kernel so we have no basis for a hash. I think the answer here is to start replacing fixed kernel C code with eBPF even in the critical path (we already talked about replacing flow dissector with eBPF).
Anyway, we have been working on this. There's Common Parser Representation in json (formerly known CPL that we talked about at Netdev). For execution logic, LLVM IR seems fine (btrw, MLIR is really useful by the way!). We're just starting to look at tables (probably also json). If there's interest I could share more...
Tom
Jamal Hadi Salim
maintainerRe: On the NACKs on P4TC patches
In reply to John FastabendQuoted 128 lines · John Fastabend
Quoted 19 lines · Anjali
Quoted 18 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.Quoted 7 lines
Quoted 3 lines
If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.Maybe more direct what Linux drivers support this? That would be a good first place to start IMO. Similarly what AMD hardware driver supports this. If I have two drivers from two vendors with P4 support this is great.
For Intel I assume this is idpf?
To be concrete can we start with Linux driver A and P4 program P. Modprobe driver A and push P4 program P so that it does something very simple, and drop a CIDR/Port range into a table. Perhaps this is so obvious in your community the trouble is in the context of a Linux driver its not immediately obvious to me and I would suspect its not obvious to many others.
I really think walking through the key steps here would really help?
1. $ p4IntelCompiler p4-dos.p4 -o myp4
2. $ modprobe idpf
3. $ ping -i eth0 10.0.0.1 // good
4. $ p4Load p4-dos.p4
5. -- load cidr into the hardware somehow -- p4rt-ctrl?
6. $ ping -i eth0 10.0.0.1 // droppedThis is an honest attempt to help fwiw. Questions would be.
For compilation do we need an artifact from Intel it seems so from docs. But maybe a typo not sure. I'm not overly stuck on it but worth mentioning if folks try to follow your docs.
For 2 I assume this is just normal every day module load nothing to see. Does it pop something up in /proc or in firmware or...? How do I know its P4 ready?
For 4. How does this actually work? Is it a file in a directory the driver pushes into firmware? How does the firmware know I've done this? Does the Linux driver already support this?
For 5 (most interesting) how does this work today. How are you currently talking to the driver/firmware to insert rules and discover the tables? And does the idpf driver do this already? Some side channel I guess? This is p4rt-ctrl?
I've seen docs for above in ipdk, but they are a bit hard to follow if I'm honest.
I assume IPDK is the source folks talk to when we mention there is hardware somewhere. Also it seems there is an IPDK BPF support as well which is interesting.
And do you know how the DPDK implementation works? Can we learn from them is it just on top of Flow API which we could easily use in devlink or some other *link I suspect.
Quoted 3 lines
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.I think many 1st order and important points have been skipped. How do you program the device is it a firmware blob, a set of firmware commands, something that comes to you on device so only vendor sees this? Maybe I can infer this from some docs and some examples (by the way I ran through some of your DPU docs and such) but its unclear how these map onto Linux networking. Jiri started into this earlier and was cut off because p4tc was not for hardware offload. Now it is apparently.
P4 is a good DSL for this sure and it has a runtime already specified which is great.
This is not a qdisc/tc its an entire hardware pipeline I don't see the reason to put it in TC at all.
Quoted 1 line
We feel P4TC approach is the path to add Linux kernel support.
I disagree with your implementation not your goals to support flexible hardware.
Quoted 3 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.None of above requires P4TC. For different architectures you build optimal backend compilers. You have a Xilenx backend, an Intel backend, and a Linux CPU based backend. I see no reason to constrain the software case to map to a pipeline model for example. Software running on a CPU has very different characteristics from something running on a TOR, or FPGA. Trying to push all these into one backend "model" will result in suboptimal result for every target. At the end of the day my .02$, P4 is a DSL it needs a target dependent compiler in front of it. I want to optimize my software pipeline the compiler should compress tables as much as possible and search for a O(1) lookup even if getting that key is somewhat expensive. Conversely a TCAM changes the game. An FPGA is going to be flexible and make lots of tradeoffs here of which I'm not an expert. Also by avoiding loading the DSL into the kernel you leave room for others to build new/better/worse DSLs as they please.
Quoted 5 lines
The P4 community writes control applicatoins on top of the runtime spec right? p4rt-ctl being the thing I found. This should abstract the endpoint away to work with hardware or software or FPGA or anything else.
For the record, _every single patchset we have posted_ specified our requirements as being s/w + h/w. A simpler version of the requirements is listed here:
https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#summary-of-our-requirements
John's content variant above is described in:
https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#summary-of-our-requirements
According to him we should not bother with the kernel at all. It's what is commonly referred to as a monday-morning quarterbacking or arm-chair lawyering "lets just do it my way and it will all be great". It's 90% of these discussions and one of the reasons I put up that page.
cheers,
jamal
Jamal Hadi Salim
maintainerRe: On the NACKs on P4TC patches
Quoted 136 lines · Jamal Hadi Salim
Quoted 128 lines · John Fastabend
Quoted 19 lines · Anjali
Quoted 18 lines
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PMQuoted 6 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program >these devices?
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example >maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally >different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions.
This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't >see the need for P4TC in this context.Quoted 7 lines
Quoted 3 lines
If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly >vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
John,
Let me start by saying production hardware exists i think Jamal posted some links but i can point you to our hardware.Maybe more direct what Linux drivers support this? That would be a good first place to start IMO. Similarly what AMD hardware driver supports this. If I have two drivers from two vendors with P4 support this is great.
For Intel I assume this is idpf?
To be concrete can we start with Linux driver A and P4 program P. Modprobe driver A and push P4 program P so that it does something very simple, and drop a CIDR/Port range into a table. Perhaps this is so obvious in your community the trouble is in the context of a Linux driver its not immediately obvious to me and I would suspect its not obvious to many others.
I really think walking through the key steps here would really help?
1. $ p4IntelCompiler p4-dos.p4 -o myp4
2. $ modprobe idpf
3. $ ping -i eth0 10.0.0.1 // good
4. $ p4Load p4-dos.p4
5. -- load cidr into the hardware somehow -- p4rt-ctrl?
6. $ ping -i eth0 10.0.0.1 // droppedThis is an honest attempt to help fwiw. Questions would be.
For compilation do we need an artifact from Intel it seems so from docs. But maybe a typo not sure. I'm not overly stuck on it but worth mentioning if folks try to follow your docs.
For 2 I assume this is just normal every day module load nothing to see. Does it pop something up in /proc or in firmware or...? How do I know its P4 ready?
For 4. How does this actually work? Is it a file in a directory the driver pushes into firmware? How does the firmware know I've done this? Does the Linux driver already support this?
For 5 (most interesting) how does this work today. How are you currently talking to the driver/firmware to insert rules and discover the tables? And does the idpf driver do this already? Some side channel I guess? This is p4rt-ctrl?
I've seen docs for above in ipdk, but they are a bit hard to follow if I'm honest.
I assume IPDK is the source folks talk to when we mention there is hardware somewhere. Also it seems there is an IPDK BPF support as well which is interesting.
And do you know how the DPDK implementation works? Can we learn from them is it just on top of Flow API which we could easily use in devlink or some other *link I suspect.
Quoted 3 lines
The hardware devices under discussion are capable of being abstracted using the P4 match-action paradigm so that's why we chose TC.
These devices are programmed using the TC/netlink interface i.e the standard TC control-driver ops apply. While it is clear to us that the P4TC abstraction suffices, we are currently discussing details that will cater for all vendors in our biweekly meetings. One big requirement is we want to avoid the flower trap - we dont want to be changingkernel/user/drivercode every time we add new datapaths.I think many 1st order and important points have been skipped. How do you program the device is it a firmware blob, a set of firmware commands, something that comes to you on device so only vendor sees this? Maybe I can infer this from some docs and some examples (by the way I ran through some of your DPU docs and such) but its unclear how these map onto Linux networking. Jiri started into this earlier and was cut off because p4tc was not for hardware offload. Now it is apparently.
P4 is a good DSL for this sure and it has a runtime already specified which is great.
This is not a qdisc/tc its an entire hardware pipeline I don't see the reason to put it in TC at all.
Quoted 1 line
We feel P4TC approach is the path to add Linux kernel support.
I disagree with your implementation not your goals to support flexible hardware.
Quoted 3 lines
The s/w path is needed as well for several reasons.
We need the same P4 program to run either in software or hardware or in both using skip_sw/skip_hw. It could be either in split mode or as an exception path as it is done today in flower or u32. Also it is common now in the P4 community that people define their datapath using their program and will write a control application that works for both hardware and software datapaths. They could be using the software datapath for testing as you said but also for the split/exception path. Chris can probably add more comments on the software datapath.None of above requires P4TC. For different architectures you build optimal backend compilers. You have a Xilenx backend, an Intel backend, and a Linux CPU based backend. I see no reason to constrain the software case to map to a pipeline model for example. Software running on a CPU has very different characteristics from something running on a TOR, or FPGA. Trying to push all these into one backend "model" will result in suboptimal result for every target. At the end of the day my .02$, P4 is a DSL it needs a target dependent compiler in front of it. I want to optimize my software pipeline the compiler should compress tables as much as possible and search for a O(1) lookup even if getting that key is somewhat expensive. Conversely a TCAM changes the game. An FPGA is going to be flexible and make lots of tradeoffs here of which I'm not an expert. Also by avoiding loading the DSL into the kernel you leave room for others to build new/better/worse DSLs as they please.
Quoted 5 lines
The P4 community writes control applicatoins on top of the runtime spec right? p4rt-ctl being the thing I found. This should abstract the endpoint away to work with hardware or software or FPGA or anything else.
Quoted 7 lines
For the record,
_everysingle patchset we haveposted_specified our requirements as being s/w + h/w. A simpler version of the requirements is listed here:
https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#summary-of-our-requirementsJohn's content variant above is described in:
https://github.com/p4tc-dev/pushback-patches?tab=readme-ov-file#summary-of-our-requirements
cheers,
jamal
Quoted 8 lines
According to him we should not bother with the kernel at all. It's what is commonly referred to as a monday-morning quarterbacking or arm-chair lawyering "lets just do it my way and it will all be great". It's 90% of these discussions and one of the reasons I put up that page.
cheers,
jamal
A message that is not in the archive (sent off-list or not yet received): MW4PR12MB71928072B3524CCC71B191F497F22@MW4PR12MB7192.namprd12.prod.outlook.com
Re: On the NACKs on P4TC patches
[Public]
Inline as <VJ2>... (was html, sorry)
________________________________________
From: John Fastabend <[email protected]>
Sent: Tuesday, May 28, 2024 1:17 PM
To: Jain, Vipin <[email protected]>; Singhai, Anjali <[email protected]>; Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: Re: On the NACKs on P4TC patches
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 36 lines · Vipin
[AMD Official Use Only - AMD Internal Distribution Only]
My apologies, earlier email used html and was blocked by the list... My response at the bottom as "VJ>"
________________________________________
From: Jain, Vipin <[email protected]>
Sent: Friday, May 24, 2024 2:28 PM
To: Singhai, Anjali <[email protected]>; Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: Re: On the NACKs on P4TC patches[AMD Official Use Only - AMD Internal Distribution Only]
I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC.
Happy to elaborate more if needed.
Thank you,
Vipin Jain
Sr Fellow Engineer, AMD
________________________________________
From: Singhai, Anjali <[email protected]>
Sent: Wednesday, May 22, 2024 5:30 PM
To: Hadi Salim, Jamal <[email protected]>; Jakub Kicinski <[email protected]>
Cc: Paolo Abeni <[email protected]>; Alexei Starovoitov <[email protected]>; Network Development <[email protected]>; Chatterjee, Deb <[email protected]>; Limaye, Namrata <[email protected]>; tom Herbert <[email protected]>; Marcelo Ricardo Leitner <[email protected]>; Shirshyad, Mahesh <[email protected]>; Osinski, Tomasz <[email protected]>; Jiri Pirko <[email protected]>; Cong Wang <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Vlad Buslov <[email protected]>; Simon Horman <[email protected]>; Khalid Manaa <[email protected]>; Toke Høiland-Jørgensen <[email protected]>; Victor Nogueira <[email protected]>; Tammela, Pedro <[email protected]>; Jain, Vipin <[email protected]>; Daly, Dan <[email protected]>; Andy Fingerhut <[email protected]>; Sommers, Chris <[email protected]>; Matty Kadosh <[email protected]>; bpf <[email protected]>; [email protected] <[email protected]>
Subject: RE: On the NACKs on P4TC patchesCaution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 1 line · Jakub Kicinski
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
Quoted 8 lines
Quoted 2 lines · Jamal Hadi Salim
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project >responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. >Sample:
https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/FWIW, Intel is in full support of P4TC as we have stated several times in the past.
VJ> I can ascertain (from AMD) that we have stated interest in, and are in full support of P4TC. Happy to elaborate more if needed. VJ> Thanks, Vipin
Anjali and Vipin is your support for HW support of P4 or a Linux SW implementation of P4. If its for HW support what drivers would we want to support? Can you describe how to program these devices?
<VJ2> Both, Sw implementation of P4 would allow prototyping and implementing without DPU/IPU hardware. Users have been deploying AMD P4 DPUs since 2019. They user standard (pensando or virtio) driver on the host and on the sideband compile and run/load P4 programs.
At the moment there hasn't been any movement on Linux hardware P4 support side as far as I can tell. Yes there are some SDKs and build kits floating around for FPGAs. For example maybe start with what drivers in kernel tree run the DPUs that have this support? I think this would be a productive direction to go if we in fact have hardware support in the works.
<VJ2> You guessed it correctly; AMD DPUs have SDK/build-kits to add hardware P4 support. For cloud infrastructure use case this works well, because one needs to expose a simple NIC (vanilla Ethernet NIC) on the host and therefore the users are okay (and prefer) compiling/loading it out-of-band. But P4TC effort, I am hoping can bring P4 to the users wanting to do inline acceleration.
If you want a SW implementation in Linux my opinion is still pushing a DSL into the kernel datapath via qdisc/tc is the wrong direction. Mapping P4 onto hardware blocks is fundamentally different architecture from mapping P4 onto general purpose CPU and registers. My opinion -- to handle this you need a per architecture backend/JIT to compile the P4 to native instructions. This will give you the most flexibility to define new constructs, best performance, and lowest overhead runtime. We have a P4 BPF backend already and JITs for most architectures I don't see the need for P4TC in this context.
<VJ2> Well, I am not really a linux network or kernel datapath expert. But my hope certainly is that if a use case this succesful (with deployment in production for 4 years) can be mainstreamed with software (and hardware, if available), it could benefit the end users.
If the end goal is a hardware offload control plane I'm skeptical we even need something specific just for SW datapath. I would propose a devlink or new infra to program the device directly vs overhead and complexity of abstracting through 'tc'. If you want to emulate your device use BPF or user space datapath.
.John
RE: On the NACKs on P4TC patches
In reply to Jamal Hadi SalimApologies for resending as plain text, the first try was HTML and got rejected by bots.
Quoted 37 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$
+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Quoted 6 lines
I am not sure where the parser thing comes from - the parser is generated as eBPF.
Quoted 3 lines
So from my perspective, the submission is neither technically strong enough, nor broadly useful enough to consider making questionable precedents for, i.e. to override maintainers on how their subsystems are extended.
I disagree vehemently on the "broadly useful enough" comment.
Quoted 14 lines
I believe as a community nobody should just have the power to nack things just because - as i stated in the page, not even Linus. That code doesnt touch anything to do with eBPF maintainers (meaning things they have to fix when an issue shows up) neither does it "extend" as you state any ebpf code and it is all part of the networking subsystem. Sure, anybody has the right to nack but I contend that nacks should be based on technical reasons. I have listed all the objections in that page and how i have responded to them over time. Someone needs to look at those objectively and say if they are valid. The arguement made so far(By Paolo and now by you) is "we cant override maintainers on how their subsystems are used" then we are in uncharted territory, thats why i am asking for arbitration.
Quoted 2 lines
cheers,
jamal
Maintainers: I am perplexed and dismayed that this is getting so much pushback. None of the objections, regardless of their merits (or not) seem to outweigh the potential benefits to end-users. I am extremely interested in using P4TC, it adds a lot of value and reuses so much existing Linux infra. The custom extern model is compelling. The control plane CRUDXPS will tie nicely into P4Runtime and TDI. I have an application which needs to run purely in SW - no HW offload, so prior suggestions to wait for it to "approve" this is frustrating. I could use this yesterday. Furthermore, as an active contributor to sonic-dash, where we model the pipeline in P4, I can state that P4TC could be a compelling alternative to bmv2, which is slow, long in the tooth and lacks PNA support.
I beseech the NACKers to take a deep breath, reevaluate any entrenched positions and consider how much goodness this will add, even if this is not your preference for implementing datapaths. It doesn't have to be. That can and should be decided by the larger community. This could open the door to thousands of creative developers who are comfortable in P4 but not adept in low-level networking code. P4 had a significant impact on democratizing network programming, and that was just on bmv2 and Tofino, which is EOL. Making performant and powerful P4TC ubiquitous on virtually any Linux server could have a similar effect, just like eBPF opened a lot of doors to non-kernel programmers to do interesting things. Be a part of that transformation!
A message that is not in the archive (sent off-list or not yet received): SN6PR17MB211069668AF4C8031B116B9D96EB2@SN6PR17MB2110.namprd17.prod.outlook.com
Re: On the NACKs on P4TC patches
Quoted 46 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$
Quoted 8 lines
+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting? Considering that P4 syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs). Personally, I don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://netdevconf.info/0x16/papers/11/High%20Performance%20Programmable%20Parsers.pdf.
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use P4->eBPF compiler.
Tom
Quoted 27 lines
Quoted 7 lines
I am not sure where the parser thing comes from - the parser is generated as eBPF.
Quoted 3 lines
So from my perspective, the submission is neither technically strong enough, nor broadly useful enough to consider making questionable precedents for, i.e. to override maintainers on how their subsystems are extended.
I disagree vehemently on the "broadly useful enough" comment.
Quoted 14 lines
I believe as a community nobody should just have the power to nack things just because - as i stated in the page, not even Linus. That code doesnt touch anything to do with eBPF maintainers (meaning things they have to fix when an issue shows up) neither does it "extend" as you state any ebpf code and it is all part of the networking subsystem. Sure, anybody has the right to nack but I contend that nacks should be based on technical reasons. I have listed all the objections in that page and how i have responded to them over time. Someone needs to look at those objectively and say if they are valid. The arguement made so far(By Paolo and now by you) is "we cant override maintainers on how their subsystems are used" then we are in uncharted territory, thats why i am asking for arbitration.
Quoted 2 lines
cheers,
jamalMaintainers: I am perplexed and dismayed that this is getting so much pushback. None of the objections, regardless of their merits (or not) seem to outweigh the potential benefits to end-users. I am extremely interested in using P4TC, it adds a lot of value and reuses so much existing Linux infra. The custom extern model is compelling. The control plane CRUDXPS will tie nicely into P4Runtime and TDI. I have an application which needs to run purely in SW - no HW offload, so prior suggestions to wait for it to "approve" this is frustrating. I could use this yesterday. Furthermore, as an active contributor to sonic-dash, where we model the pipeline in P4, I can state that P4TC could be a compelling alternative to bmv2, which is slow, long in the tooth and lacks PNA support.
I beseech the NACKers to take a deep breath, reevaluate any entrenched positions and consider how much goodness this will add, even if this is not your preference for implementing datapaths. It doesn't have to be. That can and should be decided by the larger community. This could open the door to thousands of creative developers who are comfortable in P4 but not adept in low-level networking code. P4 had a significant impact on democratizing network programming, and that was just on bmv2 and Tofino, which is EOL. Making performant and powerful P4TC ubiquitous on virtually any Linux server could have a similar effect, just like eBPF opened a lot of doors to non-kernel programmers to do interesting things. Be a part of that transformation!
Jamal Hadi Salim
maintainerDSL vs low level language WAS(Re: On the NACKs on P4TC patches
Quoted 82 lines · Tom Herbert
Quoted 46 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$Quoted 8 lines
+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting? Considering that P4 syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs). Personally, I don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://netdevconf.info/0x16/papers/11/High%20Performance%20Programmable%20Parsers.pdf.
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.
cheers,
jamal
cheers,
jamal
RE: DSL vs low level language WAS(Re: On the NACKs on P4TC patches
Quoted 57 lines · Tom Herbert
Quoted 56 lines
Quoted 53 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting?
Hi Tom, thanks for the dialog. To answer your question, it took seconds to compile and deploy, not learn P4. Adding the parsing for several headers took minutes. If you want to compare learning curve, learning to write P4 code and let the framework handle all the painful low-level Linux details is way easier than trying to learn how to write c code for Linux networking. It’s not even close. I’ve written C for 40 years, P4 for 7 years, and dabbled in eBPF so I can attest to the ease of learning and using P4. I’ve onboarded and mentored engineers who barely knew C, to develop complex networking products using P4, and built the automation APIs (REST, gRPC) to manage them. One person can develop an entire commercial product by themselves in months. P4 has expanded the reach of programmers such that both HW and SW engineers can easily learn P4 and become pretty adept at it. I would not expect even experienced c programmers to be able to master Linux internals very quickly. Writing a P4-TC program and injecting it via tc was like magic the first time.
Quoted 7 lines
Quoted 7 lines
Considering that P4
syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.
I think your statement about “typical network programmers” overlooks the fact that since P4 was introduced, it has been taught in many universities to teach networking and possibly enabled a whole new breed of “network engineers” who can solve real problems without even knowing C programming. Without P4 they might never have gone this route. A class in network stack programming using c would have so many prerequisites to even get to parsing, compared to P4, where it could be demonstrated in one lesson. These “networking programmers” are not typical by your standards, but there are many such. They have just as much claim to the title "network programmer” as a C programmer. Similarly, an assembly language programmer is no less than a C or Python programmer. People writing P4 are usually focused on applications, and it is very useful and productive for that. Why should someone have to learn low-level C or eBPF to solve their problem?
Quoted 8 lines
Quoted 8 lines
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs).
A pretty strong opinion, but DSLs arise to fill a need and P4 did so. It's still going strong.
Quoted 5 lines
Quoted 5 lines
Personally, I
don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://urldefense.com/v3/__https://netdevconf.info/0x16/papers/11/High*20Performance*20Programmable*20Parsers.pdf__;JSUl!!I5pVk4LIGAfnvw!m9zrSDvddfzSt_sMBjOEvqw31RzAwWlEDM4ah5IJ2kqsmq6XtPIVJd-1_ZoGWBXKLyda77RYLvGR83Ginw$.
CPL (slide11) looks like a DSL wrapped in JSON to me. “Solution: Common Parser Language (CPL); Parser representation in declarative .json” So I am confused. It is either a new language a.k.a. DSL, or it's not. Nothing against it, I'm sure it is great, but let's call it what it is. We already have parser representations in declarative p4. And it's used and known worldwide. And has a respectable specification, any users and working groups. And it's formally provable (https://github.com/verified-network-toolchain/petr4)
Quoted 5 lines
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.
“ubiquitous way to program the kernel” – is not my goal. I don’t even want to know about the kernel when I am writing p4 - it's just a means to an end. I want to manipulate packets on a Linux host. P4DPDK, P4-eBPF, P4-TC – all let me do that. I LOVE the fact that P4-TC would be available in every Linux distro once upstreamed. It would solve so many deployment issues, benefit from regression testing, etc. So much goodness.
" and for those users that want to use P4 they can use P4->eBPF compiler." -I'd really like to choose for myself and not have someone make that choice for me. P4-TC checks all the boxes for me.
Thanks for the point of view, it's healthy to debate. Cheers,
Chris
Quoted 11 lines
Quoted 1 line
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.
cheers,
jamal
Re: DSL vs low level language WAS(Re: On the NACKs on P4TC patches
Quoted 76 lines · Chris Sommers
Quoted 58 lines · Tom Herbert
Quoted 56 lines
Quoted 53 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting?
Hi Tom, thanks for the dialog. To answer your question, it took seconds to compile and deploy, not learn P4. Adding the parsing for several headers took minutes. If you want to compare learning curve, learning to write P4 code and let the framework handle all the painful low-level Linux details is way easier than trying to learn how to write c code for Linux networking. It’s not even close. I’ve written C for 40 years, P4 for 7 years, and dabbled in eBPF so I can attest to the ease of learning and using P4. I’ve onboarded and mentored engineers who barely knew C, to develop complex networking products using P4, and built the automation APIs (REST, gRPC) to manage them. One person can develop an entire commercial product by themselves in months. P4 has expanded the reach of programmers such that both HW and SW engineers can easily learn P4 and become pretty adept at it. I would not expect even experienced c programmers to be able to master Linux internals very quickly. Writing a P4-TC program and injecting it via tc was like magic the first time.
Quoted 7 lines
Quoted 7 lines
Considering that P4
syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.
Quoted 1 line
I think your statement about “typical network programmers” overlooks the fact that since P4 was introduced, it has been taught in many universities to teach networking and possibly enabled a whole new breed of “network engineers” who can solve real problems without even knowing C programming. Without P4 they might never have gone this route. A class in network stack programming using c would have so many prerequisites to even get to parsing, compared to P4, where it could be demonstrated in one lesson. These “networking programmers” are not typical by your standards, but there are many such. They have just as much claim to the title "network programmer” as a C programmer. Similarly, an assembly language programmer is no less than a C or Python programmer. People writing P4 are usually focused on applications, and it is very useful and productive for that. Why should someone have to learn low-level C or eBPF to solve their problem?
Hio Chris,
You're comparing learning a completely new language versus programming in a subset of an established language, they're really not comparable. When one programs in Restricted-C they just need to understand what features of C are supported.
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs).
A pretty strong opinion, but DSLs arise to fill a need and P4 did so. It's still going strong.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Personally, I
don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://urldefense.com/v3/__https://netdevconf.info/0x16/papers/11/High*20Performance*20Programmable*20Parsers.pdf__;JSUl!!I5pVk4LIGAfnvw!m9zrSDvddfzSt_sMBjOEvqw31RzAwWlEDM4ah5IJ2kqsmq6XtPIVJd-1_ZoGWBXKLyda77RYLvGR83Ginw$.CPL (slide11) looks like a DSL wrapped in JSON to me. “Solution: Common Parser Language (CPL); Parser representation in declarative .json” So I am confused. It is either a new language a.k.a. DSL, or it's not. Nothing against it, I'm sure it is great, but let's call it what it is.
Correct, it's not a new language. We've since renamed it Common Parser Representation.
Quoted 11 lines
We already have parser representations in declarative p4. And it's used and known worldwide. And has a respectable specification, any users and working groups. And it's formally provable (https://github.com/verified-network-toolchain/petr4)
Quoted 5 lines
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.“ubiquitous way to program the kernel” – is not my goal. I don’t even want to know about the kernel when I am writing p4 - it's just a means to an end. I want to manipulate packets on a Linux host. P4DPDK, P4-eBPF, P4-TC – all let me do that. I LOVE the fact that P4-TC would be available in every Linux distro once upstreamed. It would solve so many deployment issues, benefit from regression testing, etc. So much goodness
" and for those users that want to use P4 they can use
P4->eBPFcompiler." -I'd really like to choose for myself and not have someone make that choice for me. P4-TC checks all the boxes for me.
Sure, but this is a lot of kernel code and that will require support and maintenance. It needs to be justified, and the fact that someone wants it just to have a choice is, frankly, not much of a justification. I think a justification needs to start with "Why isn't P4->eBPF sufficient?" (the question has been raised several times, but it still doesn't seem like there's a strong answer).
Tom
Quoted 17 lines
Thanks for the point of view, it's healthy to debate. Cheers,
ChrisQuoted 11 lines
Quoted 1 line
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.cheers,
jamal
Re: DSL vs low level language WAS(Re: On the NACKs on P4TC patches
Hi Chris,
P4 was created to support programming the hardware data path in high end routers, but P4-TC would enable the use of P4 across all Linux devices. Since this is potentially a lot of code going into the kernel to support it, I believe it's entirely fair for us to evaluate and give feedback on the P4 language and its suitability for the broader user community including environments where there will never be a need for P4 hardware. Note that I am questioning the design decisions of P4 in the context of supporting a DSL in the kernel via P4-TC, if the P4->eBPF compiler is used then then these concerns are less pertinent. Nevertheless, I would suggest that the P4 folks take the points being raised as constructive feedback on the language.
I took a cursory look at several P4 programs including tutorials, switch code, firewalls, etc. I have particular interest in variable length headers, so I'll use
https://github.com/jafingerhut/p4-guide/blob/master/checksum/checksum-ipv4-with-options.p4
as a reference.
The first thing I noticed about P4 is that almost everything is expressed as a bit field. Like bit<8> and bit<32>. I suppose this arises from the fact that P4 was originally intended to run in non-CPU hardware where there's no inherent unit of data like bytes. But, CPUs don't work that way; CPUs work ordinal types of bytes, half words, words, double words, etc. (__u8, __u16, __u32, __u64). That means that all mainstream computer languages fundamentally operate on ordinal types even if the variable types are explicitly declared. If someone programming in P4 needs to map original types to bit fields in P4, so if they want a __u32 they need to use a bit<32> in P4 (except they're not exactly equivalent, a __u32 in C is guaranteed to be byte aligned and I'm assuming in P4 bit<32> is not guaranteed to be byte aligned-- this seems like it might be susceptible to programming errors). I'd also point out that networking protocols are also defined using ordinal type fields, there are some exceptions, but for the most part protocol fields try to be in units of bytes (or octets if you want to be old school!). I believe life would be easier for the programmer if they could just define variables and fields with ordinal types, the fix here seems simple enough just add typedefs to P4 like "typedef __u32 bit<32>".
In the IP header definition there's "varbit<320> options;". It took me several seconds to decode this and realize this is space for forty bytes of IP options (i.e. 8 * 40 == 320). I suppose this follows the design of using bit fields for everything, but I think this is more than just an annoyance like the bit fields for ordinal types are. First off, it's not very readable. I've never heard anyone say that there's 320 bits of IP options, or seen an RFC specify that. Likewise, the standard Ethernet MTU is 1500 bytes, not 12,000 bits which would seem to be how that would be expressed in P4. So this seems very unreadable to me and potentially prone to errors. The fix for this also seems easy, why not just add varbyte to P4 so we can do varbyte<40>, varbyte<87>, varbyte<123>, etc.?
The next thing I notice about the P4 programs I surveyed is that all of them seem to define the protocol headers within the protocol. Every program seems to have "header ethernet_t" and "header ipv4_t" and other protocols that are used and protocol constants like Ethertypes also seem to be spelled out in each program. Sometimes these are in include files within the program. What I don't see is that P4 has a standard set of include files for defining protocol headers. For instance, in Linux C we would just do "#include <linux/if_ether.h>" and "#include <linux/ip.h>" to get the definitions of the Ethernet header and IPv4 header. In fact, if someone were to submit a patch to Netdev that included its own definition of Ethernet or an IP header structure they would almost certainly get pushback. It's a fundamental programming principle, not just in networking but pretty much everywhere, to not continuously redefine common and standard constructs-- just put common things in header files that can be shared by multiple programs (to do otherwise substantially increases the possibility of errors, bloats code, and reduces readability).
Marshalling up common definitions into header files that are common in the P4 development environment seems simple enough (maybe it's already done?), but I would also point out that Linux has included files that describe protocol formats and header structures for almost every protocol under the sun that are well tested. It would be great if somehow we could somehow leverage that work. For instance, in the P4 samples I looked at srcAddr and dstAddr are defined for IP addresses, but in linux/ip.h their saddr and daddr are the respective field names. Why not just base the P4 definition on the Linux one? Then when someone is porting code from Linux to P4 they can use the same field names-- this makes things a lot easier on the programmer! I'll also mention that we wrote a little Python script to generate P4 header and constant definitions from Linux headers. It almost worked, the snag we hit was that P4 has some limits on nesting structures and unions so we couldn't translate some of the C structures to P4 (if you're interested I can provide the details on the problem we hit).
The IPv4 header checksum code was a real head scratcher for me. Do we really need to state each field in the IP header just to compute the checksum? (and not just do this once, but twice :-( ). See code below for verifyChecksum and updateChecksum.
In C, verifying and setting the IP header checksum is really easy:
if (checksum(iphdr, 0, iphdr->ihl << 4))
goto bad_csum;
ip->csum = checksum(iphdr, 0, iphdr->ihl << 4);
Relative to the C code, the P4 code seems very convoluted to me and prone to errors. What if someone accidentally omits a field? What if fields become slightly out of order? Also, no one would ever describe the IPv4 checksum as taking the checksum over the IHL, diffserv, totalLen, ... That is *way* too complicated for an algorithm that is really simple-- from RFC791: "The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header.". Reverse engineering the design, the clue seems to be HashAlgorithm.csum16. Maybe in P4 the IP checksum is just considered another form of hash, and I suspect the input to hash computation is specified as sort of data structure to make things generic (for instance, how we create a substructure in flow keys in flow_dissector to compute a SipHash over the TCP and UDP tuple). But, the IPv4 checksum isn't just another hash-- on a host, we need to compute the checksum for *every* IPv4 packet. This has to be fast and simple, we can do this in as few as five instructions or less. So even if the code below is correct, I have to wonder how easy it is to emit an efficient executable. Would a compiler easily realize that all the fields in the pseudo structure are contiguous without holes such that it can omit those five instructions?
I don't know how prevalent this method of listing all the fields in a data structure as arguments to a function is in P4, but, by almost any objective measure, I have to say that the code below is bad and bloated. Maybe there's a better way to do it in P4, but if there's not then this is a deficiency in the P4 language.
Tom
control verifyChecksum(inout headers hdr,
inout metadata meta)
{
apply {
// There is code similar to this in Github repo p4lang/p4c in
// file testdata/p4_16_samples/flowlet_switching-bmv2.p4
// However in that file it is only for a fixed length IPv4
// header with no options.
verify_checksum(true,
{ hdr.ipv4.version,
hdr.ipv4.ihl,
hdr.ipv4.diffserv,
hdr.ipv4.totalLen,
hdr.ipv4.identification,
hdr.ipv4.flags,
hdr.ipv4.fragOffset,
hdr.ipv4.ttl,
hdr.ipv4.protocol,
hdr.ipv4.srcAddr,
hdr.ipv4.dstAddr
#ifdef ALLOW_IPV4_OPTIONS
, hdr.ipv4.options
#endif /* ALLOW_IPV4_OPTIONS */
},
hdr.ipv4.hdrChecksum, HashAlgorithm.csum16);
}
}control updateChecksum(inout headers hdr,
inout metadata meta)
{
apply {
update_checksum(true,
{ hdr.ipv4.version,
hdr.ipv4.ihl,
hdr.ipv4.diffserv,
hdr.ipv4.totalLen,
hdr.ipv4.identification,
hdr.ipv4.flags,
hdr.ipv4.fragOffset,
hdr.ipv4.ttl,
hdr.ipv4.protocol,
hdr.ipv4.srcAddr,
hdr.ipv4.dstAddr
#ifdef ALLOW_IPV4_OPTIONS
, hdr.ipv4.options
#endif /* ALLOW_IPV4_OPTIONS */
},
hdr.ipv4.hdrChecksum, HashAlgorithm.csum16);
}
}Quoted 148 lines · Tom Herbert
Quoted 76 lines · Chris Sommers
Quoted 58 lines · Tom Herbert
Quoted 56 lines
Quoted 53 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting?
Hi Tom, thanks for the dialog. To answer your question, it took seconds to compile and deploy, not learn P4. Adding the parsing for several headers took minutes. If you want to compare learning curve, learning to write P4 code and let the framework handle all the painful low-level Linux details is way easier than trying to learn how to write c code for Linux networking. It’s not even close. I’ve written C for 40 years, P4 for 7 years, and dabbled in eBPF so I can attest to the ease of learning and using P4. I’ve onboarded and mentored engineers who barely knew C, to develop complex networking products using P4, and built the automation APIs (REST, gRPC) to manage them. One person can develop an entire commercial product by themselves in months. P4 has expanded the reach of programmers such that both HW and SW engineers can easily learn P4 and become pretty adept at it. I would not expect even experienced c programmers to be able to master Linux internals very quickly. Writing a P4-TC program and injecting it via tc was like magic the first time.
Quoted 7 lines
Quoted 7 lines
Considering that P4
syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.Quoted 1 line
I think your statement about “typical network programmers” overlooks the fact that since P4 was introduced, it has been taught in many universities to teach networking and possibly enabled a whole new breed of “network engineers” who can solve real problems without even knowing C programming. Without P4 they might never have gone this route. A class in network stack programming using c would have so many prerequisites to even get to parsing, compared to P4, where it could be demonstrated in one lesson. These “networking programmers” are not typical by your standards, but there are many such. They have just as much claim to the title "network programmer” as a C programmer. Similarly, an assembly language programmer is no less than a C or Python programmer. People writing P4 are usually focused on applications, and it is very useful and productive for that. Why should someone have to learn low-level C or eBPF to solve their problem?
Hio Chris,
You're comparing learning a completely new language versus programming in a subset of an established language, they're really not comparable. When one programs in Restricted-C they just need to understand what features of C are supported.
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs).
A pretty strong opinion, but DSLs arise to fill a need and P4 did so. It's still going strong.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Personally, I
don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://urldefense.com/v3/__https://netdevconf.info/0x16/papers/11/High*20Performance*20Programmable*20Parsers.pdf__;JSUl!!I5pVk4LIGAfnvw!m9zrSDvddfzSt_sMBjOEvqw31RzAwWlEDM4ah5IJ2kqsmq6XtPIVJd-1_ZoGWBXKLyda77RYLvGR83Ginw$.CPL (slide11) looks like a DSL wrapped in JSON to me. “Solution: Common Parser Language (CPL); Parser representation in declarative .json” So I am confused. It is either a new language a.k.a. DSL, or it's not. Nothing against it, I'm sure it is great, but let's call it what it is.
Correct, it's not a new language. We've since renamed it Common Parser Representation.
Quoted 11 lines
We already have parser representations in declarative p4. And it's used and known worldwide. And has a respectable specification, any users and working groups. And it's formally provable (https://github.com/verified-network-toolchain/petr4)
Quoted 5 lines
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.“ubiquitous way to program the kernel” – is not my goal. I don’t even want to know about the kernel when I am writing p4 - it's just a means to an end. I want to manipulate packets on a Linux host. P4DPDK, P4-eBPF, P4-TC – all let me do that. I LOVE the fact that P4-TC would be available in every Linux distro once upstreamed. It would solve so many deployment issues, benefit from regression testing, etc. So much goodness
" and for those users that want to use P4 they can use
P4->eBPFcompiler." -I'd really like to choose for myself and not have someone make that choice for me. P4-TC checks all the boxes for me.Sure, but this is a lot of kernel code and that will require support and maintenance. It needs to be justified, and the fact that someone wants it just to have a choice is, frankly, not much of a justification. I think a justification needs to start with "Why isn't
P4->eBPFsufficient?" (the question has been raised several times, but it still doesn't seem like there's a strong answer).Tom
Quoted 17 lines
Thanks for the point of view, it's healthy to debate. Cheers,
ChrisQuoted 11 lines
Quoted 1 line
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.cheers,
jamal
Jamal Hadi Salim
maintainerRe: DSL vs low level language WAS(Re: On the NACKs on P4TC patches
Quoted 15 lines · Tom Herbert
Hi Chris,
P4 was created to support programming the hardware data path in high end routers, but P4-TC would enable the use of P4 across all Linux devices. Since this is potentially a lot of code going into the kernel to support it, I believe it's entirely fair for us to evaluate and give feedback on the P4 language and its suitability for the broader user community including environments where there will never be a need for P4 hardware. Note that I am questioning the design decisions of P4 in the context of supporting a DSL in the kernel via P4-TC, if the
P4->eBPFcompiler is used then then these concerns are less pertinent. Nevertheless, I would suggest that the P4 folks take the points being raised as constructive feedback on the language.
A lot of misleading info there. The P4 PNA architecture is for end hosts not routers. For some NIC vendors you can go as far as writting hardware GRO or TSO offload or variations of your liking using P4 (cretainly not a middle feature). That notwithstanding the idea of offloading match-action via TC is not new and has been widely used/adopted for end hosts.
Tom, you want to perhaps disclose that you have a competing product? That will help provide better context on your angle.
TBH, I am confused by what your end game is - is your view that a crusade against P4 will make you sell more of your product? I have 3 NICs here with me (from 2 vendors) that are P4 programmable. You can be as negative as you want about P4 but you are not going to make it go away, sorry.
I will let Chris or whoever else on Cc respond to the P4 bits if they wishe because there's misunderstanding there as well.
cheers,
jamal
Quoted 317 lines
I took a cursory look at several P4 programs including tutorials, switch code, firewalls, etc. I have particular interest in variable length headers, so I'll use
https://github.com/jafingerhut/p4-guide/blob/master/checksum/checksum-ipv4-with-options.p4
as a reference.The first thing I noticed about P4 is that almost everything is expressed as a bit field. Like bit<8> and bit<32>. I suppose this arises from the fact that P4 was originally intended to run in non-CPU hardware where there's no inherent unit of data like bytes. But, CPUs don't work that way; CPUs work ordinal types of bytes, half words, words, double words, etc. (
__u8,__u16,__u32,__u64). That means that all mainstream computer languages fundamentally operate on ordinal types even if the variable types are explicitly declared. If someone programming in P4 needs to map original types to bit fields in P4, so if they want a__u32they need to use a bit<32> in P4 (except they're not exactly equivalent, a__u32in C is guaranteed to be byte aligned and I'm assuming in P4 bit<32> is not guaranteed to be byte aligned-- this seems like it might be susceptible to programming errors). I'd also point out that networking protocols are also defined using ordinal type fields, there are some exceptions, but for the most part protocol fields try to be in units of bytes (or octets if you want to be old school!). I believe life would be easier for the programmer if they could just define variables and fields with ordinal types, the fix here seems simple enough just add typedefs to P4 like "typedef__u32bit<32>".In the IP header definition there's "varbit<320> options;". It took me several seconds to decode this and realize this is space for forty bytes of IP options (i.e. 8 * 40 == 320). I suppose this follows the design of using bit fields for everything, but I think this is more than just an annoyance like the bit fields for ordinal types are. First off, it's not very readable. I've never heard anyone say that there's 320 bits of IP options, or seen an RFC specify that. Likewise, the standard Ethernet MTU is 1500 bytes, not 12,000 bits which would seem to be how that would be expressed in P4. So this seems very unreadable to me and potentially prone to errors. The fix for this also seems easy, why not just add varbyte to P4 so we can do varbyte<40>, varbyte<87>, varbyte<123>, etc.?
The next thing I notice about the P4 programs I surveyed is that all of them seem to define the protocol headers within the protocol. Every program seems to have "header
ethernet_t" and "headeripv4_t" and other protocols that are used and protocol constants like Ethertypes also seem to be spelled out in each program. Sometimes these are in include files within the program. What I don't see is that P4 has a standard set of include files for defining protocol headers. For instance, in Linux C we would just do "#include <linux/if_ether.h>" and "#include <linux/ip.h>" to get the definitions of the Ethernet header and IPv4 header. In fact, if someone were to submit a patch to Netdev that included its own definition of Ethernet or an IP header structure they would almost certainly get pushback. It's a fundamental programming principle, not just in networking but pretty much everywhere, to not continuously redefine common and standard constructs-- just put common things in header files that can be shared by multiple programs (to do otherwise substantially increases the possibility of errors, bloats code, and reduces readability).Marshalling up common definitions into header files that are common in the P4 development environment seems simple enough (maybe it's already done?), but I would also point out that Linux has included files that describe protocol formats and header structures for almost every protocol under the sun that are well tested. It would be great if somehow we could somehow leverage that work. For instance, in the P4 samples I looked at srcAddr and dstAddr are defined for IP addresses, but in
linux/ip.htheir saddr and daddr are the respective field names. Why not just base the P4 definition on the Linux one? Then when someone is porting code from Linux to P4 they can use the same field names-- this makes things a lot easier on the programmer! I'll also mention that we wrote a little Python script to generate P4 header and constant definitions from Linux headers. It almost worked, the snag we hit was that P4 has some limits on nesting structures and unions so we couldn't translate some of the C structures to P4 (if you're interested I can provide the details on the problem we hit).The IPv4 header checksum code was a real head scratcher for me. Do we really need to state each field in the IP header just to compute the checksum? (and not just do this once, but twice :-( ). See code below for verifyChecksum and updateChecksum.
In C, verifying and setting the IP header checksum is really easy:
if (
checksum(iphdr, 0, iphdr->ihl << 4))goto bad_csum;
ip->csum=checksum(iphdr, 0, iphdr->ihl << 4);Relative to the C code, the P4 code seems very convoluted to me and prone to errors. What if someone accidentally omits a field? What if fields become slightly out of order? Also, no one would ever describe the IPv4 checksum as taking the checksum over the IHL, diffserv, totalLen, ... That is *way* too complicated for an algorithm that is really simple-- from RFC791: "The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header.". Reverse engineering the design, the clue seems to be HashAlgorithm.csum16. Maybe in P4 the IP checksum is just considered another form of hash, and I suspect the input to hash computation is specified as sort of data structure to make things generic (for instance, how we create a substructure in flow keys in
flow_dissectorto compute a SipHash over the TCP and UDP tuple). But, the IPv4 checksum isn't just another hash-- on a host, we need to compute the checksum for *every* IPv4 packet. This has to be fast and simple, we can do this in as few as five instructions or less. So even if the code below is correct, I have to wonder how easy it is to emit an efficient executable. Would a compiler easily realize that all the fields in the pseudo structure are contiguous without holes such that it can omit those five instructions?I don't know how prevalent this method of listing all the fields in a data structure as arguments to a function is in P4, but, by almost any objective measure, I have to say that the code below is bad and bloated. Maybe there's a better way to do it in P4, but if there's not then this is a deficiency in the P4 language.
Tom
control verifyChecksum(inout headers hdr,
inout metadata meta){ apply { // There is code similar to this in Github repo p4lang/p4c in // file testdata/p4_16_samples/flowlet_switching-bmv2.p4 // However in that file it is only for a fixed length IPv4 // header with no options. verify_checksum(true, { hdr.ipv4.version, hdr.ipv4.ihl, hdr.ipv4.diffserv, hdr.ipv4.totalLen, hdr.ipv4.identification, hdr.ipv4.flags, hdr.ipv4.fragOffset, hdr.ipv4.ttl, hdr.ipv4.protocol, hdr.ipv4.srcAddr, hdr.ipv4.dstAddr #ifdef ALLOW_IPV4_OPTIONS , hdr.ipv4.options #endif /* ALLOW_IPV4_OPTIONS */ }, hdr.ipv4.hdrChecksum, HashAlgorithm.csum16); } }control updateChecksum(inout headers hdr,
inout metadata meta){ apply { update_checksum(true, { hdr.ipv4.version, hdr.ipv4.ihl, hdr.ipv4.diffserv, hdr.ipv4.totalLen, hdr.ipv4.identification, hdr.ipv4.flags, hdr.ipv4.fragOffset, hdr.ipv4.ttl, hdr.ipv4.protocol, hdr.ipv4.srcAddr, hdr.ipv4.dstAddr #ifdef ALLOW_IPV4_OPTIONS , hdr.ipv4.options #endif /* ALLOW_IPV4_OPTIONS */ }, hdr.ipv4.hdrChecksum, HashAlgorithm.csum16); } }Quoted 148 lines · Tom Herbert
Quoted 76 lines · Chris Sommers
Quoted 58 lines · Tom Herbert
Quoted 56 lines
Quoted 53 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting?
Hi Tom, thanks for the dialog. To answer your question, it took seconds to compile and deploy, not learn P4. Adding the parsing for several headers took minutes. If you want to compare learning curve, learning to write P4 code and let the framework handle all the painful low-level Linux details is way easier than trying to learn how to write c code for Linux networking. It’s not even close. I’ve written C for 40 years, P4 for 7 years, and dabbled in eBPF so I can attest to the ease of learning and using P4. I’ve onboarded and mentored engineers who barely knew C, to develop complex networking products using P4, and built the automation APIs (REST, gRPC) to manage them. One person can develop an entire commercial product by themselves in months. P4 has expanded the reach of programmers such that both HW and SW engineers can easily learn P4 and become pretty adept at it. I would not expect even experienced c programmers to be able to master Linux internals very quickly. Writing a P4-TC program and injecting it via tc was like magic the first time.
Quoted 7 lines
Quoted 7 lines
Considering that P4
syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.Quoted 1 line
I think your statement about “typical network programmers” overlooks the fact that since P4 was introduced, it has been taught in many universities to teach networking and possibly enabled a whole new breed of “network engineers” who can solve real problems without even knowing C programming. Without P4 they might never have gone this route. A class in network stack programming using c would have so many prerequisites to even get to parsing, compared to P4, where it could be demonstrated in one lesson. These “networking programmers” are not typical by your standards, but there are many such. They have just as much claim to the title "network programmer” as a C programmer. Similarly, an assembly language programmer is no less than a C or Python programmer. People writing P4 are usually focused on applications, and it is very useful and productive for that. Why should someone have to learn low-level C or eBPF to solve their problem?
Hio Chris,
You're comparing learning a completely new language versus programming in a subset of an established language, they're really not comparable. When one programs in Restricted-C they just need to understand what features of C are supported.
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs).
A pretty strong opinion, but DSLs arise to fill a need and P4 did so. It's still going strong.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Personally, I
don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://urldefense.com/v3/__https://netdevconf.info/0x16/papers/11/High*20Performance*20Programmable*20Parsers.pdf__;JSUl!!I5pVk4LIGAfnvw!m9zrSDvddfzSt_sMBjOEvqw31RzAwWlEDM4ah5IJ2kqsmq6XtPIVJd-1_ZoGWBXKLyda77RYLvGR83Ginw$.CPL (slide11) looks like a DSL wrapped in JSON to me. “Solution: Common Parser Language (CPL); Parser representation in declarative .json” So I am confused. It is either a new language a.k.a. DSL, or it's not. Nothing against it, I'm sure it is great, but let's call it what it is.
Correct, it's not a new language. We've since renamed it Common Parser Representation.
Quoted 11 lines
We already have parser representations in declarative p4. And it's used and known worldwide. And has a respectable specification, any users and working groups. And it's formally provable (https://github.com/verified-network-toolchain/petr4)
Quoted 5 lines
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.“ubiquitous way to program the kernel” – is not my goal. I don’t even want to know about the kernel when I am writing p4 - it's just a means to an end. I want to manipulate packets on a Linux host. P4DPDK, P4-eBPF, P4-TC – all let me do that. I LOVE the fact that P4-TC would be available in every Linux distro once upstreamed. It would solve so many deployment issues, benefit from regression testing, etc. So much goodness
" and for those users that want to use P4 they can use
P4->eBPFcompiler." -I'd really like to choose for myself and not have someone make that choice for me. P4-TC checks all the boxes for me.Sure, but this is a lot of kernel code and that will require support and maintenance. It needs to be justified, and the fact that someone wants it just to have a choice is, frankly, not much of a justification. I think a justification needs to start with "Why isn't
P4->eBPFsufficient?" (the question has been raised several times, but it still doesn't seem like there's a strong answer).Tom
Quoted 17 lines
Thanks for the point of view, it's healthy to debate. Cheers,
ChrisQuoted 11 lines
Quoted 1 line
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.cheers,
jamal
RE: DSL vs low level language WAS(Re: On the NACKs on P4TC patches
In reply to Tom HerbertOops, resending as plaintext. Sigh...
-
Quoted 16 lines · Tom Herbert
Quoted 15 lines
Hi Chris,
P4 was created to support programming the hardware data path in high end routers, but P4-TC would enable the use of P4 across all Linux devices. Since this is potentially a lot of code going into the kernel to support it, I believe it's entirely fair for us to evaluate and give feedback on the P4 language and its suitability for the broader user community including environments where there will never be a need for P4 hardware. Note that I am questioning the design decisions of P4 in the context of supporting a DSL in the kernel via P4-TC, if the
P4->eBPFcompiler is used then then these concerns are less pertinent. Nevertheless, I would suggest that the P4 folks take the points being raised as constructive feedback on the language.
Hi Tom,
RE: Your observations and feedback on P4 language and prevalent coding practices, the most constructive approach would be to attend P4 working group meetings where your opinions and ideas will be respectfully considered and your offer to help gratefully accepted. You can also file issues or pull requests on GitHub. The Language and Architecture working groups would probably be the best places to participate. We are an open-minded and welcoming group of volunteers from industry and academia who are always looking for new members. It sounds like you have lots of relevant experience and a different point of view which could add hybrid vigor. Chris Sommers
Distinguished SW Engineer
Quoted 340 lines
A lot of misleading info there. The P4 PNA architecture is for end hosts not routers. For some NIC vendors you can go as far as writting hardware GRO or TSO offload or variations of your liking using P4 (cretainly not a middle feature). That notwithstanding the idea of offloading match-action via TC is not new and has been widely used/adopted for end hosts.
Tom, you want to perhaps disclose that you have a competing product? That will help provide better context on your angle.
TBH, I am confused by what your end game is - is your view that a crusade against P4 will make you sell more of your product? I have 3 NICs here with me (from 2 vendors) that are P4 programmable. You can be as negative as you want about P4 but you are not going to make it go away, sorry.I will let Chris or whoever else on Cc respond to the P4 bits if they wishe because there's misunderstanding there as well.
cheers,
jamalQuoted 317 lines
I took a cursory look at several P4 programs including tutorials, switch code, firewalls, etc. I have particular interest in variable length headers, so I'll use
https://urldefense.com/v3/__https://github.com/jafingerhut/p4-guide/blob/master/checksum/checksum-ipv4-with-options.p4__;!!I5pVk4LIGAfnvw!juhSwk9UTheuI8-0mudbGTSZ_GBx3Z6hmcOAgiaAW14Ecter6K8iJ8DSzakN1d4GCE4uFJ05wkE81N6KNw$
as a reference.The first thing I noticed about P4 is that almost everything is expressed as a bit field. Like bit<8> and bit<32>. I suppose this arises from the fact that P4 was originally intended to run in non-CPU hardware where there's no inherent unit of data like bytes. But, CPUs don't work that way; CPUs work ordinal types of bytes, half words, words, double words, etc. (
__u8,__u16,__u32,__u64). That means that all mainstream computer languages fundamentally operate on ordinal types even if the variable types are explicitly declared. If someone programming in P4 needs to map original types to bit fields in P4, so if they want a__u32they need to use a bit<32> in P4 (except they're not exactly equivalent, a__u32in C is guaranteed to be byte aligned and I'm assuming in P4 bit<32> is not guaranteed to be byte aligned-- this seems like it might be susceptible to programming errors). I'd also point out that networking protocols are also defined using ordinal type fields, there are some exceptions, but for the most part protocol fields try to be in units of bytes (or octets if you want to be old school!). I believe life would be easier for the programmer if they could just define variables and fields with ordinal types, the fix here seems simple enough just add typedefs to P4 like "typedef__u32bit<32>".In the IP header definition there's "varbit<320> options;". It took me several seconds to decode this and realize this is space for forty bytes of IP options (i.e. 8 * 40 == 320). I suppose this follows the design of using bit fields for everything, but I think this is more than just an annoyance like the bit fields for ordinal types are. First off, it's not very readable. I've never heard anyone say that there's 320 bits of IP options, or seen an RFC specify that. Likewise, the standard Ethernet MTU is 1500 bytes, not 12,000 bits which would seem to be how that would be expressed in P4. So this seems very unreadable to me and potentially prone to errors. The fix for this also seems easy, why not just add varbyte to P4 so we can do varbyte<40>, varbyte<87>, varbyte<123>, etc.?
The next thing I notice about the P4 programs I surveyed is that all of them seem to define the protocol headers within the protocol. Every program seems to have "header
ethernet_t" and "headeripv4_t" and other protocols that are used and protocol constants like Ethertypes also seem to be spelled out in each program. Sometimes these are in include files within the program. What I don't see is that P4 has a standard set of include files for defining protocol headers. For instance, in Linux C we would just do "#include <linux/if_ether.h>" and "#include <linux/ip.h>" to get the definitions of the Ethernet header and IPv4 header. In fact, if someone were to submit a patch to Netdev that included its own definition of Ethernet or an IP header structure they would almost certainly get pushback. It's a fundamental programming principle, not just in networking but pretty much everywhere, to not continuously redefine common and standard constructs-- just put common things in header files that can be shared by multiple programs (to do otherwise substantially increases the possibility of errors, bloats code, and reduces readability).Marshalling up common definitions into header files that are common in the P4 development environment seems simple enough (maybe it's already done?), but I would also point out that Linux has included files that describe protocol formats and header structures for almost every protocol under the sun that are well tested. It would be great if somehow we could somehow leverage that work. For instance, in the P4 samples I looked at srcAddr and dstAddr are defined for IP addresses, but in
linux/ip.htheir saddr and daddr are the respective field names. Why not just base the P4 definition on the Linux one? Then when someone is porting code from Linux to P4 they can use the same field names-- this makes things a lot easier on the programmer! I'll also mention that we wrote a little Python script to generate P4 header and constant definitions from Linux headers. It almost worked, the snag we hit was that P4 has some limits on nesting structures and unions so we couldn't translate some of the C structures to P4 (if you're interested I can provide the details on the problem we hit).The IPv4 header checksum code was a real head scratcher for me. Do we really need to state each field in the IP header just to compute the checksum? (and not just do this once, but twice :-( ). See code below for verifyChecksum and updateChecksum.
In C, verifying and setting the IP header checksum is really easy:
if (
checksum(iphdr, 0, iphdr->ihl << 4))goto bad_csum;
ip->csum=checksum(iphdr, 0, iphdr->ihl << 4);Relative to the C code, the P4 code seems very convoluted to me and prone to errors. What if someone accidentally omits a field? What if fields become slightly out of order? Also, no one would ever describe the IPv4 checksum as taking the checksum over the IHL, diffserv, totalLen, ... That is *way* too complicated for an algorithm that is really simple-- from RFC791: "The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header.". Reverse engineering the design, the clue seems to be HashAlgorithm.csum16. Maybe in P4 the IP checksum is just considered another form of hash, and I suspect the input to hash computation is specified as sort of data structure to make things generic (for instance, how we create a substructure in flow keys in
flow_dissectorto compute a SipHash over the TCP and UDP tuple). But, the IPv4 checksum isn't just another hash-- on a host, we need to compute the checksum for *every* IPv4 packet. This has to be fast and simple, we can do this in as few as five instructions or less. So even if the code below is correct, I have to wonder how easy it is to emit an efficient executable. Would a compiler easily realize that all the fields in the pseudo structure are contiguous without holes such that it can omit those five instructions?I don't know how prevalent this method of listing all the fields in a data structure as arguments to a function is in P4, but, by almost any objective measure, I have to say that the code below is bad and bloated. Maybe there's a better way to do it in P4, but if there's not then this is a deficiency in the P4 language.
Tom
control verifyChecksum(inout headers hdr,
inout metadata meta){ apply { // There is code similar to this in Github repo p4lang/p4c in // file testdata/p4_16_samples/flowlet_switching-bmv2.p4 // However in that file it is only for a fixed length IPv4 // header with no options. verify_checksum(true, { hdr.ipv4.version, hdr.ipv4.ihl, hdr.ipv4.diffserv, hdr.ipv4.totalLen, hdr.ipv4.identification, hdr.ipv4.flags, hdr.ipv4.fragOffset, hdr.ipv4.ttl, hdr.ipv4.protocol, hdr.ipv4.srcAddr, hdr.ipv4.dstAddr #ifdef ALLOW_IPV4_OPTIONS , hdr.ipv4.options #endif /* ALLOW_IPV4_OPTIONS */ }, hdr.ipv4.hdrChecksum, HashAlgorithm.csum16); } }control updateChecksum(inout headers hdr,
inout metadata meta){ apply { update_checksum(true, { hdr.ipv4.version, hdr.ipv4.ihl, hdr.ipv4.diffserv, hdr.ipv4.totalLen, hdr.ipv4.identification, hdr.ipv4.flags, hdr.ipv4.fragOffset, hdr.ipv4.ttl, hdr.ipv4.protocol, hdr.ipv4.srcAddr, hdr.ipv4.dstAddr #ifdef ALLOW_IPV4_OPTIONS , hdr.ipv4.options #endif /* ALLOW_IPV4_OPTIONS */ }, hdr.ipv4.hdrChecksum, HashAlgorithm.csum16); } }Quoted 148 lines · Tom Herbert
Quoted 76 lines · Chris Sommers
Quoted 58 lines · Tom Herbert
Quoted 56 lines
Quoted 53 lines · Chris Sommers
Quoted 38 lines · Jakub Kicinski
Quoted 15 lines
Hi Jamal!
Quoted 6 lines · On Tue, 21 May 2024 08:35:07 -0400 Jamal Hadi Salim wrote:
At that point(v16) i asked for the series to be applied despite the Nacks because, frankly, the Nacks have no merit. Paolo was not comfortable applying patches with Nacks and tried to mediate. In his mediation effort he asked if we could remove eBPF - and our answer was no because after all that time we have become dependent on it and frankly there was no technical reason not to use eBPF.
I'm not fully clear on who you're appealing to, and I may be missing some points. But maybe it will be more useful than hurtful if I clarify my point of view.
Quoted 6 lines
AFAIU BPF folks disagree with the use of their subsystem, and they point out that P4 pipelines can be implemented using BPF in the first place.
To which you reply that you like (a highly dated type of) a netlink interface, and (handwavey) ability to configure the data path SW or HW via the same interface.It's not what I "like" , rather it is a requirement to support both
s/w and h/w offload. The TC model is the traditional approach to
deploy these models. I addressed the same comment you are making above
in #1a and #1b (https://urldefense.com/v3/__https://github.com/p4tc-dev/pushback-patches__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs0-w2zKP2A$).OTOH, "BPF folks disagree with the use of their subsystem" is a problematic statement. Is BPF infra for the kernel community or is it something the ebpf folks can decide, at their whim, to allow who they like to use or not. We are not changing any BPF code. And there's already a case where the interfaces are used exactly as we used them in the conntrack code i pointed to in the page (we literally copied that code). Why is it ok for conntrack code to use exactly the same approach but not us?
Quoted 8 lines
Quoted 1 line
AFAICT there's some but not very strong support for P4TC,
I dont agree. Paolo asked this question and afaik Intel, AMD (both build P4-native NICs) and the folks interested in the MS DASH project responded saying they are in support. Look at who is being Cced. A lot of these folks who attend biweekly discussion calls on P4TC. Sample:
https://urldefense.com/v3/__https://lore.kernel.org/netdev/IA0PR17MB7070B51A955FB8595FFBA5FB965E2@IA0PR17MB7070.namprd17.prod.outlook.com/__;!!I5pVk4LIGAfnvw!kaZ6EmPxEqGLG8JMw-_L0BgYq48Pe25wj6pHMF6BVei5WsRgwMeLQupmvgvLyN-LgXacKBzzs09TFzoQBw$+1
Quoted 5 lines
Quoted 4 lines
and it
doesn't benefit or solve any problems of the broader networking stack (e.g. expressing or configuring parser graphs in general)Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Huh? As a DSL, P4 has already been proven to be an extremely effective and popular way to express parse graphs, stack manipulation, and stateful programming. Yesterday, I used the P4TC dev branch to implement something in one sitting, which includes parsing RoCEv2 network stacks. I just cut and pasted P4 code originally written for a P4 ASIC into a working P4TC example to add functionality. It took mere seconds to compile and launch it, and a few minutes to test it. I know of no other workflow which provides such quick turnaround and is so accessible. I'd like it to be as ubiquitous as eBPF itself.
Chris,
When you say "it took mere seconds to compile and launch" are you taking into account the ramp up time that it takes to learn P4 and become proficient to do something interesting?
Hi Tom, thanks for the dialog. To answer your question, it took seconds to compile and deploy, not learn P4. Adding the parsing for several headers took minutes. If you want to compare learning curve, learning to write P4 code and let the framework handle all the painful low-level Linux details is way easier than trying to learn how to write c code for Linux networking. It’s not even close. I’ve written C for 40 years, P4 for 7 years, and dabbled in eBPF so I can attest to the ease of learning and using P4. I’ve onboarded and mentored engineers who barely knew C, to develop complex networking products using P4, and built the automation APIs (REST, gRPC) to manage them. One person can develop an entire commercial product by themselves in months. P4 has expanded the reach of programmers such that both HW and SW engineers can easily learn P4 and become pretty adept at it. I would not expect even experienced c programmers to be able to master Linux internals very quickly. Writing a P4-TC program and injecting it via tc was like magic the first time.
Quoted 7 lines
Quoted 7 lines
Considering that P4
syntax is very different from typical languages than networking programmers are typically familiar with, this ramp up time is non-zero. OTOH, eBPF is ubiquitous because it's primarily programmed in Restricted C-- this makes it easy for many programmers since they don't have to learn a completely new language and so the ramp up time for the average networking programmer is much less for using eBPF.Quoted 1 line
I think your statement about “typical network programmers” overlooks the fact that since P4 was introduced, it has been taught in many universities to teach networking and possibly enabled a whole new breed of “network engineers” who can solve real problems without even knowing C programming. Without P4 they might never have gone this route. A class in network stack programming using c would have so many prerequisites to even get to parsing, compared to P4, where it could be demonstrated in one lesson. These “networking programmers” are not typical by your standards, but there are many such. They have just as much claim to the title "network programmer” as a C programmer. Similarly, an assembly language programmer is no less than a C or Python programmer. People writing P4 are usually focused on applications, and it is very useful and productive for that. Why should someone have to learn low-level C or eBPF to solve their problem?
Hio Chris,
You're comparing learning a completely new language versus programming in a subset of an established language, they're really not comparable. When one programs in Restricted-C they just need to understand what features of C are supported.
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
This is really the fundamental problem with DSLs, they require specialized skill sets in a programming language for a narrow use case (and specialized compilers, tool chains, debugging, etc)-- this means a DSL only makes sense if there is no other means to accomplish the same effects using a commodity language with perhaps a specialized library (it's not just in the networking realm, consider the advantages of using CUDA-C instead of a DLS for GPUs).
A pretty strong opinion, but DSLs arise to fill a need and P4 did so. It's still going strong.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Personally, I
don't believe that P4 has yet to be proven necessary for programming a datapath-- for instance we can program a parser in declarative representation in C,
https://urldefense.com/v3/__https://netdevconf.info/0x16/papers/11/High*20Performance*20Programmable*20Parsers.pdf__;JSUl!!I5pVk4LIGAfnvw!m9zrSDvddfzSt_sMBjOEvqw31RzAwWlEDM4ah5IJ2kqsmq6XtPIVJd-1_ZoGWBXKLyda77RYLvGR83Ginw$.CPL (slide11) looks like a DSL wrapped in JSON to me. “Solution: Common Parser Language (CPL); Parser representation in declarative .json” So I am confused. It is either a new language a.k.a. DSL, or it's not. Nothing against it, I'm sure it is great, but let's call it what it is.
Correct, it's not a new language. We've since renamed it Common Parser Representation.
Quoted 11 lines
We already have parser representations in declarative p4. And it's used and known worldwide. And has a respectable specification, any users and working groups. And it's formally provable (https://urldefense.com/v3/__https://github.com/verified-network-toolchain/petr4__;!!I5pVk4LIGAfnvw!juhSwk9UTheuI8-0mudbGTSZ_GBx3Z6hmcOAgiaAW14Ecter6K8iJ8DSzakN1d4GCE4uFJ05wkE9mvn6Vw$)
Quoted 5 lines
Quoted 5 lines
So unless P4 is proven necessary, then I'm doubtful it will ever be a ubiquitous way to program the kernel-- it seems much more likely that people will continue to use C and eBPF, and for those users that want to use P4 they can use
P4->eBPFcompiler.“ubiquitous way to program the kernel” – is not my goal. I don’t even want to know about the kernel when I am writing p4 - it's just a means to an end. I want to manipulate packets on a Linux host. P4DPDK, P4-eBPF, P4-TC – all let me do that. I LOVE the fact that P4-TC would be available in every Linux distro once upstreamed. It would solve so many deployment issues, benefit from regression testing, etc. So much goodness
" and for those users that want to use P4 they can use
P4->eBPFcompiler." -I'd really like to choose for myself and not have someone make that choice for me. P4-TC checks all the boxes for me.Sure, but this is a lot of kernel code and that will require support and maintenance. It needs to be justified, and the fact that someone wants it just to have a choice is, frankly, not much of a justification. I think a justification needs to start with "Why isn't
P4->eBPFsufficient?" (the question has been raised several times, but it still doesn't seem like there's a strong answer).Tom
Quoted 17 lines
Thanks for the point of view, it's healthy to debate. Cheers,
ChrisQuoted 11 lines
Quoted 1 line
Tom,
I cant stop the distraction of this thread becoming a discussion on the merits of DSL vs a lower level language (and I know you are not a P4 fan) but please change the subject so we dont loose the main focus which is a discussion on the patches. I have done it for you. Chris if you wish to respond please respond under the new thread subject.cheers,
jamal
Jakub Kicinski
maintainerSince the inevitable LWN article has been written, let me put more detail into what I already mentioned here:
https://lore.kernel.org/all/[email protected]/
for the benefit of non-networking people.
Quoted 3 lines · On Wed, 10 Apr 2024 10:01:26 -0400 Jamal Hadi Salim wrote:
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath.
The point of having SW datapath is to provide a blueprint for the behavior. This is completely moot for P4 which comes as a standard.
Besides we already have 5 (or more) flow offloads, we don't need a 6th, completely disconnected from the existing ones. Leaving users guessing which one to use, and how they interact.
In my opinion, reasonable way to implement programmable parser for Linux is:
1. User writes their parser in whatever DSL they want
2. User compiles the parser in user space
2.1 Compiler embeds a representation of the graph in the blob
3. User puts the blob in /lib/firmware
4. devlink dev $dev reload action parser-fetch $filename
5. devlink loads the file, parses it to extract the representation from 2.1, and passes the blob to the driver
5.1 driver/fw reinitializes the HW parser
5.2 user can inspect the graph by dumping the common representation
from 2.1 (via something like devlink dpipe, perhaps)
6. The parser tables are annotated with Linux offload targets (routes, classic ntuple, nftables, flower etc.) with some tables being left as "raw"* (* better name would be great)
7. ethtool ntuple is extended to support insertion of arbitrary rules into the "raw" tables
8. The other tables can only be inserted into using the subsystem they are annotated for
This builds on how some devices _already_ operate. Gives the benefits of expressing parser information and ability to insert rules for uncommon protocols also for devices which are not programmable. And it uses ethtool ntuple, which SW people actually want to use.
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
Jamal Hadi Salim
maintainerQuoted 13 lines · Jakub Kicinski
Since the inevitable LWN article has been written, let me put more detail into what I already mentioned here:
https://lore.kernel.org/all/[email protected]/
for the benefit of non-networking people.
Quoted 3 lines · On Wed, 10 Apr 2024 10:01:26 -0400 Jamal Hadi Salim wrote:
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath.
Quoted 8 lines
The point of having SW datapath is to provide a blueprint for the behavior. This is completely moot for P4 which comes as a standard.
Besides we already have 5 (or more) flow offloads, we don't need a 6th, completely disconnected from the existing ones. Leaving users guessing which one to use, and how they interact.
In my opinion, reasonable way to implement programmable parser for
You have mentioned "parser" before - are you referring to the DDP patches earlier from Intel?
In P4 the parser is just one of the objects.
Quoted 20 lines
Linux is:
1. User writes their parser in whatever DSL they want
2. User compiles the parser in user space
2.1 Compiler embeds a representation of the graph in the blob
3. User puts the blob in/lib/firmware
4. devlink dev $dev reload action parser-fetch $filename
5. devlink loads the file, parses it to extract the representation from 2.1, and passes the blob to the driver
5.1 driver/fw reinitializes the HW parser
5.2 user can inspect the graph by dumping the common representation
from 2.1 (via something like devlink dpipe, perhaps)
6. The parser tables are annotated with Linux offload targets (routes, classic ntuple, nftables, flower etc.) with some tables being left as "raw"* (* better name would be great)
7. ethtool ntuple is extended to support insertion of arbitrary rules into the "raw" tables
8. The other tables can only be inserted into using the subsystem they are annotated for
Quoted 8 lines
This builds on how some devices _already_ operate. Gives the benefits of expressing parser information and ability to insert rules for uncommon protocols also for devices which are not programmable. And it uses ethtool ntuple, which SW people actually want to use.
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
The scope is much bigger than just parsers though, it is about P4 in which the parser is but one object.
Limiting what we can do just to fit a narrow definition of "offload" is not the right direction.
P4 is well understood, hardware exists for P4 and is used to specify hardware specs and is deployed(See Vipin's comment).
cheers,
jamal
Jakub Kicinski
maintainerQuoted 6 lines · On Tue, 11 Jun 2024 11:10:35 -0400 Jamal Hadi Salim wrote:
Quoted 3 lines
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
The scope is much bigger than just parsers though, it is about P4 in which the parser is but one object.
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
Quoted 2 lines
Limiting what we can do just to fit a narrow definition of "offload" is not the right direction.
This is how Linux development works. You implement small, useful slice which helps the overall project. Then you implement the next, and another.
On the technical level, putting the code into devlink rather than TC does not impose any meaningful limitations. But I really don't want you to lift and shift the entire pile of code at once.
Quoted 2 lines
P4 is well understood, hardware exists for P4 and is used to specify hardware specs and is deployed(See Vipin's comment).
"Hardware exists for P4" is about as meaningful as "hardware exists for C++".
Jamal Hadi Salim
maintainerQuoted 6 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 11 Jun 2024 11:10:35 -0400 Jamal Hadi Salim wrote:
Quoted 3 lines
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
Quoted 6 lines
Quoted 2 lines
The scope is much bigger than just parsers though, it is about P4 in which the parser is but one object.
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
I think it is an important detail though.
You wouldnt say PSP shouldnt start small by first taking care of TLS or IPSec because it is not the target.
Quoted 11 lines
Quoted 2 lines
Limiting what we can do just to fit a narrow definition of "offload" is not the right direction.
This is how Linux development works. You implement small, useful slice which helps the overall project. Then you implement the next, and another.
On the technical level, putting the code into devlink rather than TC does not impose any meaningful limitations. But I really don't want you to lift and shift the entire pile of code at once.
Yes, the binary blob is going via devlink or some other scheme.
Quoted 5 lines
Quoted 2 lines
P4 is well understood, hardware exists for P4 and is used to specify hardware specs and is deployed(See Vipin's comment).
"Hardware exists for P4" is about as meaningful as "hardware exists for C++".
We'll have to agree to disagree. Take a look at this for example.
https://www.servethehome.com/pensando-distributed-services-architecture-smartnic/
cheers,
jamal
Quoted 11 lines · Jamal Hadi Salim
Quoted 9 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 11 Jun 2024 11:10:35 -0400 Jamal Hadi Salim wrote:
Quoted 3 lines
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
The scope is much bigger than just parsers though, it is about P4 in which the parser is but one object.
Quoted 10 lines
Quoted 3 lines
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
I think it is an important detail though.
You wouldnt say PSP shouldnt start small by first taking care of TLS or IPSec because it is not the target.Quoted 2 lines
Quoted 2 lines
Limiting what we can do just to fit a narrow definition of "offload" is not the right direction.
Jamal,
I think you might be missing Jakub's point. His plan wouldn't narrow the definition of "offload", but actually would increase applicability and use cases of offload. The best way to do an offload is allow flexibility on both sides of the equation: Let the user write their data path code in whatever language they want, and allow them offload to arbitrary software or programmable hardware targets.
For example, if a user already has P4 hardware for their high end server then by all means they should write their datapath in P4. But, there might also be a user that wants to offload TCP keepalive to a lower powered CPU on a Smartphone; in this case a simple C program maybe running in eBPF on the CPU should do the trick-- forcing them to write their program in P4 or even worse force them to put P4 hardware into their smartphone is not good. We should be able to define a common offload infrastructure to be both language and target agnostic that would handle both these use cases of offload and everything in between. P4 could certainly be one option for both programming language and offload target, but it shouldn't be the only option.
Tom
Quoted 23 lines
Quoted 9 lines
This is how Linux development works. You implement small, useful slice which helps the overall project. Then you implement the next, and another.
On the technical level, putting the code into devlink rather than TC does not impose any meaningful limitations. But I really don't want you to lift and shift the entire pile of code at once.
Yes, the binary blob is going via devlink or some other scheme.
Quoted 5 lines
Quoted 2 lines
P4 is well understood, hardware exists for P4 and is used to specify hardware specs and is deployed(See Vipin's comment).
"Hardware exists for P4" is about as meaningful as "hardware exists for C++".
We'll have to agree to disagree. Take a look at this for example.
https://www.servethehome.com/pensando-distributed-services-architecture-smartnic/cheers,
jamal
Quoted 23 lines · Tom Herbert
Quoted 11 lines · Jamal Hadi Salim
Quoted 9 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 11 Jun 2024 11:10:35 -0400 Jamal Hadi Salim wrote:
Quoted 3 lines
Before the tin foil hats gather - we have no use for any of this at Meta, I'm not trying to twist the design to fit the use cases of big bad hyperscalers.
The scope is much bigger than just parsers though, it is about P4 in which the parser is but one object.
Quoted 10 lines
Quoted 3 lines
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
I think it is an important detail though.
You wouldnt say PSP shouldnt start small by first taking care of TLS or IPSec because it is not the target.Quoted 2 lines
Quoted 2 lines
Limiting what we can do just to fit a narrow definition of "offload" is not the right direction.
Quoted 8 lines
Jamal,
I think you might be missing Jakub's point. His plan wouldn't narrow the definition of "offload", but actually would increase applicability and use cases of offload. The best way to do an offload is allow flexibility on both sides of the equation: Let the user write their data path code in whatever language they want, and allow them offload to arbitrary software or programmable hardware targets.
+1.
Quoted 12 lines
For example, if a user already has P4 hardware for their high end server then by all means they should write their datapath in P4. But, there might also be a user that wants to offload TCP keepalive to a lower powered CPU on a Smartphone; in this case a simple C program maybe running in eBPF on the CPU should do the trick-- forcing them to write their program in P4 or even worse force them to put P4 hardware into their smartphone is not good. We should be able to define a common offload infrastructure to be both language and target agnostic that would handle both these use cases of offload and everything in between. P4 could certainly be one option for both programming language and offload target, but it shouldn't be the only option.
Agree major benefit of proposal here is it doesn't dictate the language. My DSL preference is P4 but no need to push that here.
Quoted 2 lines
Tom
My $.02 Jakub's proposal is a very pragmatic way to get support for P4 enabled hardware I'm all for it. I can't actually think up anything in the P4 hardware side that couldn't go through the table notion in (7). We might want bulk updates and the likes at some point, but starting with basics should be good enough.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This is how Linux development works. You implement small, useful slice which helps the overall project. Then you implement the next, and another.
+1.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
On the technical level, putting the code into devlink rather than TC does not impose any meaningful limitations. But I really don't want you to lift and shift the entire pile of code at once.
devlink or an improved n_tuple (n_table?) mechanism would be great. Happy to help here.
Quoted 15 lines
Quoted 14 lines
Yes, the binary blob is going via devlink or some other scheme.
Quoted 5 lines
Quoted 2 lines
P4 is well understood, hardware exists for P4 and is used to specify hardware specs and is deployed(See Vipin's comment).
"Hardware exists for P4" is about as meaningful as "hardware exists for C++".
We'll have to agree to disagree. Take a look at this for example.
https://www.servethehome.com/pensando-distributed-services-architecture-smartnic/cheers,
jamal
Jakub Kicinski
maintainerQuoted 6 lines · On Tue, 11 Jun 2024 11:53:28 -0400 Jamal Hadi Salim wrote:
Quoted 2 lines
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
I think it is an important detail though.
You wouldnt say PSP shouldnt start small by first taking care of TLS or IPSec because it is not the target.
I really don't see any parallel with PSP. And it _is_ small, 4kLoC.
First you complain that community is "political" and doesn't give you technical feedback, and then when you get technical feedback you attack the work of the maintainer helping you.
Do you not see how these kind of retaliatory responses are exactly the reason why people were afraid to give you clear feedback earlier? Maybe one of the upcoming conferences should give out mirrors instead of t-shirts as swag.
Jamal Hadi Salim
maintainerQuoted 9 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 11 Jun 2024 11:53:28 -0400 Jamal Hadi Salim wrote:
Quoted 2 lines
For me it's very much not "about P4". I don't care what DSL user prefers and whether the device the offloads targets is built by a P4 vendor.
I think it is an important detail though.
You wouldnt say PSP shouldnt start small by first taking care of TLS or IPSec because it is not the target.
Quoted 6 lines
I really don't see any parallel with PSP. And it _is_ small, 4kLoC.
First you complain that community is "political" and doesn't give you technical feedback, and then when you get technical feedback you attack the work of the maintainer helping you.
You made a proposal saying it was a "start small" approach. I responded saying that it doesnt really cover our requirements and pointed to a sample h/w to show why. I only used PSP to illustrate why "start small" doesnt work for what we are targeting. I was not in any way attacking your work.
We are not trying to cover the whole world of offloads. It is a very specific niche -P4- which uses the existing tc model because that's how match-action tables are offloaded today. The actions and tables are dynamically defined by the users P4 program whereas in flower they are hardcoded in the kernel. I dont see any other way to achieve these goals with flower or other existing approaches. Flower for example could be written as a single P4 program and the goal here is to support a wider range of programs without making kernel changes.
cheers,
jamal