Introducing P4TC (series 1)
Subsystems (6)
Jamal Hadi Salim
maintainerv12 cover Introducing P4TC (series 1)
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Description of these Patches__
Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.
The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.
Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------
The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
What is P4TC?
-------------
P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware acquisition(example [12]).
Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC, with these patches, provides _kernel and user space code change independence_. Meaning:
A P4 program describes headers, parsers, etc alongside the datapath processing; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".
There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
__P4TC Architecture__
The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch description later..
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TC Workflow__
The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above..
b) the parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"
See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TC Runtime Control Path__
The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+) Read(</path/to/object>, [optional filter]) Update(</path/to/object>, DATA>+) Delete(</path/to/object>, [optional filter]) Subscribe(</path/to/object>, [optional filter])
Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted. "Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1
tc p4ctrl create myprog/table/mytable \
dstAddr 10.0.1.2/32 action send_to_port param port eno1
..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 action send_to_port param port eno1 \
entry dstAddr 10.1.10.2/32 action send_to_port param port eno10 \
entry dstAddr 10.0.2.2/32 action send_to_port param port eno2
..Get an entry (note "read" is interchangeably used as "get" which is a common
semantic in tc):
tc p4ctrl read myprog/table/mytable \
dstAddr 10.0.2.2/32
..dump mytable
tc p4ctrl read myprog/table/mytable
..dump mytable for all entries whose key fits within 10.1.0.0/16
tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16
..dump all mytable entries which have an action send_to_port with param "eno1"
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"
The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1" && \
key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filter msecs_since < 10000
Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types). All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TC Datapath__
The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address in labelled "dstAddr" above).
The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__
Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath).
Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.
In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__References__
[1]https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf [2]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#historical-perspective-for-p4tc [3]https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation [4]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#so-why-p4-and-how-does-p4-help-here [5]https://lore.kernel.org/netdev/[email protected]/T/#mf59be7abc5df3473cff3879c8cc3e2369c0640a6 [6]https://lore.kernel.org/netdev/[email protected]/T/#m783cfd79e9d755cf0e7afc1a7d5404635a5b1919 [7]https://lore.kernel.org/netdev/[email protected]/T/#ma8c84df0f7043d17b98f3d67aab0f4904c600469 [8]https://github.com/p4lang/p4c/tree/main/backends/tc
[9]https://p4.org/ [10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
[12]https://github.com/sonic-net/DASH/tree/main
[13]https://github.com/p4lang/p4c/tree/main/backends/ebpf [14]https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html [15]https://dl.acm.org/doi/10.1145/3630047.3630193 [16]https://lore.kernel.org/netdev/[email protected]/ [17.a]https://netdevconf.info/0x13/session.html?talk-tc-u-classifier
[17.b]man tc-u32
[18]man tc-pedit
[19] https://lore.kernel.org/netdev/[email protected]/T/#m86e71743d1d83b728bb29d5b877797cb4942e835
[20.a] https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html
[20.b] https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html
--------
HISTORY
--------
Changes in Version 12
----------------------
0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.
2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.
2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).
2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------
1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to p4tc_xxx for uapi and p4a_xxx for actions to provide consistency(Jiri).
4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------
1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------
0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------
1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------
1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------
1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------
These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------
Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).
Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".
This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.
The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via Kconfig).
Jamal Hadi Salim (15):
net: sched: act_api: Introduce P4 actions list
net/sched: act_api: increase action kind string length net/sched: act_api: Update tc_action_ops to account for P4 actions net/sched: act_api: add struct p4tc_action_ops as a parameter to
lookup callback
net: sched: act_api: Add support for preallocated P4 action instances
p4tc: add P4 data types
p4tc: add template API
p4tc: add template pipeline create, get, update, delete
p4tc: add template action create, update, delete, get, flush and dump
p4tc: add runtime action support
p4tc: add template table create, update, delete, get, flush and dump
p4tc: add runtime table entry create and update
p4tc: add runtime table entry get, delete, flush and dump
p4tc: add set of P4TC table kfuncs
p4tc: add P4 classifier
include/linux/bitops.h | 1 + include/net/act_api.h | 23 +- include/net/p4tc.h | 675 +++++++ include/net/p4tc_types.h | 91 + include/net/tc_act/p4tc.h | 78 + include/uapi/linux/p4tc.h | 442 +++++ include/uapi/linux/pkt_cls.h | 15 + include/uapi/linux/rtnetlink.h | 18 + include/uapi/linux/tc_act/tc_p4.h | 11 + net/sched/Kconfig | 23 + net/sched/Makefile | 3 + net/sched/act_api.c | 192 +- net/sched/cls_api.c | 2 +- net/sched/cls_p4.c | 305 +++ net/sched/p4tc/Makefile | 8 + net/sched/p4tc/p4tc_action.c | 2397 +++++++++++++++++++++++ net/sched/p4tc/p4tc_bpf.c | 342 ++++ net/sched/p4tc/p4tc_filter.c | 870 ++++++++ net/sched/p4tc/p4tc_pipeline.c | 700 +++++++ net/sched/p4tc/p4tc_runtime_api.c | 145 ++ net/sched/p4tc/p4tc_table.c | 1834 +++++++++++++++++ net/sched/p4tc/p4tc_tbl_entry.c | 3047 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 440 +++++ net/sched/p4tc/p4tc_types.c | 1407 +++++++++++++ net/sched/p4tc/trace.c | 10 + net/sched/p4tc/trace.h | 44 + security/selinux/nlmsgtab.c | 10 +- 27 files changed, 13097 insertions(+), 36 deletions(-) create mode 100644 include/net/p4tc.h create mode 100644 include/net/p4tc_types.h create mode 100644 include/net/tc_act/p4tc.h create mode 100644 include/uapi/linux/p4tc.h create mode 100644 include/uapi/linux/tc_act/tc_p4.h create mode 100644 net/sched/cls_p4.c create mode 100644 net/sched/p4tc/Makefile create mode 100644 net/sched/p4tc/p4tc_action.c create mode 100644 net/sched/p4tc/p4tc_bpf.c create mode 100644 net/sched/p4tc/p4tc_filter.c create mode 100644 net/sched/p4tc/p4tc_pipeline.c create mode 100644 net/sched/p4tc/p4tc_runtime_api.c create mode 100644 net/sched/p4tc/p4tc_table.c create mode 100644 net/sched/p4tc/p4tc_tbl_entry.c create mode 100644 net/sched/p4tc/p4tc_tmpl_api.c create mode 100644 net/sched/p4tc/p4tc_types.c create mode 100644 net/sched/p4tc/trace.c create mode 100644 net/sched/p4tc/trace.h
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 1/15 net: sched: act_api: Introduce P4 actions list
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)
include/net/act_api.h +6 -2
@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;
@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);
@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,
net/sched/act_api.c +109 -14
215 lines not on this page · raw message
net/sched/cls_api.c +1 -1
@@ -3330,7 +3330,7 @@ int tcf_exts_validate_ex(struct net *net, struct tcf_proto *tp, struct nlattr **struct tc_action_ops *a_o; flags |=TCA_ACT_FLAGS_POLICE |TCA_ACT_FLAGS_BIND ; - a_o =tc_action_load_ops (tb[exts->police], flags, + a_o =tc_action_load_ops (net, tb[exts->police], flags, extack);if (IS_ERR (a_o))return PTR_ERR (a_o);
Signature · 1 line
2.34.1
Paolo Abeni
maintainerRe: [PATCH net-next v12 01/15] net: sched: act_api: Introduce P4 actions list
Quoted 61 lines · Jamal Hadi Salim
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)include/net/act_api.h +6
-2@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,net/sched/act_api.c +0
-0
net/sched/act_api.c static void tcf_free_cookie_rcu(struct rcu_head *p) · line 64
Quoted 8 lines
net/sched/act_api.c +5
-0@@ -57,6 +57,40 @@ static void tcf_free_cookie_rcu(struct rcu_head *p)kfree (cookie); } +static unsigned int p4_act_net_id; + +struct tcf_p4_act_net { +struct list_head act_base; +rwlock_t act_mod_lock;
Note that rwlock in networking code is discouraged, as they have to be unfair, see commit 0daf07e527095e64ee8927ce297ab626643e9f51.
In this specific case I think there should be no problems, as is extremely hard/impossible to have serious contention on the write side,. Also there is already an existing rwlock nearby, no not a blocker but IMHO worthy to be noted.
Cheers,
Paolo
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 01/15] net: sched: act_api: Introduce P4 actions list
Quoted 75 lines · Paolo Abeni
Quoted 61 lines · Jamal Hadi Salim
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)include/net/act_api.h +6
-2@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,net/sched/act_api.c +0
-0net/sched/act_api.c static void tcf_free_cookie_rcu(struct rcu_head *p) · line 64
Quoted 8 lines
net/sched/act_api.c +5
-0@@ -57,6 +57,40 @@ static void tcf_free_cookie_rcu(struct rcu_head *p)kfree (cookie); } +static unsigned int p4_act_net_id; + +struct tcf_p4_act_net { +struct list_head act_base; +rwlock_t act_mod_lock;Note that rwlock in networking code is discouraged, as they have to be unfair, see commit
0daf07e527095e64ee8927ce297ab626643e9f51.
Quoted 5 lines
In this specific case I think there should be no problems, as is extremely hard/impossible to have serious contention on the write side,. Also there is already an existing rwlock nearby, no not a blocker but IMHO worthy to be noted.
Sure - we can replace it. What's the preference? Spinlock?
cheers,
jamal
Quoted 4 lines
Cheers,
Paolo
Paolo Abeni
maintainerRe: [PATCH net-next v12 01/15] net: sched: act_api: Introduce P4 actions list
Quoted 76 lines · Jamal Hadi Salim
Quoted 75 lines · Paolo Abeni
Quoted 61 lines · Jamal Hadi Salim
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)include/net/act_api.h +6
-2@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,net/sched/act_api.c +0
-0net/sched/act_api.c static void tcf_free_cookie_rcu(struct rcu_head *p) · line 64
Quoted 8 lines
net/sched/act_api.c +5
-0@@ -57,6 +57,40 @@ static void tcf_free_cookie_rcu(struct rcu_head *p)kfree (cookie); } +static unsigned int p4_act_net_id; + +struct tcf_p4_act_net { +struct list_head act_base; +rwlock_t act_mod_lock;Note that rwlock in networking code is discouraged, as they have to be unfair, see commit
0daf07e527095e64ee8927ce297ab626643e9f51.
Quoted 7 lines
Quoted 5 lines
In this specific case I think there should be no problems, as is extremely hard/impossible to have serious contention on the write side,. Also there is already an existing rwlock nearby, no not a blocker but IMHO worthy to be noted.
Sure - we can replace it. What's the preference? Spinlock?
Plain spinlock will work. Using spinlock + RCU should be quite straight forward and will provide faster lookup.
Cheers,
Paolo
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 01/15] net: sched: act_api: Introduce P4 actions list
Quoted 84 lines · Paolo Abeni
Quoted 82 lines · Jamal Hadi Salim
Quoted 80 lines · Paolo Abeni
Quoted 61 lines · Jamal Hadi Salim
In P4 we require to generate new actions "on the fly" based on the specified P4 action definition. P4 action kinds, like the pipeline they are attached to, must be per net namespace, as opposed to native action kinds which are global. For that reason, we chose to create a separate structure to store P4 actions.
Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 8 ++- net/sched/act_api.c | 123 +++++++++++++++++++++++++++++++++++++----- net/sched/cls_api.c | 2 +- 3 files changed, 116 insertions(+), 17 deletions(-)include/net/act_api.h +6
-2@@ -105,6 +105,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head; +struct list_head p4_head;char kind[IFNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;@@ -199,10 +200,12 @@ int tcf_idr_check_alloc(struct tc_action_net *tn, u32 *index,int tcf_idr_release (struct tc_action *a,bool bind);int tcf_register_action (struct tc_action_ops *a,struct pernet_operations *ops); +int tcf_register_p4_action (struct net *net,struct tc_action_ops *act);int tcf_unregister_action (struct tc_action_ops *a,struct pernet_operations *ops);#define NET_ACT_ALIAS_PREFIX "net-act-"#define MODULE_ALIAS_NET_ACT (kind) MODULE_ALIAS(NET_ACT_ALIAS_PREFIX kind) +void tcf_unregister_p4_action (struct net *net,struct tc_action_ops *act);int tcf_action_destroy (struct tc_action *actions[],int bind);int tcf_action_exec (struct sk_buff *skb,struct tc_action **actions,int nr_actions,struct tcf_result *res);@@ -210,8 +213,9 @@ int tcf_action_init(struct net *net, struct tcf_proto *tp, struct nlattr *nla,struct nlattr *est, struct tc_action *actions[], int init_res[], size_t *attr_size, u32 flags, u32 fl_flags, struct netlink_ext_ack *extack); -struct tc_action_ops *tc_action_load_ops (struct nlattr *nla,u32 flags, -struct netlink_ext_ack *extack); +struct tc_action_ops * +tc_action_load_ops (struct net *net,struct nlattr *nla, +u32 flags,struct netlink_ext_ack *extack); struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, struct nlattr *nla, struct nlattr *est, struct tc_action_ops *a_o, int *init_res,net/sched/act_api.c +0
-0net/sched/act_api.c static void tcf_free_cookie_rcu(struct rcu_head *p) · line 64
Quoted 8 lines
net/sched/act_api.c +5
-0@@ -57,6 +57,40 @@ static void tcf_free_cookie_rcu(struct rcu_head *p)kfree (cookie); } +static unsigned int p4_act_net_id; + +struct tcf_p4_act_net { +struct list_head act_base; +rwlock_t act_mod_lock;Note that rwlock in networking code is discouraged, as they have to be unfair, see commit
0daf07e527095e64ee8927ce297ab626643e9f51.In this specific case I think there should be no problems, as is extremely hard/impossible to have serious contention on the write side,. Also there is already an existing rwlock nearby, no not a blocker but IMHO worthy to be noted.
Quoted 5 lines
Quoted 1 line
Sure - we can replace it. What's the preference? Spinlock?
Plain spinlock will work. Using spinlock + RCU should be quite straight forward and will provide faster lookup.
rcu + spinlock sounds like a bit of overkill but we'll look into it.
cheers,
jamal
Quoted 4 lines
Cheers,
Paolo
Jamal Hadi Salim
maintainerPATCH v12 2/15 net/sched: act_api: increase action kind string length
In reply to Jamal Hadi SalimIncrease action kind string length from IFNAMSIZ to 64
The new P4 actions, created via templates, will have longer names of format: "pipeline_name/act_name". IFNAMSIZ is currently 16 and is most of the times undersized for the above format.
So, to conform to this new format, we increase the maximum name length and change its definition from IFNAMSIZ to ACTNAMSIZ to account for this extra string (pipeline name) and the '/' character.
--- include/net/act_api.h | 2 +- include/uapi/linux/pkt_cls.h | 1 + net/sched/act_api.c | 8 ++++---- 3 files changed, 6 insertions(+), 5 deletions(-)
include/net/act_api.h +1 -1
@@ -106,7 +106,7 @@ typedef void (*tc_action_priv_destructor)(void *priv);struct tc_action_ops {struct list_head head;struct list_head p4_head; -char kind[IFNAMSIZ ]; +char kind[ACTNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id;size_t size;
include/uapi/linux/pkt_cls.h +1 -0
@@ -6,6 +6,7 @@#include <linux/pkt_sched.h> #define TC_COOKIE_MAX_SIZE 16 +#define ACTNAMSIZ 64/* Action attributes */ enum {
net/sched/act_api.c +4 -4
@@ -476,7 +476,7 @@ static size_t tcf_action_shared_attrs_size(const struct tc_action *act)rcu_read_unlock ();return nla_total_size (0 )/* action number nested */ - +nla_total_size (IFNAMSIZ )/* TCA_ACT_KIND */ + +nla_total_size (ACTNAMSIZ )/* TCA_ACT_KIND */ + cookie_len/* TCA_ACT_COOKIE */ +nla_total_size (sizeof (struct nla_bitfield32 ))/* TCA_ACT_HW_STATS */ +nla_total_size (0 )/* TCA_ACT_STATS nested */
@@ -1424,7 +1424,7 @@ tc_action_load_ops(struct net *net, struct nlattr *nla,bool police = flags &TCA_ACT_FLAGS_POLICE ;struct nlattr *tb[TCA_ACT_MAX +1 ];struct tc_action_ops *a_o; -char act_name[IFNAMSIZ ]; +char act_name[ACTNAMSIZ ];struct nlattr *kind;int err;
@@ -1439,12 +1439,12 @@ tc_action_load_ops(struct net *net, struct nlattr *nla,NL_SET_ERR_MSG (extack,"TC action kind must be specified" );return ERR_PTR (err); } -if (nla_strscpy (act_name, kind,IFNAMSIZ ) <0 ) { +if (nla_strscpy (act_name, kind,ACTNAMSIZ ) <0 ) {NL_SET_ERR_MSG (extack,"TC action name too long" );return ERR_PTR (err); } }else { -if (strscpy (act_name,"police" ,IFNAMSIZ ) <0 ) { +if (strscpy (act_name,"police" ,ACTNAMSIZ ) <0 ) {NL_SET_ERR_MSG (extack,"TC action name too long" );return ERR_PTR (-EINVAL ); }
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 3/15 net/sched: act_api: Update tc_action_ops to account for P4 actions
In reply to Jamal Hadi SalimThe initialisation of P4TC action instances require access to a struct p4tc_act (which appears in later patches) to help us to retrieve information like the P4 action parameters etc. In order to retrieve struct p4tc_act we need the pipeline name or id and the action name or id. Also recall that P4TC action IDs are P4 and are net namespace specific and not global like standard tc actions.
The init callback from tc_action_ops parameters had no way of supplying us that information. To solve this issue, we decided to create a new tc_action_ops callback (init_ops), that provies us with the tc_action_ops struct which then provides us with the pipeline and action name. In addition we add a new refcount to struct tc_action_ops called dyn_ref, which accounts for how many action instances we have of a specific action.
--- include/net/act_api.h | 6 ++++++ net/sched/act_api.c | 14 +++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)
include/net/act_api.h +6 -0
@@ -109,6 +109,7 @@ struct tc_action_ops {char kind[ACTNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id; +refcount_t p4_ref;size_t size;struct module *owner; int (*act)(struct sk_buff *, const struct tc_action *,
@@ -120,6 +121,11 @@ struct tc_action_ops { struct nlattr *est, struct tc_action **act, struct tcf_proto *tp, u32 flags, struct netlink_ext_ack *extack); +/* This should be merged with the original init action */ +int (*init_ops)(struct net *net,struct nlattr *nla, +struct nlattr *est,struct tc_action **act, +struct tcf_proto *tp,struct tc_action_ops *ops, +u32 flags,struct netlink_ext_ack *extack); int (*walk)(struct net *, struct sk_buff *, struct netlink_callback *, int, const struct tc_action_ops *,
net/sched/act_api.c +11 -3
@@ -1044,7 +1044,7 @@ int tcf_register_action(struct tc_action_ops *act,struct tc_action_ops *a;int ret; -if (!act->act || !act->dump || !act->init) +if (!act->act || !act->dump || (!act->init && !act->init_ops))return -EINVAL ;/* We have to register pernet ops before making the action ops visible,
@@ -1517,8 +1517,16 @@ struct tc_action *tcf_action_init_1(struct net *net, struct tcf_proto *tp, } } - err = a_o->init (net, tb[TCA_ACT_OPTIONS ], est, &a, tp, - userflags.value | flags, extack); +/* When we arrive here we guarantee that a_o->init or +* a_o->init_ops exist. +*/ +if (a_o->init) + err = a_o->init (net, tb[TCA_ACT_OPTIONS ], est, &a, tp, + userflags.value | flags, extack); +else + err = a_o->init_ops (net, tb[TCA_ACT_OPTIONS ], est, &a, + tp, a_o, userflags.value | flags, + extack); }else { err = a_o->init (net, nla, est, &a, tp, userflags.value | flags, extack);
Signature · 1 line
2.34.1
Paolo Abeni
maintainerRe: [PATCH net-next v12 03/15] net/sched: act_api: Update tc_action_ops to account for P4 actions
Quoted 11 lines · Jamal Hadi Salim
The initialisation of P4TC action instances require access to a struct
p4tc_act(which appears in later patches) to help us to retrieve information like the P4 action parameters etc. In order to retrieve structp4tc_actwe need the pipeline name or id and the action name or id. Also recall that P4TC action IDs are P4 and are net namespace specific and not global like standard tc actions.
The init callback fromtc_action_opsparameters had no way of supplying us that information. To solve this issue, we decided to create a newtc_action_opscallback (init_ops), that provies us with thetc_action_opsstruct which then provides us with the pipeline and action name.
The new init ops looks a bit unfortunate. I *think* it would be better adding the new argument to the existing init op
Quoted 29 lines
In addition we add a new refcount to struct
tc_action_opscalleddyn_ref, which accounts for how many action instances we have of a specific action.Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 6 ++++++ net/sched/act_api.c | 14 +++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)include/net/act_api.h +1
-0@@ -109,6 +109,7 @@ struct tc_action_ops {char kind[ACTNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id; +refcount_t p4_ref;size_t size;struct module *owner; int (*act)(struct sk_buff *, const struct tc_action *,
include/net/act_api.h struct tc_action_ops { · line 127
Quoted 7 lines
include/net/act_api.h +4
-0@@ -120,6 +121,11 @@ struct tc_action_ops { struct nlattr *est, struct tc_action **act, struct tcf_proto *tp, u32 flags, struct netlink_ext_ack *extack); +/* This should be merged with the original init action */ + int (*init_ops)(struct net *net, struct nlattr *nla, + struct nlattr *est, struct tc_action **act, + struct tcf_proto *tp, struct tc_action_ops *ops,
shouldn't the 'ops' argument be 'const'?
Thanks,
Paolo
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 03/15] net/sched: act_api: Update tc_action_ops to account for P4 actions
Quoted 14 lines · Paolo Abeni
Quoted 11 lines · Jamal Hadi Salim
The initialisation of P4TC action instances require access to a struct
p4tc_act(which appears in later patches) to help us to retrieve information like the P4 action parameters etc. In order to retrieve structp4tc_actwe need the pipeline name or id and the action name or id. Also recall that P4TC action IDs are P4 and are net namespace specific and not global like standard tc actions.
The init callback fromtc_action_opsparameters had no way of supplying us that information. To solve this issue, we decided to create a newtc_action_opscallback (init_ops), that provies us with thetc_action_opsstruct which then provides us with the pipeline and action name.
Quoted 3 lines
The new init ops looks a bit unfortunate. I *think* it would be better adding the new argument to the existing init op
Our initial goal was to avoid creating a much larger patch by changing any other action's code and we observe that ->init() already has 8 params already ;-> And only dynamic actions need this extra extension. If you still feel the change is needed, sure we can make that change.
Quoted 30 lines
Quoted 30 lines
In addition we add a new refcount to struct
tc_action_opscalleddyn_ref, which accounts for how many action instances we have of a specific action.Co-developed-by Victor Nogueira <[email protected]>Signed-off-by Victor Nogueira <[email protected]>Co-developed-by Pedro Tammela <[email protected]>Signed-off-by Pedro Tammela <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>Reviewed-by Vlad Buslov <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>--- include/net/act_api.h | 6 ++++++ net/sched/act_api.c | 14 +++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)include/net/act_api.h +1
-0@@ -109,6 +109,7 @@ struct tc_action_ops {char kind[ACTNAMSIZ ];enum tca_id id;/* identifier should match kind */ unsigned int net_id; +refcount_t p4_ref;size_t size;struct module *owner; int (*act)(struct sk_buff *, const struct tc_action *,@@ -120,6 +121,11 @@ struct tc_action_ops { struct nlattr *est, struct tc_action **act,
Quoted 9 lines
include/net/act_api.h struct tc_action_ops { · line 127
Quoted 6 lines
include/net/act_api.h +4
-0struct tcf_proto *tp, u32 flags, struct netlink_ext_ack *extack); +/* This should be merged with the original init action */ + int (*init_ops)(struct net *net, struct nlattr *nla, + struct nlattr *est, struct tc_action **act, + struct tcf_proto *tp, struct tc_action_ops *ops,shouldn't the 'ops' argument be 'const'?
As it is right now this would be hard to do because we carry around a refcnt in that struct. We will think about it..
cheers,
jamal
Quoted 4 lines
Thanks,
Paolo
Jamal Hadi Salim
maintainerPATCH v12 4/15 net/sched: act_api: add struct p4tc_action_ops as a parameter to lookup callback
In reply to Jamal Hadi SalimFor P4 actions, we require information from struct tc_action_ops, specifically the action kind, to find and locate the P4 action information for the lookup operation.
--- include/net/act_api.h | 3 ++- net/sched/act_api.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
include/net/act_api.h +2 -1
@@ -116,7 +116,8 @@ struct tc_action_ops { struct tcf_result *); /* called under RCU BH lock*/int (*dump)(struct sk_buff *,struct tc_action *,int ,int );void (*cleanup)(struct tc_action *); -int (*lookup)(struct net *net,struct tc_action **a,u32 index); +int (*lookup)(struct net *net,const struct tc_action_ops *ops, +struct tc_action **a,u32 index); int (*init)(struct net *net, struct nlattr *nla, struct nlattr *est, struct tc_action **act, struct tcf_proto *tp,
net/sched/act_api.c +1 -1
@@ -726,7 +726,7 @@ static int __tcf_idr_search(struct net *net,struct tc_action_net *tn =net_generic (net, ops->net_id);if (unlikely (ops->lookup)) -return ops->lookup (net, a, index); +return ops->lookup (net, ops, a, index);return tcf_idr_search (tn, a, index); }
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 5/15 net: sched: act_api: Add support for preallocated P4 action instances
In reply to Jamal Hadi SalimIn P4, actions are assumed to pre exist and have an upper bound number of instances. Typically if you a table defined with 1M table entries you want to allocate enough action instances to cover the 1M entries. However, this is a big waste of memory if the action instances are not in use. So for our case, we allow the user to specify a minimal amount of actions in the template and then if more P4 action instances are needed then they will be added on demand as in the current approach with tc filter-action relationship.
Add the necessary code to preallocate actions instances for P4 actions.
We add 2 new actions flags:
- TCA_ACT_FLAGS_PREALLOC: Indicates the action instance is a P4 action and was preallocated for future use the templating phase of P4TC
- TCA_ACT_FLAGS_UNREFERENCED: Indicates the action instance was preallocated and is currently not being referenced by any other object. Which means it won't show up in an action instance dump.
Once an action instance is created we don't free it when the last table entry referring to it is deleted.
Instead we add it to the pool/cache of action instances for that specific action kind i.e it counts as if it is preallocated. Preallocated actions can't be deleted by the tc actions runtime commands and a dump or a get will only show preallocated actions instances which are being used (i.e TCA_ACT_FLAGS_UNREFERENCED == false).
The preallocated actions will be deleted once the pipeline is deleted (which will purge the P4 action kind and its instances).
For example, if we were to create a P4 action that preallocates 128 elements and dumped:
$ tc -j p4template get action/myprog/send_nh | jq .
We'd see the following:
[
{
"obj": "action template",
"pname": "myprog",
"pipeid": 1
},
{
"templates": [
{
"aname": "myprog/send_nh",
"actid": 1,
"params": [
{
"name": "port",
"type": "dev",
"id": 1
}
],
"prealloc": 128
}
] }
]
If we try to dump the P4 action instances, we won't see any:
$ tc -j actions ls action myprog/send_nh | jq .
[]
However, if we create a table entry which references this action kind:
$ tc p4ctrl create myprog/table/cb/FDB \ dstAddr d2:96:91:5d:02:86 action myprog/send_nh \ param port type dev dummy0
Dumping the action instance will now show this one instance which is associated with the table entry:
$ tc -j actions ls action myprog/send_nh | jq .
[
{
"total acts": 1
},
{
"actions": [
{
"order": 0,
"kind": "myprog/send_nh",
"index": 1,
"ref": 1,
"bind": 1,
"params": [
{
"name": "port",
"type": "dev",
"value": "dummy0",
"id": 1
}
],
"not_in_hw": true
}
] }
]
--- include/net/act_api.h | 3 +++ net/sched/act_api.c | 45 +++++++++++++++++++++++++++++++++++-------- 2 files changed, 40 insertions(+), 8 deletions(-)
include/net/act_api.h +3 -0
@@ -68,6 +68,8 @@ struct tc_action {#define TCA_ACT_FLAGS_REPLACE (1U << (TCA_ACT_FLAGS_USER_BITS + 2))#define TCA_ACT_FLAGS_NO_RTNL (1U << (TCA_ACT_FLAGS_USER_BITS + 3))#define TCA_ACT_FLAGS_AT_INGRESS (1U << (TCA_ACT_FLAGS_USER_BITS + 4)) +#define TCA_ACT_FLAGS_PREALLOC (1U << (TCA_ACT_FLAGS_USER_BITS + 5)) +#define TCA_ACT_FLAGS_UNREFERENCED (1U << (TCA_ACT_FLAGS_USER_BITS + 6))/* Update lastuse only if needed, to avoid dirtying a cache line. * We use a temp variable to avoid fetching jiffies twice.
@@ -201,6 +203,7 @@ int tcf_idr_create_from_flags(struct tc_action_net *tn, u32 index, const struct tc_action_ops *ops, int bind, u32 flags);void tcf_idr_insert_many (struct tc_action *actions[],int init_res[]); +void tcf_idr_insert_n (struct tc_action *actions[],const u32 n);void tcf_idr_cleanup (struct tc_action_net *tn,u32 index);int tcf_idr_check_alloc (struct tc_action_net *tn,u32 *index,struct tc_action **a,int bind);
net/sched/act_api.c +37 -8
82 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 6/15 p4tc: add P4 data types
In reply to Jamal Hadi SalimIntroduce abstraction that represents P4 data types.
This also introduces the Kconfig and Makefile which later patches use. Numeric types could be little, host or big endian definitions. The abstraction also supports defining:
a) bitstrings using P4 annotations that look like "bit<X>" where X is the number of bits defined in a type
b) bitslices such that one can define in P4 as bit<8>[0-3] and bit<16>[4-9]. A 4-bit slice from bits 0-3 and a 6-bit slice from bits 4-9 respectively.
c) speacialized types like dev (which stands for a netdev), key, etc
Each type has a bitsize, a name (for debugging purposes), an ID and methods/ops. The P4 types will be used by externs, dynamic actions, packet headers and other parts of P4TC.
Each type has four ops:
- validate_p4t: Which validates if a given value of a specific type meets valid boundary conditions.
- create_bitops: Which, given a bitsize, bitstart and bitend allocates and returns a mask and a shift value. For example, if we have type bit<8>[3-3] meaning bitstart = 3 and bitend = 3, we'll create a mask which would only give us the fourth bit of a bit8 value, that is, 0x08. Since we are interested in the fourth bit, the bit shift value will be 3. This is also useful if an "irregular" bitsize is used, for example, bit24. In that case bitstart = 0 and bitend = 23. Shift will be 0 and the mask will be 0xFFFFFF00 if the machine is big endian.
- host_read : Which reads the value of a given type and transforms it to host order (if needed)
- host_write : Which writes a provided host order value and transforms it to the type's native order (if needed)
--- include/net/p4tc_types.h | 91 +++ include/uapi/linux/p4tc.h | 33 + net/sched/Kconfig | 11 + net/sched/Makefile | 2 + net/sched/p4tc/Makefile | 3 + net/sched/p4tc/p4tc_types.c | 1407 +++++++++++++++++++++++++++++++++++ 6 files changed, 1547 insertions(+) create mode 100644 include/net/p4tc_types.h create mode 100644 include/uapi/linux/p4tc.h create mode 100644 net/sched/p4tc/Makefile create mode 100644 net/sched/p4tc/p4tc_types.c
include/net/p4tc_types.h new file +91 -0
@@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_P4TYPES_H +#define __NET_P4TYPES_H + +#include <linux/netlink.h> +#include <linux/pkt_cls.h> +#include <linux/types.h> + +#include <uapi/linux/p4tc.h> + +#define P4TC_T_MAX_BITSZ 128 + +struct p4tc_type_mask_shift { +void *mask; +u8 shift; +}; + +struct p4tc_type ; +struct p4tc_type_ops { +int (*validate_p4t)(struct p4tc_type *container,void *value, +u16 startbit,u16 endbit, +struct netlink_ext_ack *extack); +struct p4tc_type_mask_shift *(*create_bitops)(u16 bitsz,u16 bitstart, +u16 bitend, +struct netlink_ext_ack *extack); +void (*host_read)(struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +void (*host_write)(struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +void (*print)(struct net *net,struct p4tc_type *container, +const char *prefix,void *val); +}; + +#define P4TC_T_MAX_STR_SZ 32 +struct p4tc_type { +char name[P4TC_T_MAX_STR_SZ ]; +const struct p4tc_type_ops *ops; +size_t container_bitsz; +size_t bitsz; +int typeid; +}; + +struct p4tc_type *p4type_find_byid (int id); +bool p4tc_is_type_unsigned_he (int typeid); +bool p4tc_is_type_numeric (int typeid); + +void p4t_copy (struct p4tc_type_mask_shift *dst_mask_shift, +struct p4tc_type *dst_t,void *dstv, +struct p4tc_type_mask_shift *src_mask_shift, +struct p4tc_type *src_t,void *srcv); +int p4t_cmp (struct p4tc_type_mask_shift *dst_mask_shift, +struct p4tc_type *dst_t,void *dstv, +struct p4tc_type_mask_shift *src_mask_shift, +struct p4tc_type *src_t,void *srcv); +void p4t_release (struct p4tc_type_mask_shift *mask_shift); + +int p4tc_register_types (void ); +void p4tc_unregister_types (void ); + +#ifdef CONFIG_RETPOLINE +void __p4tc_type_host_read (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +void __p4tc_type_host_write (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift,void *sval, +void *dval); +#else +static inline void +__p4tc_type_host_read (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift, +void *sval,void *dval) +{ +return ops->host_read (container, mask_shift, sval, dval); +} + +static inline void +__p4tc_type_host_write (const struct p4tc_type_ops *ops, +struct p4tc_type *container, +struct p4tc_type_mask_shift *mask_shift, +void *sval,void *dval) +{ +return ops->host_write (container, mask_shift, sval, dval); +} +#endif + +#endif
include/uapi/linux/p4tc.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef __LINUX_P4TC_H +#define __LINUX_P4TC_H + +#define P4TC_MAX_KEYSZ 512 + +enum { +P4TC_T_UNSPEC , +P4TC_T_U8 , +P4TC_T_U16 , +P4TC_T_U32 , +P4TC_T_U64 , +P4TC_T_STRING , +P4TC_T_S8 , +P4TC_T_S16 , +P4TC_T_S32 , +P4TC_T_S64 , +P4TC_T_MACADDR , +P4TC_T_IPV4ADDR , +P4TC_T_BE16 , +P4TC_T_BE32 , +P4TC_T_BE64 , +P4TC_T_U128 , +P4TC_T_S128 , +P4TC_T_BOOL , +P4TC_T_DEV , +P4TC_T_KEY , + __P4TC_T_MAX, +}; + +#define P4TC_T_MAX (__P4TC_T_MAX - 1) + +#endif
net/sched/Kconfig +11 -0
@@ -675,6 +675,17 @@ config NET_EMATCH_IPT To compile this code as a module, choose M here: the module will be called em_ipt. +config NET_P4TC + bool "P4TC support" + select NET_CLS_ACT + help + Say Y here if you want to use P4 features on top of TC. + P4 is an open source, domain-specific programming language for + specifying data plane behavior. By enabling P4TC you will be able to + write a P4 program, use a P4 compiler that supports P4TC backend to + generate all needed artificats, which when loaded allow you to + introduce a new kernel datapath that can be controlled via TC. + config NET_CLS_ACT bool "Actions" select NET_CLS
net/sched/Makefile +2 -0
@@ -81,3 +81,5 @@ obj-$(CONFIG_NET_EMATCH_TEXT) += em_text.o obj-$(CONFIG_NET_EMATCH_CANID) += em_canid.o obj-$(CONFIG_NET_EMATCH_IPSET) += em_ipset.o obj-$(CONFIG_NET_EMATCH_IPT) += em_ipt.o + +obj-$(CONFIG_NET_P4TC) += p4tc/
net/sched/p4tc/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-y := p4tc_types.o
net/sched/p4tc/p4tc_types.c new file +1407 -0
1,407 lines not on this page · raw message
Signature · 1 line
2.34.1
Paolo Abeni
maintainerRe: [PATCH net-next v12 06/15] p4tc: add P4 data types
Quoted 32 lines · Jamal Hadi Salim
Introduce abstraction that represents P4 data types.
This also introduces theKconfigandMakefilewhich later patches use. Numeric types could be little, host or big endian definitions. The abstraction also supports defining:a) bitstrings using P4 annotations that look like "bit<X>" where X is the number of bits defined in a type
b) bitslices such that one can define in P4 as bit<8>[0-3] and bit<16>[4-9]. A 4-bit slice from bits 0-3 and a 6-bit slice from bits 4-9 respectively.
c) speacialized types like dev (which stands for a netdev), key, etc
Each type has a bitsize, a name (for debugging purposes), an ID and methods/ops. The P4 types will be used by externs, dynamic actions, packet headers and other parts of P4TC.
Each type has four ops:
-
validate_p4t: Which validates if a given value of a specific type meets valid boundary conditions.-
create_bitops: Which, given a bitsize, bitstart and bitend allocates and returns a mask and a shift value. For example, if we have type bit<8>[3-3] meaning bitstart = 3 and bitend = 3, we'll create a mask which would only give us the fourth bit of a bit8 value, that is,0x08. Since we are interested in the fourth bit, the bit shift value will be 3. This is also useful if an "irregular" bitsize is used, for example, bit24. In that case bitstart = 0 and bitend = 23. Shift will be 0 and the mask will be0xFFFFFF00if the machine is big endian.
Quoted 5 lines
-
host_read: Which reads the value of a given type and transforms it to host order (if needed)-
host_write: Which writes a provided host order value and transforms it to the type's native order (if needed)
The type has a 'print' op, but I can't easily find where such op is used and its role?!?
Thanks,
Paolo
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 06/15] p4tc: add P4 data types
Quoted 34 lines · Paolo Abeni
Quoted 32 lines · Jamal Hadi Salim
Introduce abstraction that represents P4 data types.
This also introduces theKconfigandMakefilewhich later patches use. Numeric types could be little, host or big endian definitions. The abstraction also supports defining:a) bitstrings using P4 annotations that look like "bit<X>" where X is the number of bits defined in a type
b) bitslices such that one can define in P4 as bit<8>[0-3] and bit<16>[4-9]. A 4-bit slice from bits 0-3 and a 6-bit slice from bits 4-9 respectively.
c) speacialized types like dev (which stands for a netdev), key, etc
Each type has a bitsize, a name (for debugging purposes), an ID and methods/ops. The P4 types will be used by externs, dynamic actions, packet headers and other parts of P4TC.
Each type has four ops:
-
validate_p4t: Which validates if a given value of a specific type meets valid boundary conditions.-
create_bitops: Which, given a bitsize, bitstart and bitend allocates and returns a mask and a shift value. For example, if we have type bit<8>[3-3] meaning bitstart = 3 and bitend = 3, we'll create a mask which would only give us the fourth bit of a bit8 value, that is,0x08. Since we are interested in the fourth bit, the bit shift value will be 3. This is also useful if an "irregular" bitsize is used, for example, bit24. In that case bitstart = 0 and bitend = 23. Shift will be 0 and the mask will be0xFFFFFF00if the machine is big endian.
Quoted 9 lines
Quoted 5 lines
-
host_read: Which reads the value of a given type and transforms it to host order (if needed)-
host_write: Which writes a provided host order value and transforms it to the type's native order (if needed)The type has a 'print' op, but I can't easily find where such op is used and its role?!?
Thanks for catching that. We'll remove it. It was part of an operational debugging patch that was not submitted.
cheers,
jamal
Quoted 4 lines
Thanks,
Paolo
Jamal Hadi Salim
maintainerPATCH v12 7/15 p4tc: add template API
In reply to Jamal Hadi SalimAdd p4tc template API that will serve as infrastructure for all future template objects (in this set pipeline, table, action, and more later)
This commit is not functional by itself. It needs the subsequent patch to be of any use. This commit's purpose is to ease review. If something were to break and you do git bisect to this patch it will not be helpful. In the next release we are planning to merge the two back again.
The template API infrastructure follows the CRUD (Create, Read/get, Update, and Delete) commands.
To issue a p4template create command the user will follow the below grammar:
tc p4template create objtype/[objpath] [objid] objparams
To show a more concrete example, to create a new pipeline (pipelines come in the next commit), the user would issue the following command:
tc p4template create pipeline/aP4proggie pipeid 1 numtables 1 ...
Note that the user may specify an optional ID to the obj ("pipeid 1" above), if none is specified, the kernel will assign one.
The command for update is analogous:
tc p4template update objtype/[objpath] [objid] objparams
Note that for the user may refer to the object by name (in the objpath) or directly by ID.
Delete is also analogous:
tc p4template delete objtype/[objpath] [objid]
As is get:
tc p4template get objtype/[objpath] [objid]
One can also dump or flush template objects. This will be better exposed in the object specific commits in this patchset
--- include/net/p4tc.h | 53 +++++ include/uapi/linux/p4tc.h | 42 ++++ include/uapi/linux/rtnetlink.h | 9 + net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_tmpl_api.c | 366 +++++++++++++++++++++++++++++++++ security/selinux/nlmsgtab.c | 6 +- 6 files changed, 476 insertions(+), 2 deletions(-) create mode 100644 include/net/p4tc.h create mode 100644 net/sched/p4tc/p4tc_tmpl_api.c
include/net/p4tc.h new file +53 -0
@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_P4TC_H +#define __NET_P4TC_H + +#include <uapi/linux/p4tc.h> +#include <linux/workqueue.h> +#include <net/sch_generic.h> +#include <net/net_namespace.h> +#include <linux/refcount.h> +#include <linux/rhashtable.h> +#include <linux/rhashtable-types.h> + +#define P4TC_PATH_MAX 3 + +struct p4tc_dump_ctx { +u32 ids[P4TC_PATH_MAX ]; +}; + +struct p4tc_template_common ; + +struct p4tc_template_ops { +struct p4tc_template_common *(*cu)(struct net *net,struct nlmsghdr *n, +struct nlattr *nla, +struct netlink_ext_ack *extack); +int (*put)(struct p4tc_template_common *tmpl, +struct netlink_ext_ack *extack); +int (*gd)(struct net *net,struct sk_buff *skb,struct nlmsghdr *n, +struct nlattr *nla,struct netlink_ext_ack *extack); +int (*fill_nlmsg)(struct net *net,struct sk_buff *skb, +struct p4tc_template_common *tmpl, +struct netlink_ext_ack *extack); +int (*dump)(struct sk_buff *skb,struct p4tc_dump_ctx *ctx, +struct nlattr *nla,u32 *ids, +struct netlink_ext_ack *extack); +int (*dump_1)(struct sk_buff *skb,struct p4tc_template_common *common); +u32 obj_id; +}; + +struct p4tc_template_common { +char name[P4TC_TMPL_NAMSZ ]; +struct p4tc_template_ops *ops; +}; + +static inline bool p4tc_tmpl_msg_is_update (struct nlmsghdr *n) +{ +return n->nlmsg_type ==RTM_UPDATEP4TEMPLATE ; +} + +int p4tc_tmpl_generic_dump (struct sk_buff *skb,struct p4tc_dump_ctx *ctx, +struct idr *idr,int idx, +struct netlink_ext_ack *extack); + +#endif
include/uapi/linux/p4tc.h +42 -0
54 lines not on this page · raw message
include/uapi/linux/rtnetlink.h +9 -0
@@ -194,6 +194,15 @@ enum { RTM_GETTUNNEL,#define RTM_GETTUNNEL RTM_GETTUNNEL + RTM_CREATEP4TEMPLATE = 124, +#define RTM_CREATEP4TEMPLATE RTM_CREATEP4TEMPLATE + RTM_DELP4TEMPLATE, +#define RTM_DELP4TEMPLATE RTM_DELP4TEMPLATE + RTM_GETP4TEMPLATE, +#define RTM_GETP4TEMPLATE RTM_GETP4TEMPLATE + RTM_UPDATEP4TEMPLATE, +#define RTM_UPDATEP4TEMPLATE RTM_UPDATEP4TEMPLATE + __RTM_MAX,#define RTM_MAX (((__RTM_MAX + 3) & ~3) - 1) };
net/sched/p4tc/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o +obj-y := p4tc_types.o p4tc_tmpl_api.o
net/sched/p4tc/p4tc_tmpl_api.c new file +366 -0
366 lines not on this page · raw message
security/selinux/nlmsgtab.c +5 -1
@@ -94,6 +94,10 @@ static const struct nlmsg_perm nlmsg_route_perms[] = { { RTM_NEWTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_DELTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_GETTUNNEL, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_CREATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_DELP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_GETP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_UPDATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, }; static const struct nlmsg_perm nlmsg_tcpdiag_perms[] = {
@@ -177,7 +181,7 @@ int selinux_nlmsg_lookup(u16 sclass, u16 nlmsg_type, u32 *perm)* structures at the top of this file with the new mappings * before updating the BUILD_BUG_ON() macro! */ -BUILD_BUG_ON (RTM_MAX != (RTM_NEWTUNNEL +3 )); +BUILD_BUG_ON (RTM_MAX != (RTM_CREATEP4TEMPLATE +3 )); err =nlmsg_perm (nlmsg_type, perm, nlmsg_route_perms,sizeof (nlmsg_route_perms));break ;
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 8/15 p4tc: add template pipeline create, get, update, delete
In reply to Jamal Hadi Salim__Introducing P4 TC Pipeline__
This commit introduces P4 TC pipelines, which emulate the semantics of a P4 program/pipeline using the TC infrastructure.
This patch relies on the previous one to be functional. They were split to ease review.
One can refer to P4 programs/pipelines using their names or their specific pipeline ids (pipeid)
P4 template CRUD (Create, Read/get, Update and Delete) commands apply on a pipeline.
As an example, to create a P4 program/pipeline named aP4proggie with a single table in its pipeline, one would use the following command from user space tc (as generated by the compiler):
tc p4template create pipeline/aP4proggie numtables 1 pipeid 1
Note that, in the above command, the numtables is set as 1; the default is 0 because it is feasible to have a P4 program with no tables at all.
Note: if no pipeid is specified, the kernel will issue one. To see what pipeline ID is issued, one would add -echo option and the response back from the kernel will contain the details:
tc -echo p4template create pipeline/aP4proggie numtables 1
To Read pipeline aP4proggie attributes, one would retrieve those details as follows:
tc p4template get pipeline/[aP4proggie] [pipeid 1]
Note that in the above command one may specify pipeline ID, name or both.
To Update aP4proggie pipeline from 1 to 10 tables, one would use the following command:
tc p4template update pipeline/[aP4proggie] [pipeid 1] numtables 10
Note that, in the above command, one could use the P4 program/pipeline name, id or both to specify which P4 program/pipeline to update.
To Delete a P4 program/pipeline named aP4proggie with a pipeid of 1, one would use the following command:
tc p4template del pipeline/[aP4proggie] [pipeid 1]
Note that, in the above command, one could use the P4 program/pipeline name, id or both to specify which P4 program/pipeline to delete
If one wished to dump all the created P4 programs/pipelines, one would use the following command:
tc p4template get pipeline/
__Pipeline Lifetime__
After _Create_ is issued, one can Read/get, Update and Delete pipeline objects; however the pipeline can only be put to use after it is "sealed". To seal a pipeline, one would issue the following command:
tc p4template update pipeline/aP4proggie state ready
After a pipeline is sealed it can be instantiated via the TC P4 classifier. For example:
tc filter add $DEV ingress protocol any prio 6 p4 pname aP4proggie \
action bpf obj $PARSER.o section p4tc/parse
action bpf obj $PROGNAME.o section p4tc/main
Instantiates aP4proggie in the ingress of $DEV. One could also attach it to a block of ports (example tc block 22) as such:
tc filter add block 22 ingress protocol all prio 6 p4 pname aP4proggie \
action bpf obj $PARSER.o section p4tc/parse
action bpf obj $PROGNAME.o section p4tc/main
We can, add a table entry after the pipeline is sealed (even before instantiating). Like, for example:
tc p4ctrl create aP4proggie/table/cb/aP4table \
dstAddr 10.10.10.0/24 srcAddr 192.168.0.0/16 prio 16 \
action dropOnce the pipeline is instantiated on a device or block it cannot be deleted. It becomes Read-only from the control plane/user space.
The pipeline can be deleted when there are no longer any users left by destroying all instances (i.e all instantiated filters are deleted).
--- include/net/p4tc.h | 78 +++- include/uapi/linux/p4tc.h | 24 ++ net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_pipeline.c | 636 +++++++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 104 ++++-- 5 files changed, 820 insertions(+), 24 deletions(-) create mode 100644 net/sched/p4tc/p4tc_pipeline.c
include/net/p4tc.h +75 -3
@@ -10,27 +10,43 @@#include <linux/rhashtable.h> #include <linux/rhashtable-types.h> +#define P4TC_DEFAULT_NUM_TABLES P4TC_MINTABLES_COUNT +#define P4TC_DEFAULT_MAX_RULES 1#define P4TC_PATH_MAX 3 +#define P4TC_KERNEL_PIPEID 0 + +#define P4TC_PID_IDX 0 +struct p4tc_dump_ctx {u32 ids[P4TC_PATH_MAX ]; };struct p4tc_template_common ; +struct p4tc_path_nlattrs { +char *pname; +u32 *ids; +bool pname_passed; +}; + +struct p4tc_pipeline ;struct p4tc_template_ops {struct p4tc_template_common *(*cu)(struct net *net,struct nlmsghdr *n,struct nlattr *nla, +struct p4tc_path_nlattrs *nl_pname,struct netlink_ext_ack *extack); -int (*put)(struct p4tc_template_common *tmpl, +int (*put)(struct p4tc_pipeline *pipeline, +struct p4tc_template_common *tmpl,struct netlink_ext_ack *extack);int (*gd)(struct net *net,struct sk_buff *skb,struct nlmsghdr *n, -struct nlattr *nla,struct netlink_ext_ack *extack); +struct nlattr *nla,struct p4tc_path_nlattrs *nl_pname, +struct netlink_ext_ack *extack);int (*fill_nlmsg)(struct net *net,struct sk_buff *skb,struct p4tc_template_common *tmpl,struct netlink_ext_ack *extack);int (*dump)(struct sk_buff *skb,struct p4tc_dump_ctx *ctx, -struct nlattr *nla,u32 *ids, +struct nlattr *nla,char **p_name,u32 *ids,struct netlink_ext_ack *extack);int (*dump_1)(struct sk_buff *skb,struct p4tc_template_common *common);u32 obj_id;
@@ -39,6 +55,25 @@ struct p4tc_template_ops {struct p4tc_template_common {char name[P4TC_TMPL_NAMSZ ];struct p4tc_template_ops *ops; +u32 p_id; +u32 __pad0; +}; + +struct p4tc_pipeline { +struct p4tc_template_common common; +struct rcu_head rcu; +struct net *net; +/* Accounts for how many entities are referencing this pipeline. +* As for now only P4 filters can refer to pipelines. +*/ +refcount_t p_ctrl_ref; +u16 num_tables; +u16 curr_tables; +u8 p_state; +}; + +struct p4tc_pipeline_net { +struct idr pipeline_idr; };static inline bool p4tc_tmpl_msg_is_update (struct nlmsghdr *n)
@@ -46,8 +81,45 @@ static inline bool p4tc_tmpl_msg_is_update(struct nlmsghdr *n)return n->nlmsg_type ==RTM_UPDATEP4TEMPLATE ; } +int p4tc_tmpl_register_ops (const struct p4tc_template_ops *tmpl_ops); +int p4tc_tmpl_generic_dump (struct sk_buff *skb,struct p4tc_dump_ctx *ctx,struct idr *idr,int idx,struct netlink_ext_ack *extack); +struct p4tc_pipeline *p4tc_pipeline_find_byany (struct net *net, +const char *p_name, +const u32 pipeid, +struct netlink_ext_ack *extack); +struct p4tc_pipeline *p4tc_pipeline_find_byid (struct net *net, +const u32 pipeid); +struct p4tc_pipeline * +p4tc_pipeline_find_get (struct net *net,const char *p_name, +const u32 pipeid,struct netlink_ext_ack *extack); + +static inline bool p4tc_pipeline_get (struct p4tc_pipeline *pipeline) +{ +return refcount_inc_not_zero (&pipeline->p_ctrl_ref); +} + +void p4tc_pipeline_put (struct p4tc_pipeline *pipeline); +struct p4tc_pipeline * +p4tc_pipeline_find_byany_unsealed (struct net *net,const char *p_name, +const u32 pipeid, +struct netlink_ext_ack *extack); + +static inline int p4tc_action_destroy (struct tc_action **acts) +{ +int ret =0 ; + +if (acts) { + ret =tcf_action_destroy (acts,TCA_ACT_UNBIND ); +kfree (acts); + } + +return ret; +} + +#define to_pipeline (t) ((struct p4tc_pipeline *)t) +#endif
include/uapi/linux/p4tc.h +24 -0
@@ -7,19 +7,25 @@/* pipeline header */ struct p4tcmsg { +__u32 pipeid;__u32 obj; }; +#define P4TC_MAXPIPELINE_COUNT 32 +#define P4TC_MAXTABLES_COUNT 32 +#define P4TC_MINTABLES_COUNT 0#define P4TC_MSGBATCH_SIZE 16#define P4TC_MAX_KEYSZ 512#define P4TC_TMPL_NAMSZ 32 +#define P4TC_PIPELINE_NAMSIZ P4TC_TMPL_NAMSZ/* Root attributes */ enum {P4TC_ROOT_UNSPEC ,P4TC_ROOT ,/* nested messages */ +P4TC_ROOT_PNAME ,/* string - mandatory for pipeline create */ __P4TC_ROOT_MAX, };
@@ -28,6 +34,7 @@ enum {/* P4 Object types */ enum {P4TC_OBJ_UNSPEC , +P4TC_OBJ_PIPELINE , __P4TC_OBJ_MAX, };
@@ -43,6 +50,23 @@ enum {#define P4TC_MAX (__P4TC_MAX - 1) +/* PIPELINE attributes */ +enum { +P4TC_PIPELINE_UNSPEC , +P4TC_PIPELINE_NUMTABLES ,/* u16 */ +P4TC_PIPELINE_STATE ,/* u8 */ +P4TC_PIPELINE_NAME ,/* string only used for pipeline dump */ + __P4TC_PIPELINE_MAX +}; + +#define P4TC_PIPELINE_MAX (__P4TC_PIPELINE_MAX - 1) + +/* PIPELINE states */ +enum { +P4TC_STATE_NOT_READY , +P4TC_STATE_READY , +}; +enum {P4TC_T_UNSPEC ,P4TC_T_U8 ,
net/sched/p4tc/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o p4tc_tmpl_api.o +obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o
net/sched/p4tc/p4tc_pipeline.c new file +636 -0
636 lines not on this page · raw message
net/sched/p4tc/p4tc_tmpl_api.c +84 -20
251 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 9/15 p4tc: add template action create, update, delete, get, flush and dump
In reply to Jamal Hadi SalimThis commit allows the control plane to create, update, delete, get, flush and dump action templates based on P4 action definitions.
Visualize the following action in a P4 program named aP4Proggie:
action send_nh(@tc_type("macaddr) bit<48> dstAddr, @tc_type("dev") bit<8> port)
{
hdr.ethernet.dstAddr = dstMac;
send_to_port(port);
}The above is an action called send_nh which receives as parameters a bit<48> dstAddr (a mac address) and a bit<8> port (something close to ifindex).
which is applied on a P4 table match as such:
table mytable {
key = {
hdr.ipv4.dstAddr @tc_type("ipv4"): lpm;
}
actions = {
send_nh;
drop;
NoAction;
}
size = 1024;
}The mechanics of actions follow the CRUD semantics which are illustrated in the text below.
___P4 ACTION KIND CREATION___
In this stage we create the p4 action kind by specifying the action name, its ID, its parameters and the parameter types.
So for the send_nh action, the creation would look something like this:
tc p4template create action/aP4proggie/send_nh \
param dstAddr type macaddr id 1 param port type dev id 2
All the template commands (tc p4template) are generated by the p4c compiler (but of course could be hand coded by humans).
Also note that an action name has to specify the program name since P4 actions are unique to a program. As an example, the above command creates an action template that is bounded to pipeline/program named "aP4proggie".
Note2: In P4, actions are assumed to pre-exist and have an upper bound number of instances. Typically, if you have a max of 1024 "mytable" table entries you want to allocate enough action instances to cover the 1024 entries. However, this is a big waste of memory when we have low table occupancy. We pick a middle ground by providing pre-allocation control via attribute "num_prealloc".
The compiler generated template does not specify it and by default we preallocate 16 entries. The user can override this value by editing the generated text, for example to change the number to 128 as such:
tc p4template create action/aP4proggie/send_nh num_prealloc 128 \
param dstAddr type macaddr id 1 param port type dev id 2
When all preallocated action instances are exhausted (used in table entries) then the behavior switches to the current tc action approach i.e for every table entry created a new action instance is dynamically allocated if no "index" attribute is specified. Once an instance is created it is added to the pool and never freed.
Note, Current tc action behavior is maintained:
a) If the user wishes to preallocate more action instances later at runtime to take advantage of a faster table entry creation (by avoiding dynamic allocation at table entry creation time), they will have to individually create actions via the control plane using the classical "tc actions" command.
For example:
tc actions add action aP4proggie/send_nh \
param dstAddr AA:BB:CC:DD:EE:DD param port eth1
The action is added to the pool of action aP4proggie/send_nh instances and any table entry creation will grab it. The parameters specified above will be replaced when the table entry is created.
b) Sharing of action instances works the same way i.e you could autobind to any action instance in a table entry creation by specifying the action "index".
___ACTION KIND ACTIVATION___
Once we provided all the necessary information for the new p4 action, we can go to the final stage: action activation. In this template stage, we activate the p4 action and make it available for instantiation. To activate the action template, we issue the following command:
tc p4template update action/aP4proggie/send_nh state active
___OTHER CONTROL COMMANDS___
The lifetime of the p4 action is tied to its pipeline
(see earlier patches). As with all pipeline components, write operations to action templates (i.e action kinds), such as create, update and delete, can only be executed if the pipeline is not sealed. Read/get can be issued even after the pipeline is sealed.
If, after we are done with our action template we want to delete it, we could issue the following command:
tc p4template del action/aP4proggie/send_nh
Note: If any instance was created for this action (as illustrated earlier) than this action cannot be deleted, unless you delete all instances first.
If we had created more action templates and wanted to flush all of the action templates from pipeline aP4proggie, one would use the following command:
tc p4template del action/aP4proggie/
After creating or updating a p4 actions, if one wishes to verify that the p4 action was created correctly, one would use the following sample command:
tc p4template get action/aP4proggie/send_nh
The above command will display the relevant data for the action, such as parameter names, types, etc.
If one wanted to check which action templates were associated to a specific pipeline, one could use the following command:
tc p4template get action/aP4proggie/
Note that this command will only display the name of these action templates. To verify their specific details, one should use the get command, which was previously described.
--- include/net/act_api.h | 1 + include/net/p4tc.h | 78 +- include/net/tc_act/p4tc.h | 27 + include/uapi/linux/p4tc.h | 50 ++ include/uapi/linux/tc_act/tc_p4.h | 11 + net/sched/p4tc/Makefile | 3 +- net/sched/p4tc/p4tc_action.c | 1109 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_pipeline.c | 16 +- net/sched/p4tc/p4tc_tmpl_api.c | 10 + 9 files changed, 1294 insertions(+), 11 deletions(-) create mode 100644 include/net/tc_act/p4tc.h create mode 100644 include/uapi/linux/tc_act/tc_p4.h create mode 100644 net/sched/p4tc/p4tc_action.c
include/net/act_api.h +1 -0
@@ -70,6 +70,7 @@ struct tc_action {#define TCA_ACT_FLAGS_AT_INGRESS (1U << (TCA_ACT_FLAGS_USER_BITS + 4))#define TCA_ACT_FLAGS_PREALLOC (1U << (TCA_ACT_FLAGS_USER_BITS + 5))#define TCA_ACT_FLAGS_UNREFERENCED (1U << (TCA_ACT_FLAGS_USER_BITS + 6)) +#define TCA_ACT_FLAGS_FROM_P4TC (1U << (TCA_ACT_FLAGS_USER_BITS + 7))/* Update lastuse only if needed, to avoid dirtying a cache line. * We use a temp variable to avoid fetching jiffies twice.
include/net/p4tc.h +70 -8
113 lines not on this page · raw message
include/net/tc_act/p4tc.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __NET_TC_ACT_P4_H +#define __NET_TC_ACT_P4_H + +#include <net/pkt_cls.h> +#include <net/act_api.h> + +struct tcf_p4act_params { +struct idr params_idr; +struct p4tc_act_param **params_array; +struct rcu_head rcu; +u32 num_params; +u32 tot_params_sz; +}; + +struct tcf_p4act { +struct tc_action common; +/* Params IDR reference passed during runtime */ +struct tcf_p4act_params __rcu *params; +u32 p_id; +u32 act_id; +struct list_head node; +}; + +#define to_p4act (a) ((struct tcf_p4act *)a) + +#endif /* __NET_TC_ACT_P4_H */
include/uapi/linux/p4tc.h +50 -0
83 lines not on this page · raw message
include/uapi/linux/tc_act/tc_p4.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef __LINUX_TC_P4_H +#define __LINUX_TC_P4_H + +#include <linux/pkt_cls.h> + +struct tc_act_p4 { +tc_gen ; +}; + +#endif
net/sched/p4tc/Makefile +2 -1
@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o +obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ + p4tc_action.o
net/sched/p4tc/p4tc_action.c new file +1109 -0
1,109 lines not on this page · raw message
net/sched/p4tc/p4tc_pipeline.c +14 -2
@@ -75,6 +75,8 @@ static const struct nla_policy tc_pipeline_policy[P4TC_PIPELINE_MAX + 1] = {static void p4tc_pipeline_destroy (struct p4tc_pipeline *pipeline) { +idr_destroy (&pipeline->p_act_idr); +kfree (pipeline); }
@@ -96,8 +98,12 @@ static void p4tc_pipeline_teardown(struct p4tc_pipeline *pipeline,struct net *net = pipeline->net;struct p4tc_pipeline_net *pipe_net =net_generic (net, pipeline_net_id);struct net *pipeline_net =maybe_get_net (net); +unsigned long iter_act_id; +struct p4tc_act *act; +unsigned long tmp; -idr_remove (&pipe_net->pipeline_idr, pipeline->common.p_id); +idr_for_each_entry_ul (&pipeline->p_act_idr, act, tmp, iter_act_id) + act->common.ops->put (pipeline, &act->common, extack);/* If we are on netns cleanup we can't touch the pipeline_idr. * On pre_exit we will destroy the idr but never call into teardown
@@ -154,6 +160,7 @@ static int pipeline_try_set_state_ready(struct p4tc_pipeline *pipeline, } pipeline->p_state =P4TC_STATE_READY ; +return true; }
@@ -253,6 +260,10 @@ p4tc_pipeline_create(struct net *net, struct nlmsghdr *n,else pipeline->num_tables =P4TC_DEFAULT_NUM_TABLES ; +idr_init (&pipeline->p_act_idr); + + pipeline->num_created_acts =0 ; + pipeline->p_state =P4TC_STATE_NOT_READY ; pipeline->net = net;
@@ -507,7 +518,8 @@ static int p4tc_pipeline_gd(struct net *net, struct sk_buff *skb,return PTR_ERR (pipeline); tmpl = (struct p4tc_template_common *)pipeline; -if (p4tc_pipeline_fill_nlmsg (net, skb, tmpl, extack) <0 ) + ret =p4tc_pipeline_fill_nlmsg (net, skb, tmpl, extack); +if (ret <0 )return -1 ;if (!ids[P4TC_PID_IDX ])
net/sched/p4tc/p4tc_tmpl_api.c +10 -0
@@ -158,6 +158,11 @@ static int tc_ctl_p4_tmpl_1(struct sk_buff *skb, struct nlmsghdr *n, ids[P4TC_PID_IDX ] = t->pipeid; +if (tb[P4TC_PATH ]) { +const u32 *arg_ids =nla_data (tb[P4TC_PATH ]); + +memcpy (&ids[P4TC_PID_IDX +1 ], arg_ids,nla_len (tb[P4TC_PATH ])); + } nl_path_attrs.ids = ids; nskb =alloc_skb (NLMSG_GOODSIZE ,GFP_KERNEL );
@@ -365,6 +370,11 @@ static int tc_ctl_p4_tmpl_dump_1(struct sk_buff *skb, struct nlattr *arg, root =nla_nest_start (skb,P4TC_ROOT ); ids[P4TC_PID_IDX ] = t->pipeid; +if (tb[P4TC_PATH ]) { +const u32 *arg_ids =nla_data (tb[P4TC_PATH ]); + +memcpy (&ids[P4TC_PID_IDX +1 ], arg_ids,nla_len (tb[P4TC_PATH ])); + } obj_op = (struct p4tc_template_ops *)p4tc_ops[t->obj]; ret = obj_op->dump (skb, ctx, tb[P4TC_PARAMS ], &p_name, ids, extack);
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 10/15 p4tc: add runtime action support
In reply to Jamal Hadi SalimThis commit deals with the runtime part of P4 actions i.e instantiation and binding of action kinds which are created via templates (see previous patch).
For illustration we repeat the P4 code snippet from the action template commit:
action send_nh(@tc_type("macaddr) bit<48> dstAddr, @tc_type("dev") bit<8> port)
{
hdr.ethernet.dstAddr = dstMac;
send_to_port(port);
}table mytable {
key = {
hdr.ipv4.dstAddr @tc_type("ipv4"): lpm;
}
actions = {
send_nh;
drop;
NoAction;
}
size = 1024;
}One could create a table entry alongside an action instance as follows:
tc p4ctrl create aP4proggie/table/mycontrol/mytable \
srcAddr 10.10.10.0/24 \
action send_nh param dstAddr AA:BB:CC:DD:EE:FF param port eth0
As previously stated, we refer to the action by it's "full name" (pipeline_name/action_name). In the above the pipeline_name is inherited from the table patch ("aP4proggie"), so it doesn't need to be specified. Above we are creating an instance of the send_nh action specifying as parameter values AA:BB:CC:DD:EE:FF for dstAddr and eth0 for port.
We could also create the action instance outside of the table entry which is then added to the pool of send_nh action kind's instances as follows:
tc actions add action aP4proggie/send_nh \
param dstAddr AA:BB:CC:DD:EE:FF param port eth0
Observe these are _exactly the same semantics_ as what tc today already provides with a caveat that we have a keyword "param" to precede the appropriate parameters.
Note: We can create as many instances for action templates as we wish, as long as we do not exceed the maximum allowed actions - in this specific case 1024 for table "mytable". Any creation of action instances above the max allowed will be rejected by the kernel.
Action sharing still works the same way (as in classical tc). For example if we know the action index of the previous instance is 100 then we can bind it to a table entry, for example
tc p4ctrl create aP4proggie/table/mycontrol/mytable \
srcAddr 11.11.0.0/16 action send_nh index 100
--- include/net/p4tc.h | 114 ++++ include/uapi/linux/p4tc.h | 13 + net/sched/p4tc/p4tc_action.c | 1239 +++++++++++++++++++++++++++++++++- 3 files changed, 1356 insertions(+), 10 deletions(-)
include/net/p4tc.h +114 -0
@@ -116,6 +116,56 @@ p4tc_pipeline_find_byany_unsealed(struct net *net, const char *p_name,const u32 pipeid, struct netlink_ext_ack *extack); +struct p4tc_act *p4a_runt_find (struct net *net, +const struct tc_action_ops *a_o, +struct netlink_ext_ack *extack); +void +p4a_runt_prealloc_put (struct p4tc_act *act,struct tcf_p4act *p4_act); + +static inline int p4tc_action_destroy (struct tc_action *acts[]) +{ +struct tc_action *acts_non_prealloc[TCA_ACT_MAX_PRIO ] = {NULL }; +struct tc_action *a; +int ret =0 ; +int j =0 ; +int i; + +tcf_act_for_each_action (i, a, acts) { +if (acts[i]->tcfa_flags &TCA_ACT_FLAGS_PREALLOC ) { +struct tcf_p4act *p4act; +struct p4tc_act *act; +struct net *net; + + p4act = (struct tcf_p4act *)acts[i]; + net =maybe_get_net (acts[i]->idrinfo->net); + +if (net) { +const struct tc_action_ops *ops; + + ops = acts[i]->ops; + act =p4a_runt_find (net, ops,NULL ); +p4a_runt_prealloc_put (act, p4act); +put_net (net); + }else { +/* If net is coming down, template +* action will be deleted, so no need to +* remove from prealloc list, just decr +* refcounts. +*/ + acts_non_prealloc[j] = acts[i]; + j++; + } + }else { + acts_non_prealloc[j] = acts[i]; + j++; + } + } + + ret =tcf_action_destroy (acts_non_prealloc,TCA_ACT_UNBIND ); + +return ret; +} +struct p4tc_act_param {struct list_head head;struct rcu_head rcu;
@@ -169,6 +219,62 @@ struct p4tc_act {char fullname[ACTNAMSIZ ]; }; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla, +struct tc_action *acts[],u32 pipeid, +u32 flags,struct netlink_ext_ack *extack) +{ +struct nlattr *tb[TCA_ACT_MAX_PRIO +1 ] = {}; +int init_res[TCA_ACT_MAX_PRIO ]; +struct tc_action *a; +size_t attrs_size; +size_t nacts =0 ; +int ret; +int i; + + ret =nla_parse_nested_deprecated (tb,TCA_ACT_MAX_PRIO , nla,NULL , + extack); +if (ret <0 ) +return ret; + +for (i =1 ; i <TCA_ACT_MAX_PRIO +1 ; i++) + nacts += !!tb[i]; + +if (nacts >1 ) { +NL_SET_ERR_MSG (extack,"Only one action is allowed" ); +return -E2BIG ; + } + +/* If action was already created, just bind to existing one */ + flags |=TCA_ACT_FLAGS_BIND ; + flags |=TCA_ACT_FLAGS_FROM_P4TC ; + ret =tcf_action_init (net,NULL , nla,NULL , acts, init_res, &attrs_size, + flags,0 , extack); + +/* Check if we are trying to bind to dynamic action from different +* pipeline. +*/ +tcf_act_for_each_action (i, a, acts) { +struct tcf_p4act *p; + +if (a->ops->id <=TCA_ID_MAX ) +continue ; + + p =to_p4act (a); +if (p->p_id != pipeid) { +NL_SET_ERR_MSG (extack, +"Unable to bind to dynact from different pipeline" ); + ret = -EPERM ; + goto destroy_acts; + } + } + +return ret; + +destroy_acts: +p4tc_action_destroy (acts); +return ret; +} +struct p4tc_act *p4a_tmpl_get (struct p4tc_pipeline *pipeline,const char *act_name,const u32 a_id,struct netlink_ext_ack *extack);
@@ -180,6 +286,14 @@ static inline bool p4tc_action_put_ref(struct p4tc_act *act)return refcount_dec_not_one (&act->a_ref); } +struct tcf_p4act * +p4a_runt_prealloc_get_next (struct p4tc_act *act); +void p4a_runt_init_flags (struct tcf_p4act *p4act); +void p4a_runt_parm_destroy (struct p4tc_act_param *parm); +struct p4tc_act_param * +p4a_runt_parm_init (struct net *net,struct p4tc_act *act, +struct nlattr *nla,struct netlink_ext_ack *extack); +#define to_pipeline (t) ((struct p4tc_pipeline *)t)#define to_hdrfield (t) ((struct p4tc_hdrfield *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t)
include/uapi/linux/p4tc.h +13 -0
@@ -115,6 +115,17 @@ enum { };#define P4TC_ACT_MAX (__P4TC_ACT_MAX - 1) + +/* Action params value attributes */ + +enum { +P4TC_ACT_PARAMS_VALUE_UNSPEC , +P4TC_ACT_PARAMS_VALUE_RAW ,/* binary */ + __P4TC_ACT_PARAMS_VALUE_MAX +}; + +#define P4TC_ACT_VALUE_PARAMS_MAX (__P4TC_ACT_PARAMS_VALUE_MAX - 1) +enum {P4TC_ACT_PARAMS_TYPE_UNSPEC ,P4TC_ACT_PARAMS_TYPE_BITEND ,/* u16 */
@@ -138,6 +149,8 @@ enum {P4TC_ACT_PARAMS_ID ,/* u32 */ P4TC_ACT_PARAMS_TYPE ,/* nested type - mandatory for params create */ P4TC_ACT_PARAMS_FLAGS ,/* u8 */ +P4TC_ACT_PARAMS_VALUE ,/* bytes - mandatory for runtime params create */ +P4TC_ACT_PARAMS_MASK ,/* bytes */ __P4TC_ACT_PARAMS_MAX };
net/sched/p4tc/p4tc_action.c +1229 -10
1,364 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 11/15 p4tc: add template table create, update, delete, get, flush and dump
In reply to Jamal Hadi SalimThis commit introduces code to creation and maintenance of P4 tables defined in a P4 program.
As with all other P4TC objects, tables' lifetimes conform to extended CRUD operations and are maintained via templates.
It's important to note that write operations, such as create, update and delete can only be made if the pipeline is not sealed.
Per the P4 specification, tables prefix their name with the control block (although this could be overridden by P4 annotations).
As an example, if one were to create a table named table1 in a pipeline named myprog1, on control block "mycontrol", one would use the following command:
tc p4template create table/myprog1/mycontrol/table1 tblid 1 \
keysz 32 nummasks 8 tentries 8192
Above says that we are creating a table (table1) attached to pipeline myprog1 on control block mycontrol. Table1's key size is 32 bits wide and it can have up to 8 associated masks and 8192 entries. The table id for table1 is 1. The table id is typically provided by the compiler.
Parameters such as nummasks (number of masks this table may have) and tentries (maximum number of entries this table may have); either may be omitted in which case it defaults to 8 masks and 256 entries respectfully.
P4 tables have many associated attributes that further refine how these tables operate. Some attributes are per table and others per entry. Attributes include:
- Timer profiles: The timer profiles for the table entries belonging to the table. These profiles contain the supported aging values for the table. There are currently 4 different built-in profiles ID0=30s(default), ID1=60s, ID2=120s and ID3=180s.
The user can override the defaults by specifying the number of profiles per table. In that case, the kernel will generate the specified number of profiles with their aging values abiding by the following rule:
- Profile ID 0 - 30000ms
- Profile ID(n) - Profile ID(n - 1) + 30000ms
So, for example, if the user specified num_timer_profiles as 5, the profile IDs and aging values would be the following:
- Profile ID 0 - 30000ms (default profile)
- Profile ID 1 - 60000ms
- Profile ID 2 - 90000ms
- Profile ID 3 - 120000ms
- Profile ID 4 - 150000ms
The values of the different profiles could be changed at runtime. The default profile is used if the user specifies an out of range value.
- Match key type: The match type of the table key (exact, LPM, ternary, etc).
Note in P4 a key may constitute multiple (sub)key types, eg matching on srcip using a LPM prefix and an exact match for dstip. In such a case it is up to the compiler to generalize the key used (For example in this case the overall key may endup being LPM or ternary).
- Direct Counter: Table counter instances used directly by the table, when specified in the P4 program, there will be one counter per entry.
- Direct Meter: Table meter instances used directly by the table, when specified in the P4 program, there will be one meter per entry.
- CRUDXPS Permissions both for specific entries and tables. The permissions are applicable to both the control plane and the datapath (see "Table Permissions" further below).
- Allowed Actions List. This will be a list of all possible actions that can be added to table entries that are added to the specified table.
- Action profiles. When defined in a P4 program, action profiles provide a mechanism to share action instances.
- Actions Selectors. When defined in a P4 program can be used to select the execution of one or more action instance selected at table lookup time by using a hash computation.
- Default hit action. When a default hit action is defined it is used when a matched table entry did not define an action. Depending on the P4 program the default hit action can be updated at runtime (in addition to being specified in the template).
- Default miss action. When a default miss action is defined it is used when a lookup that table fails.
- Action scope. In addition to actions being annotated as default hit or miss they can also be annotated to be either specific to a table of globally available to multiple tables within the same P4 program.
- Max entries. This is an upper bound for number of entries a specific table allows.
- Num masks. In the case of LPM or ternary matches, this defines the maximum allowed masks for that table.
- Timers. When defined in a P4 program, each entry has an associated timer. Depending on the programmed timer profile (see above), an entry gets a timeout. The timer attribute specifies the behavior of a table entry expiration.
The timer is refreshed every time there's a hit. After an idle period the P4 program can define using this attribute, whether it wants to have an event generated to user space (and have user space delete the entry), or whether it wants the kernel to delete it and send the event to announce the deletion. The default in P4TC is to both delete and generate an event.
- per entry "static" vs "dynamic" entry. By default all entries created from the control plane are "static" unless otherwise specified. All entries added from datapath are "dynamic" unless otherwise specified. "Dynamic" entries are subject to deletion when idle (subject to the rules specified in "Timers" above).
If one were to retrieve the template details of a table named table1 (before or after the pipeline is sealed) one would use the following command:
tc p4template get table/myprog1/mycontrol/table1
If one were to dump all the tables from a pipeline named myprog1, one would use the following command:
tc p4template get table/myprog1
If one were to update table1 (before the pipeline is sealed) one would use the following command:
tc p4template update table/myprog1/mycontrol/table1 ....
If one were to delete table1 (before the pipeline is sealed) one would use the following command:
tc p4template del table/myprog1/mycontrol/table1
If one were to flush all the tables from a pipeline named myprog1, control block "mycontrol" one would use the following command:
tc p4template del table/myprog1/mycontrol/
___Table Permissions___
Tables can have permissions which apply to all the entries in the specified table. Permissions are defined for both what the control plane (user space) as well as the data path are allowed to do on the table.
The permissions field is a 16bit value which will hold CRUDXPS (create, read, update, delete, execute, publish and subscribe) permissions for control and data path. Bits 13-7 will have the CRUDXPS values for control and bits 6-0 will have CRUDXPS values for data path. By default each table has the following permissions:
CRUD-PS-R--X--
Which means the control plane can perform CRUDPS operations whereas the data path (-R--X--) can only Read and execute on the entries. The user can override these permissions when creating the table or when updating.
For example, the following command will create a table which will not allow the datapath to create, update or delete entries but give full CRUDS permissions for the control plane.
$TC p4template create table/aP4proggie/cb/tname tblid 1 keysz 64 type lpm \
permissions 0x3D24 ...
Recall that these permissions come in the form of CRUDXPSCRUDXPS, where the first CRUDXPS block is for control and the last is for data path.
So 0x34A4 is equivalent to CR-D--S-R--X--
If we were to issue a read command on a table (tname):
$TC -j p4template get table/aP4proggie/cb/tname | jq .
The output would be the following:
[
{
"obj": "table",
"pname": "aP4Proggie",
"pipeid": 22
},
{
"templates": [
{
"tblid": 1,
"tname": "cb/tname",
"keysz": 64,
"max_entries": 256,
"masks": 8,
"entries": 0,
"permissions": "CRUD--S-R--X--",
"table_type": "lpm",
"acts_list": []
}
] }
]
Note, the permissions concept is more powerful than classical const definition currently taken by P4 which makes everything in a table read-only.
___Initial Table Entries___
Templating can create initial table entries. For example:
tc p4template create table/myprog/cb/tname \
entry srcAddr 10.10.10.10/32 dstAddr 1.1.1.0/24 prio 17
In this command we are creating table tname with an entry which will be present before even the P4 program starts executing. This entry has as its key IPv4 srcAddr and dstAddr and prio 17.
If one was to read back the entry by issuing the following runtime command:
tc p4ctrl get myprog/table/cb/tname
They would get:
...
entry priority 17[permissions-RUD--S-R--X--]
entry key
srcAddr id:1 size:32b type:ipv4 exact fieldval 10.10.10.10/32
dstAddr id:2 size:32b type:ipv4 exact fieldval 1.1.1.0/24We will explain the p4ctrl commands in more details on the next patches.
Before the pipeline is sealed we can only do a p4ctrl get. The write commands (create, update, delete and flush) and not allowed before sealing.
___Table Actions List___
P4 tables can be programmed to allow only a specified list of actions to be part of match entry on a table. P4 also defines default actions to be executed when no entries match; we have extended this concept to have a default hit, which is executed upon matching an entry which has no action associated with it.
We also allow flags for each of the actions in this list that specify if the action can be added only as a table entry (tableonly), or only as a default action (defaultonly). If no flags are specified, it is assumed that the action can be used in both contexts.
Both default hit and default missactions are optional.
An example of specifying a default miss action is as follows:
tc p4template update table/myprog/cb/mytable \
default_miss_action permissions 0x1124 action dropThe above will drop packets if the entry is not found in mytable. Note the above makes the default action a const. Meaning the control plane can neither replace it nor delete it.
tc p4template update table/myprog/mytable \
default_hit_action permissions 0x3004 action ok
Whereas the above allows a default hit action to accept the packet. The permission 0x3004 (binary 11000000000100) means we have only Create and Read permissions in the control plane and eXecute permissions in the data plane. This means, for example, that now we can only delete the default hit action from the control plane.
--- include/net/p4tc.h | 156 +++- include/net/p4tc_types.h | 2 +- include/uapi/linux/p4tc.h | 119 +++ net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_action.c | 4 +- net/sched/p4tc/p4tc_pipeline.c | 23 +- net/sched/p4tc/p4tc_table.c | 1606 ++++++++++++++++++++++++++++++++ 7 files changed, 1900 insertions(+), 12 deletions(-) create mode 100644 net/sched/p4tc/p4tc_table.c
include/net/p4tc.h +155 -1
@@ -16,10 +16,23 @@#define P4TC_DEFAULT_MAX_RULES 1#define P4TC_PATH_MAX 3#define P4TC_MAX_TENTRIES 0x2000000 +#define P4TC_DEFAULT_TENTRIES 256 +#define P4TC_MAX_TMASKS 1024 +#define P4TC_DEFAULT_TMASKS 8 +#define P4TC_MAX_T_AGING_MS 864000000 +#define P4TC_DEFAULT_T_AGING_MS 30000 +#define P4TC_DEFAULT_NUM_TIMER_PROFILES 4 +#define P4TC_MAX_NUM_TIMER_PROFILES P4TC_MSGBATCH_SIZE + +#define P4TC_TIMER_PROFILE_ZERO_AGING_MS 30000 +#define P4TC_DEFAULT_TIMER_PROFILE_ID 0 + +#define P4TC_MAX_PERMISSION (GENMASK(P4TC_PERM_MAX_BIT, 0))#define P4TC_KERNEL_PIPEID 0#define P4TC_PID_IDX 0 +#define P4TC_TBLID_IDX 1#define P4TC_AID_IDX 1#define P4TC_PARSEID_IDX 1
@@ -68,6 +81,7 @@ struct p4tc_template_common {struct p4tc_pipeline {struct p4tc_template_common common;struct idr p_act_idr; +struct idr p_tbl_idr;struct rcu_head rcu;struct net *net;u32 num_created_acts;
@@ -122,6 +136,11 @@ struct p4tc_act *p4a_runt_find(struct net *net,void p4a_runt_prealloc_put (struct p4tc_act *act,struct tcf_p4act *p4_act); +static inline bool p4tc_pipeline_sealed (struct p4tc_pipeline *pipeline) +{ +return pipeline->p_state ==P4TC_STATE_READY ; +} +static inline int p4tc_action_destroy (struct tc_action *acts[]) {struct tc_action *acts_non_prealloc[TCA_ACT_MAX_PRIO ] = {NULL };
@@ -166,6 +185,65 @@ static inline int p4tc_action_destroy(struct tc_action *acts[])return ret; } +#define P4TC_CONTROL_PERMISSIONS (GENMASK(13, 7)) +#define P4TC_DATA_PERMISSIONS (GENMASK(6, 0)) + +#define P4TC_TABLE_DEFAULT_PERMISSIONS \ + ((GENMASK(P4TC_CTRL_PERM_C_BIT, P4TC_CTRL_PERM_D_BIT)) | \ + P4TC_CTRL_PERM_P | P4TC_CTRL_PERM_S | P4TC_DATA_PERM_R | \ + P4TC_DATA_PERM_X) + +#define P4TC_PERMISSIONS_UNINIT (1 << P4TC_PERM_MAX_BIT) + +struct p4tc_table_defact { +struct tc_action *acts[2 ]; +/* Will have two 7 bits blocks containing CRUDXPS (Create, read, update, +* delete, execute, publish and subscribe) permissions for control plane +* and data plane. The first 5 bits are for control and the next five +* are for data plane. |crudxpscrudxps| if we were to denote it as UNIX +* permission flags. +*/ +__u16 perm; +struct rcu_head rcu; +}; + +struct p4tc_table_perm { +__u16 permissions; +struct rcu_head rcu; +}; + +struct p4tc_table { +struct p4tc_template_common common; +struct list_head tbl_acts_list; +struct idr tbl_masks_idr; +struct idr tbl_prio_idr; +struct xarray tbl_profiles_xa; +struct rhltable tbl_entries; +/* Mutex that protects tbl_profiles_xa */ +struct mutex tbl_profiles_xa_lock; +struct p4tc_table_defact __rcu *tbl_dflt_hitact; +struct p4tc_table_defact __rcu *tbl_dflt_missact; +struct p4tc_table_perm __rcu *tbl_permissions; +struct p4tc_table_entry_mask __rcu **tbl_masks_array; +unsigned long __rcu *tbl_free_masks_bitmap; +/* Locks the available masks IDR which will be used when adding and +* deleting table entries. +*/ +spinlock_t tbl_masks_idr_lock; +u32 tbl_keysz; +u32 tbl_id; +u32 tbl_max_entries; +u32 tbl_max_masks; +u32 tbl_curr_num_masks; +atomic_t tbl_num_timer_profiles; +/* Accounts for how many entities refer to this table. Usually just the +* pipeline it belongs to. +*/ +refcount_t tbl_ctrl_ref; +u16 tbl_type; +u16 __pad0; +}; +struct p4tc_act_param {struct list_head head;struct rcu_head rcu;
@@ -219,6 +297,18 @@ struct p4tc_act {char fullname[ACTNAMSIZ ]; }; +struct p4tc_table_act { +struct list_head node; +struct p4tc_act *act; +u8 flags; +}; + +struct p4tc_table_timer_profile { +struct rcu_head rcu; +u64 aging_ms; +u32 profile_id; +}; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla,struct tc_action *acts[],u32 pipeid,u32 flags,struct netlink_ext_ack *extack)
@@ -286,6 +376,70 @@ static inline bool p4tc_action_put_ref(struct p4tc_act *act)return refcount_dec_not_one (&act->a_ref); } +struct p4tc_act_param *p4a_parm_find_byid (struct idr *params_idr, +const u32 param_id); +struct p4tc_act_param * +p4a_parm_find_byany (struct p4tc_act *act,const char *param_name, +const u32 param_id,struct netlink_ext_ack *extack); + +struct p4tc_table *p4tc_table_find_byany (struct p4tc_pipeline *pipeline, +const char *tblname,const u32 tbl_id, +struct netlink_ext_ack *extack); +struct p4tc_table *p4tc_table_find_byid (struct p4tc_pipeline *pipeline, +const u32 tbl_id); +int p4tc_table_try_set_state_ready (struct p4tc_pipeline *pipeline, +struct netlink_ext_ack *extack); +void p4tc_table_put_mask_array (struct p4tc_pipeline *pipeline); +struct p4tc_table *p4tc_table_find_get (struct p4tc_pipeline *pipeline, +const char *tblname,const u32 tbl_id, +struct netlink_ext_ack *extack); + +static inline bool p4tc_table_put_ref (struct p4tc_table *table) +{ +return refcount_dec_not_one (&table->tbl_ctrl_ref); +} + +struct p4tc_table_defact_params { +struct p4tc_table_defact *hitact; +struct p4tc_table_defact *missact; +struct nlattr *nla_hit; +struct nlattr *nla_miss; +}; + +int p4tc_table_init_default_acts (struct net *net, +struct p4tc_table_defact_params *dflt, +struct p4tc_table *table, +struct list_head *acts_list, +struct netlink_ext_ack *extack); + +static inline bool p4tc_table_act_is_noaction (struct p4tc_table_act *table_act) +{ +return !table_act->act->common.p_id && !table_act->act->a_id; +} + +static inline bool p4tc_table_defact_is_noaction (struct tcf_p4act *p4_defact) +{ +return !p4_defact->p_id && !p4_defact->act_id; +} + +static inline void p4tc_table_defacts_acts_copy (struct p4tc_table_defact *dst, +struct p4tc_table_defact *src) +{ + dst->acts[0 ] = src->acts[0 ]; + dst->acts[1 ] =NULL ; +} + +void p4tc_table_replace_default_acts (struct p4tc_table *table, +struct p4tc_table_defact_params *dflt, +bool lock_rtnl); + +struct p4tc_table_perm * +p4tc_table_init_permissions (struct p4tc_table *table,u16 permissions, +struct netlink_ext_ack *extack); +void p4tc_table_replace_permissions (struct p4tc_table *table, +struct p4tc_table_perm *tbl_perm, +bool lock_rtnl); +struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);void p4a_runt_init_flags (struct tcf_p4act *p4act);
@@ -295,7 +449,7 @@ p4a_runt_parm_init(struct net *net, struct p4tc_act *act, struct nlattr *nla, struct netlink_ext_ack *extack);#define to_pipeline (t) ((struct p4tc_pipeline *)t) -#define to_hdrfield (t) ((struct p4tc_hdrfield *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t) +#define p4tc_to_table (t) ((struct p4tc_table *)t)#endif
include/net/p4tc_types.h +1 -1
@@ -8,7 +8,7 @@#include <uapi/linux/p4tc.h> -#define P4TC_T_MAX_BITSZ 128 +#define P4TC_T_MAX_BITSZ P4TC_MAX_KEYSZstruct p4tc_type_mask_shift {void *mask;
include/uapi/linux/p4tc.h +119 -0
137 lines not on this page · raw message
net/sched/p4tc/Makefile +1 -1
@@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ - p4tc_action.o + p4tc_action.o p4tc_table.o
net/sched/p4tc/p4tc_action.c +2 -2
@@ -661,13 +661,13 @@ p4a_parm_find_byname(struct idr *params_idr, const char *param_name)return NULL ; } -static struct p4tc_act_param * +struct p4tc_act_param *p4a_parm_find_byid (struct idr *params_idr,const u32 param_id) {return idr_find (params_idr, param_id); } -static struct p4tc_act_param * +struct p4tc_act_param *p4a_parm_find_byany (struct p4tc_act *act,const char *param_name,const u32 param_id,struct netlink_ext_ack *extack) {
net/sched/p4tc/p4tc_pipeline.c +16 -7
@@ -76,6 +76,7 @@ static const struct nla_policy tc_pipeline_policy[P4TC_PIPELINE_MAX + 1] = {static void p4tc_pipeline_destroy (struct p4tc_pipeline *pipeline) {idr_destroy (&pipeline->p_act_idr); +idr_destroy (&pipeline->p_tbl_idr);kfree (pipeline); }
@@ -98,9 +99,13 @@ static void p4tc_pipeline_teardown(struct p4tc_pipeline *pipeline,struct net *net = pipeline->net;struct p4tc_pipeline_net *pipe_net =net_generic (net, pipeline_net_id);struct net *pipeline_net =maybe_get_net (net); -unsigned long iter_act_id; +unsigned long iter_act_id, tmp; +struct p4tc_table *table;struct p4tc_act *act; -unsigned long tmp; +unsigned long tbl_id; + +idr_for_each_entry_ul (&pipeline->p_tbl_idr, table, tmp, tbl_id) + table->common.ops->put (pipeline, &table->common, extack);idr_for_each_entry_ul (&pipeline->p_act_idr, act, tmp, iter_act_id) act->common.ops->put (pipeline, &act->common, extack);
@@ -153,22 +158,23 @@ static int __p4tc_pipeline_put(struct p4tc_pipeline *pipeline,static int pipeline_try_set_state_ready (struct p4tc_pipeline *pipeline,struct netlink_ext_ack *extack) { +int ret; +if (pipeline->curr_tables != pipeline->num_tables) {NL_SET_ERR_MSG (extack,"Must have all table defined to update state to ready" );return -EINVAL ; } + ret =p4tc_table_try_set_state_ready (pipeline, extack); +if (ret <0 ) +return ret; + pipeline->p_state =P4TC_STATE_READY ;return true; } -static bool p4tc_pipeline_sealed (struct p4tc_pipeline *pipeline) -{ -return pipeline->p_state ==P4TC_STATE_READY ; -} -struct p4tc_pipeline *p4tc_pipeline_find_byid (struct net *net,const u32 pipeid) {struct p4tc_pipeline_net *pipe_net;
@@ -262,6 +268,9 @@ p4tc_pipeline_create(struct net *net, struct nlmsghdr *n,idr_init (&pipeline->p_act_idr); +idr_init (&pipeline->p_tbl_idr); + pipeline->curr_tables =0 ; + pipeline->num_created_acts =0 ; pipeline->p_state =P4TC_STATE_NOT_READY ;
net/sched/p4tc/p4tc_table.c new file +1606 -0
1,606 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 12/15 p4tc: add runtime table entry create and update
In reply to Jamal Hadi SalimTables are conceptually similar to TCAMs and this implementation could be labelled as an "algorithmic" TCAM. Tables have a key of a specific size, maximum number of entries and masks allowed. The basic P4 key types are supported (exact, LPM, ternary, and ranges) although the kernel side is oblivious of all that and sees only bit blobs which it masks before a lookup is performed.
This commit allows users to create and update table _entries_ (templates were described in earlier patch). We'll add get, delete and dump in the next commit
Note that table entries can only be created once the pipeline template is sealed.
For example, a user issuing the following command:
tc p4ctrl create myprog/table/cb/tname \
dstAddr 10.10.10.0/24 srcAddr 192.168.0.0/16 prio 16 \
action send param port port1
indicates we are creating a table entry in table "cb/tname" (a table residing in control block "cb") on a pipeline named "myprog".
User space tc will create a key which has a value of 0x0a0a0a00c0a00000 (10.10.10.0 concatenated with 192.168.0.0) and a mask value of 0xffffff00ffff0000 (/24 concatenated with /16) that will be sent to the kernel. In addition a priority field of 16 is passed to the kernel as well as the action definition.
The priority field is needed to disambiguate in case two entries match. In that case, the kernel will choose the one with lowest priority number.
If the user wanted to, for example, update our just created entry with an action, they'd issue the following command:
tc p4ctrl update myprog/table/cb/tname srcAddr 10.10.10.0/24 \
dstAddr 192.168.0.0/16 prio 16 action send param port port5
In this case, the user needs to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry.
__Table Attributes__
Just for context, these are the table attributes (presented in the table template commit):
- Timer profiles: The timer profiles for the table entries belonging to the table. These profiles contain the supported aging values for the table. There are currently 4 different built-in profiles ID0=30s(default), ID1=60s, ID2=120s and ID3=180s.
The user can override the defaults by specifying the number of profiles per table. In that case, the kernel will generate the specified number of profiles with their aging values abiding by the following rule:
- Profile ID 0 - 30000ms
- Profile ID(n) - Profile ID(n - 1) + 30000ms
So, for example, if the user specified num_timer_profiles as 5, the profile IDs and aging values would be the following:
- Profile ID 0 - 30000ms (default profile)
- Profile ID 1 - 60000ms
- Profile ID 2 - 90000ms
- Profile ID 3 - 120000ms
- Profile ID 4 - 150000ms
The values of the different profiles could be changed at runtime. The default profile is used if the user specifies an out of range value.
- Match key type: The match type of the table key (exact, LPM, ternary, etc).
Note in P4 a key may constitute multiple (sub)key types, eg matching on srcip using a prefix and an exact match for dstip. In such a case it is up to the compiler to generalize the key used (For example in this case the overall key may endup being LPM or ternary).
- Direct Counter: Table counter instances used directly by the table, when specified in the P4 program, there will be one counter per entry.
- Direct Meter: Table meter instances used directly by the table, when specified in the P4 program, there will be one meter per entry.
- CRUDXPS Permissions both for specific entries and tables. The permissions are applicable to both the control plane and the datapath (see "Table Permissions" further below).
- Allowed Actions List. This will be a list of all possible actions that can be added to table entries that are added to the specified table.
- Action profiles. When defined in a P4 program, action profiles provide a mechanism to share action instances.
- Actions Selectors. When defined in a P4 program can be used to select the execution of one or more action instance selected at table lookup time by using a hash computation.
- Default hit action. When a default hit action is defined it is used when a matched table entry did not define an action. Depending on the P4 program the default hit action can be updated at runtime (in addition to being specified in the template).
- Default miss action. When a default miss action is defined it is used when a lookup that table fails.
- Action scope. In addition to actions being annotated as default hit or miss they can also be annotated to be either specific to a table of globally available to multiple tables within the same P4 program.
- Max entries. This is an upper bound for number of entries a specific table allows.
- Num masks. In the case of LPM or ternary matches, this defines the maximum allowed masks for that table.
- Timers. When defined in a P4 program, each entry has an associated timer. Depending on the programmed timer profile (see above), an entry gets a timeout. The timer attribute specifies the behavior of a table entry expiration.
The timer is refreshed every time there's a hit. After an idle period the P4 program can define using this attribute, whether it wants to have an event generated to user space (and have user space delete the entry), or whether it wants the kernel to delete it and send the event to announce the deletion. The default in P4TC is to both delete and generate an event.
- per entry "static" vs "dynamic" entry. By default all entries created from the control plane are "static" unless otherwise specified. All entries added from datapath are "dynamic" unless otherwise specified. "Dynamic" entries are subject to deletion when idle (subject to the rules specified in "Timers" above).
__Table Entry Permissions__
Whereas templates can define the default permissions for tables, runtime has ability to define table entry permissions as they are added.
To reiterate there are two types of permissions:
- Table permissions which are a property of the table (think directory in file systems). These are set by the template (see earlier patch on table template).
- Table entry permissions which are specific to a table entry (think a file in a directory). This patch describes those permissions.
Furthermore in both cases the permissions are split into datapath vs control path. The template definition can set either one. For example, one could allow for adding table entries by the datapath in case of PNA add-on-miss is needed.
By default tables entries have control plane RUD, meaning the control plane can Read, Update or Delete entries. By default, as well, the control plane can create new entries unless specified otherwise by the template.
Lets see an example of which creates the table "cb/tname" at template time:
tc p4template create table/aP4proggie/cb/tname tblid 1 keysz 64 \
permissions 0x3C24 ...Above is setting the table tname's permission to be 0x3C24 is equivalent to CRUD----R--X-- meaning:
The control plane can Create, Read, Update, Delete The datapath can only Read and Execute table entries. If one was to dump this table with:
tc -j p4template get table/aP4proggie/cb/tname | jq .
The output would be the following:
[
{
"obj": "table",
"pname": "aP4proggie",
"pipeid": 22
},
{
"templates": [
{
"tblid": 1,
"tname": "cb/tname",
"keysz": 64,
"max_entries": 256,
"masks": 8,
"entries": 0,
"permissions": "CRUD----R--X--",
"table_type": "exact",
"acts_list": []
}
] }
]
The expressed permissions above are probably the most practical for most use cases.
__Constant Tables And P4-programmed Defined Entries__
If one wanted to restrict the table to be an equivalent to a "const" then the permissions would be set to be: -R------R--X-- by the template.
In such a case, typically the P4 program will have some entries defined (see the famous P4 calculator example). The "initial entries" specified in the P4 program will have to be added by the template (as generated by the compiler), as such:
tc p4template create table/aP4proggie/cb/tname \
entry srcAddr 10.10.10.10/24 dstAddr 1.1.1.0/24 prio 17
This table cannot be updated at runtime. Any attempt to add an entry of a table which is read-only at runtime will get a "permission denied" response back from the kernel.
Note: If one was to create an equivalent for PNA add-on-miss feature for this table, then the template would issue table permissions as:
-R-----CR--X-- PNA doesn't specify whether the datapath can also delete or update entries, but if it did then more appropriate permissions will be:
-R-----CRUDX--
__Mix And Match of RW vs Constant Entries__
Lets look at other scenarios; lets say the table has CRUD----R--X-- permissions as defined by the template...
At runtime the user could add entries which are "const" - by specifying the entry's permission as -R------R--X-- example:
tc p4ctrl create aP4proggie/table/cb/tname srcAddr 10.10.10.10/24 \
dstAddr 1.1.1.0/24 prio 17 permissions 0x1024 action drop
or not specify permissions at all as such:
tc p4ctrl create aP4proggie/table/cb/tname srcAddr 10.10.10.10/24 \
dstAddr 1.1.1.0/24 prio 17 action drop
in which case the table's permissions defined at template time(CRUD----R--X--) are assumed; meaning the table entry can be deleted or updated by the control plane.
__Entries permissions Allowed On A Table Entry Creation At Runtime__
When entries are created without specifying permissions they inherit the table permissions set by the template.
When an entry is added with expressed permissions it has at most to have what the template table definition expressed but could ask for less permission. For example, assuming a table with templated specified permissions of CR-D----R--X--:
An entry created at runtime with permission of -R------R--X-- is allowed but an entry with -RUD----R--X-- will be rejected.
--- include/net/p4tc.h | 119 +- include/uapi/linux/p4tc.h | 73 +- include/uapi/linux/rtnetlink.h | 9 + net/sched/p4tc/Makefile | 3 +- net/sched/p4tc/p4tc_runtime_api.c | 82 ++ net/sched/p4tc/p4tc_table.c | 96 +- net/sched/p4tc/p4tc_tbl_entry.c | 1804 +++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_tmpl_api.c | 4 +- security/selinux/nlmsgtab.c | 6 +- 9 files changed, 2179 insertions(+), 17 deletions(-) create mode 100644 net/sched/p4tc/p4tc_runtime_api.c create mode 100644 net/sched/p4tc/p4tc_tbl_entry.c
include/net/p4tc.h +117 -2
@@ -124,6 +124,11 @@ static inline bool p4tc_pipeline_get(struct p4tc_pipeline *pipeline)return refcount_inc_not_zero (&pipeline->p_ctrl_ref); } +static inline void p4tc_pipeline_put_ref (struct p4tc_pipeline *pipeline) +{ +refcount_dec (&pipeline->p_ctrl_ref); +} +void p4tc_pipeline_put (struct p4tc_pipeline *pipeline);struct p4tc_pipeline * p4tc_pipeline_find_byany_unsealed(struct net *net, const char *p_name,
@@ -195,6 +200,8 @@ static inline int p4tc_action_destroy(struct tc_action *acts[])#define P4TC_PERMISSIONS_UNINIT (1 << P4TC_PERM_MAX_BIT) +#define P4TC_MAX_PARAM_DATA_SIZE 124 +struct p4tc_table_defact {struct tc_action *acts[2 ];/* Will have two 7 bits blocks containing CRUDXPS (Create, read, update,
@@ -216,11 +223,12 @@ struct p4tc_table {struct p4tc_template_common common;struct list_head tbl_acts_list;struct idr tbl_masks_idr; -struct idr tbl_prio_idr; +struct ida tbl_prio_idr;struct xarray tbl_profiles_xa;struct rhltable tbl_entries;/* Mutex that protects tbl_profiles_xa */ struct mutex tbl_profiles_xa_lock; +struct p4tc_table_entry *tbl_entry;struct p4tc_table_defact __rcu *tbl_dflt_hitact;struct p4tc_table_defact __rcu *tbl_dflt_missact;struct p4tc_table_perm __rcu *tbl_permissions;
@@ -236,6 +244,8 @@ struct p4tc_table {u32 tbl_max_masks;u32 tbl_curr_num_masks;atomic_t tbl_num_timer_profiles; +/* Accounts for how many entries this table has */ +atomic_t tbl_nelems;/* Accounts for how many entities refer to this table. Usually just the * pipeline it belongs to. */
@@ -309,6 +319,76 @@ struct p4tc_table_timer_profile {u32 profile_id; }; +extern const struct rhashtable_params entry_hlt_params; + +struct p4tc_table_entry ; +struct p4tc_table_entry_work { +struct work_struct work; +struct p4tc_pipeline *pipeline; +struct p4tc_table_entry *entry; +struct p4tc_table *table; +u16 who_deleted; +bool send_event; +}; + +struct p4tc_table_entry_key { +u32 keysz; +/* Key start */ +u32 maskid; +unsigned char fa_key[]__aligned(8) ; +}; + +struct p4tc_table_entry_value { +struct tc_action *acts[2 ]; +/* Accounts for how many entities are referencing it +* eg: Data path, one or more control path and timer. +*/ +u32 prio; +refcount_t entries_ref; +u32 permissions; +u32 __pad0; +struct p4tc_table_entry_tm __rcu *tm; +struct p4tc_table_entry_work *entry_work; +u64 aging_ms; +struct hrtimer entry_timer; +bool tmpl_created; +}; + +struct p4tc_table_entry_mask { +struct rcu_head rcu; +u32 sz; +u32 mask_index; +/* Accounts for how many entries are using this mask */ +refcount_t mask_ref; +u32 mask_id; +unsigned char fa_value[]__aligned(8) ; +}; + +struct p4tc_table_entry { +struct rcu_head rcu; +struct rhlist_head ht_node; +struct p4tc_table_entry_key key; +/* fallthrough: key data + value */ +}; + +#define P4TC_KEYSZ_BYTES (bits) (round_up(BITS_TO_BYTES(bits), 8)) + +static inline void *p4tc_table_entry_value (struct p4tc_table_entry *entry) +{ +return entry->key.fa_key +P4TC_KEYSZ_BYTES (entry->key.keysz); +} + +static inline struct p4tc_table_entry_work * +p4tc_table_entry_work (struct p4tc_table_entry *entry) +{ +struct p4tc_table_entry_value *value =p4tc_table_entry_value (entry); + +return value->entry_work; +} + +extern const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ]; +extern const struct nla_policy p4tc_policy[P4TC_MAX +1 ]; +static inline int p4tc_action_init (struct net *net,struct nlattr *nla,struct tc_action *acts[],u32 pipeid,u32 flags,struct netlink_ext_ack *extack)
@@ -422,6 +502,21 @@ static inline bool p4tc_table_defact_is_noaction(struct tcf_p4act *p4_defact)return !p4_defact->p_id && !p4_defact->act_id; } +static inline void p4tc_table_defact_destroy (struct p4tc_table_defact *defact) +{ +if (defact) { +if (defact->acts[0 ]) { +struct tcf_p4act *p4_defact =to_p4act (defact->acts[0 ]); + +if (p4tc_table_defact_is_noaction (p4_defact)) +kfree (p4_defact); +else +p4tc_action_destroy (defact->acts); + } +kfree (defact); + } +} +static inline void p4tc_table_defacts_acts_copy (struct p4tc_table_defact *dst,struct p4tc_table_defact *src) {
@@ -435,10 +530,25 @@ void p4tc_table_replace_default_acts(struct p4tc_table *table,struct p4tc_table_perm *p4tc_table_init_permissions (struct p4tc_table *table,u16 permissions, -struct netlink_ext_ack *extack); +struct netlink_ext_ack *extack);void p4tc_table_replace_permissions (struct p4tc_table *table,struct p4tc_table_perm *tbl_perm,bool lock_rtnl); +void p4tc_table_entry_destroy_hash (void *ptr,void *arg); + +struct p4tc_table_entry * +p4tc_tmpl_table_entry_cu (struct net *net,struct nlattr *arg, +struct p4tc_pipeline *pipeline, +struct p4tc_table *table, +struct netlink_ext_ack *extack); +int p4tc_tbl_entry_root (struct net *net,struct sk_buff *skb, +struct nlmsghdr *n,int cmd, +struct netlink_ext_ack *extack); +int p4tc_tbl_entry_dumpit (struct net *net,struct sk_buff *skb, +struct netlink_callback *cb, +struct nlattr *arg,char *p_name); +int p4tc_tbl_entry_fill (struct sk_buff *skb,struct p4tc_table *table, +struct p4tc_table_entry *entry,u32 tbl_id);struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);
@@ -448,6 +558,11 @@ struct p4tc_act_param * p4a_runt_parm_init(struct net *net, struct p4tc_act *act, struct nlattr *nla, struct netlink_ext_ack *extack); +static inline bool p4tc_runtime_msg_is_update (struct nlmsghdr *n) +{ +return n->nlmsg_type ==RTM_P4TC_UPDATE ; +} +#define to_pipeline (t) ((struct p4tc_pipeline *)t)#define p4tc_to_act (t) ((struct p4tc_act *)t)#define p4tc_to_table (t) ((struct p4tc_table *)t)
include/uapi/linux/p4tc.h +72 -1
103 lines not on this page · raw message
include/uapi/linux/rtnetlink.h +9 -0
@@ -203,6 +203,15 @@ enum { RTM_UPDATEP4TEMPLATE,#define RTM_UPDATEP4TEMPLATE RTM_UPDATEP4TEMPLATE + RTM_P4TC_CREATE = 128, +#define RTM_P4TC_CREATE RTM_P4TC_CREATE + RTM_P4TC_DEL, +#define RTM_P4TC_DEL RTM_P4TC_DEL + RTM_P4TC_GET, +#define RTM_P4TC_GET RTM_P4TC_GET + RTM_P4TC_UPDATE, +#define RTM_P4TC_UPDATE RTM_P4TC_UPDATE + __RTM_MAX,#define RTM_MAX (((__RTM_MAX + 3) & ~3) - 1) };
net/sched/p4tc/Makefile +2 -1
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ - p4tc_action.o p4tc_table.o + p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ + p4tc_runtime_api.o
net/sched/p4tc/p4tc_runtime_api.c new file +82 -0
82 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +86 -10
164 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c new file +1804 -0
1,804 lines not on this page · raw message
net/sched/p4tc/p4tc_tmpl_api.c +2 -2
@@ -27,12 +27,12 @@#include <net/netlink.h> #include <net/flow_offload.h> -static const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ] = { +const struct nla_policy p4tc_root_policy[P4TC_ROOT_MAX +1 ] = { [P4TC_ROOT ] = { .type =NLA_NESTED }, [P4TC_ROOT_PNAME ] = { .type =NLA_STRING , .len =P4TC_PIPELINE_NAMSIZ }, }; -static const struct nla_policy p4tc_policy[P4TC_MAX +1 ] = { +const struct nla_policy p4tc_policy[P4TC_MAX +1 ] = { [P4TC_PATH ] = { .type =NLA_BINARY , .len =P4TC_PATH_MAX *sizeof (u32) }, [P4TC_PARAMS ] = { .type =NLA_NESTED },
security/selinux/nlmsgtab.c +5 -1
@@ -98,6 +98,10 @@ static const struct nlmsg_perm nlmsg_route_perms[] = { { RTM_DELP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, { RTM_GETP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_READ }, { RTM_UPDATEP4TEMPLATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_CREATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_DEL, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, + { RTM_P4TC_GET, NETLINK_ROUTE_SOCKET__NLMSG_READ }, + { RTM_P4TC_UPDATE, NETLINK_ROUTE_SOCKET__NLMSG_WRITE }, }; static const struct nlmsg_perm nlmsg_tcpdiag_perms[] = {
@@ -181,7 +185,7 @@ int selinux_nlmsg_lookup(u16 sclass, u16 nlmsg_type, u32 *perm)* structures at the top of this file with the new mappings * before updating the BUILD_BUG_ON() macro! */ -BUILD_BUG_ON (RTM_MAX != (RTM_CREATEP4TEMPLATE +3 )); +BUILD_BUG_ON (RTM_MAX != (RTM_P4TC_CREATE +3 )); err =nlmsg_perm (nlmsg_type, perm, nlmsg_route_perms,sizeof (nlmsg_route_perms));break ;
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 13/15 p4tc: add runtime table entry get, delete, flush and dump
In reply to Jamal Hadi SalimThis commit allows users to get, delete, flush and dump table _entries_ (templates were described in earlier patch and so were create and update).
If the user wants to, for example, read a table entry from a table (nh_table) which has as key nh_index (bit32), belongs to pipeline (routing) and has as its key an nh_index of 1, they'd issue the following command:
tc p4ctrl get routing/table/Main/nh_table nh_index 1
Which will give us the following output:
$TC p4ctrl get routing/table/Main/nh_table
pipeline: routing(id 1)
table: Main/nh_table(id 1)entry priority 64000[permissions -RUD-PS-R--X--]
entry key
nh_index id:1 size:32b type:bit32 exact fieldval 1
entry actions:
action order 1: routing/Main/set_nh index 2 ref 1 bind 1
params:
dmac type macaddr value: 13:37:13:37:13:37 id 1
port type dev value: port1 id 2
created by: tc (id 2)
dynamic false
created 20338 sec used 39 sec
Note that, as with create and update, we need to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry. Also, in this case, the entry had an action which has a parameter dmac (a mac address) and port1 (a net device).
If the user wanted to delete the same table entry, they'd issue the following command:
tc p4ctrl del routing/table/Main/nh_table nh_index 1
Note that, again, we need to specify the pipeline name, the table name, the key and the priority, so that we can locate the table entry.
We can also flush all the table entries from a specific table. To flush the table entries of table tname ane pipeline ptables, the user would issue the following command:
tc p4ctrl del routing/table/Main/nh_table
Likewise, we can also dump all the table entries from a specific table. To dump the table entries of table tname and pipeline myprog, the user would issue the following command:
tc p4ctrl get routing/table/Main/nh_table
Which will give the following output (in case we had one entry with nh_index key value 1):
pipeline: routing(id 1)
table: Main/nh_table(id 1)entry priority 64000[permissions -RUD-PS-R--X--]
entry key
nh_index id:1 size:32b type:bit32 exact fieldval 1
entry actions:
action order 1: routing/Main/set_nh index 2 ref 1 bind 1
params:
dmac type macaddr value: 13:37:13:37:13:37 id 1
port type dev value: port1 id 2
created by: tc (id 2)
dynamic false
created 20338 sec used 39 sec
--- include/net/p4tc.h | 28 +- include/uapi/linux/p4tc.h | 90 +++- net/sched/p4tc/Makefile | 2 +- net/sched/p4tc/p4tc_filter.c | 870 ++++++++++++++++++++++++++++++ net/sched/p4tc/p4tc_runtime_api.c | 63 +++ net/sched/p4tc/p4tc_table.c | 113 +++- net/sched/p4tc/p4tc_tbl_entry.c | 853 ++++++++++++++++++++++++++++- 7 files changed, 2010 insertions(+), 9 deletions(-) create mode 100644 net/sched/p4tc/p4tc_filter.c
include/net/p4tc.h +27 -1
@@ -36,9 +36,15 @@#define P4TC_AID_IDX 1#define P4TC_PARSEID_IDX 1 +struct p4tc_filter { +struct p4tc_filter_oper *operation; +int cmd;/* CRUD command */ +}; +struct p4tc_dump_ctx {u32 ids[P4TC_PATH_MAX ];struct rhashtable_iter *iter; +struct p4tc_filter *entry_filter; };struct p4tc_template_common ;
@@ -351,6 +357,7 @@ struct p4tc_table_entry_value {struct p4tc_table_entry_work *entry_work;u64 aging_ms;struct hrtimer entry_timer; +bool is_dyn;bool tmpl_created; };
@@ -534,6 +541,15 @@ p4tc_table_init_permissions(struct p4tc_table *table, u16 permissions,void p4tc_table_replace_permissions (struct p4tc_table *table,struct p4tc_table_perm *tbl_perm,bool lock_rtnl); +int p4tc_table_timer_profile_update (struct p4tc_table *table, +struct nlattr *nla, +struct netlink_ext_ack *extack); +struct p4tc_table_timer_profile * +p4tc_table_timer_profile_find_byaging (struct p4tc_table *table, +u64 aging_ms); +struct p4tc_table_timer_profile * +p4tc_table_timer_profile_find (struct p4tc_table *table,u32 profile_id); +void p4tc_table_entry_destroy_hash (void *ptr,void *arg); struct p4tc_table_entry *
@@ -548,7 +564,17 @@ int p4tc_tbl_entry_dumpit(struct net *net, struct sk_buff *skb,struct netlink_callback *cb, struct nlattr *arg, char *p_name);int p4tc_tbl_entry_fill (struct sk_buff *skb,struct p4tc_table *table, -struct p4tc_table_entry *entry,u32 tbl_id); +struct p4tc_table_entry *entry,u32 tbl_id, +u16 who_deleted); +void p4tc_tbl_entry_mask_key (u8 *masked_key,u8 *key,const u8 *mask, +u32 masksz); + +struct p4tc_filter * +p4tc_filter_build (struct p4tc_pipeline *pipeline,struct p4tc_table *table, +struct nlattr *nla,struct netlink_ext_ack *extack); +bool p4tc_filter_exec (struct p4tc_filter *filter, +struct p4tc_table_entry *entry); +void p4tc_filter_destroy (struct p4tc_filter *filter);struct tcf_p4act *p4a_runt_prealloc_get_next (struct p4tc_act *act);
include/uapi/linux/p4tc.h +89 -1
119 lines not on this page · raw message
net/sched/p4tc/Makefile +1 -1
@@ -2,4 +2,4 @@ obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ - p4tc_runtime_api.o + p4tc_filter.o p4tc_runtime_api.o
net/sched/p4tc/p4tc_filter.c new file +870 -0
870 lines not on this page · raw message
net/sched/p4tc/p4tc_runtime_api.c +63 -0
81 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +112 -1
131 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c +848 -5
1,004 lines not on this page · raw message
Signature · 1 line
2.34.1
Jamal Hadi Salim
maintainerPATCH v12 14/15 p4tc: add set of P4TC table kfuncs
In reply to Jamal Hadi SalimWe add an initial set of kfuncs to allow interactions from eBPF programs to the P4TC domain.
- bpf_p4tc_tbl_read: Used to lookup a table entry from a BPF program installed in TC. To find the table entry we take in an skb, the pipeline ID, the table ID, a key and a key size.
We use the skb to get the network namespace structure where all the pipelines are stored. After that we use the pipeline ID and the table ID, to find the table. We then use the key to search for the entry. We return an entry on success and NULL on failure.
- xdp_p4tc_tbl_read: Used to lookup a table entry from a BPF program installed in XDP. To find the table entry we take in an xdp_md, the pipeline ID, the table ID, a key and a key size.
We use struct xdp_md to get the network namespace structure where all the pipelines are stored. After that we use the pipeline ID and the table ID, to find the table. We then use the key to search for the entry. We return an entry on success and NULL on failure.
- bpf_p4tc_entry_create: Used to create a table entry from a BPF program installed in TC. To create the table entry we take an skb, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_create: Used to create a table entry from a BPF program installed in XDP. To create the table entry we take an xdp_md, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_create_on_miss: conforms to PNA "add on miss". First does a lookup using the passed key and upon a miss will add the entry to the table.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_create_on_miss: conforms to PNA "add on miss". First does a lookup using the passed key and upon a miss will add the entry to the table.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_update: Used to update a table entry from a BPF program installed in TC. To update the table entry we take an skb, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_update: Used to update a table entry from a BPF program installed in XDP. To update the table entry we take an xdp_md, the pipeline ID, the table ID, a key and its size, and an action which will be associated with the new entry.
We return 0 on success and a negative errno on failure
- bpf_p4tc_entry_delete: Used to delete a table entry from a BPF program installed in TC. To delete the table entry we take an skb, the pipeline ID, the table ID, a key and a key size.
We return 0 on success and a negative errno on failure
- xdp_p4tc_entry_delete: Used to delete a table entry from a BPF program installed in XDP. To delete the table entry we take an xdp_md, the pipeline ID, the table ID, a key and a key size.
We return 0 on success and a negative errno on failure
--- include/linux/bitops.h | 1 + include/net/p4tc.h | 89 ++++++- include/net/tc_act/p4tc.h | 51 ++++ include/uapi/linux/p4tc.h | 2 + net/sched/p4tc/Makefile | 1 + net/sched/p4tc/p4tc_action.c | 71 ++++- net/sched/p4tc/p4tc_bpf.c | 342 ++++++++++++++++++++++++ net/sched/p4tc/p4tc_pipeline.c | 43 +++ net/sched/p4tc/p4tc_table.c | 41 +++ net/sched/p4tc/p4tc_tbl_entry.c | 454 ++++++++++++++++++++++++++++++-- 10 files changed, 1062 insertions(+), 33 deletions(-) create mode 100644 net/sched/p4tc/p4tc_bpf.c
include/linux/bitops.h +1 -0
@@ -19,6 +19,7 @@#define BITS_TO_LONGS (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(long))#define BITS_TO_U64 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u64))#define BITS_TO_U32 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u32)) +#define BITS_TO_U16 (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(u16))#define BITS_TO_BYTES (nr) __KERNEL_DIV_ROUND_UP(nr, BITS_PER_TYPE(char))extern unsigned int __sw_hweight8 (unsigned int w);
include/net/p4tc.h +84 -5
136 lines not on this page · raw message
include/net/tc_act/p4tc.h +51 -0
65 lines not on this page · raw message
include/uapi/linux/p4tc.h +2 -0
@@ -19,6 +19,8 @@ struct p4tcmsg {#define P4TC_MINTABLES_COUNT 0#define P4TC_MSGBATCH_SIZE 16 +#define P4TC_ACT_MAX_NUM_PARAMS P4TC_MSGBATCH_SIZE +#define P4TC_MAX_KEYSZ 512#define P4TC_DEFAULT_NUM_PREALLOC 16
net/sched/p4tc/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ p4tc_filter.o p4tc_runtime_api.o +obj-$(CONFIG_DEBUG_INFO_BTF) += p4tc_bpf.o
net/sched/p4tc/p4tc_action.c +70 -1
133 lines not on this page · raw message
net/sched/p4tc/p4tc_bpf.c new file +342 -0
342 lines not on this page · raw message
net/sched/p4tc/p4tc_pipeline.c +43 -0
55 lines not on this page · raw message
net/sched/p4tc/p4tc_table.c +41 -0
99 lines not on this page · raw message
net/sched/p4tc/p4tc_tbl_entry.c +427 -27
512 lines not on this page · raw message
Signature · 1 line
2.34.1
Martin KaFai Lau
reviewerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 1 line · Jamal Hadi Salim
+struct
p4tc_table_entry_act_bpf_params{
Will this struct be extended in the future?
Quoted 10 lines
include/net/p4tc.h +10
-0+u32 pipeid; +u32 tblid; +}; + +struct p4tc_table_entry_create_bpf_params { +u32 profile_id; +u32 pipeid; +u32 tblid; +}; +
[ ... ]
Quoted 54 lines
include/net/tc_act/p4tc.h +35
-0@@ -13,10 +13,26 @@ struct tcf_p4act_params {u32 tot_params_sz; }; +#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );
[ ... ]
include/net/tc_act/p4tc.h
Quoted 2 lines
include/net/tc_act/p4tc.h +2
-0+__bpf_kfunc static struct p4tc_table_entry_act_bpf * +bpf_p4tc_tbl_read(struct __sk_buff *skb_ctx,
The argument could be "struct sk_buff *skb" instead of __sk_buff. Take a look at commit 2f4643934670.
Quoted 42 lines
include/net/tc_act/p4tc.h +42
-0+struct p4tc_table_entry_act_bpf_params *params, + void *key, const u32 key__sz) +{ + struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *caller_net; + + caller_net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +__bpf_kfunc static struct p4tc_table_entry_act_bpf * +xdp_p4tc_tbl_read (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_act_bpf_params *params, +void *key,const u32 key__sz) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *caller_net; + + caller_net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ;
include/net/tc_act/p4tc.h
Quoted 10 lines
include/net/tc_act/p4tc.h +10
-0+ + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ; + +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);
My understanding is this kfunc will allocate a "struct p4tc_table_entry_act_bpf_kern" object. If the bpf_p4tc_entry_delete() kfunc is never called and the bpf prog is unloaded, how the act_bpf object will be cleaned up?
Quoted 36 lines
include/net/tc_act/p4tc.h +36
-0+} + +__bpf_kfunc static int +bpf_p4tc_entry_create (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +xdp_p4tc_entry_create (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +bpf_p4tc_entry_create_on_miss (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{
include/net/tc_act/p4tc.h
Quoted 10 lines
include/net/tc_act/p4tc.h +10
-0+struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +xdp_p4tc_entry_create_on_miss(struct xdp_md *xdp_ctx,
Same here. "struct xdp_buff *xdp".
Quoted 12 lines
include/net/tc_act/p4tc.h +12
-0+struct p4tc_table_entry_create_bpf_params *params, + void *key, const u32 key__sz, + struct p4tc_table_entry_act_bpf *act_bpf) +{ + struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} +
[ ... ]
Quoted 17 lines
include/net/tc_act/p4tc.h +17
-0+__bpf_kfunc static int +bpf_p4tc_entry_delete (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz) +{ +struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_entry_delete (net, params, key, key__sz); +} + +__bpf_kfunc static int +xdp_p4tc_entry_delete (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz)
include/net/tc_act/p4tc.h
Quoted 10 lines
include/net/tc_act/p4tc.h +10
-0+{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_entry_delete (net, params, key, key__sz); +} + +BTF_SET8_START (p4tc_kfunc_check_tbl_set_skb)
This soon will be broken with the latest change in bpf-next. It is replaced by BTF_KFUNCS_START. commit a05e90427ef6.
What is the plan on the selftest ?
Quoted 19 lines
include/net/tc_act/p4tc.h +19
-0+BTF_ID_FLAGS (func, bpf_p4tc_tbl_read,KF_RET_NULL ); +BTF_ID_FLAGS (func, bpf_p4tc_entry_create); +BTF_ID_FLAGS (func, bpf_p4tc_entry_create_on_miss); +BTF_ID_FLAGS (func, bpf_p4tc_entry_update); +BTF_ID_FLAGS (func, bpf_p4tc_entry_delete); +BTF_SET8_END (p4tc_kfunc_check_tbl_set_skb) + +static const struct btf_kfunc_id_set p4tc_kfunc_tbl_set_skb = { + .owner =THIS_MODULE , + .set = &p4tc_kfunc_check_tbl_set_skb, +}; + +BTF_SET8_START (p4tc_kfunc_check_tbl_set_xdp) +BTF_ID_FLAGS (func, xdp_p4tc_tbl_read, KF_RET_NULL); +BTF_ID_FLAGS (func, xdp_p4tc_entry_create); +BTF_ID_FLAGS (func, xdp_p4tc_entry_create_on_miss); +BTF_ID_FLAGS (func, xdp_p4tc_entry_update); +BTF_ID_FLAGS (func, xdp_p4tc_entry_delete); +BTF_SET8_END (p4tc_kfunc_check_tbl_set_xdp)
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 10 lines · Martin KaFai Lau
Quoted 1 line · Jamal Hadi Salim
+struct
p4tc_table_entry_act_bpf_params{Will this struct be extended in the future?
include/net/p4tc.h struct p4tc_pipeline { · line 109
Quoted 4 lines
include/net/p4tc.h +4
-0+u32 pipeid; +u32 tblid; +}; +
Not that i can think of. We probably want to have the option to do so if needed. Do you see any harm if we were to make changes for whatever reason in the future?
Quoted 67 lines
Quoted 6 lines
include/net/p4tc.h +6
-0+struct p4tc_table_entry_create_bpf_params { +u32 profile_id; +u32 pipeid; +u32 tblid; +}; +[ ... ]
Quoted 54 lines
include/net/tc_act/p4tc.h +35
-0@@ -13,10 +13,26 @@ struct tcf_p4act_params {u32 tot_params_sz; }; +#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );[ ... ]
Quoted 5 lines
include/net/tc_act/p4tc.h
Quoted 2 lines
include/net/tc_act/p4tc.h +2
-0+__bpf_kfunc static struct p4tc_table_entry_act_bpf * +bpf_p4tc_tbl_read(struct __sk_buff *skb_ctx,The argument could be "struct
sk_buff*skb" instead of__sk_buff. Take a look at commit2f4643934670.
We'll make that change.
Quoted 50 lines
Quoted 50 lines
include/net/tc_act/p4tc.h +49
-0+struct p4tc_table_entry_act_bpf_params *params, + void *key, const u32 key__sz) +{ + struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *caller_net; + + caller_net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +__bpf_kfunc static struct p4tc_table_entry_act_bpf * +xdp_p4tc_tbl_read (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_act_bpf_params *params, +void *key,const u32 key__sz) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *caller_net; + + caller_net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ; + + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ;
Quoted 9 lines
include/net/tc_act/p4tc.h
Quoted 3 lines
include/net/tc_act/p4tc.h +3
-0+ +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
Quoted 40 lines
Quoted 40 lines
include/net/tc_act/p4tc.h +40
-0+} + +__bpf_kfunc static int +bpf_p4tc_entry_create (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +xdp_p4tc_entry_create (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +bpf_p4tc_entry_create_on_miss (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk);
Quoted 9 lines
include/net/tc_act/p4tc.h
Quoted 6 lines
include/net/tc_act/p4tc.h +6
-0+ +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} + +__bpf_kfunc static int +xdp_p4tc_entry_create_on_miss(struct xdp_md *xdp_ctx,Same here. "struct
xdp_buff*xdp".
ACK
Quoted 37 lines
Quoted 12 lines
include/net/tc_act/p4tc.h +12
-0+struct p4tc_table_entry_create_bpf_params *params, + void *key, const u32 key__sz, + struct p4tc_table_entry_act_bpf *act_bpf) +{ + struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_entry_create (net, params, key, key__sz, act_bpf); +} +[ ... ]
Quoted 22 lines
include/net/tc_act/p4tc.h +22
-0+__bpf_kfunc static int +bpf_p4tc_entry_delete (struct __sk_buff *skb_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz) +{ +struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *net; + + net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_entry_delete (net, params, key, key__sz); +} + +__bpf_kfunc static int +xdp_p4tc_entry_delete (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *net; + + net =dev_net (ctx->rxq->dev);
Quoted 9 lines
include/net/tc_act/p4tc.h
Quoted 5 lines
include/net/tc_act/p4tc.h +5
-0+ +return __bpf_p4tc_entry_delete (net, params, key, key__sz); +} + +BTF_SET8_START (p4tc_kfunc_check_tbl_set_skb)This soon will be broken with the latest change in bpf-next. It is replaced by
BTF_KFUNCS_START. commita05e90427ef6.
Ok, this wasnt in net-next when we pushed. We base our changes on net-next. When do you plan to merge that into net-next?
Quoted 2 lines
What is the plan on the selftest ?
We may need some guidance. How do you see us writing a selftest for this? We have extensive testing on the control side which is netlink (not part of the current series).
Overall: I thank you for taking time to review - it is the kind of feedback we were hoping for from the ebpf side.
cheers,
jamal
Quoted 20 lines
Quoted 19 lines
include/net/tc_act/p4tc.h +19
-0+BTF_ID_FLAGS (func, bpf_p4tc_tbl_read,KF_RET_NULL ); +BTF_ID_FLAGS (func, bpf_p4tc_entry_create); +BTF_ID_FLAGS (func, bpf_p4tc_entry_create_on_miss); +BTF_ID_FLAGS (func, bpf_p4tc_entry_update); +BTF_ID_FLAGS (func, bpf_p4tc_entry_delete); +BTF_SET8_END (p4tc_kfunc_check_tbl_set_skb) + +static const struct btf_kfunc_id_set p4tc_kfunc_tbl_set_skb = { + .owner =THIS_MODULE , + .set = &p4tc_kfunc_check_tbl_set_skb, +}; + +BTF_SET8_START (p4tc_kfunc_check_tbl_set_xdp) +BTF_ID_FLAGS (func, xdp_p4tc_tbl_read, KF_RET_NULL); +BTF_ID_FLAGS (func, xdp_p4tc_entry_create); +BTF_ID_FLAGS (func, xdp_p4tc_entry_create_on_miss); +BTF_ID_FLAGS (func, xdp_p4tc_entry_update); +BTF_ID_FLAGS (func, xdp_p4tc_entry_delete); +BTF_SET8_END (p4tc_kfunc_check_tbl_set_xdp)
Martin KaFai Lau
reviewerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 7 lines · Jamal Hadi Salim
Quoted 6 lines · Martin KaFai Lau
Quoted 1 line · Jamal Hadi Salim
+struct
p4tc_table_entry_act_bpf_params{Will this struct be extended in the future?
Quoted 8 lines
Quoted 4 lines
include/net/p4tc.h struct p4tc_pipeline { · line 109
Quoted 4 lines
include/net/p4tc.h +4
-0+u32 pipeid; +u32 tblid; +}; +Not that i can think of. We probably want to have the option to do so if needed. Do you see any harm if we were to make changes for whatever reason in the future?
It will be useful to add an argument named with "__sz" suffix to the kfunc.
Take a look at how the kfunc in nf_conntrack_bpf.c is handling the "opts" and "opts__sz" argument in its kfunc.
Quoted 136 lines
Quoted 68 lines
Quoted 7 lines
include/net/p4tc.h +6
-0+struct p4tc_table_entry_create_bpf_params { +u32 profile_id; +u32 pipeid; +u32 tblid; +}; +[ ... ]
Quoted 54 lines
include/net/tc_act/p4tc.h +35
-0@@ -13,10 +13,26 @@ struct tcf_p4act_params {u32 tot_params_sz; }; +#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );[ ... ]
Quoted 5 lines
include/net/tc_act/p4tc.h
Quoted 2 lines
include/net/tc_act/p4tc.h +2
-0+__bpf_kfunc static struct p4tc_table_entry_act_bpf * +bpf_p4tc_tbl_read(struct __sk_buff *skb_ctx,The argument could be "struct
sk_buff*skb" instead of__sk_buff. Take a look at commit2f4643934670.We'll make that change.
Quoted 50 lines
Quoted 50 lines
include/net/tc_act/p4tc.h +49
-0+struct p4tc_table_entry_act_bpf_params *params, + void *key, const u32 key__sz) +{ + struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *caller_net; + + caller_net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +__bpf_kfunc static struct p4tc_table_entry_act_bpf * +xdp_p4tc_tbl_read (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_act_bpf_params *params, +void *key,const u32 key__sz) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *caller_net; + + caller_net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ; + + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ;Quoted 9 lines
include/net/tc_act/p4tc.h
Quoted 3 lines
include/net/tc_act/p4tc.h +3
-0+ +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?
Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where the act_bpf_kern object resided) is gone.
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
[ ... ]
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
+BTF_SET8_START(p4tc_kfunc_check_tbl_set_skb)
This soon will be broken with the latest change in bpf-next. It is replaced by
BTF_KFUNCS_START. commita05e90427ef6.
It has already been included in the latest bpf-next pull-request, so should reach net-next soon.
Quoted 11 lines
Quoted 1 line
Ok, this wasnt in net-next when we pushed. We base our changes on net-next. When do you plan to merge that into net-next?
Quoted 2 lines
What is the plan on the selftest ?
We may need some guidance. How do you see us writing a selftest for this? We have extensive testing on the control side which is netlink (not part of the current series).
There are examples in tools/testing/selftests/bpf, e.g. the test_bpf_nf.c to test the kfuncs in nf_conntrack_bpf mentioned above. There are also selftests doing netlink to setup the test. The bpf/test_progs tries to avoid external dependency as much as possible, so linking to an extra external library and using an extra tool/binary will be unacceptable. and only the bpf/test_progs binary will be run by bpf CI.
The selftest does not have to be complicated. It can exercise the kfunc and show how the new struct (e.g. struct p4tc_table_entry_bpf_*) will be used. There is BPF_PROG_RUN for the tc and xdp prog, so should be quite doable.
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 18 lines · Martin KaFai Lau
Quoted 7 lines · Jamal Hadi Salim
Quoted 6 lines · Martin KaFai Lau
Quoted 1 line · Jamal Hadi Salim
+struct
p4tc_table_entry_act_bpf_params{Will this struct be extended in the future?
Quoted 8 lines
Quoted 4 lines
include/net/p4tc.h struct p4tc_pipeline { · line 109
Quoted 4 lines
include/net/p4tc.h +4
-0+u32 pipeid; +u32 tblid; +}; +Not that i can think of. We probably want to have the option to do so if needed. Do you see any harm if we were to make changes for whatever reason in the future?
Quoted 4 lines
It will be useful to add an argument named with "
__sz" suffix to the kfunc. Take a look at how the kfunc innf_conntrack_bpf.cis handling the "opts" and "opts__sz" argument in its kfunc.
Ok, will look.
Quoted 147 lines
Quoted 136 lines
Quoted 68 lines
Quoted 7 lines
include/net/p4tc.h +6
-0+struct p4tc_table_entry_create_bpf_params { +u32 profile_id; +u32 pipeid; +u32 tblid; +}; +[ ... ]
Quoted 54 lines
include/net/tc_act/p4tc.h +35
-0@@ -13,10 +13,26 @@ struct tcf_p4act_params {u32 tot_params_sz; }; +#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );[ ... ]
Quoted 5 lines
include/net/tc_act/p4tc.h
Quoted 2 lines
include/net/tc_act/p4tc.h +2
-0+__bpf_kfunc static struct p4tc_table_entry_act_bpf * +bpf_p4tc_tbl_read(struct __sk_buff *skb_ctx,The argument could be "struct
sk_buff*skb" instead of__sk_buff. Take a look at commit2f4643934670.We'll make that change.
Quoted 50 lines
Quoted 50 lines
include/net/tc_act/p4tc.h +49
-0+struct p4tc_table_entry_act_bpf_params *params, + void *key, const u32 key__sz) +{ + struct sk_buff *skb = (struct sk_buff *)skb_ctx; +struct net *caller_net; + + caller_net = skb->dev ?dev_net (skb->dev) :sock_net (skb->sk); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +__bpf_kfunc static struct p4tc_table_entry_act_bpf * +xdp_p4tc_tbl_read (struct xdp_md *xdp_ctx, +struct p4tc_table_entry_act_bpf_params *params, +void *key,const u32 key__sz) +{ +struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; +struct net *caller_net; + + caller_net =dev_net (ctx->rxq->dev); + +return __bpf_p4tc_tbl_read (caller_net, params, key, key__sz); +} + +static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ; + + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ;Quoted 9 lines
include/net/tc_act/p4tc.h
Quoted 3 lines
include/net/tc_act/p4tc.h +3
-0+ +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct
p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where theact_bpf_kernobject resided) is gone.
Quoted 4 lines
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
The table (residing on the TC side) could be shared by multiple bpf programs. Entries are allocated on the TC side of the fence. IOW, the memory is not owned by the bpf prog but rather by pipeline. We do have a "whodunnit" field, i.e we keep track of which entity added an entry and we are capable of deleting all entries when we detect a bpf program being deleted (this would be via deleting the tc filter). But my thinking is we should make that a policy decision as opposed to something which is default.
Quoted 10 lines
[ ... ]
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
+BTF_SET8_START(p4tc_kfunc_check_tbl_set_skb)
This soon will be broken with the latest change in bpf-next. It is replaced by
BTF_KFUNCS_START. commita05e90427ef6.It has already been included in the latest bpf-next pull-request, so should reach net-next soon.
Ok, we'll wait for it.
Quoted 11 lines
Quoted 3 lines
We may need some guidance. How do you see us writing a selftest for this? We have extensive testing on the control side which is netlink (not part of the current series).
There are examples in
tools/testing/selftests/bpf, e.g. thetest_bpf_nf.cto test the kfuncs innf_conntrack_bpfmentioned above. There are also selftests doing netlink to setup the test. The bpf/test_progs tries to avoid external dependency as much as possible, so linking to an extra external library and using an extra tool/binary will be unacceptable.
and only the bpf/test_progs binary will be run by bpf CI.
Quoted 3 lines
The selftest does not have to be complicated. It can exercise the kfunc and show how the new struct (e.g. struct p4tc_table_entry_bpf_*) will be used. There is
BPF_PROG_RUNfor the tc and xdp prog, so should be quite doable.
We will look into it.
Thanks for your feedback.
cheers,
jamal
Martin KaFai Lau
reviewerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 102 lines · Jamal Hadi Salim
Quoted 97 lines
Quoted 86 lines
Quoted 76 lines
Quoted 46 lines
include/net/tc_act/p4tc.h +35
-0+#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );[ ... ]
Quoted 25 lines
net/sched/p4tc/p4tc_bpf.c new file +25
-0+static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ; + + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ;Quoted 9 lines
net/sched/p4tc/p4tc_bpf.c line 131
Quoted 3 lines
net/sched/p4tc/p4tc_bpf.c new file +3
-0+ +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct
p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where theact_bpf_kernobject resided) is gone.Quoted 4 lines
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
Quoted 5 lines
The table (residing on the TC side) could be shared by multiple bpf programs. Entries are allocated on the TC side of the fence.
IOW, the memory is not owned by the bpf prog but rather by pipeline.
The struct p4tc_table_entry_act_(bpf_kern) object is allocated by bpf_p4tc_entry_create() kfunc and only bpf prog can use it, no? afaict, this is bpf objects.
Quoted 5 lines
We do have a "whodunnit" field, i.e we keep track of which entity added an entry and we are capable of deleting all entries when we detect a bpf program being deleted (this would be via deleting the tc filter). But my thinking is we should make that a policy decision as opposed to something which is default.
afaik, this policy decision or cleanup upon tc filter delete has not been done yet. I will leave it to you to figure out how to track what was allocated by a particular bpf prog on the TC side. It is not immediately clear to me and I probably won't have a good idea either.
Just to be clear that it is almost certain to be unacceptable to extend and make changes on the bpf side in the future to handle specific resource cleanup/tracking/sharing of the bpf objects allocated by these kfuncs. This problem has already been solved and works for different bpf program types, tc/cgroup/tracing...etc. Adding a refcnted bpf prog pointer alongside the act_bpf_kern object will be a non-starter.
I think multiple people have already commented that these kfuncs (create/update/delete...) resemble the existing bpf map. If these kfuncs are replaced with the bpf map ops, this bpf resource management has already been handled and will be consistent with other bpf program types.
I expect the act_bpf_kern object probably will grow in size over time also.
Considering this new p4 pipeline and table is residing on the TC side, I will leave it up to others to decide if it is acceptable to have some unused bpf objects left attached there.
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 109 lines · Martin KaFai Lau
Quoted 104 lines · Jamal Hadi Salim
Quoted 97 lines
Quoted 86 lines
Quoted 76 lines
Quoted 46 lines
include/net/tc_act/p4tc.h +35
-0+#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { +u32 act_id; +u32 hit:1 , + is_default_miss_act:1 , + is_default_hit_act:1 ; +u8 params[P4TC_MAX_PARAM_DATA_SIZE ]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { +struct rcu_head rcu; +struct p4tc_table_entry_act_bpf act_bpf; +}; +struct tcf_p4act {struct tc_action common;/* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; +struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf;u32 p_id;u32 act_id;struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act {#define to_p4act (a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf (struct tc_action *action) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =rcu_dereference (p4act->act_bpf); + +return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags (struct tc_action *action,u32 hit, +u32 dflt_miss,u32 dflt_hit) +{ +struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; +struct tcf_p4act *p4act =to_p4act (action); + + act_bpf =kzalloc (sizeof (*act_bpf),GFP_KERNEL );[ ... ]
Quoted 25 lines
net/sched/p4tc/p4tc_bpf.c new file +25
-0+static int +__bpf_p4tc_entry_create (struct net *net, +struct p4tc_table_entry_create_bpf_params *params, +void *key,const u32 key__sz, +struct p4tc_table_entry_act_bpf *act_bpf) +{ +struct p4tc_table_entry_key *entry_key = key; +struct p4tc_pipeline *pipeline; +struct p4tc_table *table; + +if (!params || !key) +return -EINVAL ; +if (key__sz !=P4TC_ENTRY_KEY_SZ_BYTES (entry_key->keysz)) +return -EINVAL ; + + pipeline =p4tc_pipeline_find_byid (net, params->pipeid); +if (!pipeline) +return -ENOENT ; + + table =p4tc_tbl_cache_lookup (net, params->pipeid, params->tblid); +if (!table) +return -ENOENT ; + +if (entry_key->keysz != table->tbl_keysz) +return -EINVAL ;Quoted 9 lines
net/sched/p4tc/p4tc_bpf.c line 131
Quoted 3 lines
net/sched/p4tc/p4tc_bpf.c new file +3
-0+ +return p4tc_table_entry_create_bpf (pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct
p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where theact_bpf_kernobject resided) is gone.Quoted 4 lines
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
The table (residing on the TC side) could be shared by multiple bpf programs. Entries are allocated on the TC side of the fence.
Quoted 6 lines
Quoted 1 line
IOW, the memory is not owned by the bpf prog but rather by pipeline.
The struct
p4tc_table_entry_act_(bpf_kern)object is allocated bybpf_p4tc_entry_create()kfunc and only bpf prog can use it, no? afaict, this is bpf objects.
Bear with me because i am not sure i am following.
When we looked at conntrack as guidance we noticed they do things slightly differently. They have an allocate kfunc and an insert function. If you have alloc then you need a complimentary release. The existence of the release in conntrack, correct me if i am wrong, seems to be based on the need to free the object if an insert fails. In our case the insert does first allocate then inserts all in one operation. If either fails it's not the concern of the bpf side to worry about it. IOW, i see the ownership as belonging to the P4TC side (it is both allocated, updated and freed by that side). Likely i am missing something..
Quoted 11 lines
Quoted 5 lines
We do have a "whodunnit" field, i.e we keep track of which entity added an entry and we are capable of deleting all entries when we detect a bpf program being deleted (this would be via deleting the tc filter). But my thinking is we should make that a policy decision as opposed to something which is default.
afaik, this policy decision or cleanup upon tc filter delete has not been done yet. I will leave it to you to figure out how to track what was allocated by a particular bpf prog on the TC side. It is not immediately clear to me and I probably won't have a good idea either.
I am looking at the conntrack code and i dont see how they release entries from the cotrack table when the bpf prog goes away.
Quoted 12 lines
Just to be clear that it is almost certain to be unacceptable to extend and make changes on the bpf side in the future to handle specific resource cleanup/tracking/sharing of the bpf objects allocated by these kfuncs. This problem has already been solved and works for different bpf program types, tc/cgroup/tracing...etc. Adding a refcnted bpf prog pointer alongside the
act_bpf_kernobject will be a non-starter.I think multiple people have already commented that these kfuncs (create/update/delete...) resemble the existing bpf map. If these kfuncs are replaced with the bpf map ops, this bpf resource management has already been handled and will be consistent with other bpf program types.
Quoted 4 lines
I expect the
act_bpf_kernobject probably will grow in size over time also. Considering this new p4 pipeline and table is residing on the TC side, I will leave it up to others to decide if it is acceptable to have some unused bpf objects left attached there.
There should be no dangling things at all.
Probably not a very good example, but this would be analogous to pinning a map which is shared by many bpf progs. Deleting one or all the bpf progs doesnt delete the contents of the bpf map, you have to explicitly remove it. Deleting the pipeline will be equivalent to deleting the map. IOW, resource cleanup is tied to the pipeline...
cheers,
jamal
Martin KaFai Lau
reviewerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 117 lines · Jamal Hadi Salim
Quoted 109 lines · Martin KaFai Lau
Quoted 104 lines · Jamal Hadi Salim
Quoted 97 lines
Quoted 86 lines
Quoted 76 lines
Quoted 46 lines
(quoted diff) +35
-0+#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { + u32 act_id; + u32 hit:1, + is_default_miss_act:1, + is_default_hit_act:1; + u8 params[P4TC_MAX_PARAM_DATA_SIZE]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { + struct rcu_head rcu; + struct p4tc_table_entry_act_bpf act_bpf; +}; + struct tcf_p4act { struct tc_action common; /* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; + struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf; u32 p_id; u32 act_id; struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act { #define to_p4act(a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf(struct tc_action *action) +{ + struct p4tc_table_entry_act_bpf_kern *act_bpf; + struct tcf_p4act *p4act = to_p4act(action); + + act_bpf = rcu_dereference(p4act->act_bpf); + + return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags(struct tc_action *action, u32 hit, + u32 dflt_miss, u32 dflt_hit) +{ + struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; + struct tcf_p4act *p4act = to_p4act(action); + + act_bpf = kzalloc(sizeof(*act_bpf), GFP_KERNEL);[ ... ]
Quoted 25 lines
(quoted diff) +25
-0+static int +__bpf_p4tc_entry_create(struct net *net, + struct p4tc_table_entry_create_bpf_params *params, + void *key, const u32 key__sz, + struct p4tc_table_entry_act_bpf *act_bpf) +{ + struct p4tc_table_entry_key *entry_key = key; + struct p4tc_pipeline *pipeline; + struct p4tc_table *table; + + if (!params || !key) + return -EINVAL; + if (key__sz != P4TC_ENTRY_KEY_SZ_BYTES(entry_key->keysz)) + return -EINVAL; + + pipeline = p4tc_pipeline_find_byid(net, params->pipeid); + if (!pipeline) + return -ENOENT; + + table = p4tc_tbl_cache_lookup(net, params->pipeid, params->tblid); + if (!table) + return -ENOENT; + + if (entry_key->keysz != table->tbl_keysz) + return -EINVAL;Quoted 9 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ + return p4tc_table_entry_create_bpf(pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct
p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where theact_bpf_kernobject resided) is gone.Quoted 4 lines
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
The table (residing on the TC side) could be shared by multiple bpf programs. Entries are allocated on the TC side of the fence.
Quoted 6 lines
Quoted 1 line
IOW, the memory is not owned by the bpf prog but rather by pipeline.
The struct
p4tc_table_entry_act_(bpf_kern)object is allocated bybpf_p4tc_entry_create()kfunc and only bpf prog can use it, no? afaict, this is bpf objects.
Quoted 11 lines
Bear with me because i am not sure i am following.
When we looked at conntrack as guidance we noticed they do things slightly differently. They have an allocate kfunc and an insert function. If you have alloc then you need a complimentary release. The existence of the release in conntrack, correct me if i am wrong, seems to be based on the need to free the object if an insert fails. In our case the insert does first allocate then inserts all in one operation. If either fails it's not the concern of the bpf side to worry about it. IOW, i see the ownership as belonging to the P4TC side (it is both allocated, updated and freed by that side). Likely i am missing something..
It is not the concern about the kfuncs may leak object.
I think my question was, who can use the act_bpf_kern object when all tc bpf prog is unloaded? If no one can use it, it should as well be cleaned up when the bpf prog is unloaded.
or the kernel p4 pipeline can use the act_bpf_kern object even when there is no bpf prog loaded?
Quoted 33 lines
Quoted 7 lines
Quoted 6 lines
We do have a "whodunnit" field, i.e we keep track of which entity added an entry and we are capable of deleting all entries when we detect a bpf program being deleted (this would be via deleting the tc filter). But my thinking is we should make that a policy decision as opposed to something which is default.
Quoted 5 lines
afaik, this policy decision or cleanup upon tc filter delete has not been done yet. I will leave it to you to figure out how to track what was allocated by a particular bpf prog on the TC side. It is not immediately clear to me and I probably won't have a good idea either.
I am looking at the conntrack code and i dont see how they release entries from the cotrack table when the bpf prog goes away.
Quoted 12 lines
Just to be clear that it is almost certain to be unacceptable to extend and make changes on the bpf side in the future to handle specific resource cleanup/tracking/sharing of the bpf objects allocated by these kfuncs. This problem has already been solved and works for different bpf program types, tc/cgroup/tracing...etc. Adding a refcnted bpf prog pointer alongside the
act_bpf_kernobject will be a non-starter.I think multiple people have already commented that these kfuncs (create/update/delete...) resemble the existing bpf map. If these kfuncs are replaced with the bpf map ops, this bpf resource management has already been handled and will be consistent with other bpf program types.
Quoted 4 lines
I expect the
act_bpf_kernobject probably will grow in size over time also. Considering this new p4 pipeline and table is residing on the TC side, I will leave it up to others to decide if it is acceptable to have some unused bpf objects left attached there.
Quoted 6 lines
There should be no dangling things at all.
Probably not a very good example, but this would be analogous to pinning a map which is shared by many bpf progs. Deleting one or all the bpf progs doesnt delete the contents of the bpf map, you have to explicitly remove it. Deleting the pipeline will be equivalent to deleting the map. IOW, resource cleanup is tied to the pipeline..
bpf is also used by many subsystems (e.g. tracing/cgroup/...). The bpf users have a common expectation on how bpf resources will be cleaned up when writing bpf for different subsystems, i.e. map/link/pinned-file. Thus, p4 pipeline is not the same as a pinned bpf map here. The p4-tc bpf user cannot depend on the common bpf ecosystem to cleanup all resources.
It is going back to how link/fd and the map ops discussion by others in the earlier revisions which we probably don't want to redo here. I think I have been making enough noise such that we don't have to discuss potential future changes about how to release this resources when the bpf prog is unloaded.
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 181 lines · Martin KaFai Lau
Quoted 117 lines · Jamal Hadi Salim
Quoted 109 lines · Martin KaFai Lau
Quoted 104 lines · Jamal Hadi Salim
Quoted 97 lines
Quoted 86 lines
Quoted 76 lines
Quoted 46 lines
(quoted diff) +35
-0+#define P4TC_MAX_PARAM_DATA_SIZE 124 + +struct p4tc_table_entry_act_bpf { + u32 act_id; + u32 hit:1, + is_default_miss_act:1, + is_default_hit_act:1; + u8 params[P4TC_MAX_PARAM_DATA_SIZE]; +} __packed; + +struct p4tc_table_entry_act_bpf_kern { + struct rcu_head rcu; + struct p4tc_table_entry_act_bpf act_bpf; +}; + struct tcf_p4act { struct tc_action common; /* Params IDR reference passed during runtime */ struct tcf_p4act_params __rcu *params; + struct p4tc_table_entry_act_bpf_kern __rcu *act_bpf; u32 p_id; u32 act_id; struct list_head node;@@ -24,4 +40,39 @@ struct tcf_p4act { #define to_p4act(a) ((struct tcf_p4act *)a) +static inline struct p4tc_table_entry_act_bpf * +p4tc_table_entry_act_bpf(struct tc_action *action) +{ + struct p4tc_table_entry_act_bpf_kern *act_bpf; + struct tcf_p4act *p4act = to_p4act(action); + + act_bpf = rcu_dereference(p4act->act_bpf); + + return &act_bpf->act_bpf; +} + +static inline int +p4tc_table_entry_act_bpf_change_flags(struct tc_action *action, u32 hit, + u32 dflt_miss, u32 dflt_hit) +{ + struct p4tc_table_entry_act_bpf_kern *act_bpf, *act_bpf_old; + struct tcf_p4act *p4act = to_p4act(action); + + act_bpf = kzalloc(sizeof(*act_bpf), GFP_KERNEL);[ ... ]
Quoted 25 lines
(quoted diff) +25
-0+static int +__bpf_p4tc_entry_create(struct net *net, + struct p4tc_table_entry_create_bpf_params *params, + void *key, const u32 key__sz, + struct p4tc_table_entry_act_bpf *act_bpf) +{ + struct p4tc_table_entry_key *entry_key = key; + struct p4tc_pipeline *pipeline; + struct p4tc_table *table; + + if (!params || !key) + return -EINVAL; + if (key__sz != P4TC_ENTRY_KEY_SZ_BYTES(entry_key->keysz)) + return -EINVAL; + + pipeline = p4tc_pipeline_find_byid(net, params->pipeid); + if (!pipeline) + return -ENOENT; + + table = p4tc_tbl_cache_lookup(net, params->pipeid, params->tblid); + if (!table) + return -ENOENT; + + if (entry_key->keysz != table->tbl_keysz) + return -EINVAL;Quoted 9 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ + return p4tc_table_entry_create_bpf(pipeline, table, entry_key, act_bpf, + params->profile_id);My understanding is this kfunc will allocate a "struct
p4tc_table_entry_act_bpf_kern" object. If thebpf_p4tc_entry_delete()kfunc is never called and the bpf prog is unloaded, how theact_bpfobject will be cleaned up?Quoted 4 lines
The TC code takes care of this. Unloading the bpf prog does not affect the deletion, it is the TC control side that will take care of it. If we delete the pipeline otoh then not just this entry but all entries will be flushed.
It looks like the "struct
p4tc_table_entry_act_bpf_kern" object is allocated by the bpf prog through kfunc and will only be useful for the bpf prog but not other parts of the kernel. However, if the bpf prog is unloaded, these bpf specific objects will be left over in the kernel until the tc pipeline (where theact_bpf_kernobject resided) is gone.Quoted 4 lines
It is the expectation on bpf prog (not only tc/xdp bpf prog) about resources clean up that these bpf objects will be gone after unloading the bpf prog and unpinning its bpf map.
The table (residing on the TC side) could be shared by multiple bpf programs. Entries are allocated on the TC side of the fence.
Quoted 6 lines
Quoted 1 line
IOW, the memory is not owned by the bpf prog but rather by pipeline.
The struct
p4tc_table_entry_act_(bpf_kern)object is allocated bybpf_p4tc_entry_create()kfunc and only bpf prog can use it, no? afaict, this is bpf objects.Quoted 11 lines
Bear with me because i am not sure i am following.
When we looked at conntrack as guidance we noticed they do things slightly differently. They have an allocate kfunc and an insert function. If you have alloc then you need a complimentary release. The existence of the release in conntrack, correct me if i am wrong, seems to be based on the need to free the object if an insert fails. In our case the insert does first allocate then inserts all in one operation. If either fails it's not the concern of the bpf side to worry about it. IOW, i see the ownership as belonging to the P4TC side (it is both allocated, updated and freed by that side). Likely i am missing something..It is not the concern about the kfuncs may leak object.
I think my question was, who can use the
act_bpf_kernobject when all tc bpf prog is unloaded? If no one can use it, it should as well be cleaned up when the bpf prog is unloaded.or the kernel p4 pipeline can use the
act_bpf_kernobject even when there is no bpf prog loaded?Quoted 33 lines
Quoted 7 lines
Quoted 6 lines
We do have a "whodunnit" field, i.e we keep track of which entity added an entry and we are capable of deleting all entries when we detect a bpf program being deleted (this would be via deleting the tc filter). But my thinking is we should make that a policy decision as opposed to something which is default.
Quoted 5 lines
afaik, this policy decision or cleanup upon tc filter delete has not been done yet. I will leave it to you to figure out how to track what was allocated by a particular bpf prog on the TC side. It is not immediately clear to me and I probably won't have a good idea either.
I am looking at the conntrack code and i dont see how they release entries from the cotrack table when the bpf prog goes away.
Quoted 12 lines
Just to be clear that it is almost certain to be unacceptable to extend and make changes on the bpf side in the future to handle specific resource cleanup/tracking/sharing of the bpf objects allocated by these kfuncs. This problem has already been solved and works for different bpf program types, tc/cgroup/tracing...etc. Adding a refcnted bpf prog pointer alongside the
act_bpf_kernobject will be a non-starter.I think multiple people have already commented that these kfuncs (create/update/delete...) resemble the existing bpf map. If these kfuncs are replaced with the bpf map ops, this bpf resource management has already been handled and will be consistent with other bpf program types.
Quoted 4 lines
I expect the
act_bpf_kernobject probably will grow in size over time also. Considering this new p4 pipeline and table is residing on the TC side, I will leave it up to others to decide if it is acceptable to have some unused bpf objects left attached there.Quoted 6 lines
There should be no dangling things at all.
Probably not a very good example, but this would be analogous to pinning a map which is shared by many bpf progs. Deleting one or all the bpf progs doesnt delete the contents of the bpf map, you have to explicitly remove it. Deleting the pipeline will be equivalent to deleting the map. IOW, resource cleanup is tied to the pipeline..
Quoted 6 lines
bpf is also used by many subsystems (e.g. tracing/cgroup/...). The bpf users have a common expectation on how bpf resources will be cleaned up when writing bpf for different subsystems, i.e. map/link/pinned-file. Thus, p4 pipeline is not the same as a pinned bpf map here. The p4-tc bpf user cannot depend on the common bpf ecosystem to cleanup all resources.
I am not trying to be difficult. Sincerely trying to understand and very puzzled - and it is not that we cant do what you are suggesting just trying to understand the reasoning to make sure it fits our requirements.
I asked earlier about conntrack (where we took the inspiration from):
How is what we are doing different from contrack? If you can help me understand that i am more than willing to make the change. Conntrack entries can be added via the kfunc(same for us). Contrack entries can also be added from the control plane and can be found by ebpf lookups(same for us). They can be deleted by the control plane, timers, entry evictions to make space for new entries, etc (same for us). Not sure if they can be deleted by ebpf side (we can). Perusing the conntrack code, I could not find anything that indicated that entries created from ebpf are deleted when the ebpf program goes away.
To re-emphasize: Maybe there's something subtle i am missing that we are not doing that conntrack is doing?
Conntrack does one small thing we dont: It allocs and returns to ebpf the memory for insertion. I dont see that as particularly useful for our case (and more importantly how that results in the entries being deleted when the ebpf prog goes away)
cheers,
jamal
Quoted 4 lines
It is going back to how link/fd and the map ops discussion by others in the earlier revisions which we probably don't want to redo here. I think I have been making enough noise such that we don't have to discuss potential future changes about how to release this resources when the bpf prog is unloaded.
Martin KaFai Lau
reviewerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 6 lines · Jamal Hadi Salim
Quoted 6 lines
I think my question was, who can use the
act_bpf_kernobject when all tc bpf prog is unloaded? If no one can use it, it should as well be cleaned up when the bpf prog is unloaded.or the kernel p4 pipeline can use the
act_bpf_kernobject even when there is no bpf prog loaded?
[ ... ]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I am looking at the conntrack code and i dont see how they release entries from the cotrack table when the bpf prog goes away.
[ ... ]
Quoted 11 lines
I asked earlier about conntrack (where we took the inspiration from):
How is what we are doing different from contrack? If you can help me understand that i am more than willing to make the change. Conntrack entries can be added via the kfunc(same for us). Contrack entries can also be added from the control plane and can be found by ebpf lookups(same for us). They can be deleted by the control plane, timers, entry evictions to make space for new entries, etc (same for us). Not sure if they can be deleted by ebpf side (we can). Perusing the conntrack code, I could not find anything that indicated that entries created from ebpf are deleted when the ebpf program goes away.
Quoted 6 lines
To re-emphasize: Maybe there's something subtle i am missing that we are not doing that conntrack is doing?
Conntrack does one small thing we dont: It allocs and returns to ebpf the memory for insertion. I dont see that as particularly useful for our case (and more importantly how that results in the entries being deleted when the ebpf prog goes away)
afaik, the conntrack kfunc inserts "struct nf_conn" that can also be used by other kernel parts, so it is reasonable to go through the kernel existing eviction logic. It is why my earlier question on "is the act_bpf_kern object only useful for the bpf prog alone but not other kernel parts". From reading patch 14, it seems to be only usable by bpf prog. When all bpf program is unloaded, who will still read it and do something useful? If I mis-understood it, this will be useful to capture in the commit message to explain how it could be used by other kernel parts without bpf prog running.
Jamal Hadi Salim
maintainerRe: [PATCH net-next v12 14/15] p4tc: add set of P4TC table kfuncs
Quoted 34 lines · Martin KaFai Lau
Quoted 6 lines · Jamal Hadi Salim
Quoted 6 lines
I think my question was, who can use the
act_bpf_kernobject when all tc bpf prog is unloaded? If no one can use it, it should as well be cleaned up when the bpf prog is unloaded.or the kernel p4 pipeline can use the
act_bpf_kernobject even when there is no bpf prog loaded?[ ... ]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I am looking at the conntrack code and i dont see how they release entries from the cotrack table when the bpf prog goes away.
[ ... ]
Quoted 11 lines
I asked earlier about conntrack (where we took the inspiration from):
How is what we are doing different from contrack? If you can help me understand that i am more than willing to make the change. Conntrack entries can be added via the kfunc(same for us). Contrack entries can also be added from the control plane and can be found by ebpf lookups(same for us). They can be deleted by the control plane, timers, entry evictions to make space for new entries, etc (same for us). Not sure if they can be deleted by ebpf side (we can). Perusing the conntrack code, I could not find anything that indicated that entries created from ebpf are deleted when the ebpf program goes away.Quoted 6 lines
To re-emphasize: Maybe there's something subtle i am missing that we are not doing that conntrack is doing?
Conntrack does one small thing we dont: It allocs and returns to ebpf the memory for insertion. I dont see that as particularly useful for our case (and more importantly how that results in the entries being deleted when the ebpf prog goes away)
Quoted 8 lines
afaik, the conntrack kfunc inserts "struct
nf_conn" that can also be used by other kernel parts, so it is reasonable to go through the kernel existing eviction logic. It is why my earlier question on "is theact_bpf_kernobject only useful for the bpf prog alone but not other kernel parts". From reading patch 14, it seems to be only usable by bpf prog. When all bpf program is unloaded, who will still read it and do something useful? If I mis-understood it, this will be useful to capture in the commit message to explain how it could be used by other kernel parts without bpf prog running.
Ok, I think i may have got the issue. Sigh. I didnt do a good job explaining p4tc_table_entry_act_bpf_kern which has been the crux of our back and forth. Sorry I know you said this several times and i was busy describing things around it instead.
A multiple of these structure p4tc_table_entry_act_bpf_kern arepreallocated(to match the P4 architecture, patch #9 describes some of the subtleties involved) by the p4tc control plane and put in a kernel pool. Their purpose is to hold the action parameters that are returned to ebpf when there is a successful table lookup. When the table entry is deleted the act_bpf_kern is recycled to a pool to be reused for the next table entry. The only time the pool memory is released is when the pipeline is deleted. So it is not allocated via the kfunc at all.
I am not sure if that helps, if it does and you feel it should go in the commit message we can do that. If not, please a little more patience with me..
cheers,
jamal
Jamal Hadi Salim
maintainerPATCH v12 15/15 p4tc: add P4 classifier
In reply to Jamal Hadi SalimIntroduce P4 tc classifier. The main task of this classifier is to manage the lifetime of pipeline instances across one or more netdev ports.
Note: a defined pipeline may be instantiated multiple times across one or more tc chains and different priorities.
Note that part or whole of the P4 pipeline could reside in tc, XDP or even hardware depending on how the P4 program was compiled. This classifier only deals with tc layer.
To use the P4 classifier you must specify a pipeline name that will be associated to the filter instance, a s/w parser (eBPF) and datapath P4 control block program (eBPF) program. Although this patchset does not deal with offloads, it is also possible to load the h/w part using this filter. We will illustrate a few examples further below to clarify. Please treat the illustrated split as an example - there are probably more pragmatic approaches to splitting the pipeline; however, regardless of where the different pieces of the pipeline are placed (tc, XDP, HW) and what each layer will implement (what part of the pipeline) - these examples are merely showing what is possible.
The pipeline is assumed to have already been created via a template.
For example, if we were to add a filter to ingress of a group of netdevs (tc block 22) and associate it to P4 pipeline simple_l3 we could issue the following command:
tc filter add block 22 parent ffff: protocol all prio 6 p4 pname simple_l3 \
action bpf obj $PARSER.o ... \
action bpf obj $PROGNAME.o section p4tc/main
The above uses the classical tc action mechanism in which the first action runs the P4 parser and if that goes well then the P4 control block is executed. Note, although not shown above, one could also append the command line with other traditional tc actions.
Given one of the objectives of this classifier is to manage the lifetime of the p4 program and said program may be split across tc:xdp:hardware we allow specification of where the xdp (and in the future hardware) programs can be found. For this reason when instantiating one could specify where the associated XDP program using they syntax "prog type xdp progname", where progname refers to the XDP ebpf program name. The control plane side (below we show iproute2) will be responsible for loading the XDP program. The kernel is unaware of the XDP side.
There is an ongoing discussion in the P4TC community biweekly meetings which is likely going to have us add another location definition "prog type hw" which will specify the hardware object file name and other related attributes. The current discussion is that this h/w piece will go via the p4 classifier.
An example using xdp and tc:
tc filter add dev $P0 ingress protocol all prio 1 p4 pname simple_l3 \
prog type xdp obj $PARSER.o section p4tc/parse-xdp \
action bpf obj $PROGNAME.o section p4tc/main
In this case, the parser will be executed in the XDP layer and the rest of P4 control block as a tc action.
For illustration sake, the hw one looks as follows (please note there's still a lot of discussions going on in the meetings - the example is here merely to illustrate the tc filter functionality):
tc filter add block 22 ingress protocol all prio 1 p4 pname simple_l3 \
prog type hw filename "mypnameprog.o" ... \
prog type xdp obj $PARSER.o section p4tc/parse-xdp \
action bpf obj $PROGNAME.o section p4tc/main
The theory of operations is as follows:
================================1. PARSING================================
The packet first encounters the parser.
The parser is implemented in ebpf residing either at the TC or XDP level. The parsed header values are stored in a shared per-cpu eBPF map. When the parser runs at XDP level, we load it into XDP using the control plane (tc filter command) and pin it to a file.
=============================2. ACTIONS=============================
In the above example, the P4 program (minus the parser) is encoded in an action($PROGNAME.o). It should be noted that classical tc actions continue to work:
IOW, someone could decide to add a mirred action to mirror all packets after or before the ebpf action.
tc filter add dev $P0 parent ffff: protocol all prio 6 p4 pname simple_l3 \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main \
action mirred egress mirror index 1 dev $P1 \
action bpf obj $ANOTHERPROG.o section mysect/section-1It should also be noted that it is feasible to split some of the ingress datapath into XDP first and more into TC later (as was shown above for example where the parser runs at XDP level). YMMV. Regardless of choice of which scheme to use, none of these will affect UAPI. It will all depend on whether you generate code to load on XDP vs tc, etc. We expect the compiler to evolve over time (but that has nothing to do with the kernel part).
--- include/uapi/linux/pkt_cls.h | 14 ++ net/sched/Kconfig | 12 ++ net/sched/Makefile | 1 + net/sched/cls_p4.c | 305 +++++++++++++++++++++++++++++++++++ net/sched/p4tc/Makefile | 4 +- net/sched/p4tc/trace.c | 10 ++ net/sched/p4tc/trace.h | 44 +++++ 7 files changed, 389 insertions(+), 1 deletion(-) create mode 100644 net/sched/cls_p4.c create mode 100644 net/sched/p4tc/trace.c create mode 100644 net/sched/p4tc/trace.h
include/uapi/linux/pkt_cls.h +14 -0
@@ -692,6 +692,20 @@ enum {#define TCA_MATCHALL_MAX (__TCA_MATCHALL_MAX - 1) +/* P4 classifier */ + +enum { +TCA_P4_UNSPEC , +TCA_P4_CLASSID , +TCA_P4_ACT , +TCA_P4_PNAME , +TCA_P4_PIPEID , +TCA_P4_PAD , + __TCA_P4_MAX, +}; + +#define TCA_P4_MAX (__TCA_P4_MAX - 1) +/* Extended Matches */ struct tcf_ematch_tree_hdr {
net/sched/Kconfig +12 -0
@@ -565,6 +565,18 @@ config NET_CLS_MATCHALL To compile this code as a module, choose M here: the module will be called cls_matchall. +config NET_CLS_P4 + tristate "P4 classifier" + select NET_CLS + select NET_P4TC + help + If you say Y here, you will be able to bind a P4 pipeline + program. You will need to install a P4 template representing the + program successfully to use this feature. + + To compile this code as a module, choose M here: the module will + be called cls_p4. + config NET_EMATCH bool "Extended Matches" select NET_CLS
net/sched/Makefile +1 -0
@@ -72,6 +72,7 @@ obj-$(CONFIG_NET_CLS_CGROUP) += cls_cgroup.o obj-$(CONFIG_NET_CLS_BPF) += cls_bpf.o obj-$(CONFIG_NET_CLS_FLOWER) += cls_flower.o obj-$(CONFIG_NET_CLS_MATCHALL) += cls_matchall.o +obj-$(CONFIG_NET_CLS_P4) += cls_p4.o obj-$(CONFIG_NET_EMATCH) += ematch.o obj-$(CONFIG_NET_EMATCH_CMP) += em_cmp.o obj-$(CONFIG_NET_EMATCH_NBYTE) += em_nbyte.o
net/sched/cls_p4.c new file +305 -0
305 lines not on this page · raw message
net/sched/p4tc/Makefile +3 -1
@@ -1,6 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 +CFLAGS_trace.o := -I$(src) + obj-y := p4tc_types.o p4tc_tmpl_api.o p4tc_pipeline.o \ p4tc_action.o p4tc_table.o p4tc_tbl_entry.o \ - p4tc_filter.o p4tc_runtime_api.o + p4tc_filter.o p4tc_runtime_api.o trace.o obj-$(CONFIG_DEBUG_INFO_BTF) += p4tc_bpf.o
net/sched/p4tc/trace.c new file +10 -0
@@ -0,0 +1,10 @@ +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) + +#include <net/p4tc.h> + +#ifndef __CHECKER__ + +#define CREATE_TRACE_POINTS +#include "trace.h" +EXPORT_TRACEPOINT_SYMBOL_GPL (p4_classify); +#endif
net/sched/p4tc/trace.h new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 44 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
Quoted 8 lines
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
[...]
Although I appreciate a good amount of work went into building above I'll add my concerns here so they are not lost. These are architecture concerns not this line of code needs some tweak.
- It encodes a DSL into the kernel. Its unclear how we pick which DSL gets pushed into the kernel and which do not. Do we take any DSL folks can code up?
I would prefer a lower level intermediate langauge. My view is this is a lesson we should have learned from OVS. OVS had wider adoption and still struggled in some ways my belief is this is very similar to OVS. (Also OVS was novel/great at a lot of things fwiw.)
- We have a general purpose language in BPF that can implement the P4 DSL already. I don't see any need for another set of code when the end goal is running P4 in Linux network stack is doable. Typically we reject duplicate things when they don't have concrete benefits.
- P4 as a DSL is not optimized for general purpose CPUs, but rather hardware pipelines. Although it can be optimized for CPUs its a harder problem. A review of some of the VPP/DPDK work here is useful.
- P4 infrastructure already has a p4c backend this is adding another P4 backend instead of getting the rather small group of people to work on a single backend we are now creating another one.
- Common reasons I think would justify a new P4 backend and implementation would be: speed efficiency, or expressiveness. I think this implementation is neither more efficient nor more expressive. Concrete examples on expressiveness would be interesting, but I don't see any. Loops were mentioned once but latest kernels have loop support.
- The main talking point for many slide decks about p4tc is hardware offload. This seems like the main benefit of pushing the P4 DSL into the kernel. But, we have no hw implementation, not even a vendor stepping up to comment on this implementation and how it will work for them. HW introduces all sorts of interesting problems that I don't see how we solve in this framework. For example a few off the top of my head:
syncing current state into tc, how does operator program tc inside constraints, who writes the p4 models for these hardware devices, do they fit into this 'tc' infrastructure, partial updates into hardware seems unlikely to work for most hardware, ...
- The kfuncs are mostly duplicates of map ops we already have in BPF API. The motivation by my read is to use netlink instead of bpf commands. I don't agree with this, optimizing for some low level debug a developer uses is the wrong design space. Actual users should not be deploying this via ssh into boxes. The workflow will not scale and really we need tooling and infra to land P4 programs across the network. This is orders of more pain if its an endpoint solution and not a middlebox/switch solution. As a switch solution I don't see how p4tc sw scales to even TOR packet rates. So you need tooling on top and user interact with the tooling not the Linux widget/debugger at the bottom.
- There is no performance analysis: The comment was functionality before performance which I disagree with. If it was a first implementation and we didn't have a way to do P4 DSL already than I might agree, but here we have an existing solution so it should be at least as good and should be better than existing backend. A software datapath adoption is going to be critically based on performance. I don't see taking even a 5% hit when porting over to P4 from existing datapath.
Commentary: I think its 100% correct to debate how the P4 DSL is implemented in the kernel. I can't see why this is off limits somehow this patch set proposes an approach there could be many approaches. BPF comes up not because I'm some BPF zealot that needs P4 DSL in BPF, but because it exists today there is even a P4 backend. Fundamentally I don't see the value add we get by creating two P4 pipelines this is going to create duplication all the way up to the P4 tooling/infra through to the kernel. From your side you keep saying I'm bike shedding and demanding BPF, but from my perspective your introducing another entire toolchain simply because you want some low level debug commands that 99% of P4 users should not be using or caring about.
To try and be constructive some things that would change my mind would be a vendor showing how hardware can be used. This would be compelling. Or performance showing its somehow gets a more performant implementation. Or lastly if the current p4c implementation is fundamentally broken somehow.
Thanks
John
Jamal Hadi Salim
maintainerQuoted 130 lines · John Fastabend
Quoted 44 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
Quoted 8 lines
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
[...]
Although I appreciate a good amount of work went into building above I'll add my concerns here so they are not lost. These are architecture concerns not this line of code needs some tweak.
- It encodes a DSL into the kernel. Its unclear how we pick which DSL gets pushed into the kernel and which do not. Do we take any DSL folks can code up?
I would prefer a lower level intermediate langauge. My view is this is a lesson we should have learned from OVS. OVS had wider adoption and still struggled in some ways my belief is this is very similar to OVS. (Also OVS was novel/great at a lot of things fwiw.)- We have a general purpose language in BPF that can implement the P4 DSL already. I don't see any need for another set of code when the end goal is running P4 in Linux network stack is doable. Typically we reject duplicate things when they don't have concrete benefits.
- P4 as a DSL is not optimized for general purpose CPUs, but rather hardware pipelines. Although it can be optimized for CPUs its a harder problem. A review of some of the VPP/DPDK work here is useful.
- P4 infrastructure already has a p4c backend this is adding another P4 backend instead of getting the rather small group of people to work on a single backend we are now creating another one.
- Common reasons I think would justify a new P4 backend and implementation would be: speed efficiency, or expressiveness. I think this implementation is neither more efficient nor more expressive. Concrete examples on expressiveness would be interesting, but I don't see any. Loops were mentioned once but latest kernels have loop support.
- The main talking point for many slide decks about p4tc is hardware offload. This seems like the main benefit of pushing the P4 DSL into the kernel. But, we have no hw implementation, not even a vendor stepping up to comment on this implementation and how it will work for them. HW introduces all sorts of interesting problems that I don't see how we solve in this framework. For example a few off the top of my head:
syncing current state into tc, how does operator program tc inside constraints, who writes the p4 models for these hardware devices, do they fit into this 'tc' infrastructure, partial updates into hardware seems unlikely to work for most hardware, ...- The kfuncs are mostly duplicates of map ops we already have in BPF API. The motivation by my read is to use netlink instead of bpf commands. I don't agree with this, optimizing for some low level debug a developer uses is the wrong design space. Actual users should not be deploying this via ssh into boxes. The workflow will not scale and really we need tooling and infra to land P4 programs across the network. This is orders of more pain if its an endpoint solution and not a middlebox/switch solution. As a switch solution I don't see how p4tc sw scales to even TOR packet rates. So you need tooling on top and user interact with the tooling not the Linux widget/debugger at the bottom.
- There is no performance analysis: The comment was functionality before performance which I disagree with. If it was a first implementation and we didn't have a way to do P4 DSL already than I might agree, but here we have an existing solution so it should be at least as good and should be better than existing backend. A software datapath adoption is going to be critically based on performance. I don't see taking even a 5% hit when porting over to P4 from existing datapath.
Commentary: I think its 100% correct to debate how the P4 DSL is implemented in the kernel. I can't see why this is off limits somehow this patch set proposes an approach there could be many approaches. BPF comes up not because I'm some BPF zealot that needs P4 DSL in BPF, but because it exists today there is even a P4 backend. Fundamentally I don't see the value add we get by creating two P4 pipelines this is going to create duplication all the way up to the P4 tooling/infra through to the kernel. From your side you keep saying I'm bike shedding and demanding BPF, but from my perspective your introducing another entire toolchain simply because you want some low level debug commands that 99% of P4 users should not be using or caring about.
Quoted 6 lines
To try and be constructive some things that would change my mind would be a vendor showing how hardware can be used. This would be compelling. Or performance showing its somehow gets a more performant implementation. Or lastly if the current p4c implementation is fundamentally broken somehow.
John,
With all due respect we are going back again over the same points, recycled many times over to which i have responded to you many times. It's gettting tiring. This is exactly why i called it bikeshedding. Let's just agree to disagree.
cheers,
jamal
Quoted 2 lines
Thanks
John
Quoted 139 lines · Jamal Hadi Salim
Quoted 130 lines · John Fastabend
Quoted 44 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
Quoted 8 lines
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
[...]
Although I appreciate a good amount of work went into building above I'll add my concerns here so they are not lost. These are architecture concerns not this line of code needs some tweak.
- It encodes a DSL into the kernel. Its unclear how we pick which DSL gets pushed into the kernel and which do not. Do we take any DSL folks can code up?
I would prefer a lower level intermediate langauge. My view is this is a lesson we should have learned from OVS. OVS had wider adoption and still struggled in some ways my belief is this is very similar to OVS. (Also OVS was novel/great at a lot of things fwiw.)- We have a general purpose language in BPF that can implement the P4 DSL already. I don't see any need for another set of code when the end goal is running P4 in Linux network stack is doable. Typically we reject duplicate things when they don't have concrete benefits.
- P4 as a DSL is not optimized for general purpose CPUs, but rather hardware pipelines. Although it can be optimized for CPUs its a harder problem. A review of some of the VPP/DPDK work here is useful.
- P4 infrastructure already has a p4c backend this is adding another P4 backend instead of getting the rather small group of people to work on a single backend we are now creating another one.
- Common reasons I think would justify a new P4 backend and implementation would be: speed efficiency, or expressiveness. I think this implementation is neither more efficient nor more expressive. Concrete examples on expressiveness would be interesting, but I don't see any. Loops were mentioned once but latest kernels have loop support.
- The main talking point for many slide decks about p4tc is hardware offload. This seems like the main benefit of pushing the P4 DSL into the kernel. But, we have no hw implementation, not even a vendor stepping up to comment on this implementation and how it will work for them. HW introduces all sorts of interesting problems that I don't see how we solve in this framework. For example a few off the top of my head:
syncing current state into tc, how does operator program tc inside constraints, who writes the p4 models for these hardware devices, do they fit into this 'tc' infrastructure, partial updates into hardware seems unlikely to work for most hardware, ...- The kfuncs are mostly duplicates of map ops we already have in BPF API. The motivation by my read is to use netlink instead of bpf commands. I don't agree with this, optimizing for some low level debug a developer uses is the wrong design space. Actual users should not be deploying this via ssh into boxes. The workflow will not scale and really we need tooling and infra to land P4 programs across the network. This is orders of more pain if its an endpoint solution and not a middlebox/switch solution. As a switch solution I don't see how p4tc sw scales to even TOR packet rates. So you need tooling on top and user interact with the tooling not the Linux widget/debugger at the bottom.
- There is no performance analysis: The comment was functionality before performance which I disagree with. If it was a first implementation and we didn't have a way to do P4 DSL already than I might agree, but here we have an existing solution so it should be at least as good and should be better than existing backend. A software datapath adoption is going to be critically based on performance. I don't see taking even a 5% hit when porting over to P4 from existing datapath.
Commentary: I think its 100% correct to debate how the P4 DSL is implemented in the kernel. I can't see why this is off limits somehow this patch set proposes an approach there could be many approaches. BPF comes up not because I'm some BPF zealot that needs P4 DSL in BPF, but because it exists today there is even a P4 backend. Fundamentally I don't see the value add we get by creating two P4 pipelines this is going to create duplication all the way up to the P4 tooling/infra through to the kernel. From your side you keep saying I'm bike shedding and demanding BPF, but from my perspective your introducing another entire toolchain simply because you want some low level debug commands that 99% of P4 users should not be using or caring about.
Quoted 6 lines
To try and be constructive some things that would change my mind would be a vendor showing how hardware can be used. This would be compelling. Or performance showing its somehow gets a more performant implementation. Or lastly if the current p4c implementation is fundamentally broken somehow.
Quoted 5 lines
John,
With all due respect we are going back again over the same points, recycled many times over to which i have responded to you many times. It's gettting tiring. This is exactly why i called it bikeshedding. Let's just agree to disagree.
Yep we agree to disagree and I put them them as a summary so others can see them and think it over/decide where they stand on it. In the end you don't need my ACK here, but I wanted my opinion summarized.
Quoted 6 lines
cheers,
jamalQuoted 2 lines
Thanks
John
Martin KaFai Lau
reviewerQuoted 2 lines · John Fastabend
- The kfuncs are mostly duplicates of map ops we already have in BPF API. The motivation by my read is to use netlink instead of bpf commands. I
I also have similar thought on the kfuncs (create/update/delete) which is mostly bpf map ops. It could have one single kfunc to allocate a kernel specific p4 entry/object and then store that in a bpf map. With the bpf_rbtree, bpf_list, and other recent advancements, it should be able to describe them in a bpf map. The reply in v9 was that the p4 table will also be used in the future HW piece/driver but the HW piece is not ready yet, bpf is the only consumer of the kernel p4 table now and this makes mimicking the bpf map api to kfuncs not convincing. bpf "tc / xdp" program uses netlink to attach/detach and the policy also stays in the bpf map.
When there is a HW piece that consumes the p4 table, that will be a better time to discuss the kfunc interface.
Quoted 8 lines
don't agree with this, optimizing for some low level debug a developer uses is the wrong design space. Actual users should not be deploying this via ssh into boxes. The workflow will not scale and really we need tooling and infra to land P4 programs across the network. This is orders of more pain if its an endpoint solution and not a middlebox/switch solution. As a switch solution I don't see how p4tc sw scales to even TOR packet rates. So you need tooling on top and user interact with the tooling not the Linux widget/debugger at the bottom.
Jamal Hadi Salim
maintainerQuoted 5 lines · Martin KaFai Lau
Quoted 2 lines · John Fastabend
- The kfuncs are mostly duplicates of map ops we already have in BPF API. The motivation by my read is to use netlink instead of bpf commands. I
Quoted 10 lines
I also have similar thought on the kfuncs (create/update/delete) which is mostly bpf map ops. It could have one single kfunc to allocate a kernel specific p4 entry/object and then store that in a bpf map. With the
bpf_rbtree,bpf_list, and other recent advancements, it should be able to describe them in a bpf map. The reply in v9 was that the p4 table will also be used in the future HW piece/driver but the HW piece is not ready yet, bpf is the only consumer of the kernel p4 table now and this makes mimicking the bpf map api to kfuncs not convincing. bpf "tc / xdp" program uses netlink to attach/detach and the policy also stays in the bpf map.
It's a lot more complex than just attaching/detaching. Our control plane uses netlink (regardless of whether it is offloaded or not) for all object controls (not just table entries) for the many reasons that have been stated in the cover letters since the beginning. I unfortunately took out some of the text after v10 to try and shorten the text. I will be adding it back. If you cant find it i could cutnpaste and send privately.
cheers,
jamal
Quoted 12 lines
When there is a HW piece that consumes the p4 table, that will be a better time to discuss the kfunc interface.
Quoted 8 lines
don't agree with this, optimizing for some low level debug a developer uses is the wrong design space. Actual users should not be deploying this via ssh into boxes. The workflow will not scale and really we need tooling and infra to land P4 programs across the network. This is orders of more pain if its an endpoint solution and not a middlebox/switch solution. As a switch solution I don't see how p4tc sw scales to even TOR packet rates. So you need tooling on top and user interact with the tooling not the Linux widget/debugger at the bottom.
Paolo Abeni
maintainerQuoted 472 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware
acquisition(example [12]).Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC, with these patches, provides
_kerneland user space code changeindependence_. Meaning:
A P4 program describes headers, parsers, etc alongside the datapath processing; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
__P4TCArchitecture__The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch description later..
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TCWorkflow__The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above..
b) the parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TCRuntime ControlPath__The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+)Read(</path/to/object>, [optional filter])Update(</path/to/object>, DATA>+)Delete(</path/to/object>, [optional filter])Subscribe(</path/to/object>, [optional filter])Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted. "Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1
tc p4ctrl create myprog/table/mytable \ dstAddr 10.0.1.2/32 action send_to_port param port eno1..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 actionsend_to_portparam port eno1 \
entry dstAddr 10.1.10.2/32 actionsend_to_portparam port eno10 \
entry dstAddr 10.0.2.2/32 actionsend_to_portparam port eno2..Get an entry (note "read" is interchangeably used as "get" which is a common
semantic in tc):tc p4ctrl read myprog/table/mytable \ dstAddr 10.0.2.2/32..dump mytable
tc p4ctrl read myprog/table/mytable..dump mytable for all entries whose key fits within 10.1.0.0/16
tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16..dump all mytable entries which have an action
send_to_portwith param "eno1"
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \ filter param/act/myprog/send_to_port/port = "eno1" && \ key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filtermsecs_since< 10000Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "
send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types). All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TCDatapath__The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address in labelled "dstAddr" above).
The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath).
Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__References__[1]https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf [2]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#historical-perspective-for-p4tc [3]https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation [4]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#so-why-p4-and-how-does-p4-help-here [5]https://lore.kernel.org/netdev/[email protected]/T/#mf59be7abc5df3473cff3879c8cc3e2369c0640a6 [6]https://lore.kernel.org/netdev/[email protected]/T/#m783cfd79e9d755cf0e7afc1a7d5404635a5b1919 [7]https://lore.kernel.org/netdev/[email protected]/T/#ma8c84df0f7043d17b98f3d67aab0f4904c600469 [8]https://github.com/p4lang/p4c/tree/main/backends/tc
[9]https://p4.org/ [10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
[12]https://github.com/sonic-net/DASH/tree/main
[13]https://github.com/p4lang/p4c/tree/main/backends/ebpf [14]https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html [15]https://dl.acm.org/doi/10.1145/3630047.3630193 [16]https://lore.kernel.org/netdev/[email protected]/ [17.a]https://netdevconf.info/0x13/session.html?talk-tc-u-classifier
[17.b]man tc-u32
[18]man tc-pedit
[19] https://lore.kernel.org/netdev/[email protected]/T/#m86e71743d1d83b728bb29d5b877797cb4942e835
[20.a] https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html
[20.b] https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html--------
HISTORY
--------Changes in Version 12
----------------------0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to
p4tc_xxxfor uapi andp4a_xxxfor actions to provide consistency(Jiri).4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.
Quoted 5 lines
The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via
Kconfig).
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Thanks,
Paolo
Jamal Hadi Salim
maintainerQuoted 480 lines · Paolo Abeni
Quoted 472 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware
acquisition(example [12]).Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC, with these patches, provides
_kerneland user space code changeindependence_. Meaning:
A P4 program describes headers, parsers, etc alongside the datapath processing; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
__P4TCArchitecture__The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch description later..
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TCWorkflow__The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above..
b) the parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TCRuntime ControlPath__The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+)Read(</path/to/object>, [optional filter])Update(</path/to/object>, DATA>+)Delete(</path/to/object>, [optional filter])Subscribe(</path/to/object>, [optional filter])Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted. "Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1
tc p4ctrl create myprog/table/mytable \ dstAddr 10.0.1.2/32 action send_to_port param port eno1..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 actionsend_to_portparam port eno1 \
entry dstAddr 10.1.10.2/32 actionsend_to_portparam port eno10 \
entry dstAddr 10.0.2.2/32 actionsend_to_portparam port eno2..Get an entry (note "read" is interchangeably used as "get" which is a common
semantic in tc):tc p4ctrl read myprog/table/mytable \ dstAddr 10.0.2.2/32..dump mytable
tc p4ctrl read myprog/table/mytable..dump mytable for all entries whose key fits within 10.1.0.0/16
tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16..dump all mytable entries which have an action
send_to_portwith param "eno1"
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \ filter param/act/myprog/send_to_port/port = "eno1" && \ key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filtermsecs_since< 10000Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "
send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types). All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TCDatapath__The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address in labelled "dstAddr" above).
The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath).
Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__References__[1]https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf [2]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#historical-perspective-for-p4tc [3]https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation [4]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#so-why-p4-and-how-does-p4-help-here [5]https://lore.kernel.org/netdev/[email protected]/T/#mf59be7abc5df3473cff3879c8cc3e2369c0640a6 [6]https://lore.kernel.org/netdev/[email protected]/T/#m783cfd79e9d755cf0e7afc1a7d5404635a5b1919 [7]https://lore.kernel.org/netdev/[email protected]/T/#ma8c84df0f7043d17b98f3d67aab0f4904c600469 [8]https://github.com/p4lang/p4c/tree/main/backends/tc
[9]https://p4.org/ [10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
[12]https://github.com/sonic-net/DASH/tree/main
[13]https://github.com/p4lang/p4c/tree/main/backends/ebpf [14]https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html [15]https://dl.acm.org/doi/10.1145/3630047.3630193 [16]https://lore.kernel.org/netdev/[email protected]/ [17.a]https://netdevconf.info/0x13/session.html?talk-tc-u-classifier
[17.b]man tc-u32
[18]man tc-pedit
[19] https://lore.kernel.org/netdev/[email protected]/T/#m86e71743d1d83b728bb29d5b877797cb4942e835
[20.a] https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html
[20.b] https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html--------
HISTORY
--------Changes in Version 12
----------------------0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to
p4tc_xxxfor uapi andp4a_xxxfor actions to provide consistency(Jiri).4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.Quoted 5 lines
The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via
Kconfig).
Quoted 7 lines
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Some of the people who attend our meetings and have vested interest in this are on Cc. But the cover letter is clear on this (right at the top under "What is P4" and "what is P4TC").
cheers,
jamal
Quoted 5 lines
Thanks,
Paolo
Quoted 481 lines · Jamal Hadi Salim
Quoted 480 lines · Paolo Abeni
Quoted 477 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware
acquisition(example [12]).Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC, with these patches, provides
_kerneland user space code changeindependence_. Meaning:
A P4 program describes headers, parsers, etc alongside the datapath processing; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md
__P4TCArchitecture__The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch description later..
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TCWorkflow__The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above..
b) the parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TCRuntime ControlPath__The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+)Read(</path/to/object>, [optional filter])Update(</path/to/object>, DATA>+)Delete(</path/to/object>, [optional filter])Subscribe(</path/to/object>, [optional filter])Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted. "Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1
tc p4ctrl create myprog/table/mytable \ dstAddr 10.0.1.2/32 action send_to_port param port eno1..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 actionsend_to_portparam port eno1 \
entry dstAddr 10.1.10.2/32 actionsend_to_portparam port eno10 \
entry dstAddr 10.0.2.2/32 actionsend_to_portparam port eno2..Get an entry (note "read" is interchangeably used as "get" which is a common
semantic in tc):tc p4ctrl read myprog/table/mytable \ dstAddr 10.0.2.2/32..dump mytable
tc p4ctrl read myprog/table/mytable..dump mytable for all entries whose key fits within 10.1.0.0/16
tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16..dump all mytable entries which have an action
send_to_portwith param "eno1"
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \ filter param/act/myprog/send_to_port/port = "eno1" && \ key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filtermsecs_since< 10000Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "
send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types). All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TCDatapath__The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address in labelled "dstAddr" above).
The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath).
Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__References__[1]https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf [2]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#historical-perspective-for-p4tc [3]https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation [4]https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md#so-why-p4-and-how-does-p4-help-here [5]https://lore.kernel.org/netdev/[email protected]/T/#mf59be7abc5df3473cff3879c8cc3e2369c0640a6 [6]https://lore.kernel.org/netdev/[email protected]/T/#m783cfd79e9d755cf0e7afc1a7d5404635a5b1919 [7]https://lore.kernel.org/netdev/[email protected]/T/#ma8c84df0f7043d17b98f3d67aab0f4904c600469 [8]https://github.com/p4lang/p4c/tree/main/backends/tc
[9]https://p4.org/ [10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
[12]https://github.com/sonic-net/DASH/tree/main
[13]https://github.com/p4lang/p4c/tree/main/backends/ebpf [14]https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html [15]https://dl.acm.org/doi/10.1145/3630047.3630193 [16]https://lore.kernel.org/netdev/[email protected]/ [17.a]https://netdevconf.info/0x13/session.html?talk-tc-u-classifier
[17.b]man tc-u32
[18]man tc-pedit
[19] https://lore.kernel.org/netdev/[email protected]/T/#m86e71743d1d83b728bb29d5b877797cb4942e835
[20.a] https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html
[20.b] https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html--------
HISTORY
--------Changes in Version 12
----------------------0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to
p4tc_xxxfor uapi andp4a_xxxfor actions to provide consistency(Jiri).4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via
Kconfig).
Quoted 7 lines
Quoted 7 lines
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Yeah agree with h/w comment would be interested to hear these folks that have h/w. For me to get on board obvious things that would be interesting.
(a) hardware offload (b) some fundamental problem with exisiing p4c backend we already have or (c) significant performance improvement.
Quoted 15 lines
Some of the people who attend our meetings and have vested interest in this are on Cc. But the cover letter is clear on this (right at the top under "What is P4" and "what is P4TC").
cheers,
jamalQuoted 5 lines
Thanks,
Paolo
From: Paolo Abeni <[email protected]>
Quoted 5 lines
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Hi,
Intel/AMD definitely need the p4tc offload support and a kernel SW pipeline, as a lot of customers using programmable pipeline (smart switch and smart NIC) prefer kernel standard APIs and interfaces (netlink and tc ndo). Intel and other vendors have native P4 capable HW and are invested in P4 as a dataplane specification.
- Customers run P4 dataplane in multiple targets including SW pipeline as well as programmable Switches and DPUs.
- A standardized kernel APIs and implementation brings in portability across vendors and across targets (CPU/SW and DPUs).
- A P4 pipeline can be built using both SW and HW (DPU/switch) components and the P4 pipeline should seamlessly move between the two.
- This patch series helps create a SW pipeline and standard API.
Thanks,
Anjali
Quoted 10 lines · Anjali
From: Paolo Abeni <[email protected]>
Quoted 5 lines
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Hi,
Intel/AMD definitely need the p4tc offload support and a kernel SW pipeline, as a lot of customers using programmable pipeline (smart switch and smart NIC) prefer kernel standard APIs and interfaces (netlink and tc ndo). Intel and other vendors have native P4 capable HW and are invested in P4 as a dataplane specification.
Great what hardware/driver and how do we get that code here so we can see it working? Is the hardware available e.g. can I get ahold of one?
What is programmable on your devices? Is this 'just' the parser graph or are you slicing up tables and so on. Is it a FPGA, DPU architecture or a TCAM architecture? How do you reprogram the device? I somehow doubt its through a piecemeal ndo. But let me know if I'm wrong maybe my internal architecture details are dated. Fully speculating the interface is a FW big thunk to the device?
Without any details its difficult to get community feedback on how the hw programmable interface should work. The only reason I've even bothered with this thread is I want to see P4 working.
Who owns the AMD side or some other vendor so we can get something that works across at least two vendors which is our usual bar for adding hw offload things.
Note if you just want a kernel SW pipeline we already have that so I'm not seeing that as paticularly motivating. Again my point of view. P4 as a dataplane specification is great but I don't see the connection to this patchset without real hardware in a driver.
Quoted 9 lines
- Customers run P4 dataplane in multiple targets including SW pipeline as well as programmable Switches and DPUs.
- A standardized kernel APIs and implementation brings in portability across vendors and across targets (CPU/SW and DPUs).
- A P4 pipeline can be built using both SW and HW (DPU/switch) components and the P4 pipeline should seamlessly move between the two.
- This patch series helps create a SW pipeline and standard API.Thanks,
Anjali
Jamal Hadi Salim
maintainerQuoted 27 lines · John Fastabend
Quoted 10 lines · Anjali
From: Paolo Abeni <[email protected]>
Quoted 5 lines
I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Hi,
Intel/AMD definitely need the p4tc offload support and a kernel SW pipeline, as a lot of customers using programmable pipeline (smart switch and smart NIC) prefer kernel standard APIs and interfaces (netlink and tc ndo). Intel and other vendors have native P4 capable HW and are invested in P4 as a dataplane specification.Great what hardware/driver and how do we get that code here so we can see it working? Is the hardware available e.g. can I get ahold of one?
What is programmable on your devices? Is this 'just' the parser graph or are you slicing up tables and so on. Is it a FPGA, DPU architecture or a TCAM architecture? How do you reprogram the device? I somehow doubt its through a piecemeal ndo. But let me know if I'm wrong maybe my internal architecture details are dated. Fully speculating the interface is a FW big thunk to the device?
Without any details its difficult to get community feedback on how the hw programmable interface should work. The only reason I've even bothered with this thread is I want to see P4 working.
Quoted 8 lines
Who owns the AMD side or some other vendor so we can get something that works across at least two vendors which is our usual bar for adding hw offload things.
Note if you just want a kernel SW pipeline we already have that so I'm not seeing that as paticularly motivating. Again my point of view. P4 as a dataplane specification is great but I don't see the connection to this patchset without real hardware in a driver.
Here's what you can buy on the market that are native P4 (not that it
hasnt been mentioned from day 1 on patch 0 references):
[10]https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html [11]https://www.amd.com/en/accelerators/pensando
I want to emphasize again these patches are about the P4 s/w pipeline that is intended to work seamlessly with hw offload. If you are interested in h/w offload and want to contribute just show up at the meetings - they are open to all. The current offloadable piece is the match-action tables. The P4 specs may change to include parsers in the future or other objects etc (but not sure why we should discuss this in the thread).
cheers,
jamal
A message that is not in the archive (sent off-list or not yet received): CAOuuhY8qbsYCjdUYUZv8J3jz8HGXmtxLmTDP6LKgN5uRVZwMnQ@mail.gmail.com
Jakub Kicinski
maintainerQuoted 10 lines · On Thu, 29 Feb 2024 19:00:50 -0800 Tom Herbert wrote:
Quoted 7 lines
I want to emphasize again these patches are about the P4 s/w pipeline that is intended to work seamlessly with hw offload. If you are interested in h/w offload and want to contribute just show up at the meetings - they are open to all. The current offloadable piece is the match-action tables. The P4 specs may change to include parsers in the future or other objects etc (but not sure why we should discuss this in the thread).
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from /lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
Jamal Hadi Salim
maintainerQuoted 10 lines · Jakub Kicinski
Quoted 8 lines · On Thu, 29 Feb 2024 19:00:50 -0800 Tom Herbert wrote:
Quoted 7 lines
I want to emphasize again these patches are about the P4 s/w pipeline that is intended to work seamlessly with hw offload. If you are interested in h/w offload and want to contribute just show up at the meetings - they are open to all. The current offloadable piece is the match-action tables. The P4 specs may change to include parsers in the future or other objects etc (but not sure why we should discuss this in the thread).
Quoted 5 lines
Quoted 2 lines
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
I should clarify what i meant by "seamless". It means the same control API is used for s/w or h/w. This is a feature of tc, and is not being introduced by P4TC. P4 control only deals with Match-action tables - just as TC does.
Quoted 5 lines
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from
/lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.
BTW: I just want to say that these patches are about s/w - not offload. Someone asked about offload so as in normal discussions we steered in that direction. The hardware piece will require additional patchsets which still require discussions. I hope we dont steer off too much, otherwise i can start a new thread just to discuss current view of the h/w.
Its not the device telling the kernel what it has. Its the other way around. From the P4 program you generate the s/w (the ebpf code and other auxillary stuff) and h/w pieces using a compiler.
You compile ebpf, etc, then load.
The current point of discussion is the hw binary is to be "activated" through the same tc filter that does the s/w. So one could say:
tc filter add block 22 ingress protocol all prio 1 p4 pname simple_l3 \
prog type hw filename "simple_l3.o" ... \
action bpf obj $PARSER.o section p4tc/parser \
action bpf obj $PROGNAME.o section p4tc/main
And that would through tc driver callbacks signal to the driver to find the binary possibly via /lib/firmware
Some of the original discussion was to use devlink for loading the binary - but that went nowhere.
Once you have this in place then netlink with tc skip_sw/hw. This is what i meant by "seamless"
Quoted 7 lines
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
The current (maybe i should say initial) thought is the P4 program does not touch the existing kernel infra such as fdb etc. Of course we can model the kernel datapath using P4 but you wont be using "ip route add..." or "bridge fdb...".
In the future, P4 extern could be used to model existing infra and we should be able to use the same tooling. That is a discussion that comes on/off (i think it did in the last meeting).
cheers,
jamal
Jakub Kicinski
maintainerQuoted 10 lines · On Fri, 1 Mar 2024 12:39:56 -0500 Jamal Hadi Salim wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
I should clarify what i meant by "seamless". It means the same control API is used for s/w or h/w. This is a feature of tc, and is not being introduced by P4TC. P4 control only deals with Match-action tables - just as TC does.
Right, and the compiled P4 pipeline is tacked onto that API. Loading that presumably implies a pipeline reset. There's no precedent for loading things into TC resulting a device datapath reset.
Quoted 5 lines
Quoted 4 lines
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from
/lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.
Quoted 8 lines
BTW: I just want to say that these patches are about s/w - not offload. Someone asked about offload so as in normal discussions we steered in that direction. The hardware piece will require additional patchsets which still require discussions. I hope we dont steer off too much, otherwise i can start a new thread just to discuss current view of the h/w.
Its not the device telling the kernel what it has. Its the other way around.
Yes, I'm describing how I'd have designed it :) If it was the same as what you've already implemented - why would I be typing it into an email.. ? :)
Quoted 3 lines
From the P4 program you generate the s/w (the ebpf code and other auxillary stuff) and h/w pieces using a compiler.
You compile ebpf, etc, then load.
That part is fine.
Quoted 9 lines
The current point of discussion is the hw binary is to be "activated" through the same tc filter that does the s/w. So one could say:
tc filter add block 22 ingress protocol all prio 1 p4 pname
simple_l3\ prog type hw filename "simple_l3.o" ... \ action bpf obj $PARSER.o section p4tc/parser \ action bpf obj $PROGNAME.o section p4tc/main
Quoted 4 lines
And that would through tc driver callbacks signal to the driver to find the binary possibly via
/lib/firmware
Some of the original discussion was to use devlink for loading the binary - but that went nowhere.
Back to the device reset, unless the load has no impact on inflight traffic the loading doesn't belong in TC, IMO. Plus you're going to run into (what IIRC was Jiri's complaint) that you're loading arbitrary binary blobs, opaque to the kernel.
Quoted 11 lines
Once you have this in place then netlink with tc skip_sw/hw. This is what i meant by "seamless"
Quoted 7 lines
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
Quoted 2 lines
The current (maybe i should say initial) thought is the P4 program does not touch the existing kernel infra such as fdb etc.
It's off to the side thing. Ignoring the fact that *all*, networking devices already have parsers which would benefit from being accurately described.
Quoted 5 lines
Of course we can model the kernel datapath using P4 but you wont be using "ip route add..." or "bridge fdb...".
In the future, P4 extern could be used to model existing infra and we should be able to use the same tooling. That is a discussion that comes on/off (i think it did in the last meeting).
Maybe, IDK. I thought prevailing wisdom, at least for offloads, is to offload the existing networking stack, and fill in the gaps. Not build a completely new implementation from scratch, and "integrate later". Or at least "fill in the gaps" is how I like to think.
I can't quite fit together in my head how this is okay, but OvS was not allowed to add their offload API. And what's supposed to be part of TC and what isn't, where you only expect to have one filter here, and create a whole new object universe inside TC.
But that's just my opinions. The way things work we may wake up one day and find out that Dave has applied this :)
Quoted 72 lines · Jakub Kicinski
Quoted 10 lines · On Fri, 1 Mar 2024 12:39:56 -0500 Jamal Hadi Salim wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
I should clarify what i meant by "seamless". It means the same control API is used for s/w or h/w. This is a feature of tc, and is not being introduced by P4TC. P4 control only deals with Match-action tables - just as TC does.
Right, and the compiled P4 pipeline is tacked onto that API. Loading that presumably implies a pipeline reset. There's no precedent for loading things into TC resulting a device datapath reset.
Quoted 5 lines
Quoted 4 lines
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from
/lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.Quoted 8 lines
BTW: I just want to say that these patches are about s/w - not offload. Someone asked about offload so as in normal discussions we steered in that direction. The hardware piece will require additional patchsets which still require discussions. I hope we dont steer off too much, otherwise i can start a new thread just to discuss current view of the h/w.
Its not the device telling the kernel what it has. Its the other way around.
Yes, I'm describing how I'd have designed it :) If it was the same as what you've already implemented - why would I be typing it into an email.. ? :)
Quoted 3 lines
From the P4 program you generate the s/w (the ebpf code and other auxillary stuff) and h/w pieces using a compiler.
You compile ebpf, etc, then load.That part is fine.
Quoted 9 lines
The current point of discussion is the hw binary is to be "activated" through the same tc filter that does the s/w. So one could say:
tc filter add block 22 ingress protocol all prio 1 p4 pname
simple_l3\ prog type hw filename "simple_l3.o" ... \ action bpf obj $PARSER.o section p4tc/parser \ action bpf obj $PROGNAME.o section p4tc/mainQuoted 4 lines
And that would through tc driver callbacks signal to the driver to find the binary possibly via
/lib/firmware
Some of the original discussion was to use devlink for loading the binary - but that went nowhere.Back to the device reset, unless the load has no impact on inflight traffic the loading doesn't belong in TC, IMO. Plus you're going to run into (what IIRC was Jiri's complaint) that you're loading arbitrary binary blobs, opaque to the kernel.
Quoted 11 lines
Once you have this in place then netlink with tc skip_sw/hw. This is what i meant by "seamless"
Quoted 7 lines
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
Quoted 6 lines
Quoted 2 lines
The current (maybe i should say initial) thought is the P4 program does not touch the existing kernel infra such as fdb etc.
It's off to the side thing. Ignoring the fact that *all*, networking devices already have parsers which would benefit from being accurately described.
Jakub,
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Tom
Quoted 19 lines
Quoted 6 lines
Of course we can model the kernel datapath using P4 but you wont be using "ip route add..." or "bridge fdb...".
In the future, P4 extern could be used to model existing infra and we should be able to use the same tooling. That is a discussion that comes on/off (i think it did in the last meeting).Maybe, IDK. I thought prevailing wisdom, at least for offloads, is to offload the existing networking stack, and fill in the gaps. Not build a completely new implementation from scratch, and "integrate later". Or at least "fill in the gaps" is how I like to think.
I can't quite fit together in my head how this is okay, but OvS was not allowed to add their offload API. And what's supposed to be part of TC and what isn't, where you only expect to have one filter here, and create a whole new object universe inside TC.
But that's just my opinions. The way things work we may wake up one day and find out that Dave has applied this :)
Jakub Kicinski
maintainerQuoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32. Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Quoted 27 lines · Jakub Kicinski
Quoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Quoted 2 lines
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32.
Yes. Parsing things like TLVs, GRE flag field, or nested protobufs isn't conducive to u32. We also want the advantages of compiler optimizations to unroll loops, squash nodes in the parse graph, etc.
Quoted 8 lines
Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Hi Jakub,
I believe the solution is to replace kernel code with eBPF in cases where we need programmability. This effectively means that we would ship eBPF code as part of the kernel. So in the case of TSO, the kernel would include a standard implementation in eBPF that could be compiled into the kernel by default. The restricted C source code is tagged with a hash, so if someone wants to offload TSO they could compile the source into their target and retain the hash. At runtime it's a matter of querying the driver to see if the device supports the TSO program the kernel is running by comparing hash values. Scaling this, a device could support a catalogue of programs: TSO, LRO, parser, IPtables, etc., If the kernel can match the hash of its eBPF code to one reported by the driver then it can assume functionality is offloadable. This is an elaboration of "device features", but instead of the device telling us they think they support an adequate GRO implementation by reporting NETIF_F_GRO, the device would tell the kernel that they not only support GRO but they provide identical functionality of the kernel GRO (which IMO is the first requirement of kernel offload).
Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable. Since the code is in eBPF, the kernel can be reprogrammed at runtime which could be controlled by TC. This allows local customization of kernel features, but also is the simplest way to "patch" the kernel with security and bug fixes (nobody is ever excited to do a kernel rebase in their datacenter!). Flow dissector is a prime candidate for this, and I am still planning to replace it with an all eBPF program
(https://netdevconf.info/0x15/slides/16/Flow%20dissector_PANDA%20parser.pdf).
Tom
Jakub Kicinski
maintainerQuoted 2 lines · On Sun, 3 Mar 2024 08:31:11 -0800 Tom Herbert wrote:
Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable.
I like some aspects of what you're describing, but my understanding is that it'd be a noticeable shift in direction.
I'm not sure if merging P4TC is the most effective way of taking a first step in that direction. (I mean that in the literal sense of lack of confidence, not polite way to indicate holding a conviction to the contrary.)
eBPF to implement core functionility WAS Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 11 lines · Jakub Kicinski
Quoted 2 lines · On Sun, 3 Mar 2024 08:31:11 -0800 Tom Herbert wrote:
Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable.
I like some aspects of what you're describing, but my understanding is that it'd be a noticeable shift in direction.
I'm not sure if merging P4TC is the most effective way of taking a first step in that direction. (I mean that in the literal sense of lack of confidence, not polite way to indicate holding a conviction to the contrary.)
Jakub,
My comments were with regards to making the kernel offloadable by first making it programmable. The P4TC patches are very good for describing processing that is table driven like filtering or IPtables, but I was thinking more of kernel datapath processing that isn't table driven like GSO, GRO, flow dissector, and even up to revisiting TCP offload.
Basically, I'm proposing that instead of eBPF always being side functionality, there are cases where it could natively be used to implement the main functionality of the kernel datapath! It is a noticeable shift in direction, but I also think it's the logical outcome of eBPF :-).
Tom
Quoted 71 lines · Tom Herbert
Quoted 27 lines · Jakub Kicinski
Quoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Quoted 2 lines
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32.
Yes. Parsing things like TLVs, GRE flag field, or nested protobufs isn't conducive to u32. We also want the advantages of compiler optimizations to unroll loops, squash nodes in the parse graph, etc.
Quoted 8 lines
Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Hi Jakub,
I believe the solution is to replace kernel code with eBPF in cases where we need programmability. This effectively means that we would ship eBPF code as part of the kernel. So in the case of TSO, the kernel would include a standard implementation in eBPF that could be compiled into the kernel by default. The restricted C source code is tagged with a hash, so if someone wants to offload TSO they could compile the source into their target and retain the hash. At runtime it's a matter of querying the driver to see if the device supports the TSO program the kernel is running by comparing hash values. Scaling this, a device could support a catalogue of programs: TSO, LRO, parser, IPtables, etc., If the kernel can match the hash of its eBPF code to one reported by the driver then it can assume functionality is offloadable. This is an elaboration of "device features", but instead of the device telling us they think they support an adequate GRO implementation by reporting
NETIF_F_GRO, the device would tell the kernel that they not only support GRO but they provide identical functionality of the kernel GRO (which IMO is the first requirement of kernel offload).Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable. Since the code is in eBPF, the kernel can be reprogrammed at runtime which could be controlled by TC. This allows local customization of kernel features, but also is the simplest way to "patch" the kernel with security and bug fixes (nobody is ever excited to do a kernel
[..]
Quoted 3 lines
rebase in their datacenter!). Flow dissector is a prime candidate for this, and I am still planning to replace it with an all eBPF program
(https://netdevconf.info/0x15/slides/16/Flow%20dissector_PANDA%20parser.pdf).
So you're suggesting to bundle (and extend) tools/testing/selftests/bpf/progs/bpf_flow.c? We were thinking along similar lines here. We load this program manually right now, shipping and autoloading with the kernel will be easer.
Quoted 76 lines · Stanislav Fomichev
Quoted 71 lines · Tom Herbert
Quoted 27 lines · Jakub Kicinski
Quoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Quoted 2 lines
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32.
Yes. Parsing things like TLVs, GRE flag field, or nested protobufs isn't conducive to u32. We also want the advantages of compiler optimizations to unroll loops, squash nodes in the parse graph, etc.
Quoted 8 lines
Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Hi Jakub,
I believe the solution is to replace kernel code with eBPF in cases where we need programmability. This effectively means that we would ship eBPF code as part of the kernel. So in the case of TSO, the kernel would include a standard implementation in eBPF that could be compiled into the kernel by default. The restricted C source code is tagged with a hash, so if someone wants to offload TSO they could compile the source into their target and retain the hash. At runtime it's a matter of querying the driver to see if the device supports the TSO program the kernel is running by comparing hash values. Scaling this, a device could support a catalogue of programs: TSO, LRO, parser, IPtables, etc., If the kernel can match the hash of its eBPF code to one reported by the driver then it can assume functionality is offloadable. This is an elaboration of "device features", but instead of the device telling us they think they support an adequate GRO implementation by reporting
NETIF_F_GRO, the device would tell the kernel that they not only support GRO but they provide identical functionality of the kernel GRO (which IMO is the first requirement of kernel offload).Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable. Since the code is in eBPF, the kernel can be reprogrammed at runtime which could be controlled by TC. This allows local customization of kernel features, but also is the simplest way to "patch" the kernel with security and bug fixes (nobody is ever excited to do a kernel
[..]
Quoted 8 lines
Quoted 3 lines
rebase in their datacenter!). Flow dissector is a prime candidate for this, and I am still planning to replace it with an all eBPF program
(https://netdevconf.info/0x15/slides/16/Flow%20dissector_PANDA%20parser.pdf).So you're suggesting to bundle (and extend)
tools/testing/selftests/bpf/progs/bpf_flow.c? We were thinking along similar lines here. We load this program manually right now, shipping and autoloading with the kernel will be easer.
Hi Stanislav,
Yes, I envision that we would have a standard implementation of flow-dissector in eBPF that is shipped with the kernel and autoloaded. However, for the front end source I want to move away from imperative code. As I mentioned in the presentation flow_dissector.c is spaghetti code and has been prone to bugs over the years especially whenever someone adds support for a new fringe protocol (I take the liberty to call it spaghetti code since I'm partially responsible for creating this mess ;-) ).
The problem is that parsers are much better represented by a declarative rather than an imperative representation. To that end, we defined PANDA which allows constructing a parser (parse graph) in data structures in C. We use the "PANDA parser" to compile C to restricted C code which looks more like eBPF in imperative code. With this method we abstract out all the bookkeeping that was often the source of bugs (like pulling up skbufs, checking length limits, etc.). The other advantage is that we're able to find a lot more optimizations if we start with a right representation of the problem.
If you're interested, the video presentation on this is in
https://www.youtube.com/watch?v=zVnmVDSEoXc.
Tom
Quoted 107 lines · Tom Herbert
Quoted 76 lines · Stanislav Fomichev
Quoted 71 lines · Tom Herbert
Quoted 27 lines · Jakub Kicinski
Quoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Quoted 2 lines
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32.
Yes. Parsing things like TLVs, GRE flag field, or nested protobufs isn't conducive to u32. We also want the advantages of compiler optimizations to unroll loops, squash nodes in the parse graph, etc.
Quoted 8 lines
Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Hi Jakub,
I believe the solution is to replace kernel code with eBPF in cases where we need programmability. This effectively means that we would ship eBPF code as part of the kernel. So in the case of TSO, the kernel would include a standard implementation in eBPF that could be compiled into the kernel by default. The restricted C source code is tagged with a hash, so if someone wants to offload TSO they could compile the source into their target and retain the hash. At runtime it's a matter of querying the driver to see if the device supports the TSO program the kernel is running by comparing hash values. Scaling this, a device could support a catalogue of programs: TSO, LRO, parser, IPtables, etc., If the kernel can match the hash of its eBPF code to one reported by the driver then it can assume functionality is offloadable. This is an elaboration of "device features", but instead of the device telling us they think they support an adequate GRO implementation by reporting
NETIF_F_GRO, the device would tell the kernel that they not only support GRO but they provide identical functionality of the kernel GRO (which IMO is the first requirement of kernel offload).Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable. Since the code is in eBPF, the kernel can be reprogrammed at runtime which could be controlled by TC. This allows local customization of kernel features, but also is the simplest way to "patch" the kernel with security and bug fixes (nobody is ever excited to do a kernel
[..]
Quoted 8 lines
Quoted 3 lines
rebase in their datacenter!). Flow dissector is a prime candidate for this, and I am still planning to replace it with an all eBPF program
(https://netdevconf.info/0x15/slides/16/Flow%20dissector_PANDA%20parser.pdf).So you're suggesting to bundle (and extend)
tools/testing/selftests/bpf/progs/bpf_flow.c? We were thinking along similar lines here. We load this program manually right now, shipping and autoloading with the kernel will be easer.Hi Stanislav,
Yes, I envision that we would have a standard implementation of flow-dissector in eBPF that is shipped with the kernel and autoloaded. However, for the front end source I want to move away from imperative code. As I mentioned in the presentation
flow_dissector.cis spaghetti code and has been prone to bugs over the years especially whenever someone adds support for a new fringe protocol (I take the liberty to call it spaghetti code since I'm partially responsible for creating this mess ;-) ).The problem is that parsers are much better represented by a declarative rather than an imperative representation. To that end, we defined PANDA which allows constructing a parser (parse graph) in data structures in C. We use the "PANDA parser" to compile C to restricted C code which looks more like eBPF in imperative code. With this method we abstract out all the bookkeeping that was often the source of bugs (like pulling up skbufs, checking length limits, etc.). The other advantage is that we're able to find a lot more optimizations if we start with a right representation of the problem.
Quoted 2 lines
If you're interested, the video presentation on this is in
https://www.youtube.com/watch?v=zVnmVDSEoXc.
Oh, yeah, I've seen this one. Agreed that the C implementation is not pleasant and generating a parser from some declarative spec is a better idea.
From my pow, the biggest win we get from making bpf flow dissector pluggable is the fact that we can now actually write some tests for it (and, maybe, fuzz it?). We should also probably spend more time properly defining the behavior of the existing C implementation. We've seen some interesting bugs like this one:
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=9fa02892857ae2b3b699630e5ede28f72106e7e7
Quoted 112 lines · Stanislav Fomichev
Quoted 107 lines · Tom Herbert
Quoted 76 lines · Stanislav Fomichev
Quoted 71 lines · Tom Herbert
Quoted 27 lines · Jakub Kicinski
Quoted 11 lines · On Fri, 1 Mar 2024 18:20:36 -0800 Tom Herbert wrote:
This is configurability versus programmability. The table driven approach as input (configurability) might work fine for generic match-action tables up to the point that tables are expressive enough to satisfy the requirements. But parsing doesn't fall into the table driven paradigm: parsers want to be *programmed*. This is why we removed kParser from this patch set and fell back to eBPF for parsing. But the problem we quickly hit that eBPF is not offloadable to network devices, for example when we compile P4 in an eBPF parser we've lost the declarative representation that parsers in the devices could consume (they're not CPUs running eBPF).
Quoted 13 lines
I think the key here is what we mean by kernel offload. When we do kernel offload, is it the kernel implementation or the kernel functionality that's being offloaded? If it's the latter then we have a lot more flexibility. What we'd need is a safe and secure way to synchronize with that offload device that precisely supports the kernel functionality we'd like to offload. This can be done if both the kernel bits and programmed offload are derived from the same source (i.e. tag source code with a sha-1). For example, if someone writes a parser in P4, we can compile that into both eBPF and a P4 backend using independent tool chains and program download. At runtime, the kernel can safely offload the functionality of the eBPF parser to the device if it matches the hash to that reported by the device
Quoted 2 lines
Good points. If I understand you correctly you're saying that parsers are more complex than just a basic parsing tree a'la u32.
Yes. Parsing things like TLVs, GRE flag field, or nested protobufs isn't conducive to u32. We also want the advantages of compiler optimizations to unroll loops, squash nodes in the parse graph, etc.
Quoted 8 lines
Then we can take this argument further. P4 has grown to encompass a lot of functionality of quite complex devices. How do we square that with the kernel functionality offload model. If the entire device is modeled, including f.e. TSO, an offload would mean that the user has to write a TSO implementation which they then load into TC? That seems odd.
IOW I don't quite know how to square in my head the "total functionality" with being a TC-based "plugin".
Hi Jakub,
I believe the solution is to replace kernel code with eBPF in cases where we need programmability. This effectively means that we would ship eBPF code as part of the kernel. So in the case of TSO, the kernel would include a standard implementation in eBPF that could be compiled into the kernel by default. The restricted C source code is tagged with a hash, so if someone wants to offload TSO they could compile the source into their target and retain the hash. At runtime it's a matter of querying the driver to see if the device supports the TSO program the kernel is running by comparing hash values. Scaling this, a device could support a catalogue of programs: TSO, LRO, parser, IPtables, etc., If the kernel can match the hash of its eBPF code to one reported by the driver then it can assume functionality is offloadable. This is an elaboration of "device features", but instead of the device telling us they think they support an adequate GRO implementation by reporting
NETIF_F_GRO, the device would tell the kernel that they not only support GRO but they provide identical functionality of the kernel GRO (which IMO is the first requirement of kernel offload).Even before considering hardware offload, I think this approach addresses a more fundamental problem to make the kernel programmable. Since the code is in eBPF, the kernel can be reprogrammed at runtime which could be controlled by TC. This allows local customization of kernel features, but also is the simplest way to "patch" the kernel with security and bug fixes (nobody is ever excited to do a kernel
[..]
Quoted 8 lines
Quoted 3 lines
rebase in their datacenter!). Flow dissector is a prime candidate for this, and I am still planning to replace it with an all eBPF program
(https://netdevconf.info/0x15/slides/16/Flow%20dissector_PANDA%20parser.pdf).So you're suggesting to bundle (and extend)
tools/testing/selftests/bpf/progs/bpf_flow.c? We were thinking along similar lines here. We load this program manually right now, shipping and autoloading with the kernel will be easer.Hi Stanislav,
Yes, I envision that we would have a standard implementation of flow-dissector in eBPF that is shipped with the kernel and autoloaded. However, for the front end source I want to move away from imperative code. As I mentioned in the presentation
flow_dissector.cis spaghetti code and has been prone to bugs over the years especially whenever someone adds support for a new fringe protocol (I take the liberty to call it spaghetti code since I'm partially responsible for creating this mess ;-) ).The problem is that parsers are much better represented by a declarative rather than an imperative representation. To that end, we defined PANDA which allows constructing a parser (parse graph) in data structures in C. We use the "PANDA parser" to compile C to restricted C code which looks more like eBPF in imperative code. With this method we abstract out all the bookkeeping that was often the source of bugs (like pulling up skbufs, checking length limits, etc.). The other advantage is that we're able to find a lot more optimizations if we start with a right representation of the problem.
Quoted 2 lines
If you're interested, the video presentation on this is in
https://www.youtube.com/watch?v=zVnmVDSEoXc.
Quoted 6 lines
Oh, yeah, I've seen this one. Agreed that the C implementation is not pleasant and generating a parser from some declarative spec is a better idea.
From my pow, the biggest win we get from making bpf flow dissector pluggable is the fact that we can now actually write some tests for it
Yes, extracting out functions from the kernel allows them to be independently unit tested. It's an even bigger win if the same source code is used for offloading the functionality as I described. We can call this "Test once, run anywhere!"
Tom
Quoted 4 lines
(and, maybe, fuzz it?). We should also probably spend more time properly defining the behavior of the existing C implementation. We've seen some interesting bugs like this one:
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=9fa02892857ae2b3b699630e5ede28f72106e7e7
Jamal Hadi Salim
maintainerHardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
In reply to Jakub KicinskiQuoted 13 lines · Jakub Kicinski
Quoted 10 lines · On Fri, 1 Mar 2024 12:39:56 -0500 Jamal Hadi Salim wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
I should clarify what i meant by "seamless". It means the same control API is used for s/w or h/w. This is a feature of tc, and is not being introduced by P4TC. P4 control only deals with Match-action tables - just as TC does.
Quoted 4 lines
Right, and the compiled P4 pipeline is tacked onto that API. Loading that presumably implies a pipeline reset. There's no precedent for loading things into TC resulting a device datapath reset.
Ive changed the subject to reflect this discussion is about h/w offload so we dont drift too much from the intent of the patches.
AFAIK, all these devices have some HA built in to do program replacement. i.e. afaik, no device reset.
I believe the tofino switch in the earlier generations may have needed resets which caused a few packet drops in a live environment update. Granted there may be devices (not that i am aware) that may not be able to do HA. All this needs to be considered for offloads.
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from
/lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.BTW: I just want to say that these patches are about s/w - not offload. Someone asked about offload so as in normal discussions we steered in that direction. The hardware piece will require additional patchsets which still require discussions. I hope we dont steer off too much, otherwise i can start a new thread just to discuss current view of the h/w.
Quoted 6 lines
Quoted 1 line
Its not the device telling the kernel what it has. Its the other way around.
Yes, I'm describing how I'd have designed it :) If it was the same as what you've already implemented - why would I be typing it into an email.. ? :)
I think i misunderstood you and thought I needed to provide context. The P4 pipelines are meant to be able to be re-programmed multiple times in a live environment. IOW, I should be able to delete/create a pipeline while another is running. Some hardware may require that the parser is shared etc, but you can certainly replace the match action tables or add an entirely new logic. In any case this is all still under discussion and can be further refined.
Quoted 20 lines
Quoted 3 lines
From the P4 program you generate the s/w (the ebpf code and other auxillary stuff) and h/w pieces using a compiler.
You compile ebpf, etc, then load.That part is fine.
Quoted 9 lines
The current point of discussion is the hw binary is to be "activated" through the same tc filter that does the s/w. So one could say:
tc filter add block 22 ingress protocol all prio 1 p4 pname
simple_l3\ prog type hw filename "simple_l3.o" ... \ action bpf obj $PARSER.o section p4tc/parser \ action bpf obj $PROGNAME.o section p4tc/mainQuoted 4 lines
And that would through tc driver callbacks signal to the driver to find the binary possibly via
/lib/firmware
Some of the original discussion was to use devlink for loading the binary - but that went nowhere.
Quoted 5 lines
Back to the device reset, unless the load has no impact on inflight traffic the loading doesn't belong in TC, IMO. Plus you're going to run into (what IIRC was Jiri's complaint) that you're loading arbitrary binary blobs, opaque to the kernel.
And you said at that time binary blobs are already a way of life. Let's take DDP as a use case: They load the firmware (via ethtool) and we were recently discussing whether they should use flower or u32 etc. I would say this is in the same spirit. Doing ethtool may be a bit disconnected. But that is up for discussion as well. There has been concern that we need to have some authentication in some of the discussions. Is that what you mean?
Quoted 11 lines
Quoted 11 lines
Once you have this in place then netlink with tc skip_sw/hw. This is what i meant by "seamless"
Quoted 7 lines
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
Quoted 7 lines
Quoted 2 lines
The current (maybe i should say initial) thought is the P4 program does not touch the existing kernel infra such as fdb etc.
It's off to the side thing. Ignoring the fact that *all*, networking devices already have parsers which would benefit from being accurately described.
I am not following this point.
Quoted 11 lines
Quoted 5 lines
Of course we can model the kernel datapath using P4 but you wont be using "ip route add..." or "bridge fdb...".
In the future, P4 extern could be used to model existing infra and we should be able to use the same tooling. That is a discussion that comes on/off (i think it did in the last meeting).Maybe, IDK. I thought prevailing wisdom, at least for offloads, is to offload the existing networking stack, and fill in the gaps. Not build a completely new implementation from scratch, and "integrate later". Or at least "fill in the gaps" is how I like to think.
Quoted 5 lines
I can't quite fit together in my head how this is okay, but OvS was not allowed to add their offload API. And what's supposed to be part of TC and what isn't, where you only expect to have one filter here, and create a whole new object universe inside TC.
I was there.
Ovs matched what tc already had functionally, 10 years after tc existed, and they were busy rewriting what tc offered. So naturally we pushed for them to use what TC had. You still need to write whatever extensions needed into the kernel etc in order to support what the hardware can offer.
I hope i am not stating the obvious: P4 provides a more malleable approach. Assume a blank template in h/w and s/w and where you specify what you need then both the s/w and hardware support it. Flower is analogous to a "fixed pipeline" meaning you can extend flower by changing the kernel and datapath. Often it is not covering all potential hw match actions engines and often we see patches to do one more thing requiring more kernel changes. If you replace flower with P4 you remove the need to update the kernel, user space etc for the same features that flower needs to be extended for today. You just tell the compiler what you need (within hardware capacity of course). So i dont see P4 as "offload the existing kernel infra aka flower" but rather remove the limitations that flower constrains us with today. As far as other kernel infra (fdb etc), that can be added as i stated - it is just not a starting point.
cheers,
jamal
Quoted 2 lines
But that's just my opinions. The way things work we may wake up one day and find out that Dave has applied this :)
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 136 lines · Jamal Hadi Salim
Quoted 13 lines · Jakub Kicinski
Quoted 10 lines · On Fri, 1 Mar 2024 12:39:56 -0500 Jamal Hadi Salim wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Pardon my ignorance, but doesn't P4 want to be compiled to a backend target? How does going through TC make this seamless?
+1
I should clarify what i meant by "seamless". It means the same control API is used for s/w or h/w. This is a feature of tc, and is not being introduced by P4TC. P4 control only deals with Match-action tables - just as TC does.
Quoted 4 lines
Right, and the compiled P4 pipeline is tacked onto that API. Loading that presumably implies a pipeline reset. There's no precedent for loading things into TC resulting a device datapath reset.
Ive changed the subject to reflect this discussion is about h/w offload so we dont drift too much from the intent of the patches.
AFAIK, all these devices have some HA built in to do program replacement. i.e. afaik, no device reset.
I believe the tofino switch in the earlier generations may have needed resets which caused a few packet drops in a live environment update. Granted there may be devices (not that i am aware) that may not be able to do HA. All this needs to be considered for offloads.Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
My intuition is that for offload the device would be programmed at start-of-day / probe. By loading the compiled P4 from
/lib/firmware. Then the _device_ tells the kernel what tables and parser graph it's got.BTW: I just want to say that these patches are about s/w - not offload. Someone asked about offload so as in normal discussions we steered in that direction. The hardware piece will require additional patchsets which still require discussions. I hope we dont steer off too much, otherwise i can start a new thread just to discuss current view of the h/w.
Quoted 6 lines
Quoted 1 line
Its not the device telling the kernel what it has. Its the other way around.
Yes, I'm describing how I'd have designed it :) If it was the same as what you've already implemented - why would I be typing it into an email.. ? :)
I think i misunderstood you and thought I needed to provide context. The P4 pipelines are meant to be able to be re-programmed multiple times in a live environment. IOW, I should be able to delete/create a pipeline while another is running. Some hardware may require that the parser is shared etc, but you can certainly replace the match action tables or add an entirely new logic. In any case this is all still under discussion and can be further refined.
Quoted 20 lines
Quoted 3 lines
From the P4 program you generate the s/w (the ebpf code and other auxillary stuff) and h/w pieces using a compiler.
You compile ebpf, etc, then load.That part is fine.
Quoted 9 lines
The current point of discussion is the hw binary is to be "activated" through the same tc filter that does the s/w. So one could say:
tc filter add block 22 ingress protocol all prio 1 p4 pname
simple_l3\ prog type hw filename "simple_l3.o" ... \ action bpf obj $PARSER.o section p4tc/parser \ action bpf obj $PROGNAME.o section p4tc/mainQuoted 4 lines
And that would through tc driver callbacks signal to the driver to find the binary possibly via
/lib/firmware
Some of the original discussion was to use devlink for loading the binary - but that went nowhere.Quoted 5 lines
Back to the device reset, unless the load has no impact on inflight traffic the loading doesn't belong in TC, IMO. Plus you're going to run into (what IIRC was Jiri's complaint) that you're loading arbitrary binary blobs, opaque to the kernel.
And you said at that time binary blobs are already a way of life. Let's take DDP as a use case: They load the firmware (via ethtool) and we were recently discussing whether they should use flower or u32 etc. I would say this is in the same spirit. Doing ethtool may be a bit disconnected. But that is up for discussion as well. There has been concern that we need to have some authentication in some of the discussions. Is that what you mean?
Quoted 11 lines
Quoted 11 lines
Once you have this in place then netlink with tc skip_sw/hw. This is what i meant by "seamless"
Quoted 7 lines
Plus, if we're talking about offloads, aren't we getting back into the same controversies we had when merging OvS (not that I was around). The "standalone stack to the side" problem. Some of the tables in the pipeline may be for routing, not ACLs. Should they be fed from the routing stack? How is that integration going to work? The parsing graph feels a bit like global device configuration, not a piece of functionality that should sit under sub-sub-system in the corner.
Quoted 7 lines
Quoted 2 lines
The current (maybe i should say initial) thought is the P4 program does not touch the existing kernel infra such as fdb etc.
It's off to the side thing. Ignoring the fact that *all*, networking devices already have parsers which would benefit from being accurately described.
I am not following this point.
Quoted 11 lines
Quoted 5 lines
Of course we can model the kernel datapath using P4 but you wont be using "ip route add..." or "bridge fdb...".
In the future, P4 extern could be used to model existing infra and we should be able to use the same tooling. That is a discussion that comes on/off (i think it did in the last meeting).Maybe, IDK. I thought prevailing wisdom, at least for offloads, is to offload the existing networking stack, and fill in the gaps. Not build a completely new implementation from scratch, and "integrate later". Or at least "fill in the gaps" is how I like to think.
Quoted 5 lines
I can't quite fit together in my head how this is okay, but OvS was not allowed to add their offload API. And what's supposed to be part of TC and what isn't, where you only expect to have one filter here, and create a whole new object universe inside TC.
I was there.
Ovs matched what tc already had functionally, 10 years after tc existed, and they were busy rewriting what tc offered. So naturally we pushed for them to use what TC had. You still need to write whatever extensions needed into the kernel etc in order to support what the hardware can offer.
Quoted 15 lines
I hope i am not stating the obvious: P4 provides a more malleable approach. Assume a blank template in h/w and s/w and where you specify what you need then both the s/w and hardware support it. Flower is analogous to a "fixed pipeline" meaning you can extend flower by changing the kernel and datapath. Often it is not covering all potential hw match actions engines and often we see patches to do one more thing requiring more kernel changes. If you replace flower with P4 you remove the need to update the kernel, user space etc for the same features that flower needs to be extended for today. You just tell the compiler what you need (within hardware capacity of course). So i dont see P4 as "offload the existing kernel infra aka flower" but rather remove the limitations that flower constrains us with today. As far as other kernel infra (fdb etc), that can be added as i stated - it is just not a starting point.
Sorry, after getting some coffee i believe I mumbled too much in my previous email. Let me summarize your points and reduce the mumbling:
1)Your point on: Triggering the pipeline re/programming via the filter would require a reset of the device on a live environment. AFAIK, the "P4 native" devices that I know of do allow multiple programs and have operational schemes to allow updates without resets. I will gather more info and post it after one of our meetings. Having said that, we really have not paid much attention to this detail so it is a valid concern that needs to be ironed out. It is even more imperative if we want to support a device that is not "P4 native" or one that requires a reset whether it is P4 native or not then what you referred to as "programmed at start-of-day / probe" is a valid concern.
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user. At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have very strong user bases with strong infra tooling which took years to get in shape. So they must be supported. I see two approaches:
- you can leave those "legacy" ndo ops alone and not go via the tc ndo ops used by P4TC.
- or write a P4 program that looks _exactly_ like what current bridging looks like and add helpers to allow existing tools to continue to work via tc ndo and then phase out the "fixed datapath" ndos. This will take a long long time but it could be a goal.
There is another caveat: Often different vendor hardware has slightly different features which cant be exposed because either they are very specific to the vendor or it's just very hard to express with existing "legacy" without making intrusive changes. So we are going to be able to allow these vendors/users to expose as much or as little as is needed for a specific deployment without affecting anyone else with new kernel/user code.
On the "integrate later" aspect: That is probably because most of the times we want to avoid doing intrusive niche changes (which is resolvable with the above).
cheers,
jamal
Jakub Kicinski
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the device
User can still do BPF kfuncs to look up in the tables (like in FIB), but call them from cls_bpf.
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 20 lines
very strong user bases with strong infra tooling which took years to get in shape. So they must be supported. I see two approaches:
- you can leave those "legacy" ndo ops alone and not go via the tc ndo ops used by P4TC.
- or write a P4 program that looks _exactly_ like what current bridging looks like and add helpers to allow existing tools to continue to work via tc ndo and then phase out the "fixed datapath" ndos. This will take a long long time but it could be a goal.There is another caveat: Often different vendor hardware has slightly different features which cant be exposed because either they are very specific to the vendor or it's just very hard to express with existing "legacy" without making intrusive changes. So we are going to be able to allow these vendors/users to expose as much or as little as is needed for a specific deployment without affecting anyone else with new kernel/user code.
On the "integrate later" aspect: That is probably because most of the times we want to avoid doing intrusive niche changes (which is resolvable with the above).
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.
This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do skip_sw and it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).
A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.
Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
cheers,
jamal
Re: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy. At this point in its lifetime, eBPF had far more examples of real world use cases publically available. That being said, there's nothing unique about P4 supporting the network calculator. We could just as easily write this in eBPF (either plain C or P4) and "offload" it to an ARM core on a SmartNIC.
If we are going to support programmable device offload in the Linux kernel then I maintain it should be a generic mechanism that's agnostic to *both* the frontend programming language as well as the backend target. For frontend languages we want to let the user program in a language that's convenient for *them*, which honestly in most cases isn't going to be a narrow use case DSL (i.e. typically users want to code in C/C++, Python, Rust, etc.). For the backend it's the same story, maybe we're compiling to run in host, maybe we're offloading to P4 runtime, maybe we're offloading to another CPU, maybe we're offloading some other programmable NPU. The only real requirement is a compiler that can take the frontend code and compile for the desired backend target, but above all we want this to be easy for the programmer, the compiler needs to do the heavy lifting and we should never require the user to understand the nuances of a target.
IMO, the model we want for programmable kernel offload is "write once, run anywhere, run well". Which is the Java tagline amended with "run well". Users write one program for their datapath processing, it runs on various targets, for any given target we run to run at the highest performance levels possible given the target's capabilities.
Tom
Quoted 3 lines
cheers,
jamal
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 81 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Quoted 4 lines
Jamal,
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Quoted 6 lines
At
this point in its lifetime, eBPF had far more examples of real world use cases publically available. That being said, there's nothing unique about P4 supporting the network calculator. We could just as easily write this in eBPF (either plain C or P4) and "offload" it to an ARM core on a SmartNIC.
With current port speeds hitting 800gbps you want to use Arm cores as your offload engine?;-> Running the generated ebpf on the arm core is a valid P4 target. i.e there is no contradiction.
Note: P4 is a DSL specialized for datapath definition; it is not a competition to ebpf, two different worlds. I see ebpf as an infrastructure tool, nothing more.
Quoted 7 lines
If we are going to support programmable device offload in the Linux kernel then I maintain it should be a generic mechanism that's agnostic to *both* the frontend programming language as well as the backend target. For frontend languages we want to let the user program in a language that's convenient for *them*, which honestly in most cases isn't going to be a narrow use case DSL (i.e. typically users want to code in C/C++, Python, Rust, etc.).
You and I have never agreed philosophically on this point, ever. Developers are expensive and not economically scalable. IOW, In the era of automation (generative AI, etc) tooling is king. Let's build the right tooling. Whenever you make this statement i get the vision of Steve Balmer ranting on the stage with "developers! developers! developers!" but that was eons ago. To use your strong view: Learn compilers! And the future is probably to replace compilers with AI.
Quoted 9 lines
For the backend it's the
same story, maybe we're compiling to run in host, maybe we're offloading to P4 runtime, maybe we're offloading to another CPU, maybe we're offloading some other programmable NPU. The only real requirement is a compiler that can take the frontend code and compile for the desired backend target, but above all we want this to be easy for the programmer, the compiler needs to do the heavy lifting and we should never require the user to understand the nuances of a target.
Agreed, it is possible to use other languages in the frontend. It is also possible to extend P4.
Quoted 6 lines
IMO, the model we want for programmable kernel offload is "write once, run anywhere, run well". Which is the Java tagline amended with "run well". Users write one program for their datapath processing, it runs on various targets, for any given target we run to run at the highest performance levels possible given the target's capabilities.
I would like to emphasize: Our target is P4 - vendors have put out hardware, people are deploying and evolving things. It is real today with deployments, not some science project. I am not arguing you cant do what you suggested but we want to initially focus on P4. Neither am i saying we cant influence P4 to be more Linux friendly. But none of that matters. We are only concerned about P4.
cheers,
jamal
Quoted 5 lines
Tom
Quoted 3 lines
cheers,
jamal
Jakub Kicinski
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 7 lines · On Sun, 3 Mar 2024 14:04:11 -0500 Jamal Hadi Salim wrote:
Quoted 6 lines
At
this point in its lifetime, eBPF had far more examples of real world use cases publically available. That being said, there's nothing unique about P4 supporting the network calculator. We could just as easily write this in eBPF (either plain C or P4) and "offload" it to an ARM core on a SmartNIC.
Quoted 6 lines
With current port speeds hitting 800gbps you want to use Arm cores as your offload engine?;-> Running the generated ebpf on the arm core is a valid P4 target. i.e there is no contradiction.
Note: P4 is a DSL specialized for datapath definition; it is not a competition to ebpf, two different worlds. I see ebpf as an infrastructure tool, nothing more.
I wonder how much we're benefiting of calling this thing P4 and how much we should focus on filling in the tech gaps. Exactly like you said, BPF is not competition, but neither does the kernel "support P4", any more than it supports bpftrace and:
$ git grep --files-with-matches bpftrace Documentation/bpf/redirect.rst tools/testing/selftests/bpf/progs/test_xdp_attach_fail.c
Filling in tech gaps would also help DPP, IDK how much DPP is based or using P4, neither should I have to care, frankly :S
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 16 lines · Jakub Kicinski
Quoted 7 lines · On Sun, 3 Mar 2024 14:04:11 -0500 Jamal Hadi Salim wrote:
Quoted 6 lines
At
this point in its lifetime, eBPF had far more examples of real world use cases publically available. That being said, there's nothing unique about P4 supporting the network calculator. We could just as easily write this in eBPF (either plain C or P4) and "offload" it to an ARM core on a SmartNIC.Quoted 6 lines
With current port speeds hitting 800gbps you want to use Arm cores as your offload engine?;-> Running the generated ebpf on the arm core is a valid P4 target. i.e there is no contradiction.
Note: P4 is a DSL specialized for datapath definition; it is not a competition to ebpf, two different worlds. I see ebpf as an infrastructure tool, nothing more.
Quoted 2 lines
I wonder how much we're benefiting of calling this thing P4 and how much we should focus on filling in the tech gaps.
We are implementing based on the P4 standard specification. I fear it is confusing to call it something else if everyone else is calling it P4 (including the vendors whose devices are being targeted in case of offload).
If the name is an issue, sure we can change.
It just so happens that TC has similar semantics to P4 (match action tables) - hence the name P4TC and implementation encompassing code that fits nicely with TC.
Quoted 3 lines
Exactly like you said, BPF is not competition, but neither does the kernel "support P4", any more than it supports bpftrace and:
Like i said if name is an issue, let's change the name;->
Quoted 6 lines
$ git grep --files-with-matches bpftrace Documentation/bpf/redirect.rst tools/testing/selftests/bpf/progs/test_xdp_attach_fail.cFilling in tech gaps would also help DPP, IDK how much DPP is based or using P4, neither should I have to care, frankly :S
DDP is an Intel specific approach, pre-P4. P4: at least two vendors(on Cc) AMD have NICs with P4 specification and there FPGA variants out there as well.
From my discussions with folks at Intel it is easy to transform DDP to P4. My understanding is it is the same compiler folks. The beauty being you dont have to use the intel version of the loaded program to offload if you wanted to change what the hardware does custom to you (within constraints of what hardware can do).
cheers,
jamal
Re: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
In reply to Jamal Hadi SalimQuoted 84 lines · Jamal Hadi Salim
Quoted 83 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Quoted 8 lines
Quoted 2 lines
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Shameless plug. I have a more crazy example with bpf:
https://github.com/fomichev/xdp-btc-miner
A good way to ensure all those smartnic cycles are not wasted :-D I wish we had more nics with xdp bpf offloads :-(
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 95 lines · Stanislav Fomichev
Quoted 84 lines · Jamal Hadi Salim
Quoted 83 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Quoted 8 lines
Quoted 2 lines
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Quoted 4 lines
Shameless plug. I have a more crazy example with bpf:
Hrm - this looks crazy interesting;-> Tempting. I guess to port this to P4 we'd need the sha256 in h/w (which most of these vendors have already). Is there any other acceleration would you need? Would have been more fun if you invented you own headers too ;->
cheers,
jamal
Quoted 2 lines
A good way to ensure all those smartnic cycles are not wasted :-D I wish we had more nics with xdp bpf offloads :-(
Re: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 98 lines · Jamal Hadi Salim
Quoted 97 lines · Stanislav Fomichev
Quoted 84 lines · Jamal Hadi Salim
Quoted 83 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Quoted 8 lines
Quoted 2 lines
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Shameless plug. I have a more crazy example with bpf:
Quoted 7 lines
Quoted 2 lines
Hrm - this looks crazy interesting;-> Tempting. I guess to port this to P4 we'd need the sha256 in h/w (which most of these vendors have already). Is there any other acceleration would you need? Would have been more fun if you invented you own headers too ;->
Yeah, some way to do sha256(sha256(at_some_fixed_packet_offset + 80 bytes)) is one thing. And the other is some way to compare that sha256 vs some hard-coded (difficulty) number (as a 256-byte uint). But I have no clue how well that maps into declarative p4 language. Most likely possible if you're saying that the calculator is possible?
I'm assuming that even sha256 can possibly be implemented in p4 without any extra support from the vendor? It's just a bunch of xors and rotations over a fix-sized input buffer.
Jamal Hadi Salim
maintainerRe: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 103 lines · Stanislav Fomichev
Quoted 101 lines · Jamal Hadi Salim
Quoted 99 lines · Stanislav Fomichev
Quoted 84 lines · Jamal Hadi Salim
Quoted 83 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Quoted 8 lines
Quoted 2 lines
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Shameless plug. I have a more crazy example with bpf:
Quoted 6 lines
Quoted 4 lines
Hrm - this looks crazy interesting;-> Tempting. I guess to port this to P4 we'd need the sha256 in h/w (which most of these vendors have already). Is there any other acceleration would you need? Would have been more fun if you invented you own headers too ;->
Yeah, some way to do
sha256(sha256(at_some_fixed_packet_offset + 80 bytes))
This part is straight forward.
Quoted 2 lines
is one thing. And the other is some way to compare that sha256 vs some hard-coded (difficulty) number (as a 256-byte uint).
The compiler may have issues with this comparison - will have to look (I am pretty sure it's fixable though).
Quoted 3 lines
But I have no
clue how well that maps into declarative p4 language. Most likely possible if you're saying that the calculator is possible?
The calculator basically is written as a set of match-action tables. You parse your header, construct a key based on the operator field of the header (eg "+"), invoke an action which takes the operands from the headers(eg "1" and "2"), the action returns you results(3"). You stash the result in a new packet and send it back to the source.
So my thinking is the computation you need would be modelled on an action.
Quoted 3 lines
I'm assuming that even sha256 can possibly be implemented in p4 without any extra support from the vendor? It's just a bunch of xors and rotations over a fix-sized input buffer.
True, and I think those would be fast. But if the h/w offers it as an interface why not.
It's not that you are running out of instruction space - and my memory is hazy - but iirc, there is sha256 support in the kernel Crypto API - does it not make sense to kfunc into that?
cheers,
jamal
Re: Hardware Offload discussion WAS(Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)
Quoted 135 lines · Jamal Hadi Salim
Quoted 103 lines · Stanislav Fomichev
Quoted 101 lines · Jamal Hadi Salim
Quoted 99 lines · Stanislav Fomichev
Quoted 84 lines · Jamal Hadi Salim
Quoted 83 lines · Tom Herbert
Quoted 70 lines · Jamal Hadi Salim
Quoted 20 lines · Jakub Kicinski
Quoted 4 lines · On Sat, 2 Mar 2024 09:36:53 -0500 Jamal Hadi Salim wrote:
2) Your point on: "integrate later", or at least "fill in the gaps" This part i am probably going to mumble on. I am going to consider more than just doing ACLs/MAT via flower/u32 for the sake of discussion.
Quoted 8 lines
True, "fill the gaps" has been our model so far. It requires kernel changes, user space code changes etc justifiably so because most of the time such datapaths are subject to standardization via IETF, IEEE, etc and new extensions come in on a regular basis. And sometimes we do add features that one or two users or a single vendor has need for at the cost of kernel and user/control extension. Given our work process, any features added this way take a long time to make it to the end user.
What I had in mind was more of a DDP model. The device loads it binary blob FW in whatever way it does, then it tells the kernel its parser graph, and tables. The kernel exposes those tables to user space. All dynamic, no need to change the kernel for each new protocol.
Quoted 8 lines
But that's different in two ways:
1. the device tells kernel the tables, no "dynamic reprogramming"
2. you don't need the SW side, the only use of the API is to interact with the deviceUser can still do BPF kfuncs to look up in the tables (like in FIB), but call them from
cls_bpf.This is not far off from what is envisioned today in the discussions. The main issue is who loads the binary? We went from devlink to the filter doing the loading. DDP is ethtool. We still need to tie a PCI device/tc block to the "program" so we can do
skip_swand it works. Meaning a device that is capable of handling multiple programs can have multiple blobs loaded. A "program" is mapped to a tc filter and MAT control works the same way as it does today (netlink/tc ndo).A program in P4 has a name, ID and people have been suggesting a sha1 identity (or a signature of some kind should be generated by the compiler). So the upward propagation could be tied to discovering these 3 tuples from the driver. Then the control plane targets a program via those tuples via netlink (as we do currently).
I do note, using the DDP sample space, currently whatever gets loaded is "trusted" and really you need to have human knowledge of what the NIC's parsing + MAT is to send the control. With P4 that is all visible/programmable by the end user (i am not a proponent of vendors "shipping" things or calling them for support) - so should be sufficient to just discover what is in the binary and send the correct control messages down.
Quoted 2 lines
I think in P4 terms that may be something more akin to only providing the runtime API? I seem to recall they had some distinction...
There are several solutions out there (ex: TDI, P4runtime) - our API is netlink and those could be written on top of netlink, there's no controversy there.
So the starting point is defining the datapath using P4, generating the binary blob and whatever constraints needed using the vendor backend and for s/w equivalent generating the eBPF datapath.Quoted 6 lines
Quoted 3 lines
At the cost of this sounding controversial, i am going to call things like fdb, fib, etc which have fixed datapaths in the kernel "legacy". These "legacy" datapaths almost all the time have
The cynic in me sometimes thinks that the biggest problem with "legacy" protocols is that it's hard to make money on them :)
Quoted 8 lines
That's a big motivation without a doubt, but also there are people that want to experiment with things. One of the craziest examples we have is someone who created a P4 program for "in network calculator", essentially a calculator in the datapath. You send it two operands and an operator using custom headers, it does the math and responds with a result in a new header. By itself this program is a toy but it demonstrates that if one wanted to, they could have something custom in hardware and/or kernel datapath.
Jamal,
Quoted 8 lines
Quoted 2 lines
Given how long P4 has been around it's surprising that the best publicly available code example is "the network calculator" toy.
Come on Tom ;-> That was just an example of something "crazy" to demonstrate freedom. I can run that in any of the P4 friendly NICs today. You are probably being facetious - There are some serious publicly available projects out there, some of which I quote on the cover letter (like DASH).
Shameless plug. I have a more crazy example with bpf:
Quoted 6 lines
Quoted 4 lines
Hrm - this looks crazy interesting;-> Tempting. I guess to port this to P4 we'd need the sha256 in h/w (which most of these vendors have already). Is there any other acceleration would you need? Would have been more fun if you invented you own headers too ;->
Yeah, some way to do
sha256(sha256(at_some_fixed_packet_offset + 80 bytes))This part is straight forward.
Quoted 2 lines
is one thing. And the other is some way to compare that sha256 vs some hard-coded (difficulty) number (as a 256-byte uint).
The compiler may have issues with this comparison - will have to look (I am pretty sure it's fixable though).
Quoted 3 lines
But I have no
clue how well that maps into declarative p4 language. Most likely possible if you're saying that the calculator is possible?The calculator basically is written as a set of match-action tables. You parse your header, construct a key based on the operator field of the header (eg "+"), invoke an action which takes the operands from the
headers(eg "1" and "2"), the action returns youresults(3"). You stash the result in a new packet and send it back to the source.So my thinking is the computation you need would be modelled on an action.
Quoted 3 lines
I'm assuming that even sha256 can possibly be implemented in p4 without any extra support from the vendor? It's just a bunch of xors and rotations over a fix-sized input buffer.
[..]
Quoted 5 lines
True, and I think those would be fast. But if the h/w offers it as an interface why not.
It's not that you are running out of instruction space - and my memory is hazy - but iirc, there is sha256 support in the kernel Crypto API - does it not make sense to kfunc into that?
Oh yeah, that's definitely a better path if somebody were do to it "properly". It's still fun, though, to see how far we can push the bpf vm/verifier without using any extra helpers :-D
Quoted 496 lines
From: Paolo Abeni mailto:[email protected]
Sent: Thursday, February 29, 2024 9:14 AM
To: Jamal Hadi Salim mailto:[email protected]; mailto:[email protected]
Cc: mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; Chris Sommers mailto:[email protected]; mailto:[email protected]; mailto:[email protected]
Subject: Re: [PATCH net-next v12 00/15] Introducing P4TC (series 1)On Sun, 2024-02-25 at 11: 54 -0500, Jamal Hadi Salim wrote: > This is the first patchset of two. In this patch we are submitting 15 which > cover the minimal viable P4 PNA architecture. > >
__Descriptionof thesePatches__> > ZjQcmQRYFpfptBannerStart
This Message is From an External Sender: Use caution opening files, clicking links or responding to requests.ZjQcmQRYFpfptBannerEnd
Quoted 472 lines · Jamal Hadi Salim
This is the first patchset of two. In this patch we are submitting 15 which cover the minimal viable P4 PNA architecture.
__Descriptionof thesePatches__Patch #1 adds infrastructure for per-netns P4 actions that can be created on as need basis for the P4 program requirement. This patch makes a small incision into
act_api. Patches 2-4 are minimalist enablers for P4TC and have no effect the classical tc action (example patch#2 just increases the size of the action names from 16->64B).
Patch 5 adds infrastructure support for preallocation of dynamic actions.The core P4TC code implements several P4 objects.
1) Patch #6 introduces P4 data types which are consumed by the rest of the code
2) Patch #7 introduces the templating API. i.e. CRUD commands for templates
3) Patch #8 introduces the concept of templating Pipelines. i.e CRUD commands for P4 pipelines.
4) Patch #9 introduces the action templates and associated CRUD commands.
5) Patch #10 introduce the action runtime infrastructure.
6) Patch #11 introduces the concept of P4 table templates and associated CRUD commands for tables.
7) Patch #12 introduces runtime table entry infra and associated CU commands.
8) Patch #13 introduces runtime table entry infra and associated RD commands.
9) Patch #14 introduces interaction of eBPF to P4TC tables via kfunc.
10) Patch #15 introduces the TC classifier P4 used at runtime.Daniel, please look again at patch #15.
There are a few more patches (5) not in this patchset that deal with test cases, etc.
What is P4?
-----------The Programming Protocol-independent Packet Processors (P4) is an open source, domain-specific programming language for specifying data plane behavior.
The current P4 landscape includes an extensive range of deployments, products, projects and services, etc[9][12]. Two major NIC vendors, Intel[10] and AMD[11] currently offer P4-native NICs. P4 is currently curated by the Linux Foundation[9].
On why P4 - see small treatise here:[4].
What is P4TC?
-------------P4TC is a net-namespace aware P4 implementation over TC; meaning, a P4 program and its associated objects and state are attachend to a kernel _netns_ structure. IOW, if we had two programs across netns' or within a netns they have no visibility to each others objects (unlike for example TC actions whose kinds are "global" in nature or eBPF maps visavis bpftool).
P4TC builds on top of many years of Linux TC experiences of a netlink control path interface coupled with a software datapath with an equivalent offloadable hardware datapath. In this patch series we are focussing only on the s/w datapath. The s/w and h/w path equivalence that TC provides is relevant for a primary use case of P4 where some (currently) large consumers of NICs provide vendors their datapath specs in P4. In such a case one could generate specified datapaths in s/w and test/validate the requirements before hardware
acquisition(example [12]).Unlike other approaches such as TC Flower which require kernel and user space changes when new datapath objects like packet headers are introduced P4TC, with these patches, provides
_kerneland user space code changeindependence_. Meaning:
A P4 program describes headers, parsers, etc alongside the datapath processing; the compiler uses the P4 program as input and generates several artifacts which are then loaded into the kernel to manifest the intended datapath. In addition to the generated datapath, control path constructs are generated. The process is described further below in "P4TC Workflow".There have been many discussions and meetings within the community since about 2015 in regards to P4 over TC[2] and we are finally proving to the naysayers that we do get stuff done!
A lot more of the P4TC motivation is captured at:
https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7ZSCV8wc$
__P4TCArchitecture__The current architecture was described at netdevconf 0x17[14] and if you prefer academic conference papers, a short paper is available here[15].
There are 4 parts:
1) A Template CRUD provisioning API for manifesting a P4 program and its associated objects in the kernel. The template provisioning API uses netlink. See patch in part 2.
2) A Runtime CRUD+ API code which is used for controlling the different runtime behavior of the P4 objects. The runtime API uses netlink. See notes further down. See patch description later..
3) P4 objects and their control interfaces: tables, actions, externs, etc. Any object that requires control plane interaction resides in the TC domain and is subject to the CRUD runtime API. The intended goal is to make use of the tc semantics of skip_sw/hw to target P4 program objects either in s/w or h/w.
4) S/W Datapath code hooks. The s/w datapath is eBPF based and is generated by a compiler based on the P4 spec. When accessing any P4 object that requires control plane interfaces, the eBPF code accesses the P4TC side from #3 above using kfuncs.
The generated eBPF code is derived from [13] with enhancements and fixes to meet our requirements.
__P4TCWorkflow__The Development and instantiation workflow for P4TC is as follows:
A) A developer writes a P4 program, "myprog"
B) Compiles it using the P4C compiler[8]. The compiler generates 3 outputs:
a) A shell script which form template definitions for the different P4 objects "myprog" utilizes (tables, externs, actions etc). See #1 above..
b) the parser and the rest of the datapath are generated as eBPF and need to be compiled into binaries. At the moment the parser and the main control block are generated as separate eBPF program but this could change in the future (without affecting any kernel code). See #4 above.
c) A json introspection file used for the control plane (by iproute2/tc).
C) At this point the artifacts from #1,#4 could be handed to an operator (the operator could be the same person as the developer from #A, #B).
i) For the eBPF part, either the operator is handed an ebpf binary or source which they compile at this point into a binary. The operator executes the shell script(s) to manifest the functional "myprog" into the kernel.
ii) The operator instantiates "myprog" pipeline via the tc P4 filter to ingress/egress (depending on P4 arch) of one or more netdevs/ports (illustrated below as "block 22").
Example instantion where the parser is a separate action:
"tc filter add block 22 ingress protocol all prio 10 p4 pname myprog \
action bpf obj $PARSER.o section p4tc/parse \
action bpf obj $PROGNAME.o section p4tc/main"See individual patches in partc for more examples tc vs xdp etc. Also see section on "challenges" (further below on this cover letter).
Once "myprog" P4 program is instantiated one can start performing operations on table entries and/or actions at runtime as described below.
__P4TCRuntime ControlPath__The control interface builds on past tc experience and tries to get things right from the beginning (example filtering is separated from depending on existing object TLVs and made generic); also the code is written in such a way it is mostly lockless.
The P4TC control interface, using netlink, provides what we call a CRUDPS abstraction which stands for: Create, Read(get), Update, Delete, Subscribe, Publish. From a high level PoV the following describes a conformant high level API (both on netlink data model and code level):
Create(</path/to/object, DATA>+)
Read(</path/to/object>, [optional filter])
Update(</path/to/object>, DATA>+)
Delete(</path/to/object>, [optional filter])
Subscribe(</path/to/object>, [optional filter])Note, we _dont_ treat "dump" or "flush" as speacial. If "path/to/object" points to a table then a "Delete" implies "flush" and a "Read" implies dump but if it points to an entry (by specifying a key) then "Delete" implies deleting and entry and "Read" implies reading that single entry. It should be noted that both "Delete" and "Read" take an optional filter parameter. The filter can define further refinements to what the control plane wants read or deleted. "Subscribe" uses built in netlink event management. It, as well, takes a filter which can further refine what events get generated to the control plane (taken out of this patchset, to be re-added with consideration of [16]).
Lets show some runtime samples:
..create an entry, if we match ip address 10.0.1.2 send packet out eno1 tc p4ctrl create myprog/table/mytable \
dstAddr 10.0.1.2/32 actionsend_to_portparam port eno1..Batch create entries
tc p4ctrl create myprog/table/mytable \
entry dstAddr 10.1.1.2/32 actionsend_to_portparam port eno1 \
entry dstAddr 10.1.10.2/32 actionsend_to_portparam port eno10 \
entry dstAddr 10.0.2.2/32 actionsend_to_portparam port eno2..Get an entry (note "read" is interchangeably used as "get" which is a common semantic in tc):
tc p4ctrl read myprog/table/mytable \
dstAddr 10.0.2.2/32..dump mytable
tc p4ctrl read myprog/table/mytable..dump mytable for all entries whose key fits within 10.1.0.0/16 tc p4ctrl read myprog/table/mytable \
filter key/myprog/mytable/dstAddr = 10.1.0.0/16..dump all mytable entries which have an action
send_to_portwith param "eno1" tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1"The filter expression is powerful, f.e you could say:
tc p4ctrl get myprog/table/mytable \
filter param/act/myprog/send_to_port/port = "eno1" && \
key/myprog/mytable/dstAddr = 10.1.0.0/16It also works on built in metadata, example in the following case dumping entries from mytable that have seen activity in the last 10 secs:
tc p4ctrl get myprog/table/mytable \
filtermsecs_since< 10000Delete follows the same syntax as get/read, so for sake of brevity we won't show more example than how to flush mytable:
tc p4ctrl delete myprog/table/mytable
Mystery question: How do we achieve iproute2-kernel independence and how does "tc p4ctrl" as a cli know how to program the kernel given an arbitrary command line as shown above? Answer(s): It queries the compiler generated json file in "P4TC Workflow" #B.c above. The json file has enough details to figure out that we have a program called "myprog" which has a table "mytable" that has a key name "dstAddr" which happens to be type ipv4 address prefix. The json file also provides details to show that the table "mytable" supports an action called "
send_to_port" which accepts a parameter "port" of type netdev (see the types patch for all supported P4 data types). All P4 components have names, IDs, and types - so this makes it very easy to map into netlink.
Once user space tc/p4ctrl validates the human command input, it creates standard binary netlink structures (TLVs etc) which are sent to the kernel. See the runtime table entry patch for more details.
__P4TCDatapath__The P4TC s/w datapath execution is generated as eBPF. Any objects that require control interfacing reside in the "P4TC domain" and are controlled via netlink as described above. Per packet execution and state and even objects that do not require control interfacing (like the P4 parser) are generated as eBPF.
A packet arriving on s/w ingress of any of the ports on block 22 will first be exercised via the (generated eBPF) parser component to extract the headers (the ip destination address in labelled "dstAddr" above).
The datapath then proceeds to use "dstAddr", table ID and pipeline ID as a key to do a lookup in myprog's "mytable" which returns the action params which are then used to execute the action in the eBPF datapath (eventually sending out packets to eno1).
On a table miss, mytable's default miss action (not described) is executed.
__Testing__Speaking of testing - we have 2-300 tdc test cases (which will be in the second patchset).
These tests are run on our CICD system on pull requests and after commits are approved. The CICD does a lot of other tests (more since v2, thanks to Simon's input)including:
checkpatch, sparse, smatch, coccinelle, 32 bit and 64 bit builds tested on both X86, ARM 64 and emulated BE via qemu s390. We trigger performance testing in the CICD to catch performance regressions (currently only on the control path, but in the future for the datapath).
Syzkaller runs 24/7 on dedicated hardware, originally we focussed only on memory sanitizer but recently added support for concurrency sanitizer. Before main releases we ensure each patch will compile on its own to help in git bisect and run the xmas tree tool. We eventually put the code via coverity.In addition we are working on enabling a tool that will take a P4 program, run it through the compiler, and generate permutations of traffic patterns via symbolic execution that will test both positive and negative datapath code paths. The test generator tool integration is still work in progress.
Also: We have other code that test parallelization etc which we are trying to find a fit for in the kernel tree's testing infra.
__References__[1]https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/p4-conference-2023/2023P4WorkshopP4TC.pdf__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7bPf6Tk4$ [2]https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md*historical-perspective-for-p4tc__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7LkM5QJk$ [3]https://urldefense.com/v3/__https://2023p4workshop.sched.com/event/1KsAe/p4tc-linux-kernel-p4-implementation-approaches-and-evaluation__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O73gpmAKE$ [4]https://urldefense.com/v3/__https://github.com/p4tc-dev/docs/blob/main/why-p4tc.md*so-why-p4-and-how-does-p4-help-here__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7fvy73gU$ [5]https://urldefense.com/v3/__https://lore.kernel.org/netdev/[email protected]/T/*mf59be7abc5df3473cff3879c8cc3e2369c0640a6__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7rJJDxSc$ [6]https://urldefense.com/v3/__https://lore.kernel.org/netdev/[email protected]/T/*m783cfd79e9d755cf0e7afc1a7d5404635a5b1919__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O74EMrBVI$ [7]https://urldefense.com/v3/__https://lore.kernel.org/netdev/[email protected]/T/*ma8c84df0f7043d17b98f3d67aab0f4904c600469__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7-6T3BD8$ [8]https://urldefense.com/v3/__https://github.com/p4lang/p4c/tree/main/backends/tc__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7EsGj_yE$ [9]https://urldefense.com/v3/__https://p4.org/__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7MA51wp8$ [10]https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/products/details/network-io/ipu/e2000-asic.html__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7HaJpkWg$ [11]https://urldefense.com/v3/__https://www.amd.com/en/accelerators/pensando__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7u8agJlY$ [12]https://urldefense.com/v3/__https://github.com/sonic-net/DASH/tree/main__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O77NF6LU0$ [13]https://urldefense.com/v3/__https://github.com/p4lang/p4c/tree/main/backends/ebpf__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7Hn8dxDI$ [14]https://urldefense.com/v3/__https://netdevconf.info/0x17/sessions/talk/integrating-ebpf-into-the-p4tc-datapath.html__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7dDtnoik$ [15]https://urldefense.com/v3/__https://dl.acm.org/doi/10.1145/3630047.3630193__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7zb87EuI$ [16]https://urldefense.com/v3/__https://lore.kernel.org/netdev/[email protected]/__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7mLYrgl8$ [17.a]https://urldefense.com/v3/__https://netdevconf.info/0x13/session.html?talk-tc-u-classifier__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7qSaba8A$
[17.b]man tc-u32
[18]man tc-pedit
[19] https://urldefense.com/v3/__https://lore.kernel.org/netdev/[email protected]/T/*m86e71743d1d83b728bb29d5b877797cb4942e835__;Iw!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7Uc3-7Vg$
[20.a] https://urldefense.com/v3/__https://netdevconf.info/0x16/sessions/talk/your-network-datapath-will-be-p4-scripted.html__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7YIAkKuc$
[20.b] https://urldefense.com/v3/__https://netdevconf.info/0x16/sessions/workshop/p4tc-workshop.html__;!!I5pVk4LIGAfnvw!nUhTLe4nLBPIqlW8u7cJYnSCbFNQSDW318aG2nqxbMZuptPK41snJiHein5mNazAYWIwAfJ3o8O7_8aEvEI$--------
HISTORY
--------Changes in Version 12
----------------------0) Introduce back 15 patches (v11 had 5)
1) From discussions with Daniel:
i) Remove the XDP programs association alltogether. No refcounting. nothing. ii) Remove prog type tc - everything is now an ebpf tc action.2) s/PAD0/__pad0/g. Thanks to Marcelo.
3) Add extack to specify how many entries (N of M) specified in a batch for any of requested Create/Update/Delete succeeded. Prior to this it would only tell us the batch failed to complete without giving us details of which of M failed. Added as a debug aid.
Changes in Version 11
----------------------
1) Split the series into two. Original patches 1-5 in this patchset. The rest will go out after this is merged.2) Change any references of IFNAMSIZ in the action code when referencing the action name size to ACTNAMSIZ. Thanks to Marcelo.
Changes in Version 10
----------------------
1) A couple of patches from the earlier version were clean enough to submit, so we did. This gave us room to split the two largest patches each into two. Even though the split is not git-bisactable and really some of it didn't make much sense (eg spliting a create, and update in one patch and delete and get into another) we made sure each of the split patches compiled independently. The idea is to reduce the number of lines of code to review and when we get sufficient reviews we will put the splits together again. See patch #12 and #13 as well as patches #7 and #8).2) Add more context in patch 0. Please READ!
3) Added dump/delete filters back to the code - we had taken them out in the earlier patches to reduce the amount of code for review - but in retrospect we feel they are important enough to push earlier rather than later.
Changes In version 9
---------------------1) Remove the largest patch (externs) to ease review.
2) Break up action patches into two to ease review bringing down the patches that need more scrutiny to 8 (the first 7 are almost trivial).
3) Fixup prefix naming convention to
p4tc_xxxfor uapi andp4a_xxxfor actions to provide consistency(Jiri).4) Silence sparse warning "was not declared. Should it be static?" for kfuncs by making them static. TBH, not sure if this is the right solution but it makes sparse happy and hopefully someone will comment.
Changes In Version 8
---------------------1) Fix all the patchwork warnings and improve our ci to catch them in the future
2) Reduce the number of patches to basic max(15) to ease review.
Changes In Version 7
-------------------------0) First time removing the RFC tag!
1) Removed XDP cookie. It turns out as was pointed out by Toke(Thanks!) - that using bpf links was sufficient to protect us from someone replacing or deleting a eBPF program after it has been bound to a netdev.
2) Add some reviewed-bys from Vlad.
3) Small bug fixes from v6 based on testing for ebpf.
4) Added the counter extern as a sample extern. Illustrating this example because it is slightly complex since it is possible to invoke it directly from the P4TC domain (in case of direct counters) or from eBPF (indirect counters). It is not exactly the most efficient implementation (a reasonable counter impl should be per-cpu).
Changes In RFC Version 6
-------------------------1) Completed integration from scriptable view to eBPF. Completed integration of externs integration.
2) Small bug fixes from v5 based on testing.
Changes In RFC Version 5
-------------------------1) More integration from scriptable view to eBPF. Small bug fixes from last integration.
2) More streamlining support of externs via kfunc (create-on-miss, etc)
3) eBPF linking for XDP.
There is more eBPF integration/streamlining coming (we are getting close to conversion from scriptable domain).
Changes In RFC Version 4
-------------------------1) More integration from scriptable to eBPF. Small bug fixes.
2) More streamlining support of externs via kfunc (one additional kfunc).
3) Removed per-cpu scratchpad per Toke's suggestion and instead use XDP metadata.
There is more eBPF integration coming. One thing we looked at but is not in this patchset but should be in the next is use of eBPF link in our loading (see "challenge #1" further below).
Changes In RFC Version 3
-------------------------These patches are still in a little bit of flux as we adjust to integrating eBPF. So there are small constructs that are used in V1 and 2 but no longer used in this version. We will make a V4 which will remove those. The changes from V2 are as follows:
1) Feedback we got in V2 is to try stick to one of the two modes. In this version we are taking one more step and going the path of mode2 vs v2 where we had 2 modes.
2) The P4 Register extern is no longer standalone. Instead, as part of integrating into eBPF we introduce another kfunc which encapsulates Register as part of the extern interface.
3) We have improved our CICD to include tools pointed to us by Simon. See "Testing" further below. Thanks to Simon for that and other issues he caught. Simon, we discussed on issue [7] but decided to keep that log since we think it is useful.
4) A lot of small cleanups. Thanks Marcelo. There are two things we need to re-discuss though; see: [5], [6].
5) We removed the need for a range of IDs for dynamic actions. Thanks Jakub.
6) Clarify ambiguity caused by smatch in an if(A) else if(B) condition. We are guaranteed that either A or B must exist; however, lets make smatch happy. Thanks to Simon and Dan Carpenter.
Changes In RFC Version 2
-------------------------Version 2 is the initial integration of the eBPF datapath.
We took into consideration suggestions provided to use eBPF and put effort into analyzing eBPF as datapath which involved extensive testing.
We implemented 6 approaches with eBPF and ran performance analysis and presented our results at the P4 2023 workshop in Santa Clara[see: 1, 3] on each of the 6 vs the scriptable P4TC and concluded that 2 of the approaches are sensible (4 if you account for XDP or TC separately).Conclusions from the exercise: We lose the simple operational model we had prior to integrating eBPF. We do gain performance in most cases when the datapath is less compute-bound.
For more discussion on our requirements vs journeying the eBPF path please scroll down to "Restating Our Requirements" and "Challenges".This patch set presented two modes.
mode1: the parser is entirely based on eBPF - whereas the rest of the SW datapath stays as _scriptable_ as in Version 1.
mode2: All of the kernel s/w datapath (including parser) is in eBPF.Quoted 5 lines
The key ingredient for eBPF, that we did not have access to in the past, is kfunc (it made a big difference for us to reconsider eBPF).
In V2 the two modes are mutually exclusive (IOW, you get to choose one or the other via
Kconfig).I think/fear that this series has a "quorum" problem: different voices raises opposition, and nobody (?) outside the authors supported the code and the feature.
Quoted 8 lines
Could be the missing of H/W offload support in the current form the root cause for such lack support? Or there are parties interested that have been quite so far?
Thanks,
Paolo
Hi Paolo, thanks. I am one of those "parties interested that have been quite so far."
I wanted to voice my staunch support for accepting P4TC into the kernel. None of the present objections in the various threads reduce my enthusiasm. I find the following aspects most compelling:
- Performant, highly functional, pure-SW P4 dataplane
- Near-ubiquitous availability on all platforms, once it's upstreamed. Saves having to install a bunch of other p4 ecosystem tools, lowers the barrier to entry, and increases the likelihood an application can run on any platform.
- larger dev community. Anything added to the Linux kernel benefits from a large, thriving community, vast and rigorous regression testing, long-term support, etc.
- well-conceived CRUDX northbound API and clever use of existing well-understood netlink, easy to overlay other northbound APIs such as TDI (Table driven interface) used in IPDK; P4Runtime gRPC API; etc.
- integration with popular and well-understood tc provides a good impedance match for users.
- extensibility, ability to add externs, and interface to eBPF. The ability to add externs is especially compelling. It is not easy to do so in current backends such as bmv2, P4DPDK or p4-ebpf.
- roadmap to hardware offload for even greater performance. Even _without_ offload, the above benefits justify it in my mind. There are many applications for a pure-SW P4 dataplane, both in userland like P4DPDK, and the proposed P4TC - running as part of the kernel is _exciting_. Vendors have already voiced their support for offload and this initial set of patches paves the way and lets the community benefit from it and start to make it better, now.
It is possible the detractors of P4TC are not active P4 users, so I hope to provide a bit of perspective. Besides the pioneering switch ASIC (Tofino) use-cases which provided the initial impetus, P4 is used extensively in at least two commercial IPUs/DPUs. In addition, there are multiple toolchains to run P4 code on FPGAs. The dream is to write P4 code which can be run in a scalable fashion on a range of targets. It shouldn’t be necessary to “prove” P4 is worthy, those who’ve already embraced it know this.
There are several use-cases for a SW implementation of a P4 dataplane, including behavioral modeling and production uses. P4 allows one to write core functionality which can run on multiple platforms: pure SW, FPGAs, offload NICs/DPUs/IPUs, switch ASICs.
Behavioral modeling of a pipeline using P4:
- The SONiC-DASH project (https://github.com/sonic-net/DASH) is a thriving, multi-vendor collaboration which specifies advanced, high-performance features to accelerate datacenter services. These overlay services are specified using a P4 program which allows all concerned to agree on the packet pipeline and even the control-plane APIs (using SAI, the Switch Abstraction Interface). The actual implementation on a vendor's offload device (DPU/IPU) may or may not use any of the reference P4 code, but that is not important. What is important is that we specify the dataplane in P4, and execute it on the bmv2 backend in a container. We run conformance and regression suites with standard test vectors, which can also be run against actual production implementations to verify compliance. The bmv2 backend has many limitations, including performance and difficulty to extend its functionality. As a major contributor to this project, I am helping to explore alternatives.
- Large-scale cloud-service providers use P4 extensively as a dataplane (fabric switch) modeling language. One of the driving use-cases in the P4-API working group (I’m a co-chair) is to control SDN switches using P4-Runtime. The switches’ pipelines are modeled in P4 by some users, similar to the DASH use-case. Having a performant, pure-SW implementation is invaluable for modeling and simulation.
Running P4 code in pure SW for production use-cases (not just modeling):
There are many use-cases for running a custom dataplane written in P4. The productivity of P4 code cannot be overstated. With the right framework, P4 apps can be developed (and controlled/managed) in literally hours. It is much more productive than writing, say c or eBPF. I can do all three, and P4 is way more productive for certain applications.
In conclusion, I hope we can upstream P4-TC soon. Please move this forward with all due speed. Thanks!
Chris Sommers
Keysight Technologies