net: flow_dissector: opt-in byte-identical fast paths for common shapes
Subsystems (4)
cover net: flow_dissector: opt-in byte-identical fast paths for common shapes
G'day,
__skb_flow_dissect() parses packet headers with a generic protocol-graph parser. The graph walk is what makes the dissector so flexible, but most machines spend that flexibility on the same traffic all day long: plain eth + IPv4/IPv6 + TCP/UDP, perhaps with a VLAN tag or a tunnel header in front. The intuition behind this series is that the common shapes can be dissected by straight-line code instead, without giving up the graph walk for everything else.
Concretely: the series adds an opt-in, static-branch-gated fast path per packet shape, covering the common cases and the IP-in-IP/GRE encapsulations. Each shape has its own gate under /proc/sys/net/flow_dissector/, every gate is disabled by default, and when a gate is off its added cost is one not-taken branch per dissect. The gates are manual in this series; a follow-up RFC will propose an optional auto mode that flips them from the measured traffic mix (via the patch-8 counters), so nobody has to hand-tune seven knobs (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre). Nothing
here depends on it -- it is where this is heading, not a prerequisite.
The series is byte-identical by contract: for any packet, the fast path either writes exactly the flow_keys bytes the slow path would have written, or it returns false and the slow path runs. That contract is enforced in-tree by a KUnit equivalence suite (patch 10).
Patch 1 stands slightly apart: it is a small win for the existing path -- today every dissect takes rcu_read_lock() and does two dependent rcu_dereference() loads to look for a netns BPF flow dissector program even when none is attached anywhere; a static key (mirroring bpf_sk_lookup_enabled) skips all of it in the common no-program case.
Patches 2-7 are the byte-identical common shapes (eth+IPv4/IPv6 x TCP/UDP, VLAN/QinQ, PPPoE, single MPLS, IP-in-IP, plain GRE). Patch 8 adds per-shape counters (/proc/net/flow_dissector_stats); patch 9 bounds the fast-path tunnel recursion at MAX_FLOW_DISSECT_HDRS.
A companion series (posted separately) builds on this one to add opt-in inner-flow descent for the UDP tunnels (VXLAN, Geneve, GTP-U, FOU/GUE) -- that one changes hashing when enabled, so it is a separate policy discussion; this series is purely a transparent optimisation.
Why not just optimise the existing loop
========================================
The slow path is a generic graph walk by design: one loop, one proto/nhoff state machine, re-entered per header, able to express every protocol relationship the dissector knows. That generality is the cost -- per-header dispatch, state saves, and branchy key-writing that the compiler cannot flatten because the shape is only known at runtime. A per-shape straight-line function knows the whole shape at compile time, so the loads, checks and key writes schedule as one block. The two approaches are complementary: the loop stays the single source of semantic truth for every shape, and a handful of high-volume shapes get a verified shortcut.
Performance summary
===================
Measured across 3 ISAs and 8 microarchitectures in total, with byte-identical verification per shape (userspace A/B harness compiling the dissector both ways from the same source). Two instruments, each run on 7 of the 8 microarchitectures: "allshapes" compiles every fast path in (the realistic icache footprint) and measures each shape in isolation, on x86 Zen1/Zen2/Skylake, ARM Cortex-A53/A72/A76 and RISC-V X60; "isolated A/B" compiles one shape's path against the unmodified dissector (the upper bound for that shape), on the same set with Haswell in place of Zen 1.
Reduction in dissector CPU cost, allshapes microbench. Each row is the range across its 7 microarchitectures, worst to best:
shape worst best
------ ------ ------
eth_ip -4.7% -31.6%
vlan -18.1% -38.5%
qinq -16.1% -39.8%
pppoe -8.6% -28.0%
mpls -6.7% -15.0%
ipip -36.8% -47.0% largest: slow path re-parses inner IP
gre (byte-identical descent family, tracks ipip; verified
by the KUnit suite rather than separately benchmarked)Isolated A/B, eth + IPv4 + TCP (same source compiled two ways), nanoseconds per packet, lower is better:
microarchitecture slow fast delta
------------------------------------ ------ ------ -----
x86 Zen 2 (Threadripper PRO 3945WX) 12.4 6.6 -47%
x86 Skylake (Core i9-10885H) 10.6 5.6 -47%
x86 Haswell (Celeron 2955U) 35.9 18.0 -50%
ARM Cortex-A53 119.0 61.3 -48%
ARM Cortex-A72 42.4 19.1 -55%
ARM Cortex-A76 19.4 9.3 -52%
RISC-V SpacemiT X60 98.2 49.3 -50%The isolated dissect costs 47-55% less per packet, consistent across the three ISAs and the seven microarchitectures measured with that instrument; the relative reduction is largest on in-order cores, indicating the saving is fewer instructions rather than a microarchitectural artifact. IPv6 uses the same gate and is byte-identical (x86 -39%, RISC-V -25%).
flow_keys -> skb->hash via skb_get_hash() is the flow identity behind RPS/RFS/aRFS, the fq / fq_codel / cake fairness queues, bonding/LAG transmit hashing and more; those callers pass no restricting flags, so each of them gets the per-dissect saving at once. Dissector callers that pass STOP flags (ECMP multipath, xfrm policy lookup) or run their own dissector instances (tc-flower) deliberately stay on the slow path -- unchanged, not regressed.
Counters (patch 8) are the one part with a cost in the all-gates-off
default: one this_cpu_inc per dissect plus one per matched shape. Measured on a CPU-bound pktgen soak that is +0.74% dissector time (+0.44 sigma -- within run-to-run noise). The gates themselves stay
free: off means one not-taken static branch.
Relationship to the BPF flow dissector
======================================
The fast path is invoked strictly after the netns BPF flow dissector hook in __skb_flow_dissect(): if a program is attached and returns any verdict other than BPF_FLOW_DISSECTOR_CONTINUE, the function returns before flow_dissect_fast() is reached. Attached BPF dissectors therefore always take precedence, and a system running one sees bit-for-bit unchanged behaviour from this series (patch 1 additionally makes its program lookup cheaper for everyone else). Dissects fully handled by a BPF program are deliberately not counted in /proc/net/flow_dissector_stats (patch 8).
The gates are global (a static key patches code shared by every netns); per-netns dissector policy already has a mechanism -- the netns BPF flow dissector -- and it keeps full precedence here.
Maintenance burden: why two paths won't diverge
===============================================
The slow path remains the single source of semantic truth and the fast path is verified against it, not vice versa. Three layers enforce that:
1. The eligibility gate: only the two standard dissectors and a flag subset ever enter flow_dissect_fast(); custom tc-flower dissector instances are structurally unaffected.
2. Every shape is a default-off static branch: a kernel with gates off executes the not-taken-branch slow path only.
3. The KUnit equivalence suite (patch 10) dissects a corpus of eligible shapes, deliberate fast-path misses, truncations at every byte boundary, and skb-mode entry conditions through both paths and memcmp()s the resulting struct flow_keys (and their flow_hash_from_keys()). Future divergence is a failing test, not a silent behaviour change.
The test earns its place: during development it caught real divergences the out-of-tree A/B harness had missed -- the MPLS fast path skipping the slow path's out_good terminal writes, the 4in6 descent not reproducing the outer-IPv6 residue, and -- caught by a hardware boot smoke with counter positive-controls, then pinned by a regression test -- the dispatcher rejecting the STOP_AT_FLOW_LABEL flag that skb_get_hash() passes on every dissect, which had made the fast path unreachable from RPS/RFS/fq/cake entirely. All fixed in this posting.
Robustness and safety limits
============================
The fast path descends into IP-in-IP and GRE by recursion. The slow path bounds header descents at MAX_FLOW_DISSECT_HDRS; patch 9 gives the fast path the same bound, so a crafted nested chain falls back to the slow path at the same depth rather than driving one C stack frame per ~20 bytes of linear header. The KUnit suite covers the bound boundary, the non-linear-skb fallback, a gates-off negative (the fast path must never run when disabled, observed via the counters), and a seeded fuzzer as a general guard.
Patch 11 adds Documentation/networking/flow_dissector.rst -- there was no general flow_dissector overview in the tree. It explains what the dissector is and who consumes its output, then documents this series' fast paths: the byte-identical contract and the break-even model for when a gate is worth enabling. The per-knob reference stays in Documentation/admin-guide/sysctl/net.rst; the new doc links to it.
On tooling: this series was developed with substantial LLM assistance (Claude, Anthropic) -- identifying the optimisation opportunity, drafting the patches, and building and running the test matrix. Every patch carries an Assisted-by: trailer in the Documentation/process/coding-assistants.rst format. All of it was human-reviewed and human-measured: the performance numbers come from months of runs across the hardware above, and the byte-identical contract is enforced mechanically by the in-tree KUnit suite rather than by trust in either the tool or the author.
This series grew out of work on Tom Herbert's XDP2 parser project:
porting the kernel flow dissector into XDP2's userspace benchmarking harness, to compare parsers on equal footing, is what made the per-shape cost of the generic loop visible -- this opt-in fast path is the kernel-side result of that analysis.
Testing: 3 ISAs / 8 uarches (x86 Zen1/Zen2/Skylake/Haswell, ARM Cortex-A53/A72/A76, RISC-V X60), in-tree KUnit (patch 10: 55 tests, also run under KASAN+UBSAN; per-patch W=1, sparse and smatch clean; CONFIG_NET_FOU n/m/y config matrix for the companion series) and out-of-tree A/B. Supporting data (flow-distribution study, per-shape/per-arch matrix, break-even derivation) lives in the author's XDP2 fork: https://github.com/randomizedcoder/xdp2 , under perf-results/ .
Dave Seddon (11):
net: flow_dissector: gate BPF program lookup behind a static key
net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}
net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP
net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} +
TCP/UDP
net: flow_dissector: add fast-path for single MPLS label + IP
net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 /
6in4)
net: flow_dissector: add byte-identical fast-path for plain GRE inner
net: flow_dissector: per-shape counters +
/proc/net/flow_dissector_stats
net: flow_dissector: bound fast-path tunnel recursion
net: flow_dissector: add KUnit fast/slow path equivalence tests
Documentation: networking: add flow_dissector overview and fast-path
guide
Documentation/admin-guide/sysctl/net.rst | 178 ++- Documentation/bpf/index.rst | 1 + Documentation/bpf/prog_flow_dissector.rst | 2 + Documentation/networking/flow_dissector.rst | 133 +++ Documentation/networking/index.rst | 1 + include/linux/skbuff.h | 1 + include/net/flow_dissector.h | 34 + kernel/bpf/net_namespace.c | 17 +- net/Kconfig | 12 + net/core/Makefile | 1 + net/core/flow_dissector.c | 1116 ++++++++++++++++++- net/core/flow_dissector_test.c | 1053 +++++++++++++++++ 12 files changed, 2500 insertions(+), 49 deletions(-) create mode 100644 Documentation/networking/flow_dissector.rst create mode 100644 net/core/flow_dissector_test.c
base-commit: f6f3b36c15ed44de1fbb44e645e4fae8c4a4453e
Signature · 1 line
2.54.0
PATCH 1/11 net: flow_dissector: gate BPF program lookup behind a static key
__skb_flow_dissect() takes rcu_read_lock(), resolves the netns from skb->dev/skb->sk, and does two dependent rcu_dereference() loads of the netns-BPF run_array (init_net's first - a shared cacheline - then the packet netns') on every dissect, even though the overwhelmingly common case is that no BPF flow dissector program is attached anywhere on the system.
Add a static key that counts attached flow dissector programs and skip the whole lookup block when it is zero, exactly as bpf_sk_lookup_enabled does for the sibling NETNS_BPF_SK_LOOKUP attach type (also see cgroup_bpf_enabled_key[] for the same pattern per cgroup-BPF attach type).
The sk_lookup type is link-only, so its inc/dec sit only in netns_bpf_attach_type_need()/unneed(), which the link attach/release paths already call. The flow dissector type also supports legacy BPF_PROG_ATTACH, so additionally inc on a fresh (non-replacement) prog attach, dec in __netns_bpf_prog_detach(), and dec for a remaining legacy prog in netns_bpf_pernet_pre_exit() (links are already handled by the existing per-link unneed loop there). All these paths run under netns_bpf_mutex.
As with sk_lookup, there is a brief window where a just-attached program can miss a concurrent dissect (key flips after rcu_assign_pointer) or a just-detached one can cause a wasted lookup; both are benign and match the existing semantics of the sibling keys.
DEBUG_NET_WARN_ON_ONCE(!net) moves inside the gated block: it exists to catch a missing netns for the BPF hook, which is only meaningful when a program can actually be attached.
With the key disabled this removes the rcu_read_lock/unlock pair, the netns resolution, and both run_array loads from the per-packet path.
--- include/linux/skbuff.h | 1 + kernel/bpf/net_namespace.c | 17 +++++- net/core/flow_dissector.c | 104 +++++++++++++++++++++---------------- 3 files changed, 75 insertions(+), 47 deletions(-)
include/linux/skbuff.h +1 -0
@@ -1588,6 +1588,7 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector, unsigned int key_count);struct bpf_flow_dissector ; +extern struct static_key_false netns_bpf_flow_dissector_enabled;u32 bpf_flow_dissect (struct bpf_prog *prog,struct bpf_flow_dissector *ctx,__be16 proto,int nhoff,int hlen,unsigned int flags);
kernel/bpf/net_namespace.c +16 -1
@@ -28,6 +28,9 @@ DEFINE_MUTEX(netns_bpf_mutex);static void netns_bpf_attach_type_unneed (enum netns_bpf_attach_type type) {switch (type) { +case NETNS_BPF_FLOW_DISSECTOR : +static_branch_dec (&netns_bpf_flow_dissector_enabled); +break ;#ifdef CONFIG_INET case NETNS_BPF_SK_LOOKUP :static_branch_dec (&bpf_sk_lookup_enabled);
@@ -41,6 +44,9 @@ static void netns_bpf_attach_type_unneed(enum netns_bpf_attach_type type)static void netns_bpf_attach_type_need (enum netns_bpf_attach_type type) {switch (type) { +case NETNS_BPF_FLOW_DISSECTOR : +static_branch_inc (&netns_bpf_flow_dissector_enabled); +break ;#ifdef CONFIG_INET case NETNS_BPF_SK_LOOKUP :static_branch_inc (&bpf_sk_lookup_enabled);
@@ -352,6 +358,11 @@ int netns_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog) net->bpf.progs[type] = prog;if (attached)bpf_prog_put (attached); +else +/* Mark attach point as used on a fresh attach; a +* replacement keeps the existing count. +*/ +netns_bpf_attach_type_need (type); out_unlock:mutex_unlock (&netns_bpf_mutex);
@@ -373,6 +384,8 @@ static int __netns_bpf_prog_detach(struct net *net, attached = net->bpf.progs[type];if (!attached || attached != old)return -ENOENT ; +/* Mark attach point as unused */ +netns_bpf_attach_type_unneed (type);netns_bpf_run_array_detach (net, type); net->bpf.progs[type] =NULL ;bpf_prog_put (attached);
@@ -546,8 +559,10 @@ static void __net_exit netns_bpf_pernet_pre_exit(struct net *net) net_link->net =NULL ;/* auto-detach link */ netns_bpf_attach_type_unneed (type); } -if (net->bpf.progs[type]) +if (net->bpf.progs[type]) { +netns_bpf_attach_type_unneed (type);bpf_prog_put (net->bpf.progs[type]); + } }mutex_unlock (&netns_bpf_mutex); }
net/core/flow_dissector.c +58 -46
123 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 2/11 net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}
In reply to Dave SeddonA small targeted optimisation to __skb_flow_dissect(). A new flow_dissect_fast() dispatcher routes the standard flow_keys dissector calls through per-shape straight-line extractors for the common eth + IPv4/IPv6 + TCP/UDP packet shapes. On any miss, falls back to the existing graph walk. Output is byte-identical to the slow path for the shapes handled.
This is the parent of the fast-path series. Subsequent patches add per-shape extensions for single VLAN, QinQ, PPPoE, MPLS, IP-in-IP and GRE. Each patch ships its own per-shape sysctl in the new ``/proc/sys/net/flow_dissector/`` subtree; this patch establishes the subtree and registers the first entry, ``eth_ip``.
The fast path is invoked strictly after the netns BPF flow dissector
hook: if a BPF program is attached and returns a verdict other than BPF_FLOW_DISSECTOR_CONTINUE, __skb_flow_dissect() returns before flow_dissect_fast() is reached. Attached BPF dissectors therefore always take precedence and observe no behaviour change from this series.
- What this does. Adds an opt-in fast-path for eth + IPv4 (IHL=5, not fragmented) + TCP/UDP and eth + IPv6 (no extension headers) + TCP/UDP. When the per-shape sysctl is enabled and a packet matches, the helper fills struct flow_keys byte-identically to the slow path and returns. On any miss it defers to the existing graph walk. Only the two standard dissectors are eligible (flow_keys_dissector and flow_keys_dissector_symmetric); custom dissectors and slow-path-only flags defer unconditionally.
- How it gates. A new DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key), externed in <net/flow_dissector.h>, toggled by /proc/sys/net/flow_dissector/eth_ip with proc_do_static_key. When the sysctl is 0 the dispatcher's per-shape case is a forward not- taken JMP — same slow-path cost as today. When 1, the helper runs inline.
- Namespace. The sysctl subtree is /proc/sys/net/flow_dissector/ (registered by a new late_initcall in net/core/flow_dissector.c via register_net_sysctl(&init_net, "net/flow_dissector", tbl) — same shape as net/mpls, net/bridge).
- Forward declarations. flow_dissect_fast_ipv4 / _ipv6 are forward- declared at the top of the file (after includes). Subsequent patches add flow_dissect_fast_vlan etc. that call these helpers; anchoring the declarations at top of file makes the patch insertion order robust against kernel-version line-number drift.
- Cost model. On a miss, dispatcher overhead is one ethertype switch (~5 cycles) plus the per-case static_branch test (~not-taken JMP when 0). On a hit, the straight-line body saves ~30 ns/skb of graph-walk cost on the common shape (Zen 2 / Cortex-A76; see the cover letter for the cross-uarch microbench table).
- Output contract. Byte-identical with the slow path for the eligible shape. The IPv6 fast-path defers on any non-zero flow
label: callers passing FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- skb_get_hash(), i.e. RPS/RFS, fq, fq_codel, cake -- make the slow path stop at a non-zero label, and dissectors requesting the label key write it; deferring on the label itself keeps byte-identity for every combination without threading flags into the helpers, and zero-label IPv6 stays on the fast path. Fragmented IPv4 packets defer. Custom dissectors defer, as do FLOW_DISSECTOR_F_* flags other than PARSE_1ST_FRAG and STOP_AT_FLOW_LABEL. Admitting STOP_AT_FLOW_LABEL is what makes the fast path reachable from skb_get_hash() at all -- the kernel's main dissection consumer.
--- Documentation/admin-guide/sysctl/net.rst | 27 ++- include/net/flow_dissector.h | 5 + net/core/flow_dissector.c | 273 ++++++++++++++++++++++- 3 files changed, 302 insertions(+), 3 deletions(-)
Documentation/admin-guide/sysctl/net.rst +26 -1
@@ -460,7 +460,32 @@ historical importance. Default: 0 -2. /proc/sys/net/unix - Parameters for Unix domain sockets +2. /proc/sys/net/flow_dissector - Flow dissector fast-path per-shape sysctls +--------------------------------------------------------------------------- + +The flow dissector is invoked on every packet that needs a flow hash +(RPS, RFS, ECMP, sch_cake, cls_flower, ...). For common shapes +(Eth+IPv4/IPv6+TCP/UDP, plus per-shape extensions for VLAN/QinQ, +PPPoE, MPLS, IP-in-IP and GRE added by this series), an opt-in +fast-path bypasses the slow-path graph walk and writes +``struct flow_keys`` directly. Each shape is gated by its own static +key + sysctl so operators enable only what their deployment uses. + +All defaults are 0 (off). Flipping any of these on takes effect +immediately; no kernel rebuild needed. + +eth_ip +~~~~~~ + +Eth + IPv4 (IHL=5, no fragmentation) + TCP/UDP, and Eth + IPv6 (no +extension headers) + TCP/UDP. Output is byte-identical to the slow +path on the eligible shape; non-matching packets fall through to the +slow path unchanged. The dispatcher costs one not-taken JMP per call +when this is 0. + +Default: 0 + +3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- There is only one file in this directory.
include/net/flow_dissector.h +5 -0
@@ -425,6 +425,11 @@ __be32 flow_get_u32_dst(const struct flow_keys *flow);extern struct flow_dissector flow_keys_dissector;extern struct flow_dissector flow_keys_basic_dissector; +/* Per-shape fast-path static-branch keys, toggled via +* /proc/sys/net/flow_dissector/<shape>. All default off. +*/ +extern struct static_key_false flow_dissector_eth_ip_key; +/* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a
net/core/flow_dissector.c +271 -2
300 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 3/11 net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP
In reply to Dave SeddonExtension of the parent series covering the VLAN-tagged shapes: a single ETH_P_8021Q or ETH_P_8021AD tag, or two stacked tags (the canonical QinQ S-tag + C-tag), in front of IPv4/IPv6 + TCP/UDP. Adds flow_dissect_fast_vlan() that consumes the tag(s) and dispatches to the existing flow_dissect_fast_ipv4 / _ipv6 helpers on the inner ethertype.
- What this covers. VLAN-tagged Eth (8021Q or 8021AD, depth 1 or
2) + IPv4 or IPv6 (per parent's shape constraints) + TCP/UDP. Both hardware-stripped (TCI in skb metadata, inner ethertype in skb->protocol; only ever the outermost tag) and in-band (4-byte vlan_hdr at data + nhoff) tag forms.
- Keys written. Outer tag: FLOW_DISSECTOR_KEY_VLAN (vlan_id / vlan_priority / vlan_tpid / vlan_eth_type); inner tag (depth 1):
FLOW_DISSECTOR_KEY_CVLAN, mirroring the slow-path's MAX -> VLAN -> CVLAN state machine. Bumps FLOW_DISSECTOR_KEY_NUM_OF_VLANS when requested by the dissector. Byte-identical with the slow path. Past depth 2 the helper returns false (defers to the slow path).
- Gates. Two static keys, each with its own sysctl:
DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key) (/proc/sys/net/flow_dissector/vlan) gates entry at depth 0, and DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key) (/proc/sys/net/flow_dissector/qinq) gates the depth-1 tag. Defaults 0; off has no behaviour change beyond one extra not-taken JMP in the dispatcher switch.
- Sibling-key handling. The two gates are coupled in their proc
handlers: writing qinq=1 auto-enables the vlan key (QinQ extends VLAN; depth-0 entry is gated by vlan), and turning vlan off clears the qinq key (depth-2 cannot fire when the outer entry is off). Reads always reflect the underlying key state.
- Cost. On a miss, one ethertype switch + the static_branch test. On a hit, ~3-5 ns added to the IPv4/IPv6 fast-path body per tag consumed, while the slow-path's per-tag handling (state-machine transition + proto_again rewind) is skipped -- net saving ~30 ns/skb on the single-VLAN shape and ~50 ns/skb on QinQ.
Verified byte-identical by the KUnit equivalence suite. Isolated dissector cost on the allshapes microbench (all fast paths compiled in, range across 7 measured microarchitectures): vlan -18.1%..-38.5%, qinq -16.1%..-39.8% -- see the cover letter's performance summary.
--- Documentation/admin-guide/sysctl/net.rst | 31 +++++ include/net/flow_dissector.h | 2 + net/core/flow_dissector.c | 147 +++++++++++++++++++++++ 3 files changed, 180 insertions(+)
Documentation/admin-guide/sysctl/net.rst +31 -0
@@ -485,6 +485,37 @@ when this is 0. Default: 0 +vlan +~~~~ + +Single 802.1Q or 802.1AD tagged frames over Eth + IPv4/IPv6 + +TCP/UDP. Handles both the hardware-stripped form (tag in skb +metadata, inner ethertype in skb->protocol) and the in-band form +(4-byte vlan_hdr at data + nhoff). Writes FLOW_DISSECTOR_KEY_VLAN +and bumps FLOW_DISSECTOR_KEY_NUM_OF_VLANS when those keys are +requested by the dissector. Byte-identical with the slow path for +the eligible shape; defers on any miss (QinQ defers; see the qinq +sysctl). + +Default: 0 + +qinq +~~~~ + +Lifts the vlan fast-path's depth limit from 1 to 2 so two stacked +802.1AD / 802.1Q tags (canonical QinQ S-tag + C-tag, or two stacked +802.1Q tags) are also fast-pathed. Outer tag is written to +FLOW_DISSECTOR_KEY_VLAN, inner to FLOW_DISSECTOR_KEY_CVLAN, mirroring +the slow-path's MAX → VLAN → CVLAN state machine. Byte-identical +with the slow path for the eligible shape. + +Auto-toggle: writing qinq=1 also enables vlan (QinQ extends VLAN; +the depth-0 entry of the helper is gated by the vlan key). Turning +vlan off (writing 0) also clears qinq (depth-2 cannot fire when +the outer entry is off). + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +2 -0
@@ -429,6 +429,8 @@ extern struct flow_dissector flow_keys_basic_dissector;* /proc/sys/net/flow_dissector/<shape>. All default off. */ extern struct static_key_false flow_dissector_eth_ip_key; +extern struct static_key_false flow_dissector_vlan_key; +extern struct static_key_false flow_dissector_qinq_key;/* struct flow_keys_digest: *
net/core/flow_dissector.c +147 -0
177 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 4/11 net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} + TCP/UDP
In reply to Dave SeddonAdds flow_dissect_fast_pppoe() — a straight-line extractor for PPPoE session frames (ETH_P_PPP_SES, EtherType 0x8864 per RFC 2516) carrying IPv4 or IPv6 inside PPP. Strips the 6-byte pppoe_hdr + 2-byte PPP protocol field, writes FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto, type) when requested, and tail-calls into the eth_ip fast-path for the inner IPv4 or IPv6 + TCP/UDP.
Byte-identical with the slow path's case htons(ETH_P_PPP_SES) block for the eligible shape (RFC-2516-valid header + PPP_IP / PPP_IPV6 + straight-line IP). On any miss — invalid header, PFC-compressed PPP proto, PPP_MPLS_UC / PPP_MPLS_MC (defer until MPLS fast-path lands), LCP / IPCP / other control frames — returns false so the dispatcher falls through to the slow path. Output is exactly what the slow path would produce.
Why this matters: PPPoE is the dominant L2 over millions of ISP last- mile DSL/FTTH links, and the existing slow-path PPPoE block does a __skb_header_pointer + per-field switch on every packet at the graph-walk entry. The fast-path saves the walk overhead for the common IPv4 / IPv6 inner case while keeping the slow path as the escape valve for control / less-common inner protos.
Gated by:
- new DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key)
- new sysctl /proc/sys/net/flow_dissector/pppoe (default 0)
- dispatcher case htons(ETH_P_PPP_SES) with static_branch_unlikely(&flow_dissector_pppoe_key) guard
When the sysctl is 0, the dispatcher hits a forward not-taken JMP — same per-call cost as the vlan / qinq cases. The new dispatcher case also doesn't widen the switch table (it adds one case to a switch the compiler can already lower to a jump table on the ETH_P_* constants).
Documentation/admin-guide/sysctl/net.rst grows a pppoe subsection under the existing /proc/sys/net/flow_dissector/ chapter, between qinq and the next chapter, matching the existing per-shape layout.
--- Documentation/admin-guide/sysctl/net.rst | 18 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 76 ++++++++++++++++++++++++ 3 files changed, 95 insertions(+)
Documentation/admin-guide/sysctl/net.rst +18 -0
@@ -516,6 +516,24 @@ the outer entry is off). Default: 0 +pppoe +~~~~~ + +PPPoE session frames (RFC 2516, EtherType ``ETH_P_PPP_SES`` 0x8864) +carrying IPv4 or IPv6 inside PPP. The fast-path strips the 6-byte +PPPoE session header + 2-byte PPP protocol field, writes +FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto, type) when the +dissector requests it, and tail-calls into the eth_ip fast-path for +the inner IPv4 / IPv6 + TCP/UDP. Byte-identical with the slow path +for the eligible shape. + +Defers to slow path for: PPP_MPLS_UC / PPP_MPLS_MC (until an MPLS +fast-path lands), PFC-compressed PPP protocol fields, LCP/IPCP/etc. +control frames, and any malformed PPPoE header (ver != 1, type != 1, +code != 0). + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +1 -0
@@ -431,6 +431,7 @@ extern struct flow_dissector flow_keys_basic_dissector;extern struct static_key_false flow_dissector_eth_ip_key;extern struct static_key_false flow_dissector_vlan_key;extern struct static_key_false flow_dissector_qinq_key; +extern struct static_key_false flow_dissector_pppoe_key;/* struct flow_keys_digest: *
net/core/flow_dissector.c +76 -0
100 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 5/11 net: flow_dissector: add fast-path for single MPLS label + IP
In reply to Dave SeddonAdds flow_dissect_fast_mpls() — straight-line read of one MPLS label entry (ETH_P_MPLS_UC 0x8847 / ETH_P_MPLS_MC 0x8848), extracting label / TC / TTL / BoS and writing to FLOW_DISSECTOR_KEY_MPLS lse[0] when the dissector requests it.
Byte-identical with the slow path's __skb_flow_dissect_mpls() called with lse_index=0, bos=1: the slow path returns FLOW_DISSECT_RET_OUT_GOOD when BoS is set without descending into the inner packet (MPLS flow hashing is on the label stack, not the inner 5-tuple), so this fast-path does the same — no IP tail-call.
Multi-label stacks (BoS=0 on the first label) defer to the slow path. A depth-2+ variant can land later, mirroring the vlan -> qinq staging used for the VLAN gates.
Why this matters: MPLS is the dominant SP backbone L2.5 and the existing slow-path traverses an iteration loop per label even when the consumer just wants the label-stack flow key. The fast-path saves the iteration overhead for the common single-label case while keeping the slow path as escape for multi-label and the entropy-label case (which can only fire from label 2+ anyway).
Gated by:
- new DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key)
- new sysctl /proc/sys/net/flow_dissector/mpls (default 0)
- dispatcher case htons(ETH_P_MPLS_UC) / htons(ETH_P_MPLS_MC) with static_branch_unlikely(&flow_dissector_mpls_key) guard
When the sysctl is 0, dispatcher cost is one not-taken JMP per matching packet — same per-shape pattern as the other fast-path patches.
Documentation/admin-guide/sysctl/net.rst grows an mpls subsection under /proc/sys/net/flow_dissector/.
The helper also mirrors the slow path's out_good terminal writes: for the standard flow_keys dissectors (which request no MPLS key), the slow path returns OUT_GOOD after the first LSE with nhoff advanced past it, so thoff, n_proto (the MPLS ethertype) and ip_proto (0) are still written at the exit label. Skipping those writes is observable in struct flow_keys and would break the byte-identical contract; the in-tree KUnit equivalence test added later in this series catches exactly this.
--- Documentation/admin-guide/sysctl/net.rst | 19 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 83 ++++++++++++++++++++++++ 3 files changed, 103 insertions(+)
Documentation/admin-guide/sysctl/net.rst +19 -0
@@ -534,6 +534,25 @@ code != 0). Default: 0 +mpls +~~~~ + +Single-label MPLS frames (EtherType ``ETH_P_MPLS_UC`` 0x8847 or +``ETH_P_MPLS_MC`` 0x8848 with bottom-of-stack bit set on the first +label). The fast-path reads the 4-byte label entry, extracts +label / TC / TTL / BoS, and writes them to FLOW_DISSECTOR_KEY_MPLS +lse[0] when requested. Byte-identical with the slow path's +__skb_flow_dissect_mpls() for lse_index=0, bos=1: the slow path +returns OUT_GOOD on BoS without descending into the inner packet +(MPLS hashing is on the label stack, not inner 5-tuple), so this +fast-path does the same. + +Multi-label stacks (BoS=0 on the first label) defer to slow path — +extending to depth-2 or more is a follow-up, mirroring the +vlan -> qinq staging used earlier in this series. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +1 -0
@@ -432,6 +432,7 @@ extern struct static_key_false flow_dissector_eth_ip_key;extern struct static_key_false flow_dissector_vlan_key;extern struct static_key_false flow_dissector_qinq_key;extern struct static_key_false flow_dissector_pppoe_key; +extern struct static_key_false flow_dissector_mpls_key;/* struct flow_keys_digest: *
net/core/flow_dissector.c +83 -0
107 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 6/11 net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 / 6in4)
In reply to Dave SeddonAdds flow_dissect_fast_ipip_inner() — a small recursion helper invoked from the existing flow_dissect_fast_ipv4() and flow_dissect_fast_ipv6() helpers when the outer IP's protocol (IPv4) or nexthdr (IPv6) is IPPROTO_IPIP or IPPROTO_IPV6 and the ipip gate is on. Tail-calls into the matching inner IP fast-path then stamps FLOW_DIS_ENCAPSULATION on key_control->flags. This is the series' first tunnel descent: the fast path parses through the tunnel header and dissects the inner flow, the pattern the GRE and UDP-tunnel patches later in the series reuse.
Byte-identical with the slow path's IPPROTO_IPIP and IPPROTO_IPV6 switch cases in __skb_flow_dissect(): the slow path also lets the inner IP overwrite outer addrs/basic and OR's ENCAP into the control flags. The ENCAP write must happen *after* the inner returns because the existing fast-path helpers unconditionally set key_control->flags = 0 on entry (correct for the top-level call but clobbers the encap flag during recursion); the ipip_inner helper re-establishes ENCAP after the inner returns.
Coverage:
outer IPv4, protocol == IPIP -> inner IPv4 outer IPv4, protocol == IPV6 -> inner IPv6 (6in4) outer IPv6, nexthdr == IPIP -> inner IPv4 (4in6) outer IPv6, nexthdr == IPV6 -> inner IPv6
FLOW_DISSECTOR_F_STOP_BEFORE_ENCAP and STOP_AT_ENCAP don't apply because the top-level dispatcher already rejects anything beyond F_PARSE_1ST_FRAG, so the fast-path entry never sees the encap-stop flags - packets that request them go to the slow path unchanged.
Gated by:
- new DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key)
- new sysctl /proc/sys/net/flow_dissector/ipip (default 0)
- static_branch_unlikely guard inside the v4/v6 helpers, hit only when the outer protocol/nexthdr is not TCP/UDP (i.e. the TCP/UDP fast-path was about to return false anyway)
Cost when off: the not-TCP/UDP fall-through grows by one static_branch_unlikely check. The TCP/UDP hot path is unchanged — the new branch is below the existing TCP/UDP fast-out.
The IPv6 helper's standalone not-TCP/UDP guard becomes unreachable once the new block handles that case in full, so it is removed here.
Documentation/admin-guide/sysctl/net.rst grows an ipip subsection under /proc/sys/net/flow_dissector/.
The IPv6-outer descent mirrors the slow path's outer writes (v6addrs, addr_type) before recursing into the inner header. The slow path fills these for the outer header and the inner pass then overwrites only what it uses: an inner IPv4 overwrites just the first 8 bytes of the addrs union, leaving outer-v6 bytes in the tail. Byte-identical output means reproducing exactly that, residue included — the KUnit equivalence test's 4in6 cases fail otherwise. (A non-zero outer flow label defers before any descent — see the eth_ip patch's output contract — so no label residue exists. The IPv4-outer descent needs no mirroring: an inner IPv4 or IPv6 write always fully covers the outer's 8 address bytes.)
--- Documentation/admin-guide/sysctl/net.rst | 19 ++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 117 ++++++++++++++++++++++- 3 files changed, 134 insertions(+), 3 deletions(-)
Documentation/admin-guide/sysctl/net.rst +19 -0
@@ -553,6 +553,25 @@ vlan -> qinq staging used earlier in this series. Default: 0 +ipip +~~~~ + +IPv4-in-IPv4 (``IPPROTO_IPIP`` 4), IPv6-in-IPv4 (``IPPROTO_IPV6`` 41 +inside an IPv4 outer), IPv4-in-IPv6 and IPv6-in-IPv6 tunnels. When +the outer IP fast-path notices the outer protocol/nexthdr is one of +these and this gate is on, it skips the outer header and tail-calls +into the inner IPv4 / IPv6 fast-path. The flow_keys ends up with the +inner 5-tuple (matching the slow path's behaviour, where the inner +parse overwrites the outer addrs/basic), plus +FLOW_DIS_ENCAPSULATION set on key_control->flags. + +Byte-identical with the slow path's IPPROTO_IPIP / IPPROTO_IPV6 +switch cases for the eligible shape. Defers on any miss — including +fragmented outer, unusual outer IHL, or unsupported inner shapes +that the inner IP fast-path itself would defer. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +1 -0
@@ -433,6 +433,7 @@ extern struct static_key_false flow_dissector_vlan_key;extern struct static_key_false flow_dissector_qinq_key;extern struct static_key_false flow_dissector_pppoe_key;extern struct static_key_false flow_dissector_mpls_key; +extern struct static_key_false flow_dissector_ipip_key;/* struct flow_keys_digest: *
net/core/flow_dissector.c +114 -3
154 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 7/11 net: flow_dissector: add byte-identical fast-path for plain GRE inner
In reply to Dave SeddonAdds flow_dissect_fast_gre_inner() — invoked from inside the existing flow_dissect_fast_ipv4() / flow_dissect_fast_ipv6() helpers when the outer IP's protocol/nexthdr is IPPROTO_GRE and the GRE gate is on.
Byte-identical with the slow path's __skb_flow_dissect_gre() for the subset where:
- GRE version == 0 (v1 = PPTP defers to slow path)
- All GRE flags clear (no CSUM, KEY, SEQ, ROUTING)
- protocol == ETH_P_IP or ETH_P_IPV6 (no TEB, PPP, MPLS-over-GRE)
In that subset, the GRE base header is exactly 4 bytes; slow path also descends to inner IP (sets *p_proto = hdr->protocol and returns PROTO_AGAIN) and stamps FLOW_DIS_ENCAPSULATION on key_control->flags. The fast-path produces the same output: skips the 4-byte header, tail-calls into the inner IP fast-path, re-establishes ENCAP on key_control after the inner overwrites it (same pattern as the IPIP family fast-path from the prior patch).
Why this matters: plain GRE (no key, no checksum) is the dominant shape for cloud backbone tunnels (AWS GRE between regions, GCP inter-region overlays) and corporate site-to-site VPNs that don't need per-tunnel keying. The existing slow-path GRE handler does a __skb_header_pointer + flag-walk per packet; the fast-path saves that for the common case while keeping the slow path as the escape valve for GRE-with-flags, v1/PPTP, TEB, and PPP-over-GRE.
Gated by:
- new DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key)
- new sysctl /proc/sys/net/flow_dissector/gre (default 0)
- static_branch_unlikely guard inside the v4/v6 helpers, on the same not-TCP/UDP fall-through path as the IPIP gate (so the eth_ip hot path remains unchanged when this is off)
Cost when off: the not-TCP/UDP fall-through grows by one static_branch_unlikely check on top of the IPIP check. The TCP/UDP hot path is untouched.
GRE-with-KEY (used in some MPLS-over-GRE and cloud-overlay deployments) is a natural follow-up: same descent shape with the optional 4-byte key field read and a write to FLOW_DISSECTOR_KEY_GRE_KEYID.
Documentation/admin-guide/sysctl/net.rst grows a gre subsection under /proc/sys/net/flow_dissector/.
Classification note: the slow path already descends through plain GRE unconditionally, so this fast path is byte-identical with today's behaviour — a pure CPU saving. That is unlike the UDP-tunnel descents later in the series (VXLAN/Geneve/GTP-U/FOU/GUE), where the slow path stops at the outer UDP header today and the descent itself is the new, gated behaviour.
--- Documentation/admin-guide/sysctl/net.rst | 25 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 101 +++++++++++++++++++++-- 3 files changed, 121 insertions(+), 6 deletions(-)
Documentation/admin-guide/sysctl/net.rst +25 -0
@@ -572,6 +572,31 @@ that the inner IP fast-path itself would defer. Default: 0 +gre +~~~ + +GRE-encapsulated inner IP (``IPPROTO_GRE`` 47). The fast-path +mirrors the slow path's __skb_flow_dissect_gre() descent for the +common subset: + +- GRE version 0 (the v1 PPTP variant defers to slow path) +- All GRE flags clear (no GRE_CSUM, GRE_KEY, GRE_SEQ, GRE_ROUTING — + i.e. plain 4-byte GRE base header) +- protocol field is ``ETH_P_IP`` 0x0800 or ``ETH_P_IPV6`` 0x86DD + (no Transparent Ethernet Bridging, no PPP-over-GRE, no MPLS- + over-GRE) + +In that subset, slow path also descends to inner IP and stamps +``key_control->flags |= FLOW_DIS_ENCAPSULATION``; the fast-path +produces the same output. + +GRE-with-KEY (common in MPLS-over-GRE deployments and some cloud +overlays) is a follow-up patch — same descent shape with an +additional 4-byte key field read and a write to +``FLOW_DISSECTOR_KEY_GRE_KEYID``. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +1 -0
@@ -434,6 +434,7 @@ extern struct static_key_false flow_dissector_qinq_key;extern struct static_key_false flow_dissector_pppoe_key;extern struct static_key_false flow_dissector_mpls_key;extern struct static_key_false flow_dissector_ipip_key; +extern struct static_key_false flow_dissector_gre_key;/* struct flow_keys_digest: *
net/core/flow_dissector.c +95 -6
144 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 8/11 net: flow_dissector: per-shape counters + /proc/net/flow_dissector_stats
In reply to Dave SeddonThe opt-in fast-path (this series) gates each byte-identical shape behind its own net.flow_dissector.* sysctl. Operators need to see how traffic is actually composed to decide which gates to enable — and that same signal lets a policy layer (a userspace agent, or the auto controller proposed in a separate RFC thread) enable/disable the gates automatically instead of hand-tuning knobs.
Add per-cpu counters, keyed by a small enum flow_dissector_shape covering the byte-identical shapes (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre):
occurrences[] a shape handled by the SLOW path. Measured while its
gate is off, this approximates the eligible-fraction
signal: how much traffic that shape's fast body would have handled. The slow counts sit at protocol recognition points, which are a superset of fast-path eligibility (a VLAN+ARP frame counts as vlan; an options-bearing IPv4 counts as eth_ip), so eligible% is an upper bound -- the exact hit rate is fast_hits / (occurrences + fast_hits) once a gate is on. Deliberately so: matching the fast path's full eligibility conditions in the slow path would mean re-implementing the fast path's checks there.
fast_hits[] the fast body actually ran for that shape (gate on).
dissects total dissects — the denominator.
Because the fast path returns before the slow-path classification, exactly one of occurrences/fast_hits is incremented per shaped packet depending on gate state, so (occurrences + fast_hits) / dissects is a gate-invariant eligible fraction. Placement mirrors the two paths precisely: the slow counts sit at each protocol's dissection point -- for eth_ip that is the
out: exit label, taken only on a top-level, non-encapsulated, TCP/UDP terminal, so a packet counted as VLAN/IPIP/GRE is never also counted as eth_ip; the fast counts sit at each shape's success terminal, so a fall-through to the slow path is never double-counted.
Expose the summed counters read-only at /proc/net/flow_dissector_stats (documented in Documentation/admin-guide/sysctl/net.rst), one line per shape with occurrences, fast_hits, computed eligible%, and the current gate state. Counters and the file are global (init_net), matching the global gates; the file is visible in the init netns only, so containers do not see it. Per-netns is a noted follow-up.
Dissects fully handled by an attached netns BPF flow dissector program are deliberately not counted: the dissects increment sits after the BPF hook's early return, so the file counts only dissects that reach the in-kernel dissector. That keeps fast_hits/dissects an undiluted hit-rate for the code these gates control — the signal an operator (or the auto controller proposed in a separate RFC thread) acts on.
The counters are unconditional by design, not oversight: their whole purpose is observing traffic composition while the gates are OFF, so a "stats gate" would blind exactly the signal an operator (or the auto-enable controller proposed separately) needs before enabling anything.
Cost: one unconditional this_cpu_inc per dissect (the denominator) plus one this_cpu_inc at the matched shape's classification point, summed only on read. Measured on a CPU-bound pktgen soak this is +0.74% dissector time (+0.44 sigma -- within run-to-run noise); see the cover letter. The gates themselves are unaffected: an off gate stays a not-taken static branch.
--- Documentation/admin-guide/sysctl/net.rst | 39 ++++ include/net/flow_dissector.h | 17 ++ net/core/flow_dissector.c | 236 ++++++++++++++++++++--- 3 files changed, 270 insertions(+), 22 deletions(-)
Documentation/admin-guide/sysctl/net.rst +39 -0
@@ -597,6 +597,45 @@ additional 4-byte key field read and a write to Default: 0 +The ``/proc/net/flow_dissector_stats`` observability file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Read-only. Reports, per byte-identical shape, the number of packets the +slow path handled (``occurrences``), the number the fast body handled +(``fast_hits``), the resulting eligible fraction, and the current gate +state, plus a total dissect count. Because the fast path returns before +the slow-path classification, exactly one of the two counters is +incremented per shaped packet depending on gate state, so +``(occurrences + fast_hits) / dissects`` is a gate-invariant measure of +how much traffic each shape's fast body would handle -- the signal an +operator uses to decide whether a gate is worth enabling. The slow-path +counts sit at protocol recognition points, a superset of fast-path +eligibility (a VLAN+ARP frame counts as vlan), so the eligible fraction +is an upper bound; the exact hit rate once a gate is on is +``fast_hits / (occurrences + fast_hits)``. The counters +are per-cpu and summed on read; the per-packet cost is one increment on +the already-hot classification path. + +An example snapshot, from a host that mostly sees plain Eth + IP traffic +with some VLAN, before any gate has been enabled:: + + # cat /proc/net/flow_dissector_stats + shape occurrences fast_hits eligible% gate + eth_ip 1866909 0 75.71 off + vlan 598847 0 24.28 off + qinq 0 0 0.00 off + pppoe 0 0 0.00 off + mpls 0 0 0.00 off + ipip 0 0 0.00 off + gre 0 0 0.00 off + dissects: 2465765 + +Read top-down: 76% of dissects are plain Eth + IP and 24% are +VLAN-tagged, so ``eth_ip`` and ``vlan`` are the shapes worth enabling on +this host; the other shapes are absent. Every gate is still off, so all +the traffic sits under ``occurrences``; enabling a gate moves that +shape's packets to ``fast_hits``. + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/flow_dissector.h +17 -0
@@ -436,6 +436,23 @@ extern struct static_key_false flow_dissector_mpls_key;extern struct static_key_false flow_dissector_ipip_key;extern struct static_key_false flow_dissector_gre_key; +/* Per-shape counters for /proc/net/flow_dissector_stats. +* occurrences[]: shape seen by the slow path (the eligible-fraction +* signal while its gate is off); fast_hits[]: fast path ran. +* Byte-identical shapes only; the behaviour-changing descents are +* never counted or auto-managed. +*/ +enum flow_dissector_shape { +FLOW_DISSECTOR_SHAPE_ETH_IP , +FLOW_DISSECTOR_SHAPE_VLAN , +FLOW_DISSECTOR_SHAPE_QINQ , +FLOW_DISSECTOR_SHAPE_PPPOE , +FLOW_DISSECTOR_SHAPE_MPLS , +FLOW_DISSECTOR_SHAPE_IPIP , +FLOW_DISSECTOR_SHAPE_GRE , +FLOW_DISSECTOR_SHAPE__MAX , +}; +/* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a
net/core/flow_dissector.c +214 -22
387 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 9/11 net: flow_dissector: bound fast-path tunnel recursion
In reply to Dave SeddonThe opt-in fast path descends into IP-in-IP and GRE tunnels by
recursion: flow_dissect_fast_ipv4()/ipv6() call flow_dissect_fast_ipip_inner()/gre_inner(), which parse the inner IP header and call back into flow_dissect_fast_ipv4()/ipv6(). Unlike the slow path, which caps total header descents at MAX_FLOW_DISSECT_HDRS via skb_flow_dissect_allowed(), the fast helpers had no such bound.
A crafted frame with a deeply nested tunnel chain (IP-in-IP-in-IP-in-..., or GRE) therefore drives one C stack frame per ~20 bytes of linear header. hlen is skb_headlen(), so a large, GRO-coalesced or locally crafted skb yields hundreds to thousands of levels and can exhaust the kernel stack. The inner helpers also do work after the recursive call (they stamp FLOW_DIS_ENCAPSULATION), so the frames are not tail-call eliminable. It is additionally an output
divergence: past 15 headers the slow path stops and reports the 15th-level keys, while the fast path would recurse to the innermost, breaking the byte-identical contract.
Thread a num_hdrs counter through the tunnel-reachable helpers and, at each tunnel descent, mirror skb_flow_dissect_allowed(): once the count exceeds MAX_FLOW_DISSECT_HDRS the helper returns false and the dissect falls back to the slow path (which caps at the same limit). For chains within the limit both paths descend fully and agree; beyond it the fast path defers to the slow path's result, so the output stays identical and the recursion is bounded.
The initial count at each fast-path entry is set at or above the number of headers the slow path has already consumed (outer VLAN tags, PPPoE, the outer IP), so the fast path never descends past the point the slow path would have capped.
The added KUnit equivalence test's deep-nest case exercises IPIP and GRE chains across the cap boundary; with the bound removed it fails at 16 levels, confirming it guards this fix.
--- net/core/flow_dissector.c | 61 ++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 24 deletions(-)
net/core/flow_dissector.c +37 -24
@@ -91,28 +91,31 @@ static inline void flow_dissector_count_fast(enum flow_dissector_shape shape)/* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 -/* Fast-path helper forward declarations. */ +/* @num_hdrs: protocol headers consumed so far (Ethernet = 1, each VLAN +* tag, each outer IP header). Tunnel descents increment it and defer +* past MAX_FLOW_DISSECT_HDRS, so both paths stop at the same depth. +*/ static bool flow_dissect_fast_ipv4 (const struct sk_buff *skb,struct flow_dissector *flow_dissector,void *target_container,const void *data, -int nhoff,int hlen); +int nhoff,int hlen,int num_hdrs);static bool flow_dissect_fast_ipv6 (const struct sk_buff *skb,struct flow_dissector *flow_dissector,void *target_container,const void *data, -int nhoff,int hlen); +int nhoff,int hlen,int num_hdrs);static bool flow_dissect_fast_ipip_inner (const struct sk_buff *skb,struct flow_dissector *flow_dissector,void *target_container,const void *data,__be16 inner_eth_proto, -int inner_nhoff,int hlen); +int inner_nhoff,int hlen,int num_hdrs);static bool flow_dissect_fast_gre_inner (const struct sk_buff *skb,struct flow_dissector *flow_dissector,void *target_container,const void *data, -int nhoff,int hlen); +int nhoff,int hlen,int num_hdrs);/* One of the two dissectors the fast-path eligibility check admits; * defined here so flow_dissect_fast() below can reference it (its keys
@@ -1141,7 +1144,7 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control;struct flow_dissector_key_addrs *key_addrs;
@@ -1172,18 +1175,18 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,return flow_dissect_fast_ipip_inner (skb, flow_dissector, target_container, data,htons (ETH_P_IP ), - nhoff + (int )sizeof (*iph), hlen); + nhoff + (int )sizeof (*iph), hlen, num_hdrs);if (iph->protocol ==IPPROTO_IPV6 )return flow_dissect_fast_ipip_inner (skb, flow_dissector, target_container, data,htons (ETH_P_IPV6 ), - nhoff + (int )sizeof (*iph), hlen); + nhoff + (int )sizeof (*iph), hlen, num_hdrs); }if (static_branch_unlikely (&flow_dissector_gre_key) && iph->protocol ==IPPROTO_GRE )return flow_dissect_fast_gre_inner (skb, flow_dissector, target_container, data, - nhoff + (int )sizeof (*iph), hlen); + nhoff + (int )sizeof (*iph), hlen, num_hdrs);return false; }
@@ -1241,7 +1244,7 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control;struct flow_dissector_key_addrs *key_addrs;
@@ -1317,10 +1320,10 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, data, iph->nexthdr ==IPPROTO_IPIP ?htons (ETH_P_IP ) :htons (ETH_P_IPV6 ), - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs);return flow_dissect_fast_gre_inner (skb, flow_dissector, target_container, data, - nhoff + (int )sizeof (*iph), hlen); + nhoff + (int )sizeof (*iph), hlen, num_hdrs); } thoff = nhoff + (int )sizeof (*iph);
@@ -1439,12 +1442,12 @@ static bool flow_dissect_fast_vlan(const struct sk_buff *skb,case htons (ETH_P_IP ): ok =flow_dissect_fast_ipv4 (skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, vlan_depth +2 );break ;case htons (ETH_P_IPV6 ): ok =flow_dissect_fast_ipv6 (skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, vlan_depth +2 );break ;case htons (ETH_P_8021Q ):case htons (ETH_P_8021AD ):
@@ -1524,11 +1527,11 @@ static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,if (inner_eth_proto ==htons (ETH_P_IP )) ok =flow_dissect_fast_ipv4 (skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen,2 );else ok =flow_dissect_fast_ipv6 (skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen,2 );if (ok)flow_dissector_count_fast (FLOW_DISSECTOR_SHAPE_PPPOE );
@@ -1612,19 +1615,25 @@ static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,void *target_container, const void *data, __be16 inner_eth_proto, - int inner_nhoff, int hlen) + int inner_nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control;bool ok; +/* Mirror the slow path's MAX_FLOW_DISSECT_HDRS budget; past the cap +* defer, so both paths stop at the same depth. +*/ +if (++num_hdrs >MAX_FLOW_DISSECT_HDRS ) +return false; +if (inner_eth_proto ==htons (ETH_P_IP )) ok =flow_dissect_fast_ipv4 (skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs);else if (inner_eth_proto ==htons (ETH_P_IPV6 )) ok =flow_dissect_fast_ipv6 (skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs);else return false;
@@ -1651,7 +1660,7 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control;const struct gre_base_hdr *hdr;
@@ -1659,6 +1668,10 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,int inner_nhoff;bool ok; +/* Same MAX_FLOW_DISSECT_HDRS budget as flow_dissect_fast_ipip_inner(). */ +if (++num_hdrs >MAX_FLOW_DISSECT_HDRS ) +return false; +if (unlikely (hlen - nhoff < (int )sizeof (*hdr)))return false; hdr = (const struct gre_base_hdr *)((const u8 *)data + nhoff);
@@ -1686,11 +1699,11 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,if (inner_proto ==htons (ETH_P_IP )) ok =flow_dissect_fast_ipv4 (skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs);else ok =flow_dissect_fast_ipv6 (skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs);if (!ok)return false;
@@ -1764,7 +1777,7 @@ static bool flow_dissect_fast(const struct sk_buff *skb,return false;if (!flow_dissect_fast_ipv4 (skb, flow_dissector, target_container, data, - nhoff, hlen)) + nhoff, hlen,1 ))return false;/* Plain top-level eth+IP only; descents count under their own shape. */ if (!flow_dissect_fast_is_encap (flow_dissector, target_container))
@@ -1775,7 +1788,7 @@ static bool flow_dissect_fast(const struct sk_buff *skb,return false;if (!flow_dissect_fast_ipv6 (skb, flow_dissector, target_container, data, - nhoff, hlen)) + nhoff, hlen,1 ))return false;if (!flow_dissect_fast_is_encap (flow_dissector, target_container))flow_dissector_count_fast (FLOW_DISSECTOR_SHAPE_ETH_IP );
Signature · 1 line
2.54.0
PATCH 10/11 net: flow_dissector: add KUnit fast/slow path equivalence tests
In reply to Dave SeddonThe per-shape fast paths contract to be byte-identical with the slow
path: for any packet, either struct flow_keys is filled with exactly the bytes the slow path would have written, or the fast path returns false and the slow path runs. Until now that contract was only verified out of tree (a userspace A/B harness compiling the dissector both ways). Enforce it in-tree so any future divergence between the two implementations is a failing test, not a silent behaviour change.
Each corpus packet is dissected twice into zeroed containers - once with every fast-path gate disabled and once with every gate enabled - and the return values and full struct flow_keys must match exactly. (A fast-path miss cannot leak partial writes to callers:flow_dissect_fast() is only invoked from __skb_flow_dissect(), which falls through to the full slow-path walk on the same container whenever the fast path returns false.) The corpus covers the eligible shapes (eth+IPv4/IPv6 x TCP/UDP, VLAN, QinQ, PPPoE session, single MPLS, IPIP/4in6/6in4, plain GRE), deliberate fast-path misses (IP options, fragments with and without FLOW_DISSECTOR_F_PARSE_1ST_FRAG, IPv6 extension headers and non-zero flow labels, triple VLAN, multi-label MPLS, non-IP PPP, flagged GRE, ICMP), a truncation sweep cutting every packet at every byte boundary, and skb-mode cases for the hardware-stripped VLAN tag entry conditions raw-data mode cannot reach.
Every equivalence case runs both with and without FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- the flag skb_get_hash() passes on every dissect -- against both eligible dissectors, and the gates-off test's positive control asserts the fast path actually hits for that flagship flag shape (it caught the dispatcher rejecting the flag outright, which had made the fast path unreachable from RPS/RFS/fq/cake).
Beyond byte equality, every check also asserts flow_hash_from_keys() equality (the hash is what consumers act on). A gates-off pass over the whole corpus proves the fast path never runs when disabled (observed via the per-shape counters), and a seeded, reproducible fuzzer mutates corpus packets and re-checks equivalence every iteration, as a guard for divergences the hand-written corpus does not name.
The file-static symmetric dissector and the summed fast-hit counters are exposed to the suite through accessors compiled only under CONFIG_FLOW_DISSECTOR_KUNIT_TEST.
Run with:
./tools/testing/kunit/kunit.py run --arch=x86_64 \
--kconfig_add CONFIG_NET=y \
--kconfig_add CONFIG_FLOW_DISSECTOR_KUNIT_TEST=y \
flow_dissector_fastpath--- include/net/flow_dissector.h | 6 + net/Kconfig | 12 + net/core/Makefile | 1 + net/core/flow_dissector.c | 30 + net/core/flow_dissector_test.c | 1053 ++++++++++++++++++++++++++++++++ 5 files changed, 1102 insertions(+) create mode 100644 net/core/flow_dissector_test.c
include/net/flow_dissector.h +6 -0
@@ -453,6 +453,12 @@ enum flow_dissector_shape { FLOW_DISSECTOR_SHAPE__MAX, }; +#if IS_ENABLED (CONFIG_FLOW_DISSECTOR_KUNIT_TEST ) +/* Test-only accessors, defined in flow_dissector.c. */ +struct flow_dissector *flow_keys_dissector_symmetric_kunit (void ); +u64 flow_dissector_fast_hits_kunit (void ); +#endif +/* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a
net/Kconfig +12 -0
@@ -540,4 +540,16 @@ config NET_TEST If unsure, say N. +config FLOW_DISSECTOR_KUNIT_TEST + tristate "KUnit tests for flow dissector fast/slow path equivalence" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + KUnit tests asserting the flow dissector's opt-in per-shape + fast paths produce byte-identical struct flow_keys output to + the slow path for every corpus packet, with the gates on and + off. + + If unsure, say N. + endif # if NET
net/core/Makefile +1 -0
@@ -46,6 +46,7 @@ obj-$(CONFIG_BPF_SYSCALL) += sock_map.o obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o obj-$(CONFIG_OF) += of_net.o obj-$(CONFIG_NET_TEST) += net_test.o +obj-$(CONFIG_FLOW_DISSECTOR_KUNIT_TEST) += flow_dissector_test.o obj-$(CONFIG_NET_DEVMEM) += devmem.o obj-$(CONFIG_DEBUG_NET) += lock_debug.o obj-$(CONFIG_FAIL_SKB_REALLOC) += skb_fault_injection.o
net/core/flow_dissector.c +30 -0
@@ -2909,6 +2909,36 @@ EXPORT_SYMBOL(flow_keys_dissector);struct flow_dissector flow_keys_basic_dissector__read_mostly ;EXPORT_SYMBOL (flow_keys_basic_dissector); +#if IS_ENABLED (CONFIG_FLOW_DISSECTOR_KUNIT_TEST ) +/* Test-only accessor: the symmetric dissector is file-static but is +* the second dissector the fast-path eligibility gate admits. +*/ +struct flow_dissector *flow_keys_dissector_symmetric_kunit (void ) +{ +return &flow_keys_dissector_symmetric; +} +EXPORT_SYMBOL_GPL (flow_keys_dissector_symmetric_kunit); + +/* Test-only accessor: summed fast-path hits, the observable for the +* suite's gates-off negative. +*/ +u64 flow_dissector_fast_hits_kunit (void ) +{ +u64 sum =0 ; +int cpu, i; + +for_each_possible_cpu (cpu) { +const struct flow_dissector_stats *s = +per_cpu_ptr (&flow_dissector_pcpu_stats, cpu); + +for (i =0 ; i <FLOW_DISSECTOR_SHAPE__MAX ; i++) + sum +=READ_ONCE (s->fast_hits[i]); + } +return sum; +} +EXPORT_SYMBOL_GPL (flow_dissector_fast_hits_kunit); +#endif +static int __init init_default_flow_dissectors (void ) { skb_flow_dissector_init(&flow_keys_dissector,
net/core/flow_dissector_test.c new file +1053 -0
1,053 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 11/11 Documentation: networking: add flow_dissector overview and fast-path guide
In reply to Dave SeddonThe kernel has no general flow_dissector documentation: Documentation/bpf/ prog_flow_dissector.rst covers only the BPF override mechanism (and was itself in no toctree). Add Documentation/networking/flow_dissector.rst as a standalone overview -- what the flow dissector is, struct flow_keys and the consumers of skb->hash (RSS/RPS/RFS, ECMP/multipath, bonding/LAG, tc-flower,
aRFS), the generic protocol-graph parser, and the relationship to the BPF flow dissector -- and document the opt-in fast paths added by this series:
the byte-identical per-shape gates, the
break-even model for when to enable a gate (with the observable /proc/net/flow_dissector_stats). The per-knob
reference stays in Documentation/admin-guide/sysctl/net.rst; this doc links to it rather than duplicating it.
Also index the previously-orphaned Documentation/bpf/prog_flow_dissector.rst and cross-link it with the new overview (the bpf/index.rst hunk can be split out if the BPF maintainers prefer).
--- Documentation/bpf/index.rst | 1 + Documentation/bpf/prog_flow_dissector.rst | 2 + Documentation/networking/flow_dissector.rst | 133 ++++++++++++++++++++ Documentation/networking/index.rst | 1 + 4 files changed, 137 insertions(+) create mode 100644 Documentation/networking/flow_dissector.rst
Documentation/bpf/index.rst +1 -0
@@ -23,6 +23,7 @@ that goes into great technical depth about the BPF Architecture. cpumasks fs_kfuncs programs + prog_flow_dissector maps bpf_prog_run classic_vs_extended.rst
Documentation/bpf/prog_flow_dissector.rst +2 -0
@@ -9,6 +9,8 @@ Overview Flow dissector is a routine that parses metadata out of the packets. It's used in the various places in the networking subsystem (RFS, flow hash, etc). +See :doc:`/networking/flow_dissector` for an overview of the flow dissector +and the built-in C implementation this program type overrides. BPF flow dissector is an attempt to reimplement C-based flow dissector logic in BPF to gain all the benefits of BPF verifier (namely, limits on the
Documentation/networking/flow_dissector.rst new file +133 -0
133 lines not on this page · raw message
Documentation/networking/index.rst +1 -0
@@ -52,6 +52,7 @@ Contents: eql fib_trie filter + flow_dissector generic-hdlc generic_netlink ../netlink/specs/index
Signature · 1 line
2.54.0
Quoted 33 lines · Dave Seddon
G'day,
__skb_flow_dissect()parses packet headers with a generic protocol-graph parser. The graph walk is what makes the dissector so flexible, but most machines spend that flexibility on the same traffic all day long: plain eth + IPv4/IPv6 + TCP/UDP, perhaps with a VLAN tag or a tunnel header in front. The intuition behind this series is that the common shapes can be dissected by straight-line code instead, without giving up the graph walk for everything else.Concretely: the series adds an opt-in, static-branch-gated fast path per packet shape, covering the common cases and the IP-in-IP/GRE encapsulations. Each shape has its own gate under
/proc/sys/net/flow_dissector/, every gate is disabled by default, and when a gate is off its added cost is one not-taken branch per dissect. The gates are manual in this series; a follow-up RFC will propose an optional auto mode that flips them from the measured traffic mix (via the patch-8 counters), so nobody has to hand-tune seven knobs (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre). Nothing
here depends on it -- it is where this is heading, not a prerequisite.Relationship to the BPF flow dissector
======================================
The fast path is invoked strictly after the netns BPF flow dissector hook in__skb_flow_dissect(): if a program is attached and returns any verdict other thanBPF_FLOW_DISSECTOR_CONTINUE, the function returns beforeflow_dissect_fast()is reached. Attached BPF dissectors therefore always take precedence, and a system running one sees bit-for-bit unchanged behaviour from this series (patch 1 additionally makes its program lookup cheaper for everyone else). Dissects fully handled by a BPF program are deliberately not counted in/proc/net/flow_dissector_stats(patch 8).
Quoted 3 lines
The gates are global (a static key patches code shared by every netns); per-netns dissector policy already has a mechanism -- the netns BPF flow dissector -- and it keeps full precedence here.
This series adds a lot of code: +2500 LoC.
If you want a linear fast path, the BPF dissector offers that.
I don't think the purported benefit justifies the significant new code, I'm afraid.
The code is duplicative of existing paths, so there may be additional maintenance cost keeping the two consistent.
And which paths justify a fast paths and which do not is highly subjective. Why is GRE included, for instance?
Thanks Willem, for the quick response and the feedback.
I think your point about where to draw the protocol boundary is a good one.
Eth + IPv4/IPv6 + TCP/UDP is clearly the common case. Beyond that, choosing what belongs in-kernel becomes increasingly subjective: VLAN/QinQ, PPPoE, MPLS, GRE, GTP-U, etc. all have substantial deployments, but that's also an argument for leaving those cases to the BPF flow dissector rather than continuing to grow a parallel parser.
On the +2500 lines, most of that isn't actually the fast-path parser:
01 +75 gate BPF lookup behind a static key
02 +302 Eth + IPv4/IPv6 + TCP/UDP fast path
03 +180 + VLAN / QinQ
04 +95 + PPPoE
05 +103 + MPLS
06 +134 + IP-in-IP (4in4 / 6in4 / 4in6)
07 +121 + GRE
08 +270 per-shape counters (/proc/net)
09 +37 bound tunnel recursion
10 +1102 KUnit fast/slow-path equivalence test
11 +137 Documentation
------
~+2500The KUnit test is therefore a large fraction of the diff. I don't want to dismiss that as "just tests" -- it is code that has to be maintained -- but its purpose is specifically to make maintaining two implementations safer:
the same corpus is dissected through both paths and the resulting flow_keys are compared byte-for-byte.
More importantly, if I take your protocol-selection concern to its logical conclusion, most of this series disappears.
A v2 could contain only:
1. the static-key optimization for the existing BPF lookup; and
2. one fast path for Eth + IPv4/IPv6 + TCP/UDP.
That is roughly:
+377 / -50 across 5 files
plus whatever reduced KUnit coverage we decide is appropriate for that one shape.
I think there is a useful distinction between that case and the less common protocols. A network operator or vendor with a specialized encapsulation can reasonably deploy a BPF flow dissector. Requiring BPF in order for ordinary hosts to optimize the overwhelmingly common Ethernet/IP/TCP/UDP case is a higher bar, especially since users of RPS/RFS, fq/fq_codel/cake and bonding benefit indirectly through skb_get_hash() without necessarily knowing that the flow dissector is involved.
The performance result is also fairly consistent across architectures. In the isolated Eth + IPv4 + TCP benchmark the straight-line path reduced dissection time by 47-55% across x86, ARM and RISC-V. With all shapes compiled in, the Eth/IP path still improved the tested CPUs, although by a wider 4.7-31.6% range.
I agree that duplicating parsing logic has a maintenance cost. The reason I think the single common shape may still be a reasonable trade is that the surface is small, the fast path is deliberately allowed to bail out to the generic dissector whenever the packet doesn't match, and equivalence with the generic path can be continuously tested rather than assumed.
For the other protocols, I've implemented the same idea as a loadable BPF flow dissector:
https://github.com/randomizedcoder/flow_dissector_ebpf
That seems like a better home for experimenting with specialized shapes without growing the in-kernel parser.
So rather than trying to justify seven in-kernel fast paths, I'd propose shrinking v2 to the static-key BPF lookup optimization plus the single Eth + IPv4/IPv6 + TCP/UDP fast path, with a correspondingly smaller equivalence test.
Would that reduced series be worth posting for review?
Thanks,
Dave Seddon
Quoted 75 lines · Dave Seddon
Thanks Willem, for the quick response and the feedback.
I think your point about where to draw the protocol boundary is a good one.
Eth + IPv4/IPv6 + TCP/UDP is clearly the common case. Beyond that, choosing what belongs in-kernel becomes increasingly subjective: VLAN/QinQ, PPPoE, MPLS, GRE, GTP-U, etc. all have substantial deployments, but that's also an argument for leaving those cases to the BPF flow dissector rather than continuing to grow a parallel parser.
On the +2500 lines, most of that isn't actually the fast-path parser:
01 +75 gate BPF lookup behind a static key 02 +302 Eth + IPv4/IPv6 + TCP/UDP fast path 03 +180 + VLAN / QinQ 04 +95 + PPPoE 05 +103 + MPLS 06 +134 + IP-in-IP (4in4 / 6in4 / 4in6) 07 +121 + GRE 08 +270 per-shape counters (/proc/net) 09 +37 bound tunnel recursion 10 +1102 KUnit fast/slow-path equivalence test 11 +137 Documentation ------ ~+2500The KUnit test is therefore a large fraction of the diff. I don't want to dismiss that as "just tests" -- it is code that has to be maintained -- but its purpose is specifically to make maintaining two implementations safer:
the same corpus is dissected through both paths and the resultingflow_keysare compared byte-for-byte.More importantly, if I take your protocol-selection concern to its logical conclusion, most of this series disappears.
A v2 could contain only:
1. the static-key optimization for the existing BPF lookup; and
2. one fast path for Eth + IPv4/IPv6 + TCP/UDP.That is roughly:
+377 / -50 across 5 filesplus whatever reduced KUnit coverage we decide is appropriate for that one shape.
I think there is a useful distinction between that case and the less common protocols. A network operator or vendor with a specialized encapsulation can reasonably deploy a BPF flow dissector. Requiring BPF in order for ordinary hosts to optimize the overwhelmingly common Ethernet/IP/TCP/UDP case is a higher bar, especially since users of RPS/RFS, fq/fq_codel/cake and bonding benefit indirectly through
skb_get_hash()without necessarily knowing that the flow dissector is involved.The performance result is also fairly consistent across architectures. In the isolated Eth + IPv4 + TCP benchmark the straight-line path reduced dissection time by 47-55% across x86, ARM and RISC-V. With all shapes compiled in, the Eth/IP path still improved the tested CPUs, although by a wider 4.7-31.6% range.
I agree that duplicating parsing logic has a maintenance cost. The reason I think the single common shape may still be a reasonable trade is that the surface is small, the fast path is deliberately allowed to bail out to the generic dissector whenever the packet doesn't match, and equivalence with the generic path can be continuously tested rather than assumed.
For the other protocols, I've implemented the same idea as a loadable BPF flow dissector:
https://github.com/randomizedcoder/flow_dissector_ebpf
That seems like a better home for experimenting with specialized shapes without growing the in-kernel parser.
Quoted 6 lines
So rather than trying to justify seven in-kernel fast paths, I'd propose shrinking v2 to the static-key BPF lookup optimization plus the single Eth + IPv4/IPv6 + TCP/UDP fast path, with a correspondingly smaller equivalence test.
Would that reduced series be worth posting for review?
Mine is just one opinion and not the most important one at that.
Existing non-linear code can also be optimized quite well, e.g., with direct-call optimizations to indirect calls, FDO, etc. That may be a more promising path than logic duplication.
That said, showing a significant performance benefit end-to-end for a representative workload benchmark may help. Say, a on a TCP_RR transport test. It is harder to judge the real world impact of the flow dissection micro-benchmark on its own. With transport level improvement, a single fast patch could warrant a look.
And same independently for the BPF static-key. BPF hooks have been highly optimized by default, I'm a bit skeptical without data that is worthwhile.
But I think the bar would be high.
In my ideal world we would ship a BPF program with the kernel, deprecate the whole C parser and have the BPF parser be autoloaded at boot.