Skip to contentSkip to conversation

BPF link support for tc BPF programs

PATCH Superseded Daniel Borkmann 10 participants 49 messages Jun 7, 2023 – Jul 6, 2023 1 message not in the archive

Subsystems (10)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Daniel Borkmann

author maintainer
#0 lore raw

v2 cover BPF link support for tc BPF programs

This series adds BPF link support for tc BPF programs. We initially presented the motivation, related work and design at last year's LPC conference in the networking & BPF track [0], and a recent update on our progress of the rework during this year's LSF/MM/BPF summit [1]. The main changes are in first two patches and the last two have an extensive batch of test cases we developed along with it, please see individual patches for details. We tested this series with tc-testing selftest suite as well as BPF CI/selftests. Thanks!

[0] https://lpc.events/event/16/contributions/1353/
[1] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

Daniel Borkmann (7):
bpf: Add generic attach/detach/query API for multi-progs
bpf: Add fd-based tcx multi-prog infra with link support
libbpf: Add opts-based attach/detach/query API for tcx
libbpf: Add link-based API for tcx
bpftool: Extend net dump with tcx progs selftests/bpf: Add mprog API tests for BPF tcx opts selftests/bpf: Add mprog API tests for BPF tcx links

 MAINTAINERS                                   |    5 +-
 include/linux/bpf_mprog.h                     |  245 ++
 include/linux/netdevice.h                     |   15 +-
 include/linux/skbuff.h                        |    4 +-
 include/net/sch_generic.h                     |    2 +-
 include/net/tcx.h                             |  157 +
 include/uapi/linux/bpf.h                      |   72 +-
 kernel/bpf/Kconfig                            |    1 +
 kernel/bpf/Makefile                           |    3 +-
 kernel/bpf/mprog.c                            |  476 +++
 kernel/bpf/syscall.c                          |   95 +-
 kernel/bpf/tcx.c                              |  347 +++
 net/Kconfig                                   |    5 +
 net/core/dev.c                                |  267 +-
 net/core/filter.c                             |    4 +-
 net/sched/Kconfig                             |    4 +-
 net/sched/sch_ingress.c                       |   45 +-
 tools/bpf/bpftool/net.c                       |   92 +-
 tools/include/uapi/linux/bpf.h                |   72 +-
 tools/lib/bpf/bpf.c                           |   83 +-
 tools/lib/bpf/bpf.h                           |   61 +-
 tools/lib/bpf/libbpf.c                        |   50 +-
 tools/lib/bpf/libbpf.h                        |   17 +
 tools/lib/bpf/libbpf.map                      |    2 +
 .../selftests/bpf/prog_tests/tc_helpers.h     |   72 +
 .../selftests/bpf/prog_tests/tc_links.c       | 2279 ++++++++++++++
 .../selftests/bpf/prog_tests/tc_opts.c        | 2698 +++++++++++++++++
 .../selftests/bpf/progs/test_tc_link.c        |   40 +
 28 files changed, 6995 insertions(+), 218 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/mprog.c
 create mode 100644 kernel/bpf/tcx.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_links.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_opts.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_tc_link.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v2 1/7 bpf: Add generic attach/detach/query API for multi-progs

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c
MAINTAINERS +1 -0
index c904dba1733b..754a9eeca0a1 100644
@@ -3733,6 +3733,7 @@ F:	include/linux/filter.h
 F:	include/linux/tnum.h
 F:	kernel/bpf/core.c
 F:	kernel/bpf/dispatcher.c
+F:	kernel/bpf/mprog.c
 F:	kernel/bpf/syscall.c
 F:	kernel/bpf/tnum.c
 F:	kernel/bpf/trampoline.c
include/linux/bpf_mprog.h new file +245 -0

245 lines not on this page · raw message

include/uapi/linux/bpf.h +29 -8
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_FIRST		(1U << 5)
+#define BPF_F_LAST		(1U << 6)
+#define BPF_F_ID		(1U << 7)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1433,14 +1440,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+			__u32	replace_bpf_fd;
+		};
+		__u32		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1486,16 +1498,25 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		revision;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
kernel/bpf/Makefile +1 -1
index 1d3892168d32..1bea2eb912cd 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += disasm.o
+obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
 obj-$(CONFIG_BPF_JIT) += dispatcher.o
kernel/bpf/mprog.c new file +476 -0

476 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +29 -8
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_FIRST		(1U << 5)
+#define BPF_F_LAST		(1U << 6)
+#define BPF_F_ID		(1U << 7)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1433,14 +1440,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+			__u32	replace_bpf_fd;
+		};
+		__u32		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1486,16 +1498,25 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		revision;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 277 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c
MAINTAINERS +1 -0
index c904dba1733b..754a9eeca0a1 100644
@@ -3733,6 +3733,7 @@ F:	include/linux/filter.h
 F:	include/linux/tnum.h
 F:	kernel/bpf/core.c
 F:	kernel/bpf/dispatcher.c
+F:	kernel/bpf/mprog.c
 F:	kernel/bpf/syscall.c
 F:	kernel/bpf/tnum.c
 F:	kernel/bpf/trampoline.c
include/linux/bpf_mprog.h new file +166 -0
new file mode 100644
index 000000000000..7399181d8e6c
@@ -0,0 +1,245 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef __BPF_MPROG_H
+#define __BPF_MPROG_H
+
+#include <linux/bpf.h>
+
+#define BPF_MPROG_MAX	64
+#define BPF_MPROG_SWAP	1
+#define BPF_MPROG_FREE	2
+
+struct bpf_mprog_fp {
+	struct bpf_prog *prog;
+};
+
+struct bpf_mprog_cp {
+	struct bpf_link *link;
+	u32 flags;
+};
+
+struct bpf_mprog_entry {
+	struct bpf_mprog_fp fp_items[BPF_MPROG_MAX] ____cacheline_aligned;
+	struct bpf_mprog_cp cp_items[BPF_MPROG_MAX] ____cacheline_aligned;
+	struct bpf_mprog_bundle *parent;
+};
+
+struct bpf_mprog_bundle {
+	struct bpf_mprog_entry a;
+	struct bpf_mprog_entry b;
+	struct rcu_head rcu;
+	struct bpf_prog *ref;
+	atomic_t revision;
+};
+
+struct bpf_tuple {
+	struct bpf_prog *prog;
+	struct bpf_link *link;
+};
+
+static inline struct bpf_mprog_entry *
+bpf_mprog_peer(const struct bpf_mprog_entry *entry)
+{
+	if (entry == &entry->parent->a)
+		return &entry->parent->b;
+	else
+		return &entry->parent->a;
+}
+
+#define bpf_mprog_foreach_tuple(entry, fp, cp, t)			\
+	for (fp = &entry->fp_items[0], cp = &entry->cp_items[0];	\
+	     ({								\
+		t.prog = READ_ONCE(fp->prog);				\
+		t.link = cp->link;					\
+		t.prog;							\
+	      });							\
+	     fp++, cp++)
+
+#define bpf_mprog_foreach_prog(entry, fp, p)				\
+	for (fp = &entry->fp_items[0];					\
+	     (p = READ_ONCE(fp->prog));					\
+	     fp++)
+
+static inline struct bpf_mprog_entry *bpf_mprog_create(size_t extra_size)
+{
+	struct bpf_mprog_bundle *bundle;
+
+	/* Fast-path items are not extensible, must only contain prog pointer! */
+	BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+	/* Control-path items can be extended w/o affecting fast-path. */
+	BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) != ARRAY_SIZE(bundle->a.cp_items));
+
+	bundle = kzalloc(sizeof(*bundle) + extra_size, GFP_KERNEL);
+	if (bundle) {
+		atomic_set(&bundle->revision, 1);
+		bundle->a.parent = bundle;
+		bundle->b.parent = bundle;
+		return &bundle->a;
+	}
+	return NULL;
+}
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+	kfree_rcu(entry->parent, rcu);
+}
+
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+				      struct bpf_prog *prog)
+{
+	WARN_ON_ONCE(entry->parent->ref);
+	entry->parent->ref = prog;
+}
+
+static inline u32 bpf_mprog_flags(u32 cur_flags, u32 req_flags, u32 flag)
+{
+	if (req_flags & flag)
+		cur_flags |= flag;
+	else
+		cur_flags &= ~flag;
+	return cur_flags;
+}
+
+static inline u32 bpf_mprog_max(void)
+{
+	return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline struct bpf_prog *bpf_mprog_first(struct bpf_mprog_entry *entry)
+{
+	return READ_ONCE(entry->fp_items[0].prog);
+}
+
+static inline struct bpf_prog *bpf_mprog_last(struct bpf_mprog_entry *entry)
+{
+	struct bpf_prog *tmp, *prog = NULL;
+	struct bpf_mprog_fp *fp;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp)
+		prog = tmp;
+	return prog;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+				    struct bpf_prog *prog)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *tmp;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp) {
+		if (tmp == prog)
+			return true;
+	}
+	return false;
+}
+
+static inline struct bpf_prog *bpf_mprog_first_reg(struct bpf_mprog_entry *entry)
+{
+	struct bpf_tuple tuple = {};
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+
+	bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+		if (cp->flags & BPF_F_FIRST)
+			continue;
+		return tuple.prog;
+	}
+	return NULL;
+}
+
+static inline struct bpf_prog *bpf_mprog_last_reg(struct bpf_mprog_entry *entry)
+{
+	struct bpf_tuple tuple = {};
+	struct bpf_prog *prog = NULL;
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+
+	bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+		if (cp->flags & BPF_F_LAST)
+			break;
+		prog = tuple.prog;
+	}
+	return prog;
+}
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{

[..]

include/linux/bpf_mprog.h line 169

Quoted 3 lines
include/linux/bpf_mprog.h new file +3 -0
+	do {
+		atomic_inc(&entry->parent->revision);
+	} while (atomic_read(&entry->parent->revision) == 0);

Can you explain more what's going on here? Maybe with a comment?

include/linux/bpf_mprog.h line 174

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+	synchronize_rcu();
+	if (entry->parent->ref) {
+		bpf_prog_put(entry->parent->ref);
+		entry->parent->ref = NULL;
+	}

I'm assuming this is to guard the detach path? But isn't bpf_prog_put already doing the deferred dealloc? So calling it without synchronize_rcu here should be ok?

Quoted 27 lines
include/linux/bpf_mprog.h new file +27 -0
+}
+
+static inline void bpf_mprog_entry_clear(struct bpf_mprog_entry *entry)
+{
+	memset(entry->fp_items, 0, sizeof(entry->fp_items));
+	memset(entry->cp_items, 0, sizeof(entry->cp_items));
+}
+
+static inline u64 bpf_mprog_revision(struct bpf_mprog_entry *entry)
+{
+	return atomic_read(&entry->parent->revision);
+}
+
+static inline void bpf_mprog_read(struct bpf_mprog_entry *entry, u32 which,
+				  struct bpf_mprog_fp **fp_dst,
+				  struct bpf_mprog_cp **cp_dst)
+{
+	*fp_dst = &entry->fp_items[which];
+	*cp_dst = &entry->cp_items[which];
+}
+
+static inline void bpf_mprog_write(struct bpf_mprog_fp *fp_dst,
+				   struct bpf_mprog_cp *cp_dst,
+				   struct bpf_tuple *tuple, u32 flags)
+{
+	WRITE_ONCE(fp_dst->prog, tuple->prog);
+	cp_dst->link  = tuple->link;

include/linux/bpf_mprog.h line 211

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+	cp_dst->flags = flags;
+}
+
+static inline void bpf_mprog_copy(struct bpf_mprog_fp *fp_dst,
+				  struct bpf_mprog_cp *cp_dst,
+				  struct bpf_mprog_fp *fp_src,
+				  struct bpf_mprog_cp *cp_src)
+{
+	WRITE_ONCE(fp_dst->prog, READ_ONCE(fp_src->prog));
+	memcpy(cp_dst, cp_src, sizeof(*cp_src));

nit: why not simply *cp_dst = *cp_src? memcpy somewhat implies (in my mind) that we are copying several entries..

Quoted 614 lines
include/linux/bpf_mprog.h new file +34 -0
+}
+
+static inline void bpf_mprog_copy_range(struct bpf_mprog_entry *peer,
+					struct bpf_mprog_entry *entry,
+					u32 idx_peer, u32 idx_entry, u32 num)
+{
+	memcpy(&peer->fp_items[idx_peer], &entry->fp_items[idx_entry],
+	       num * sizeof(peer->fp_items[0]));
+	memcpy(&peer->cp_items[idx_peer], &entry->cp_items[idx_entry],
+	       num * sizeof(peer->cp_items[0]));
+}
+
+static inline u32 bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *tmp;
+	u32 num = 0;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp)
+		num++;
+	return num;
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object,
+		     u32 expected_revision);
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object,
+		     u32 expected_revision);
+
+int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
+		    struct bpf_mprog_entry *entry);
+
+#endif /* __BPF_MPROG_H */
include/uapi/linux/bpf.h +29 -8
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_FIRST		(1U << 5)
+#define BPF_F_LAST		(1U << 6)
+#define BPF_F_ID		(1U << 7)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1433,14 +1440,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+			__u32	replace_bpf_fd;
+		};
+		__u32		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1486,16 +1498,25 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		revision;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
kernel/bpf/Makefile +1 -1
index 1d3892168d32..1bea2eb912cd 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += disasm.o
+obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
 obj-$(CONFIG_BPF_JIT) += dispatcher.o
kernel/bpf/mprog.c new file +476 -0
new file mode 100644
index 000000000000..efc3b73f8bf5
@@ -0,0 +1,476 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+#include <linux/filter.h>
+
+static int bpf_mprog_tuple_relative(struct bpf_tuple *tuple,
+				    u32 object, u32 flags,
+				    enum bpf_prog_type type)
+{
+	struct bpf_prog *prog;
+	struct bpf_link *link;
+
+	memset(tuple, 0, sizeof(*tuple));
+	if (!(flags & (BPF_F_REPLACE | BPF_F_BEFORE | BPF_F_AFTER)))
+		return object || (flags & (BPF_F_ID | BPF_F_LINK)) ?
+		       -EINVAL : 0;
+	if (flags & BPF_F_LINK) {
+		if (flags & BPF_F_ID)
+			link = bpf_link_by_id(object);
+		else
+			link = bpf_link_get_from_fd(object);
+		if (IS_ERR(link))
+			return PTR_ERR(link);
+		if (type && link->prog->type != type) {
+			bpf_link_put(link);
+			return -EINVAL;
+		}
+		tuple->link = link;
+		tuple->prog = link->prog;
+	} else {
+		if (flags & BPF_F_ID)
+			prog = bpf_prog_by_id(object);
+		else
+			prog = bpf_prog_get(object);
+		if (IS_ERR(prog)) {
+			if (!object &&
+			    !(flags & BPF_F_ID))
+				return 0;
+			return PTR_ERR(prog);
+		}
+		if (type && prog->type != type) {
+			bpf_prog_put(prog);
+			return -EINVAL;
+		}
+		tuple->link = NULL;
+		tuple->prog = prog;
+	}
+	return 0;
+}
+
+static void bpf_mprog_tuple_put(struct bpf_tuple *tuple)
+{
+	if (tuple->link)
+		bpf_link_put(tuple->link);
+	else if (tuple->prog)
+		bpf_prog_put(tuple->prog);
+}
+
+static int bpf_mprog_replace(struct bpf_mprog_entry *entry,
+			     struct bpf_tuple *ntuple,
+			     struct bpf_tuple *rtuple, u32 rflags)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	struct bpf_prog *oprog;
+	u32 iflags;
+	int i;
+
+	if (rflags & (BPF_F_BEFORE | BPF_F_AFTER | BPF_F_LINK))
+		return -EINVAL;
+	if (rtuple->prog != ntuple->prog &&
+	    bpf_mprog_exists(entry, ntuple->prog))
+		return -EEXIST;
+	for (i = 0; i < bpf_mprog_max(); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		oprog = READ_ONCE(fp->prog);
+		if (!oprog)
+			break;
+		if (oprog != rtuple->prog)
+			continue;
+		if (cp->link != ntuple->link)
+			return -EBUSY;
+		iflags = cp->flags;
+		if ((iflags & BPF_F_FIRST) !=
+		    (rflags & BPF_F_FIRST)) {
+			iflags = bpf_mprog_flags(iflags, rflags,
+						 BPF_F_FIRST);
+			if ((iflags & BPF_F_FIRST) &&
+			    rtuple->prog != bpf_mprog_first(entry))
+				return -EACCES;
+		}
+		if ((iflags & BPF_F_LAST) !=
+		    (rflags & BPF_F_LAST)) {
+			iflags = bpf_mprog_flags(iflags, rflags,
+						 BPF_F_LAST);
+			if ((iflags & BPF_F_LAST) &&
+			    rtuple->prog != bpf_mprog_last(entry))
+				return -EACCES;
+		}
+		bpf_mprog_write(fp, cp, ntuple, iflags);
+		if (!ntuple->link)
+			bpf_prog_put(oprog);
+		return 0;
+	}
+	return -ENOENT;
+}
+
+static int bpf_mprog_head_tail(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *ntuple,
+			       struct bpf_tuple *rtuple, u32 aflags)
+{
+	struct bpf_mprog_entry *peer;
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	struct bpf_prog *oprog;
+	u32 iflags, items;
+
+	if (bpf_mprog_exists(entry, ntuple->prog))
+		return -EEXIST;
+	items = bpf_mprog_total(entry);
+	peer = bpf_mprog_peer(entry);
+	bpf_mprog_entry_clear(peer);
+	if (aflags & BPF_F_FIRST) {
+		if (aflags & BPF_F_AFTER)
+			return -EINVAL;
+		bpf_mprog_read(entry, 0, &fp, &cp);
+		iflags = cp->flags;
+		if (iflags & BPF_F_FIRST)
+			return -EBUSY;
+		if (aflags & BPF_F_LAST) {
+			if (aflags & BPF_F_BEFORE)
+				return -EINVAL;
+			if (items)
+				return -EBUSY;
+			bpf_mprog_read(peer, 0, &fp, &cp);
+			bpf_mprog_write(fp, cp, ntuple,
+					BPF_F_FIRST | BPF_F_LAST);
+			return BPF_MPROG_SWAP;
+		}
+		if (aflags & BPF_F_BEFORE) {
+			oprog = READ_ONCE(fp->prog);
+			if (oprog != rtuple->prog ||
+			    (rtuple->link &&
+			     rtuple->link != cp->link))
+				return -EBUSY;
+		}
+		if (items >= bpf_mprog_max())
+			return -ENOSPC;
+		bpf_mprog_read(peer, 0, &fp, &cp);
+		bpf_mprog_write(fp, cp, ntuple, BPF_F_FIRST);
+		bpf_mprog_copy_range(peer, entry, 1, 0, items);
+		return BPF_MPROG_SWAP;
+	}
+	if (aflags & BPF_F_LAST) {
+		if (aflags & BPF_F_BEFORE)
+			return -EINVAL;
+		if (items) {
+			bpf_mprog_read(entry, items - 1, &fp, &cp);
+			iflags = cp->flags;
+			if (iflags & BPF_F_LAST)
+				return -EBUSY;
+			if (aflags & BPF_F_AFTER) {
+				oprog = READ_ONCE(fp->prog);
+				if (oprog != rtuple->prog ||
+				    (rtuple->link &&
+				     rtuple->link != cp->link))
+					return -EBUSY;
+			}
+			if (items >= bpf_mprog_max())
+				return -ENOSPC;
+		} else {
+			if (aflags & BPF_F_AFTER)
+				return -EBUSY;
+		}
+		bpf_mprog_read(peer, items, &fp, &cp);
+		bpf_mprog_write(fp, cp, ntuple, BPF_F_LAST);
+		bpf_mprog_copy_range(peer, entry, 0, 0, items);
+		return BPF_MPROG_SWAP;
+	}
+	return -ENOENT;
+}
+
+static int bpf_mprog_add(struct bpf_mprog_entry *entry,
+			 struct bpf_tuple *ntuple,
+			 struct bpf_tuple *rtuple, u32 aflags)
+{
+	struct bpf_mprog_fp *fp_dst, *fp_src;
+	struct bpf_mprog_cp *cp_dst, *cp_src;
+	struct bpf_mprog_entry *peer;
+	struct bpf_prog *oprog;
+	bool found = false;
+	u32 items;
+	int i, j;
+
+	items = bpf_mprog_total(entry);
+	if (items >= bpf_mprog_max())
+		return -ENOSPC;
+	if ((aflags & (BPF_F_BEFORE | BPF_F_AFTER)) ==
+	    (BPF_F_BEFORE | BPF_F_AFTER))
+		return -EINVAL;
+	if (bpf_mprog_exists(entry, ntuple->prog))
+		return -EEXIST;
+	if (!rtuple->prog && (aflags & (BPF_F_BEFORE | BPF_F_AFTER))) {
+		if (!items)
+			aflags &= ~(BPF_F_AFTER | BPF_F_BEFORE);
+		if (aflags & BPF_F_BEFORE)
+			rtuple->prog = bpf_mprog_first_reg(entry);
+		if (aflags & BPF_F_AFTER)
+			rtuple->prog = bpf_mprog_last_reg(entry);
+		if (!rtuple->prog)
+			aflags &= ~(BPF_F_AFTER | BPF_F_BEFORE);
+		else
+			bpf_prog_inc(rtuple->prog);
+	}
+	peer = bpf_mprog_peer(entry);
+	bpf_mprog_entry_clear(peer);
+	for (i = 0, j = 0; i < bpf_mprog_max(); i++, j++) {
+		bpf_mprog_read(entry, i, &fp_src, &cp_src);
+		bpf_mprog_read(peer,  j, &fp_dst, &cp_dst);
+		oprog = READ_ONCE(fp_src->prog);
+		if (!oprog) {
+			if (i != j)
+				break;
+			if (i > 0) {
+				bpf_mprog_read(entry, i - 1,
+					       &fp_src, &cp_src);
+				if (cp_src->flags & BPF_F_LAST) {
+					if (cp_src->flags & BPF_F_FIRST)
+						return -EBUSY;
+					bpf_mprog_copy(fp_dst, cp_dst,
+						       fp_src, cp_src);
+					bpf_mprog_read(peer, --j,
+						       &fp_dst, &cp_dst);
+				}
+			}
+			bpf_mprog_write(fp_dst, cp_dst, ntuple, 0);
+			break;
+		}
+		if (aflags & (BPF_F_BEFORE | BPF_F_AFTER)) {
+			if (rtuple->prog != oprog ||
+			    (rtuple->link &&
+			     rtuple->link != cp_src->link))
+				goto next;
+			found = true;
+			if (aflags & BPF_F_BEFORE) {
+				if (cp_src->flags & BPF_F_FIRST)
+					return -EBUSY;
+				bpf_mprog_write(fp_dst, cp_dst, ntuple, 0);
+				bpf_mprog_read(peer, ++j, &fp_dst, &cp_dst);
+				goto next;
+			}
+			if (aflags & BPF_F_AFTER) {
+				if (cp_src->flags & BPF_F_LAST)
+					return -EBUSY;
+				bpf_mprog_copy(fp_dst, cp_dst,
+					       fp_src, cp_src);
+				bpf_mprog_read(peer, ++j, &fp_dst, &cp_dst);
+				bpf_mprog_write(fp_dst, cp_dst, ntuple, 0);
+				continue;
+			}
+		}
+next:
+		bpf_mprog_copy(fp_dst, cp_dst,
+			       fp_src, cp_src);
+	}
+	if (rtuple->prog && !found)
+		return -ENOENT;
+	return BPF_MPROG_SWAP;
+}
+
+static int bpf_mprog_del(struct bpf_mprog_entry *entry,
+			 struct bpf_tuple *dtuple,
+			 struct bpf_tuple *rtuple, u32 dflags)
+{
+	struct bpf_mprog_fp *fp_dst, *fp_src;
+	struct bpf_mprog_cp *cp_dst, *cp_src;
+	struct bpf_mprog_entry *peer;
+	struct bpf_prog *oprog;
+	bool found = false;
+	int i, j, ret;
+
+	if (dflags & BPF_F_REPLACE)
+		return -EINVAL;
+	if (dflags & BPF_F_FIRST) {
+		oprog = bpf_mprog_first(entry);
+		if (dtuple->prog &&
+		    dtuple->prog != oprog)
+			return -ENOENT;
+		dtuple->prog = oprog;
+	}
+	if (dflags & BPF_F_LAST) {
+		oprog = bpf_mprog_last(entry);
+		if (dtuple->prog &&
+		    dtuple->prog != oprog)
+			return -ENOENT;
+		dtuple->prog = oprog;
+	}
+	if (!rtuple->prog && (dflags & (BPF_F_BEFORE | BPF_F_AFTER))) {
+		if (dtuple->prog)
+			return -EINVAL;
+		if (dflags & BPF_F_BEFORE)
+			dtuple->prog = bpf_mprog_first_reg(entry);
+		if (dflags & BPF_F_AFTER)
+			dtuple->prog = bpf_mprog_last_reg(entry);
+		if (dtuple->prog)
+			dflags &= ~(BPF_F_AFTER | BPF_F_BEFORE);
+	}
+	for (i = 0; i < bpf_mprog_max(); i++) {
+		bpf_mprog_read(entry, i, &fp_src, &cp_src);
+		oprog = READ_ONCE(fp_src->prog);
+		if (!oprog)
+			break;
+		if (dflags & (BPF_F_BEFORE | BPF_F_AFTER)) {
+			if (rtuple->prog != oprog ||
+			    (rtuple->link &&
+			     rtuple->link != cp_src->link))
+				continue;
+			found = true;
+			if (dflags & BPF_F_BEFORE) {
+				if (!i)
+					return -ENOENT;
+				bpf_mprog_read(entry, i - 1,
+					       &fp_src, &cp_src);
+				oprog = READ_ONCE(fp_src->prog);
+				if (dtuple->prog &&
+				    dtuple->prog != oprog)
+					return -ENOENT;
+				dtuple->prog = oprog;
+				break;
+			}
+			if (dflags & BPF_F_AFTER) {
+				bpf_mprog_read(entry, i + 1,
+					       &fp_src, &cp_src);
+				oprog = READ_ONCE(fp_src->prog);
+				if (dtuple->prog &&
+				    dtuple->prog != oprog)
+					return -ENOENT;
+				dtuple->prog = oprog;
+				break;
+			}
+		}
+	}
+	if (!dtuple->prog || (rtuple->prog && !found))
+		return -ENOENT;
+	peer = bpf_mprog_peer(entry);
+	bpf_mprog_entry_clear(peer);
+	ret = -ENOENT;
+	for (i = 0, j = 0; i < bpf_mprog_max(); i++) {
+		bpf_mprog_read(entry, i, &fp_src, &cp_src);
+		bpf_mprog_read(peer,  j, &fp_dst, &cp_dst);
+		oprog = READ_ONCE(fp_src->prog);
+		if (!oprog)
+			break;
+		if (oprog != dtuple->prog) {
+			bpf_mprog_copy(fp_dst, cp_dst,
+				       fp_src, cp_src);
+			j++;
+		} else {
+			if (cp_src->link != dtuple->link)
+				return -EBUSY;
+			if (!cp_src->link)
+				bpf_mprog_mark_ref(entry, dtuple->prog);
+			ret = BPF_MPROG_SWAP;
+		}
+	}
+	if (!bpf_mprog_total(peer))
+		ret = BPF_MPROG_FREE;
+	return ret;
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object,
+		     u32 expected_revision)
+{
+	struct bpf_tuple rtuple, ntuple = {
+		.prog = prog,
+		.link = link,
+	};
+	int ret;
+
+	if (expected_revision &&
+	    expected_revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	ret = bpf_mprog_tuple_relative(&rtuple, object, flags, prog->type);
+	if (ret)
+		return ret;
+	if (flags & BPF_F_REPLACE)
+		ret = bpf_mprog_replace(entry, &ntuple, &rtuple, flags);
+	else if (flags & (BPF_F_FIRST | BPF_F_LAST))
+		ret = bpf_mprog_head_tail(entry, &ntuple, &rtuple, flags);
+	else
+		ret = bpf_mprog_add(entry, &ntuple, &rtuple, flags);
+	bpf_mprog_tuple_put(&rtuple);
+	return ret;
+}
+
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object,
+		     u32 expected_revision)
+{
+	struct bpf_tuple rtuple, dtuple = {
+		.prog = prog,
+		.link = link,
+	};
+	int ret;
+
+	if (expected_revision &&
+	    expected_revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	ret = bpf_mprog_tuple_relative(&rtuple, object, flags,
+				       prog ? prog->type :
+				       BPF_PROG_TYPE_UNSPEC);
+	if (ret)
+		return ret;
+	ret = bpf_mprog_del(entry, &dtuple, &rtuple, flags);
+	bpf_mprog_tuple_put(&rtuple);
+	return ret;
+}
+
+int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
+		    struct bpf_mprog_entry *entry)
+{
+	u32 i, id, flags = 0, count, revision;
+	u32 __user *uprog_id, *uprog_af;
+	u32 __user *ulink_id, *ulink_af;
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	struct bpf_prog *prog;
+	int ret = 0;
+
+	if (attr->query.query_flags || attr->query.attach_flags)
+		return -EINVAL;
+	revision = bpf_mprog_revision(entry);
+	count = bpf_mprog_total(entry);
+	if (copy_to_user(&uattr->query.attach_flags, &flags, sizeof(flags)))
+		return -EFAULT;
+	if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
+		return -EFAULT;
+	if (copy_to_user(&uattr->query.count, &count, sizeof(count)))
+		return -EFAULT;
+	uprog_id = u64_to_user_ptr(attr->query.prog_ids);
+	if (attr->query.count == 0 || !uprog_id || !count)
+		return 0;
+	if (attr->query.count < count) {
+		count = attr->query.count;
+		ret = -ENOSPC;
+	}
+	uprog_af = u64_to_user_ptr(attr->query.prog_attach_flags);
+	ulink_id = u64_to_user_ptr(attr->query.link_ids);
+	ulink_af = u64_to_user_ptr(attr->query.link_attach_flags);
+	for (i = 0; i < ARRAY_SIZE(entry->fp_items); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		prog = READ_ONCE(fp->prog);
+		if (!prog)
+			break;
+		id = prog->aux->id;
+		if (copy_to_user(uprog_id + i, &id, sizeof(id)))
+			return -EFAULT;
+		id = cp->link ? cp->link->id : 0;
+		if (ulink_id &&
+		    copy_to_user(ulink_id + i, &id, sizeof(id)))
+			return -EFAULT;
+		flags = cp->flags;
+		if (uprog_af && !id &&
+		    copy_to_user(uprog_af + i, &flags, sizeof(flags)))
+			return -EFAULT;
+		if (ulink_af && id &&
+		    copy_to_user(ulink_af + i, &flags, sizeof(flags)))
+			return -EFAULT;
+		if (i + 1 == count)
+			break;
+	}
+	return ret;
+}
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)

[..]

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST		(1U << 5)
+#define BPF_F_LAST		(1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

But if everyone besides myself is on board with first/last, maybe at least put a comment here saying that only a single program can be first/last? And the users are advised not to use these unless they really really really need to be first/last. (IOW, feels like first/last should be reserved for observability tools/etc).

Quoted 62 lines
tools/include/uapi/linux/bpf.h +24 -8
+#define BPF_F_ID		(1U << 7)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1433,14 +1440,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+			__u32	replace_bpf_fd;
+		};
+		__u32		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1486,16 +1498,25 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		revision;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 95 lines · Stanislav Fomichev
Quoted 93 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE (1U << 0)
 #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Quoted 6 lines

But if everyone besides myself is on board with first/last, maybe at least put a comment here saying that only a single program can be first/last? And the users are advised not to use these unless they really really really need to be first/last. (IOW, feels like first/last should be reserved for observability tools/etc).

+1, we can definitely make it clear in API that this will prevent anyone else from being attached as FIRST/LAST, so it's not cooperative in nature and has to be very consciously evaluated.

Quoted 6 lines
Quoted 6 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_ID             (1U << 7)
+#define BPF_F_LINK           BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel

[...]

#4 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 129 lines · Andrii Nakryiko
Quoted 95 lines · Stanislav Fomichev
Quoted 93 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE (1U << 0)
 #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

Quoted 8 lines

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

Quoted 18 lines
Quoted 5 lines

But if everyone besides myself is on board with first/last, maybe at least put a comment here saying that only a single program can be first/last? And the users are advised not to use these unless they really really really need to be first/last. (IOW, feels like first/last should be reserved for observability tools/etc).

+1, we can definitely make it clear in API that this will prevent anyone else from being attached as FIRST/LAST, so it's not cooperative in nature and has to be very consciously evaluated.

Quoted 6 lines
Quoted 6 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_ID             (1U << 7)
+#define BPF_F_LINK           BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel

[...]

#5 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 136 lines · Stanislav Fomichev
Quoted 134 lines · Andrii Nakryiko
Quoted 95 lines · Stanislav Fomichev
Quoted 93 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE (1U << 0)
 #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 3 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 5 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 19 lines
Quoted 19 lines
Quoted 6 lines

But if everyone besides myself is on board with first/last, maybe at least put a comment here saying that only a single program can be first/last? And the users are advised not to use these unless they really really really need to be first/last. (IOW, feels like first/last should be reserved for observability tools/etc).

+1, we can definitely make it clear in API that this will prevent anyone else from being attached as FIRST/LAST, so it's not cooperative in nature and has to be very consciously evaluated.

Quoted 6 lines
Quoted 6 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_ID             (1U << 7)
+#define BPF_F_LINK           BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel

[...]

#6 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 165 lines · Andrii Nakryiko
Quoted 136 lines · Stanislav Fomichev
Quoted 134 lines · Andrii Nakryiko
Quoted 95 lines · Stanislav Fomichev
Quoted 93 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE (1U << 0)
 #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 3 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 5 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

Quoted 9 lines

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

But, if you think you need F_FIRST/F_LAST, let's have them. I just personally don't see us using them (nor do I see why they have to be used upstream). The only thing that makes sense is probably for cilium to do F_FIRST|F_LAST to prevent other things from breaking it?

#7 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 10 lines · Stanislav Fomichev

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

+1.
I have the same concerns as expressed during lsfmmbpf. This first/last is a foot gun.
It puts the whole API back into a single user situation. Without "first api" the users are forced to talk to each other and come up with an arbitration mechanism. A daemon to control the order or something like that.
With "first api" there is no incentive to do so.

#8 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 17 lines · Alexei Starovoitov
Quoted 10 lines · Stanislav Fomichev

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

Quoted 8 lines

+1.
I have the same concerns as expressed during lsfmmbpf. This first/last is a foot gun.
It puts the whole API back into a single user situation. Without "first api" the users are forced to talk to each other and come up with an arbitration mechanism. A daemon to control the order or something like that.
With "first api" there is no incentive to do so.

If Cilium and some other company X both produce, say, anti-DDOS solution which cannot co-exist with any other anti-DDOS program and either of them needs to guarantee that their program runs first, then FIRST is what would be used by both to prevent accidental breakage of each other (which is basically what happened with Cilium and some other networking solution, don't remember the name). It's better for one of them to loudly fail to attach than silently break other solution with end users struggling to understand what's going on.

You and Stanislav keep insisting that any combination of any BPF programs should co-exist, and I don't understand why we can or should presume that. I think we are conflating generic API (and kernel *not* making any assumptions about such API usage) with encouraging collaborative BPF attachment policies. They are orthogonal and are not in conflict with each other.

But we lived without FIRST/LAST guarantees till now, that's fine, I'll stop fighting this.

#9 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 44 lines · Andrii Nakryiko
Quoted 17 lines · Alexei Starovoitov
Quoted 10 lines · Stanislav Fomichev

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

Quoted 8 lines

+1.
I have the same concerns as expressed during lsfmmbpf. This first/last is a foot gun.
It puts the whole API back into a single user situation. Without "first api" the users are forced to talk to each other and come up with an arbitration mechanism. A daemon to control the order or something like that.
With "first api" there is no incentive to do so.

If Cilium and some other company X both produce, say, anti-DDOS solution which cannot co-exist with any other anti-DDOS program and either of them needs to guarantee that their program runs first, then FIRST is what would be used by both to prevent accidental breakage of each other (which is basically what happened with Cilium and some other networking solution, don't remember the name). It's better for one of them to loudly fail to attach than silently break other solution with end users struggling to understand what's going on.

You and Stanislav keep insisting that any combination of any BPF programs should co-exist, and I don't understand why we can or should presume that. I think we are conflating generic API (and kernel *not* making any assumptions about such API usage) with encouraging collaborative BPF attachment policies. They are orthogonal and are not in conflict with each other.

Quoted 2 lines

But we lived without FIRST/LAST guarantees till now, that's fine, I'll stop fighting this.

I'm not saying this situation where there are several incompatible programs doesn't exist. All I'm saying is that imo this is a policy that doesn't belong to the kernel. Or maybe even let's put it that way: F_FIRST and F_LAST isn't flexible enough to express this policy. External systemd-like arbiter should express the dependencies/ordering/conflicts/etc. And F_BEFORE and F_AFTER is enough for that sysmted-like entity to do the rest.

#10 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Stanislav Fomichev
Quoted 185 lines · Stanislav Fomichev
Quoted 165 lines · Andrii Nakryiko
Quoted 136 lines · Stanislav Fomichev
Quoted 134 lines · Andrii Nakryiko
Quoted 95 lines · Stanislav Fomichev
Quoted 93 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE (1U << 0)
 #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 3 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 5 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

Quoted 9 lines

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me to be policy in the kernel, which would be much better handled in userspace like we do for so many other things. So we should rather expose a hook to allow userspace to set the policy, as we've discussed before; I definitely think we should add that at some point! Although obviously it doesn't have to be part of this series...

-Toke

author maintainer
#11 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 172 lines · Toke Høiland-Jørgensen
Quoted 171 lines · Stanislav Fomichev
Quoted 169 lines · Andrii Nakryiko
Quoted 133 lines · Stanislav Fomichev
Quoted 132 lines · Andrii Nakryiko
Quoted 93 lines · Stanislav Fomichev
Quoted 92 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

   [0] https://lore.kernel.org/bpf/[email protected]/
   [1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
   [2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
---
  MAINTAINERS                    |   1 +
  include/linux/bpf_mprog.h      | 245 +++++++++++++++++
  include/uapi/linux/bpf.h       |  37 ++-
  kernel/bpf/Makefile            |   2 +-
  kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
  tools/include/uapi/linux/bpf.h |  37 ++-
  6 files changed, 781 insertions(+), 17 deletions(-)

create mode 100644 include/linux/bpf_mprog.h
create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
   */
  #define BPF_F_ALLOW_OVERRIDE (1U << 0)
  #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
  #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

Quoted 7 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

Quoted 5 lines

to be policy in the kernel, which would be much better handled in userspace like we do for so many other things. So we should rather expose a hook to allow userspace to set the policy, as we've discussed before; I definitely think we should add that at some point! Although obviously it doesn't have to be part of this series...

Imo, it would be better if we could avoid that.. it feels like we're trying to shoot sparrows with cannon, e.g. when this API gets reused for other attach hooks, then for each of them you need yet another policy program. I don't think that's a good user experience, and I presume this is then single-user program, thus you'll run into the same race in the end - whichever management daemon or application gets to install this policy program first wins. This is potentially just shifting the same issue one level higher, imo.

Thanks,
Daniel

author maintainer
#12 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 193 lines · Daniel Borkmann
Quoted 172 lines · Toke Høiland-Jørgensen
Quoted 171 lines · Stanislav Fomichev
Quoted 169 lines · Andrii Nakryiko
Quoted 133 lines · Stanislav Fomichev
Quoted 132 lines · Andrii Nakryiko
Quoted 93 lines · Stanislav Fomichev
Quoted 92 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

   I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

   The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

   - Support prog-based attach/detach and link API
   - Dependency directives (can also be combined):
     - BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
       - BPF_F_ID flag as {fd,id} toggle
       - BPF_F_LINK flag as {prog,link} toggle
       - If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
       - Enforced only at attach time
     - BPF_F_{FIRST,LAST}
       - Enforced throughout the bpf_mprog state's lifetime
       - Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
   - Internal revision counter and optionally being able to pass expected_revision
   - User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
   - Query also gets extension for link_ids array and link_attach_flags:
     - prog_ids are always filled with program IDs
     - link_ids are filled with link IDs when link was used, otherwise 0
     - {prog,link}_attach_flags for holding {prog,link}-specific flags
   - Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

   [0] https://lore.kernel.org/bpf/[email protected]/
   [1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
   [2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
  MAINTAINERS                    |   1 + include/linux/bpf_mprog.h      | 245 +++++++++++++++++ include/uapi/linux/bpf.h       |  37 ++- kernel/bpf/Makefile            |   2 +-
  kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++ tools/include/uapi/linux/bpf.h |  37 ++-

  6 files changed, 781 insertions(+), 17 deletions(-)

  create mode 100644 include/linux/bpf_mprog.h
  create mode 100644 kernel/bpf/mprog.c

[...]

Quoted 15 lines
Quoted 12 lines
tools/include/uapi/linux/bpf.h +3 -0
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
    */
   #define BPF_F_ALLOW_OVERRIDE (1U << 0)
   #define BPF_F_ALLOW_MULTI    (1U << 1)
+/* Generic attachment flags. */
   #define BPF_F_REPLACE                (1U << 2)
+#define BPF_F_BEFORE         (1U << 3)
+#define BPF_F_AFTER          (1U << 4)

[..]

Quoted 8 lines

tools/include/uapi/linux/bpf.h

Quoted 2 lines
tools/include/uapi/linux/bpf.h +2 -0
+#define BPF_F_FIRST          (1U << 5)
+#define BPF_F_LAST           (1U << 6)

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair  for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 11 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

Quoted 2 lines
Quoted 2 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?).

Just to add, in K8s there can be multiple CNIs chained together, and there is also no common management daemon as you have in G or Meta. So yes, K8s is special snowflake, but everyone outside of the big hyperscalers are relying on it as a platform, so we do need to have a solution for the trivial, above- mentioned scenario if we drop the first/last.

Quoted 15 lines
Quoted 5 lines
Quoted 2 lines

And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

Quoted 5 lines

to be policy in the kernel, which would be much better handled in userspace like we do for so many other things. So we should rather expose a hook to allow userspace to set the policy, as we've discussed before; I definitely think we should add that at some point! Although obviously it doesn't have to be part of this series...

Quoted 8 lines

Imo, it would be better if we could avoid that.. it feels like we're trying to shoot sparrows with cannon, e.g. when this API gets reused for other attach hooks, then for each of them you need yet another policy program. I don't think that's a good user experience, and I presume this is then single-user program, thus you'll run into the same race in the end - whichever management daemon or application gets to install this policy program first wins. This is potentially just shifting the same issue one level higher, imo.

Thanks,
Daniel

#13 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 73 lines · Daniel Borkmann
Quoted 53 lines
Quoted 53 lines
Quoted 52 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 7 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 10 lines
Quoted 5 lines

to be policy in the kernel, which would be much better handled in userspace like we do for so many other things. So we should rather expose a hook to allow userspace to set the policy, as we've discussed before; I definitely think we should add that at some point! Although obviously it doesn't have to be part of this series...

Imo, it would be better if we could avoid that.. it feels like we're trying to shoot sparrows with cannon, e.g. when this API gets reused for other attach hooks, then for each of them you need yet another policy program.

Or a single one that understands multiple program types. Sharing the multi-prog implementation is helpful here.

Quoted 5 lines

I don't think that's a good user experience, and I presume this is then single-user program, thus you'll run into the same race in the end - whichever management daemon or application gets to install this policy program first wins. This is potentially just shifting the same issue one level higher, imo.

Sure, we're shifting the problem one level higher, i.e., out of the kernel. That's the point: this is better solved in userspace, so different environments can solve it according to their needs :)

I'm not against having one policy agent on the system, I just don't think the kernel should hard-code one particular solution to the policy problem. Much better to merge this without it, and then iterate on different options (and happy to help with this!), instead of locking the UAPI into a single solution straight away.

-Toke

#14 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 81 lines · Toke Høiland-Jørgensen
Quoted 66 lines · Daniel Borkmann
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached, the most reliable option is basically to blindly replace the attachment, unless you have the possibility to inspect the attached prog and try to figure out if it belongs to another tool. This is fragile in and of itself, and only possible on more recent kernels iirc.

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing link, and F_FIRST is taken, bail out with an error. The owner of the existing F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear API to allow it to coordinate and make the right decisions.

To circle back to the observability case: in offline discussions with Daniel, I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!). Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST | F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST | F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

This is still perfectly possible to implement on top of Daniel's proposal, and to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Please give this some consideration; we've been trying to figure out a way forward for years at this point. Try not to defer to a daemon too much, it won't actually address any of the pain points with developing k8s tooling.

Thanks,

T

Quoted 29 lines
Quoted 9 lines
Quoted 5 lines

to be policy in the kernel, which would be much better handled in userspace like we do for so many other things. So we should rather expose a hook to allow userspace to set the policy, as we've discussed before; I definitely think we should add that at some point! Although obviously it doesn't have to be part of this series...

Imo, it would be better if we could avoid that.. it feels like we're trying to shoot sparrows with cannon, e.g. when this API gets reused for other attach hooks, then for each of them you need yet another policy program.

Or a single one that understands multiple program types. Sharing the multi-prog implementation is helpful here.

Quoted 5 lines

I don't think that's a good user experience, and I presume this is then single-user program, thus you'll run into the same race in the end - whichever management daemon or application gets to install this policy program first wins. This is potentially just shifting the same issue one level higher, imo.

Sure, we're shifting the problem one level higher, i.e., out of the kernel. That's the point: this is better solved in userspace, so different environments can solve it according to their needs :)

I'm not against having one policy agent on the system, I just don't think the kernel should hard-code one particular solution to the policy problem. Much better to merge this without it, and then iterate on different options (and happy to help with this!), instead of locking the UAPI into a single solution straight away.

-Toke

#15 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 109 lines · Timo Beckers
Quoted 81 lines · Toke Høiland-Jørgensen
Quoted 66 lines · Daniel Borkmann
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

-Toke

author maintainer
#16 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

Quoted 31 lines
Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

Quoted 7 lines

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Fwiw, it's deterministic, and I think this 1000x better than silently having a non-deterministic deployment where the two programs ship with before + 0. That is much harder to debug.

Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

That's okay and I'll do that for v3 to move on.

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

Thanks,
Daniel

#17 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 244 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

Quoted 31 lines
Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

Quoted 7 lines

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Fwiw, it's deterministic, and I think this 1000x better than silently having a non-deterministic deployment where the two programs ship with before + 0. That is much harder to debug.

Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

Quoted 4 lines

That's okay and I'll do that for v3 to move on.

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

With your cilium vs ddos example, maybe all we really need is for the program to have some signal about whether it's ok to have somebody modify/drop the packets before it?
For example, the verifier, depending on whether it sees that the program writes to the data, uses some helpers, or returns TC_ACT_SHOT/etc can classify the program as readonly or non-readonly. And then, we'll have some extra flag during program load/attach that cilium will pass to express "I'm not ok with having a non-readonly program before me".

Seems doable? If it makes sense, we can try to do this as a follow up. It should solve some simple cases without an external arbiter.

#18 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 251 lines · Stanislav Fomichev
Quoted 244 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

Quoted 31 lines
Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

Quoted 7 lines

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Fwiw, it's deterministic, and I think this 1000x better than silently having a non-deterministic deployment where the two programs ship with before + 0. That is much harder to debug.

Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

Quoted 4 lines

That's okay and I'll do that for v3 to move on.

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

Quoted 9 lines

With your cilium vs ddos example, maybe all we really need is for the program to have some signal about whether it's ok to have somebody modify/drop the packets before it?
For example, the verifier, depending on whether it sees that the program writes to the data, uses some helpers, or returns TC_ACT_SHOT/etc can classify the program as readonly or non-readonly. And then, we'll have some extra flag during program load/attach that cilium will pass to express "I'm not ok with having a non-readonly program before me".

So this is what Timo is proposing with F_READONLY. And I agree, that makes sense and we've discussed the need for something like this internally. Specific use case was setsockopt programs. Sometimes they should just observe, and we'd like to enforce that.

Once we have this F_READONLY flag support and enforce that during BPF program validation, then "I'm not ok with having a non-readonly program before me" is exactly F_FIRST. We just say that the F_READONLY program can be inserted anywhere because it has no effect on the state of the system.

Quoted 3 lines

Seems doable? If it makes sense, we can try to do this as a follow up. It should solve some simple cases without an external arbiter.

Yes, and we can add that on top of current F_FIRST/F_LAST. Currently we have to pessimistically assume that every program is non-readonly and F_FIRST/F_LAST applies to just non-readonly programs.

#19 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Hi all,

Quoted 267 lines · Andrii Nakryiko
Quoted 251 lines · Stanislav Fomichev
Quoted 244 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really
useful. My worry is that some prog will just use BPF_F_FIRST which
would prevent the rest of the users.. (starting with only
F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really
need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

Quoted 31 lines
Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

Quoted 7 lines

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Fwiw, it's deterministic, and I think this 1000x better than silently having a non-deterministic deployment where the two programs ship with before + 0. That is much harder to debug.

Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

Quoted 4 lines

That's okay and I'll do that for v3 to move on.

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

Quoted 9 lines

With your cilium vs ddos example, maybe all we really need is for the program to have some signal about whether it's ok to have somebody modify/drop the packets before it?
For example, the verifier, depending on whether it sees that the program writes to the data, uses some helpers, or returns TC_ACT_SHOT/etc can classify the program as readonly or non-readonly. And then, we'll have some extra flag during program load/attach that cilium will pass to express "I'm not ok with having a non-readonly program before me".

So this is what Timo is proposing with F_READONLY. And I agree, that makes sense and we've discussed the need for something like this internally. Specific use case was setsockopt programs. Sometimes they should just observe, and we'd like to enforce that.

Quoted 5 lines

Once we have this F_READONLY flag support and enforce that during BPF program validation, then "I'm not ok with having a non-readonly program before me" is exactly F_FIRST. We just say that the F_READONLY program can be inserted anywhere because it has no effect on the state of the system.

I have a different use case for something like F_READONLY. Basically I would like to be able to accept precompiled BPF progs from semi-trusted sources and run / attach the prog in a trusted context. Example could be telling the customer:
"give me a prog that you'd like to run against every packet that enters your network and I will orchestrate / distribute it across your infrastructure". F_READONLY could be used as one of the mechanisms to uphold invariants like not being able to bring down the network.

Thanks,
Daniel

#20 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 209 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

Quoted 31 lines
Quoted 24 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

Quoted 7 lines

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Quoted 4 lines

Fwiw, it's deterministic, and I think this 1000x better than silently having a non-deterministic deployment where the two programs ship with before + 0. That is much harder to debug.

Totally agree. Silent overriding of the BPF program while user-space is still running (and completely unaware) is MUCH WORSE than not being able to start if your spot is taken.

The latter is explicit and gives a lot of signal early on. The former is confusing and results in hours of painful guessing on what's going on.

How is this even controversial?

Quoted 38 lines
Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

That's okay and I'll do that for v3 to move on.

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

Thanks,
Daniel

#21 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 160 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 136 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that

Quoted 8 lines

there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

See my reply to Andrii: I'm not actually against having an API where an application can say "please always run me first", I'm against the kernel making a hard (UAPI) promise to honour that request.

Quoted 31 lines
Quoted 21 lines
Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

Quoted 9 lines

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

Quoted 1 line

That's okay and I'll do that for v3 to move on.

Sounds good.

Quoted 2 lines

I feel we might repeat the same discussion with no good solution for K8s users once we come back to this point again.

FWIW I do understand that we need to solve the problem for k8s as well, and I'll try to get some people from RH who are working more with the k8s side of things to look at this as well...

-Toke

#22 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 160 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 128 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Quoted 9 lines

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.
My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that

Quoted 8 lines

there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

As someone deeply familiar with the K8s side of the equation you’re greatly oversimplifying.

You can’t just “run a daemon-set” for eBPF-enabled software and expect it to work. <Insert Boromir stating “One does not simply walk into Mordor”>

First off, you need to find out which privileges it needs.

Just CAP_BPF? Pttf nope.
Depending on the program type likely it will need more, up to and including CAP_SYS_ADMIN. Scary stuff.

Beyond that, you’ll also need some “special” paths from the host mounted into your container for vmlinux, tracefs maybe even a bpffs etc…

Furthermore, all of these things above are usually restricted/discouraged in most K8s distros so you need to wade into the depths of how to disable these protections.

The poor platform engineer in this case will be forced to learn all of these concepts on-the-fly. So the assumption of them being oblivious to eBPF being run in their cluster should be dismissed.

Clearly explaining the following in documentation would make coming up with policies much easier:
1. Which priority you choose if not instructed otherwise via configuration
2. The risks of attaching other programs ahead/behind this one
3. The risks of having a conflicting priority with another application

Even from the bpf-enabled software vendor standpoint, the status-quo is annoying because you’ll need to provide recipes to deploy your software on every different K8s distro.

I’ve been working on bpfd [1] + it’s kube integration for the past year to solve these problems for users/vendors.

From a kernel standpoint, give me an array that does something like this:
- If no priority is provided picks the first free from upper 16 bits of the priority range
- If priority is provided, attach at that priority
- If conflict, use flags to decide what to do where the options are something like:
- BPF_F_ERR_ON_CONFLICT
- BPF_F_ASSIGN_ON_CONFLICT

That solves the immediate problem since given a block of u32 priorities I’m sure affected vendors can pick one within the lower 16 bits that would produce the desired ordering.

As for how this works with a system daemon (and by extension in K8s), I’m of the opinion that the only viable option is to move program load and attachment to some other API, be it varlink, gRPC, or the K8s API.

It’s at that layer that policy decisions about priority are made and the kernel semantics can remain as above.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change

Quoted 8 lines
Quoted 8 lines

dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.
So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

I’m sorry but again I have to strongly disagree here.

Tools can provide hints at where they should be placed in a chain of programs, but that eventual placement should always be down to the user.

The examples you’ve cited are large, specialised applications… but consider for a moment how this works for smaller programs.

Let’s say you’ve got 3 programs:
- Firewall
- Load-Balancer
- Packet Logger

There are 6 ways that I can order these programs, each of which will have a very different effect. How can any of these tools individually understand what the user actually wants?

- Dave

[1]: https://github.com/bpfd-dev/bpfd

author maintainer
#23 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 177 lines · Dave Tucker
Quoted 157 lines · Daniel Borkmann
Quoted 3 lines · Toke Høiland-Jørgensen
Quoted 2 lines · Timo Beckers
Quoted 1 line · Toke Høiland-Jørgensen

Daniel Borkmann <[email protected]> writes:

[...]

Quoted 128 lines
Quoted 106 lines
Quoted 79 lines
Quoted 66 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Quoted 9 lines

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.
My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

I would come with the point of view of the K8s cluster operator or platform engineer, if you will. Someone deeply familiar with K8s, but not necessarily knowing about kernel internals. I know my org needs to run container A and container B, so I'll deploy the daemon-sets for both and they get deployed into my cluster. That platform engineer might have never heard of BPF or might not even know that container A or container B ships software with BPF. As mentioned, K8s itself has no concept of Pod ordering as its paradigm is that everything is loosely coupled. We are now expecting from that person to make a concrete decision about some BPF kernel internals on various hooks in which order they should be executed given if they don't then the system becomes non-deterministic. I think that is quite a big burden and ask to understand. Eventually that person will say that he/she cannot make this technical decision and that only one of the two containers can be deployed. I agree with you that

Quoted 8 lines

there should be an option for a technically versed person to be able to change ordering to avoid lock out, but I don't think it will fly asking users to come up on their own with policies of BPF software in the wild ... similar as you probably don't want having to deal with writing systemd unit files for software xyz before you can use your laptop. It's a burden. You expect this to magically work by default and only if needed for good reasons to make custom changes. Just the one difference is that the latter ships with the OS (a priori known / tight-ship analogy).

As someone deeply familiar with the K8s side of the equation you’re greatly oversimplifying.

You can’t just “run a daemon-set” for eBPF-enabled software and expect it to work. <Insert Boromir stating “One does not simply walk into Mordor”>

First off, you need to find out which privileges it needs.

Just CAP_BPF? Pttf nope.
Depending on the program type likely it will need more, up to and including CAP_SYS_ADMIN. Scary stuff.

Quoted 8 lines

Beyond that, you’ll also need some “special” paths from the host mounted into your container for vmlinux, tracefs maybe even a bpffs etc…

Furthermore, all of these things above are usually restricted/discouraged in most K8s distros so you need to wade into the depths of how to disable these protections.

The poor platform engineer in this case will be forced to learn all of these concepts on-the-fly. So the assumption of them being oblivious to eBPF being run in their cluster should be dismissed.

Sure, a lot of the above is irrelevant to the discussion, though. Yes, things need to align and in many cases this is done by the existing projects shipping via Helm charts or other means to make installation easier. Unless you build something from scratch, ofc.

Quoted 17 lines

Clearly explaining the following in documentation would make coming up with policies much easier:
1. Which priority you choose if not instructed otherwise via configuration
2. The risks of attaching other programs ahead/behind this one
3. The risks of having a conflicting priority with another application >
Even from the bpf-enabled software vendor standpoint, the status-quo is annoying because you’ll need to provide recipes to deploy your software on every different K8s distro.

I’ve been working on bpfd [1] + it’s kube integration for the past year to solve these problems for users/vendors.

From a kernel standpoint, give me an array that does something like this:
- If no priority is provided picks the first free from upper 16 bits of the priority range
- If priority is provided, attach at that priority
- If conflict, use flags to decide what to do where the options are something like:
- BPF_F_ERR_ON_CONFLICT
- BPF_F_ASSIGN_ON_CONFLICT

Quoted 2 lines

That solves the immediate problem since given a block of u32 priorities I’m sure affected vendors can pick one within the lower 16 bits that would produce the desired ordering.

See the discussion that we're moving away from priorities. I used them in v1, but the community feedback was that priorities are discouraged.

Quoted 3 lines

As for how this works with a system daemon (and by extension in K8s), I’m of the opinion that the only viable option is to move program load and attachment to some other API, be it varlink, gRPC, or the K8s API.

Fwiw, I doubt that this will fly to be honest. You would have to rewrite libbpf, and all loaders out in the wild to standardize and implement this rpc protocol, and you also still need to support the non-rpc way of loading your BPF programs for the case where you need to deal with old/existing environments where this is not the case. If bpfd ships as a 3rd party software, then fingers crossed that eventually all k8s distros would actually adopt it. If it's something that comes natively with K8s it would probably have a better chance, but still you'd force every project to support loading via both ways. It's not as straight forward, and for it to be the only viable path you need a clean way for projects to transition to this w/o breaking.

Quoted 46 lines

It’s at that layer that policy decisions about priority are made and the kernel semantics can remain as above.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change

Quoted 8 lines
Quoted 8 lines

dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.
So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

I’m sorry but again I have to strongly disagree here.

Tools can provide hints at where they should be placed in a chain of programs, but that eventual placement should always be down to the user.

The examples you’ve cited are large, specialised applications… but consider for a moment how this works for smaller programs.

Let’s say you’ve got 3 programs:
- Firewall
- Load-Balancer
- Packet Logger

There are 6 ways that I can order these programs, each of which will have a very different effect. How can any of these tools individually understand what the user actually wants?

- Dave

[1]: https://github.com/bpfd-dev/bpfd

#24 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Toke Høiland-Jørgensen
Quoted 120 lines · Toke Høiland-Jørgensen
Quoted 109 lines · Timo Beckers
Quoted 81 lines · Toke Høiland-Jørgensen
Quoted 66 lines · Daniel Borkmann
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

I'm still not sure whether the hard semantics of first/last is really useful. My worry is that some prog will just use BPF_F_FIRST which would prevent the rest of the users.. (starting with only F_BEFORE/F_AFTER feels 'safer'; we can iterate later on if we really need first/laste).

Without FIRST/LAST some scenarios cannot be guaranteed to be safely implemented. E.g., if I have some hard audit requirements and I need to guarantee that my program runs first and observes each event, I'll enforce BPF_F_FIRST when attaching it. And if that attachment fails, then server setup is broken and my application cannot function.

In a setup where we expect multiple applications to co-exist, it should be a rule that no one is using FIRST/LAST (unless it's absolutely required). And if someone doesn't comply, then that's a bug and has to be reported to application owners.

Quoted 6 lines
Quoted 3 lines

But it's not up to the kernel to enforce this cooperation by disallowing FIRST/LAST semantics, because that semantics is critical for some applications, IMO.

Maybe that's something that should be done by some other mechanism? (and as a follow up, if needed) Something akin to what Toke mentioned with another program doing sorting or similar.

The goal of this API is to avoid needing some extra special program to do this sorting

Quoted 2 lines

Otherwise, those first/last are just plain simple old priority bands; only we have two now, not u16.

I think it's different. FIRST/LAST has to be used judiciously, of course, but when they are needed, they will have no alternative.

Also, specifying FIRST + LAST is the way to say "I want my program to be the only one attached". Should we encourage such use cases? No, of course. But I think it's fair for users to be able to express this.

Quoted 4 lines

I'm mostly coming from the observability point: imagine I have my fancy tc_ingress_tcpdump program that I want to attach as a first program to debug some issue, but it won't work because there is already a 'first' program installed.. Or the assumption that I'd do F_REPLACE | F_FIRST ?

If your production setup requires that some important program has to be FIRST, then yeah, your "let me debug something" program shouldn't interfere with it (assuming that FIRST requirement is a real requirement and not someone just thinking they need to be first; but that's up to user space to decide). Maybe the solution for you in that case would be freplace program installed on top of that stubborn FIRST program? And if we are talking about local debugging and development, then you are a sysadmin and you should be able to force-detach that program that is getting in the way.

Quoted 8 lines
Quoted 8 lines

I'm not really concerned about our production environment. It's pretty controlled and restricted and I'm pretty certain we can avoid doing something stupid. Probably the same for your env.

I'm mostly fantasizing about upstream world where different users don't know about each other and start doing stupid things like F_FIRST where they don't really have to be first. It's that "used judiciously" part that I'm a bit skeptical about :-D

But in the end how is that different from just attaching themselves blindly into the first position (e.g. with before and relative_fd as 0 or the fd/id of the current first program) - same, they don't really have to be first. How would that not result in doing something stupid? ;) To add to Andrii's earlier DDoS mitigation example ... think of K8s environment: one project is implementing DDoS mitigation with BPF, another one wants to monitor/ sample traffic to user space with BPF. Both install as first position by default (before + 0). In K8s, there is no built-in Pod dependency management so you cannot guarantee whether Pod A comes up before Pod B. So you'll end up in a situation where sometimes the monitor runs before the DDoS mitigation and on some other nodes it's vice versa. The other case where this gets

Quoted 8 lines

broken (assuming a node where we get first the DDoS mitigation, then the monitoring) is when you need to upgrade one of the Pods: monitoring Pod gets a new stable update and is being re-rolled out, then it inserts itself before the DDoS mitigation mechanism, potentially causing outage. With the first/last mechanism these two situations cannot happen. The DDoS mitigation software uses first and the monitoring uses before + 0, then no matter the re-rollouts or the ordering in which Pods come up, it's always at the expected/correct location.

I'm not disputing that these kinds of policy issues need to be solved somehow. But adding the first/last pinning as part of the kernel hooks doesn't solve the policy problem, it just hard-codes a solution for one particular instance of the problem.

Quoted 13 lines

Taking your example from above, what happens when someone wants to deploy those tools in reverse order? Say the monitoring tool counts packets and someone wants to also count the DDOS traffic; but the DDOS protection tool has decided for itself (by setting the FIRST) flag that it can *only* run as the first program, so there is no way to achieve this without modifying the application itself.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Because even with this new ordering scheme, there still should be some entity to do relative ordering (systemd-style, maybe CNI?). And if it does the ordering, I don't really see why we need F_FIRST/F_LAST.

I can see I'm a bit late to the party, but FWIW I agree with this:
FIRST/LAST will definitely be abused if we add it. It also seems to me

It's in the prisoners' best interest to collaborate (and they do! see
https://www.youtube.com/watch?v=YK7GyEJdJGo), except the current prio system is limiting and turns out to be really fragile in practice.

If your tool wants to attach to tc prio 1 and there's already a prog attached,
the most reliable option is basically to blindly replace the attachment, unless
you have the possibility to inspect the attached prog and try to figure out if it
belongs to another tool. This is fragile in and of itself, and only possible on
more recent kernels iirc.

Quoted 7 lines

With tcx, Cilium could make an initial attachment using F_FIRST and simply update a link at well-known path on subsequent startups. If there's no existing
link, and F_FIRST is taken, bail out with an error. The owner of the existing
F_FIRST program can be queried and logged; we know for sure the program doesn't belong to Cilium, and we have no interest in detaching it.

Quoted 2 lines

That's conflating the benefit of F_FIRST with that of bpf_link, though; you can have the replace thing without the exclusive locking.

I think Timo says that he wants to install his bpf_link as the very first decision-making BPF program (with F_FIRST) and make sure that that spot stays the very first decision-making BPF program. And then he can just do LINK_UPDATE to upgrade the underlying program.

I don't see anything being conflated here.

Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Quoted 4 lines

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

What if some combinations of programs just cannot co-exist?

Me, Daniel, Timo are arguing that there are real situations where you have to be first or need to die. And the counter argument we are getting is "but someone can accidentally or in bad faith overuse F_FIRST". The former is causing real problems and silent failures. The latter is about fixing bugs and/or fighting bad actors. We don't propose any real solution for the real first problem, because we are afraid of hypothetical bad actors. The former has a technical solution (F_FIRST/F_LAST), the latter is a matter of bug fixing and pushing back on bad actors. This is where distros can actually help by making sure that bad actors that don't really need F_FIRST/F_LAST are not using them.

It's disturbing that we use the hypothetical "but users can be bad" argument to prevent a solution to a technical problem that already happened and will keep happening because there is no solution available.

And the mythical user-space daemon that will solve all these problems is not a convincing argument. I haven't seen any concrete proposals beyond hand-wavy arguments.

Quoted 6 lines

My assumption is basically this: A system administrator installs packages A and B that both use the TC hook. The developers of A and B have never heard about each other. It should be possible for that admin to run A and B in whichever order they like, without making any changes to A and B themselves.

That's impossible if A and B just cannot co-exist. E.g., if both A and B are setting some socket options that are fundamentally in conflict. You will have to either choose A or B, but not both. Or make sure A and B somehow don't step on each other's toe. But it's not an ordering problem. It's something that developers of A and B have to coordinate between each other outside of the kernel.

Quoted 72 lines
Quoted 25 lines

Regardless of merit or feasability of a system-wide bpf management daemon for k8s, there _is no ordering configuration possible_. K8s is not a distro where package maintainers (or anyone else, really) can coordinate on correctly defining priority of each of the tools they ship. This is effectively
the prisoner's dilemma. I feel like most of the discussion so far has been very hand-wavy in 'user space should solve it'. Well, we are user space, and we're here trying to solve it. :)

A hypothetical policy/gatekeeper/ordering daemon doesn't possess implicit knowledge about which program needs to go where in the chain, nor is there an obvious heuristic about how to order things. Maintaining such a configuration for all cloud-native tooling out there that possibly uses bpf is simply impossible, as even a tool like Cilium can change dramatically from one release to the next. Having to manage this too would put a significant burden on velocity and flexibility for arguably little benefit to the user.

So, daemon/kernel will need to be told how to order things, preferably by the tools (Cilium/datadog-agent) themselves, since the user/admin of the system cannot be expected to know where to position the hundreds of progs loaded by Cilium and how they might interfere with other tools. Figuring this out is the job of the tool, daemon or not.

Quoted 6 lines

The prisoners _must_ communicate (so, not abuse F_FIRST) for things to work correctly, and it's 100% in their best interest in doing so. Let's not pretend like we're able to solve game theory on this mailing list. :) We'll have to settle for the next-best thing: give user space a safe and clear
API to allow it to coordinate and make the right decisions.

But "always first" is not a meaningful concept. It's just what we have today (everyone picks priority 1), except now if there are two programs that want the same hook, it will be the first program that wins the contest (by locking the second one out), instead of the second program winning (by overriding the first one) as is the case with the silent override semantics we have with TC today. So we haven't solved the problem, we've just shifted the breakage.

Quoted 16 lines

To circle back to the observability case: in offline discussions with Daniel,
I've mentioned the need for 'shadow' progs that only collect data and pump it to user space, attached at specific points in the chain (still within tcx!).
Their retcodes would be ignored, and context modifications would be rejected, so attaching multiple to the same hook can always succeed, much like cgroup multi. Consider the following:

To attach a shadow prog before F_FIRST, a caller could use F_BEFORE | F_FIRST |
F_RDONLY. Attaching between first and the 'relative' section: F_AFTER | F_FIRST |
F_RDONLY, etc. The rdonly flag could even be made redundant if a new prog/ attach type is added for progs like these.

Quoted 4 lines

This is still perfectly possible to implement on top of Daniel's proposal, and
to me looks like it could address many of the concerns around ordering of progs I've seen in this thread, many mention data exfiltration.

It may well be that semantics like this will turn out to be enough. Or it may not (I personally believe we'll need something more expressive still, and where the system admin has the option to override things; but I may turn out to be wrong). Ultimately, my main point wrt this series is that this kind of policy decision can be added later, and it's better to merge the TCX infrastructure without it, instead of locking ourselves into an API that is way too limited today. TCX (and in-kernel XDP multiprog) has value without it, so let's merge that first and iterate on the policy aspects.

-Toke

#25 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 3 lines · Andrii Nakryiko

Me, Daniel, Timo are arguing that there are real situations where you have to be first or need to die.

afaik out of all xdp and tc progs there is not a single prog in the fb fleet that has to be first.
fb's ddos and firewall don't have to be first.
cilium and datadog progs don't have to be first either.
The race between cilium and datadog was not the race to the first position, but the conflict due to the same prio.
In all cases, I'm aware, prog owners care a lot about ordering, but never about strict first.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#27 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

Quoted 21 lines · Prankur gupta
Quoted 5 lines
Quoted 4 lines

Me, Daniel, Timo are arguing that there are real situations where you have to be first or need to die.

Quoted 8 lines

afaik out of all xdp and tc progs there is not a single prog in the fb fleet that has to be first.
fb's ddos and firewall don't have to be first.
cilium and datadog progs don't have to be first either.
The race between cilium and datadog was not the race to the first position, but the conflict due to the same prio.
In all cases, I'm aware, prog owners care a lot about ordering, but never about strict first.

One usecase which we actively use in Meta(fb) fleet is avoiding double writer for cgroup/sockops bpf programs. For ex: we can have multiple BPF programs setting skops->reply field resulting in stepping on each other for ex: for ECN callback one program can set it 1 and other can set it to 0.
We do that by creating a pre-func and post-func before
executing sockops BPF program in our custom built chainer.

Quoted 7 lines

We want these functions to be executed first and last respectively which actually makes the above functionality useful for us.

Hypothetical usecase for cgroup/sockops - Middle BPF programs will not set skops->reply and the final BPF program based on results from each of the middle
BPF program can set the appropriate reply to skops->reply, thus making sure all the middle programs executed and the final reply is correct.

cgroup progs are more complicated than a simple list of progs in tc/xdp. It is not really possible for the kernel to guarantee absolute last and first in a hierarchy of cgroups. In theory that's possible within a cgroup, but not when children and parents are involved and progs can be attached anywhere in the hierarchy and we need to keep
uapi of BPF_F_ALLOW_OVERRIDE, BPF_F_ALLOW_MULTI intact.
The absolute first/last is not the answer for this skops issue.
A different solution is necessary.

#28 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Andrii Nakryiko
Quoted 19 lines
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

See above on the issues w/o the first/last. How would you work around them in practice so they cannot happen?

By having an ordering configuration that is deterministic. Enforced by the system-wide management daemon by whichever mechanism suits it. We could implement a minimal reference policy agent that just reads a config file in /etc somewhere, and *that* could implement FIRST/LAST semantics.

Quoted 6 lines

I think this particular perspective is what's deadlocking this discussion. To me, it looks like distros and hyperscalers are in the same boat with regards to the possibility of coordination between tools. Distros are only responsible for the tools they package themselves, and hyperscalers run a tight ship with mostly in-house tooling already. When it comes to projects out in the wild, that all goes out the window.

Quoted 4 lines

Not really: from the distro PoV we absolutely care about arbitrary combinations of programs with different authors. Which is why I'm arguing against putting anything into the kernel where the first program to come along can just grab a hook and lock everyone out.

Quoted 5 lines

What if some combinations of programs just cannot co-exist?

Me, Daniel, Timo are arguing that there are real situations where you have to be first or need to die.

Right, and what I'm saying is that this decision should not be up to individual applications to decide for the whole system. I'm OK with an application *requesting* that, but it should be possible for a system-level policy to override that request. I don't actually care so much about the mechanism for doing so; I just don't want to expose a flag in UAPI that comes with such a "lock everything" promise, because that explicitly prevents such system overrides.

Quoted 9 lines

And the counter argument we are getting is "but someone can accidentally or in bad faith overuse F_FIRST". The former is causing real problems and silent failures. The latter is about fixing bugs and/or fighting bad actors. We don't propose any real solution for the real first problem, because we are afraid of hypothetical bad actors. The former has a technical solution (F_FIRST/F_LAST), the latter is a matter of bug fixing and pushing back on bad actors. This is where distros can actually help by making sure that bad actors that don't really need F_FIRST/F_LAST are not using them.

It's not about "bad actors" in the malicious sense, it's about decisions being made in one context not being valid in another. Take Daniel's example of a DDOS application. It's probably a quite legitimate choice for the developers of such an application to say "it only makes sense to run a DDOS application first, so of course we'll set the FIRST flag". But it is just as legitimate for a user/admin to say "I actually want to run this DDOS application after this other application I wrote for that specific purpose". If we enforce the FIRST flag semantics at the kernel level we're making a decision that the first case is legitimate and the second isn't, and that's just not true.

The whole datadog/cilium issue shows exactly that this kind of conflict *is* how things play out in practice.

-Toke

#29 lore raw

Re: [PATCH bpf-next v2 1/7] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 83 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for the case of attaching
- Enforced only at attach time
- BPF_F_{FIRST,LAST}
- Enforced throughout the bpf_mprog state's lifetime
- Admin override possible (e.g. link detach, prog-based BPF_F_REPLACE)
- Internal revision counter and optionally being able to pass expected_revision
- User space daemon can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure) and bpf_mprog_cp (control-path structure). Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficieny. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF. The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggy- backs on rtnl. The nitty-gritty details are in the bpf_mprog_{replace,head_tail, add,del} implementation and an extensive test suite for checking all aspects of this API for prog-based attach/detach and link API as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management daemon.

[0] https://lore.kernel.org/bpf/[email protected]/
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

Quoted 11 lines
---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 245 +++++++++++++++++
 include/uapi/linux/bpf.h       |  37 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 476 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  37 ++-
 6 files changed, 781 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c

I like the API itself, I think it strikes the right balance. My questions and comments below are mostly about specific implementation details only.

Quoted 37 lines
MAINTAINERS +1 -0
index c904dba1733b..754a9eeca0a1 100644
@@ -3733,6 +3733,7 @@ F:        include/linux/filter.h
 F:     include/linux/tnum.h
 F:     kernel/bpf/core.c
 F:     kernel/bpf/dispatcher.c
+F:     kernel/bpf/mprog.c
 F:     kernel/bpf/syscall.c
 F:     kernel/bpf/tnum.c
 F:     kernel/bpf/trampoline.c
include/linux/bpf_mprog.h new file +19 -0
new file mode 100644
index 000000000000..7399181d8e6c
@@ -0,0 +1,245 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef __BPF_MPROG_H
+#define __BPF_MPROG_H
+
+#include <linux/bpf.h>
+
+#define BPF_MPROG_MAX  64
+#define BPF_MPROG_SWAP 1
+#define BPF_MPROG_FREE 2
+
+struct bpf_mprog_fp {
+       struct bpf_prog *prog;
+};
+
+struct bpf_mprog_cp {
+       struct bpf_link *link;
+       u32 flags;
+};

include/linux/bpf_mprog.h line 29

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+
+struct bpf_mprog_entry {
+       struct bpf_mprog_fp fp_items[BPF_MPROG_MAX] ____cacheline_aligned;
+       struct bpf_mprog_cp cp_items[BPF_MPROG_MAX] ____cacheline_aligned;
+       struct bpf_mprog_bundle *parent;
+};
+
+struct bpf_mprog_bundle {
+       struct bpf_mprog_entry a;
+       struct bpf_mprog_entry b;

I get why we want a and b for bpf_mprog_fd, as we don't want to modify "effective" array while it might be read by bpf_prog_run(). But can't we just modify bpf_mprog_cp items in place instead of having an entire BPF_MPROG_MAX of copies?

Quoted 130 lines
include/linux/bpf_mprog.h new file +130 -0
+       struct rcu_head rcu;
+       struct bpf_prog *ref;
+       atomic_t revision;
+};
+
+struct bpf_tuple {
+       struct bpf_prog *prog;
+       struct bpf_link *link;
+};
+
+static inline struct bpf_mprog_entry *
+bpf_mprog_peer(const struct bpf_mprog_entry *entry)
+{
+       if (entry == &entry->parent->a)
+               return &entry->parent->b;
+       else
+               return &entry->parent->a;
+}
+
+#define bpf_mprog_foreach_tuple(entry, fp, cp, t)                      \
+       for (fp = &entry->fp_items[0], cp = &entry->cp_items[0];        \
+            ({                                                         \
+               t.prog = READ_ONCE(fp->prog);                           \
+               t.link = cp->link;                                      \
+               t.prog;                                                 \
+             });                                                       \
+            fp++, cp++)
+
+#define bpf_mprog_foreach_prog(entry, fp, p)                           \
+       for (fp = &entry->fp_items[0];                                  \
+            (p = READ_ONCE(fp->prog));                                 \
+            fp++)
+
+static inline struct bpf_mprog_entry *bpf_mprog_create(size_t extra_size)
+{
+       struct bpf_mprog_bundle *bundle;
+
+       /* Fast-path items are not extensible, must only contain prog pointer! */
+       BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+       /* Control-path items can be extended w/o affecting fast-path. */
+       BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) != ARRAY_SIZE(bundle->a.cp_items));
+
+       bundle = kzalloc(sizeof(*bundle) + extra_size, GFP_KERNEL);
+       if (bundle) {
+               atomic_set(&bundle->revision, 1);
+               bundle->a.parent = bundle;
+               bundle->b.parent = bundle;
+               return &bundle->a;
+       }
+       return NULL;
+}
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+       kfree_rcu(entry->parent, rcu);
+}
+
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                                     struct bpf_prog *prog)
+{
+       WARN_ON_ONCE(entry->parent->ref);
+       entry->parent->ref = prog;
+}
+
+static inline u32 bpf_mprog_flags(u32 cur_flags, u32 req_flags, u32 flag)
+{
+       if (req_flags & flag)
+               cur_flags |= flag;
+       else
+               cur_flags &= ~flag;
+       return cur_flags;
+}
+
+static inline u32 bpf_mprog_max(void)
+{
+       return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline struct bpf_prog *bpf_mprog_first(struct bpf_mprog_entry *entry)
+{
+       return READ_ONCE(entry->fp_items[0].prog);
+}
+
+static inline struct bpf_prog *bpf_mprog_last(struct bpf_mprog_entry *entry)
+{
+       struct bpf_prog *tmp, *prog = NULL;
+       struct bpf_mprog_fp *fp;
+
+       bpf_mprog_foreach_prog(entry, fp, tmp)
+               prog = tmp;
+       return prog;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                                   struct bpf_prog *prog)
+{
+       const struct bpf_mprog_fp *fp;
+       const struct bpf_prog *tmp;
+
+       bpf_mprog_foreach_prog(entry, fp, tmp) {
+               if (tmp == prog)
+                       return true;
+       }
+       return false;
+}
+
+static inline struct bpf_prog *bpf_mprog_first_reg(struct bpf_mprog_entry *entry)
+{
+       struct bpf_tuple tuple = {};
+       struct bpf_mprog_fp *fp;
+       struct bpf_mprog_cp *cp;
+
+       bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+               if (cp->flags & BPF_F_FIRST)
+                       continue;
+               return tuple.prog;
+       }
+       return NULL;
+}
+
+static inline struct bpf_prog *bpf_mprog_last_reg(struct bpf_mprog_entry *entry)
+{
+       struct bpf_tuple tuple = {};
+       struct bpf_prog *prog = NULL;
+       struct bpf_mprog_fp *fp;
+       struct bpf_mprog_cp *cp;
+
+       bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+               if (cp->flags & BPF_F_LAST)
+                       break;

include/linux/bpf_mprog.h line 169

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+               prog = tuple.prog;
+       }
+       return prog;
+}
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+       do {
+               atomic_inc(&entry->parent->revision);
+       } while (atomic_read(&entry->parent->revision) == 0);

why not just use atomic64_t and never care about zero wrap-around?

Quoted 36 lines
include/linux/bpf_mprog.h new file +36 -0
+       synchronize_rcu();
+       if (entry->parent->ref) {
+               bpf_prog_put(entry->parent->ref);
+               entry->parent->ref = NULL;
+       }
+}
+
+static inline void bpf_mprog_entry_clear(struct bpf_mprog_entry *entry)
+{
+       memset(entry->fp_items, 0, sizeof(entry->fp_items));
+       memset(entry->cp_items, 0, sizeof(entry->cp_items));
+}
+
+static inline u64 bpf_mprog_revision(struct bpf_mprog_entry *entry)
+{
+       return atomic_read(&entry->parent->revision);
+}
+
+static inline void bpf_mprog_read(struct bpf_mprog_entry *entry, u32 which,
+                                 struct bpf_mprog_fp **fp_dst,
+                                 struct bpf_mprog_cp **cp_dst)
+{
+       *fp_dst = &entry->fp_items[which];
+       *cp_dst = &entry->cp_items[which];
+}
+
+static inline void bpf_mprog_write(struct bpf_mprog_fp *fp_dst,
+                                  struct bpf_mprog_cp *cp_dst,
+                                  struct bpf_tuple *tuple, u32 flags)
+{
+       WRITE_ONCE(fp_dst->prog, tuple->prog);
+       cp_dst->link  = tuple->link;
+       cp_dst->flags = flags;
+}
+
+static inline void bpf_mprog_copy(struct bpf_mprog_fp *fp_dst,

include/linux/bpf_mprog.h line 215

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+                                 struct bpf_mprog_cp *cp_dst,
+                                 struct bpf_mprog_fp *fp_src,
+                                 struct bpf_mprog_cp *cp_src)
+{
+       WRITE_ONCE(fp_dst->prog, READ_ONCE(fp_src->prog));
+       memcpy(cp_dst, cp_src, sizeof(*cp_src));
+}
+
+static inline void bpf_mprog_copy_range(struct bpf_mprog_entry *peer,
+                                       struct bpf_mprog_entry *entry,

it's not clear what is source and what is destination, why not just use "src" and "dst" naming?

Quoted 90 lines
include/linux/bpf_mprog.h new file +30 -0
+                                       u32 idx_peer, u32 idx_entry, u32 num)
+{
+       memcpy(&peer->fp_items[idx_peer], &entry->fp_items[idx_entry],
+              num * sizeof(peer->fp_items[0]));
+       memcpy(&peer->cp_items[idx_peer], &entry->cp_items[idx_entry],
+              num * sizeof(peer->cp_items[0]));
+}
+
+static inline u32 bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+       const struct bpf_mprog_fp *fp;
+       const struct bpf_prog *tmp;
+       u32 num = 0;
+
+       bpf_mprog_foreach_prog(entry, fp, tmp)
+               num++;
+       return num;
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+                    struct bpf_link *link, u32 flags, u32 object,
+                    u32 expected_revision);
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+                    struct bpf_link *link, u32 flags, u32 object,
+                    u32 expected_revision);
+
+int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
+                   struct bpf_mprog_entry *entry);
+
+#endif /* __BPF_MPROG_H */
include/uapi/linux/bpf.h +23 -8
index a7b5e91dd768..207f8a37b327 100644
@@ -1102,7 +1102,14 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE   (1U << 0)
 #define BPF_F_ALLOW_MULTI      (1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE          (1U << 2)
+#define BPF_F_BEFORE           (1U << 3)
+#define BPF_F_AFTER            (1U << 4)
+#define BPF_F_FIRST            (1U << 5)
+#define BPF_F_LAST             (1U << 6)
+#define BPF_F_ID               (1U << 7)
+#define BPF_F_LINK             BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1433,14 +1440,19 @@ union bpf_attr {
        };
 
        struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-               __u32           target_fd;      /* container object to attach to */
-               __u32           attach_bpf_fd;  /* eBPF program to attach */
+               union {
+                       __u32   target_fd;      /* target object to attach to or ... */
+                       __u32   target_ifindex; /* target ifindex */
+               };
+               __u32           attach_bpf_fd;
                __u32           attach_type;
                __u32           attach_flags;
-               __u32           replace_bpf_fd; /* previously attached eBPF
-                                                * program to replace if
-                                                * BPF_F_REPLACE is used
-                                                */
+               union {
+                       __u32   relative_fd;
+                       __u32   relative_id;
+                       __u32   replace_bpf_fd;
+               };
+               __u32           expected_revision;
        };
 
        struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1486,16 +1498,25 @@ union bpf_attr {
        } info;
 
        struct { /* anonymous struct used by BPF_PROG_QUERY command */
-               __u32           target_fd;      /* container object to query */
+               union {
+                       __u32   target_fd;      /* target object to query or ... */
+                       __u32   target_ifindex; /* target ifindex */
+               };
                __u32           attach_type;
                __u32           query_flags;
                __u32           attach_flags;
                __aligned_u64   prog_ids;
-               __u32           prog_cnt;
+               union {

include/uapi/linux/bpf.h union bpf_attr { · line 1519

Quoted 10 lines
include/uapi/linux/bpf.h +6 -0
+                       __u32   prog_cnt;
+                       __u32   count;
+               };
+               __u32           revision;
                /* output: per-program attach_flags.
                 * not allowed to be set during effective query.
                 */
                __aligned_u64   prog_attach_flags;
+               __aligned_u64   link_ids;
+               __aligned_u64   link_attach_flags;

flags are 32-bit, no?

Quoted 88 lines
include/uapi/linux/bpf.h +0 -0
        } query;
 
        struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
kernel/bpf/Makefile +1 -1
index 1d3892168d32..1bea2eb912cd 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}    += bpf_inode_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += disasm.o
+obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
 obj-$(CONFIG_BPF_JIT) += dispatcher.o
kernel/bpf/mprog.c new file +66 -0
new file mode 100644
index 000000000000..efc3b73f8bf5
@@ -0,0 +1,476 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+#include <linux/filter.h>
+
+static int bpf_mprog_tuple_relative(struct bpf_tuple *tuple,
+                                   u32 object, u32 flags,
+                                   enum bpf_prog_type type)
+{
+       struct bpf_prog *prog;
+       struct bpf_link *link;
+
+       memset(tuple, 0, sizeof(*tuple));
+       if (!(flags & (BPF_F_REPLACE | BPF_F_BEFORE | BPF_F_AFTER)))
+               return object || (flags & (BPF_F_ID | BPF_F_LINK)) ?
+                      -EINVAL : 0;
+       if (flags & BPF_F_LINK) {
+               if (flags & BPF_F_ID)
+                       link = bpf_link_by_id(object);
+               else
+                       link = bpf_link_get_from_fd(object);
+               if (IS_ERR(link))
+                       return PTR_ERR(link);
+               if (type && link->prog->type != type) {
+                       bpf_link_put(link);
+                       return -EINVAL;
+               }
+               tuple->link = link;
+               tuple->prog = link->prog;
+       } else {
+               if (flags & BPF_F_ID)
+                       prog = bpf_prog_by_id(object);
+               else
+                       prog = bpf_prog_get(object);
+               if (IS_ERR(prog)) {
+                       if (!object &&
+                           !(flags & BPF_F_ID))
+                               return 0;
+                       return PTR_ERR(prog);
+               }
+               if (type && prog->type != type) {
+                       bpf_prog_put(prog);
+                       return -EINVAL;
+               }
+               tuple->link = NULL;
+               tuple->prog = prog;
+       }
+       return 0;
+}
+
+static void bpf_mprog_tuple_put(struct bpf_tuple *tuple)
+{
+       if (tuple->link)
+               bpf_link_put(tuple->link);
+       else if (tuple->prog)
+               bpf_prog_put(tuple->prog);
+}
+
+static int bpf_mprog_replace(struct bpf_mprog_entry *entry,
+                            struct bpf_tuple *ntuple,
+                            struct bpf_tuple *rtuple, u32 rflags)
+{
+       struct bpf_mprog_fp *fp;
+       struct bpf_mprog_cp *cp;

kernel/bpf/mprog.c line 76

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+       struct bpf_prog *oprog;
+       u32 iflags;
+       int i;
+
+       if (rflags & (BPF_F_BEFORE | BPF_F_AFTER | BPF_F_LINK))
+               return -EINVAL;
+       if (rtuple->prog != ntuple->prog &&
+           bpf_mprog_exists(entry, ntuple->prog))
+               return -EEXIST;
+       for (i = 0; i < bpf_mprog_max(); i++) {

why not just keep track of actual count in bpf_mprob_bundle? that will speed up bpf_mprog_last() as well

Quoted 33 lines
kernel/bpf/mprog.c new file +33 -0
+               bpf_mprog_read(entry, i, &fp, &cp);
+               oprog = READ_ONCE(fp->prog);
+               if (!oprog)
+                       break;
+               if (oprog != rtuple->prog)
+                       continue;
+               if (cp->link != ntuple->link)
+                       return -EBUSY;
+               iflags = cp->flags;
+               if ((iflags & BPF_F_FIRST) !=
+                   (rflags & BPF_F_FIRST)) {
+                       iflags = bpf_mprog_flags(iflags, rflags,
+                                                BPF_F_FIRST);
+                       if ((iflags & BPF_F_FIRST) &&
+                           rtuple->prog != bpf_mprog_first(entry))
+                               return -EACCES;
+               }
+               if ((iflags & BPF_F_LAST) !=
+                   (rflags & BPF_F_LAST)) {
+                       iflags = bpf_mprog_flags(iflags, rflags,
+                                                BPF_F_LAST);
+                       if ((iflags & BPF_F_LAST) &&
+                           rtuple->prog != bpf_mprog_last(entry))
+                               return -EACCES;
+               }
+               bpf_mprog_write(fp, cp, ntuple, iflags);
+               if (!ntuple->link)
+                       bpf_prog_put(oprog);
+               return 0;
+       }
+       return -ENOENT;
+}
+

[...]

Quoted 16 lines
kernel/bpf/mprog.c new file +16 -0
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+                    struct bpf_link *link, u32 flags, u32 object,
+                    u32 expected_revision)
+{
+       struct bpf_tuple rtuple, ntuple = {
+               .prog = prog,
+               .link = link,
+       };
+       int ret;
+
+       if (expected_revision &&
+           expected_revision != bpf_mprog_revision(entry))
+               return -ESTALE;
+       ret = bpf_mprog_tuple_relative(&rtuple, object, flags, prog->type);
+       if (ret)

kernel/bpf/mprog.c line 397

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+               return ret;
+       if (flags & BPF_F_REPLACE)
+               ret = bpf_mprog_replace(entry, &ntuple, &rtuple, flags);
+       else if (flags & (BPF_F_FIRST | BPF_F_LAST))
+               ret = bpf_mprog_head_tail(entry, &ntuple, &rtuple, flags);
+       else
+               ret = bpf_mprog_add(entry, &ntuple, &rtuple, flags);
+       bpf_mprog_tuple_put(&rtuple);
+       return ret;
+}

We chatted about this a bit offline, but let me expand a bit on this in writing. I still find the need to have mprog_add, mprog_head_tail and mprog_replace variants a bit unnecessary. Especially that each of them kind of reimplements all these BEFORE/AFTER and FIRST/LAST constraints each time.

I was thinking if it would be more straightforward to evaluate target position for the prog/link that needs to be replaced/inserted/deleted for each "rule" independently. And if all rules agree on that position or existing element, then enact replacement/addition/deletion. Something like below, in very pseudo-code-like manner:

int idx = -1, tidx;

if (flag & BPF_F_REPLACE) {
    tidx = mprog_pos_exact(object, flags);
    if (tidx < 0 || (idx >= 0 && tidx != idx))
        return -EINVAL;
    idx = tidx;
}
if (flag & BPF_F_BEFORE) {
    tidx = mprog_pos_before(object, flags);
    if (tidx < 0 || (idx >= 0 && tidx != idx))
        return -EINVAL;
    idx = tidx;
}
if (flag & BPF_F_AFTER) {
    tidx = mprog_pos_after(object, flags);
    if (tidx < 0 || (idx >= 0 && tidx != idx))
        return -EINVAL;
    idx = tidx;
}
if (flag & BPF_F_FIRST) {
    if (idx >= 0 && idx != 0)
        return -EINVAL;
    idx = 0;
    if (idx < bpf_mprog_cnt() && (prog_flag_at(idx) & BPF_F_FIRST))
        return -EBUSY;
}
if (flag & BPF_F_LAST) {
    if (idx >= 0 && idx != bpf_mprog_cnt())
        return -EINVAL;
    idx = bpf_mprog_cnt();
    if (idx < bpf_mprog_cnt() && (prog_flag_at(idx) & BPF_F_LAST))
        return -EBUSY;
}

if (flag & BPF_F_REPLACE)
replace_in_place(idx, flags & (BPF_F_FIRST | BPF_F_LAST));
else /* if delete */
delete_at_pos(idx);
else /* add new element */
insert_at_pos(idx, flags & (BPF_F_FIRST | BPF_F_LAST));

Each of those mprog_pos_{exact,before,after} should be trivial to implement (they will just find position that satisfies one of those conditions). idx will mean either position at which we need to insert new element (and so everything at that position and to the right should be shifted), or if it's a replacement/deletion where we expect to find existing prog/link, we'll have to find matching element there.

I guess REPLACE and BEFORE/AFTER are currently fundamentally incompatible because they reuse the same field to specify FD/ID, so we'd have to check that both are not specified at the same time. Or we can choose to have separate replace_fd and relative_fd/id. And then you could even express "replace prog if it's before another prog X", similarly how you can express REPLACE + FIRST (replace if it's first).

You mentioned actual implementation gets hard, so I'm curious which parts with such approach are becoming convoluted in actual implementation?

Quoted 24 lines
kernel/bpf/mprog.c new file +24 -0
+
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+                    struct bpf_link *link, u32 flags, u32 object,
+                    u32 expected_revision)
+{
+       struct bpf_tuple rtuple, dtuple = {
+               .prog = prog,
+               .link = link,
+       };
+       int ret;
+
+       if (expected_revision &&
+           expected_revision != bpf_mprog_revision(entry))
+               return -ESTALE;
+       ret = bpf_mprog_tuple_relative(&rtuple, object, flags,
+                                      prog ? prog->type :
+                                      BPF_PROG_TYPE_UNSPEC);
+       if (ret)
+               return ret;
+       ret = bpf_mprog_del(entry, &dtuple, &rtuple, flags);
+       bpf_mprog_tuple_put(&rtuple);
+       return ret;
+}
+

[...]

author maintainer
#30 lore raw

PATCH v2 2/7 bpf: Add fd-based tcx multi-prog infra with link support

In reply to Daniel Borkmann

This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment.

Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example:

- From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1]

- From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2]

BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware.

Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different.

We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient.

For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one.

For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective.

The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB.

Kudos also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work.

[0] https://lpc.events/event/16/contributions/1353/
[1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com/
[2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog
[3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
[4] https://lore.kernel.org/bpf/[email protected]/

---
 MAINTAINERS                    |   4 +-
 include/linux/netdevice.h      |  15 +-
 include/linux/skbuff.h         |   4 +-
 include/net/sch_generic.h      |   2 +-
 include/net/tcx.h              | 157 +++++++++++++++
 include/uapi/linux/bpf.h       |  35 +++-
 kernel/bpf/Kconfig             |   1 +
 kernel/bpf/Makefile            |   1 +
 kernel/bpf/syscall.c           |  95 +++++++--
 kernel/bpf/tcx.c               | 347 +++++++++++++++++++++++++++++++++
 net/Kconfig                    |   5 +
 net/core/dev.c                 | 267 +++++++++++++++----------
 net/core/filter.c              |   4 +-
 net/sched/Kconfig              |   4 +-
 net/sched/sch_ingress.c        |  45 ++++-
 tools/include/uapi/linux/bpf.h |  35 +++-
 16 files changed, 877 insertions(+), 144 deletions(-)
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/tcx.c
MAINTAINERS +3 -1
index 754a9eeca0a1..7a0d0b0c5a5e 100644
@@ -3827,13 +3827,15 @@ L:	[email protected]
 S:	Maintained
 F:	kernel/bpf/bpf_struct*
 
-BPF [NETWORKING] (tc BPF, sock_addr)
+BPF [NETWORKING] (tcx & tc BPF, sock_addr)
 M:	Martin KaFai Lau <[email protected]>
 M:	Daniel Borkmann <[email protected]>
 R:	John Fastabend <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
+F:	include/net/tcx.h
+F:	kernel/bpf/tcx.c
 F:	net/core/filter.c
 F:	net/sched/act_bpf.c
 F:	net/sched/cls_bpf.c
include/linux/netdevice.h +6 -9
index 08fbd4622ccf..fd4281d1cdbb 100644
@@ -1927,8 +1927,7 @@ enum netdev_ml_priv_type {
  *
  *	@rx_handler:		handler for received packets
  *	@rx_handler_data: 	XXX: need comments on this one
- *	@miniq_ingress:		ingress/clsact qdisc specific data for
- *				ingress processing
+ *	@tcx_ingress:		BPF & clsact qdisc specific data for ingress processing
  *	@ingress_queue:		XXX: need comments on this one
  *	@nf_hooks_ingress:	netfilter hooks executed for ingress packets
  *	@broadcast:		hw bcast address
@@ -1949,8 +1948,7 @@ enum netdev_ml_priv_type {
  *	@xps_maps:		all CPUs/RXQs maps for XPS device
  *
  *	@xps_maps:	XXX: need comments on this one
- *	@miniq_egress:		clsact qdisc specific data for
- *				egress processing
+ *	@tcx_egress:		BPF & clsact qdisc specific data for egress processing
  *	@nf_hooks_egress:	netfilter hooks executed for egress packets
  *	@qdisc_hash:		qdisc hash table
  *	@watchdog_timeo:	Represents the timeout that is used by
@@ -2249,9 +2247,8 @@ struct net_device {
 	unsigned int		gro_ipv4_max_size;
 	rx_handler_func_t __rcu	*rx_handler;
 	void __rcu		*rx_handler_data;
-
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_ingress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_ingress;
 #endif
 	struct netdev_queue __rcu *ingress_queue;
 #ifdef CONFIG_NETFILTER_INGRESS
@@ -2279,8 +2276,8 @@ struct net_device {
 #ifdef CONFIG_XPS
 	struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
 #endif
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_egress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_egress;
 #endif
 #ifdef CONFIG_NETFILTER_EGRESS
 	struct nf_hook_entries __rcu *nf_hooks_egress;
include/linux/skbuff.h +2 -2
index 5951904413ab..48c3e307f057 100644
@@ -943,7 +943,7 @@ struct sk_buff {
 	__u8			__mono_tc_offset[0];
 	/* public: */
 	__u8			mono_delivery_time:1;	/* See SKB_MONO_DELIVERY_TIME_MASK */
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	__u8			tc_at_ingress:1;	/* See TC_AT_INGRESS_MASK */
 	__u8			tc_skip_classify:1;
 #endif
@@ -992,7 +992,7 @@ struct sk_buff {
 	__u8			csum_not_inet:1;
 #endif
 
-#ifdef CONFIG_NET_SCHED
+#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
 	__u16			tc_index;	/* traffic control index */
 #endif
 
include/net/sch_generic.h +1 -1
index fab5ba3e61b7..0ade5d1a72b2 100644
@@ -695,7 +695,7 @@ int skb_do_redirect(struct sk_buff *);
 
 static inline bool skb_at_tc_ingress(const struct sk_buff *skb)
 {
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	return skb->tc_at_ingress;
 #else
 	return false;
include/net/tcx.h new file +157 -0

157 lines not on this page · raw message

include/uapi/linux/bpf.h +31 -4
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u32		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+			__u32 flags;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
kernel/bpf/Kconfig +1 -0
index 2dfe1079f772..6a906ff93006 100644
@@ -31,6 +31,7 @@ config BPF_SYSCALL
 	select TASKS_TRACE_RCU
 	select BINARY_PRINTF
 	select NET_SOCK_MSG if NET
+	select NET_XGRESS if NET
 	select PAGE_POOL if NET
 	default n
 	help
kernel/bpf/Makefile +1 -0
index 1bea2eb912cd..f526b7573e97 100644
@@ -21,6 +21,7 @@ obj-$(CONFIG_BPF_SYSCALL) += devmap.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
 obj-$(CONFIG_BPF_SYSCALL) += offload.o
 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
+obj-$(CONFIG_BPF_SYSCALL) += tcx.o
 endif
 ifeq ($(CONFIG_PERF_EVENTS),y)
 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/syscall.c +82 -13

181 lines not on this page · raw message

kernel/bpf/tcx.c new file +347 -0

347 lines not on this page · raw message

net/Kconfig +5 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
 config NET_EGRESS
 	bool
 
+config NET_XGRESS
+	select NET_INGRESS
+	select NET_EGRESS
+	bool
+
 config NET_REDIRECT
 	bool
 
net/core/dev.c +165 -102

335 lines not on this page · raw message

net/core/filter.c +2 -2
index d25d52854c21..1ff9a0988ea6 100644
@@ -9233,7 +9233,7 @@ static struct bpf_insn *bpf_convert_tstamp_read(const struct bpf_prog *prog,
 	__u8 value_reg = si->dst_reg;
 	__u8 skb_reg = si->src_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, read skb->tstamp as is if tstamp_type_access is true.
@@ -9267,7 +9267,7 @@ static struct bpf_insn *bpf_convert_tstamp_write(const struct bpf_prog *prog,
 	__u8 value_reg = si->src_reg;
 	__u8 skb_reg = si->dst_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, write skb->tstamp as is if tstamp_type_access is true.
net/sched/Kconfig +2 -2
index 4b95cb1ac435..470c70deffe2 100644
@@ -347,8 +347,7 @@ config NET_SCH_FQ_PIE
 config NET_SCH_INGRESS
 	tristate "Ingress/classifier-action Qdisc"
 	depends on NET_CLS_ACT
-	select NET_INGRESS
-	select NET_EGRESS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use classifiers for incoming and/or outgoing
 	  packets. This qdisc doesn't do anything else besides running classifiers,
@@ -679,6 +678,7 @@ config NET_EMATCH_IPT
 config NET_CLS_ACT
 	bool "Actions"
 	select NET_CLS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use traffic control actions. Actions
 	  get attached to classifiers and are invoked after a successful
net/sched/sch_ingress.c +41 -4

101 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +31 -4
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u32		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+			__u32 flags;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
Signature · 1 line
2.34.1
maintainer
#31 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Daniel,

A general question (which i think i asked last time as well): who decides what comes after/before what prog in this setup? And would that same entity not have been able to make the same decision using tc priorities?

The idea of protecting programs from being unloaded is very welcome but feels would have made sense to be a separate patchset (we have good need for it). Would it be possible to use that feature in tc and xdp?

comments inline:

Quoted 240 lines · Daniel Borkmann

This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment.

Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example:

- From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1]

- From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2]

BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware.

Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different.

We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient.

For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one.

For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective.

The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB.

Kudos also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work.

[0] https://lpc.events/event/16/contributions/1353/
[1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com/
[2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog
[3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
[4] https://lore.kernel.org/bpf/[email protected]/

---
 MAINTAINERS                    |   4 +-
 include/linux/netdevice.h      |  15 +-
 include/linux/skbuff.h         |   4 +-
 include/net/sch_generic.h      |   2 +-
 include/net/tcx.h              | 157 +++++++++++++++
 include/uapi/linux/bpf.h       |  35 +++-
 kernel/bpf/Kconfig             |   1 +
 kernel/bpf/Makefile            |   1 +
 kernel/bpf/syscall.c           |  95 +++++++--
 kernel/bpf/tcx.c               | 347 +++++++++++++++++++++++++++++++++
 net/Kconfig                    |   5 +
 net/core/dev.c                 | 267 +++++++++++++++----------
 net/core/filter.c              |   4 +-
 net/sched/Kconfig              |   4 +-
 net/sched/sch_ingress.c        |  45 ++++-
 tools/include/uapi/linux/bpf.h |  35 +++-
 16 files changed, 877 insertions(+), 144 deletions(-)
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/tcx.c
MAINTAINERS +3 -1
index 754a9eeca0a1..7a0d0b0c5a5e 100644
@@ -3827,13 +3827,15 @@ L:      [email protected]
 S:     Maintained
 F:     kernel/bpf/bpf_struct*
 
-BPF [NETWORKING] (tc BPF, sock_addr)
+BPF [NETWORKING] (tcx & tc BPF, sock_addr)
 M:     Martin KaFai Lau <[email protected]>
 M:     Daniel Borkmann <[email protected]>
 R:     John Fastabend <[email protected]>
 L:     [email protected]
 L:     [email protected]
 S:     Maintained
+F:     include/net/tcx.h
+F:     kernel/bpf/tcx.c
 F:     net/core/filter.c
 F:     net/sched/act_bpf.c
 F:     net/sched/cls_bpf.c
include/linux/netdevice.h +6 -9
index 08fbd4622ccf..fd4281d1cdbb 100644
@@ -1927,8 +1927,7 @@ enum netdev_ml_priv_type {
  *
  *     @rx_handler:            handler for received packets
  *     @rx_handler_data:       XXX: need comments on this one
- *     @miniq_ingress:         ingress/clsact qdisc specific data for
- *                             ingress processing
+ *     @tcx_ingress:           BPF & clsact qdisc specific data for ingress processing
  *     @ingress_queue:         XXX: need comments on this one
  *     @nf_hooks_ingress:      netfilter hooks executed for ingress packets
  *     @broadcast:             hw bcast address
@@ -1949,8 +1948,7 @@ enum netdev_ml_priv_type {
  *     @xps_maps:              all CPUs/RXQs maps for XPS device
  *
  *     @xps_maps:      XXX: need comments on this one
- *     @miniq_egress:          clsact qdisc specific data for
- *                             egress processing
+ *     @tcx_egress:            BPF & clsact qdisc specific data for egress processing
  *     @nf_hooks_egress:       netfilter hooks executed for egress packets
  *     @qdisc_hash:            qdisc hash table
  *     @watchdog_timeo:        Represents the timeout that is used by
@@ -2249,9 +2247,8 @@ struct net_device {
        unsigned int            gro_ipv4_max_size;
        rx_handler_func_t __rcu *rx_handler;
        void __rcu              *rx_handler_data;
-
-#ifdef CONFIG_NET_CLS_ACT
-       struct mini_Qdisc __rcu *miniq_ingress;
+#ifdef CONFIG_NET_XGRESS
+       struct bpf_mprog_entry __rcu *tcx_ingress;
 #endif
        struct netdev_queue __rcu *ingress_queue;
 #ifdef CONFIG_NETFILTER_INGRESS
@@ -2279,8 +2276,8 @@ struct net_device {
 #ifdef CONFIG_XPS
        struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
 #endif
-#ifdef CONFIG_NET_CLS_ACT
-       struct mini_Qdisc __rcu *miniq_egress;
+#ifdef CONFIG_NET_XGRESS
+       struct bpf_mprog_entry __rcu *tcx_egress;
 #endif
 #ifdef CONFIG_NETFILTER_EGRESS
        struct nf_hook_entries __rcu *nf_hooks_egress;
include/linux/skbuff.h +2 -2
index 5951904413ab..48c3e307f057 100644
@@ -943,7 +943,7 @@ struct sk_buff {
        __u8                    __mono_tc_offset[0];
        /* public: */
        __u8                    mono_delivery_time:1;   /* See SKB_MONO_DELIVERY_TIME_MASK */
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
        __u8                    tc_at_ingress:1;        /* See TC_AT_INGRESS_MASK */
        __u8                    tc_skip_classify:1;
 #endif
@@ -992,7 +992,7 @@ struct sk_buff {
        __u8                    csum_not_inet:1;
 #endif
 
-#ifdef CONFIG_NET_SCHED
+#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
        __u16                   tc_index;       /* traffic control index */
 #endif
 
include/net/sch_generic.h +1 -1
index fab5ba3e61b7..0ade5d1a72b2 100644
@@ -695,7 +695,7 @@ int skb_do_redirect(struct sk_buff *);
 
 static inline bool skb_at_tc_ingress(const struct sk_buff *skb)
 {
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
        return skb->tc_at_ingress;
 #else
        return false;
include/net/tcx.h new file +7 -0
new file mode 100644
index 000000000000..27885ecedff9
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef __NET_TCX_H
+#define __NET_TCX_H
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>

include/net/tcx.h line 17

Quoted 10 lines
include/net/tcx.h new file +10 -0
+
+#include <net/sch_generic.h>
+
+struct mini_Qdisc;
+
+struct tcx_entry {
+       struct bpf_mprog_bundle         bundle;
+       struct mini_Qdisc __rcu         *miniq;
+};
+

Can you please move miniq to the front? From where i sit this looks:
struct tcx_entry {
struct bpf_mprog_bundle bundle
__attribute__((__aligned__(64))); /* 0 3264 */

        /* XXX last struct has 36 bytes of padding */

        /* --- cacheline 51 boundary (3264 bytes) --- */
        struct mini_Qdisc *        miniq;                /*  3264     8 */

        /* size: 3328, cachelines: 52, members: 2 */
        /* padding: 56 */
        /* paddings: 1, sum paddings: 36 */
        /* forced alignments: 1 */
} __attribute__((__aligned__(64)));

That is a _lot_ of cachelines - at the expense of the status quo clsact/ingress qdiscs which access miniq.

Quoted 942 lines
include/net/tcx.h new file +140 -0
+struct tcx_link {
+       struct bpf_link link;
+       struct net_device *dev;
+       u32 location;
+       u32 flags;
+};
+
+static inline struct tcx_link *tcx_link(struct bpf_link *link)
+{
+       return container_of(link, struct tcx_link, link);
+}
+
+static inline const struct tcx_link *tcx_link_const(const struct bpf_link *link)
+{
+       return tcx_link((struct bpf_link *)link);
+}
+
+static inline void tcx_set_ingress(struct sk_buff *skb, bool ingress)
+{
+#ifdef CONFIG_NET_XGRESS
+       skb->tc_at_ingress = ingress;
+#endif
+}
+
+#ifdef CONFIG_NET_XGRESS
+void tcx_inc(void);
+void tcx_dec(void);
+
+static inline struct tcx_entry *tcx_entry(struct bpf_mprog_entry *entry)
+{
+       return container_of(entry->parent, struct tcx_entry, bundle);
+}
+
+static inline void
+tcx_entry_update(struct net_device *dev, struct bpf_mprog_entry *entry, bool ingress)
+{
+       ASSERT_RTNL();
+       if (ingress)
+               rcu_assign_pointer(dev->tcx_ingress, entry);
+       else
+               rcu_assign_pointer(dev->tcx_egress, entry);
+}
+
+static inline struct bpf_mprog_entry *
+dev_tcx_entry_fetch(struct net_device *dev, bool ingress)
+{
+       ASSERT_RTNL();
+       if (ingress)
+               return rcu_dereference_rtnl(dev->tcx_ingress);
+       else
+               return rcu_dereference_rtnl(dev->tcx_egress);
+}
+
+static inline struct bpf_mprog_entry *
+dev_tcx_entry_fetch_or_create(struct net_device *dev, bool ingress, bool *created)
+{
+       struct bpf_mprog_entry *entry = dev_tcx_entry_fetch(dev, ingress);
+
+       *created = false;
+       if (!entry) {
+               entry = bpf_mprog_create(sizeof_field(struct tcx_entry,
+                                                     miniq));
+               if (!entry)
+                       return NULL;
+               *created = true;
+       }
+       return entry;
+}
+
+static inline void tcx_skeys_inc(bool ingress)
+{
+       tcx_inc();
+       if (ingress)
+               net_inc_ingress_queue();
+       else
+               net_inc_egress_queue();
+}
+
+static inline void tcx_skeys_dec(bool ingress)
+{
+       if (ingress)
+               net_dec_ingress_queue();
+       else
+               net_dec_egress_queue();
+       tcx_dec();
+}
+
+static inline enum tcx_action_base tcx_action_code(struct sk_buff *skb, int code)
+{
+       switch (code) {
+       case TCX_PASS:
+               skb->tc_index = qdisc_skb_cb(skb)->tc_classid;
+               fallthrough;
+       case TCX_DROP:
+       case TCX_REDIRECT:
+               return code;
+       case TCX_NEXT:
+       default:
+               return TCX_NEXT;
+       }
+}
+#endif /* CONFIG_NET_XGRESS */
+
+#if defined(CONFIG_NET_XGRESS) && defined(CONFIG_BPF_SYSCALL)
+int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_prog_query(const union bpf_attr *attr,
+                  union bpf_attr __user *uattr);
+void dev_tcx_uninstall(struct net_device *dev);
+#else
+static inline int tcx_prog_attach(const union bpf_attr *attr,
+                                 struct bpf_prog *prog)
+{
+       return -EINVAL;
+}
+
+static inline int tcx_link_attach(const union bpf_attr *attr,
+                                 struct bpf_prog *prog)
+{
+       return -EINVAL;
+}
+
+static inline int tcx_prog_detach(const union bpf_attr *attr,
+                                 struct bpf_prog *prog)
+{
+       return -EINVAL;
+}
+
+static inline int tcx_prog_query(const union bpf_attr *attr,
+                                union bpf_attr __user *uattr)
+{
+       return -EINVAL;
+}
+
+static inline void dev_tcx_uninstall(struct net_device *dev)
+{
+}
+#endif /* CONFIG_NET_XGRESS && CONFIG_BPF_SYSCALL */
+#endif /* __NET_TCX_H */
include/uapi/linux/bpf.h +31 -4
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
        BPF_TRACE_KPROBE_MULTI,
        BPF_LSM_CGROUP,
        BPF_STRUCT_OPS,
+       BPF_TCX_INGRESS,
+       BPF_TCX_EGRESS,
        __MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
        BPF_LINK_TYPE_KPROBE_MULTI = 8,
        BPF_LINK_TYPE_STRUCT_OPS = 9,
        BPF_LINK_TYPE_NETFILTER = 10,
-
+       BPF_LINK_TYPE_TCX = 11,
        MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
                        __u32           map_fd;         /* struct_ops to attach */
                };
                union {
-                       __u32           target_fd;      /* object to attach to */
-                       __u32           target_ifindex; /* target ifindex */
+                       __u32   target_fd;      /* target object to attach to or ... */
+                       __u32   target_ifindex; /* target ifindex */
                };
                __u32           attach_type;    /* attach type */
                __u32           flags;          /* extra flags */
                union {
-                       __u32           target_btf_id;  /* btf_id of target to attach to */
+                       __u32   target_btf_id;  /* btf_id of target to attach to */
                        struct {
                                __aligned_u64   iter_info;      /* extra bpf_iter_link_info */
                                __u32           iter_info_len;  /* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
                                __s32           priority;
                                __u32           flags;
                        } netfilter;
+                       struct {
+                               union {
+                                       __u32   relative_fd;
+                                       __u32   relative_id;
+                               };
+                               __u32           expected_revision;
+                       } tcx;
                };
        } link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
        };
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+       TCX_NEXT        = -1,
+       TCX_PASS        = 0,
+       TCX_DROP        = 2,
+       TCX_REDIRECT    = 7,
+};
+
 struct bpf_xdp_sock {
        __u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
                        __s32 priority;
                        __u32 flags;
                } netfilter;
+               struct {
+                       __u32 ifindex;
+                       __u32 attach_type;
+                       __u32 flags;
+               } tcx;
        };
 } __attribute__((aligned(8)));
 
kernel/bpf/Kconfig +1 -0
index 2dfe1079f772..6a906ff93006 100644
@@ -31,6 +31,7 @@ config BPF_SYSCALL
        select TASKS_TRACE_RCU
        select BINARY_PRINTF
        select NET_SOCK_MSG if NET
+       select NET_XGRESS if NET
        select PAGE_POOL if NET
        default n
        help
kernel/bpf/Makefile +1 -0
index 1bea2eb912cd..f526b7573e97 100644
@@ -21,6 +21,7 @@ obj-$(CONFIG_BPF_SYSCALL) += devmap.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
 obj-$(CONFIG_BPF_SYSCALL) += offload.o
 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
+obj-$(CONFIG_BPF_SYSCALL) += tcx.o
 endif
 ifeq ($(CONFIG_PERF_EVENTS),y)
 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/syscall.c +82 -13
index 92a57efc77de..e2c219d053f4 100644
@@ -37,6 +37,8 @@
 #include <linux/trace_events.h>
 #include <net/netfilter/nf_bpf_link.h>
 
+#include <net/tcx.h>
+
 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
                          (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
                          (map)->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS)
@@ -3522,31 +3524,57 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)
                return BPF_PROG_TYPE_XDP;
        case BPF_LSM_CGROUP:
                return BPF_PROG_TYPE_LSM;
+       case BPF_TCX_INGRESS:
+       case BPF_TCX_EGRESS:
+               return BPF_PROG_TYPE_SCHED_CLS;
        default:
                return BPF_PROG_TYPE_UNSPEC;
        }
 }
 
-#define BPF_PROG_ATTACH_LAST_FIELD replace_bpf_fd
+#define BPF_PROG_ATTACH_LAST_FIELD expected_revision
+
+#define BPF_F_ATTACH_MASK_BASE \
+       (BPF_F_ALLOW_OVERRIDE | \
+        BPF_F_ALLOW_MULTI |    \
+        BPF_F_REPLACE)
+
+#define BPF_F_ATTACH_MASK_MPROG        \
+       (BPF_F_REPLACE |        \
+        BPF_F_BEFORE |         \
+        BPF_F_AFTER |          \
+        BPF_F_FIRST |          \
+        BPF_F_LAST |           \
+        BPF_F_ID |             \
+        BPF_F_LINK)
 
-#define BPF_F_ATTACH_MASK \
-       (BPF_F_ALLOW_OVERRIDE | BPF_F_ALLOW_MULTI | BPF_F_REPLACE)
+static bool bpf_supports_mprog(enum bpf_prog_type ptype)
+{
+       switch (ptype) {
+       case BPF_PROG_TYPE_SCHED_CLS:
+               return true;
+       default:
+               return false;
+       }
+}
 
 static int bpf_prog_attach(const union bpf_attr *attr)
 {
        enum bpf_prog_type ptype;
        struct bpf_prog *prog;
+       u32 mask;
        int ret;
 
        if (CHECK_ATTR(BPF_PROG_ATTACH))
                return -EINVAL;
 
-       if (attr->attach_flags & ~BPF_F_ATTACH_MASK)
-               return -EINVAL;
-
        ptype = attach_type_to_prog_type(attr->attach_type);
        if (ptype == BPF_PROG_TYPE_UNSPEC)
                return -EINVAL;
+       mask = bpf_supports_mprog(ptype) ?
+              BPF_F_ATTACH_MASK_MPROG : BPF_F_ATTACH_MASK_BASE;
+       if (attr->attach_flags & ~mask)
+               return -EINVAL;
 
        prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
        if (IS_ERR(prog))
@@ -3582,6 +3610,9 @@ static int bpf_prog_attach(const union bpf_attr *attr)
                else
                        ret = cgroup_bpf_prog_attach(attr, ptype, prog);
                break;
+       case BPF_PROG_TYPE_SCHED_CLS:
+               ret = tcx_prog_attach(attr, prog);
+               break;
        default:
                ret = -EINVAL;
        }
@@ -3591,25 +3622,42 @@ static int bpf_prog_attach(const union bpf_attr *attr)
        return ret;
 }
 
-#define BPF_PROG_DETACH_LAST_FIELD attach_type
+#define BPF_PROG_DETACH_LAST_FIELD expected_revision
 
 static int bpf_prog_detach(const union bpf_attr *attr)
 {
+       struct bpf_prog *prog = NULL;
        enum bpf_prog_type ptype;
+       int ret;
 
        if (CHECK_ATTR(BPF_PROG_DETACH))
                return -EINVAL;
 
        ptype = attach_type_to_prog_type(attr->attach_type);
+       if (bpf_supports_mprog(ptype)) {
+               if (ptype == BPF_PROG_TYPE_UNSPEC)
+                       return -EINVAL;
+               if (attr->attach_flags & ~BPF_F_ATTACH_MASK_MPROG)
+                       return -EINVAL;
+               prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
+               if (IS_ERR(prog)) {
+                       if ((int)attr->attach_bpf_fd > 0)
+                               return PTR_ERR(prog);
+                       prog = NULL;
+               }
+       }
 
        switch (ptype) {
        case BPF_PROG_TYPE_SK_MSG:
        case BPF_PROG_TYPE_SK_SKB:
-               return sock_map_prog_detach(attr, ptype);
+               ret = sock_map_prog_detach(attr, ptype);
+               break;
        case BPF_PROG_TYPE_LIRC_MODE2:
-               return lirc_prog_detach(attr);
+               ret = lirc_prog_detach(attr);
+               break;
        case BPF_PROG_TYPE_FLOW_DISSECTOR:
-               return netns_bpf_prog_detach(attr, ptype);
+               ret = netns_bpf_prog_detach(attr, ptype);
+               break;
        case BPF_PROG_TYPE_CGROUP_DEVICE:
        case BPF_PROG_TYPE_CGROUP_SKB:
        case BPF_PROG_TYPE_CGROUP_SOCK:
@@ -3618,13 +3666,21 @@ static int bpf_prog_detach(const union bpf_attr *attr)
        case BPF_PROG_TYPE_CGROUP_SYSCTL:
        case BPF_PROG_TYPE_SOCK_OPS:
        case BPF_PROG_TYPE_LSM:
-               return cgroup_bpf_prog_detach(attr, ptype);
+               ret = cgroup_bpf_prog_detach(attr, ptype);
+               break;
+       case BPF_PROG_TYPE_SCHED_CLS:
+               ret = tcx_prog_detach(attr, prog);
+               break;
        default:
-               return -EINVAL;
+               ret = -EINVAL;
        }
+
+       if (prog)
+               bpf_prog_put(prog);
+       return ret;
 }
 
-#define BPF_PROG_QUERY_LAST_FIELD query.prog_attach_flags
+#define BPF_PROG_QUERY_LAST_FIELD query.link_attach_flags
 
 static int bpf_prog_query(const union bpf_attr *attr,
                          union bpf_attr __user *uattr)
@@ -3672,6 +3728,9 @@ static int bpf_prog_query(const union bpf_attr *attr,
        case BPF_SK_MSG_VERDICT:
        case BPF_SK_SKB_VERDICT:
                return sock_map_bpf_prog_query(attr, uattr);
+       case BPF_TCX_INGRESS:
+       case BPF_TCX_EGRESS:
+               return tcx_prog_query(attr, uattr);
        default:
                return -EINVAL;
        }
@@ -4629,6 +4688,13 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
                        goto out;
                }
                break;
+       case BPF_PROG_TYPE_SCHED_CLS:
+               if (attr->link_create.attach_type != BPF_TCX_INGRESS &&
+                   attr->link_create.attach_type != BPF_TCX_EGRESS) {
+                       ret = -EINVAL;
+                       goto out;
+               }
+               break;
        default:
                ptype = attach_type_to_prog_type(attr->link_create.attach_type);
                if (ptype == BPF_PROG_TYPE_UNSPEC || ptype != prog->type) {
@@ -4680,6 +4746,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
        case BPF_PROG_TYPE_XDP:
                ret = bpf_xdp_link_attach(attr, prog);
                break;
+       case BPF_PROG_TYPE_SCHED_CLS:
+               ret = tcx_link_attach(attr, prog);
+               break;
        case BPF_PROG_TYPE_NETFILTER:
                ret = bpf_nf_link_attach(attr, prog);
                break;
kernel/bpf/tcx.c new file +347 -0
new file mode 100644
index 000000000000..d3d23b4ed4f0
@@ -0,0 +1,347 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+#include <linux/netdevice.h>
+
+#include <net/tcx.h>
+
+int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       bool created, ingress = attr->attach_type == BPF_TCX_INGRESS;
+       struct net *net = current->nsproxy->net_ns;
+       struct bpf_mprog_entry *entry;
+       struct net_device *dev;
+       int ret;
+
+       rtnl_lock();
+       dev = __dev_get_by_index(net, attr->target_ifindex);
+       if (!dev) {
+               ret = -ENODEV;
+               goto out;
+       }
+       entry = dev_tcx_entry_fetch_or_create(dev, ingress, &created);
+       if (!entry) {
+               ret = -ENOMEM;
+               goto out;
+       }
+       ret = bpf_mprog_attach(entry, prog, NULL, attr->attach_flags,
+                              attr->relative_fd, attr->expected_revision);
+       if (ret >= 0) {
+               if (ret == BPF_MPROG_SWAP)
+                       tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+               bpf_mprog_commit(entry);
+               tcx_skeys_inc(ingress);
+               ret = 0;
+       } else if (created) {
+               bpf_mprog_free(entry);
+       }
+out:
+       rtnl_unlock();
+       return ret;
+}
+
+static bool tcx_release_entry(struct bpf_mprog_entry *entry, int code)
+{
+       return code == BPF_MPROG_FREE && !tcx_entry(entry)->miniq;
+}
+
+int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       bool tcx_release, ingress = attr->attach_type == BPF_TCX_INGRESS;
+       struct net *net = current->nsproxy->net_ns;
+       struct bpf_mprog_entry *entry, *peer;
+       struct net_device *dev;
+       int ret;
+
+       rtnl_lock();
+       dev = __dev_get_by_index(net, attr->target_ifindex);
+       if (!dev) {
+               ret = -ENODEV;
+               goto out;
+       }
+       entry = dev_tcx_entry_fetch(dev, ingress);
+       if (!entry) {
+               ret = -ENOENT;
+               goto out;
+       }
+       ret = bpf_mprog_detach(entry, prog, NULL, attr->attach_flags,
+                              attr->relative_fd, attr->expected_revision);
+       if (ret >= 0) {
+               tcx_release = tcx_release_entry(entry, ret);
+               peer = tcx_release ? NULL : bpf_mprog_peer(entry);
+               if (ret == BPF_MPROG_SWAP || ret == BPF_MPROG_FREE)
+                       tcx_entry_update(dev, peer, ingress);
+               bpf_mprog_commit(entry);
+               tcx_skeys_dec(ingress);
+               if (tcx_release)
+                       bpf_mprog_free(entry);
+               ret = 0;
+       }
+out:
+       rtnl_unlock();
+       return ret;
+}
+
+static void tcx_uninstall(struct net_device *dev, bool ingress)
+{
+       struct bpf_tuple tuple = {};
+       struct bpf_mprog_entry *entry;
+       struct bpf_mprog_fp *fp;
+       struct bpf_mprog_cp *cp;
+
+       entry = dev_tcx_entry_fetch(dev, ingress);
+       if (!entry)
+               return;
+       tcx_entry_update(dev, NULL, ingress);
+       bpf_mprog_commit(entry);
+       bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+               if (tuple.link)
+                       tcx_link(tuple.link)->dev = NULL;
+               else
+                       bpf_prog_put(tuple.prog);
+               tcx_skeys_dec(ingress);
+       }
+       WARN_ON_ONCE(tcx_entry(entry)->miniq);
+       bpf_mprog_free(entry);
+}
+
+void dev_tcx_uninstall(struct net_device *dev)
+{
+       ASSERT_RTNL();
+       tcx_uninstall(dev, true);
+       tcx_uninstall(dev, false);
+}
+
+int tcx_prog_query(const union bpf_attr *attr, union bpf_attr __user *uattr)
+{
+       bool ingress = attr->query.attach_type == BPF_TCX_INGRESS;
+       struct net *net = current->nsproxy->net_ns;
+       struct bpf_mprog_entry *entry;
+       struct net_device *dev;
+       int ret;
+
+       rtnl_lock();
+       dev = __dev_get_by_index(net, attr->query.target_ifindex);
+       if (!dev) {
+               ret = -ENODEV;
+               goto out;
+       }
+       entry = dev_tcx_entry_fetch(dev, ingress);
+       if (!entry) {
+               ret = -ENOENT;
+               goto out;
+       }
+       ret = bpf_mprog_query(attr, uattr, entry);
+out:
+       rtnl_unlock();
+       return ret;
+}
+
+static int tcx_link_prog_attach(struct bpf_link *l, u32 flags, u32 object,
+                               u32 expected_revision)
+{
+       struct tcx_link *link = tcx_link(l);
+       bool created, ingress = link->location == BPF_TCX_INGRESS;
+       struct net_device *dev = link->dev;
+       struct bpf_mprog_entry *entry;
+       int ret;
+
+       ASSERT_RTNL();
+       entry = dev_tcx_entry_fetch_or_create(dev, ingress, &created);
+       if (!entry)
+               return -ENOMEM;
+       ret = bpf_mprog_attach(entry, l->prog, l, flags, object,
+                              expected_revision);
+       if (ret >= 0) {
+               if (ret == BPF_MPROG_SWAP)
+                       tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+               bpf_mprog_commit(entry);
+               tcx_skeys_inc(ingress);
+               ret = 0;
+       } else if (created) {
+               bpf_mprog_free(entry);
+       }
+       return ret;
+}
+
+static void tcx_link_release(struct bpf_link *l)
+{
+       struct tcx_link *link = tcx_link(l);
+       bool tcx_release, ingress = link->location == BPF_TCX_INGRESS;
+       struct bpf_mprog_entry *entry, *peer;
+       struct net_device *dev;
+       int ret = 0;
+
+       rtnl_lock();
+       dev = link->dev;
+       if (!dev)
+               goto out;
+       entry = dev_tcx_entry_fetch(dev, ingress);
+       if (!entry) {
+               ret = -ENOENT;
+               goto out;
+       }
+       ret = bpf_mprog_detach(entry, l->prog, l, link->flags, 0, 0);
+       if (ret >= 0) {
+               tcx_release = tcx_release_entry(entry, ret);
+               peer = tcx_release ? NULL : bpf_mprog_peer(entry);
+               if (ret == BPF_MPROG_SWAP || ret == BPF_MPROG_FREE)
+                       tcx_entry_update(dev, peer, ingress);
+               bpf_mprog_commit(entry);
+               tcx_skeys_dec(ingress);
+               if (tcx_release)
+                       bpf_mprog_free(entry);
+               link->dev = NULL;
+               ret = 0;
+       }
+out:
+       WARN_ON_ONCE(ret);
+       rtnl_unlock();
+}
+
+static int tcx_link_update(struct bpf_link *l, struct bpf_prog *nprog,
+                          struct bpf_prog *oprog)
+{
+       struct tcx_link *link = tcx_link(l);
+       bool ingress = link->location == BPF_TCX_INGRESS;
+       struct net_device *dev = link->dev;
+       struct bpf_mprog_entry *entry;
+       int ret = 0;
+
+       rtnl_lock();
+       if (!link->dev) {
+               ret = -ENOLINK;
+               goto out;
+       }
+       if (oprog && l->prog != oprog) {
+               ret = -EPERM;
+               goto out;
+       }
+       oprog = l->prog;
+       if (oprog == nprog) {
+               bpf_prog_put(nprog);
+               goto out;
+       }
+       entry = dev_tcx_entry_fetch(dev, ingress);
+       if (!entry) {
+               ret = -ENOENT;
+               goto out;
+       }
+       ret = bpf_mprog_attach(entry, nprog, l,
+                              BPF_F_REPLACE | BPF_F_ID | link->flags,
+                              l->prog->aux->id, 0);
+       if (ret >= 0) {
+               if (ret == BPF_MPROG_SWAP)
+                       tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+               bpf_mprog_commit(entry);
+               tcx_skeys_inc(ingress);
+               oprog = xchg(&l->prog, nprog);
+               bpf_prog_put(oprog);
+               ret = 0;
+       }
+out:
+       rtnl_unlock();
+       return ret;
+}
+
+static void tcx_link_dealloc(struct bpf_link *l)
+{
+       kfree(tcx_link(l));
+}
+
+static void tcx_link_fdinfo(const struct bpf_link *l, struct seq_file *seq)
+{
+       const struct tcx_link *link = tcx_link_const(l);
+       u32 ifindex = 0;
+
+       rtnl_lock();
+       if (link->dev)
+               ifindex = link->dev->ifindex;
+       rtnl_unlock();
+
+       seq_printf(seq, "ifindex:\t%u\n", ifindex);
+       seq_printf(seq, "attach_type:\t%u (%s)\n",
+                  link->location,
+                  link->location == BPF_TCX_INGRESS ? "ingress" : "egress");
+       seq_printf(seq, "flags:\t%u\n", link->flags);
+}
+
+static int tcx_link_fill_info(const struct bpf_link *l,
+                             struct bpf_link_info *info)
+{
+       const struct tcx_link *link = tcx_link_const(l);
+       u32 ifindex = 0;
+
+       rtnl_lock();
+       if (link->dev)
+               ifindex = link->dev->ifindex;
+       rtnl_unlock();
+
+       info->tcx.ifindex = ifindex;
+       info->tcx.attach_type = link->location;
+       info->tcx.flags = link->flags;
+       return 0;
+}
+
+static int tcx_link_detach(struct bpf_link *l)
+{
+       tcx_link_release(l);
+       return 0;
+}
+
+static const struct bpf_link_ops tcx_link_lops = {
+       .release        = tcx_link_release,
+       .detach         = tcx_link_detach,
+       .dealloc        = tcx_link_dealloc,
+       .update_prog    = tcx_link_update,
+       .show_fdinfo    = tcx_link_fdinfo,
+       .fill_link_info = tcx_link_fill_info,
+};
+
+int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       struct net *net = current->nsproxy->net_ns;
+       struct bpf_link_primer link_primer;
+       struct net_device *dev;
+       struct tcx_link *link;
+       int fd, err;
+
+       dev = dev_get_by_index(net, attr->link_create.target_ifindex);
+       if (!dev)
+               return -EINVAL;
+       link = kzalloc(sizeof(*link), GFP_USER);
+       if (!link) {
+               err = -ENOMEM;
+               goto out_put;
+       }
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_TCX, &tcx_link_lops, prog);
+       link->location = attr->link_create.attach_type;
+       link->flags = attr->link_create.flags & (BPF_F_FIRST | BPF_F_LAST);
+       link->dev = dev;
+
+       err = bpf_link_prime(&link->link, &link_primer);
+       if (err) {
+               kfree(link);
+               goto out_put;
+       }
+       rtnl_lock();
+       err = tcx_link_prog_attach(&link->link, attr->link_create.flags,
+                                  attr->link_create.tcx.relative_fd,
+                                  attr->link_create.tcx.expected_revision);
+       if (!err)
+               fd = bpf_link_settle(&link_primer);
+       rtnl_unlock();
+       if (err) {
+               link->dev = NULL;
+               bpf_link_cleanup(&link_primer);
+               goto out_put;
+       }
+       dev_put(dev);
+       return fd;
+out_put:
+       dev_put(dev);
+       return err;
+}
net/Kconfig +5 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
 config NET_EGRESS
        bool
 
+config NET_XGRESS
+       select NET_INGRESS
+       select NET_EGRESS
+       bool
+
 config NET_REDIRECT
        bool
 
net/core/dev.c +88 -10
index 3393c2f3dbe8..95c7e3189884 100644
@@ -107,6 +107,7 @@
 #include <net/pkt_cls.h>
 #include <net/checksum.h>
 #include <net/xfrm.h>
+#include <net/tcx.h>
 #include <linux/highmem.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -154,7 +155,6 @@
 #include "dev.h"
 #include "net-sysfs.h"
 
-
 static DEFINE_SPINLOCK(ptype_lock);
 struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
 struct list_head ptype_all __read_mostly;      /* Taps */
@@ -3923,69 +3923,200 @@ int dev_loopback_xmit(struct net *net, struct sock *sk, struct sk_buff *skb)
 EXPORT_SYMBOL(dev_loopback_xmit);
 
 #ifdef CONFIG_NET_EGRESS
-static struct sk_buff *
-sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
+static struct netdev_queue *
+netdev_tx_queue_mapping(struct net_device *dev, struct sk_buff *skb)
+{
+       int qm = skb_get_queue_mapping(skb);
+
+       return netdev_get_tx_queue(dev, netdev_cap_txqueue(dev, qm));
+}
+
+static bool netdev_xmit_txqueue_skipped(void)
 {
+       return __this_cpu_read(softnet_data.xmit.skip_txqueue);
+}
+
+void netdev_xmit_skip_txqueue(bool skip)
+{
+       __this_cpu_write(softnet_data.xmit.skip_txqueue, skip);
+}
+EXPORT_SYMBOL_GPL(netdev_xmit_skip_txqueue);
+#endif /* CONFIG_NET_EGRESS */
+
+#ifdef CONFIG_NET_XGRESS
+static int tc_run(struct tcx_entry *entry, struct sk_buff *skb)
+{
+       int ret = TC_ACT_UNSPEC;
 #ifdef CONFIG_NET_CLS_ACT
-       struct mini_Qdisc *miniq = rcu_dereference_bh(dev->miniq_egress);
-       struct tcf_result cl_res;
+       struct mini_Qdisc *miniq = rcu_dereference_bh(entry->miniq);
+       struct tcf_result res;
 
        if (!miniq)
-               return skb;
+               return ret;
 
-       /* qdisc_skb_cb(skb)->pkt_len was already set by the caller. */
        tc_skb_cb(skb)->mru = 0;
        tc_skb_cb(skb)->post_ct = false;
-       mini_qdisc_bstats_cpu_update(miniq, skb);
 
-       switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
+       mini_qdisc_bstats_cpu_update(miniq, skb);
+       ret = tcf_classify(skb, miniq->block, miniq->filter_list, &res, false);
+       /* Only tcf related quirks below. */
+       switch (ret) {
+       case TC_ACT_SHOT:
+               mini_qdisc_qstats_cpu_drop(miniq);
+               break;
        case TC_ACT_OK:
         case TC_ACT_RECLASSIFY:
-               skb->tc_index = TC_H_MIN(cl_res.classid);
+               skb->tc_index = TC_H_MIN(res.classid);
                break;
+       }
+#endif /* CONFIG_NET_CLS_ACT */
+       return ret;
+}
+
+static DEFINE_STATIC_KEY_FALSE(tcx_needed_key);
+
+void tcx_inc(void)
+{
+       static_branch_inc(&tcx_needed_key);
+}
+EXPORT_SYMBOL_GPL(tcx_inc);
+
+void tcx_dec(void)
+{
+       static_branch_dec(&tcx_needed_key);
+}
+EXPORT_SYMBOL_GPL(tcx_dec);
+
+static __always_inline enum tcx_action_base
+tcx_run(const struct bpf_mprog_entry *entry, struct sk_buff *skb,
+       const bool needs_mac)
+{
+       const struct bpf_mprog_fp *fp;
+       const struct bpf_prog *prog;
+       int ret = TCX_NEXT;
+
+       if (needs_mac)
+               __skb_push(skb, skb->mac_len);
+       bpf_mprog_foreach_prog(entry, fp, prog) {
+               bpf_compute_data_pointers(skb);
+               ret = bpf_prog_run(prog, skb);
+               if (ret != TCX_NEXT)
+                       break;
+       }
+       if (needs_mac)
+               __skb_pull(skb, skb->mac_len);
+       return tcx_action_code(skb, ret);
+}
+
+static __always_inline struct sk_buff *
+sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
+                  struct net_device *orig_dev, bool *another)
+{
+       struct bpf_mprog_entry *entry = rcu_dereference_bh(skb->dev->tcx_ingress);
+       int sch_ret;
+
+       if (!entry)
+               return skb;
+       if (*pt_prev) {
+               *ret = deliver_skb(skb, *pt_prev, orig_dev);
+               *pt_prev = NULL;
+       }

net/core/dev.c int dev_loopback_xmit(struct net *net, struct sock *sk, struct sk_buff *skb) · line 4033

Quoted 10 lines
net/core/dev.c +10 -0
+
+       qdisc_skb_cb(skb)->pkt_len = skb->len;
+       tcx_set_ingress(skb, true);
+
+       if (static_branch_unlikely(&tcx_needed_key)) {
+               sch_ret = tcx_run(entry, skb, true);
+               if (sch_ret != TC_ACT_UNSPEC)
+                       goto ingress_verdict;
+       }
+       sch_ret = tc_run(container_of(entry->parent, struct tcx_entry, bundle), skb);

This...
Essentially if we have a mix of tc and txc, tcx gets to run first/faster. It would be fairer to have tc infront. Same for egress...

cheers,
jamal

Quoted 448 lines
net/core/dev.c +67 -92
+ingress_verdict:
+       switch (sch_ret) {
+       case TC_ACT_REDIRECT:
+               /* skb_mac_header check was done by BPF, so we can safely
+                * push the L2 header back before redirecting to another
+                * netdev.
+                */
+               __skb_push(skb, skb->mac_len);
+               if (skb_do_redirect(skb) == -EAGAIN) {
+                       __skb_pull(skb, skb->mac_len);
+                       *another = true;
+                       break;
+               }
+               *ret = NET_RX_SUCCESS;
+               return NULL;
        case TC_ACT_SHOT:
-               mini_qdisc_qstats_cpu_drop(miniq);
-               *ret = NET_XMIT_DROP;
-               kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+               kfree_skb_reason(skb, SKB_DROP_REASON_TC_INGRESS);
+               *ret = NET_RX_DROP;
                return NULL;
+       /* used by tc_run */
        case TC_ACT_STOLEN:
        case TC_ACT_QUEUED:
        case TC_ACT_TRAP:
-               *ret = NET_XMIT_SUCCESS;
                consume_skb(skb);
+               fallthrough;
+       case TC_ACT_CONSUMED:
+               *ret = NET_RX_SUCCESS;
                return NULL;
+       }
+
+       return skb;
+}
+
+static __always_inline struct sk_buff *
+sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
+{
+       struct bpf_mprog_entry *entry = rcu_dereference_bh(dev->tcx_egress);
+       int sch_ret;
+
+       if (!entry)
+               return skb;
+
+       /* qdisc_skb_cb(skb)->pkt_len & tcx_set_ingress() was
+        * already set by the caller.
+        */
+       if (static_branch_unlikely(&tcx_needed_key)) {
+               sch_ret = tcx_run(entry, skb, false);
+               if (sch_ret != TC_ACT_UNSPEC)
+                       goto egress_verdict;
+       }
+       sch_ret = tc_run(container_of(entry->parent, struct tcx_entry, bundle), skb);
+egress_verdict:
+       switch (sch_ret) {
        case TC_ACT_REDIRECT:
                /* No need to push/pop skb's mac_header here on egress! */
                skb_do_redirect(skb);
                *ret = NET_XMIT_SUCCESS;
                return NULL;
-       default:
-               break;
+       case TC_ACT_SHOT:
+               kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+               *ret = NET_XMIT_DROP;
+               return NULL;
+       /* used by tc_run */
+       case TC_ACT_STOLEN:
+       case TC_ACT_QUEUED:
+       case TC_ACT_TRAP:
+               *ret = NET_XMIT_SUCCESS;
+               return NULL;
        }
-#endif /* CONFIG_NET_CLS_ACT */
 
        return skb;
 }
-
-static struct netdev_queue *
-netdev_tx_queue_mapping(struct net_device *dev, struct sk_buff *skb)
-{
-       int qm = skb_get_queue_mapping(skb);
-
-       return netdev_get_tx_queue(dev, netdev_cap_txqueue(dev, qm));
-}
-
-static bool netdev_xmit_txqueue_skipped(void)
+#else
+static __always_inline struct sk_buff *
+sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
+                  struct net_device *orig_dev, bool *another)
 {
-       return __this_cpu_read(softnet_data.xmit.skip_txqueue);
+       return skb;
 }
 
-void netdev_xmit_skip_txqueue(bool skip)
+static __always_inline struct sk_buff *
+sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
 {
-       __this_cpu_write(softnet_data.xmit.skip_txqueue, skip);
+       return skb;
 }
-EXPORT_SYMBOL_GPL(netdev_xmit_skip_txqueue);
-#endif /* CONFIG_NET_EGRESS */
+#endif /* CONFIG_NET_XGRESS */
 
 #ifdef CONFIG_XPS
 static int __get_xps_queue_idx(struct net_device *dev, struct sk_buff *skb,
@@ -4169,9 +4300,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
        skb_update_prio(skb);
 
        qdisc_pkt_len_init(skb);
-#ifdef CONFIG_NET_CLS_ACT
-       skb->tc_at_ingress = 0;
-#endif
+       tcx_set_ingress(skb, false);
 #ifdef CONFIG_NET_EGRESS
        if (static_branch_unlikely(&egress_needed_key)) {
                if (nf_hook_egress_active()) {
@@ -5103,72 +5232,6 @@ int (*br_fdb_test_addr_hook)(struct net_device *dev,
 EXPORT_SYMBOL_GPL(br_fdb_test_addr_hook);
 #endif
 
-static inline struct sk_buff *
-sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
-                  struct net_device *orig_dev, bool *another)
-{
-#ifdef CONFIG_NET_CLS_ACT
-       struct mini_Qdisc *miniq = rcu_dereference_bh(skb->dev->miniq_ingress);
-       struct tcf_result cl_res;
-
-       /* If there's at least one ingress present somewhere (so
-        * we get here via enabled static key), remaining devices
-        * that are not configured with an ingress qdisc will bail
-        * out here.
-        */
-       if (!miniq)
-               return skb;
-
-       if (*pt_prev) {
-               *ret = deliver_skb(skb, *pt_prev, orig_dev);
-               *pt_prev = NULL;
-       }
-
-       qdisc_skb_cb(skb)->pkt_len = skb->len;
-       tc_skb_cb(skb)->mru = 0;
-       tc_skb_cb(skb)->post_ct = false;
-       skb->tc_at_ingress = 1;
-       mini_qdisc_bstats_cpu_update(miniq, skb);
-
-       switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
-       case TC_ACT_OK:
-       case TC_ACT_RECLASSIFY:
-               skb->tc_index = TC_H_MIN(cl_res.classid);
-               break;
-       case TC_ACT_SHOT:
-               mini_qdisc_qstats_cpu_drop(miniq);
-               kfree_skb_reason(skb, SKB_DROP_REASON_TC_INGRESS);
-               *ret = NET_RX_DROP;
-               return NULL;
-       case TC_ACT_STOLEN:
-       case TC_ACT_QUEUED:
-       case TC_ACT_TRAP:
-               consume_skb(skb);
-               *ret = NET_RX_SUCCESS;
-               return NULL;
-       case TC_ACT_REDIRECT:
-               /* skb_mac_header check was done by cls/act_bpf, so
-                * we can safely push the L2 header back before
-                * redirecting to another netdev
-                */
-               __skb_push(skb, skb->mac_len);
-               if (skb_do_redirect(skb) == -EAGAIN) {
-                       __skb_pull(skb, skb->mac_len);
-                       *another = true;
-                       break;
-               }
-               *ret = NET_RX_SUCCESS;
-               return NULL;
-       case TC_ACT_CONSUMED:
-               *ret = NET_RX_SUCCESS;
-               return NULL;
-       default:
-               break;
-       }
-#endif /* CONFIG_NET_CLS_ACT */
-       return skb;
-}
-
 /**
  *     netdev_is_rx_handler_busy - check if receive handler is registered
  *     @dev: device to check
@@ -10873,7 +10936,7 @@ void unregister_netdevice_many_notify(struct list_head *head,
 
                /* Shutdown queueing discipline. */
                dev_shutdown(dev);
-
+               dev_tcx_uninstall(dev);
                dev_xdp_uninstall(dev);
                bpf_dev_bound_netdev_unregister(dev);
 
net/core/filter.c +2 -2
index d25d52854c21..1ff9a0988ea6 100644
@@ -9233,7 +9233,7 @@ static struct bpf_insn *bpf_convert_tstamp_read(const struct bpf_prog *prog,
        __u8 value_reg = si->dst_reg;
        __u8 skb_reg = si->src_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
        /* If the tstamp_type is read,
         * the bpf prog is aware the tstamp could have delivery time.
         * Thus, read skb->tstamp as is if tstamp_type_access is true.
@@ -9267,7 +9267,7 @@ static struct bpf_insn *bpf_convert_tstamp_write(const struct bpf_prog *prog,
        __u8 value_reg = si->src_reg;
        __u8 skb_reg = si->dst_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
        /* If the tstamp_type is read,
         * the bpf prog is aware the tstamp could have delivery time.
         * Thus, write skb->tstamp as is if tstamp_type_access is true.
net/sched/Kconfig +2 -2
index 4b95cb1ac435..470c70deffe2 100644
@@ -347,8 +347,7 @@ config NET_SCH_FQ_PIE
 config NET_SCH_INGRESS
        tristate "Ingress/classifier-action Qdisc"
        depends on NET_CLS_ACT
-       select NET_INGRESS
-       select NET_EGRESS
+       select NET_XGRESS
        help
          Say Y here if you want to use classifiers for incoming and/or outgoing
          packets. This qdisc doesn't do anything else besides running classifiers,
@@ -679,6 +678,7 @@ config NET_EMATCH_IPT
 config NET_CLS_ACT
        bool "Actions"
        select NET_CLS
+       select NET_XGRESS
        help
          Say Y here if you want to use traffic control actions. Actions
          get attached to classifiers and are invoked after a successful
net/sched/sch_ingress.c +41 -4
index 84838128b9c5..4af1360f537e 100644
@@ -13,6 +13,7 @@
 #include <net/netlink.h>
 #include <net/pkt_sched.h>
 #include <net/pkt_cls.h>
+#include <net/tcx.h>
 
 struct ingress_sched_data {
        struct tcf_block *block;
@@ -78,11 +79,18 @@ static int ingress_init(struct Qdisc *sch, struct nlattr *opt,
 {
        struct ingress_sched_data *q = qdisc_priv(sch);
        struct net_device *dev = qdisc_dev(sch);
+       struct bpf_mprog_entry *entry;
+       bool created;
        int err;
 
        net_inc_ingress_queue();
 
-       mini_qdisc_pair_init(&q->miniqp, sch, &dev->miniq_ingress);
+       entry = dev_tcx_entry_fetch_or_create(dev, true, &created);
+       if (!entry)
+               return -ENOMEM;
+       mini_qdisc_pair_init(&q->miniqp, sch, &tcx_entry(entry)->miniq);
+       if (created)
+               tcx_entry_update(dev, entry, true);
 
        q->block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
        q->block_info.chain_head_change = clsact_chain_head_change;
@@ -93,15 +101,20 @@ static int ingress_init(struct Qdisc *sch, struct nlattr *opt,
                return err;
 
        mini_qdisc_pair_block_init(&q->miniqp, q->block);
-
        return 0;
 }
 
 static void ingress_destroy(struct Qdisc *sch)
 {
        struct ingress_sched_data *q = qdisc_priv(sch);
+       struct net_device *dev = qdisc_dev(sch);
+       struct bpf_mprog_entry *entry = rtnl_dereference(dev->tcx_ingress);
 
        tcf_block_put_ext(q->block, sch, &q->block_info);
+       if (entry && !bpf_mprog_total(entry)) {
+               tcx_entry_update(dev, NULL, true);
+               bpf_mprog_free(entry);
+       }
        net_dec_ingress_queue();
 }
 
@@ -217,12 +230,19 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 {
        struct clsact_sched_data *q = qdisc_priv(sch);
        struct net_device *dev = qdisc_dev(sch);
+       struct bpf_mprog_entry *entry;
+       bool created;
        int err;
 
        net_inc_ingress_queue();
        net_inc_egress_queue();
 
-       mini_qdisc_pair_init(&q->miniqp_ingress, sch, &dev->miniq_ingress);
+       entry = dev_tcx_entry_fetch_or_create(dev, true, &created);
+       if (!entry)
+               return -ENOMEM;
+       mini_qdisc_pair_init(&q->miniqp_ingress, sch, &tcx_entry(entry)->miniq);
+       if (created)
+               tcx_entry_update(dev, entry, true);
 
        q->ingress_block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
        q->ingress_block_info.chain_head_change = clsact_chain_head_change;
@@ -235,7 +255,12 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 
        mini_qdisc_pair_block_init(&q->miniqp_ingress, q->ingress_block);
 
-       mini_qdisc_pair_init(&q->miniqp_egress, sch, &dev->miniq_egress);
+       entry = dev_tcx_entry_fetch_or_create(dev, false, &created);
+       if (!entry)
+               return -ENOMEM;
+       mini_qdisc_pair_init(&q->miniqp_egress, sch, &tcx_entry(entry)->miniq);
+       if (created)
+               tcx_entry_update(dev, entry, false);
 
        q->egress_block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_EGRESS;
        q->egress_block_info.chain_head_change = clsact_chain_head_change;
@@ -247,9 +272,21 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 static void clsact_destroy(struct Qdisc *sch)
 {
        struct clsact_sched_data *q = qdisc_priv(sch);
+       struct net_device *dev = qdisc_dev(sch);
+       struct bpf_mprog_entry *ingress_entry = rtnl_dereference(dev->tcx_ingress);
+       struct bpf_mprog_entry *egress_entry = rtnl_dereference(dev->tcx_egress);
 
        tcf_block_put_ext(q->egress_block, sch, &q->egress_block_info);
+       if (egress_entry && !bpf_mprog_total(egress_entry)) {
+               tcx_entry_update(dev, NULL, false);
+               bpf_mprog_free(egress_entry);
+       }
+
        tcf_block_put_ext(q->ingress_block, sch, &q->ingress_block_info);
+       if (ingress_entry && !bpf_mprog_total(ingress_entry)) {
+               tcx_entry_update(dev, NULL, true);
+               bpf_mprog_free(ingress_entry);
+       }
 
        net_dec_ingress_queue();
        net_dec_egress_queue();
tools/include/uapi/linux/bpf.h +31 -5
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
        BPF_TRACE_KPROBE_MULTI,
        BPF_LSM_CGROUP,
        BPF_STRUCT_OPS,
+       BPF_TCX_INGRESS,
+       BPF_TCX_EGRESS,
        __MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
        BPF_LINK_TYPE_KPROBE_MULTI = 8,
        BPF_LINK_TYPE_STRUCT_OPS = 9,
        BPF_LINK_TYPE_NETFILTER = 10,
-
+       BPF_LINK_TYPE_TCX = 11,
        MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
                        __u32           map_fd;         /* struct_ops to attach */
                };
                union {
-                       __u32           target_fd;      /* object to attach to */
-                       __u32           target_ifindex; /* target ifindex */
+                       __u32   target_fd;      /* target object to attach to or ... */
+                       __u32   target_ifindex; /* target ifindex */
                };
                __u32           attach_type;    /* attach type */
                __u32           flags;          /* extra flags */
                union {
-                       __u32           target_btf_id;  /* btf_id of target to attach to */
+                       __u32   target_btf_id;  /* btf_id of target to attach to */
                        struct {
                                __aligned_u64   iter_info;      /* extra bpf_iter_link_info */
                                __u32           iter_info_len;  /* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
                                __s32           priority;
                                __u32           flags;
                        } netfilter;
+                       struct {
+                               union {
+                                       __u32   relative_fd;
+                                       __u32   relative_id;
+                               };
+                               __u32           expected_revision;
+                       } tcx;
                };
        } link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
        };
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+       TCX_NEXT        = -1,
+       TCX_PASS        = 0,
+       TCX_DROP        = 2,
+       TCX_REDIRECT    = 7,
+};
+
 struct bpf_xdp_sock {
        __u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
                        __s32 priority;
                        __u32 flags;
                } netfilter;
+               struct {
+                       __u32 ifindex;
+                       __u32 attach_type;
+                       __u32 flags;
+               } tcx;
        };
 } __attribute__((aligned(8)));
 
--

2.34.1

author maintainer
#32 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Hi Jamal,

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:
[...]

Quoted 4 lines

A general question (which i think i asked last time as well): who decides what comes after/before what prog in this setup? And would that same entity not have been able to make the same decision using tc priorities?

Back in the first version of the series I initially coded up this option that the tc_run() would basically be a fake 'bpf_prog' and it would have, say, fixed prio 1000. It would get executed via tcx_run() when iterating via bpf_mprog_foreach_prog() where bpf_prog_run() is called, and then users could pick for native BPF prio before or after that. But then the feedback was that sticking to prio is a bad user experience which led to the development of what is in patch 1 of this series (see the details there).

Quoted 4 lines

The idea of protecting programs from being unloaded is very welcome but feels would have made sense to be a separate patchset (we have good need for it). Would it be possible to use that feature in tc and xdp?

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 22 lines

include/net/tcx.h line 17

Quoted 5 lines
include/net/tcx.h new file +5 -0
+struct tcx_entry {
+       struct bpf_mprog_bundle         bundle;
+       struct mini_Qdisc __rcu         *miniq;
+};
+

Can you please move miniq to the front? From where i sit this looks:
struct tcx_entry {
struct bpf_mprog_bundle bundle
__attribute__((__aligned__(64))); /* 0 3264 */

         /* XXX last struct has 36 bytes of padding */

         /* --- cacheline 51 boundary (3264 bytes) --- */
         struct mini_Qdisc *        miniq;                /*  3264     8 */

         /* size: 3328, cachelines: 52, members: 2 */
         /* padding: 56 */
         /* paddings: 1, sum paddings: 36 */
         /* forced alignments: 1 */
} __attribute__((__aligned__(64)));
Quoted 2 lines

That is a _lot_ of cachelines - at the expense of the status quo clsact/ingress qdiscs which access miniq.

Ah yes, I'll fix this up.

Thanks,
Daniel

maintainer
#33 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Hi Daniel,

Quoted 10 lines · Daniel Borkmann

Hi Jamal,

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:
[...]

Quoted 4 lines

A general question (which i think i asked last time as well): who decides what comes after/before what prog in this setup? And would that same entity not have been able to make the same decision using tc priorities?

Quoted 8 lines

Back in the first version of the series I initially coded up this option that the tc_run() would basically be a fake 'bpf_prog' and it would have, say, fixed prio 1000. It would get executed via tcx_run() when iterating via bpf_mprog_foreach_prog() where bpf_prog_run() is called, and then users could pick for native BPF prio before or after that. But then the feedback was that sticking to prio is a bad user experience which led to the development of what is in patch 1 of this series (see the details there).

Thanks. I read the commit message in patch 1 and followed the thread back including some of the discussion we had and i am still disagreeing that this couldnt be solved with a smart priority based scheme - but i think we can move on since this is standalone and doesnt affect tc.

Daniel - i am still curious in the new scheme of things how would cilium vs datadog food fight get resolved without some arbitration entity?

Quoted 10 lines
Quoted 4 lines

The idea of protecting programs from being unloaded is very welcome but feels would have made sense to be a separate patchset (we have good need for it). Would it be possible to use that feature in tc and xdp?

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 22 lines
Quoted 22 lines

include/net/tcx.h line 17

Quoted 5 lines
include/net/tcx.h new file +5 -0
+struct tcx_entry {
+       struct bpf_mprog_bundle         bundle;
+       struct mini_Qdisc __rcu         *miniq;
+};
+

Can you please move miniq to the front? From where i sit this looks:
struct tcx_entry {
struct bpf_mprog_bundle bundle
__attribute__((__aligned__(64))); /* 0 3264 */

         /* XXX last struct has 36 bytes of padding */

         /* --- cacheline 51 boundary (3264 bytes) --- */
         struct mini_Qdisc *        miniq;                /*  3264     8 */

         /* size: 3328, cachelines: 52, members: 2 */
         /* padding: 56 */
         /* paddings: 1, sum paddings: 36 */
         /* forced alignments: 1 */
} __attribute__((__aligned__(64)));
Quoted 4 lines
Quoted 2 lines

That is a _lot_ of cachelines - at the expense of the status quo clsact/ingress qdiscs which access miniq.

Ah yes, I'll fix this up.

Thanks.

cheers,
jamal

Quoted 2 lines

Thanks,
Daniel

#34 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 44 lines · Jamal Hadi Salim

Hi Daniel,

Quoted 10 lines · Daniel Borkmann

Hi Jamal,

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:
[...]

Quoted 4 lines

A general question (which i think i asked last time as well): who decides what comes after/before what prog in this setup? And would that same entity not have been able to make the same decision using tc priorities?

Quoted 8 lines

Back in the first version of the series I initially coded up this option that the tc_run() would basically be a fake 'bpf_prog' and it would have, say, fixed prio 1000. It would get executed via tcx_run() when iterating via bpf_mprog_foreach_prog() where bpf_prog_run() is called, and then users could pick for native BPF prio before or after that. But then the feedback was that sticking to prio is a bad user experience which led to the development of what is in patch 1 of this series (see the details there).

Thanks. I read the commit message in patch 1 and followed the thread back including some of the discussion we had and i am still disagreeing that this couldnt be solved with a smart priority based scheme - but i think we can move on since this is standalone and doesnt affect tc.

Daniel - i am still curious in the new scheme of things how would cilium vs datadog food fight get resolved without some arbitration entity?

Quoted 10 lines
Quoted 4 lines

The idea of protecting programs from being unloaded is very welcome but feels would have made sense to be a separate patchset (we have good need for it). Would it be possible to use that feature in tc and xdp?

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

That is, until we allow multiple attachments of XDP programs to the same network interface. But even then, no one will be able to accidentally replace attached link, unless they have that link FD and replace underlying BPF program.

Quoted 35 lines
Quoted 24 lines
Quoted 24 lines

include/net/tcx.h line 17

Quoted 7 lines
include/net/tcx.h new file +5 -0
+struct tcx_entry {
+       struct bpf_mprog_bundle         bundle;
+       struct mini_Qdisc __rcu         *miniq;
+};
+

Can you please move miniq to the front? From where i sit this looks:
struct tcx_entry {
struct bpf_mprog_bundle bundle
__attribute__((__aligned__(64))); /* 0 3264 */

         /* XXX last struct has 36 bytes of padding */

         /* --- cacheline 51 boundary (3264 bytes) --- */
         struct mini_Qdisc *        miniq;                /*  3264     8 */

         /* size: 3328, cachelines: 52, members: 2 */
         /* padding: 56 */
         /* paddings: 1, sum paddings: 36 */
         /* forced alignments: 1 */
} __attribute__((__aligned__(64)));
Quoted 4 lines
Quoted 2 lines

That is a _lot_ of cachelines - at the expense of the status quo clsact/ingress qdiscs which access miniq.

Ah yes, I'll fix this up.

Thanks.

cheers,
jamal

Quoted 2 lines

Thanks,
Daniel

maintainer
#35 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Sorry for the late reply, but trying this out now - and have a question:

Quoted 55 lines · Andrii Nakryiko
Quoted 44 lines · Jamal Hadi Salim

Hi Daniel,

Quoted 10 lines · Daniel Borkmann

Hi Jamal,

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:
[...]

Quoted 4 lines

A general question (which i think i asked last time as well): who decides what comes after/before what prog in this setup? And would that same entity not have been able to make the same decision using tc priorities?

Quoted 8 lines

Back in the first version of the series I initially coded up this option that the tc_run() would basically be a fake 'bpf_prog' and it would have, say, fixed prio 1000. It would get executed via tcx_run() when iterating via bpf_mprog_foreach_prog() where bpf_prog_run() is called, and then users could pick for native BPF prio before or after that. But then the feedback was that sticking to prio is a bad user experience which led to the development of what is in patch 1 of this series (see the details there).

Thanks. I read the commit message in patch 1 and followed the thread back including some of the discussion we had and i am still disagreeing that this couldnt be solved with a smart priority based scheme - but i think we can move on since this is standalone and doesnt affect tc.

Daniel - i am still curious in the new scheme of things how would cilium vs datadog food fight get resolved without some arbitration entity?

Quoted 10 lines
Quoted 4 lines

The idea of protecting programs from being unloaded is very welcome but feels would have made sense to be a separate patchset (we have good need for it). Would it be possible to use that feature in tc and xdp?

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 6 lines

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

So this works as advertised. The problem is however not totally solved because it seems we need a process that's alive to hold the ownership. If we had a daemon then that would solve it i think (we dont). Alternatively, you pin the link. The pinning part can be circumvented, unless i misunderstood i,e anybody with the right permissions can remove it.

Am I missing something?

cheers,
jamal

cheers,
jamal

Quoted 40 lines

That is, until we allow multiple attachments of XDP programs to the same network interface. But even then, no one will be able to accidentally replace attached link, unless they have that link FD and replace underlying BPF program.

Quoted 35 lines
Quoted 24 lines
Quoted 24 lines

include/net/tcx.h line 17

Quoted 7 lines
include/net/tcx.h new file +5 -0
+struct tcx_entry {
+       struct bpf_mprog_bundle         bundle;
+       struct mini_Qdisc __rcu         *miniq;
+};
+

Can you please move miniq to the front? From where i sit this looks:
struct tcx_entry {
struct bpf_mprog_bundle bundle
__attribute__((__aligned__(64))); /* 0 3264 */

         /* XXX last struct has 36 bytes of padding */

         /* --- cacheline 51 boundary (3264 bytes) --- */
         struct mini_Qdisc *        miniq;                /*  3264     8 */

         /* size: 3328, cachelines: 52, members: 2 */
         /* padding: 56 */
         /* paddings: 1, sum paddings: 36 */
         /* forced alignments: 1 */
} __attribute__((__aligned__(64)));
Quoted 4 lines
Quoted 2 lines

That is a _lot_ of cachelines - at the expense of the status quo clsact/ingress qdiscs which access miniq.

Ah yes, I'll fix this up.

Thanks.

cheers,
jamal

Quoted 2 lines

Thanks,
Daniel

author maintainer
#36 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 5 lines · Jamal Hadi Salim
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Jamal Hadi Salim
Quoted 1 line · Daniel Borkmann

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:

[...]

Quoted 22 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 5 lines

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

Quoted 8 lines

So this works as advertised. The problem is however not totally solved because it seems we need a process that's alive to hold the ownership. If we had a daemon then that would solve it i think (we dont). Alternatively, you pin the link. The pinning part can be circumvented, unless i misunderstood i,e anybody with the right permissions can remove it.

Am I missing something?

It would be either of those depending on the use case, and for pinning removal, it would require right permissions/acls. Keep in mind that for your application you can also use your own bpffs mount, so you don't need to use the default /sys/fs/bpf one in hostns.

Thanks,
Daniel

maintainer
#37 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 37 lines · Daniel Borkmann
Quoted 5 lines · Jamal Hadi Salim
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Jamal Hadi Salim
Quoted 1 line · Daniel Borkmann

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:

[...]

Quoted 29 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 5 lines

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

So this works as advertised. The problem is however not totally solved because it seems we need a process that's alive to hold the ownership. If we had a daemon then that would solve it i think (we dont). Alternatively, you pin the link. The pinning part can be circumvented, unless i misunderstood i,e anybody with the right permissions can remove it.

Quoted 6 lines
Quoted 1 line

Am I missing something?

It would be either of those depending on the use case, and for pinning removal, it would require right permissions/acls. Keep in mind that for your application you can also use your own bpffs mount, so you don't need to use the default /sys/fs/bpf one in hostns.

This helps for sure - doesnt 100% solve it. It would really be nice if we could tie in a kerberos-like ticketing system for ownership of the mount or something even more fine grained like a link. Doesnt have to be kerberos but anything that would allow a digest of some verifiable credentials/token to be handed to the kernel for authorization...

cheers,
jamal

Quoted 2 lines

Thanks,
Daniel

author maintainer
#38 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 44 lines · Jamal Hadi Salim
Quoted 36 lines · Daniel Borkmann
Quoted 5 lines · Jamal Hadi Salim
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Jamal Hadi Salim
Quoted 1 line · Daniel Borkmann

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:

[...]

Quoted 29 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 5 lines

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

So this works as advertised. The problem is however not totally solved because it seems we need a process that's alive to hold the ownership. If we had a daemon then that would solve it i think (we dont). Alternatively, you pin the link. The pinning part can be circumvented, unless i misunderstood i,e anybody with the right permissions can remove it.

Quoted 6 lines
Quoted 1 line

Am I missing something?

It would be either of those depending on the use case, and for pinning removal, it would require right permissions/acls. Keep in mind that for your application you can also use your own bpffs mount, so you don't need to use the default /sys/fs/bpf one in hostns.

Quoted 5 lines

This helps for sure - doesnt 100% solve it. It would really be nice if we could tie in a kerberos-like ticketing system for ownership of the mount or something even more fine grained like a link. Doesnt have to be kerberos but anything that would allow a digest of some verifiable credentials/token to be handed to the kernel for authorization...

What is your use-case, you don't want anyone except your own orchestration application to access it, so any kind of ACLs, LSM policies or making the mount only available to your container are not enough in this scenario you have in mind?

I think the closest to that is probably the prototype which Lorenz recently built where the user space application's digest is validated via IMA [0].

[0] http://vger.kernel.org/bpfconf2023_material/Lorenz_Bauer_-_BPF_signing_using_fsverity_and_LSM_gatekeeper.pdf
https://github.com/isovalent/bpf-verity

maintainer
#39 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 52 lines · Daniel Borkmann
Quoted 44 lines · Jamal Hadi Salim
Quoted 36 lines · Daniel Borkmann
Quoted 5 lines · Jamal Hadi Salim
Quoted 3 lines · Andrii Nakryiko
Quoted 2 lines · Jamal Hadi Salim
Quoted 1 line · Daniel Borkmann

On 6/8/23 3:25 AM, Jamal Hadi Salim wrote:

[...]

Quoted 29 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

BPF links are supported for XDP today, just tc BPF is one of the few remainders where it is not the case, hence the work of this series. What XDP lacks today however is multi-prog support. With the bpf_mprog concept that could be addressed with that common/uniform api (and Andrii expressed interest in integrating this also for cgroup progs), so yes, various hook points/program types could benefit from it.

Quoted 8 lines

Is there some sample XDP related i could look at? Let me describe our use case: lets say we load an ebpf program foo attached to XDP of a netdev and then something further upstream in the stack is consuming the results of that ebpf XDP program. For some reason someone, at some point, decides to replace the XDP prog with a different one - and the new prog does a very different thing. Could we stop the replacement with the link mechanism you describe? i.e the program is still loaded but is no longer attached to the netdev.

Quoted 5 lines

If you initially attached an XDP program using BPF link api (LINK_CREATE command in bpf() syscall), then subsequent attachment to the same interface (of a new link or program with BPF_PROG_ATTACH) will fail until the current BPF link is detached through closing its last fd.

So this works as advertised. The problem is however not totally solved because it seems we need a process that's alive to hold the ownership. If we had a daemon then that would solve it i think (we dont). Alternatively, you pin the link. The pinning part can be circumvented, unless i misunderstood i,e anybody with the right permissions can remove it.

Quoted 6 lines
Quoted 1 line

Am I missing something?

It would be either of those depending on the use case, and for pinning removal, it would require right permissions/acls. Keep in mind that for your application you can also use your own bpffs mount, so you don't need to use the default /sys/fs/bpf one in hostns.

Quoted 5 lines

This helps for sure - doesnt 100% solve it. It would really be nice if we could tie in a kerberos-like ticketing system for ownership of the mount or something even more fine grained like a link. Doesnt have to be kerberos but anything that would allow a digest of some verifiable credentials/token to be handed to the kernel for authorization...

Quoted 4 lines

What is your use-case, you don't want anyone except your own orchestration application to access it, so any kind of ACLs, LSM policies or making the mount only available to your container are not enough in this scenario you have in mind?

It should work - it's not even a shared environment (unlike the situation you have to deal with). I think i got overly paranoid because we have gone through a couple of debug cases where an installed parser (using ip) in XDP (with a tc prog consuming the results) was accidentally replaced (and the tc side had expectations built on the removed prog). i.e end goal is two or more programs, in this case, one running in XDP and another at TC are interdependent; if you touch one you affect the other.
In a shared environment it could be problematic because all you need is root access to remove things.
If you have a second factor authentication etc then someone has to be both root and has more secret knowledge to displace things.

But: I do have some ulterior motive where the authentication could be used at policy level as well. Example someone can read-only a tc rule whereas someone else can read, update or even delete the rule.

Quoted 2 lines

I think the closest to that is probably the prototype which Lorenz recently built where the user space application's digest is validated via IMA [0].

This may be sufficient for the atomicity requirement if we can lock things into our own ebpf fs. I will take a look - thanks.

cheers,
jamal

Quoted 2 lines

[0] http://vger.kernel.org/bpfconf2023_material/Lorenz_Bauer_-_BPF_signing_using_fsverity_and_LSM_gatekeeper.pdf
https://github.com/isovalent/bpf-verity

#40 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

In reply to Daniel Borkmann
Quoted 303 lines · Daniel Borkmann

This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment.

Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example:

- From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1]

- From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2]

BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware.

Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different.

We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient.

For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one.

For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective.

The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB.

Kudos also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work.

[0] https://lpc.events/event/16/contributions/1353/
[1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com/
[2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog
[3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
[4] https://lore.kernel.org/bpf/[email protected]/

---
 MAINTAINERS                    |   4 +-
 include/linux/netdevice.h      |  15 +-
 include/linux/skbuff.h         |   4 +-
 include/net/sch_generic.h      |   2 +-
 include/net/tcx.h              | 157 +++++++++++++++
 include/uapi/linux/bpf.h       |  35 +++-
 kernel/bpf/Kconfig             |   1 +
 kernel/bpf/Makefile            |   1 +
 kernel/bpf/syscall.c           |  95 +++++++--
 kernel/bpf/tcx.c               | 347 +++++++++++++++++++++++++++++++++
 net/Kconfig                    |   5 +
 net/core/dev.c                 | 267 +++++++++++++++----------
 net/core/filter.c              |   4 +-
 net/sched/Kconfig              |   4 +-
 net/sched/sch_ingress.c        |  45 ++++-
 tools/include/uapi/linux/bpf.h |  35 +++-
 16 files changed, 877 insertions(+), 144 deletions(-)
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/tcx.c
MAINTAINERS +3 -1
index 754a9eeca0a1..7a0d0b0c5a5e 100644
@@ -3827,13 +3827,15 @@ L:	[email protected]
 S:	Maintained
 F:	kernel/bpf/bpf_struct*
 
-BPF [NETWORKING] (tc BPF, sock_addr)
+BPF [NETWORKING] (tcx & tc BPF, sock_addr)
 M:	Martin KaFai Lau <[email protected]>
 M:	Daniel Borkmann <[email protected]>
 R:	John Fastabend <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
+F:	include/net/tcx.h
+F:	kernel/bpf/tcx.c
 F:	net/core/filter.c
 F:	net/sched/act_bpf.c
 F:	net/sched/cls_bpf.c
include/linux/netdevice.h +6 -9
index 08fbd4622ccf..fd4281d1cdbb 100644
@@ -1927,8 +1927,7 @@ enum netdev_ml_priv_type {
  *
  *	@rx_handler:		handler for received packets
  *	@rx_handler_data: 	XXX: need comments on this one
- *	@miniq_ingress:		ingress/clsact qdisc specific data for
- *				ingress processing
+ *	@tcx_ingress:		BPF & clsact qdisc specific data for ingress processing
  *	@ingress_queue:		XXX: need comments on this one
  *	@nf_hooks_ingress:	netfilter hooks executed for ingress packets
  *	@broadcast:		hw bcast address
@@ -1949,8 +1948,7 @@ enum netdev_ml_priv_type {
  *	@xps_maps:		all CPUs/RXQs maps for XPS device
  *
  *	@xps_maps:	XXX: need comments on this one
- *	@miniq_egress:		clsact qdisc specific data for
- *				egress processing
+ *	@tcx_egress:		BPF & clsact qdisc specific data for egress processing
  *	@nf_hooks_egress:	netfilter hooks executed for egress packets
  *	@qdisc_hash:		qdisc hash table
  *	@watchdog_timeo:	Represents the timeout that is used by
@@ -2249,9 +2247,8 @@ struct net_device {
 	unsigned int		gro_ipv4_max_size;
 	rx_handler_func_t __rcu	*rx_handler;
 	void __rcu		*rx_handler_data;
-
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_ingress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_ingress;
 #endif
 	struct netdev_queue __rcu *ingress_queue;
 #ifdef CONFIG_NETFILTER_INGRESS
@@ -2279,8 +2276,8 @@ struct net_device {
 #ifdef CONFIG_XPS
 	struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
 #endif
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_egress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_egress;
 #endif
 #ifdef CONFIG_NETFILTER_EGRESS
 	struct nf_hook_entries __rcu *nf_hooks_egress;
include/linux/skbuff.h +2 -2
index 5951904413ab..48c3e307f057 100644
@@ -943,7 +943,7 @@ struct sk_buff {
 	__u8			__mono_tc_offset[0];
 	/* public: */
 	__u8			mono_delivery_time:1;	/* See SKB_MONO_DELIVERY_TIME_MASK */
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	__u8			tc_at_ingress:1;	/* See TC_AT_INGRESS_MASK */
 	__u8			tc_skip_classify:1;
 #endif
@@ -992,7 +992,7 @@ struct sk_buff {
 	__u8			csum_not_inet:1;
 #endif
 
-#ifdef CONFIG_NET_SCHED
+#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
 	__u16			tc_index;	/* traffic control index */
 #endif
 
include/net/sch_generic.h +1 -1
index fab5ba3e61b7..0ade5d1a72b2 100644
@@ -695,7 +695,7 @@ int skb_do_redirect(struct sk_buff *);
 
 static inline bool skb_at_tc_ingress(const struct sk_buff *skb)
 {
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	return skb->tc_at_ingress;
 #else
 	return false;
include/net/tcx.h new file +71 -0
new file mode 100644
index 000000000000..27885ecedff9
@@ -0,0 +1,157 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef __NET_TCX_H
+#define __NET_TCX_H
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+
+#include <net/sch_generic.h>
+
+struct mini_Qdisc;
+
+struct tcx_entry {
+	struct bpf_mprog_bundle		bundle;
+	struct mini_Qdisc __rcu		*miniq;
+};
+
+struct tcx_link {
+	struct bpf_link link;
+	struct net_device *dev;
+	u32 location;
+	u32 flags;
+};
+
+static inline struct tcx_link *tcx_link(struct bpf_link *link)
+{
+	return container_of(link, struct tcx_link, link);
+}
+
+static inline const struct tcx_link *tcx_link_const(const struct bpf_link *link)
+{
+	return tcx_link((struct bpf_link *)link);
+}
+
+static inline void tcx_set_ingress(struct sk_buff *skb, bool ingress)
+{
+#ifdef CONFIG_NET_XGRESS
+	skb->tc_at_ingress = ingress;
+#endif
+}
+
+#ifdef CONFIG_NET_XGRESS
+void tcx_inc(void);
+void tcx_dec(void);
+
+static inline struct tcx_entry *tcx_entry(struct bpf_mprog_entry *entry)
+{
+	return container_of(entry->parent, struct tcx_entry, bundle);
+}
+
+static inline void
+tcx_entry_update(struct net_device *dev, struct bpf_mprog_entry *entry, bool ingress)
+{
+	ASSERT_RTNL();
+	if (ingress)
+		rcu_assign_pointer(dev->tcx_ingress, entry);
+	else
+		rcu_assign_pointer(dev->tcx_egress, entry);
+}
+
+static inline struct bpf_mprog_entry *
+dev_tcx_entry_fetch(struct net_device *dev, bool ingress)
+{
+	ASSERT_RTNL();
+	if (ingress)
+		return rcu_dereference_rtnl(dev->tcx_ingress);
+	else
+		return rcu_dereference_rtnl(dev->tcx_egress);
+}
+
+static inline struct bpf_mprog_entry *

[..]

Quoted 1 line

+dev_tcx_entry_fetch_or_create(struct net_device *dev, bool ingress, bool *created)

Regarding 'created' argument: any reason we are not doing conventional reference counting on bpf_mprog_entry? I wonder if there is a better way to hide those places where we handle BPF_MPROG_FREE explicitly.

Btw, thinking of this a/b arrays, should we call them active/inactive?

Quoted 1,344 lines
include/net/tcx.h new file +85 -0
+{
+	struct bpf_mprog_entry *entry = dev_tcx_entry_fetch(dev, ingress);
+
+	*created = false;
+	if (!entry) {
+		entry = bpf_mprog_create(sizeof_field(struct tcx_entry,
+						      miniq));
+		if (!entry)
+			return NULL;
+		*created = true;
+	}
+	return entry;
+}
+
+static inline void tcx_skeys_inc(bool ingress)
+{
+	tcx_inc();
+	if (ingress)
+		net_inc_ingress_queue();
+	else
+		net_inc_egress_queue();
+}
+
+static inline void tcx_skeys_dec(bool ingress)
+{
+	if (ingress)
+		net_dec_ingress_queue();
+	else
+		net_dec_egress_queue();
+	tcx_dec();
+}
+
+static inline enum tcx_action_base tcx_action_code(struct sk_buff *skb, int code)
+{
+	switch (code) {
+	case TCX_PASS:
+		skb->tc_index = qdisc_skb_cb(skb)->tc_classid;
+		fallthrough;
+	case TCX_DROP:
+	case TCX_REDIRECT:
+		return code;
+	case TCX_NEXT:
+	default:
+		return TCX_NEXT;
+	}
+}
+#endif /* CONFIG_NET_XGRESS */
+
+#if defined(CONFIG_NET_XGRESS) && defined(CONFIG_BPF_SYSCALL)
+int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog);
+int tcx_prog_query(const union bpf_attr *attr,
+		   union bpf_attr __user *uattr);
+void dev_tcx_uninstall(struct net_device *dev);
+#else
+static inline int tcx_prog_attach(const union bpf_attr *attr,
+				  struct bpf_prog *prog)
+{
+	return -EINVAL;
+}
+
+static inline int tcx_link_attach(const union bpf_attr *attr,
+				  struct bpf_prog *prog)
+{
+	return -EINVAL;
+}
+
+static inline int tcx_prog_detach(const union bpf_attr *attr,
+				  struct bpf_prog *prog)
+{
+	return -EINVAL;
+}
+
+static inline int tcx_prog_query(const union bpf_attr *attr,
+				 union bpf_attr __user *uattr)
+{
+	return -EINVAL;
+}
+
+static inline void dev_tcx_uninstall(struct net_device *dev)
+{
+}
+#endif /* CONFIG_NET_XGRESS && CONFIG_BPF_SYSCALL */
+#endif /* __NET_TCX_H */
include/uapi/linux/bpf.h +31 -4
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u32		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+			__u32 flags;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
kernel/bpf/Kconfig +1 -0
index 2dfe1079f772..6a906ff93006 100644
@@ -31,6 +31,7 @@ config BPF_SYSCALL
 	select TASKS_TRACE_RCU
 	select BINARY_PRINTF
 	select NET_SOCK_MSG if NET
+	select NET_XGRESS if NET
 	select PAGE_POOL if NET
 	default n
 	help
kernel/bpf/Makefile +1 -0
index 1bea2eb912cd..f526b7573e97 100644
@@ -21,6 +21,7 @@ obj-$(CONFIG_BPF_SYSCALL) += devmap.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
 obj-$(CONFIG_BPF_SYSCALL) += offload.o
 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
+obj-$(CONFIG_BPF_SYSCALL) += tcx.o
 endif
 ifeq ($(CONFIG_PERF_EVENTS),y)
 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/syscall.c +82 -13
index 92a57efc77de..e2c219d053f4 100644
@@ -37,6 +37,8 @@
 #include <linux/trace_events.h>
 #include <net/netfilter/nf_bpf_link.h>
 
+#include <net/tcx.h>
+
 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
 			  (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
 			  (map)->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS)
@@ -3522,31 +3524,57 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)
 		return BPF_PROG_TYPE_XDP;
 	case BPF_LSM_CGROUP:
 		return BPF_PROG_TYPE_LSM;
+	case BPF_TCX_INGRESS:
+	case BPF_TCX_EGRESS:
+		return BPF_PROG_TYPE_SCHED_CLS;
 	default:
 		return BPF_PROG_TYPE_UNSPEC;
 	}
 }
 
-#define BPF_PROG_ATTACH_LAST_FIELD replace_bpf_fd
+#define BPF_PROG_ATTACH_LAST_FIELD expected_revision
+
+#define BPF_F_ATTACH_MASK_BASE	\
+	(BPF_F_ALLOW_OVERRIDE |	\
+	 BPF_F_ALLOW_MULTI |	\
+	 BPF_F_REPLACE)
+
+#define BPF_F_ATTACH_MASK_MPROG	\
+	(BPF_F_REPLACE |	\
+	 BPF_F_BEFORE |		\
+	 BPF_F_AFTER |		\
+	 BPF_F_FIRST |		\
+	 BPF_F_LAST |		\
+	 BPF_F_ID |		\
+	 BPF_F_LINK)
 
-#define BPF_F_ATTACH_MASK \
-	(BPF_F_ALLOW_OVERRIDE | BPF_F_ALLOW_MULTI | BPF_F_REPLACE)
+static bool bpf_supports_mprog(enum bpf_prog_type ptype)
+{
+	switch (ptype) {
+	case BPF_PROG_TYPE_SCHED_CLS:
+		return true;
+	default:
+		return false;
+	}
+}
 
 static int bpf_prog_attach(const union bpf_attr *attr)
 {
 	enum bpf_prog_type ptype;
 	struct bpf_prog *prog;
+	u32 mask;
 	int ret;
 
 	if (CHECK_ATTR(BPF_PROG_ATTACH))
 		return -EINVAL;
 
-	if (attr->attach_flags & ~BPF_F_ATTACH_MASK)
-		return -EINVAL;
-
 	ptype = attach_type_to_prog_type(attr->attach_type);
 	if (ptype == BPF_PROG_TYPE_UNSPEC)
 		return -EINVAL;
+	mask = bpf_supports_mprog(ptype) ?
+	       BPF_F_ATTACH_MASK_MPROG : BPF_F_ATTACH_MASK_BASE;
+	if (attr->attach_flags & ~mask)
+		return -EINVAL;
 
 	prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
 	if (IS_ERR(prog))
@@ -3582,6 +3610,9 @@ static int bpf_prog_attach(const union bpf_attr *attr)
 		else
 			ret = cgroup_bpf_prog_attach(attr, ptype, prog);
 		break;
+	case BPF_PROG_TYPE_SCHED_CLS:
+		ret = tcx_prog_attach(attr, prog);
+		break;
 	default:
 		ret = -EINVAL;
 	}
@@ -3591,25 +3622,42 @@ static int bpf_prog_attach(const union bpf_attr *attr)
 	return ret;
 }
 
-#define BPF_PROG_DETACH_LAST_FIELD attach_type
+#define BPF_PROG_DETACH_LAST_FIELD expected_revision
 
 static int bpf_prog_detach(const union bpf_attr *attr)
 {
+	struct bpf_prog *prog = NULL;
 	enum bpf_prog_type ptype;
+	int ret;
 
 	if (CHECK_ATTR(BPF_PROG_DETACH))
 		return -EINVAL;
 
 	ptype = attach_type_to_prog_type(attr->attach_type);
+	if (bpf_supports_mprog(ptype)) {
+		if (ptype == BPF_PROG_TYPE_UNSPEC)
+			return -EINVAL;
+		if (attr->attach_flags & ~BPF_F_ATTACH_MASK_MPROG)
+			return -EINVAL;
+		prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
+		if (IS_ERR(prog)) {
+			if ((int)attr->attach_bpf_fd > 0)
+				return PTR_ERR(prog);
+			prog = NULL;
+		}
+	}
 
 	switch (ptype) {
 	case BPF_PROG_TYPE_SK_MSG:
 	case BPF_PROG_TYPE_SK_SKB:
-		return sock_map_prog_detach(attr, ptype);
+		ret = sock_map_prog_detach(attr, ptype);
+		break;
 	case BPF_PROG_TYPE_LIRC_MODE2:
-		return lirc_prog_detach(attr);
+		ret = lirc_prog_detach(attr);
+		break;
 	case BPF_PROG_TYPE_FLOW_DISSECTOR:
-		return netns_bpf_prog_detach(attr, ptype);
+		ret = netns_bpf_prog_detach(attr, ptype);
+		break;
 	case BPF_PROG_TYPE_CGROUP_DEVICE:
 	case BPF_PROG_TYPE_CGROUP_SKB:
 	case BPF_PROG_TYPE_CGROUP_SOCK:
@@ -3618,13 +3666,21 @@ static int bpf_prog_detach(const union bpf_attr *attr)
 	case BPF_PROG_TYPE_CGROUP_SYSCTL:
 	case BPF_PROG_TYPE_SOCK_OPS:
 	case BPF_PROG_TYPE_LSM:
-		return cgroup_bpf_prog_detach(attr, ptype);
+		ret = cgroup_bpf_prog_detach(attr, ptype);
+		break;
+	case BPF_PROG_TYPE_SCHED_CLS:
+		ret = tcx_prog_detach(attr, prog);
+		break;
 	default:
-		return -EINVAL;
+		ret = -EINVAL;
 	}
+
+	if (prog)
+		bpf_prog_put(prog);
+	return ret;
 }
 
-#define BPF_PROG_QUERY_LAST_FIELD query.prog_attach_flags
+#define BPF_PROG_QUERY_LAST_FIELD query.link_attach_flags
 
 static int bpf_prog_query(const union bpf_attr *attr,
 			  union bpf_attr __user *uattr)
@@ -3672,6 +3728,9 @@ static int bpf_prog_query(const union bpf_attr *attr,
 	case BPF_SK_MSG_VERDICT:
 	case BPF_SK_SKB_VERDICT:
 		return sock_map_bpf_prog_query(attr, uattr);
+	case BPF_TCX_INGRESS:
+	case BPF_TCX_EGRESS:
+		return tcx_prog_query(attr, uattr);
 	default:
 		return -EINVAL;
 	}
@@ -4629,6 +4688,13 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 			goto out;
 		}
 		break;
+	case BPF_PROG_TYPE_SCHED_CLS:
+		if (attr->link_create.attach_type != BPF_TCX_INGRESS &&
+		    attr->link_create.attach_type != BPF_TCX_EGRESS) {
+			ret = -EINVAL;
+			goto out;
+		}
+		break;
 	default:
 		ptype = attach_type_to_prog_type(attr->link_create.attach_type);
 		if (ptype == BPF_PROG_TYPE_UNSPEC || ptype != prog->type) {
@@ -4680,6 +4746,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 	case BPF_PROG_TYPE_XDP:
 		ret = bpf_xdp_link_attach(attr, prog);
 		break;
+	case BPF_PROG_TYPE_SCHED_CLS:
+		ret = tcx_link_attach(attr, prog);
+		break;
 	case BPF_PROG_TYPE_NETFILTER:
 		ret = bpf_nf_link_attach(attr, prog);
 		break;
kernel/bpf/tcx.c new file +347 -0
new file mode 100644
index 000000000000..d3d23b4ed4f0
@@ -0,0 +1,347 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+#include <linux/netdevice.h>
+
+#include <net/tcx.h>
+
+int tcx_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	bool created, ingress = attr->attach_type == BPF_TCX_INGRESS;
+	struct net *net = current->nsproxy->net_ns;
+	struct bpf_mprog_entry *entry;
+	struct net_device *dev;
+	int ret;
+
+	rtnl_lock();
+	dev = __dev_get_by_index(net, attr->target_ifindex);
+	if (!dev) {
+		ret = -ENODEV;
+		goto out;
+	}
+	entry = dev_tcx_entry_fetch_or_create(dev, ingress, &created);
+	if (!entry) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	ret = bpf_mprog_attach(entry, prog, NULL, attr->attach_flags,
+			       attr->relative_fd, attr->expected_revision);
+	if (ret >= 0) {
+		if (ret == BPF_MPROG_SWAP)
+			tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+		bpf_mprog_commit(entry);
+		tcx_skeys_inc(ingress);
+		ret = 0;
+	} else if (created) {
+		bpf_mprog_free(entry);
+	}
+out:
+	rtnl_unlock();
+	return ret;
+}
+
+static bool tcx_release_entry(struct bpf_mprog_entry *entry, int code)
+{
+	return code == BPF_MPROG_FREE && !tcx_entry(entry)->miniq;
+}
+
+int tcx_prog_detach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	bool tcx_release, ingress = attr->attach_type == BPF_TCX_INGRESS;
+	struct net *net = current->nsproxy->net_ns;
+	struct bpf_mprog_entry *entry, *peer;
+	struct net_device *dev;
+	int ret;
+
+	rtnl_lock();
+	dev = __dev_get_by_index(net, attr->target_ifindex);
+	if (!dev) {
+		ret = -ENODEV;
+		goto out;
+	}
+	entry = dev_tcx_entry_fetch(dev, ingress);
+	if (!entry) {
+		ret = -ENOENT;
+		goto out;
+	}
+	ret = bpf_mprog_detach(entry, prog, NULL, attr->attach_flags,
+			       attr->relative_fd, attr->expected_revision);
+	if (ret >= 0) {
+		tcx_release = tcx_release_entry(entry, ret);
+		peer = tcx_release ? NULL : bpf_mprog_peer(entry);
+		if (ret == BPF_MPROG_SWAP || ret == BPF_MPROG_FREE)
+			tcx_entry_update(dev, peer, ingress);
+		bpf_mprog_commit(entry);
+		tcx_skeys_dec(ingress);
+		if (tcx_release)
+			bpf_mprog_free(entry);
+		ret = 0;
+	}
+out:
+	rtnl_unlock();
+	return ret;
+}
+
+static void tcx_uninstall(struct net_device *dev, bool ingress)
+{
+	struct bpf_tuple tuple = {};
+	struct bpf_mprog_entry *entry;
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+
+	entry = dev_tcx_entry_fetch(dev, ingress);
+	if (!entry)
+		return;
+	tcx_entry_update(dev, NULL, ingress);
+	bpf_mprog_commit(entry);
+	bpf_mprog_foreach_tuple(entry, fp, cp, tuple) {
+		if (tuple.link)
+			tcx_link(tuple.link)->dev = NULL;
+		else
+			bpf_prog_put(tuple.prog);
+		tcx_skeys_dec(ingress);
+	}
+	WARN_ON_ONCE(tcx_entry(entry)->miniq);
+	bpf_mprog_free(entry);
+}
+
+void dev_tcx_uninstall(struct net_device *dev)
+{
+	ASSERT_RTNL();
+	tcx_uninstall(dev, true);
+	tcx_uninstall(dev, false);
+}
+
+int tcx_prog_query(const union bpf_attr *attr, union bpf_attr __user *uattr)
+{
+	bool ingress = attr->query.attach_type == BPF_TCX_INGRESS;
+	struct net *net = current->nsproxy->net_ns;
+	struct bpf_mprog_entry *entry;
+	struct net_device *dev;
+	int ret;
+
+	rtnl_lock();
+	dev = __dev_get_by_index(net, attr->query.target_ifindex);
+	if (!dev) {
+		ret = -ENODEV;
+		goto out;
+	}
+	entry = dev_tcx_entry_fetch(dev, ingress);
+	if (!entry) {
+		ret = -ENOENT;
+		goto out;
+	}
+	ret = bpf_mprog_query(attr, uattr, entry);
+out:
+	rtnl_unlock();
+	return ret;
+}
+
+static int tcx_link_prog_attach(struct bpf_link *l, u32 flags, u32 object,
+				u32 expected_revision)
+{
+	struct tcx_link *link = tcx_link(l);
+	bool created, ingress = link->location == BPF_TCX_INGRESS;
+	struct net_device *dev = link->dev;
+	struct bpf_mprog_entry *entry;
+	int ret;
+
+	ASSERT_RTNL();
+	entry = dev_tcx_entry_fetch_or_create(dev, ingress, &created);
+	if (!entry)
+		return -ENOMEM;
+	ret = bpf_mprog_attach(entry, l->prog, l, flags, object,
+			       expected_revision);
+	if (ret >= 0) {
+		if (ret == BPF_MPROG_SWAP)
+			tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+		bpf_mprog_commit(entry);
+		tcx_skeys_inc(ingress);
+		ret = 0;
+	} else if (created) {
+		bpf_mprog_free(entry);
+	}
+	return ret;
+}
+
+static void tcx_link_release(struct bpf_link *l)
+{
+	struct tcx_link *link = tcx_link(l);
+	bool tcx_release, ingress = link->location == BPF_TCX_INGRESS;
+	struct bpf_mprog_entry *entry, *peer;
+	struct net_device *dev;
+	int ret = 0;
+
+	rtnl_lock();
+	dev = link->dev;
+	if (!dev)
+		goto out;
+	entry = dev_tcx_entry_fetch(dev, ingress);
+	if (!entry) {
+		ret = -ENOENT;
+		goto out;
+	}
+	ret = bpf_mprog_detach(entry, l->prog, l, link->flags, 0, 0);
+	if (ret >= 0) {
+		tcx_release = tcx_release_entry(entry, ret);
+		peer = tcx_release ? NULL : bpf_mprog_peer(entry);
+		if (ret == BPF_MPROG_SWAP || ret == BPF_MPROG_FREE)
+			tcx_entry_update(dev, peer, ingress);
+		bpf_mprog_commit(entry);
+		tcx_skeys_dec(ingress);
+		if (tcx_release)
+			bpf_mprog_free(entry);
+		link->dev = NULL;
+		ret = 0;
+	}
+out:
+	WARN_ON_ONCE(ret);
+	rtnl_unlock();
+}
+
+static int tcx_link_update(struct bpf_link *l, struct bpf_prog *nprog,
+			   struct bpf_prog *oprog)
+{
+	struct tcx_link *link = tcx_link(l);
+	bool ingress = link->location == BPF_TCX_INGRESS;
+	struct net_device *dev = link->dev;
+	struct bpf_mprog_entry *entry;
+	int ret = 0;
+
+	rtnl_lock();
+	if (!link->dev) {
+		ret = -ENOLINK;
+		goto out;
+	}
+	if (oprog && l->prog != oprog) {
+		ret = -EPERM;
+		goto out;
+	}
+	oprog = l->prog;
+	if (oprog == nprog) {
+		bpf_prog_put(nprog);
+		goto out;
+	}
+	entry = dev_tcx_entry_fetch(dev, ingress);
+	if (!entry) {
+		ret = -ENOENT;
+		goto out;
+	}
+	ret = bpf_mprog_attach(entry, nprog, l,
+			       BPF_F_REPLACE | BPF_F_ID | link->flags,
+			       l->prog->aux->id, 0);
+	if (ret >= 0) {
+		if (ret == BPF_MPROG_SWAP)
+			tcx_entry_update(dev, bpf_mprog_peer(entry), ingress);
+		bpf_mprog_commit(entry);
+		tcx_skeys_inc(ingress);
+		oprog = xchg(&l->prog, nprog);
+		bpf_prog_put(oprog);
+		ret = 0;
+	}
+out:
+	rtnl_unlock();
+	return ret;
+}
+
+static void tcx_link_dealloc(struct bpf_link *l)
+{
+	kfree(tcx_link(l));
+}
+
+static void tcx_link_fdinfo(const struct bpf_link *l, struct seq_file *seq)
+{
+	const struct tcx_link *link = tcx_link_const(l);
+	u32 ifindex = 0;
+
+	rtnl_lock();
+	if (link->dev)
+		ifindex = link->dev->ifindex;
+	rtnl_unlock();
+
+	seq_printf(seq, "ifindex:\t%u\n", ifindex);
+	seq_printf(seq, "attach_type:\t%u (%s)\n",
+		   link->location,
+		   link->location == BPF_TCX_INGRESS ? "ingress" : "egress");
+	seq_printf(seq, "flags:\t%u\n", link->flags);
+}
+
+static int tcx_link_fill_info(const struct bpf_link *l,
+			      struct bpf_link_info *info)
+{
+	const struct tcx_link *link = tcx_link_const(l);
+	u32 ifindex = 0;
+
+	rtnl_lock();
+	if (link->dev)
+		ifindex = link->dev->ifindex;
+	rtnl_unlock();
+
+	info->tcx.ifindex = ifindex;
+	info->tcx.attach_type = link->location;
+	info->tcx.flags = link->flags;
+	return 0;
+}
+
+static int tcx_link_detach(struct bpf_link *l)
+{
+	tcx_link_release(l);
+	return 0;
+}
+
+static const struct bpf_link_ops tcx_link_lops = {
+	.release	= tcx_link_release,
+	.detach		= tcx_link_detach,
+	.dealloc	= tcx_link_dealloc,
+	.update_prog	= tcx_link_update,
+	.show_fdinfo	= tcx_link_fdinfo,
+	.fill_link_info	= tcx_link_fill_info,
+};
+
+int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	struct net *net = current->nsproxy->net_ns;
+	struct bpf_link_primer link_primer;
+	struct net_device *dev;
+	struct tcx_link *link;
+	int fd, err;
+
+	dev = dev_get_by_index(net, attr->link_create.target_ifindex);
+	if (!dev)
+		return -EINVAL;
+	link = kzalloc(sizeof(*link), GFP_USER);
+	if (!link) {
+		err = -ENOMEM;
+		goto out_put;
+	}
+
+	bpf_link_init(&link->link, BPF_LINK_TYPE_TCX, &tcx_link_lops, prog);
+	link->location = attr->link_create.attach_type;
+	link->flags = attr->link_create.flags & (BPF_F_FIRST | BPF_F_LAST);
+	link->dev = dev;
+
+	err = bpf_link_prime(&link->link, &link_primer);
+	if (err) {
+		kfree(link);
+		goto out_put;
+	}
+	rtnl_lock();
+	err = tcx_link_prog_attach(&link->link, attr->link_create.flags,
+				   attr->link_create.tcx.relative_fd,
+				   attr->link_create.tcx.expected_revision);
+	if (!err)
+		fd = bpf_link_settle(&link_primer);
+	rtnl_unlock();
+	if (err) {
+		link->dev = NULL;
+		bpf_link_cleanup(&link_primer);
+		goto out_put;
+	}
+	dev_put(dev);
+	return fd;
+out_put:
+	dev_put(dev);
+	return err;
+}
net/Kconfig +5 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
 config NET_EGRESS
 	bool
 
+config NET_XGRESS
+	select NET_INGRESS
+	select NET_EGRESS
+	bool
+
 config NET_REDIRECT
 	bool
 
net/core/dev.c +165 -102
index 3393c2f3dbe8..95c7e3189884 100644
@@ -107,6 +107,7 @@
 #include <net/pkt_cls.h>
 #include <net/checksum.h>
 #include <net/xfrm.h>
+#include <net/tcx.h>
 #include <linux/highmem.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -154,7 +155,6 @@
 #include "dev.h"
 #include "net-sysfs.h"
 
-
 static DEFINE_SPINLOCK(ptype_lock);
 struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
 struct list_head ptype_all __read_mostly;	/* Taps */
@@ -3923,69 +3923,200 @@ int dev_loopback_xmit(struct net *net, struct sock *sk, struct sk_buff *skb)
 EXPORT_SYMBOL(dev_loopback_xmit);
 
 #ifdef CONFIG_NET_EGRESS
-static struct sk_buff *
-sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
+static struct netdev_queue *
+netdev_tx_queue_mapping(struct net_device *dev, struct sk_buff *skb)
+{
+	int qm = skb_get_queue_mapping(skb);
+
+	return netdev_get_tx_queue(dev, netdev_cap_txqueue(dev, qm));
+}
+
+static bool netdev_xmit_txqueue_skipped(void)
 {
+	return __this_cpu_read(softnet_data.xmit.skip_txqueue);
+}
+
+void netdev_xmit_skip_txqueue(bool skip)
+{
+	__this_cpu_write(softnet_data.xmit.skip_txqueue, skip);
+}
+EXPORT_SYMBOL_GPL(netdev_xmit_skip_txqueue);
+#endif /* CONFIG_NET_EGRESS */
+
+#ifdef CONFIG_NET_XGRESS
+static int tc_run(struct tcx_entry *entry, struct sk_buff *skb)
+{
+	int ret = TC_ACT_UNSPEC;
 #ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc *miniq = rcu_dereference_bh(dev->miniq_egress);
-	struct tcf_result cl_res;
+	struct mini_Qdisc *miniq = rcu_dereference_bh(entry->miniq);
+	struct tcf_result res;
 
 	if (!miniq)
-		return skb;
+		return ret;
 
-	/* qdisc_skb_cb(skb)->pkt_len was already set by the caller. */
 	tc_skb_cb(skb)->mru = 0;
 	tc_skb_cb(skb)->post_ct = false;
-	mini_qdisc_bstats_cpu_update(miniq, skb);
 
-	switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
+	mini_qdisc_bstats_cpu_update(miniq, skb);
+	ret = tcf_classify(skb, miniq->block, miniq->filter_list, &res, false);
+	/* Only tcf related quirks below. */
+	switch (ret) {
+	case TC_ACT_SHOT:
+		mini_qdisc_qstats_cpu_drop(miniq);
+		break;
 	case TC_ACT_OK:
 	case TC_ACT_RECLASSIFY:
-		skb->tc_index = TC_H_MIN(cl_res.classid);
+		skb->tc_index = TC_H_MIN(res.classid);
 		break;
+	}
+#endif /* CONFIG_NET_CLS_ACT */
+	return ret;
+}
+
+static DEFINE_STATIC_KEY_FALSE(tcx_needed_key);
+
+void tcx_inc(void)
+{
+	static_branch_inc(&tcx_needed_key);
+}
+EXPORT_SYMBOL_GPL(tcx_inc);
+
+void tcx_dec(void)
+{
+	static_branch_dec(&tcx_needed_key);
+}
+EXPORT_SYMBOL_GPL(tcx_dec);
+
+static __always_inline enum tcx_action_base
+tcx_run(const struct bpf_mprog_entry *entry, struct sk_buff *skb,
+	const bool needs_mac)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *prog;
+	int ret = TCX_NEXT;
+
+	if (needs_mac)
+		__skb_push(skb, skb->mac_len);
+	bpf_mprog_foreach_prog(entry, fp, prog) {
+		bpf_compute_data_pointers(skb);
+		ret = bpf_prog_run(prog, skb);
+		if (ret != TCX_NEXT)
+			break;
+	}
+	if (needs_mac)
+		__skb_pull(skb, skb->mac_len);
+	return tcx_action_code(skb, ret);
+}
+
+static __always_inline struct sk_buff *
+sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
+		   struct net_device *orig_dev, bool *another)
+{
+	struct bpf_mprog_entry *entry = rcu_dereference_bh(skb->dev->tcx_ingress);
+	int sch_ret;
+
+	if (!entry)
+		return skb;
+	if (*pt_prev) {
+		*ret = deliver_skb(skb, *pt_prev, orig_dev);
+		*pt_prev = NULL;
+	}
+
+	qdisc_skb_cb(skb)->pkt_len = skb->len;
+	tcx_set_ingress(skb, true);
+
+	if (static_branch_unlikely(&tcx_needed_key)) {
+		sch_ret = tcx_run(entry, skb, true);
+		if (sch_ret != TC_ACT_UNSPEC)
+			goto ingress_verdict;
+	}
+	sch_ret = tc_run(container_of(entry->parent, struct tcx_entry, bundle), skb);
+ingress_verdict:
+	switch (sch_ret) {
+	case TC_ACT_REDIRECT:
+		/* skb_mac_header check was done by BPF, so we can safely
+		 * push the L2 header back before redirecting to another
+		 * netdev.
+		 */
+		__skb_push(skb, skb->mac_len);
+		if (skb_do_redirect(skb) == -EAGAIN) {
+			__skb_pull(skb, skb->mac_len);
+			*another = true;
+			break;
+		}
+		*ret = NET_RX_SUCCESS;
+		return NULL;
 	case TC_ACT_SHOT:
-		mini_qdisc_qstats_cpu_drop(miniq);
-		*ret = NET_XMIT_DROP;
-		kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+		kfree_skb_reason(skb, SKB_DROP_REASON_TC_INGRESS);
+		*ret = NET_RX_DROP;
 		return NULL;
+	/* used by tc_run */
 	case TC_ACT_STOLEN:
 	case TC_ACT_QUEUED:
 	case TC_ACT_TRAP:
-		*ret = NET_XMIT_SUCCESS;
 		consume_skb(skb);
+		fallthrough;
+	case TC_ACT_CONSUMED:
+		*ret = NET_RX_SUCCESS;
 		return NULL;
+	}
+
+	return skb;
+}
+
+static __always_inline struct sk_buff *
+sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
+{
+	struct bpf_mprog_entry *entry = rcu_dereference_bh(dev->tcx_egress);
+	int sch_ret;
+
+	if (!entry)
+		return skb;
+
+	/* qdisc_skb_cb(skb)->pkt_len & tcx_set_ingress() was
+	 * already set by the caller.
+	 */
+	if (static_branch_unlikely(&tcx_needed_key)) {
+		sch_ret = tcx_run(entry, skb, false);
+		if (sch_ret != TC_ACT_UNSPEC)
+			goto egress_verdict;
+	}
+	sch_ret = tc_run(container_of(entry->parent, struct tcx_entry, bundle), skb);
+egress_verdict:
+	switch (sch_ret) {
 	case TC_ACT_REDIRECT:
 		/* No need to push/pop skb's mac_header here on egress! */
 		skb_do_redirect(skb);
 		*ret = NET_XMIT_SUCCESS;
 		return NULL;
-	default:
-		break;
+	case TC_ACT_SHOT:
+		kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+		*ret = NET_XMIT_DROP;
+		return NULL;
+	/* used by tc_run */
+	case TC_ACT_STOLEN:
+	case TC_ACT_QUEUED:
+	case TC_ACT_TRAP:
+		*ret = NET_XMIT_SUCCESS;
+		return NULL;
 	}
-#endif /* CONFIG_NET_CLS_ACT */
 
 	return skb;
 }
-
-static struct netdev_queue *
-netdev_tx_queue_mapping(struct net_device *dev, struct sk_buff *skb)
-{
-	int qm = skb_get_queue_mapping(skb);
-
-	return netdev_get_tx_queue(dev, netdev_cap_txqueue(dev, qm));
-}
-
-static bool netdev_xmit_txqueue_skipped(void)
+#else
+static __always_inline struct sk_buff *
+sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
+		   struct net_device *orig_dev, bool *another)
 {
-	return __this_cpu_read(softnet_data.xmit.skip_txqueue);
+	return skb;
 }
 
-void netdev_xmit_skip_txqueue(bool skip)
+static __always_inline struct sk_buff *
+sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
 {
-	__this_cpu_write(softnet_data.xmit.skip_txqueue, skip);
+	return skb;
 }
-EXPORT_SYMBOL_GPL(netdev_xmit_skip_txqueue);
-#endif /* CONFIG_NET_EGRESS */
+#endif /* CONFIG_NET_XGRESS */
 
 #ifdef CONFIG_XPS
 static int __get_xps_queue_idx(struct net_device *dev, struct sk_buff *skb,
@@ -4169,9 +4300,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
 	skb_update_prio(skb);
 
 	qdisc_pkt_len_init(skb);
-#ifdef CONFIG_NET_CLS_ACT
-	skb->tc_at_ingress = 0;
-#endif
+	tcx_set_ingress(skb, false);
 #ifdef CONFIG_NET_EGRESS
 	if (static_branch_unlikely(&egress_needed_key)) {
 		if (nf_hook_egress_active()) {
@@ -5103,72 +5232,6 @@ int (*br_fdb_test_addr_hook)(struct net_device *dev,
 EXPORT_SYMBOL_GPL(br_fdb_test_addr_hook);
 #endif
 
-static inline struct sk_buff *
-sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
-		   struct net_device *orig_dev, bool *another)
-{
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc *miniq = rcu_dereference_bh(skb->dev->miniq_ingress);
-	struct tcf_result cl_res;
-
-	/* If there's at least one ingress present somewhere (so
-	 * we get here via enabled static key), remaining devices
-	 * that are not configured with an ingress qdisc will bail
-	 * out here.
-	 */
-	if (!miniq)
-		return skb;
-
-	if (*pt_prev) {
-		*ret = deliver_skb(skb, *pt_prev, orig_dev);
-		*pt_prev = NULL;
-	}
-
-	qdisc_skb_cb(skb)->pkt_len = skb->len;
-	tc_skb_cb(skb)->mru = 0;
-	tc_skb_cb(skb)->post_ct = false;
-	skb->tc_at_ingress = 1;
-	mini_qdisc_bstats_cpu_update(miniq, skb);
-
-	switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
-	case TC_ACT_OK:
-	case TC_ACT_RECLASSIFY:
-		skb->tc_index = TC_H_MIN(cl_res.classid);
-		break;
-	case TC_ACT_SHOT:
-		mini_qdisc_qstats_cpu_drop(miniq);
-		kfree_skb_reason(skb, SKB_DROP_REASON_TC_INGRESS);
-		*ret = NET_RX_DROP;
-		return NULL;
-	case TC_ACT_STOLEN:
-	case TC_ACT_QUEUED:
-	case TC_ACT_TRAP:
-		consume_skb(skb);
-		*ret = NET_RX_SUCCESS;
-		return NULL;
-	case TC_ACT_REDIRECT:
-		/* skb_mac_header check was done by cls/act_bpf, so
-		 * we can safely push the L2 header back before
-		 * redirecting to another netdev
-		 */
-		__skb_push(skb, skb->mac_len);
-		if (skb_do_redirect(skb) == -EAGAIN) {
-			__skb_pull(skb, skb->mac_len);
-			*another = true;
-			break;
-		}
-		*ret = NET_RX_SUCCESS;
-		return NULL;
-	case TC_ACT_CONSUMED:
-		*ret = NET_RX_SUCCESS;
-		return NULL;
-	default:
-		break;
-	}
-#endif /* CONFIG_NET_CLS_ACT */
-	return skb;
-}
-
 /**
  *	netdev_is_rx_handler_busy - check if receive handler is registered
  *	@dev: device to check
@@ -10873,7 +10936,7 @@ void unregister_netdevice_many_notify(struct list_head *head,
 
 		/* Shutdown queueing discipline. */
 		dev_shutdown(dev);
-
+		dev_tcx_uninstall(dev);
 		dev_xdp_uninstall(dev);
 		bpf_dev_bound_netdev_unregister(dev);
 
net/core/filter.c +2 -2
index d25d52854c21..1ff9a0988ea6 100644
@@ -9233,7 +9233,7 @@ static struct bpf_insn *bpf_convert_tstamp_read(const struct bpf_prog *prog,
 	__u8 value_reg = si->dst_reg;
 	__u8 skb_reg = si->src_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, read skb->tstamp as is if tstamp_type_access is true.
@@ -9267,7 +9267,7 @@ static struct bpf_insn *bpf_convert_tstamp_write(const struct bpf_prog *prog,
 	__u8 value_reg = si->src_reg;
 	__u8 skb_reg = si->dst_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, write skb->tstamp as is if tstamp_type_access is true.
net/sched/Kconfig +2 -2
index 4b95cb1ac435..470c70deffe2 100644
@@ -347,8 +347,7 @@ config NET_SCH_FQ_PIE
 config NET_SCH_INGRESS
 	tristate "Ingress/classifier-action Qdisc"
 	depends on NET_CLS_ACT
-	select NET_INGRESS
-	select NET_EGRESS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use classifiers for incoming and/or outgoing
 	  packets. This qdisc doesn't do anything else besides running classifiers,
@@ -679,6 +678,7 @@ config NET_EMATCH_IPT
 config NET_CLS_ACT
 	bool "Actions"
 	select NET_CLS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use traffic control actions. Actions
 	  get attached to classifiers and are invoked after a successful
net/sched/sch_ingress.c +41 -4
index 84838128b9c5..4af1360f537e 100644
@@ -13,6 +13,7 @@
 #include <net/netlink.h>
 #include <net/pkt_sched.h>
 #include <net/pkt_cls.h>
+#include <net/tcx.h>
 
 struct ingress_sched_data {
 	struct tcf_block *block;
@@ -78,11 +79,18 @@ static int ingress_init(struct Qdisc *sch, struct nlattr *opt,
 {
 	struct ingress_sched_data *q = qdisc_priv(sch);
 	struct net_device *dev = qdisc_dev(sch);
+	struct bpf_mprog_entry *entry;
+	bool created;
 	int err;
 
 	net_inc_ingress_queue();
 
-	mini_qdisc_pair_init(&q->miniqp, sch, &dev->miniq_ingress);
+	entry = dev_tcx_entry_fetch_or_create(dev, true, &created);
+	if (!entry)
+		return -ENOMEM;
+	mini_qdisc_pair_init(&q->miniqp, sch, &tcx_entry(entry)->miniq);
+	if (created)
+		tcx_entry_update(dev, entry, true);
 
 	q->block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
 	q->block_info.chain_head_change = clsact_chain_head_change;
@@ -93,15 +101,20 @@ static int ingress_init(struct Qdisc *sch, struct nlattr *opt,
 		return err;
 
 	mini_qdisc_pair_block_init(&q->miniqp, q->block);
-
 	return 0;
 }
 
 static void ingress_destroy(struct Qdisc *sch)
 {
 	struct ingress_sched_data *q = qdisc_priv(sch);
+	struct net_device *dev = qdisc_dev(sch);
+	struct bpf_mprog_entry *entry = rtnl_dereference(dev->tcx_ingress);
 
 	tcf_block_put_ext(q->block, sch, &q->block_info);
+	if (entry && !bpf_mprog_total(entry)) {
+		tcx_entry_update(dev, NULL, true);
+		bpf_mprog_free(entry);
+	}
 	net_dec_ingress_queue();
 }
 
@@ -217,12 +230,19 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 {
 	struct clsact_sched_data *q = qdisc_priv(sch);
 	struct net_device *dev = qdisc_dev(sch);
+	struct bpf_mprog_entry *entry;
+	bool created;
 	int err;
 
 	net_inc_ingress_queue();
 	net_inc_egress_queue();
 
-	mini_qdisc_pair_init(&q->miniqp_ingress, sch, &dev->miniq_ingress);
+	entry = dev_tcx_entry_fetch_or_create(dev, true, &created);
+	if (!entry)
+		return -ENOMEM;
+	mini_qdisc_pair_init(&q->miniqp_ingress, sch, &tcx_entry(entry)->miniq);
+	if (created)
+		tcx_entry_update(dev, entry, true);
 
 	q->ingress_block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
 	q->ingress_block_info.chain_head_change = clsact_chain_head_change;
@@ -235,7 +255,12 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 
 	mini_qdisc_pair_block_init(&q->miniqp_ingress, q->ingress_block);
 
-	mini_qdisc_pair_init(&q->miniqp_egress, sch, &dev->miniq_egress);
+	entry = dev_tcx_entry_fetch_or_create(dev, false, &created);
+	if (!entry)
+		return -ENOMEM;
+	mini_qdisc_pair_init(&q->miniqp_egress, sch, &tcx_entry(entry)->miniq);
+	if (created)
+		tcx_entry_update(dev, entry, false);
 
 	q->egress_block_info.binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_EGRESS;
 	q->egress_block_info.chain_head_change = clsact_chain_head_change;
@@ -247,9 +272,21 @@ static int clsact_init(struct Qdisc *sch, struct nlattr *opt,
 static void clsact_destroy(struct Qdisc *sch)
 {
 	struct clsact_sched_data *q = qdisc_priv(sch);
+	struct net_device *dev = qdisc_dev(sch);
+	struct bpf_mprog_entry *ingress_entry = rtnl_dereference(dev->tcx_ingress);
+	struct bpf_mprog_entry *egress_entry = rtnl_dereference(dev->tcx_egress);
 
 	tcf_block_put_ext(q->egress_block, sch, &q->egress_block_info);
+	if (egress_entry && !bpf_mprog_total(egress_entry)) {
+		tcx_entry_update(dev, NULL, false);
+		bpf_mprog_free(egress_entry);
+	}
+
 	tcf_block_put_ext(q->ingress_block, sch, &q->ingress_block_info);
+	if (ingress_entry && !bpf_mprog_total(ingress_entry)) {
+		tcx_entry_update(dev, NULL, true);
+		bpf_mprog_free(ingress_entry);
+	}
 
 	net_dec_ingress_queue();
 	net_dec_egress_queue();
tools/include/uapi/linux/bpf.h +31 -4
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u32		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,11 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+			__u32 flags;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
Signature · 1 line
2.34.1
#41 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

In reply to Daniel Borkmann
Quoted 124 lines · Daniel Borkmann

This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment.

Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example:

- From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1]

- From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2]

BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware.

Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different.

We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient.

For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one.

For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective.

The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer. tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail. The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB.

Kudos also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work.

[0] https://lpc.events/event/16/contributions/1353/
[1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com/
[2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog
[3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
[4] https://lore.kernel.org/bpf/[email protected]/

---
 MAINTAINERS                    |   4 +-
 include/linux/netdevice.h      |  15 +-
 include/linux/skbuff.h         |   4 +-
 include/net/sch_generic.h      |   2 +-
 include/net/tcx.h              | 157 +++++++++++++++
 include/uapi/linux/bpf.h       |  35 +++-
 kernel/bpf/Kconfig             |   1 +
 kernel/bpf/Makefile            |   1 +
 kernel/bpf/syscall.c           |  95 +++++++--
 kernel/bpf/tcx.c               | 347 +++++++++++++++++++++++++++++++++
 net/Kconfig                    |   5 +
 net/core/dev.c                 | 267 +++++++++++++++----------
 net/core/filter.c              |   4 +-
 net/sched/Kconfig              |   4 +-
 net/sched/sch_ingress.c        |  45 ++++-
 tools/include/uapi/linux/bpf.h |  35 +++-
 16 files changed, 877 insertions(+), 144 deletions(-)
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/tcx.c

[...]

Quoted 26 lines
include/uapi/linux/bpf.h +3 -1
index 207f8a37b327..e7584e24bc83 100644
@@ -1035,6 +1035,8 @@ enum bpf_attach_type {
        BPF_TRACE_KPROBE_MULTI,
        BPF_LSM_CGROUP,
        BPF_STRUCT_OPS,
+       BPF_TCX_INGRESS,
+       BPF_TCX_EGRESS,
        __MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,7 +1054,7 @@ enum bpf_link_type {
        BPF_LINK_TYPE_KPROBE_MULTI = 8,
        BPF_LINK_TYPE_STRUCT_OPS = 9,
        BPF_LINK_TYPE_NETFILTER = 10,
-
+       BPF_LINK_TYPE_TCX = 11,
        MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
                        __u32           map_fd;         /* struct_ops to attach */
                };
                union {

include/uapi/linux/bpf.h union bpf_attr { · line 1570

Quoted 10 lines
include/uapi/linux/bpf.h +3 -3
-                       __u32           target_fd;      /* object to attach to */
-                       __u32           target_ifindex; /* target ifindex */
+                       __u32   target_fd;      /* target object to attach to or ... */
+                       __u32   target_ifindex; /* target ifindex */
                };
                __u32           attach_type;    /* attach type */
                __u32           flags;          /* extra flags */
                union {
-                       __u32           target_btf_id;  /* btf_id of target to attach to */
+                       __u32   target_btf_id;  /* btf_id of target to attach to */

nit: should this part be in patch 1?

Quoted 17 lines
include/uapi/linux/bpf.h +7 -0
                        struct {
                                __aligned_u64   iter_info;      /* extra bpf_iter_link_info */
                                __u32           iter_info_len;  /* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
                                __s32           priority;
                                __u32           flags;
                        } netfilter;
+                       struct {
+                               union {
+                                       __u32   relative_fd;
+                                       __u32   relative_id;
+                               };
+                               __u32           expected_revision;
+                       } tcx;
                };
        } link_create;
 

[...]

Quoted 23 lines
include/uapi/linux/bpf.h +23 -0
+int tcx_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+       struct net *net = current->nsproxy->net_ns;
+       struct bpf_link_primer link_primer;
+       struct net_device *dev;
+       struct tcx_link *link;
+       int fd, err;
+
+       dev = dev_get_by_index(net, attr->link_create.target_ifindex);
+       if (!dev)
+               return -EINVAL;
+       link = kzalloc(sizeof(*link), GFP_USER);
+       if (!link) {
+               err = -ENOMEM;
+               goto out_put;
+       }
+
+       bpf_link_init(&link->link, BPF_LINK_TYPE_TCX, &tcx_link_lops, prog);
+       link->location = attr->link_create.attach_type;
+       link->flags = attr->link_create.flags & (BPF_F_FIRST | BPF_F_LAST);
+       link->dev = dev;
+
+       err = bpf_link_prime(&link->link, &link_primer);

include/uapi/linux/bpf.h

Quoted 10 lines
include/uapi/linux/bpf.h +10 -0
+       if (err) {
+               kfree(link);
+               goto out_put;
+       }
+       rtnl_lock();
+       err = tcx_link_prog_attach(&link->link, attr->link_create.flags,
+                                  attr->link_create.tcx.relative_fd,
+                                  attr->link_create.tcx.expected_revision);
+       if (!err)
+               fd = bpf_link_settle(&link_primer);

why this early settle? makes the error handling logic more convoluted. Maybe leave link->dev as is and let bpf_link_cleanup() handle dev_put(dev)? Can it be just:

err = tcx_link_prog_attach(...);

rtnl_unlock();

if (err) {
    link->dev = NULL;
    bpf_link_cleanup(&link_primer);
    goto out_put;
}

dev_put(dev);
return bpf_link_settle(&link_primer);

?

Quoted 12 lines
include/uapi/linux/bpf.h +12 -0
+       rtnl_unlock();
+       if (err) {
+               link->dev = NULL;
+               bpf_link_cleanup(&link_primer);
+               goto out_put;
+       }
+       dev_put(dev);
+       return fd;
+out_put:
+       dev_put(dev);
+       return err;
+}

[...]

maintainer
#42 lore raw

Re: [PATCH bpf-next v2 2/7] bpf: Add fd-based tcx multi-prog infra with link support

In reply to Daniel Borkmann
Quoted 10 lines · On Wed, 7 Jun 2023 21:26:20 +0200 Daniel Borkmann wrote:
kernel/bpf/tcx.c new file +10 -0
+	dev = dev_get_by_index(net, attr->link_create.target_ifindex);
+	if (!dev)
+		return -EINVAL;
+	link = kzalloc(sizeof(*link), GFP_USER);
+	if (!link) {
+		err = -ENOMEM;
+		goto out_put;
+	}
+
+	bpf_link_init(&link->link, BPF_LINK_TYPE_TCX, &tcx_link_lops, prog);

kernel/bpf/tcx.c line 330

Quoted 10 lines
kernel/bpf/tcx.c new file +10 -0
+	link->location = attr->link_create.attach_type;
+	link->flags = attr->link_create.flags & (BPF_F_FIRST | BPF_F_LAST);
+	link->dev = dev;
+
+	err = bpf_link_prime(&link->link, &link_primer);
+	if (err) {
+		kfree(link);
+		goto out_put;
+	}
+	rtnl_lock();

How does this work vs device unregistering?

Best I can tell (and it is a large patch :() the device may have passed dev_tcx_uninstall() by the time we take the lock.

Quoted 6 lines
kernel/bpf/tcx.c new file +6 -0
+	err = tcx_link_prog_attach(&link->link, attr->link_create.flags,
+				   attr->link_create.tcx.relative_fd,
+				   attr->link_create.tcx.expected_revision);
+	if (!err)
+		fd = bpf_link_settle(&link_primer);
+	rtnl_unlock();
author maintainer
#43 lore raw

PATCH v2 3/7 libbpf: Add opts-based attach/detach/query API for tcx

In reply to Daniel Borkmann

Extend libbpf attach opts and add a new detach opts API so this can be used to add/remove fd-based tcx BPF programs. The old-style bpf_prog_detach and bpf_prog_detach2 APIs are refactored to reuse the detach opts internally.

The bpf_prog_query_opts API got extended to be able to handle the new link_ids, link_attach_flags and revision fields.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      | 78 ++++++++++++++++++++++------------------
 tools/lib/bpf/bpf.h      | 54 +++++++++++++++++++++-------
 tools/lib/bpf/libbpf.c   |  6 ++++
 tools/lib/bpf/libbpf.map |  1 +
 4 files changed, 91 insertions(+), 48 deletions(-)
tools/lib/bpf/bpf.c +43 -35
index ed86b37d8024..a3d1b7ebe224 100644
@@ -629,11 +629,21 @@ int bpf_prog_attach(int prog_fd, int target_fd, enum bpf_attach_type type,
 	return bpf_prog_attach_opts(prog_fd, target_fd, type, &opts);
 }
 
-int bpf_prog_attach_opts(int prog_fd, int target_fd,
-			  enum bpf_attach_type type,
-			  const struct bpf_prog_attach_opts *opts)
+int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
+{
+	return bpf_prog_detach_opts(0, target_fd, type, NULL);
+}
+
+int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
+	return bpf_prog_detach_opts(prog_fd, target_fd, type, NULL);
+}
+
+int bpf_prog_attach_opts(int prog_fd, int target,
+			 enum bpf_attach_type type,
+			 const struct bpf_prog_attach_opts *opts)
+{
+	const size_t attr_sz = offsetofend(union bpf_attr, expected_revision);
 	union bpf_attr attr;
 	int ret;
 
@@ -641,40 +651,35 @@ int bpf_prog_attach_opts(int prog_fd, int target_fd,
 		return libbpf_err(-EINVAL);
 
 	memset(&attr, 0, attr_sz);
-	attr.target_fd	   = target_fd;
-	attr.attach_bpf_fd = prog_fd;
-	attr.attach_type   = type;
-	attr.attach_flags  = OPTS_GET(opts, flags, 0);
-	attr.replace_bpf_fd = OPTS_GET(opts, replace_prog_fd, 0);
+	attr.target_fd		= target;
+	attr.attach_bpf_fd	= prog_fd;
+	attr.attach_type	= type;
+	attr.attach_flags	= OPTS_GET(opts, flags, 0);
+	attr.replace_bpf_fd	= OPTS_GET(opts, relative_fd, 0);
+	attr.expected_revision	= OPTS_GET(opts, expected_revision, 0);
 
 	ret = sys_bpf(BPF_PROG_ATTACH, &attr, attr_sz);
 	return libbpf_err_errno(ret);
 }
 
-int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
+int bpf_prog_detach_opts(int prog_fd, int target,
+			 enum bpf_attach_type type,
+			 const struct bpf_prog_detach_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
+	const size_t attr_sz = offsetofend(union bpf_attr, expected_revision);
 	union bpf_attr attr;
 	int ret;
 
-	memset(&attr, 0, attr_sz);
-	attr.target_fd	 = target_fd;
-	attr.attach_type = type;
-
-	ret = sys_bpf(BPF_PROG_DETACH, &attr, attr_sz);
-	return libbpf_err_errno(ret);
-}
-
-int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
-{
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
-	union bpf_attr attr;
-	int ret;
+	if (!OPTS_VALID(opts, bpf_prog_detach_opts))
+		return libbpf_err(-EINVAL);
 
 	memset(&attr, 0, attr_sz);
-	attr.target_fd	 = target_fd;
-	attr.attach_bpf_fd = prog_fd;
-	attr.attach_type = type;
+	attr.target_fd		= target;
+	attr.attach_bpf_fd	= prog_fd;
+	attr.attach_type	= type;
+	attr.attach_flags	= OPTS_GET(opts, flags, 0);
+	attr.replace_bpf_fd	= OPTS_GET(opts, relative_fd, 0);
+	attr.expected_revision	= OPTS_GET(opts, expected_revision, 0);
 
 	ret = sys_bpf(BPF_PROG_DETACH, &attr, attr_sz);
 	return libbpf_err_errno(ret);
@@ -833,7 +838,7 @@ int bpf_iter_create(int link_fd)
 	return libbpf_err_errno(fd);
 }
 
-int bpf_prog_query_opts(int target_fd,
+int bpf_prog_query_opts(int target,
 			enum bpf_attach_type type,
 			struct bpf_prog_query_opts *opts)
 {
@@ -846,17 +851,20 @@ int bpf_prog_query_opts(int target_fd,
 
 	memset(&attr, 0, attr_sz);
 
-	attr.query.target_fd	= target_fd;
-	attr.query.attach_type	= type;
-	attr.query.query_flags	= OPTS_GET(opts, query_flags, 0);
-	attr.query.prog_cnt	= OPTS_GET(opts, prog_cnt, 0);
-	attr.query.prog_ids	= ptr_to_u64(OPTS_GET(opts, prog_ids, NULL));
-	attr.query.prog_attach_flags = ptr_to_u64(OPTS_GET(opts, prog_attach_flags, NULL));
+	attr.query.target_fd		= target;
+	attr.query.attach_type		= type;
+	attr.query.query_flags		= OPTS_GET(opts, query_flags, 0);
+	attr.query.count		= OPTS_GET(opts, count, 0);
+	attr.query.prog_ids		= ptr_to_u64(OPTS_GET(opts, prog_ids, NULL));
+	attr.query.prog_attach_flags	= ptr_to_u64(OPTS_GET(opts, prog_attach_flags, NULL));
+	attr.query.link_ids		= ptr_to_u64(OPTS_GET(opts, link_ids, NULL));
+	attr.query.link_attach_flags	= ptr_to_u64(OPTS_GET(opts, link_attach_flags, NULL));
 
 	ret = sys_bpf(BPF_PROG_QUERY, &attr, attr_sz);
 
 	OPTS_SET(opts, attach_flags, attr.query.attach_flags);
-	OPTS_SET(opts, prog_cnt, attr.query.prog_cnt);
+	OPTS_SET(opts, revision, attr.query.revision);
+	OPTS_SET(opts, count, attr.query.count);
 
 	return libbpf_err_errno(ret);
 }
tools/lib/bpf/bpf.h +41 -13

77 lines not on this page · raw message

tools/lib/bpf/libbpf.c +6 -0
index 47632606b06d..b89127471c6a 100644
@@ -117,6 +117,8 @@ static const char * const attach_type_name[] = {
 	[BPF_PERF_EVENT]		= "perf_event",
 	[BPF_TRACE_KPROBE_MULTI]	= "trace_kprobe_multi",
 	[BPF_STRUCT_OPS]		= "struct_ops",
+	[BPF_TCX_INGRESS]		= "tcx_ingress",
+	[BPF_TCX_EGRESS]		= "tcx_egress",
 };
 
 static const char * const link_type_name[] = {
@@ -8669,6 +8671,10 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
 	SEC_DEF("tc",			SCHED_CLS, 0, SEC_NONE),
+	SEC_DEF("tc/ingress",		SCHED_CLS, BPF_TCX_INGRESS, SEC_ATTACHABLE_OPT),
+	SEC_DEF("tc/egress",		SCHED_CLS, BPF_TCX_EGRESS, SEC_ATTACHABLE_OPT),
+	SEC_DEF("tcx/ingress",		SCHED_CLS, BPF_TCX_INGRESS, SEC_ATTACHABLE_OPT),
+	SEC_DEF("tcx/egress",		SCHED_CLS, BPF_TCX_EGRESS, SEC_ATTACHABLE_OPT),
 	SEC_DEF("classifier",		SCHED_CLS, 0, SEC_NONE),
 	SEC_DEF("action",		SCHED_ACT, 0, SEC_NONE),
 	SEC_DEF("tracepoint+",		TRACEPOINT, 0, SEC_NONE, attach_tp),
tools/lib/bpf/libbpf.map +1 -0
index 7521a2fb7626..a29b90e9713c 100644
@@ -395,4 +395,5 @@ LIBBPF_1.2.0 {
 LIBBPF_1.3.0 {
 	global:
 		bpf_obj_pin_opts;
+		bpf_prog_detach_opts;
 } LIBBPF_1.2.0;
Signature · 1 line
2.34.1
#44 lore raw

Re: [PATCH bpf-next v2 3/7] libbpf: Add opts-based attach/detach/query API for tcx

Quoted 30 lines · Daniel Borkmann

Extend libbpf attach opts and add a new detach opts API so this can be used to add/remove fd-based tcx BPF programs. The old-style bpf_prog_detach and bpf_prog_detach2 APIs are refactored to reuse the detach opts internally.

The bpf_prog_query_opts API got extended to be able to handle the new link_ids, link_attach_flags and revision fields.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      | 78 ++++++++++++++++++++++------------------
 tools/lib/bpf/bpf.h      | 54 +++++++++++++++++++++-------
 tools/lib/bpf/libbpf.c   |  6 ++++
 tools/lib/bpf/libbpf.map |  1 +
 4 files changed, 91 insertions(+), 48 deletions(-)
tools/lib/bpf/bpf.c +0 -3
index ed86b37d8024..a3d1b7ebe224 100644
@@ -629,11 +629,21 @@ int bpf_prog_attach(int prog_fd, int target_fd, enum bpf_attach_type type,
        return bpf_prog_attach_opts(prog_fd, target_fd, type, &opts);
 }
 
-int bpf_prog_attach_opts(int prog_fd, int target_fd,
-                         enum bpf_attach_type type,
-                         const struct bpf_prog_attach_opts *opts)

tools/lib/bpf/bpf.c int bpf_prog_attach(int prog_fd, int target_fd, enum bpf_attach_type type, · line 640

Quoted 10 lines
tools/lib/bpf/bpf.c +8 -1
+int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
+{
+       return bpf_prog_detach_opts(0, target_fd, type, NULL);
+}
+
+int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
 {
-       const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
+       return bpf_prog_detach_opts(prog_fd, target_fd, type, NULL);
+}

Please put these wrappers after bpf_prog_detach_ops(), it will make the diff cleaner and will keep them closer to full version of bpf_prog_detach_opts().

Quoted 9 lines
tools/lib/bpf/bpf.c +6 -0
+
+int bpf_prog_attach_opts(int prog_fd, int target,
+                        enum bpf_attach_type type,
+                        const struct bpf_prog_attach_opts *opts)
+{
+       const size_t attr_sz = offsetofend(union bpf_attr, expected_revision);
        union bpf_attr attr;
        int ret;
 

[...]

Quoted 23 lines
tools/lib/bpf/bpf.h +0 -10
index 9aa0ee473754..480c584a6f7f 100644
@@ -312,22 +312,43 @@ LIBBPF_API int bpf_obj_get(const char *pathname);
 LIBBPF_API int bpf_obj_get_opts(const char *pathname,
                                const struct bpf_obj_get_opts *opts);
 
-struct bpf_prog_attach_opts {
-       size_t sz; /* size of this struct for forward/backward compatibility */
-       unsigned int flags;
-       int replace_prog_fd;
-};
-#define bpf_prog_attach_opts__last_field replace_prog_fd
-
 LIBBPF_API int bpf_prog_attach(int prog_fd, int attachable_fd,
                               enum bpf_attach_type type, unsigned int flags);
-LIBBPF_API int bpf_prog_attach_opts(int prog_fd, int attachable_fd,
-                                    enum bpf_attach_type type,
-                                    const struct bpf_prog_attach_opts *opts);
 LIBBPF_API int bpf_prog_detach(int attachable_fd, enum bpf_attach_type type);
 LIBBPF_API int bpf_prog_detach2(int prog_fd, int attachable_fd,
                                enum bpf_attach_type type);

tools/lib/bpf/bpf.h LIBBPF_API int bpf_obj_get(const char *pathname); · line 329

Quoted 10 lines
tools/lib/bpf/bpf.h +9 -0
 
+struct bpf_prog_attach_opts {
+       size_t sz; /* size of this struct for forward/backward compatibility */
+       __u32 flags;
+       union {
+               int     replace_prog_fd;
+               int     replace_fd;
+               int     relative_fd;
+               __u32   relative_id;
+       };

I tried to not use union for such cases in OPTS-based interfaces, see bpf_link_create(). Let's keep them all as separate fields and then return error if, say, both relative_fd and relative_id is specified at the same time.

It's fine to have replace_prog_fd and replace_fd as a union, as they are basically just synonyms.

tools/lib/bpf/bpf.h LIBBPF_API int bpf_obj_get(const char *pathname); · line 340

Quoted 11 lines
tools/lib/bpf/bpf.h +11 -0
+       __u32 expected_revision;
+};
+#define bpf_prog_attach_opts__last_field expected_revision
+
+struct bpf_prog_detach_opts {
+       size_t sz; /* size of this struct for forward/backward compatibility */
+       __u32 flags;
+       union {
+               int     relative_fd;
+               __u32   relative_id;
+       };

same as above

tools/lib/bpf/bpf.h LIBBPF_API int bpf_obj_get(const char *pathname); · line 345

Quoted 5 lines
tools/lib/bpf/bpf.h +5 -0
+       __u32 expected_revision;
+};
+#define bpf_prog_detach_opts__last_field expected_revision
+
+LIBBPF_API int bpf_prog_attach_opts(int prog_fd, int target,

let's add doc comments to both these APIs, where target is explained. Right now because it doesn't have "_fd" suffix it's not very clear what sort of value it is (I know why it's not target_fd anymore due to target_ifindex)

Quoted 18 lines
tools/lib/bpf/bpf.h +10 -1
+                                   enum bpf_attach_type type,
+                                   const struct bpf_prog_attach_opts *opts);
+LIBBPF_API int bpf_prog_detach_opts(int prog_fd, int target,
+                                   enum bpf_attach_type type,
+                                   const struct bpf_prog_detach_opts *opts);
+
 union bpf_iter_link_info; /* defined in up-to-date linux/bpf.h */
 struct bpf_link_create_opts {
        size_t sz; /* size of this struct for forward/backward compatibility */
@@ -489,14 +510,21 @@ struct bpf_prog_query_opts {
        __u32 query_flags;
        __u32 attach_flags; /* output argument */
        __u32 *prog_ids;
-       __u32 prog_cnt; /* input+output argument */
+       union {
+               __u32 prog_cnt; /* input+output argument */
+               __u32 count;
+       };

tools/lib/bpf/bpf.h struct bpf_prog_query_opts { · line 524

Quoted 10 lines
tools/lib/bpf/bpf.h +5 -2
        __u32 *prog_attach_flags;
+       __u32 *link_ids;
+       __u32 *link_attach_flags;
+       __u32 revision;
 };
-#define bpf_prog_query_opts__last_field prog_attach_flags
+#define bpf_prog_query_opts__last_field revision
 
-LIBBPF_API int bpf_prog_query_opts(int target_fd,
+LIBBPF_API int bpf_prog_query_opts(int target,

same here for doc comment

Quoted 20 lines
tools/lib/bpf/bpf.h +1 -0
                                   enum bpf_attach_type type,
                                   struct bpf_prog_query_opts *opts);
+
 LIBBPF_API int bpf_prog_query(int target_fd, enum bpf_attach_type type,
                              __u32 query_flags, __u32 *attach_flags,
                              __u32 *prog_ids, __u32 *prog_cnt);
tools/lib/bpf/libbpf.c +2 -0
index 47632606b06d..b89127471c6a 100644
@@ -117,6 +117,8 @@ static const char * const attach_type_name[] = {
        [BPF_PERF_EVENT]                = "perf_event",
        [BPF_TRACE_KPROBE_MULTI]        = "trace_kprobe_multi",
        [BPF_STRUCT_OPS]                = "struct_ops",
+       [BPF_TCX_INGRESS]               = "tcx_ingress",
+       [BPF_TCX_EGRESS]                = "tcx_egress",
 };
 
 static const char * const link_type_name[] = {

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 8675

Quoted 5 lines
tools/lib/bpf/libbpf.c +2 -0
@@ -8669,6 +8671,10 @@ static const struct bpf_sec_def section_defs[] = {
        SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
        SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
        SEC_DEF("tc",                   SCHED_CLS, 0, SEC_NONE),
+       SEC_DEF("tc/ingress",           SCHED_CLS, BPF_TCX_INGRESS, SEC_ATTACHABLE_OPT),
+       SEC_DEF("tc/egress",            SCHED_CLS, BPF_TCX_EGRESS, SEC_ATTACHABLE_OPT),

for tc/ingress and tc/egress, is it intentional that libbpf should set expected_attach_type to zero if kernel doesn't support BPF_TCX_INGRESS or BPF_TCX_EGRESS? Or is it just an alias to tcx/ingress and tcx/egress?

If it's an alias, why do we need it?

If not, let's replace SEC_ATTACHABLE_OPT with just SEC_EXP_ATTACH_OPT ?

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 8677

Quoted 2 lines
tools/lib/bpf/libbpf.c +2 -0
+       SEC_DEF("tcx/ingress",          SCHED_CLS, BPF_TCX_INGRESS, SEC_ATTACHABLE_OPT),
+       SEC_DEF("tcx/egress",           SCHED_CLS, BPF_TCX_EGRESS, SEC_ATTACHABLE_OPT),

at least for tcx attach_type is not optional, right? So I'd drop SEC_ATTACHABLE_OPT.

Quoted 16 lines
tools/lib/bpf/libbpf.c +0 -0
        SEC_DEF("classifier",           SCHED_CLS, 0, SEC_NONE),
        SEC_DEF("action",               SCHED_ACT, 0, SEC_NONE),
        SEC_DEF("tracepoint+",          TRACEPOINT, 0, SEC_NONE, attach_tp),
tools/lib/bpf/libbpf.map +1 -1
index 7521a2fb7626..a29b90e9713c 100644
@@ -395,4 +395,5 @@ LIBBPF_1.2.0 {
 LIBBPF_1.3.0 {
        global:
                bpf_obj_pin_opts;
+               bpf_prog_detach_opts;
 } LIBBPF_1.2.0;
--

2.34.1

author maintainer
#45 lore raw

PATCH v2 4/7 libbpf: Add link-based API for tcx

In reply to Daniel Borkmann

Implement tcx BPF link support for libbpf.

The bpf_program__attach_fd_opts() API has been refactored slightly in order to pass bpf_link_create_opts pointer as input.

A new bpf_program__attach_tcx_opts() has been added on top of this which allows for passing all relevant data via extensible struct bpf_tcx_opts.

The program sections tcx/ingress and tcx/egress correspond to the hook locations for tc ingress and egress, respectively.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      |  5 +++++
 tools/lib/bpf/bpf.h      |  7 +++++++
 tools/lib/bpf/libbpf.c   | 44 +++++++++++++++++++++++++++++++++++-----
 tools/lib/bpf/libbpf.h   | 17 ++++++++++++++++
 tools/lib/bpf/libbpf.map |  1 +
 5 files changed, 69 insertions(+), 5 deletions(-)
tools/lib/bpf/bpf.c +5 -0
index a3d1b7ebe224..c340d3cbc6bd 100644
@@ -746,6 +746,11 @@ int bpf_link_create(int prog_fd, int target_fd,
 		if (!OPTS_ZEROED(opts, tracing))
 			return libbpf_err(-EINVAL);
 		break;
+	case BPF_TCX_INGRESS:
+	case BPF_TCX_EGRESS:
+		attr.link_create.tcx.relative_fd = OPTS_GET(opts, tcx.relative_fd, 0);
+		attr.link_create.tcx.expected_revision = OPTS_GET(opts, tcx.expected_revision, 0);
+		break;
 	default:
 		if (!OPTS_ZEROED(opts, flags))
 			return libbpf_err(-EINVAL);
tools/lib/bpf/bpf.h +7 -0
index 480c584a6f7f..12591516dca0 100644
@@ -370,6 +370,13 @@ struct bpf_link_create_opts {
 		struct {
 			__u64 cookie;
 		} tracing;
+		struct {
+			union {
+				__u32 relative_fd;
+				__u32 relative_id;
+			};
+			__u32 expected_revision;
+		} tcx;
 	};
 	size_t :0;
 };
tools/lib/bpf/libbpf.c +39 -5

75 lines not on this page · raw message

tools/lib/bpf/libbpf.h +17 -0
index 754da73c643b..8ffba0f67c60 100644
@@ -718,6 +718,23 @@ LIBBPF_API struct bpf_link *
 bpf_program__attach_freplace(const struct bpf_program *prog,
 			     int target_fd, const char *attach_func_name);
 
+struct bpf_tcx_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	int ifindex;
+	__u32 flags;
+	union {
+		__u32 relative_fd;
+		__u32 relative_id;
+	};
+	__u32 expected_revision;
+};
+#define bpf_tcx_opts__last_field expected_revision
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_tcx_opts(const struct bpf_program *prog,
+			     const struct bpf_tcx_opts *opts);
+
 struct bpf_map;
 
 LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
tools/lib/bpf/libbpf.map +1 -0
index a29b90e9713c..f66b714512c2 100644
@@ -396,4 +396,5 @@ LIBBPF_1.3.0 {
 	global:
 		bpf_obj_pin_opts;
 		bpf_prog_detach_opts;
+		bpf_program__attach_tcx_opts;
 } LIBBPF_1.2.0;
Signature · 1 line
2.34.1
#46 lore raw

Re: [PATCH bpf-next v2 4/7] libbpf: Add link-based API for tcx

Quoted 29 lines · Daniel Borkmann

Implement tcx BPF link support for libbpf.

The bpf_program__attach_fd_opts() API has been refactored slightly in order to pass bpf_link_create_opts pointer as input.

A new bpf_program__attach_tcx_opts() has been added on top of this which allows for passing all relevant data via extensible struct bpf_tcx_opts.

The program sections tcx/ingress and tcx/egress correspond to the hook locations for tc ingress and egress, respectively.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      |  5 +++++
 tools/lib/bpf/bpf.h      |  7 +++++++
 tools/lib/bpf/libbpf.c   | 44 +++++++++++++++++++++++++++++++++++-----
 tools/lib/bpf/libbpf.h   | 17 ++++++++++++++++
 tools/lib/bpf/libbpf.map |  1 +
 5 files changed, 69 insertions(+), 5 deletions(-)
tools/lib/bpf/bpf.c +0 -0
index a3d1b7ebe224..c340d3cbc6bd 100644

tools/lib/bpf/bpf.c int bpf_link_create(int prog_fd, int target_fd, · line 752

Quoted 7 lines
tools/lib/bpf/bpf.c +4 -0
@@ -746,6 +746,11 @@ int bpf_link_create(int prog_fd, int target_fd,
                if (!OPTS_ZEROED(opts, tracing))
                        return libbpf_err(-EINVAL);
                break;
+       case BPF_TCX_INGRESS:
+       case BPF_TCX_EGRESS:
+               attr.link_create.tcx.relative_fd = OPTS_GET(opts, tcx.relative_fd, 0);
+               attr.link_create.tcx.expected_revision = OPTS_GET(opts, tcx.expected_revision, 0);

can you also add an OPTS_ZEROED check like for other types of links?

Quoted 9 lines
tools/lib/bpf/bpf.c +1 -0
+               break;
        default:
                if (!OPTS_ZEROED(opts, flags))
                        return libbpf_err(-EINVAL);
tools/lib/bpf/bpf.h +0 -0
index 480c584a6f7f..12591516dca0 100644

tools/lib/bpf/bpf.h struct bpf_link_create_opts { · line 377

Quoted 8 lines
tools/lib/bpf/bpf.h +5 -0
@@ -370,6 +370,13 @@ struct bpf_link_create_opts {
                struct {
                        __u64 cookie;
                } tracing;
+               struct {
+                       union {
+                               __u32 relative_fd;
+                               __u32 relative_id;
+                       };

same comment about union, let's not add it and have two separate fields

Quoted 18 lines
tools/lib/bpf/bpf.h +2 -0
+                       __u32 expected_revision;
+               } tcx;
        };
        size_t :0;
 };
tools/lib/bpf/libbpf.c +1 -0
index b89127471c6a..d7b6ff49f02e 100644
@@ -133,6 +133,7 @@ static const char * const link_type_name[] = {
        [BPF_LINK_TYPE_KPROBE_MULTI]            = "kprobe_multi",
        [BPF_LINK_TYPE_STRUCT_OPS]              = "struct_ops",
        [BPF_LINK_TYPE_NETFILTER]               = "netfilter",
+       [BPF_LINK_TYPE_TCX]                     = "tcx",
 };
 
 static const char * const map_type_name[] = {

tools/lib/bpf/libbpf.c static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_li · line 11691

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -2
@@ -11685,11 +11686,10 @@ static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_li
 }
 
 static struct bpf_link *
-bpf_program__attach_fd(const struct bpf_program *prog, int target_fd, int btf_id,
-                      const char *target_name)
+bpf_program__attach_fd_opts(const struct bpf_program *prog,
+                           const struct bpf_link_create_opts *opts,
+                           int target_fd, const char *target_name)

nit: please keep opts as the last argument

Quoted 18 lines
tools/lib/bpf/libbpf.c +1 -3
 {
-       DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts,
-                           .target_btf_id = btf_id);
        enum bpf_attach_type attach_type;
        char errmsg[STRERR_BUFSIZE];
        struct bpf_link *link;
@@ -11707,7 +11707,7 @@ bpf_program__attach_fd(const struct bpf_program *prog, int target_fd, int btf_id
        link->detach = &bpf_link__detach_fd;
 
        attach_type = bpf_program__expected_attach_type(prog);
-       link_fd = bpf_link_create(prog_fd, target_fd, attach_type, &opts);
+       link_fd = bpf_link_create(prog_fd, target_fd, attach_type, opts);
        if (link_fd < 0) {
                link_fd = -errno;
                free(link);
@@ -11720,6 +11720,17 @@ bpf_program__attach_fd(const struct bpf_program *prog, int target_fd, int btf_id
        return link;
 }

tools/lib/bpf/libbpf.c bpf_program__attach_fd(const struct bpf_program *prog, int target_fd, int btf_id · line 11731

Quoted 10 lines
tools/lib/bpf/libbpf.c +9 -0
 
+static struct bpf_link *
+bpf_program__attach_fd(const struct bpf_program *prog, int target_fd, int btf_id,
+                      const char *target_name)
+{
+       LIBBPF_OPTS(bpf_link_create_opts, opts,
+               .target_btf_id = btf_id,
+       );
+
+       return bpf_program__attach_fd_opts(prog, &opts, target_fd, target_name);

it seems like the only user of btf_id is bpf_program__attach_freplace, so I'd just inline this there, and for all other 4 cases let's just pass NULL as options?

That means we don't really need bpf_program__attach_fd_opts() and can just add opts to bpf_program__attach_fd(). We'll have shorter name. BTW, given it's not exposed API, let's drop double underscore and call it just bpf_program_attach_fd()?

tools/lib/bpf/libbpf.c struct bpf_link *bpf_program__attach_xdp(const struct bpf_program *prog, int ifi · line 11754

Quoted 12 lines
tools/lib/bpf/libbpf.c +5 -0
+}
+
 struct bpf_link *
 bpf_program__attach_cgroup(const struct bpf_program *prog, int cgroup_fd)
 {
@@ -11738,6 +11749,29 @@ struct bpf_link *bpf_program__attach_xdp(const struct bpf_program *prog, int ifi
        return bpf_program__attach_fd(prog, ifindex, 0, "xdp");
 }
 
+struct bpf_link *
+bpf_program__attach_tcx_opts(const struct bpf_program *prog,
+                            const struct bpf_tcx_opts *opts)

we don't have non-opts variant, so let's keep the name short (like we did with bpf_program__attach_netlink): bpf_program__attach_tcx().

tools/lib/bpf/libbpf.c struct bpf_link *bpf_program__attach_xdp(const struct bpf_program *prog, int ifi · line 11757

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+{
+       LIBBPF_OPTS(bpf_link_create_opts, link_create_opts);
+       int ifindex = OPTS_GET(opts, ifindex, 0);

let's not do OPTS_GET before we checked OPTS_VALID

Quoted 27 lines
tools/lib/bpf/libbpf.c +17 -0
+
+       if (!OPTS_VALID(opts, bpf_tcx_opts))
+               return libbpf_err_ptr(-EINVAL);
+       if (!ifindex) {
+               pr_warn("prog '%s': target netdevice ifindex cannot be zero\n",
+                       prog->name);
+               return libbpf_err_ptr(-EINVAL);
+       }
+
+       link_create_opts.tcx.expected_revision = OPTS_GET(opts, expected_revision, 0);
+       link_create_opts.tcx.relative_fd = OPTS_GET(opts, relative_fd, 0);
+       link_create_opts.flags = OPTS_GET(opts, flags, 0);
+
+       /* target_fd/target_ifindex use the same field in LINK_CREATE */
+       return bpf_program__attach_fd_opts(prog, &link_create_opts, ifindex, "tc");
+}
+
 struct bpf_link *bpf_program__attach_freplace(const struct bpf_program *prog,
                                              int target_fd,
                                              const char *attach_func_name)
tools/lib/bpf/libbpf.h +0 -0
index 754da73c643b..8ffba0f67c60 100644
@@ -718,6 +718,23 @@ LIBBPF_API struct bpf_link *
 bpf_program__attach_freplace(const struct bpf_program *prog,
                             int target_fd, const char *attach_func_name);

tools/lib/bpf/libbpf.h LIBBPF_API struct bpf_link * · line 729

Quoted 10 lines
tools/lib/bpf/libbpf.h +9 -0
 
+struct bpf_tcx_opts {
+       /* size of this struct, for forward/backward compatibility */
+       size_t sz;
+       int ifindex;
+       __u32 flags;
+       union {
+               __u32 relative_fd;
+               __u32 relative_id;
+       };

same thing about not using unions here :)

tools/lib/bpf/libbpf.h LIBBPF_API struct bpf_link * · line 730

Quoted 1 line
tools/lib/bpf/libbpf.h +1 -0
+       __u32 expected_revision;

and let's add size_t :0; to prevent compiler from leaving garbage values in a padding at the end of the struct (once you drop union there will be padding)

Quoted 23 lines
tools/lib/bpf/libbpf.h +7 -0
+};
+#define bpf_tcx_opts__last_field expected_revision
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_tcx_opts(const struct bpf_program *prog,
+                            const struct bpf_tcx_opts *opts);
+
 struct bpf_map;
 
 LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
tools/lib/bpf/libbpf.map +1 -1
index a29b90e9713c..f66b714512c2 100644
@@ -396,4 +396,5 @@ LIBBPF_1.3.0 {
        global:
                bpf_obj_pin_opts;
                bpf_prog_detach_opts;
+               bpf_program__attach_tcx_opts;
 } LIBBPF_1.2.0;
--

2.34.1

author maintainer
#47 lore raw

PATCH v2 5/7 bpftool: Extend net dump with tcx progs

In reply to Daniel Borkmann

Add support to dump fd-based attach types via bpftool. This includes both the tc BPF link and attach ops programs. Dumped information contain the attach location, function entry name, program ID and link ID when applicable.

Example with tc BPF link:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 784 link id 10 bond0(4) bpf/egress cil_to_netdev prog id 804 link id 11

flow_dissector:

netfilter:

Example with tc BPF attach ops:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 654 bond0(4) bpf/egress cil_to_netdev prog id 672

flow_dissector:

netfilter:

---
 tools/bpf/bpftool/net.c | 92 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 88 insertions(+), 4 deletions(-)
tools/bpf/bpftool/net.c +88 -4
index 26a49965bf71..b23b346b3ae9 100644
@@ -76,6 +76,11 @@ static const char * const attach_type_strings[] = {
 	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
 };
 
+static const char * const attach_loc_strings[] = {
+	[BPF_TCX_INGRESS]		= "bpf/ingress",
+	[BPF_TCX_EGRESS]		= "bpf/egress",
+};
+
 const size_t net_attach_type_size = ARRAY_SIZE(attach_type_strings);
 
 static enum net_attach_type parse_attach_type(const char *str)
@@ -422,8 +427,86 @@ static int dump_filter_nlmsg(void *cookie, void *msg, struct nlattr **tb)
 			      filter_info->devname, filter_info->ifindex);
 }
 
-static int show_dev_tc_bpf(int sock, unsigned int nl_pid,
-			   struct ip_devname_ifindex *dev)
+static const char *flags_strings(__u32 flags)
+{
+	if (flags == (BPF_F_FIRST | BPF_F_LAST))
+		return json_output ? "first,last" : " first last";
+	if (flags & BPF_F_FIRST)
+		return json_output ? "first" : " first";
+	if (flags & BPF_F_LAST)
+		return json_output ? "last" : " last";
+	return json_output ? "none" : "";
+}
+
+static int __show_dev_tc_bpf_name(__u32 id, char *name, size_t len)
+{
+	struct bpf_prog_info info = {};
+	__u32 ilen = sizeof(info);
+	int fd, ret;
+
+	fd = bpf_prog_get_fd_by_id(id);
+	if (fd < 0)
+		return fd;
+	ret = bpf_obj_get_info_by_fd(fd, &info, &ilen);
+	if (ret < 0)
+		goto out;
+	ret = -ENOENT;
+	if (info.name[0]) {
+		get_prog_full_name(&info, fd, name, len);
+		ret = 0;
+	}
+out:
+	close(fd);
+	return ret;
+}
+
+static void __show_dev_tc_bpf(const struct ip_devname_ifindex *dev,
+			      const enum bpf_attach_type loc)
+{
+	__u32 prog_flags[64] = {}, link_flags[64] = {}, i;
+	__u32 prog_ids[64] = {}, link_ids[64] = {};
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	char prog_name[MAX_PROG_FULL_NAME];
+	int ret;
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	ret = bpf_prog_query_opts(dev->ifindex, loc, &optq);
+	if (ret)
+		return;
+	for (i = 0; i < optq.count; i++) {
+		NET_START_OBJECT;
+		NET_DUMP_STR("devname", "%s", dev->devname);
+		NET_DUMP_UINT("ifindex", "(%u)", dev->ifindex);
+		NET_DUMP_STR("kind", " %s", attach_loc_strings[loc]);
+		ret = __show_dev_tc_bpf_name(prog_ids[i], prog_name,
+					     sizeof(prog_name));
+		if (!ret)
+			NET_DUMP_STR("name", " %s", prog_name);
+		NET_DUMP_UINT("prog_id", " prog id %u", prog_ids[i]);
+		if (prog_flags[i])
+			NET_DUMP_STR("prog_flags", "%s", flags_strings(prog_flags[i]));
+		if (link_ids[i])
+			NET_DUMP_UINT("link_id", " link id %u",
+				      link_ids[i]);
+		if (link_flags[i])
+			NET_DUMP_STR("link_flags", "%s", flags_strings(link_flags[i]));
+		NET_END_OBJECT_FINAL;
+	}
+}
+
+static void show_dev_tc_bpf(struct ip_devname_ifindex *dev)
+{
+	__show_dev_tc_bpf(dev, BPF_TCX_INGRESS);
+	__show_dev_tc_bpf(dev, BPF_TCX_EGRESS);
+}
+
+static int show_dev_tc_bpf_classic(int sock, unsigned int nl_pid,
+				   struct ip_devname_ifindex *dev)
 {
 	struct bpf_filter_t filter_info;
 	struct bpf_tcinfo_t tcinfo;
@@ -790,8 +873,9 @@ static int do_show(int argc, char **argv)
 	if (!ret) {
 		NET_START_ARRAY("tc", "%s:\n");
 		for (i = 0; i < dev_array.used_len; i++) {
-			ret = show_dev_tc_bpf(sock, nl_pid,
-					      &dev_array.devices[i]);
+			show_dev_tc_bpf(&dev_array.devices[i]);
+			ret = show_dev_tc_bpf_classic(sock, nl_pid,
+						      &dev_array.devices[i]);
 			if (ret)
 				break;
 		}
Signature · 1 line
2.34.1
author maintainer
#48 lore raw

PATCH v2 6/7 selftests/bpf: Add mprog API tests for BPF tcx opts

In reply to Daniel Borkmann

Add a big batch of test coverage to assert all aspects of the tcx opts attach, detach and query API:

  # ./vmtest.sh -- ./test_progs -t tc_opts
  [...]
  #237     tc_opts_after:OK
  #238     tc_opts_append:OK
  #239     tc_opts_basic:OK
  #240     tc_opts_before:OK
  #241     tc_opts_both:OK
  #242     tc_opts_chain_classic:OK
  #243     tc_opts_demixed:OK
  #244     tc_opts_detach:OK
  #245     tc_opts_detach_after:OK
  #246     tc_opts_detach_before:OK
  #247     tc_opts_dev_cleanup:OK
  #248     tc_opts_first:OK
  #249     tc_opts_invalid:OK
  #250     tc_opts_last:OK
  #251     tc_opts_mixed:OK
  #252     tc_opts_prepend:OK
  #253     tc_opts_replace:OK
  #254     tc_opts_revision:OK
  Summary: 18/0 PASSED, 0 SKIPPED, 0 FAILED
---
 .../selftests/bpf/prog_tests/tc_helpers.h     |   72 +
 .../selftests/bpf/prog_tests/tc_opts.c        | 2698 +++++++++++++++++
 .../selftests/bpf/progs/test_tc_link.c        |   40 +
 3 files changed, 2810 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_opts.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_tc_link.c
tools/testing/selftests/bpf/prog_tests/tc_helpers.h new file +72 -0
new file mode 100644
index 000000000000..6c93215be8a3
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef TC_HELPERS
+#define TC_HELPERS
+#include <test_progs.h>
+
+static inline __u32 id_from_prog_fd(int fd)
+{
+	struct bpf_prog_info prog_info = {};
+	__u32 prog_info_len = sizeof(prog_info);
+	int err;
+
+	err = bpf_obj_get_info_by_fd(fd, &prog_info, &prog_info_len);
+	if (!ASSERT_OK(err, "id_from_prog_fd"))
+		return 0;
+
+	ASSERT_NEQ(prog_info.id, 0, "prog_info.id");
+	return prog_info.id;
+}
+
+static inline __u32 id_from_link_fd(int fd)
+{
+	struct bpf_link_info link_info = {};
+	__u32 link_info_len = sizeof(link_info);
+	int err;
+
+	err = bpf_link_get_info_by_fd(fd, &link_info, &link_info_len);
+	if (!ASSERT_OK(err, "id_from_link_fd"))
+		return 0;
+
+	ASSERT_NEQ(link_info.id, 0, "link_info.id");
+	return link_info.id;
+}
+
+static inline __u32 ifindex_from_link_fd(int fd)
+{
+	struct bpf_link_info link_info = {};
+	__u32 link_info_len = sizeof(link_info);
+	int err;
+
+	err = bpf_link_get_info_by_fd(fd, &link_info, &link_info_len);
+	if (!ASSERT_OK(err, "id_from_link_fd"))
+		return 0;
+
+	return link_info.tcx.ifindex;
+}
+
+static inline void __assert_mprog_count(int target, int expected, bool miniq, int ifindex)
+{
+	__u32 count = 0, attach_flags = 0;
+	int err;
+
+	err = bpf_prog_query(ifindex, target, 0, &attach_flags,
+			     NULL, &count);
+	ASSERT_EQ(count, expected, "count");
+	if (!expected && !miniq)
+		ASSERT_EQ(err, -ENOENT, "prog_query");
+	else
+		ASSERT_EQ(err, 0, "prog_query");
+}
+
+static inline void assert_mprog_count(int target, int expected)
+{
+	__assert_mprog_count(target, expected, false, loopback);
+}
+
+static inline void assert_mprog_count_ifindex(int ifindex, int target, int expected)
+{
+	__assert_mprog_count(target, expected, false, ifindex);
+}
+
+#endif /* TC_HELPERS */
tools/testing/selftests/bpf/prog_tests/tc_opts.c new file +2698 -0

2,698 lines not on this page · raw message

tools/testing/selftests/bpf/progs/test_tc_link.c new file +40 -0
new file mode 100644
index 000000000000..ed1fd0e9cee9
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+char LICENSE[] SEC("license") = "GPL";
+
+bool seen_tc1;
+bool seen_tc2;
+bool seen_tc3;
+bool seen_tc4;
+
+SEC("tc/ingress")
+int tc1(struct __sk_buff *skb)
+{
+	seen_tc1 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc2(struct __sk_buff *skb)
+{
+	seen_tc2 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc3(struct __sk_buff *skb)
+{
+	seen_tc3 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc4(struct __sk_buff *skb)
+{
+	seen_tc4 = true;
+	return TCX_NEXT;
+}
Signature · 1 line
2.34.1
author maintainer
#49 lore raw

PATCH v2 7/7 selftests/bpf: Add mprog API tests for BPF tcx links

In reply to Daniel Borkmann

Add a big batch of test coverage to assert all aspects of the tcx link API:

  # ./vmtest.sh -- ./test_progs -t tc_links
  [...]
  #224     tc_links_after:OK
  #225     tc_links_append:OK
  #226     tc_links_basic:OK
  #227     tc_links_before:OK
  #228     tc_links_both:OK
  #229     tc_links_chain_classic:OK
  #230     tc_links_dev_cleanup:OK
  #231     tc_links_first:OK
  #232     tc_links_invalid:OK
  #233     tc_links_last:OK
  #234     tc_links_prepend:OK
  #235     tc_links_replace:OK
  #236     tc_links_revision:OK
  Summary: 13/0 PASSED, 0 SKIPPED, 0 FAILED
---
 .../selftests/bpf/prog_tests/tc_links.c       | 2279 +++++++++++++++++
 1 file changed, 2279 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_links.c
tools/testing/selftests/bpf/prog_tests/tc_links.c new file +2279 -0
new file mode 100644
index 000000000000..98039db6ccaf
@@ -0,0 +1,2279 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+#include <uapi/linux/if_link.h>
+#include <net/if.h>
+#include <test_progs.h>
+
+#define loopback 1
+#define ping_cmd "ping -q -c1 -w1 127.0.0.1 > /dev/null"
+
+#include "test_tc_link.skel.h"
+#include "tc_helpers.h"
+
+/* Test:
+ *
+ * Basic test which attaches a link to ingress/egress, validates
+ * that the link got attached, runs traffic through the programs,
+ * validates that traffic has been seen, and detaches everything
+ * again. Programs are attached without special flags.
+ */
+void serial_test_tc_links_basic(void)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[2], link_ids[2];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(BPF_TCX_INGRESS, 0);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+
+	ASSERT_EQ(skel->bss->seen_tc1, false, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(BPF_TCX_INGRESS, 1);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, BPF_TCX_INGRESS, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+	ASSERT_NEQ(lid1, lid2, "link_ids_1_2");
+
+	assert_mprog_count(BPF_TCX_INGRESS, 1);
+	assert_mprog_count(BPF_TCX_EGRESS, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, BPF_TCX_EGRESS, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+cleanup:
+	test_tc_link__destroy(skel);
+
+	assert_mprog_count(BPF_TCX_INGRESS, 0);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+}
+
+static void test_tc_links_first_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[3], link_flags[3];
+	__u32 prog_ids[3], link_ids[3];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	assert_mprog_count(target, 1);
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	optl.flags = BPF_F_FIRST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID | BPF_F_LINK;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = 0;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "prog_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	bpf_link__destroy(skel->links.tc2);
+	skel->links.tc2 = NULL;
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_LAST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], BPF_F_LAST, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "prog_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a prog to ingress/egress with first flag set,
+ * validates that the prog got attached, other attach attempts for
+ * this position should fail. Regular attach attempts or with last
+ * flag set should succeed. Detach everything again.
+ */
+void serial_test_tc_links_first(void)
+{
+	test_tc_links_first_target(BPF_TCX_INGRESS);
+	test_tc_links_first_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_last_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[3], link_flags[3];
+	__u32 prog_ids[3], link_ids[3];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_LAST;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	assert_mprog_count(target, 1);
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_LAST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	optl.flags = BPF_F_LAST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_ID | BPF_F_LINK;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = 0;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], BPF_F_LAST, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "prog_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	bpf_link__destroy(skel->links.tc2);
+	skel->links.tc2 = NULL;
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_FIRST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], BPF_F_LAST, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "prog_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with last flag set,
+ * validates that the prog got attached, other attach attempts for
+ * this position should fail. Regular attach attempts or with first
+ * flag set should succeed. Detach everything again.
+ */
+void serial_test_tc_links_last(void)
+{
+	test_tc_links_last_target(BPF_TCX_INGRESS);
+	test_tc_links_last_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_both_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[3], link_flags[3];
+	__u32 prog_ids[3], link_ids[3];
+	__u32 pid1, pid2, lid1;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	assert_mprog_count(target, 1);
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST | BPF_F_LAST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	optl.flags = BPF_F_LAST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_FIRST;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_ID;
+	optl.relative_id = pid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_ID | BPF_F_LINK;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID;
+	optl.relative_id = pid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID | BPF_F_LINK;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = 0;
+	optl.relative_id = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST;
+	optl.relative_id = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST | BPF_F_REPLACE;
+	optl.relative_id = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST | BPF_F_LAST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	err = bpf_link__update_program(skel->links.tc1, skel->progs.tc2);
+	if (!ASSERT_OK(err, "link_update"))
+		goto cleanup;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST | BPF_F_LAST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with first and last
+ * flag set, validates that the link got attached, other attach
+ * attempts should fail. Link update should work. Detach everything
+ * again.
+ */
+void serial_test_tc_links_both(void)
+{
+	test_tc_links_both_target(BPF_TCX_INGRESS);
+	test_tc_links_both_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_before_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[5], link_flags[5];
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID | BPF_F_LINK;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid4, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid4, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid3, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], lid3, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid2, "prog_ids[3]");
+	ASSERT_EQ(optq.prog_attach_flags[3], 0, "prog_flags[3]");
+	ASSERT_EQ(optq.link_ids[3], lid2, "link_ids[3]");
+	ASSERT_EQ(optq.link_attach_flags[3], 0, "link_flags[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.prog_attach_flags[4], 0, "prog_flags[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+	ASSERT_EQ(optq.link_attach_flags[4], 0, "link_flags[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with before flag
+ * set, validates that the link got attached in the right location.
+ * The first test inserts in the middle, then we insert to the front.
+ * Detach everything again.
+ */
+void serial_test_tc_links_before(void)
+{
+	test_tc_links_before_target(BPF_TCX_INGRESS);
+	test_tc_links_before_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_after_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[5], link_flags[5];
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	optl.flags = BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid3, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid3, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid2, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], lid2, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid4, "prog_ids[3]");
+	ASSERT_EQ(optq.prog_attach_flags[3], 0, "prog_flags[3]");
+	ASSERT_EQ(optq.link_ids[3], lid4, "link_ids[3]");
+	ASSERT_EQ(optq.link_attach_flags[3], 0, "link_flags[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.prog_attach_flags[4], 0, "prog_flags[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+	ASSERT_EQ(optq.link_attach_flags[4], 0, "link_flags[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with after flag
+ * set, validates that the link got attached in the right location.
+ * The first test inserts in the middle, then we insert to the end.
+ * Detach everything again.
+ */
+void serial_test_tc_links_after(void)
+{
+	test_tc_links_after_target(BPF_TCX_INGRESS);
+	test_tc_links_after_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_revision_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[3], link_flags[3];
+	__u32 prog_ids[3], link_ids[3];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	optl.expected_revision = 1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.flags = 0;
+	optl.expected_revision = 1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = 0;
+	optl.expected_revision = 2;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "prog_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with revision count
+ * set, validates that the link got attached and validates that
+ * when the count mismatches that the operation bails out. Detach
+ * everything again.
+ */
+void serial_test_tc_links_revision(void)
+{
+	test_tc_links_revision_target(BPF_TCX_INGRESS);
+	test_tc_links_revision_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_chain_classic(int target, bool chain_tc_old)
+{
+	LIBBPF_OPTS(bpf_tc_opts, tc_opts, .handle = 1, .priority = 1);
+	LIBBPF_OPTS(bpf_tc_hook, tc_hook, .ifindex = loopback);
+	bool hook_created = false, tc_attached = false;
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	__u32 pid1, pid2, pid3;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	if (chain_tc_old) {
+		tc_hook.attach_point = target == BPF_TCX_INGRESS ?
+				       BPF_TC_INGRESS : BPF_TC_EGRESS;
+		err = bpf_tc_hook_create(&tc_hook);
+		if (err == 0)
+			hook_created = true;
+		err = err == -EEXIST ? 0 : err;
+		if (!ASSERT_OK(err, "bpf_tc_hook_create"))
+			goto cleanup;
+
+		tc_opts.prog_fd = bpf_program__fd(skel->progs.tc3);
+		err = bpf_tc_attach(&tc_hook, &tc_opts);
+		if (!ASSERT_OK(err, "bpf_tc_attach"))
+			goto cleanup;
+		tc_attached = true;
+	}
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, chain_tc_old, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__detach(skel->links.tc2);
+	if (!ASSERT_OK(err, "prog_detach"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, chain_tc_old, "seen_tc3");
+cleanup:
+	if (tc_attached) {
+		tc_opts.flags = tc_opts.prog_fd = tc_opts.prog_id = 0;
+		err = bpf_tc_detach(&tc_hook, &tc_opts);
+		ASSERT_OK(err, "bpf_tc_detach");
+	}
+	if (hook_created) {
+		tc_hook.attach_point = BPF_TC_INGRESS | BPF_TC_EGRESS;
+		bpf_tc_hook_destroy(&tc_hook);
+	}
+	assert_mprog_count(target, 1);
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches two links to ingress/egress through the new
+ * API and one prog via classic API. Traffic runs through and it
+ * validates that the program has been executed. One of the two
+ * links gets removed and test is rerun again. Detach everything
+ * at the end.
+ */
+void serial_test_tc_links_chain_classic(void)
+{
+	test_tc_chain_classic(BPF_TCX_INGRESS, false);
+	test_tc_chain_classic(BPF_TCX_EGRESS, false);
+	test_tc_chain_classic(BPF_TCX_INGRESS, true);
+	test_tc_chain_classic(BPF_TCX_EGRESS, true);
+}
+
+static void test_tc_links_replace_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, pid3, lid1, lid2;
+	__u32 prog_flags[4], link_flags[4];
+	__u32 prog_ids[4], link_ids[4];
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	optl.expected_revision = 1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_FIRST | BPF_F_ID;
+	optl.relative_id = pid1;
+	optl.expected_revision = 2;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	optl.flags = BPF_F_REPLACE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	optl.expected_revision = 3;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 2);
+
+	optl.flags = BPF_F_REPLACE | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc2);
+	optl.expected_revision = 3;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 2);
+
+	optl.flags = BPF_F_REPLACE | BPF_F_LINK | BPF_F_FIRST | BPF_F_ID;
+	optl.relative_id = lid2;
+	optl.expected_revision = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 2);
+
+	err = bpf_link__update_program(skel->links.tc2, skel->progs.tc3);
+	if (!ASSERT_OK(err, "link_update"))
+		goto cleanup;
+
+	assert_mprog_count(target, 2);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 4, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid3, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__detach(skel->links.tc2);
+	if (!ASSERT_OK(err, "link_detach"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__update_program(skel->links.tc1, skel->progs.tc1);
+	if (!ASSERT_OK(err, "link_update_self"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress and validates
+ * replacement in combination with various ops/flags. Similar
+ * for later detachment.
+ */
+void serial_test_tc_links_replace(void)
+{
+	test_tc_links_replace_target(BPF_TCX_INGRESS);
+	test_tc_links_replace_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_invalid_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, lid1;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_AFTER;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_AFTER | BPF_F_ID;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_ID;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_LAST | BPF_F_BEFORE;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST | BPF_F_AFTER;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_FIRST | BPF_F_LAST | BPF_F_LINK;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_ID;
+	optl.relative_id = pid2;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	optl.relative_fd = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = ~0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_ID;
+	optl.relative_id = pid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = lid1;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test invalid flag combinations when attaching/detaching a
+ * link.
+ */
+void serial_test_tc_links_invalid(void)
+{
+	test_tc_links_invalid_target(BPF_TCX_INGRESS);
+	test_tc_links_invalid_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_prepend_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[5], link_flags[5];
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_BEFORE;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	optl.flags = BPF_F_FIRST;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	optl.flags = BPF_F_BEFORE;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid3, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid3, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], BPF_F_FIRST, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid4, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid4, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid2, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], lid2, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid1, "prog_ids[3]");
+	ASSERT_EQ(optq.prog_attach_flags[3], 0, "prog_flags[3]");
+	ASSERT_EQ(optq.link_ids[3], lid1, "link_ids[3]");
+	ASSERT_EQ(optq.link_attach_flags[3], 0, "link_flags[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.prog_attach_flags[4], 0, "prog_flags[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+	ASSERT_EQ(optq.link_attach_flags[4], 0, "link_flags[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with before flag
+ * set and no fd/id, validates prepend behavior that the link got
+ * attached in the right location. Detaches everything.
+ */
+void serial_test_tc_links_prepend(void)
+{
+	test_tc_links_prepend_target(BPF_TCX_INGRESS);
+	test_tc_links_prepend_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_append_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_flags[5], link_flags[5];
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = 0;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.flags = BPF_F_AFTER;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	optl.flags = BPF_F_LAST;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	optl.flags = BPF_F_AFTER;
+	link = bpf_program__attach_tcx_opts(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(prog_flags, 0, sizeof(prog_flags));
+	memset(link_ids, 0, sizeof(link_ids));
+	memset(link_flags, 0, sizeof(link_flags));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.prog_attach_flags[0], 0, "prog_flags[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.link_attach_flags[0], 0, "link_flags[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.prog_attach_flags[1], 0, "prog_flags[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.link_attach_flags[1], 0, "link_flags[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid4, "prog_ids[2]");
+	ASSERT_EQ(optq.prog_attach_flags[2], 0, "prog_flags[2]");
+	ASSERT_EQ(optq.link_ids[2], lid4, "link_ids[2]");
+	ASSERT_EQ(optq.link_attach_flags[2], 0, "link_flags[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid3, "prog_ids[3]");
+	ASSERT_EQ(optq.prog_attach_flags[3], 0, "prog_flags[3]");
+	ASSERT_EQ(optq.link_ids[3], lid3, "link_ids[3]");
+	ASSERT_EQ(optq.link_attach_flags[3], BPF_F_LAST, "link_flags[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.prog_attach_flags[4], 0, "prog_flags[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+	ASSERT_EQ(optq.link_attach_flags[4], 0, "link_flags[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+/* Test:
+ *
+ * Test which attaches a link to ingress/egress with after flag
+ * set and no fd/id, validates append behavior that the link got
+ * attached in the right location. Detaches everything.
+ */
+void serial_test_tc_links_append(void)
+{
+	test_tc_links_append_target(BPF_TCX_INGRESS);
+	test_tc_links_append_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_dev_cleanup_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, pid3, pid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err, ifindex;
+
+	ASSERT_OK(system("ip link add dev tcx_opts1 type veth peer name tcx_opts2"), "add veth");
+	ifindex = if_nametoindex("tcx_opts1");
+	ASSERT_NEQ(ifindex, 0, "non_zero_ifindex");
+	optl.ifindex = ifindex;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc1 = link;
+	assert_mprog_count_ifindex(ifindex, target, 1);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc2 = link;
+	assert_mprog_count_ifindex(ifindex, target, 2);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc3 = link;
+	assert_mprog_count_ifindex(ifindex, target, 3);
+
+	link = bpf_program__attach_tcx_opts(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+	skel->links.tc4 = link;
+	assert_mprog_count_ifindex(ifindex, target, 4);
+
+	ASSERT_OK(system("ip link del dev tcx_opts1"), "del veth");
+	ASSERT_EQ(if_nametoindex("tcx_opts1"), 0, "dev1_removed");
+	ASSERT_EQ(if_nametoindex("tcx_opts2"), 0, "dev2_removed");
+
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc1)), 0, "tc1_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc2)), 0, "tc2_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc3)), 0, "tc3_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc4)), 0, "tc4_ifindex");
+
+	test_tc_link__destroy(skel);
+	return;
+cleanup:
+	test_tc_link__destroy(skel);
+
+	ASSERT_OK(system("ip link del dev tcx_opts1"), "del veth");
+	ASSERT_EQ(if_nametoindex("tcx_opts1"), 0, "dev1_removed");
+	ASSERT_EQ(if_nametoindex("tcx_opts2"), 0, "dev2_removed");
+}
+
+/* Test:
+ *
+ * Test which attaches links to ingress/egress on a newly created
+ * device. Removes the device with attached links. Check links are
+ * indetached state.
+ */
+void serial_test_tc_links_dev_cleanup(void)
+{
+	test_tc_links_dev_cleanup_target(BPF_TCX_INGRESS);
+	test_tc_links_dev_cleanup_target(BPF_TCX_EGRESS);
+}
Signature · 1 line
2.34.1