Skip to contentSkip to conversation

BPF link support for tc BPF programs

PATCH Superseded Daniel Borkmann 3 participants 27 messages Jul 7, 2023 – Jul 10, 2023

Subsystems (10)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Daniel Borkmann

author maintainer
#0 lore raw

v3 cover BPF link support for tc BPF programs

This series adds BPF link support for tc BPF programs. We initially presented the motivation, related work and design at last year's LPC conference in the networking & BPF track [0], and a recent update on our progress of the rework during this year's LSF/MM/BPF summit [1]. The main changes are in first two patches and the last two have an extensive batch of test cases we developed along with it, please see individual patches for details. We tested this series with tc-testing selftest suite as well as BPF CI/selftests. Thanks!

v2 -> v3:
- Removal of BPF_F_FIRST/BPF_F_LAST from control UAPI (Toke, Stan)
- Along with that full rework of bpf_mprog internals to simplify dependency management, looks much nicer now imho
- Just single bpf_mprog_cp instead of two (Andrii)
- atomic64_t for revision counter (Andrii)
- Evaluate target position and reject on conflicts (Andrii)
- Keep track of actual count in bpf_mprob_bundle (Andrii)
- Make combo of REPLACE and BEFORE/AFTER work (Andrii)
- Moved miniq as first struct member (Jamal)
- Rework tcx_link_attach with regards to rtnl (Jakub, Andrii)
- Moved wrappers after bpf_prog_detach_ops (Andrii)
- Removed union for relative_fd and friends for opts and link in libbpf (Andrii)
- Add doc comments to attach/detach/query libbpf APIs (Andrii)
- Dropped SEC_ATTACHABLE_OPT (Andrii)
- Add an OPTS_ZEROED check to bpf_link_create (Andrii)
- Keep opts as the last argument in bpf_program_attach_fd (Andrii)
- Rework bpf_program_attach_fd (Andrii)
- Remove OPTS_GET before we checked OPTS_VALID in bpf_program__attach_tcx (Andrii)
- Add size_t :0; to prevent compiler from leaving garbage (Andrii)
- Add helper macro to clear opts structs which I found useful when writing tests
- Rework of both opts and link test cases to accommodate for changes

[0] https://lpc.events/event/16/contributions/1353/
[1] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

Daniel Borkmann (8):
bpf: Add generic attach/detach/query API for multi-progs
bpf: Add fd-based tcx multi-prog infra with link support
libbpf: Add opts-based attach/detach/query API for tcx
libbpf: Add link-based API for tcx
libbpf: Add helper macro to clear opts structs
bpftool: Extend net dump with tcx progs selftests/bpf: Add mprog API tests for BPF tcx opts selftests/bpf: Add mprog API tests for BPF tcx links

 MAINTAINERS                                   |    5 +-
 include/linux/bpf_mprog.h                     |  242 ++
 include/linux/netdevice.h                     |   15 +-
 include/linux/skbuff.h                        |    4 +-
 include/net/sch_generic.h                     |    2 +-
 include/net/tcx.h                             |  183 ++
 include/uapi/linux/bpf.h                      |   70 +-
 kernel/bpf/Kconfig                            |    1 +
 kernel/bpf/Makefile                           |    3 +-
 kernel/bpf/mprog.c                            |  429 ++++
 kernel/bpf/syscall.c                          |   83 +-
 kernel/bpf/tcx.c                              |  351 +++
 net/Kconfig                                   |    5 +
 net/core/dev.c                                |  267 +-
 net/core/filter.c                             |    4 +-
 net/sched/Kconfig                             |    4 +-
 net/sched/sch_ingress.c                       |   61 +-
 tools/bpf/bpftool/net.c                       |   86 +-
 tools/include/uapi/linux/bpf.h                |   70 +-
 tools/lib/bpf/bpf.c                           |  124 +-
 tools/lib/bpf/bpf.h                           |   97 +-
 tools/lib/bpf/libbpf.c                        |   74 +-
 tools/lib/bpf/libbpf.h                        |   16 +
 tools/lib/bpf/libbpf.map                      |    2 +
 tools/lib/bpf/libbpf_common.h                 |   11 +
 .../selftests/bpf/prog_tests/tc_helpers.h     |   72 +
 .../selftests/bpf/prog_tests/tc_links.c       | 1604 ++++++++++++
 .../selftests/bpf/prog_tests/tc_opts.c        | 2182 +++++++++++++++++
 .../selftests/bpf/progs/test_tc_link.c        |   40 +
 29 files changed, 5878 insertions(+), 229 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/mprog.c
 create mode 100644 kernel/bpf/tcx.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_links.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_opts.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_tc_link.c
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v3 1/8 bpf: Add generic attach/detach/query API for multi-progs

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating internally about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle; the rationale for id is so that user space application does not need CAP_SYS_ADMIN to retrieve foreign fds via bpf_*_get_fd_by_id()
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for attaching
- Enforced only at attach time
- BPF_F_REPLACE with replace_bpf_fd which can be prog, links have their own infra for replacing their internal prog
- If no flags are set, then it's default append behavior for attaching
- Internal revision counter and optionally being able to pass expected_revision
- User space application can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure). The bpf_mprog_cp (control-path structure) is part of bpf_mprog_bundle. Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficiency. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF.

The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggybacks on rtnl.

An extensive test suite for checking all aspects of this API for prog-based attach/detach and link API comes as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management.

[0] https://lore.kernel.org/bpf/[email protected]
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 233 ++++++++++++++++++
 include/uapi/linux/bpf.h       |  36 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 429 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  36 ++-
 6 files changed, 720 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c
MAINTAINERS +1 -0
index acbe54087d1c..7e5ba799d1c5 100644
@@ -3736,6 +3736,7 @@ F:	include/linux/filter.h
 F:	include/linux/tnum.h
 F:	kernel/bpf/core.c
 F:	kernel/bpf/dispatcher.c
+F:	kernel/bpf/mprog.c
 F:	kernel/bpf/syscall.c
 F:	kernel/bpf/tnum.c
 F:	kernel/bpf/trampoline.c
include/linux/bpf_mprog.h new file +233 -0

233 lines not on this page · raw message

include/uapi/linux/bpf.h +28 -8
index 60a9d59beeab..74879c538f2b 100644
@@ -1103,7 +1103,12 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_ID		(1U << 5)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1434,14 +1439,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		__u32		replace_bpf_fd;
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+		};
+		__u64		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1487,16 +1497,26 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		:32;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
+		__u64		revision;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
kernel/bpf/Makefile +1 -1
index 1d3892168d32..1bea2eb912cd 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += disasm.o
+obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
 obj-$(CONFIG_BPF_JIT) += dispatcher.o
kernel/bpf/mprog.c new file +429 -0

429 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +28 -8
index 60a9d59beeab..74879c538f2b 100644
@@ -1103,7 +1103,12 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_ID		(1U << 5)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1434,14 +1439,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		__u32		replace_bpf_fd;
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+		};
+		__u64		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1487,16 +1497,26 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		:32;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
+		__u64		revision;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 180 lines · Daniel Borkmann

This adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution. In-kernel users of the bpf_mprog don't need to care about the dependency resolution internals, they can just consume it with few API calls.

The initial idea of having a generic API sparked out of discussion [0] from an earlier revision of this work where tc's priority was reused and exposed via BPF uapi as a way to coordinate dependencies among tc BPF programs, similar as-is for classic tc BPF. The feedback was that priority provides a bad user experience and is hard to use [1], e.g.:

I cannot help but feel that priority logic copy-paste from old tc, netfilter and friends is done because "that's how things were done in the past". [...] Priority gets exposed everywhere in uapi all the way to bpftool when it's right there for users to understand. And that's the main problem with it.

The user don't want to and don't need to be aware of it, but uapi forces them to pick the priority. [...] Your cover letter [0] example proves that in real life different service pick the same priority. They simply don't know any better. Priority is an unnecessary magic that apps _have_ to pick, so they just copy-paste and everyone ends up using the same.

The course of the discussion showed more and more the need for a generic, reusable API where the "same look and feel" can be applied for various other program types beyond just tc BPF, for example XDP today does not have multi- program support in kernel, but also there was interest around this API for improving management of cgroup program types. Such common multi-program management concept is useful for BPF management daemons or user space BPF applications coordinating internally about their attachments.

Both from Cilium and Meta side [2], we've collected the following requirements for a generic attach/detach/query API for multi-progs which has been implemented as part of this work:

- Support prog-based attach/detach and link API
- Dependency directives (can also be combined):
- BPF_F_{BEFORE,AFTER} with relative_{fd,id} which can be {prog,link,none}
- BPF_F_ID flag as {fd,id} toggle; the rationale for id is so that user space application does not need CAP_SYS_ADMIN to retrieve foreign fds via bpf_*_get_fd_by_id()
- BPF_F_LINK flag as {prog,link} toggle
- If relative_{fd,id} is none, then BPF_F_BEFORE will just prepend, and BPF_F_AFTER will just append for attaching
- Enforced only at attach time
- BPF_F_REPLACE with replace_bpf_fd which can be prog, links have their own infra for replacing their internal prog
- If no flags are set, then it's default append behavior for attaching
- Internal revision counter and optionally being able to pass expected_revision
- User space application can query current state with revision, and pass it along for attachment to assert current state before doing updates
- Query also gets extension for link_ids array and link_attach_flags:
- prog_ids are always filled with program IDs
- link_ids are filled with link IDs when link was used, otherwise 0
- {prog,link}_attach_flags for holding {prog,link}-specific flags
- Must be easy to integrate/reuse for in-kernel users

The uapi-side changes needed for supporting bpf_mprog are rather minimal, consisting of the additions of the attachment flags, revision counter, and expanding existing union with relative_{fd,id} member.

The bpf_mprog framework consists of an bpf_mprog_entry object which holds an array of bpf_mprog_fp (fast-path structure). The bpf_mprog_cp (control-path structure) is part of bpf_mprog_bundle. Both have been separated, so that fast-path gets efficient packing of bpf_prog pointers for maximum cache efficiency. Also, array has been chosen instead of linked list or other structures to remove unnecessary indirections for a fast point-to-entry in tc for BPF.

The bpf_mprog_entry comes as a pair via bpf_mprog_bundle so that in case of updates the peer bpf_mprog_entry is populated and then just swapped which avoids additional allocations that could otherwise fail, for example, in detach case. bpf_mprog_{fp,cp} arrays are currently static, but they could be converted to dynamic allocation if necessary at a point in future. Locking is deferred to the in-kernel user of bpf_mprog, for example, in case of tcx which uses this API in the next patch, it piggybacks on rtnl.

An extensive test suite for checking all aspects of this API for prog-based attach/detach and link API comes as BPF selftests in this series.

Kudos also to Andrii Nakryiko for API discussions wrt Meta's BPF management.

[0] https://lore.kernel.org/bpf/[email protected]
[1] https://lore.kernel.org/bpf/CAADnVQ+gEY3FjCR=+DmjDR4gp5bOYZUFJQXj4agKFHT9CQPZBw@mail.gmail.com
[2] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf

---
 MAINTAINERS                    |   1 +
 include/linux/bpf_mprog.h      | 233 ++++++++++++++++++
 include/uapi/linux/bpf.h       |  36 ++-
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/mprog.c             | 429 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  36 ++-
 6 files changed, 720 insertions(+), 17 deletions(-)
 create mode 100644 include/linux/bpf_mprog.h
 create mode 100644 kernel/bpf/mprog.c
MAINTAINERS +1 -0
index acbe54087d1c..7e5ba799d1c5 100644
@@ -3736,6 +3736,7 @@ F:	include/linux/filter.h
 F:	include/linux/tnum.h
 F:	kernel/bpf/core.c
 F:	kernel/bpf/dispatcher.c
+F:	kernel/bpf/mprog.c
 F:	kernel/bpf/syscall.c
 F:	kernel/bpf/tnum.c
 F:	kernel/bpf/trampoline.c
include/linux/bpf_mprog.h new file +65 -0
new file mode 100644
index 000000000000..a7a9b98df1ef
@@ -0,0 +1,233 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef __BPF_MPROG_H
+#define __BPF_MPROG_H
+
+#include <linux/bpf.h>
+
+#define BPF_MPROG_KEEP	0
+#define BPF_MPROG_SWAP	1
+#define BPF_MPROG_FREE	2
+
+#define BPF_MPROG_MAX	64
+
+#define bpf_mprog_foreach_tuple(entry, fp, cp, t)			\
+	for (fp = &entry->fp_items[0], cp = &entry->parent->cp_items[0];\
+	     ({								\
+		t.prog = READ_ONCE(fp->prog);				\
+		t.link = cp->link;					\
+		t.prog;							\
+	      });							\
+	     fp++, cp++)
+
+#define bpf_mprog_foreach_prog(entry, fp, p)				\
+	for (fp = &entry->fp_items[0];					\
+	     (p = READ_ONCE(fp->prog));					\
+	     fp++)
+
+struct bpf_mprog_fp {
+	struct bpf_prog *prog;
+};
+
+struct bpf_mprog_cp {
+	struct bpf_link *link;
+};
+
+struct bpf_mprog_entry {
+	struct bpf_mprog_fp fp_items[BPF_MPROG_MAX];
+	struct bpf_mprog_bundle *parent;
+};
+
+struct bpf_mprog_bundle {
+	struct bpf_mprog_entry a;
+	struct bpf_mprog_entry b;
+	struct bpf_mprog_cp cp_items[BPF_MPROG_MAX];
+	struct rcu_head rcu;
+	struct bpf_prog *ref;
+	atomic64_t revision;
+	off_t off;
+	u32 count;
+};
+
+struct bpf_tuple {
+	struct bpf_prog *prog;
+	struct bpf_link *link;
+};
+
+static inline struct bpf_mprog_entry *
+bpf_mprog_peer(const struct bpf_mprog_entry *entry)
+{
+	if (entry == &entry->parent->a)
+		return &entry->parent->b;
+	else
+		return &entry->parent->a;
+}
+

[..]

Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+	struct bpf_mprog_bundle *bundle;
+	void *ptr;
+
+	BUILD_BUG_ON(size < sizeof(*bundle) + off);
+	BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+	BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+		     ARRAY_SIZE(bundle->cp_items));
+
+	ptr = kzalloc(size, GFP_KERNEL);
+	if (ptr) {
+		bundle = ptr + off;
+		atomic64_set(&bundle->revision, 1);
+		bundle->off = off;
+		bundle->a.parent = bundle;
+		bundle->b.parent = bundle;
+		return &bundle->a;
+	}
+	return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+	struct bpf_mprog_bundle *bundle = entry->parent;
+
+	call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

Quoted 44 lines
include/linux/bpf_mprog.h new file +44 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+				      struct bpf_tuple *tuple)
+{
+	WARN_ON_ONCE(entry->parent->ref);
+	if (!tuple->link)
+		entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+	return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+	int total = entry->parent->count;
+
+	WARN_ON_ONCE(total > bpf_mprog_max());
+	return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+				    struct bpf_prog *prog)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *tmp;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp) {
+		if (tmp == prog)
+			return true;
+	}
+	return false;
+}
+

include/linux/bpf_mprog.h line 151

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+	return code == BPF_MPROG_SWAP ||
+	       code == BPF_MPROG_FREE;
+}
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+	atomic64_inc(&entry->parent->revision);
+	synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Quoted 214 lines
include/linux/bpf_mprog.h new file +82 -0
+	if (entry->parent->ref) {
+		bpf_prog_put(entry->parent->ref);
+		entry->parent->ref = NULL;
+	}
+}
+
+static inline u64 bpf_mprog_revision(struct bpf_mprog_entry *entry)
+{
+	return atomic64_read(&entry->parent->revision);
+}
+
+static inline void bpf_mprog_entry_clear(struct bpf_mprog_entry *entry)
+{
+	memset(entry->fp_items, 0, sizeof(entry->fp_items));
+}
+
+static inline void bpf_mprog_commit_cp(struct bpf_mprog_entry *entry,
+				       struct bpf_mprog_cp *cp_items)
+{
+	memcpy(entry->parent->cp_items, cp_items,
+	       sizeof(entry->parent->cp_items));
+}
+
+static inline void bpf_mprog_read_fp(struct bpf_mprog_entry *entry, u32 idx,
+				     struct bpf_mprog_fp **fp)
+{
+	*fp = &entry->fp_items[idx];
+}
+
+static inline void bpf_mprog_read_cp(struct bpf_mprog_entry *entry, u32 idx,
+				     struct bpf_mprog_cp **cp)
+{
+	*cp = &entry->parent->cp_items[idx];
+}
+
+static inline void bpf_mprog_read(struct bpf_mprog_entry *entry, u32 idx,
+				  struct bpf_mprog_fp **fp,
+				  struct bpf_mprog_cp **cp)
+{
+	bpf_mprog_read_fp(entry, idx, fp);
+	bpf_mprog_read_cp(entry, idx, cp);
+}
+
+static inline void bpf_mprog_write_fp(struct bpf_mprog_fp *fp,
+				      struct bpf_tuple *tuple)
+{
+	WRITE_ONCE(fp->prog, tuple->prog);
+}
+
+static inline void bpf_mprog_write_cp(struct bpf_mprog_cp *cp,
+				      struct bpf_tuple *tuple)
+{
+	cp->link = tuple->link;
+}
+
+static inline void bpf_mprog_write(struct bpf_mprog_fp *fp,
+				   struct bpf_mprog_cp *cp,
+				   struct bpf_tuple *tuple)
+{
+	bpf_mprog_write_fp(fp, tuple);
+	bpf_mprog_write_cp(cp, tuple);
+}
+
+static inline void bpf_mprog_copy(struct bpf_mprog_fp *fp_dst,
+				  struct bpf_mprog_cp *cp_dst,
+				  struct bpf_mprog_fp *fp_src,
+				  struct bpf_mprog_cp *cp_src)
+{
+	WRITE_ONCE(fp_dst->prog, READ_ONCE(fp_src->prog));
+	memcpy(cp_dst, cp_src, sizeof(*cp_src));
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog_new,
+		     struct bpf_link *link, struct bpf_prog *prog_old,
+		     u32 flags, u32 object, u64 revision);
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object, u64 revision);
+
+int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
+		    struct bpf_mprog_entry *entry);
+
+#endif /* __BPF_MPROG_H */
include/uapi/linux/bpf.h +28 -8
index 60a9d59beeab..74879c538f2b 100644
@@ -1103,7 +1103,12 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_ID		(1U << 5)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1434,14 +1439,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		__u32		replace_bpf_fd;
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+		};
+		__u64		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1487,16 +1497,26 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		:32;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
+		__u64		revision;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
kernel/bpf/Makefile +1 -1
index 1d3892168d32..1bea2eb912cd 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += disasm.o
+obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
 obj-$(CONFIG_BPF_JIT) += trampoline.o
 obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o
 obj-$(CONFIG_BPF_JIT) += dispatcher.o
kernel/bpf/mprog.c new file +41 -0
new file mode 100644
index 000000000000..8795dab5237a
@@ -0,0 +1,429 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+
+#include <linux/bpf.h>
+#include <linux/bpf_mprog.h>
+
+static int bpf_mprog_link(struct bpf_tuple *tuple,
+			  u32 object, u32 flags,
+			  enum bpf_prog_type type)
+{
+	bool id = flags & BPF_F_ID;
+	struct bpf_link *link;
+
+	if (id)
+		link = bpf_link_by_id(object);
+	else
+		link = bpf_link_get_from_fd(object);
+	if (IS_ERR(link))
+		return PTR_ERR(link);
+	if (type && link->prog->type != type) {
+		bpf_link_put(link);
+		return -EINVAL;
+	}
+
+	tuple->link = link;
+	tuple->prog = link->prog;
+	return 0;
+}
+
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+			  u32 object, u32 flags,
+			  enum bpf_prog_type type)
+{
+	bool id = flags & BPF_F_ID;
+	struct bpf_prog *prog;
+
+	if (id)
+		prog = bpf_prog_by_id(object);
+	else
+		prog = bpf_prog_get(object);
+	if (IS_ERR(prog)) {

[..]

kernel/bpf/mprog.c

Quoted 2 lines
kernel/bpf/mprog.c new file +2 -0
+		if (!object && !id)
+			return 0;

What's the reason behind this?

Quoted 162 lines
kernel/bpf/mprog.c new file +162 -0
+		return PTR_ERR(prog);
+	}
+	if (type && prog->type != type) {
+		bpf_prog_put(prog);
+		return -EINVAL;
+	}
+
+	tuple->link = NULL;
+	tuple->prog = prog;
+	return 0;
+}
+
+static int bpf_mprog_tuple_relative(struct bpf_tuple *tuple,
+				    u32 object, u32 flags,
+				    enum bpf_prog_type type)
+{
+	memset(tuple, 0, sizeof(*tuple));
+	if (flags & BPF_F_LINK)
+		return bpf_mprog_link(tuple, object, flags, type);
+	return bpf_mprog_prog(tuple, object, flags, type);
+}
+
+static void bpf_mprog_tuple_put(struct bpf_tuple *tuple)
+{
+	if (tuple->link)
+		bpf_link_put(tuple->link);
+	else if (tuple->prog)
+		bpf_prog_put(tuple->prog);
+}
+
+static int bpf_mprog_replace(struct bpf_mprog_entry *entry,
+			     struct bpf_tuple *ntuple, int idx)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	struct bpf_prog *oprog;
+
+	bpf_mprog_read(entry, idx, &fp, &cp);
+	oprog = READ_ONCE(fp->prog);
+	bpf_mprog_write(fp, cp, ntuple);
+	if (!ntuple->link) {
+		WARN_ON_ONCE(cp->link);
+		bpf_prog_put(oprog);
+	}
+	return BPF_MPROG_KEEP;
+}
+
+static int bpf_mprog_insert(struct bpf_mprog_entry *entry,
+			    struct bpf_tuple *ntuple, int idx, u32 flags)
+{
+	int i, j = 0, total = bpf_mprog_total(entry);
+	struct bpf_mprog_cp *cp, cpp[BPF_MPROG_MAX] = {};
+	struct bpf_mprog_fp *fp, *fpp;
+	struct bpf_mprog_entry *peer;
+
+	peer = bpf_mprog_peer(entry);
+	bpf_mprog_entry_clear(peer);
+	if (idx < 0) {
+		bpf_mprog_read_fp(peer, j, &fpp);
+		bpf_mprog_write_fp(fpp, ntuple);
+		bpf_mprog_write_cp(&cpp[j], ntuple);
+		j++;
+	}
+	for (i = 0; i <= total; i++) {
+		bpf_mprog_read_fp(peer, j, &fpp);
+		if (idx == i && (flags & BPF_F_AFTER)) {
+			bpf_mprog_write(fpp, &cpp[j], ntuple);
+			j++;
+			bpf_mprog_read_fp(peer, j, &fpp);
+		}
+		if (i < total) {
+			bpf_mprog_read(entry, i, &fp, &cp);
+			bpf_mprog_copy(fpp, &cpp[j], fp, cp);
+			j++;
+		}
+		if (idx == i && (flags & BPF_F_BEFORE)) {
+			bpf_mprog_read_fp(peer, j, &fpp);
+			bpf_mprog_write(fpp, &cpp[j], ntuple);
+			j++;
+		}
+	}
+	bpf_mprog_commit_cp(peer, cpp);
+	bpf_mprog_inc(peer);
+	return BPF_MPROG_SWAP;
+}
+
+static int bpf_mprog_tuple_confirm(struct bpf_mprog_entry *entry,
+				   struct bpf_tuple *dtuple, int idx)
+{
+	int first = 0, last = bpf_mprog_total(entry) - 1;
+	struct bpf_mprog_cp *cp;
+	struct bpf_mprog_fp *fp;
+	struct bpf_prog *prog;
+	struct bpf_link *link;
+
+	if (idx <= first)
+		bpf_mprog_read(entry, first, &fp, &cp);
+	else if (idx >= last)
+		bpf_mprog_read(entry, last, &fp, &cp);
+	else
+		bpf_mprog_read(entry, idx, &fp, &cp);
+
+	prog = READ_ONCE(fp->prog);
+	link = cp->link;
+	if (!dtuple->link && link)
+		return -EBUSY;
+
+	WARN_ON_ONCE(dtuple->prog && dtuple->prog != prog);
+	WARN_ON_ONCE(dtuple->link && dtuple->link != link);
+
+	dtuple->prog = prog;
+	dtuple->link = link;
+	return 0;
+}
+
+static int bpf_mprog_delete(struct bpf_mprog_entry *entry,
+			    struct bpf_tuple *dtuple, int idx)
+{
+	int i = 0, j, ret, total = bpf_mprog_total(entry);
+	struct bpf_mprog_cp *cp, cpp[BPF_MPROG_MAX] = {};
+	struct bpf_mprog_fp *fp, *fpp;
+	struct bpf_mprog_entry *peer;
+
+	ret = bpf_mprog_tuple_confirm(entry, dtuple, idx);
+	if (ret)
+		return ret;
+	peer = bpf_mprog_peer(entry);
+	bpf_mprog_entry_clear(peer);
+	if (idx < 0)
+		i++;
+	if (idx == total)
+		total--;
+	for (j = 0; i < total; i++) {
+		if (idx == i)
+			continue;
+		bpf_mprog_read_fp(peer, j, &fpp);
+		bpf_mprog_read(entry, i, &fp, &cp);
+		bpf_mprog_copy(fpp, &cpp[j], fp, cp);
+		j++;
+	}
+	bpf_mprog_commit_cp(peer, cpp);
+	bpf_mprog_dec(peer);
+	bpf_mprog_mark_ref(peer, dtuple);
+	return bpf_mprog_total(peer) ?
+	       BPF_MPROG_SWAP : BPF_MPROG_FREE;
+}
+
+static int bpf_mprog_pos_exact(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog))
+			return tuple->link == cp->link ? i : -EBUSY;
+	}
+	return -ENOENT;
+}
+

kernel/bpf/mprog.c line 215

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+static int bpf_mprog_pos_before(struct bpf_mprog_entry *entry,
+				struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&

Both attach/detach happen under rtnl, why do need READ_ONCE? I'm assuming even going forwrad, attach/detach from non-tcx places will happen under lock?

(same for bpf_mprog_pos_before/bpf_mprog_pos_after)

Feels like the only place where we need WRITE_ONCE is the replace (in-place) and READ_ONCE during fast-path. Why do we need the rest?

Quoted 52 lines
kernel/bpf/mprog.c new file +52 -0
+		    (!tuple->link || tuple->link == cp->link))
+			return i - 1;
+	}
+	return tuple->prog ? -ENOENT : -1;
+}
+
+static int bpf_mprog_pos_after(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&
+		    (!tuple->link || tuple->link == cp->link))
+			return i + 1;
+	}
+	return tuple->prog ? -ENOENT : bpf_mprog_total(entry);
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog_new,
+		     struct bpf_link *link, struct bpf_prog *prog_old,
+		     u32 flags, u32 object, u64 revision)
+{
+	struct bpf_tuple rtuple, ntuple = {
+		.prog = prog_new,
+		.link = link,
+	}, otuple = {
+		.prog = prog_old,
+		.link = link,
+	};
+	int ret, idx = -2, tidx;
+
+	if (revision && revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	if (bpf_mprog_exists(entry, prog_new))
+		return -EEXIST;
+	ret = bpf_mprog_tuple_relative(&rtuple, object,
+				       flags & ~BPF_F_REPLACE,
+				       prog_new->type);
+	if (ret)
+		return ret;
+	if (flags & BPF_F_REPLACE) {
+		tidx = bpf_mprog_pos_exact(entry, &otuple);
+		if (tidx < 0) {
+			ret = tidx;
+			goto out;
+		}
+		idx = tidx;
+	}

[..]

Quoted 6 lines
kernel/bpf/mprog.c new file +6 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}

kernel/bpf/mprog.c line 283

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < 0 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}

There still seems to be some inter-dependency between F_BEFORE and F_AFTER? IOW, looks like I can pass F_BEFORE|F_AFTER|F_REPLACE. Do we need that? Why not exclusive cases?

Quoted 221 lines
kernel/bpf/mprog.c new file +146 -0
+	if (idx < -1) {
+		if (rtuple.prog || flags) {
+			ret = -EINVAL;
+			goto out;
+		}
+		idx = bpf_mprog_total(entry);
+		flags = BPF_F_AFTER;
+	}
+	if (idx >= bpf_mprog_max()) {
+		ret = -EDOM;
+		goto out;
+	}
+	if (flags & BPF_F_REPLACE)
+		ret = bpf_mprog_replace(entry, &ntuple, idx);
+	else
+		ret = bpf_mprog_insert(entry, &ntuple, idx, flags);
+out:
+	bpf_mprog_tuple_put(&rtuple);
+	return ret;
+}
+
+int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
+		     struct bpf_link *link, u32 flags, u32 object, u64 revision)
+{
+	struct bpf_tuple rtuple, dtuple = {
+		.prog = prog,
+		.link = link,
+	};
+	int ret, idx = -2, tidx;
+
+	if (flags & BPF_F_REPLACE)
+		return -EINVAL;
+	if (revision && revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	ret = bpf_mprog_tuple_relative(&rtuple, object, flags,
+				       prog ? prog->type :
+				       BPF_PROG_TYPE_UNSPEC);
+	if (ret)
+		return ret;
+	if (dtuple.prog) {
+		tidx = bpf_mprog_pos_exact(entry, &dtuple);
+		if (tidx < 0) {
+			ret = tidx;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < 0 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (idx < -1) {
+		if (rtuple.prog || flags) {
+			ret = -EINVAL;
+			goto out;
+		}
+		idx = bpf_mprog_total(entry);
+		flags = BPF_F_AFTER;
+	}
+	if (idx >= bpf_mprog_max()) {
+		ret = -EDOM;
+		goto out;
+	}
+	ret = bpf_mprog_delete(entry, &dtuple, idx);
+out:
+	bpf_mprog_tuple_put(&rtuple);
+	return ret;
+}
+
+int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
+		    struct bpf_mprog_entry *entry)
+{
+	u32 __user *uprog_flags, *ulink_flags;
+	u32 __user *uprog_id, *ulink_id;
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	struct bpf_prog *prog;
+	const u32 flags = 0;
+	int i, ret = 0;
+	u32 id, count;
+	u64 revision;
+
+	if (attr->query.query_flags || attr->query.attach_flags)
+		return -EINVAL;
+	revision = bpf_mprog_revision(entry);
+	count = bpf_mprog_total(entry);
+	if (copy_to_user(&uattr->query.attach_flags, &flags, sizeof(flags)))
+		return -EFAULT;
+	if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
+		return -EFAULT;
+	if (copy_to_user(&uattr->query.count, &count, sizeof(count)))
+		return -EFAULT;
+	uprog_id = u64_to_user_ptr(attr->query.prog_ids);
+	uprog_flags = u64_to_user_ptr(attr->query.prog_attach_flags);
+	ulink_id = u64_to_user_ptr(attr->query.link_ids);
+	ulink_flags = u64_to_user_ptr(attr->query.link_attach_flags);
+	if (attr->query.count == 0 || !uprog_id || !count)
+		return 0;
+	if (attr->query.count < count) {
+		count = attr->query.count;
+		ret = -ENOSPC;
+	}
+	for (i = 0; i < bpf_mprog_max(); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		prog = READ_ONCE(fp->prog);
+		if (!prog)
+			break;
+		id = prog->aux->id;
+		if (copy_to_user(uprog_id + i, &id, sizeof(id)))
+			return -EFAULT;
+		if (uprog_flags &&
+		    copy_to_user(uprog_flags + i, &flags, sizeof(flags)))
+			return -EFAULT;
+		id = cp->link ? cp->link->id : 0;
+		if (ulink_id &&
+		    copy_to_user(ulink_id + i, &id, sizeof(id)))
+			return -EFAULT;
+		if (ulink_flags &&
+		    copy_to_user(ulink_flags + i, &flags, sizeof(flags)))
+			return -EFAULT;
+		if (i + 1 == count)
+			break;
+	}
+	return ret;
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu)
+{
+	struct bpf_mprog_bundle *bundle =
+		container_of(rcu, struct bpf_mprog_bundle, rcu);
+	void *ptr = (void *)bundle - bundle->off;
+
+	kfree(ptr);
+}
tools/include/uapi/linux/bpf.h +28 -8
index 60a9d59beeab..74879c538f2b 100644
@@ -1103,7 +1103,12 @@ enum bpf_link_type {
  */
 #define BPF_F_ALLOW_OVERRIDE	(1U << 0)
 #define BPF_F_ALLOW_MULTI	(1U << 1)
+/* Generic attachment flags. */
 #define BPF_F_REPLACE		(1U << 2)
+#define BPF_F_BEFORE		(1U << 3)
+#define BPF_F_AFTER		(1U << 4)
+#define BPF_F_ID		(1U << 5)
+#define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
  * verifier will perform strict alignment checking as if the kernel
@@ -1434,14 +1439,19 @@ union bpf_attr {
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
-		__u32		target_fd;	/* container object to attach to */
-		__u32		attach_bpf_fd;	/* eBPF program to attach */
+		union {
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
+		__u32		attach_bpf_fd;
 		__u32		attach_type;
 		__u32		attach_flags;
-		__u32		replace_bpf_fd;	/* previously attached eBPF
-						 * program to replace if
-						 * BPF_F_REPLACE is used
-						 */
+		__u32		replace_bpf_fd;
+		union {
+			__u32	relative_fd;
+			__u32	relative_id;
+		};
+		__u64		expected_revision;
 	};
 
 	struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
@@ -1487,16 +1497,26 @@ union bpf_attr {
 	} info;
 
 	struct { /* anonymous struct used by BPF_PROG_QUERY command */
-		__u32		target_fd;	/* container object to query */
+		union {
+			__u32	target_fd;	/* target object to query or ... */
+			__u32	target_ifindex;	/* target ifindex */
+		};
 		__u32		attach_type;
 		__u32		query_flags;
 		__u32		attach_flags;
 		__aligned_u64	prog_ids;
-		__u32		prog_cnt;
+		union {
+			__u32	prog_cnt;
+			__u32	count;
+		};
+		__u32		:32;
 		/* output: per-program attach_flags.
 		 * not allowed to be set during effective query.
 		 */
 		__aligned_u64	prog_attach_flags;
+		__aligned_u64	link_ids;
+		__aligned_u64	link_attach_flags;
+		__u64		revision;
 	} query;
 
 	struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+	struct bpf_mprog_bundle *bundle;
+	void *ptr;
+
+	BUILD_BUG_ON(size < sizeof(*bundle) + off);
+	BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+	BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+		     ARRAY_SIZE(bundle->cp_items));
+
+	ptr = kzalloc(size, GFP_KERNEL);
+	if (ptr) {
+		bundle = ptr + off;
+		atomic64_set(&bundle->revision, 1);
+		bundle->off = off;
+		bundle->a.parent = bundle;
+		bundle->b.parent = bundle;
+		return &bundle->a;
+	}
+	return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+	struct bpf_mprog_bundle *bundle = entry->parent;
+
+	call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
include/linux/bpf_mprog.h new file +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+				      struct bpf_tuple *tuple)
+{
+	WARN_ON_ONCE(entry->parent->ref);
+	if (!tuple->link)
+		entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+	return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+	int total = entry->parent->count;
+
+	WARN_ON_ONCE(total > bpf_mprog_max());
+	return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+				    struct bpf_prog *prog)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *tmp;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp) {
+		if (tmp == prog)
+			return true;
+	}
+	return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+	return code == BPF_MPROG_SWAP ||
+	       code == BPF_MPROG_FREE;
+}
Quoted 8 lines

include/linux/bpf_mprog.h line 151

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+	atomic64_inc(&entry->parent->revision);
+	synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 15 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+			  u32 object, u32 flags,
+			  enum bpf_prog_type type)
+{
+	bool id = flags & BPF_F_ID;
+	struct bpf_prog *prog;
+
+	if (id)
+		prog = bpf_prog_by_id(object);
+	else
+		prog = bpf_prog_get(object);
+	if (IS_ERR(prog)) {

[..]

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		if (!object && !id)
+			return 0;

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Quoted 11 lines
Quoted 11 lines
kernel/bpf/mprog.c new file +11 -0
+		return PTR_ERR(prog);
+	}
+	if (type && prog->type != type) {
+		bpf_prog_put(prog);
+		return -EINVAL;
+	}
+
+	tuple->link = NULL;
+	tuple->prog = prog;
+	return 0;
+}

[...]

Quoted 15 lines

kernel/bpf/mprog.c line 215

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+static int bpf_mprog_pos_before(struct bpf_mprog_entry *entry,
+				struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&

Both attach/detach happen under rtnl, why do need READ_ONCE? I'm assuming even going forwrad, attach/detach from non-tcx places will happen under lock?

Quoted 4 lines

(same for bpf_mprog_pos_before/bpf_mprog_pos_after)

Feels like the only place where we need WRITE_ONCE is the replace (in-place) and READ_ONCE during fast-path. Why do we need the rest?

Yes, the replace case is via WRITE_ONCE, hence the READ_ONCE annotations. You are saying that for the cases where we are under lock we should just drop the READ_ONCE annotations? I can do that ofc, I thought the general convention was to do the {READ,WRITE}_ONCE consistently for the purpose of documenting fp->prog access.

Quoted 67 lines
Quoted 52 lines
kernel/bpf/mprog.c new file +52 -0
+		    (!tuple->link || tuple->link == cp->link))
+			return i - 1;
+	}
+	return tuple->prog ? -ENOENT : -1;
+}
+
+static int bpf_mprog_pos_after(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&
+		    (!tuple->link || tuple->link == cp->link))
+			return i + 1;
+	}
+	return tuple->prog ? -ENOENT : bpf_mprog_total(entry);
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog_new,
+		     struct bpf_link *link, struct bpf_prog *prog_old,
+		     u32 flags, u32 object, u64 revision)
+{
+	struct bpf_tuple rtuple, ntuple = {
+		.prog = prog_new,
+		.link = link,
+	}, otuple = {
+		.prog = prog_old,
+		.link = link,
+	};
+	int ret, idx = -2, tidx;
+
+	if (revision && revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	if (bpf_mprog_exists(entry, prog_new))
+		return -EEXIST;
+	ret = bpf_mprog_tuple_relative(&rtuple, object,
+				       flags & ~BPF_F_REPLACE,
+				       prog_new->type);
+	if (ret)
+		return ret;
+	if (flags & BPF_F_REPLACE) {
+		tidx = bpf_mprog_pos_exact(entry, &otuple);
+		if (tidx < 0) {
+			ret = tidx;
+			goto out;
+		}
+		idx = tidx;
+	}

[..]

Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < 0 ? tidx : -EDOM;
Quoted 8 lines

kernel/bpf/mprog.c line 267

Quoted 4 lines
kernel/bpf/mprog.c new file +4 -0
+			goto out;
+		}
+		idx = tidx;
+	}

There still seems to be some inter-dependency between F_BEFORE and F_AFTER? IOW, looks like I can pass F_BEFORE|F_AFTER|F_REPLACE. Do we need that? Why not exclusive cases?

I reworked this as per Andrii's suggestion/preference from v2 [0], iow, to calculate target index and bail out if the request cannot be resolved into a common index.

[0] https://lore.kernel.org/bpf/CAEf4BzbsUMnP7WMm3OmJznvD2b03B1qASFRNiDoVAU6XvvTZNA@mail.gmail.com/

Quoted 21 lines
Quoted 21 lines
kernel/bpf/mprog.c new file +21 -0
+	if (idx < -1) {
+		if (rtuple.prog || flags) {
+			ret = -EINVAL;
+			goto out;
+		}
+		idx = bpf_mprog_total(entry);
+		flags = BPF_F_AFTER;
+	}
+	if (idx >= bpf_mprog_max()) {
+		ret = -EDOM;
+		goto out;
+	}
+	if (flags & BPF_F_REPLACE)
+		ret = bpf_mprog_replace(entry, &ntuple, idx);
+	else
+		ret = bpf_mprog_insert(entry, &ntuple, idx, flags);
+out:
+	bpf_mprog_tuple_put(&rtuple);
+	return ret;
+}
+
author maintainer
#4 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 42 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+    struct bpf_mprog_bundle *bundle;
+    void *ptr;
+
+    BUILD_BUG_ON(size < sizeof(*bundle) + off);
+    BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+    BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+             ARRAY_SIZE(bundle->cp_items));
+
+    ptr = kzalloc(size, GFP_KERNEL);
+    if (ptr) {
+        bundle = ptr + off;
+        atomic64_set(&bundle->revision, 1);
+        bundle->off = off;
+        bundle->a.parent = bundle;
+        bundle->b.parent = bundle;
+        return &bundle->a;
+    }
+    return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+    struct bpf_mprog_bundle *bundle = entry->parent;
+
+    call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 6 lines
Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Ok, I moved this into tcx and only left bpf_mprog_bundle_init() in mprog. Looks better indeed, thanks Stan!

#5 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann
Quoted 128 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+	struct bpf_mprog_bundle *bundle;
+	void *ptr;
+
+	BUILD_BUG_ON(size < sizeof(*bundle) + off);
+	BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+	BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+		     ARRAY_SIZE(bundle->cp_items));
+
+	ptr = kzalloc(size, GFP_KERNEL);
+	if (ptr) {
+		bundle = ptr + off;
+		atomic64_set(&bundle->revision, 1);
+		bundle->off = off;
+		bundle->a.parent = bundle;
+		bundle->b.parent = bundle;
+		return &bundle->a;
+	}
+	return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+	struct bpf_mprog_bundle *bundle = entry->parent;
+
+	call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
include/linux/bpf_mprog.h new file +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+				      struct bpf_tuple *tuple)
+{
+	WARN_ON_ONCE(entry->parent->ref);
+	if (!tuple->link)
+		entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+	entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+	return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+	int total = entry->parent->count;
+
+	WARN_ON_ONCE(total > bpf_mprog_max());
+	return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+				    struct bpf_prog *prog)
+{
+	const struct bpf_mprog_fp *fp;
+	const struct bpf_prog *tmp;
+
+	bpf_mprog_foreach_prog(entry, fp, tmp) {
+		if (tmp == prog)
+			return true;
+	}
+	return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+	return code == BPF_MPROG_SWAP ||
+	       code == BPF_MPROG_FREE;
+}
Quoted 8 lines

include/linux/bpf_mprog.h line 151

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+	atomic64_inc(&entry->parent->revision);
+	synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 15 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+			  u32 object, u32 flags,
+			  enum bpf_prog_type type)
+{
+	bool id = flags & BPF_F_ID;
+	struct bpf_prog *prog;
+
+	if (id)
+		prog = bpf_prog_by_id(object);
+	else
+		prog = bpf_prog_get(object);
+	if (IS_ERR(prog)) {

[..]

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		if (!object && !id)
+			return 0;

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

Quoted 29 lines
Quoted 11 lines
Quoted 11 lines
kernel/bpf/mprog.c new file +11 -0
+		return PTR_ERR(prog);
+	}
+	if (type && prog->type != type) {
+		bpf_prog_put(prog);
+		return -EINVAL;
+	}
+
+	tuple->link = NULL;
+	tuple->prog = prog;
+	return 0;
+}

[...]

Quoted 17 lines

kernel/bpf/mprog.c line 215

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+static int bpf_mprog_pos_before(struct bpf_mprog_entry *entry,
+				struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&

Both attach/detach happen under rtnl, why do need READ_ONCE? I'm assuming even going forwrad, attach/detach from non-tcx places will happen under lock?

(same for bpf_mprog_pos_before/bpf_mprog_pos_after)

Quoted 8 lines
Quoted 2 lines

Feels like the only place where we need WRITE_ONCE is the replace (in-place) and READ_ONCE during fast-path. Why do we need the rest?

Yes, the replace case is via WRITE_ONCE, hence the READ_ONCE annotations. You are saying that for the cases where we are under lock we should just drop the READ_ONCE annotations? I can do that ofc, I thought the general convention was to do the {READ,WRITE}_ONCE consistently for the purpose of documenting fp->prog access.

I see, then maybe let's keep them. I was a bit confused because those READ_ONCE are within a locked section so I wasn't sure whether I'm missing something or it's working as intended :-)

Quoted 76 lines
Quoted 67 lines
Quoted 52 lines
kernel/bpf/mprog.c new file +52 -0
+		    (!tuple->link || tuple->link == cp->link))
+			return i - 1;
+	}
+	return tuple->prog ? -ENOENT : -1;
+}
+
+static int bpf_mprog_pos_after(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&
+		    (!tuple->link || tuple->link == cp->link))
+			return i + 1;
+	}
+	return tuple->prog ? -ENOENT : bpf_mprog_total(entry);
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog_new,
+		     struct bpf_link *link, struct bpf_prog *prog_old,
+		     u32 flags, u32 object, u64 revision)
+{
+	struct bpf_tuple rtuple, ntuple = {
+		.prog = prog_new,
+		.link = link,
+	}, otuple = {
+		.prog = prog_old,
+		.link = link,
+	};
+	int ret, idx = -2, tidx;
+
+	if (revision && revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	if (bpf_mprog_exists(entry, prog_new))
+		return -EEXIST;
+	ret = bpf_mprog_tuple_relative(&rtuple, object,
+				       flags & ~BPF_F_REPLACE,
+				       prog_new->type);
+	if (ret)
+		return ret;
+	if (flags & BPF_F_REPLACE) {
+		tidx = bpf_mprog_pos_exact(entry, &otuple);
+		if (tidx < 0) {
+			ret = tidx;
+			goto out;
+		}
+		idx = tidx;
+	}

[..]

Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < 0 ? tidx : -EDOM;
Quoted 8 lines

kernel/bpf/mprog.c line 267

Quoted 4 lines
kernel/bpf/mprog.c new file +4 -0
+			goto out;
+		}
+		idx = tidx;
+	}

There still seems to be some inter-dependency between F_BEFORE and F_AFTER? IOW, looks like I can pass F_BEFORE|F_AFTER|F_REPLACE. Do we need that? Why not exclusive cases?

Quoted 4 lines

I reworked this as per Andrii's suggestion/preference from v2 [0], iow, to calculate target index and bail out if the request cannot be resolved into a common index.

[0] https://lore.kernel.org/bpf/CAEf4BzbsUMnP7WMm3OmJznvD2b03B1qASFRNiDoVAU6XvvTZNA@mail.gmail.com/

SG! Let's maybe put a summary in the header of what the expectation is when combining them?

#6 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 133 lines · Stanislav Fomichev
Quoted 131 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
include/linux/bpf_mprog.h new file +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

include/linux/bpf_mprog.h line 151

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 18 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

No. There is nothing to fix.

#7 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 137 lines · Alexei Starovoitov
Quoted 135 lines · Stanislav Fomichev
Quoted 133 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
include/linux/bpf_mprog.h new file +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

include/linux/bpf_mprog.h line 151

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 19 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Review

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;

What's the reason behind this?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

No. There is nothing to fix.

Care to elaborate? Do we want to preserve it for consistency? Or is there some concern with asking people to put relative_fd=-1 when doing the call?
I'm fine either way; trying to understand where it's coming from. I remember it was discussed briefly at lsfmmbpf, but don't remember the details..

#8 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 146 lines · Stanislav Fomichev
Quoted 143 lines · Alexei Starovoitov
Quoted 140 lines · Stanislav Fomichev
Quoted 128 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
include/linux/bpf_mprog.h new file +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

include/linux/bpf_mprog.h line 96

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
include/linux/bpf_mprog.h new file +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

include/linux/bpf_mprog.h line 151

Quoted 5 lines
include/linux/bpf_mprog.h new file +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 15 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Quoted 7 lines
Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

Quoted 8 lines
Quoted 1 line

No. There is nothing to fix.

Care to elaborate? Do we want to preserve it for consistency? Or is there some concern with asking people to put relative_fd=-1 when doing the call?
I'm fine either way; trying to understand where it's coming from. I remember it was discussed briefly at lsfmmbpf, but don't remember the details..

0 is invalid bpf object (prog, map, link). There is nothing to "fix".

#9 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 150 lines · Alexei Starovoitov
Quoted 148 lines · Stanislav Fomichev
Quoted 144 lines · Alexei Starovoitov
Quoted 133 lines · Stanislav Fomichev
Quoted 131 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
(quoted diff) +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

Review

Quoted 10 lines
(quoted diff) +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
(quoted diff) +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 18 lines
Quoted 12 lines
(quoted diff) +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

No. There is nothing to fix.

Quoted 8 lines
Quoted 6 lines

Care to elaborate? Do we want to preserve it for consistency? Or is there some concern with asking people to put relative_fd=-1 when doing the call?
I'm fine either way; trying to understand where it's coming from. I remember it was discussed briefly at lsfmmbpf, but don't remember the details..

0 is invalid bpf object (prog, map, link). There is nothing to "fix".

It's more like it's a conditionally invalid bpf object (fd in this case) :-)

bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // returns ok and doesn't use relative_fd
dup2(prog_fd, 0);
bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // this will use prog_fd duped at 0

It seems like it might a bit cleaner to explicitly ask for -1:
bpf_program__attach_tcx(..., { ..., relative_fd = -1, ... });

But whatever, it works anyway, and that's how it's been done elsewhere it seems, so I'm not gonna waste our time on it.

#10 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 162 lines · Stanislav Fomichev
Quoted 150 lines · Alexei Starovoitov
Quoted 148 lines · Stanislav Fomichev
Quoted 144 lines · Alexei Starovoitov
Quoted 133 lines · Stanislav Fomichev
Quoted 131 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
(quoted diff) +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

Review

Quoted 10 lines
(quoted diff) +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
(quoted diff) +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 18 lines
Quoted 12 lines
(quoted diff) +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

No. There is nothing to fix.

Quoted 8 lines
Quoted 6 lines

Care to elaborate? Do we want to preserve it for consistency? Or is there some concern with asking people to put relative_fd=-1 when doing the call?
I'm fine either way; trying to understand where it's coming from. I remember it was discussed briefly at lsfmmbpf, but don't remember the details..

0 is invalid bpf object (prog, map, link). There is nothing to "fix".

Quoted 7 lines

It's more like it's a conditionally invalid bpf object (fd in this case) :-)

bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // returns ok and doesn't use relative_fd
dup2(prog_fd, 0);
bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // this will use prog_fd duped at 0

It shouldn't. I haven't checked the code, but if the patch does that it's a bug.

Quoted 5 lines

It seems like it might a bit cleaner to explicitly ask for -1:
bpf_program__attach_tcx(..., { ..., relative_fd = -1, ... });

But whatever, it works anyway, and that's how it's been done elsewhere it seems, so I'm not gonna waste our time on it.

#11 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 166 lines · Alexei Starovoitov
Quoted 164 lines · Stanislav Fomichev
Quoted 150 lines · Alexei Starovoitov
Quoted 148 lines · Stanislav Fomichev
Quoted 144 lines · Alexei Starovoitov
Quoted 133 lines · Stanislav Fomichev
Quoted 131 lines · Daniel Borkmann
Quoted 1 line · Stanislav Fomichev

On 07/07, Daniel Borkmann wrote:

[...]

Quoted 39 lines
Quoted 21 lines
(quoted diff) +21 -0
+static inline struct bpf_mprog_entry *
+bpf_mprog_create(const size_t size, const off_t off)
+{
+ struct bpf_mprog_bundle *bundle;
+ void *ptr;
+
+ BUILD_BUG_ON(size < sizeof(*bundle) + off);
+ BUILD_BUG_ON(sizeof(bundle->a.fp_items[0]) > sizeof(u64));
+ BUILD_BUG_ON(ARRAY_SIZE(bundle->a.fp_items) !=
+              ARRAY_SIZE(bundle->cp_items));
+
+ ptr = kzalloc(size, GFP_KERNEL);
+ if (ptr) {
+         bundle = ptr + off;
+         atomic64_set(&bundle->revision, 1);
+         bundle->off = off;
+         bundle->a.parent = bundle;
+         bundle->b.parent = bundle;
+         return &bundle->a;
+ }
+ return NULL;

Review

Quoted 10 lines
(quoted diff) +10 -0
+}
+
+void bpf_mprog_free_rcu(struct rcu_head *rcu);
+
+static inline void bpf_mprog_free(struct bpf_mprog_entry *entry)
+{
+ struct bpf_mprog_bundle *bundle = entry->parent;
+
+ call_rcu(&bundle->rcu, bpf_mprog_free_rcu);
+}

Any reason we're doing allocation here? Why not do bpf_mprog_init(struct bpf_mprog_bundle *) instead that simply initializes the fields? Then we can move allocation/free part to the caller (tcx) along with rcu_head.
Feels like it would be a bit more conventional/readable? bpf_mprog_free{,_rcu} will also become tcx_free{,_rcu}..

Quoted 2 lines

I guess current approach works, but it took me awhile to figure it out.. (maybe it's just me)

I found this approach quite useful for tcx case since we only fetch the bpf_mprog_entry for tcx_link_prog_attach et al, but I can take a look to see if this looks better and if it does I'll include it.

Quoted 49 lines
Quoted 49 lines
(quoted diff) +49 -0
+static inline void bpf_mprog_mark_ref(struct bpf_mprog_entry *entry,
+                               struct bpf_tuple *tuple)
+{
+ WARN_ON_ONCE(entry->parent->ref);
+ if (!tuple->link)
+         entry->parent->ref = tuple->prog;
+}
+
+static inline void bpf_mprog_inc(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count++;
+}
+
+static inline void bpf_mprog_dec(struct bpf_mprog_entry *entry)
+{
+ entry->parent->count--;
+}
+
+static inline int bpf_mprog_max(void)
+{
+ return ARRAY_SIZE(((struct bpf_mprog_entry *)NULL)->fp_items) - 1;
+}
+
+static inline int bpf_mprog_total(struct bpf_mprog_entry *entry)
+{
+ int total = entry->parent->count;
+
+ WARN_ON_ONCE(total > bpf_mprog_max());
+ return total;
+}
+
+static inline bool bpf_mprog_exists(struct bpf_mprog_entry *entry,
+                             struct bpf_prog *prog)
+{
+ const struct bpf_mprog_fp *fp;
+ const struct bpf_prog *tmp;
+
+ bpf_mprog_foreach_prog(entry, fp, tmp) {
+         if (tmp == prog)
+                 return true;
+ }
+ return false;
+}
+
+static inline bool bpf_mprog_swap_entries(const int code)
+{
+ return code == BPF_MPROG_SWAP ||
+        code == BPF_MPROG_FREE;
+}
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+static inline void bpf_mprog_commit(struct bpf_mprog_entry *entry)
+{
+ atomic64_inc(&entry->parent->revision);
+ synchronize_rcu();

Maybe add a comment on why we need to synchronize_rcu here? In general, I don't think I have a good grasp of that ->ref member.

Yeap, will add a comment. For the case where we delete the prog, we mark it in bpf_mprog_detach, but we can only drop the reference once the user swapped the bpf_mprog_entry and ensured that there are no in-flight users hence both in bpf_mprog_commit.

[...]

Quoted 18 lines
Quoted 12 lines
(quoted diff) +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+                   u32 object, u32 flags,
+                   enum bpf_prog_type type)
+{
+ bool id = flags & BPF_F_ID;
+ struct bpf_prog *prog;
+
+ if (id)
+         prog = bpf_prog_by_id(object);
+ else
+         prog = bpf_prog_get(object);
+ if (IS_ERR(prog)) {

[..]

Quoted 2 lines
(quoted diff) +2 -0
+         if (!object && !id)
+                 return 0;
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

What's the reason behind this?

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

No. There is nothing to fix.

Quoted 8 lines
Quoted 6 lines

Care to elaborate? Do we want to preserve it for consistency? Or is there some concern with asking people to put relative_fd=-1 when doing the call?
I'm fine either way; trying to understand where it's coming from. I remember it was discussed briefly at lsfmmbpf, but don't remember the details..

0 is invalid bpf object (prog, map, link). There is nothing to "fix".

It's more like it's a conditionally invalid bpf object (fd in this case) :-)

Quoted 8 lines
Quoted 5 lines

bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // returns ok and doesn't use relative_fd
dup2(prog_fd, 0);
bpf_program__attach_tcx(..., { ..., relative_fd = 0, ... }); // this will use prog_fd duped at 0

It shouldn't. I haven't checked the code, but if the patch does that it's a bug.

Daniel, am I misreading the code? It looks like it does try to resolve relative_fd=0. If it succeeds - it's being used, if not, we don't error out (that last part is ok).

So probably better to change it to:
if (id)
<resolve id>
else if (object)
<resolve fd>
else
return -EINVAL;

?

author maintainer
#12 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

In reply to Stanislav Fomichev
Quoted 2 lines · Stanislav Fomichev
Quoted 1 line · Daniel Borkmann

On 7/7/23 11:27 PM, Stanislav Fomichev wrote:

[...]

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 12 lines
kernel/bpf/mprog.c new file +12 -0
+static int bpf_mprog_prog(struct bpf_tuple *tuple,
+			  u32 object, u32 flags,
+			  enum bpf_prog_type type)
+{
+	bool id = flags & BPF_F_ID;
+	struct bpf_prog *prog;
+
+	if (id)
+		prog = bpf_prog_by_id(object);
+	else
+		prog = bpf_prog_get(object);
+	if (IS_ERR(prog)) {

[..]

Review

Quoted 2 lines
(quoted diff) +2 -0
+		if (!object && !id)
+			return 0;

What's the reason behind this?

Quoted 5 lines
Quoted 2 lines

If an fd was passed which is 0 and this was not a program fd, then we don't error out and treat it as if no fd was passed.

Is this new api an opportunity to fix that fd==0? And always treat it as valid. Or we have some other constrains elsewhere?

Not that I'm aware of, it should work fine in the new API.

Quoted 38 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines
kernel/bpf/mprog.c new file +11 -0
+		return PTR_ERR(prog);
+	}
+	if (type && prog->type != type) {
+		bpf_prog_put(prog);
+		return -EINVAL;
+	}
+
+	tuple->link = NULL;
+	tuple->prog = prog;
+	return 0;
+}

[...]

Quoted 17 lines

kernel/bpf/mprog.c line 215

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+static int bpf_mprog_pos_before(struct bpf_mprog_entry *entry,
+				struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&

Both attach/detach happen under rtnl, why do need READ_ONCE? I'm assuming even going forwrad, attach/detach from non-tcx places will happen under lock?

(same for bpf_mprog_pos_before/bpf_mprog_pos_after)

Quoted 8 lines
Quoted 2 lines

Feels like the only place where we need WRITE_ONCE is the replace (in-place) and READ_ONCE during fast-path. Why do we need the rest?

Yes, the replace case is via WRITE_ONCE, hence the READ_ONCE annotations. You are saying that for the cases where we are under lock we should just drop the READ_ONCE annotations? I can do that ofc, I thought the general convention was to do the {READ,WRITE}_ONCE consistently for the purpose of documenting fp->prog access.

Quoted 3 lines

I see, then maybe let's keep them. I was a bit confused because those READ_ONCE are within a locked section so I wasn't sure whether I'm missing something or it's working as intended :-)

Okay. I added the explanation around locking in the big comment I sent in the other thread to Alexei.

Quoted 76 lines
Quoted 76 lines
Quoted 75 lines
Quoted 52 lines
kernel/bpf/mprog.c new file +52 -0
+		    (!tuple->link || tuple->link == cp->link))
+			return i - 1;
+	}
+	return tuple->prog ? -ENOENT : -1;
+}
+
+static int bpf_mprog_pos_after(struct bpf_mprog_entry *entry,
+			       struct bpf_tuple *tuple)
+{
+	struct bpf_mprog_fp *fp;
+	struct bpf_mprog_cp *cp;
+	int i;
+
+	for (i = 0; i < bpf_mprog_total(entry); i++) {
+		bpf_mprog_read(entry, i, &fp, &cp);
+		if (tuple->prog == READ_ONCE(fp->prog) &&
+		    (!tuple->link || tuple->link == cp->link))
+			return i + 1;
+	}
+	return tuple->prog ? -ENOENT : bpf_mprog_total(entry);
+}
+
+int bpf_mprog_attach(struct bpf_mprog_entry *entry, struct bpf_prog *prog_new,
+		     struct bpf_link *link, struct bpf_prog *prog_old,
+		     u32 flags, u32 object, u64 revision)
+{
+	struct bpf_tuple rtuple, ntuple = {
+		.prog = prog_new,
+		.link = link,
+	}, otuple = {
+		.prog = prog_old,
+		.link = link,
+	};
+	int ret, idx = -2, tidx;
+
+	if (revision && revision != bpf_mprog_revision(entry))
+		return -ESTALE;
+	if (bpf_mprog_exists(entry, prog_new))
+		return -EEXIST;
+	ret = bpf_mprog_tuple_relative(&rtuple, object,
+				       flags & ~BPF_F_REPLACE,
+				       prog_new->type);
+	if (ret)
+		return ret;
+	if (flags & BPF_F_REPLACE) {
+		tidx = bpf_mprog_pos_exact(entry, &otuple);
+		if (tidx < 0) {
+			ret = tidx;
+			goto out;
+		}
+		idx = tidx;
+	}

[..]

Quoted 6 lines
kernel/bpf/mprog.c new file +6 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}

kernel/bpf/mprog.c line 283

Quoted 10 lines
kernel/bpf/mprog.c new file +10 -0
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < 0 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}

There still seems to be some inter-dependency between F_BEFORE and F_AFTER? IOW, looks like I can pass F_BEFORE|F_AFTER|F_REPLACE. Do we need that? Why not exclusive cases?

Quoted 7 lines
Quoted 4 lines

I reworked this as per Andrii's suggestion/preference from v2 [0], iow, to calculate target index and bail out if the request cannot be resolved into a common index.

[0] https://lore.kernel.org/bpf/CAEf4BzbsUMnP7WMm3OmJznvD2b03B1qASFRNiDoVAU6XvvTZNA@mail.gmail.com/

SG! Let's maybe put a summary in the header of what the expectation is when combining them?

Yes, will add a comment sounds good.

Thanks,
Daniel

#13 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

In reply to Daniel Borkmann

include/linux/bpf_mprog.h line 10

Quoted 4 lines · Daniel Borkmann
include/linux/bpf_mprog.h new file +4 -0
+
+#define BPF_MPROG_KEEP	0
+#define BPF_MPROG_SWAP	1
+#define BPF_MPROG_FREE	2

Please document how this is suppose to be used.
Patch 2 is using BPF_MPROG_FREE in tcx_entry_needs_release(). Where most of the code treats BPF_MPROG_SWAP and BPF_MPROG_FREE as equivalent. I can guess what it's for, but a comment would help.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+#define BPF_MPROG_MAX	64

we've seen buggy user space attaching thousands of tc progs to the same netdev. I suspect 64 limit will be hit much sooner than expected.

Quoted 4 lines
include/linux/bpf_mprog.h new file +4 -0
+
+#define bpf_mprog_foreach_tuple(entry, fp, cp, t)			\
+	for (fp = &entry->fp_items[0], cp = &entry->parent->cp_items[0];\
+	     ({								\

include/linux/bpf_mprog.h line 26

Quoted 10 lines
include/linux/bpf_mprog.h new file +10 -0
+		t.prog = READ_ONCE(fp->prog);				\
+		t.link = cp->link;					\
+		t.prog;							\
+	      });							\
+	     fp++, cp++)
+
+#define bpf_mprog_foreach_prog(entry, fp, p)				\
+	for (fp = &entry->fp_items[0];					\
+	     (p = READ_ONCE(fp->prog));					\
+	     fp++)

I have similar questions to Stanislav.
Looks like update/delete/query of bpf_prog should be protected by an external lock (like RTNL in case of tcx),
but what are the life time rules for 'entry'?
Looking at patch 2 sch_handle_ingress():
struct bpf_mprog_entry *entry = rcu_dereference_bh(skb->dev->tcx_ingress);
I suspect the assumption is that bpf_mprog_entry object should be accessed within RCU critical section. Since tc/tcx and XDP run in napi we have RCU protection there. In the future, for cgroups, bpf_prog_run_array_cg() will keep explicit rcu_read_lock() before accessing bpf_mprog_entry, right?
And bpf_mprog_commit() assumes that RCU protection.
All fine, but we need to document that mprog mechanism is not suitable for sleepable progs.

kernel/bpf/mprog.c line 278

Quoted 11 lines
kernel/bpf/mprog.c new file +11 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {

tidx < 0 vs tidx < -1 for _after vs _before.
Does it have to have this subtle difference?
Can _after and _before have the same semantics for return value?

Quoted 5 lines
kernel/bpf/mprog.c new file +5 -0
+			ret = tidx < 0 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
author maintainer
#14 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 10 lines · Alexei Starovoitov

include/linux/bpf_mprog.h line 10

Quoted 4 lines · Daniel Borkmann
include/linux/bpf_mprog.h new file +4 -0
+
+#define BPF_MPROG_KEEP	0
+#define BPF_MPROG_SWAP	1
+#define BPF_MPROG_FREE	2

Please document how this is suppose to be used.
Patch 2 is using BPF_MPROG_FREE in tcx_entry_needs_release(). Where most of the code treats BPF_MPROG_SWAP and BPF_MPROG_FREE as equivalent. I can guess what it's for, but a comment would help.

Ok, sounds good, will add a comment to these codes.

Quoted 4 lines
Quoted 1 line

+#define BPF_MPROG_MAX 64

we've seen buggy user space attaching thousands of tc progs to the same netdev. I suspect 64 limit will be hit much sooner than expected.

In that sense it's probably good that we'll bail out rather than draining memory as you had with the cls_bpf case, iirc. As I mentioned given this is not uapi, we can adapt this further in future when needed if you have cases where you attach more than 64 progs for a single device.

Quoted 7 lines
Quoted 7 lines
include/linux/bpf_mprog.h new file +7 -0
+#define bpf_mprog_foreach_tuple(entry, fp, cp, t)			\
+	for (fp = &entry->fp_items[0], cp = &entry->parent->cp_items[0];\
+	     ({								\
+		t.prog = READ_ONCE(fp->prog);				\
+		t.link = cp->link;					\
+		t.prog;							\
+	      });							\
Quoted 8 lines

include/linux/bpf_mprog.h line 26

Quoted 6 lines
include/linux/bpf_mprog.h new file +6 -0
+	     fp++, cp++)
+
+#define bpf_mprog_foreach_prog(entry, fp, p)				\
+	for (fp = &entry->fp_items[0];					\
+	     (p = READ_ONCE(fp->prog));					\
+	     fp++)

I have similar questions to Stanislav.

The READ_ONCE/WRITE_ONCE is for the replacement case where we don't need to swap the whole array, so I annotated all access to fp->prog.

Quoted 2 lines

Looks like update/delete/query of bpf_prog should be protected by an external lock (like RTNL in case of tcx),

Correct for tcx it's rtnl, other users also either have to piggyback on existing locking or need their own.

Quoted 5 lines

but what are the life time rules for 'entry'?
Looking at patch 2 sch_handle_ingress():
struct bpf_mprog_entry *entry = rcu_dereference_bh(skb->dev->tcx_ingress); I suspect the assumption is that bpf_mprog_entry object should be accessed within RCU critical section. Since tc/tcx and XDP run in napi we have RCU protection there.

Correct.

Quoted 3 lines

In the future, for cgroups, bpf_prog_run_array_cg() will keep explicit rcu_read_lock() before accessing bpf_mprog_entry, right?
And bpf_mprog_commit() assumes that RCU protection.

Both yes.

Quoted 1 line

All fine, but we need to document that mprog mechanism is not suitable for sleepable progs.

Ok, I'll add a comment.

Quoted 7 lines
Quoted 7 lines
kernel/bpf/mprog.c new file +7 -0
+	if (flags & BPF_F_BEFORE) {
+		tidx = bpf_mprog_pos_before(entry, &rtuple);
+		if (tidx < -1 || (idx >= -1 && tidx != idx)) {
+			ret = tidx < -1 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
Quoted 8 lines

kernel/bpf/mprog.c line 278

Quoted 4 lines
kernel/bpf/mprog.c new file +4 -0
+	}
+	if (flags & BPF_F_AFTER) {
+		tidx = bpf_mprog_pos_after(entry, &rtuple);
+		if (tidx < 0 || (idx >= -1 && tidx != idx)) {

tidx < 0 vs tidx < -1 for _after vs _before.
Does it have to have this subtle difference?
Can _after and _before have the same semantics for return value?

Yes, they can. In 'after' tidx will never be -1, but I can adapt the condition so it's the same for the two to avoid confusion.

Quoted 6 lines
Quoted 5 lines
kernel/bpf/mprog.c new file +5 -0
+			ret = tidx < 0 ? tidx : -EDOM;
+			goto out;
+		}
+		idx = tidx;
+	}
author maintainer
#15 lore raw

Re: [PATCH bpf-next v3 1/8] bpf: Add generic attach/detach/query API for multi-progs

Quoted 14 lines · Daniel Borkmann
Quoted 10 lines · Alexei Starovoitov

include/linux/bpf_mprog.h line 10

Quoted 4 lines · Daniel Borkmann
include/linux/bpf_mprog.h new file +4 -0
+
+#define BPF_MPROG_KEEP    0
+#define BPF_MPROG_SWAP    1
+#define BPF_MPROG_FREE    2

Please document how this is suppose to be used.
Patch 2 is using BPF_MPROG_FREE in tcx_entry_needs_release(). Where most of the code treats BPF_MPROG_SWAP and BPF_MPROG_FREE as equivalent. I can guess what it's for, but a comment would help.

Ok, sounds good, will add a comment to these codes.

[...]

Quoted 9 lines
Quoted 3 lines

In the future, for cgroups, bpf_prog_run_array_cg() will keep explicit rcu_read_lock() before accessing bpf_mprog_entry, right?
And bpf_mprog_commit() assumes that RCU protection.

Both yes.

Quoted 1 line

All fine, but we need to document that mprog mechanism is not suitable for sleepable progs.

Ok, I'll add a comment.

I've added this as comment for bpf_mprog.h to address the ret codes, locking and usage example :

/*
* bpf_mprog framework:
* ~~~~~~~~~~~~~~~~~~~~
*
* bpf_mprog is a generic layer for multi-program attachment. In-kernel users
* of the bpf_mprog don't need to care about the dependency resolution
* internals, they can just consume it with few API calls. Currently available
* dependency directives are BPF_F_{BEFORE,AFTER} which enable insertion of
* a BPF program or BPF link relative to an existing BPF program or BPF link
* inside the multi-program array as well as prepend and append behavior if
* no relative object was specified, see corresponding selftests for concrete
* examples (e.g. tc_links and tc_opts test cases of test_progs). *
* Usage of bpf_mprog_{attach,detach,query}() core APIs with pseudo code:
*
* Attach case:
*
* struct bpf_mprog_entry *entry, *peer;
* int ret;
*

 *   // bpf_mprog user-side lock
 *   // fetch active @entry from attach location
 *   [...]
 *   ret = bpf_mprog_attach(entry, [...]);
 *   if (ret >= 0) {
 *       peer = bpf_mprog_peer(entry);
 *       if (bpf_mprog_swap_entries(ret))
 *           // swap @entry to @peer at attach location
 *       bpf_mprog_commit(entry);
 *       ret = 0;
 *   } else {
 *       // error path, bail out, propagate @ret
 *   }
 *   // bpf_mprog user-side unlock
 *
 *  Detach case:
 *
 *   struct bpf_mprog_entry *entry, *peer;
 *   bool release;
 *   int ret;
 *
 *   // bpf_mprog user-side lock
 *   // fetch active @entry from attach location
 *   [...]
 *   ret = bpf_mprog_detach(entry, [...]);
 *   if (ret >= 0) {
 *       release = ret == BPF_MPROG_FREE;
 *       peer = release ? NULL : bpf_mprog_peer(entry);
 *       if (bpf_mprog_swap_entries(ret))
 *           // swap entry to @peer at attach location
 *       bpf_mprog_commit(entry);
 *       if (release)
 *           // free bpf_mprog_bundle
 *       ret = 0;
 *   } else {
 *       // error path, bail out, propagate @ret
 *   }
 *   // bpf_mprog user-side unlock
 *
 *  Query case:
 *
 *   struct bpf_mprog_entry *entry;
 *   int ret;
 *
 *   // bpf_mprog user-side lock
 *   // fetch active @entry from attach location
 *   [...]
 *   ret = bpf_mprog_query(attr, uattr, entry);
 *   // bpf_mprog user-side unlock
 *
 *  Data/fast path:
 *
 *   struct bpf_mprog_entry *entry;
 *   struct bpf_mprog_fp *fp;
 *   struct bpf_prog *prog;
 *   int ret = [...];
 *
 *   rcu_read_lock();
 *   // fetch active @entry from attach location
 *   [...]
 *   bpf_mprog_foreach_prog(entry, fp, prog) {
 *       ret = bpf_prog_run(prog, [...]);
 *       // process @ret from program
 *   }
 *   [...]
 *   rcu_read_unlock();
 *
 * bpf_mprog_{attach,detach}() return codes:
 *
 * Negative return code means that an error occurred and the bpf_mprog_entry
 * has not been changed. The error should be propagated to the user. A non-
 * negative return code can be one of the following:
 *
 * BPF_MPROG_KEEP:
 *   The bpf_mprog_entry does not need a/b swap, the bpf_mprog_fp item has
 *   been replaced in the current active bpf_mprog_entry.
 *
 * BPF_MPROG_SWAP:
 *   The bpf_mprog_entry does need an a/b swap and must be updated to its
 *   peer entry (peer = bpf_mprog_peer(entry)) which has been populated to
 *   the new bpf_mprog_fp item configuration.
 *
 * BPF_MPROG_FREE:
 *   The bpf_mprog_entry now does not hold any non-NULL bpf_mprog_fp items
 *   anymore. The bpf_mprog_entry should be swapped with NULL and the
 *   corresponding bpf_mprog_bundle can be freed.

*
* bpf_mprog locking considerations:
*
* bpf_mprog_{attach,detach,query}() must be protected by an external lock
* (like RTNL in case of tcx).
*
* bpf_mprog_entry pointer can be an __rcu annotated pointer (in case of tcx
* the netdevice has tcx_ingress and tcx_egress __rcu pointer) which gets
* updated via rcu_assign_pointer() pointing to the active bpf_mprog_entry of
* the bpf_mprog_bundle.
*
* Fast path accesses the active bpf_mprog_entry within RCU critical section
* (in case of tcx it runs in NAPI which provides RCU protection there,
* other users might need explicit rcu_read_lock()). The bpf_mprog_commit()
* assumes that RCU protection.
*
* The READ_ONCE()/WRITE_ONCE() pairing for bpf_mprog_fp's prog access is for
* the replacement case where we don't swap the bpf_mprog_entry. */

Hope that helps,
Daniel

author maintainer
#16 lore raw

PATCH v3 2/8 bpf: Add fd-based tcx multi-prog infra with link support

In reply to Daniel Borkmann

This work refactors and adds a lightweight extension ("tcx") to the tc BPF ingress and egress data path side for allowing BPF program management based on fds via bpf() syscall through the newly added generic multi-prog API. The main goal behind this work which we also presented at LPC [0] last year and a recent update at LSF/MM/BPF this year [3] is to support long-awaited BPF link functionality for tc BPF programs, which allows for a model of safe ownership and program detachment.

Given the rise in tc BPF users in cloud native environments, this becomes necessary to avoid hard to debug incidents either through stale leftover programs or 3rd party applications accidentally stepping on each others toes. As a recap, a BPF link represents the attachment of a BPF program to a BPF hook point. The BPF link holds a single reference to keep BPF program alive. Moreover, hook points do not reference a BPF link, only the application's fd or pinning does. A BPF link holds meta-data specific to attachment and implements operations for link creation, (atomic) BPF program update, detachment and introspection. The motivation for BPF links for tc BPF programs is multi-fold, for example:

- From Meta: "It's especially important for applications that are deployed fleet-wide and that don't "control" hosts they are deployed to. If such application crashes and no one notices and does anything about that, BPF program will keep running draining resources or even just, say, dropping packets. We at FB had outages due to such permanent BPF attachment semantics. With fd-based BPF link we are getting a framework, which allows safe, auto-detachable behavior by default, unless application explicitly opts in by pinning the BPF link." [1]

- From Cilium-side the tc BPF programs we attach to host-facing veth devices and phys devices build the core datapath for Kubernetes Pods, and they implement forwarding, load-balancing, policy, EDT-management, etc, within BPF. Currently there is no concept of 'safe' ownership, e.g. we've recently experienced hard-to-debug issues in a user's staging environment where another Kubernetes application using tc BPF attached to the same prio/handle of cls_bpf, accidentally wiping all Cilium-based BPF programs from underneath it. The goal is to establish a clear/safe ownership model via links which cannot accidentally be overridden. [0,2]

BPF links for tc can co-exist with non-link attachments, and the semantics are in line also with XDP links: BPF links cannot replace other BPF links, BPF links cannot replace non-BPF links, non-BPF links cannot replace BPF links and lastly only non-BPF links can replace non-BPF links. In case of Cilium, this would solve mentioned issue of safe ownership model as 3rd party applications would not be able to accidentally wipe Cilium programs, even if they are not BPF link aware.

Earlier attempts [4] have tried to integrate BPF links into core tc machinery to solve cls_bpf, which has been intrusive to the generic tc kernel API with extensions only specific to cls_bpf and suboptimal/complex since cls_bpf could be wiped from the qdisc also. Locking a tc BPF program in place this way, is getting into layering hacks given the two object models are vastly different.

We instead implemented the tcx (tc 'express') layer which is an fd-based tc BPF attach API, so that the BPF link implementation blends in naturally similar to other link types which are fd-based and without the need for changing core tc internal APIs. BPF programs for tc can then be successively migrated from classic cls_bpf to the new tc BPF link without needing to change the program's source code, just the BPF loader mechanics for attaching is sufficient.

For the current tc framework, there is no change in behavior with this change and neither does this change touch on tc core kernel APIs. The gist of this patch is that the ingress and egress hook have a lightweight, qdisc-less extension for BPF to attach its tc BPF programs, in other words, a minimal entry point for tc BPF. The name tcx has been suggested from discussion of earlier revisions of this work as a good fit, and to more easily differ between the classic cls_bpf attachment and the fd-based one.

For the ingress and egress tcx points, the device holds a cache-friendly array with program pointers which is separated from control plane (slow-path) data. Earlier versions of this work used priority to determine ordering and expression of dependencies similar as with classic tc, but it was challenged that for something more future-proof a better user experience is required. Hence this resulted in the design and development of the generic attach/detach/query API for multi-progs. See prior patch with its discussion on the API design. tcx is the first user and later we plan to integrate also others, for example, one candidate is multi-prog support for XDP which would benefit and have the same 'look and feel' from API perspective.

The goal with tcx is to have maximum compatibility to existing tc BPF programs, so they don't need to be rewritten specifically. Compatibility to call into classic tcf_classify() is also provided in order to allow successive migration or both to cleanly co-exist where needed given its all one logical tc layer and the tcx plus classic tc cls/act build one logical overall processing pipeline.

tcx supports the simplified return codes TCX_NEXT which is non-terminating (go to next program) and terminating ones with TCX_PASS, TCX_DROP, TCX_REDIRECT. The fd-based API is behind a static key, so that when unused the code is also not entered. The struct tcx_entry's program array is currently static, but could be made dynamic if necessary at a point in future. The a/b pair swap design has been chosen so that for detachment there are no allocations which otherwise could fail.

The work has been tested with tc-testing selftest suite which all passes, as well as the tc BPF tests from the BPF CI, and also with Cilium's L4LB.

Kudos also to Nikolay Aleksandrov and Martin Lau for in-depth early reviews of this work.

[0] https://lpc.events/event/16/contributions/1353/
[1] https://lore.kernel.org/bpf/CAEf4BzbokCJN33Nw_kg82sO=xppXnKWEncGTWCTB9vGCmLB6pw@mail.gmail.com
[2] https://colocatedeventseu2023.sched.com/event/1Jo6O/tales-from-an-ebpf-programs-murder-mystery-hemanth-malla-guillaume-fournier-datadog
[3] http://vger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf
[4] https://lore.kernel.org/bpf/[email protected]

---
 MAINTAINERS                    |   4 +-
 include/linux/bpf_mprog.h      |   9 +
 include/linux/netdevice.h      |  15 +-
 include/linux/skbuff.h         |   4 +-
 include/net/sch_generic.h      |   2 +-
 include/net/tcx.h              | 183 +++++++++++++++++
 include/uapi/linux/bpf.h       |  34 +++-
 kernel/bpf/Kconfig             |   1 +
 kernel/bpf/Makefile            |   1 +
 kernel/bpf/syscall.c           |  83 ++++++--
 kernel/bpf/tcx.c               | 351 +++++++++++++++++++++++++++++++++
 net/Kconfig                    |   5 +
 net/core/dev.c                 | 267 +++++++++++++++----------
 net/core/filter.c              |   4 +-
 net/sched/Kconfig              |   4 +-
 net/sched/sch_ingress.c        |  61 +++++-
 tools/include/uapi/linux/bpf.h |  34 +++-
 17 files changed, 918 insertions(+), 144 deletions(-)
 create mode 100644 include/net/tcx.h
 create mode 100644 kernel/bpf/tcx.c
MAINTAINERS +3 -1
index 7e5ba799d1c5..0a1327cf4e34 100644
@@ -3830,13 +3830,15 @@ L:	[email protected]
 S:	Maintained
 F:	kernel/bpf/bpf_struct*
 
-BPF [NETWORKING] (tc BPF, sock_addr)
+BPF [NETWORKING] (tcx & tc BPF, sock_addr)
 M:	Martin KaFai Lau <[email protected]>
 M:	Daniel Borkmann <[email protected]>
 R:	John Fastabend <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
+F:	include/net/tcx.h
+F:	kernel/bpf/tcx.c
 F:	net/core/filter.c
 F:	net/sched/act_bpf.c
 F:	net/sched/cls_bpf.c
include/linux/bpf_mprog.h +9 -0
index a7a9b98df1ef..6020d2602d62 100644
@@ -230,4 +230,13 @@ int bpf_mprog_detach(struct bpf_mprog_entry *entry, struct bpf_prog *prog,
 int bpf_mprog_query(const union bpf_attr *attr, union bpf_attr __user *uattr,
 		    struct bpf_mprog_entry *entry);
 
+static inline bool bpf_mprog_supported(enum bpf_prog_type type)
+{
+	switch (type) {
+	case BPF_PROG_TYPE_SCHED_CLS:
+		return true;
+	default:
+		return false;
+	}
+}
 #endif /* __BPF_MPROG_H */
include/linux/netdevice.h +6 -9
index b828c7a75be2..024314c68bc8 100644
@@ -1930,8 +1930,7 @@ enum netdev_ml_priv_type {
  *
  *	@rx_handler:		handler for received packets
  *	@rx_handler_data: 	XXX: need comments on this one
- *	@miniq_ingress:		ingress/clsact qdisc specific data for
- *				ingress processing
+ *	@tcx_ingress:		BPF & clsact qdisc specific data for ingress processing
  *	@ingress_queue:		XXX: need comments on this one
  *	@nf_hooks_ingress:	netfilter hooks executed for ingress packets
  *	@broadcast:		hw bcast address
@@ -1952,8 +1951,7 @@ enum netdev_ml_priv_type {
  *	@xps_maps:		all CPUs/RXQs maps for XPS device
  *
  *	@xps_maps:	XXX: need comments on this one
- *	@miniq_egress:		clsact qdisc specific data for
- *				egress processing
+ *	@tcx_egress:		BPF & clsact qdisc specific data for egress processing
  *	@nf_hooks_egress:	netfilter hooks executed for egress packets
  *	@qdisc_hash:		qdisc hash table
  *	@watchdog_timeo:	Represents the timeout that is used by
@@ -2252,9 +2250,8 @@ struct net_device {
 	unsigned int		gro_ipv4_max_size;
 	rx_handler_func_t __rcu	*rx_handler;
 	void __rcu		*rx_handler_data;
-
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_ingress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_ingress;
 #endif
 	struct netdev_queue __rcu *ingress_queue;
 #ifdef CONFIG_NETFILTER_INGRESS
@@ -2282,8 +2279,8 @@ struct net_device {
 #ifdef CONFIG_XPS
 	struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX];
 #endif
-#ifdef CONFIG_NET_CLS_ACT
-	struct mini_Qdisc __rcu	*miniq_egress;
+#ifdef CONFIG_NET_XGRESS
+	struct bpf_mprog_entry __rcu *tcx_egress;
 #endif
 #ifdef CONFIG_NETFILTER_EGRESS
 	struct nf_hook_entries __rcu *nf_hooks_egress;
include/linux/skbuff.h +2 -2
index 91ed66952580..ed83f1c5fc1f 100644
@@ -944,7 +944,7 @@ struct sk_buff {
 	__u8			__mono_tc_offset[0];
 	/* public: */
 	__u8			mono_delivery_time:1;	/* See SKB_MONO_DELIVERY_TIME_MASK */
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	__u8			tc_at_ingress:1;	/* See TC_AT_INGRESS_MASK */
 	__u8			tc_skip_classify:1;
 #endif
@@ -993,7 +993,7 @@ struct sk_buff {
 	__u8			csum_not_inet:1;
 #endif
 
-#ifdef CONFIG_NET_SCHED
+#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
 	__u16			tc_index;	/* traffic control index */
 #endif
 
include/net/sch_generic.h +1 -1
index e92f73bb3198..15be2d96b06d 100644
@@ -703,7 +703,7 @@ int skb_do_redirect(struct sk_buff *);
 
 static inline bool skb_at_tc_ingress(const struct sk_buff *skb)
 {
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	return skb->tc_at_ingress;
 #else
 	return false;
include/net/tcx.h new file +183 -0

183 lines not on this page · raw message

include/uapi/linux/bpf.h +30 -4
index 74879c538f2b..98c4a3a6e137 100644
@@ -1036,6 +1036,8 @@ enum bpf_attach_type {
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
 	BPF_NETFILTER,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1053,7 +1055,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u64		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,10 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
kernel/bpf/Kconfig +1 -0
index 2dfe1079f772..6a906ff93006 100644
@@ -31,6 +31,7 @@ config BPF_SYSCALL
 	select TASKS_TRACE_RCU
 	select BINARY_PRINTF
 	select NET_SOCK_MSG if NET
+	select NET_XGRESS if NET
 	select PAGE_POOL if NET
 	default n
 	help
kernel/bpf/Makefile +1 -0
index 1bea2eb912cd..f526b7573e97 100644
@@ -21,6 +21,7 @@ obj-$(CONFIG_BPF_SYSCALL) += devmap.o
 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
 obj-$(CONFIG_BPF_SYSCALL) += offload.o
 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
+obj-$(CONFIG_BPF_SYSCALL) += tcx.o
 endif
 ifeq ($(CONFIG_PERF_EVENTS),y)
 obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
kernel/bpf/syscall.c +70 -13

169 lines not on this page · raw message

kernel/bpf/tcx.c new file +351 -0

351 lines not on this page · raw message

net/Kconfig +5 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
 config NET_EGRESS
 	bool
 
+config NET_XGRESS
+	select NET_INGRESS
+	select NET_EGRESS
+	bool
+
 config NET_REDIRECT
 	bool
 
net/core/dev.c +165 -102

335 lines not on this page · raw message

net/core/filter.c +2 -2
index 06ba0e56e369..e39a8a20dd10 100644
@@ -9312,7 +9312,7 @@ static struct bpf_insn *bpf_convert_tstamp_read(const struct bpf_prog *prog,
 	__u8 value_reg = si->dst_reg;
 	__u8 skb_reg = si->src_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, read skb->tstamp as is if tstamp_type_access is true.
@@ -9346,7 +9346,7 @@ static struct bpf_insn *bpf_convert_tstamp_write(const struct bpf_prog *prog,
 	__u8 value_reg = si->src_reg;
 	__u8 skb_reg = si->dst_reg;
 
-#ifdef CONFIG_NET_CLS_ACT
+#ifdef CONFIG_NET_XGRESS
 	/* If the tstamp_type is read,
 	 * the bpf prog is aware the tstamp could have delivery time.
 	 * Thus, write skb->tstamp as is if tstamp_type_access is true.
net/sched/Kconfig +2 -2
index 4b95cb1ac435..470c70deffe2 100644
@@ -347,8 +347,7 @@ config NET_SCH_FQ_PIE
 config NET_SCH_INGRESS
 	tristate "Ingress/classifier-action Qdisc"
 	depends on NET_CLS_ACT
-	select NET_INGRESS
-	select NET_EGRESS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use classifiers for incoming and/or outgoing
 	  packets. This qdisc doesn't do anything else besides running classifiers,
@@ -679,6 +678,7 @@ config NET_EMATCH_IPT
 config NET_CLS_ACT
 	bool "Actions"
 	select NET_CLS
+	select NET_XGRESS
 	help
 	  Say Y here if you want to use traffic control actions. Actions
 	  get attached to classifiers and are invoked after a successful
net/sched/sch_ingress.c +57 -4

119 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +30 -4
index 74879c538f2b..98c4a3a6e137 100644
@@ -1036,6 +1036,8 @@ enum bpf_attach_type {
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
 	BPF_NETFILTER,
+	BPF_TCX_INGRESS,
+	BPF_TCX_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1053,7 +1055,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
-
+	BPF_LINK_TYPE_TCX = 11,
 	MAX_BPF_LINK_TYPE,
 };
 
@@ -1559,13 +1561,13 @@ union bpf_attr {
 			__u32		map_fd;		/* struct_ops to attach */
 		};
 		union {
-			__u32		target_fd;	/* object to attach to */
-			__u32		target_ifindex; /* target ifindex */
+			__u32	target_fd;	/* target object to attach to or ... */
+			__u32	target_ifindex; /* target ifindex */
 		};
 		__u32		attach_type;	/* attach type */
 		__u32		flags;		/* extra flags */
 		union {
-			__u32		target_btf_id;	/* btf_id of target to attach to */
+			__u32	target_btf_id;	/* btf_id of target to attach to */
 			struct {
 				__aligned_u64	iter_info;	/* extra bpf_iter_link_info */
 				__u32		iter_info_len;	/* iter_info length */
@@ -1599,6 +1601,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				union {
+					__u32	relative_fd;
+					__u32	relative_id;
+				};
+				__u64		expected_revision;
+			} tcx;
 		};
 	} link_create;
 
@@ -6207,6 +6216,19 @@ struct bpf_sock_tuple {
 	};
 };
 
+/* (Simplified) user return codes for tcx prog type.
+ * A valid tcx program must return one of these defined values. All other
+ * return codes are reserved for future use. Must remain compatible with
+ * their TC_ACT_* counter-parts. For compatibility in behavior, unknown
+ * return codes are mapped to TCX_NEXT.
+ */
+enum tcx_action_base {
+	TCX_NEXT	= -1,
+	TCX_PASS	= 0,
+	TCX_DROP	= 2,
+	TCX_REDIRECT	= 7,
+};
+
 struct bpf_xdp_sock {
 	__u32 queue_id;
 };
@@ -6459,6 +6481,10 @@ struct bpf_link_info {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 ifindex;
+			__u32 attach_type;
+		} tcx;
 	};
 } __attribute__((aligned(8)));
 
Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH bpf-next v3 2/8] bpf: Add fd-based tcx multi-prog infra with link support

net/Kconfig config NET_INGRESS · line 58

Quoted 12 lines · Daniel Borkmann
net/Kconfig +4 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
 config NET_EGRESS
 	bool
 
+config NET_XGRESS
+	select NET_INGRESS
+	select NET_EGRESS
+	bool

Since new kconfig is needed, can NET_INGRESS and NET_EGRESS be removed at the same time?

author maintainer
#18 lore raw

Re: [PATCH bpf-next v3 2/8] bpf: Add fd-based tcx multi-prog infra with link support

Quoted 9 lines · Alexei Starovoitov
Quoted 8 lines · Daniel Borkmann
net/Kconfig +0 -0
index 2fb25b534df5..d532ec33f1fe 100644
@@ -52,6 +52,11 @@ config NET_INGRESS
  config NET_EGRESS
  	bool
 
Quoted 6 lines

net/Kconfig config NET_INGRESS · line 58

Quoted 4 lines
net/Kconfig +4 -0
+config NET_XGRESS
+	select NET_INGRESS
+	select NET_EGRESS
+	bool

Since new kconfig is needed, can NET_INGRESS and NET_EGRESS be removed at the same time?

No because netfilter can pick these individually :

config NETFILTER_INGRESS
bool "Netfilter ingress support"
default y
select NET_INGRESS
help
This allows you to classify packets from ingress using the Netfilter infrastructure.

config NETFILTER_EGRESS
bool "Netfilter egress support"
default y
select NET_EGRESS
help
This allows you to classify packets before transmission using the Netfilter infrastructure.

Thanks,
Daniel

author maintainer
#19 lore raw

PATCH v3 3/8 libbpf: Add opts-based attach/detach/query API for tcx

In reply to Daniel Borkmann

Extend libbpf attach opts and add a new detach opts API so this can be used to add/remove fd-based tcx BPF programs. The old-style bpf_prog_detach() and bpf_prog_detach2() APIs are refactored to reuse the new bpf_prog_detach_opts() internally.

The bpf_prog_query_opts() API got extended to be able to handle the new link_ids, link_attach_flags and revision fields.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      | 105 +++++++++++++++++++++++++--------------
 tools/lib/bpf/bpf.h      |  92 ++++++++++++++++++++++++++++------
 tools/lib/bpf/libbpf.c   |  12 +++--
 tools/lib/bpf/libbpf.map |   1 +
 4 files changed, 157 insertions(+), 53 deletions(-)
tools/lib/bpf/bpf.c +69 -36
index 3b0da19715e1..3dfc43b477c3 100644
@@ -629,55 +629,87 @@ int bpf_prog_attach(int prog_fd, int target_fd, enum bpf_attach_type type,
 	return bpf_prog_attach_opts(prog_fd, target_fd, type, &opts);
 }
 
-int bpf_prog_attach_opts(int prog_fd, int target_fd,
-			  enum bpf_attach_type type,
-			  const struct bpf_prog_attach_opts *opts)
+int bpf_prog_attach_opts(int prog_fd, int target,
+			 enum bpf_attach_type type,
+			 const struct bpf_prog_attach_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
+	const size_t attr_sz = offsetofend(union bpf_attr, expected_revision);
+	__u32 relative_id, flags;
 	union bpf_attr attr;
-	int ret;
+	int ret, relative;
 
 	if (!OPTS_VALID(opts, bpf_prog_attach_opts))
 		return libbpf_err(-EINVAL);
 
+	relative_id = OPTS_GET(opts, relative_id, 0);
+	relative = OPTS_GET(opts, relative_fd, 0);
+	flags = OPTS_GET(opts, flags, 0);
+
+	/* validate we don't have unexpected combinations of non-zero fields */
+	if (relative > 0 && relative_id)
+		return libbpf_err(-EINVAL);
+	if (relative_id) {
+		relative = relative_id;
+		flags |= BPF_F_ID;
+	}
+
 	memset(&attr, 0, attr_sz);
-	attr.target_fd	   = target_fd;
-	attr.attach_bpf_fd = prog_fd;
-	attr.attach_type   = type;
-	attr.attach_flags  = OPTS_GET(opts, flags, 0);
-	attr.replace_bpf_fd = OPTS_GET(opts, replace_prog_fd, 0);
+	attr.target_fd		= target;
+	attr.attach_bpf_fd	= prog_fd;
+	attr.attach_type	= type;
+	attr.attach_flags	= flags;
+	attr.relative_fd	= relative;
+	attr.replace_bpf_fd	= OPTS_GET(opts, replace_fd, 0);
+	attr.expected_revision	= OPTS_GET(opts, expected_revision, 0);
 
 	ret = sys_bpf(BPF_PROG_ATTACH, &attr, attr_sz);
 	return libbpf_err_errno(ret);
 }
 
-int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
+int bpf_prog_detach_opts(int prog_fd, int target,
+			 enum bpf_attach_type type,
+			 const struct bpf_prog_detach_opts *opts)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
+	const size_t attr_sz = offsetofend(union bpf_attr, expected_revision);
+	__u32 relative_id, flags;
 	union bpf_attr attr;
-	int ret;
+	int ret, relative;
+
+	if (!OPTS_VALID(opts, bpf_prog_detach_opts))
+		return libbpf_err(-EINVAL);
+
+	relative_id = OPTS_GET(opts, relative_id, 0);
+	relative = OPTS_GET(opts, relative_fd, 0);
+	flags = OPTS_GET(opts, flags, 0);
+
+	/* validate we don't have unexpected combinations of non-zero fields */
+	if (relative > 0 && relative_id)
+		return libbpf_err(-EINVAL);
+	if (relative_id) {
+		relative = relative_id;
+		flags |= BPF_F_ID;
+	}
 
 	memset(&attr, 0, attr_sz);
-	attr.target_fd	 = target_fd;
-	attr.attach_type = type;
+	attr.target_fd		= target;
+	attr.attach_bpf_fd	= prog_fd;
+	attr.attach_type	= type;
+	attr.attach_flags	= flags;
+	attr.relative_fd	= relative;
+	attr.expected_revision	= OPTS_GET(opts, expected_revision, 0);
 
 	ret = sys_bpf(BPF_PROG_DETACH, &attr, attr_sz);
 	return libbpf_err_errno(ret);
 }
 
-int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
+int bpf_prog_detach(int target_fd, enum bpf_attach_type type)
 {
-	const size_t attr_sz = offsetofend(union bpf_attr, replace_bpf_fd);
-	union bpf_attr attr;
-	int ret;
-
-	memset(&attr, 0, attr_sz);
-	attr.target_fd	 = target_fd;
-	attr.attach_bpf_fd = prog_fd;
-	attr.attach_type = type;
+	return bpf_prog_detach_opts(0, target_fd, type, NULL);
+}
 
-	ret = sys_bpf(BPF_PROG_DETACH, &attr, attr_sz);
-	return libbpf_err_errno(ret);
+int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
+{
+	return bpf_prog_detach_opts(prog_fd, target_fd, type, NULL);
 }
 
 int bpf_link_create(int prog_fd, int target_fd,
@@ -841,8 +873,7 @@ int bpf_iter_create(int link_fd)
 	return libbpf_err_errno(fd);
 }
 
-int bpf_prog_query_opts(int target_fd,
-			enum bpf_attach_type type,
+int bpf_prog_query_opts(int target, enum bpf_attach_type type,
 			struct bpf_prog_query_opts *opts)
 {
 	const size_t attr_sz = offsetofend(union bpf_attr, query);
@@ -853,18 +884,20 @@ int bpf_prog_query_opts(int target_fd,
 		return libbpf_err(-EINVAL);
 
 	memset(&attr, 0, attr_sz);
-
-	attr.query.target_fd	= target_fd;
-	attr.query.attach_type	= type;
-	attr.query.query_flags	= OPTS_GET(opts, query_flags, 0);
-	attr.query.prog_cnt	= OPTS_GET(opts, prog_cnt, 0);
-	attr.query.prog_ids	= ptr_to_u64(OPTS_GET(opts, prog_ids, NULL));
-	attr.query.prog_attach_flags = ptr_to_u64(OPTS_GET(opts, prog_attach_flags, NULL));
+	attr.query.target_fd		= target;
+	attr.query.attach_type		= type;
+	attr.query.query_flags		= OPTS_GET(opts, query_flags, 0);
+	attr.query.count		= OPTS_GET(opts, count, 0);
+	attr.query.prog_ids		= ptr_to_u64(OPTS_GET(opts, prog_ids, NULL));
+	attr.query.link_ids		= ptr_to_u64(OPTS_GET(opts, link_ids, NULL));
+	attr.query.prog_attach_flags	= ptr_to_u64(OPTS_GET(opts, prog_attach_flags, NULL));
+	attr.query.link_attach_flags	= ptr_to_u64(OPTS_GET(opts, link_attach_flags, NULL));
 
 	ret = sys_bpf(BPF_PROG_QUERY, &attr, attr_sz);
 
 	OPTS_SET(opts, attach_flags, attr.query.attach_flags);
-	OPTS_SET(opts, prog_cnt, attr.query.prog_cnt);
+	OPTS_SET(opts, revision, attr.query.revision);
+	OPTS_SET(opts, count, attr.query.count);
 
 	return libbpf_err_errno(ret);
 }
tools/lib/bpf/bpf.h +78 -14

113 lines not on this page · raw message

tools/lib/bpf/libbpf.c +9 -3
index 1c6588ad3e75..208aeee31823 100644
@@ -118,6 +118,8 @@ static const char * const attach_type_name[] = {
 	[BPF_TRACE_KPROBE_MULTI]	= "trace_kprobe_multi",
 	[BPF_STRUCT_OPS]		= "struct_ops",
 	[BPF_NETFILTER]			= "netfilter",
+	[BPF_TCX_INGRESS]		= "tcx_ingress",
+	[BPF_TCX_EGRESS]		= "tcx_egress",
 };
 
 static const char * const link_type_name[] = {
@@ -8673,9 +8675,13 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("ksyscall+",		KPROBE,	0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
-	SEC_DEF("tc",			SCHED_CLS, 0, SEC_NONE),
-	SEC_DEF("classifier",		SCHED_CLS, 0, SEC_NONE),
-	SEC_DEF("action",		SCHED_ACT, 0, SEC_NONE),
+	SEC_DEF("tc/ingress",		SCHED_CLS, BPF_TCX_INGRESS, SEC_NONE), /* alias for tcx */
+	SEC_DEF("tc/egress",		SCHED_CLS, BPF_TCX_EGRESS, SEC_NONE),  /* alias for tcx */
+	SEC_DEF("tcx/ingress",		SCHED_CLS, BPF_TCX_INGRESS, SEC_NONE),
+	SEC_DEF("tcx/egress",		SCHED_CLS, BPF_TCX_EGRESS, SEC_NONE),
+	SEC_DEF("tc",			SCHED_CLS, 0, SEC_NONE), /* deprecated / legacy, use tcx */
+	SEC_DEF("classifier",		SCHED_CLS, 0, SEC_NONE), /* deprecated / legacy, use tcx */
+	SEC_DEF("action",		SCHED_ACT, 0, SEC_NONE), /* deprecated / legacy, use tcx */
 	SEC_DEF("tracepoint+",		TRACEPOINT, 0, SEC_NONE, attach_tp),
 	SEC_DEF("tp+",			TRACEPOINT, 0, SEC_NONE, attach_tp),
 	SEC_DEF("raw_tracepoint+",	RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
tools/lib/bpf/libbpf.map +1 -0
index d9ec4407befa..a95d39bbef90 100644
@@ -396,4 +396,5 @@ LIBBPF_1.3.0 {
 	global:
 		bpf_obj_pin_opts;
 		bpf_program__attach_netfilter;
+		bpf_prog_detach_opts;
 } LIBBPF_1.2.0;
Signature · 1 line
2.34.1
author maintainer
#20 lore raw

PATCH v3 4/8 libbpf: Add link-based API for tcx

In reply to Daniel Borkmann

Implement tcx BPF link support for libbpf.

The bpf_program__attach_fd() API has been refactored slightly in order to pass bpf_link_create_opts pointer as input.

A new bpf_program__attach_tcx() has been added on top of this which allows for passing all relevant data via extensible struct bpf_tcx_opts.

The program sections tcx/ingress and tcx/egress correspond to the hook locations for tc ingress and egress, respectively.

For concrete usage examples, see the extensive selftests that have been developed as part of this series.

---
 tools/lib/bpf/bpf.c      | 19 ++++++++++--
 tools/lib/bpf/bpf.h      |  5 ++++
 tools/lib/bpf/libbpf.c   | 62 ++++++++++++++++++++++++++++++++++------
 tools/lib/bpf/libbpf.h   | 16 +++++++++++
 tools/lib/bpf/libbpf.map |  1 +
 5 files changed, 92 insertions(+), 11 deletions(-)
tools/lib/bpf/bpf.c +17 -2
index 3dfc43b477c3..d513c226b9aa 100644
@@ -717,9 +717,9 @@ int bpf_link_create(int prog_fd, int target_fd,
 		    const struct bpf_link_create_opts *opts)
 {
 	const size_t attr_sz = offsetofend(union bpf_attr, link_create);
-	__u32 target_btf_id, iter_info_len;
+	__u32 target_btf_id, iter_info_len, relative_id;
+	int fd, err, relative;
 	union bpf_attr attr;
-	int fd, err;
 
 	if (!OPTS_VALID(opts, bpf_link_create_opts))
 		return libbpf_err(-EINVAL);
@@ -781,6 +781,21 @@ int bpf_link_create(int prog_fd, int target_fd,
 		if (!OPTS_ZEROED(opts, netfilter))
 			return libbpf_err(-EINVAL);
 		break;
+	case BPF_TCX_INGRESS:
+	case BPF_TCX_EGRESS:
+		relative = OPTS_GET(opts, tcx.relative_fd, 0);
+		relative_id = OPTS_GET(opts, tcx.relative_id, 0);
+		if (relative > 0 && relative_id)
+			return libbpf_err(-EINVAL);
+		if (relative_id) {
+			relative = relative_id;
+			attr.link_create.flags |= BPF_F_ID;
+		}
+		attr.link_create.tcx.relative_fd = relative;
+		attr.link_create.tcx.expected_revision = OPTS_GET(opts, tcx.expected_revision, 0);
+		if (!OPTS_ZEROED(opts, tcx))
+			return libbpf_err(-EINVAL);
+		break;
 	default:
 		if (!OPTS_ZEROED(opts, flags))
 			return libbpf_err(-EINVAL);
tools/lib/bpf/bpf.h +5 -0
index 49e9d88fd9cf..044a74ffc38a 100644
@@ -401,6 +401,11 @@ struct bpf_link_create_opts {
 			__s32 priority;
 			__u32 flags;
 		} netfilter;
+		struct {
+			__u32 relative_fd;
+			__u32 relative_id;
+			__u64 expected_revision;
+		} tcx;
 	};
 	size_t :0;
 };
tools/lib/bpf/libbpf.c +53 -9

107 lines not on this page · raw message

tools/lib/bpf/libbpf.h +16 -0
index 10642ad69d76..33f60a318e81 100644
@@ -733,6 +733,22 @@ LIBBPF_API struct bpf_link *
 bpf_program__attach_netfilter(const struct bpf_program *prog,
 			      const struct bpf_netfilter_opts *opts);
 
+struct bpf_tcx_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	int ifindex;
+	__u32 flags;
+	__u32 relative_fd;
+	__u32 relative_id;
+	__u64 expected_revision;
+	size_t :0;
+};
+#define bpf_tcx_opts__last_field expected_revision
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_tcx(const struct bpf_program *prog,
+			const struct bpf_tcx_opts *opts);
+
 struct bpf_map;
 
 LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
tools/lib/bpf/libbpf.map +1 -0
index a95d39bbef90..2a2db5c78048 100644
@@ -397,4 +397,5 @@ LIBBPF_1.3.0 {
 		bpf_obj_pin_opts;
 		bpf_program__attach_netfilter;
 		bpf_prog_detach_opts;
+		bpf_program__attach_tcx;
 } LIBBPF_1.2.0;
Signature · 1 line
2.34.1
author maintainer
#21 lore raw

PATCH v3 5/8 libbpf: Add helper macro to clear opts structs

In reply to Daniel Borkmann

Add a small and generic LIBBPF_OPTS_CLEAR() helper macros which clears an opts structure and reinitializes its .sz member to place the structure size. I found this very useful when developing selftests, but it is also generic enough as a macro next to the existing LIBBPF_OPTS() which hides the .sz initialization, too.

---
 tools/lib/bpf/libbpf_common.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/lib/bpf/libbpf_common.h +11 -0
index 9a7937f339df..eb180023aa97 100644
@@ -70,4 +70,15 @@
 		};							    \
 	})
 
+/* Helper macro to clear a libbpf options struct
+ *
+ * Small helper macro to reset all fields and to reinitialize the common
+ * structure size member.
+ */
+#define LIBBPF_OPTS_CLEAR(NAME)						    \
+	do {								    \
+		memset(&NAME, 0, sizeof(NAME));				    \
+		NAME.sz = sizeof(NAME);					    \
+	} while (0)
+
 #endif /* __LIBBPF_LIBBPF_COMMON_H */
Signature · 1 line
2.34.1
author maintainer
#22 lore raw

PATCH v3 6/8 bpftool: Extend net dump with tcx progs

In reply to Daniel Borkmann

Add support to dump fd-based attach types via bpftool. This includes both the tc BPF link and attach ops programs. Dumped information contain the attach location, function entry name, program ID and link ID when applicable.

Example with tc BPF link:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 784 link id 10 bond0(4) bpf/egress cil_to_netdev prog id 804 link id 11

flow_dissector:

netfilter:

Example with tc BPF attach ops:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 654 bond0(4) bpf/egress cil_to_netdev prog id 672

flow_dissector:

netfilter:

---
 tools/bpf/bpftool/net.c | 86 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 82 insertions(+), 4 deletions(-)
tools/bpf/bpftool/net.c +82 -4
index 26a49965bf71..1ef1e880de61 100644
@@ -76,6 +76,11 @@ static const char * const attach_type_strings[] = {
 	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
 };
 
+static const char * const attach_loc_strings[] = {
+	[BPF_TCX_INGRESS]		= "bpf/ingress",
+	[BPF_TCX_EGRESS]		= "bpf/egress",
+};
+
 const size_t net_attach_type_size = ARRAY_SIZE(attach_type_strings);
 
 static enum net_attach_type parse_attach_type(const char *str)
@@ -422,8 +427,80 @@ static int dump_filter_nlmsg(void *cookie, void *msg, struct nlattr **tb)
 			      filter_info->devname, filter_info->ifindex);
 }
 
-static int show_dev_tc_bpf(int sock, unsigned int nl_pid,
-			   struct ip_devname_ifindex *dev)
+static const char *flags_strings(__u32 flags)
+{
+	return json_output ? "none" : "";
+}
+
+static int __show_dev_tc_bpf_name(__u32 id, char *name, size_t len)
+{
+	struct bpf_prog_info info = {};
+	__u32 ilen = sizeof(info);
+	int fd, ret;
+
+	fd = bpf_prog_get_fd_by_id(id);
+	if (fd < 0)
+		return fd;
+	ret = bpf_obj_get_info_by_fd(fd, &info, &ilen);
+	if (ret < 0)
+		goto out;
+	ret = -ENOENT;
+	if (info.name[0]) {
+		get_prog_full_name(&info, fd, name, len);
+		ret = 0;
+	}
+out:
+	close(fd);
+	return ret;
+}
+
+static void __show_dev_tc_bpf(const struct ip_devname_ifindex *dev,
+			      const enum bpf_attach_type loc)
+{
+	__u32 prog_flags[64] = {}, link_flags[64] = {}, i;
+	__u32 prog_ids[64] = {}, link_ids[64] = {};
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	char prog_name[MAX_PROG_FULL_NAME];
+	int ret;
+
+	optq.prog_ids = prog_ids;
+	optq.prog_attach_flags = prog_flags;
+	optq.link_ids = link_ids;
+	optq.link_attach_flags = link_flags;
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	ret = bpf_prog_query_opts(dev->ifindex, loc, &optq);
+	if (ret)
+		return;
+	for (i = 0; i < optq.count; i++) {
+		NET_START_OBJECT;
+		NET_DUMP_STR("devname", "%s", dev->devname);
+		NET_DUMP_UINT("ifindex", "(%u)", dev->ifindex);
+		NET_DUMP_STR("kind", " %s", attach_loc_strings[loc]);
+		ret = __show_dev_tc_bpf_name(prog_ids[i], prog_name,
+					     sizeof(prog_name));
+		if (!ret)
+			NET_DUMP_STR("name", " %s", prog_name);
+		NET_DUMP_UINT("prog_id", " prog id %u", prog_ids[i]);
+		if (prog_flags[i])
+			NET_DUMP_STR("prog_flags", "%s", flags_strings(prog_flags[i]));
+		if (link_ids[i])
+			NET_DUMP_UINT("link_id", " link id %u",
+				      link_ids[i]);
+		if (link_flags[i])
+			NET_DUMP_STR("link_flags", "%s", flags_strings(link_flags[i]));
+		NET_END_OBJECT_FINAL;
+	}
+}
+
+static void show_dev_tc_bpf(struct ip_devname_ifindex *dev)
+{
+	__show_dev_tc_bpf(dev, BPF_TCX_INGRESS);
+	__show_dev_tc_bpf(dev, BPF_TCX_EGRESS);
+}
+
+static int show_dev_tc_bpf_classic(int sock, unsigned int nl_pid,
+				   struct ip_devname_ifindex *dev)
 {
 	struct bpf_filter_t filter_info;
 	struct bpf_tcinfo_t tcinfo;
@@ -790,8 +867,9 @@ static int do_show(int argc, char **argv)
 	if (!ret) {
 		NET_START_ARRAY("tc", "%s:\n");
 		for (i = 0; i < dev_array.used_len; i++) {
-			ret = show_dev_tc_bpf(sock, nl_pid,
-					      &dev_array.devices[i]);
+			show_dev_tc_bpf(&dev_array.devices[i]);
+			ret = show_dev_tc_bpf_classic(sock, nl_pid,
+						      &dev_array.devices[i]);
 			if (ret)
 				break;
 		}
Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH bpf-next v3 6/8] bpftool: Extend net dump with tcx progs

Quoted 40 lines · Daniel Borkmann

Add support to dump fd-based attach types via bpftool. This includes both the tc BPF link and attach ops programs. Dumped information contain the attach location, function entry name, program ID and link ID when applicable.

Example with tc BPF link:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 784 link id 10 bond0(4) bpf/egress cil_to_netdev prog id 804 link id 11

flow_dissector:

netfilter:

Example with tc BPF attach ops:

  # ./bpftool net
  xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 654 bond0(4) bpf/egress cil_to_netdev prog id 672

flow_dissector:

netfilter:

---
 tools/bpf/bpftool/net.c | 86 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 82 insertions(+), 4 deletions(-)
tools/bpf/bpftool/net.c +0 -0
index 26a49965bf71..1ef1e880de61 100644

tools/bpf/bpftool/net.c static const char * const attach_type_strings[] = { · line 81

Quoted 6 lines
tools/bpf/bpftool/net.c +3 -0
@@ -76,6 +76,11 @@ static const char * const attach_type_strings[] = {
 	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
 };
 
+static const char * const attach_loc_strings[] = {
+	[BPF_TCX_INGRESS]		= "bpf/ingress",
+	[BPF_TCX_EGRESS]		= "bpf/egress",

Any reason we are not doing tcx/ingress & egress? To match the section names.

author maintainer
#24 lore raw

Re: [PATCH bpf-next v3 6/8] bpftool: Extend net dump with tcx progs

Quoted 44 lines · Stanislav Fomichev
Quoted 43 lines · Daniel Borkmann

Add support to dump fd-based attach types via bpftool. This includes both the tc BPF link and attach ops programs. Dumped information contain the attach location, function entry name, program ID and link ID when applicable.

Example with tc BPF link:

   # ./bpftool net
   xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 784 link id 10 bond0(4) bpf/egress cil_to_netdev prog id 804 link id 11

flow_dissector:

netfilter:

Example with tc BPF attach ops:

   # ./bpftool net
   xdp:

tc:
bond0(4) bpf/ingress cil_from_netdev prog id 654 bond0(4) bpf/egress cil_to_netdev prog id 672

flow_dissector:

netfilter:

---
  tools/bpf/bpftool/net.c | 86 +++++++++++++++++++++++++++++++++++++++--
  1 file changed, 82 insertions(+), 4 deletions(-)
tools/bpf/bpftool/net.c +0 -0
index 26a49965bf71..1ef1e880de61 100644
@@ -76,6 +76,11 @@ static const char * const attach_type_strings[] = {
  	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
  };
 
Quoted 6 lines

tools/bpf/bpftool/net.c static const char * const attach_type_strings[] = { · line 81

Quoted 3 lines
tools/bpf/bpftool/net.c +3 -0
+static const char * const attach_loc_strings[] = {
+	[BPF_TCX_INGRESS]		= "bpf/ingress",
+	[BPF_TCX_EGRESS]		= "bpf/egress",

Any reason we are not doing tcx/ingress & egress? To match the section names.

Ok, will change for v4 to tcx/{ingress,egress}.

author maintainer
#25 lore raw

PATCH v3 7/8 selftests/bpf: Add mprog API tests for BPF tcx opts

In reply to Daniel Borkmann

Add a big batch of test coverage to assert all aspects of the tcx opts attach, detach and query API:

  # ./vmtest.sh -- ./test_progs -t tc_opts
  [...]
  #238     tc_opts_after:OK
  #239     tc_opts_append:OK
  #240     tc_opts_basic:OK
  #241     tc_opts_before:OK
  #242     tc_opts_chain_classic:OK
  #243     tc_opts_demixed:OK
  #244     tc_opts_detach:OK
  #245     tc_opts_detach_after:OK
  #246     tc_opts_detach_before:OK
  #247     tc_opts_dev_cleanup:OK
  #248     tc_opts_invalid:OK
  #249     tc_opts_mixed:OK
  #250     tc_opts_prepend:OK
  #251     tc_opts_replace:OK
  #252     tc_opts_revision:OK
  Summary: 15/0 PASSED, 0 SKIPPED, 0 FAILED
---
 .../selftests/bpf/prog_tests/tc_helpers.h     |   72 +
 .../selftests/bpf/prog_tests/tc_opts.c        | 2182 +++++++++++++++++
 .../selftests/bpf/progs/test_tc_link.c        |   40 +
 3 files changed, 2294 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_opts.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_tc_link.c
tools/testing/selftests/bpf/prog_tests/tc_helpers.h new file +72 -0
new file mode 100644
index 000000000000..6c93215be8a3
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2023 Isovalent */
+#ifndef TC_HELPERS
+#define TC_HELPERS
+#include <test_progs.h>
+
+static inline __u32 id_from_prog_fd(int fd)
+{
+	struct bpf_prog_info prog_info = {};
+	__u32 prog_info_len = sizeof(prog_info);
+	int err;
+
+	err = bpf_obj_get_info_by_fd(fd, &prog_info, &prog_info_len);
+	if (!ASSERT_OK(err, "id_from_prog_fd"))
+		return 0;
+
+	ASSERT_NEQ(prog_info.id, 0, "prog_info.id");
+	return prog_info.id;
+}
+
+static inline __u32 id_from_link_fd(int fd)
+{
+	struct bpf_link_info link_info = {};
+	__u32 link_info_len = sizeof(link_info);
+	int err;
+
+	err = bpf_link_get_info_by_fd(fd, &link_info, &link_info_len);
+	if (!ASSERT_OK(err, "id_from_link_fd"))
+		return 0;
+
+	ASSERT_NEQ(link_info.id, 0, "link_info.id");
+	return link_info.id;
+}
+
+static inline __u32 ifindex_from_link_fd(int fd)
+{
+	struct bpf_link_info link_info = {};
+	__u32 link_info_len = sizeof(link_info);
+	int err;
+
+	err = bpf_link_get_info_by_fd(fd, &link_info, &link_info_len);
+	if (!ASSERT_OK(err, "id_from_link_fd"))
+		return 0;
+
+	return link_info.tcx.ifindex;
+}
+
+static inline void __assert_mprog_count(int target, int expected, bool miniq, int ifindex)
+{
+	__u32 count = 0, attach_flags = 0;
+	int err;
+
+	err = bpf_prog_query(ifindex, target, 0, &attach_flags,
+			     NULL, &count);
+	ASSERT_EQ(count, expected, "count");
+	if (!expected && !miniq)
+		ASSERT_EQ(err, -ENOENT, "prog_query");
+	else
+		ASSERT_EQ(err, 0, "prog_query");
+}
+
+static inline void assert_mprog_count(int target, int expected)
+{
+	__assert_mprog_count(target, expected, false, loopback);
+}
+
+static inline void assert_mprog_count_ifindex(int ifindex, int target, int expected)
+{
+	__assert_mprog_count(target, expected, false, ifindex);
+}
+
+#endif /* TC_HELPERS */
tools/testing/selftests/bpf/prog_tests/tc_opts.c new file +2182 -0

2,182 lines not on this page · raw message

tools/testing/selftests/bpf/progs/test_tc_link.c new file +40 -0
new file mode 100644
index 000000000000..ed1fd0e9cee9
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+char LICENSE[] SEC("license") = "GPL";
+
+bool seen_tc1;
+bool seen_tc2;
+bool seen_tc3;
+bool seen_tc4;
+
+SEC("tc/ingress")
+int tc1(struct __sk_buff *skb)
+{
+	seen_tc1 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc2(struct __sk_buff *skb)
+{
+	seen_tc2 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc3(struct __sk_buff *skb)
+{
+	seen_tc3 = true;
+	return TCX_NEXT;
+}
+
+SEC("tc/egress")
+int tc4(struct __sk_buff *skb)
+{
+	seen_tc4 = true;
+	return TCX_NEXT;
+}
Signature · 1 line
2.34.1
author maintainer
#26 lore raw

PATCH v3 8/8 selftests/bpf: Add mprog API tests for BPF tcx links

In reply to Daniel Borkmann

Add a big batch of test coverage to assert all aspects of the tcx link API:

  # ./vmtest.sh -- ./test_progs -t tc_links
  [...]
  #225     tc_links_after:OK
  #226     tc_links_append:OK
  #227     tc_links_basic:OK
  #228     tc_links_before:OK
  #229     tc_links_chain_classic:OK
  #230     tc_links_dev_cleanup:OK
  #231     tc_links_invalid:OK
  #232     tc_links_prepend:OK
  #233     tc_links_replace:OK
  #234     tc_links_revision:OK
  Summary: 10/0 PASSED, 0 SKIPPED, 0 FAILED
---
 .../selftests/bpf/prog_tests/tc_links.c       | 1604 +++++++++++++++++
 1 file changed, 1604 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/tc_links.c
tools/testing/selftests/bpf/prog_tests/tc_links.c new file +1604 -0
new file mode 100644
index 000000000000..6a05f75492b5
@@ -0,0 +1,1604 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2023 Isovalent */
+#include <uapi/linux/if_link.h>
+#include <net/if.h>
+#include <test_progs.h>
+
+#define loopback 1
+#define ping_cmd "ping -q -c1 -w1 127.0.0.1 > /dev/null"
+
+#include "test_tc_link.skel.h"
+#include "tc_helpers.h"
+
+void serial_test_tc_links_basic(void)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[2], link_ids[2];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(BPF_TCX_INGRESS, 0);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+
+	ASSERT_EQ(skel->bss->seen_tc1, false, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(BPF_TCX_INGRESS, 1);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, BPF_TCX_INGRESS, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+	ASSERT_NEQ(lid1, lid2, "link_ids_1_2");
+
+	assert_mprog_count(BPF_TCX_INGRESS, 1);
+	assert_mprog_count(BPF_TCX_EGRESS, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, BPF_TCX_EGRESS, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 2, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+cleanup:
+	test_tc_link__destroy(skel);
+
+	assert_mprog_count(BPF_TCX_INGRESS, 0);
+	assert_mprog_count(BPF_TCX_EGRESS, 0);
+}
+
+static void test_tc_links_before_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK;
+	optl.relative_id = lid1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid4, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid4, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid3, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], lid3, "link_ids[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid2, "prog_ids[3]");
+	ASSERT_EQ(optq.link_ids[3], lid2, "link_ids[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_before(void)
+{
+	test_tc_links_before_target(BPF_TCX_INGRESS);
+	test_tc_links_before_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_after_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid3, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid3, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid2, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], lid2, "link_ids[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid4, "prog_ids[3]");
+	ASSERT_EQ(optq.link_ids[3], lid4, "link_ids[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_after(void)
+{
+	test_tc_links_after_target(BPF_TCX_INGRESS);
+	test_tc_links_after_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_revision_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[3], link_ids[3];
+	__u32 pid1, pid2, lid1, lid2;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.expected_revision = 1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	optl.expected_revision = 1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 1);
+
+	optl.expected_revision = 2;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "prog_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_revision(void)
+{
+	test_tc_links_revision_target(BPF_TCX_INGRESS);
+	test_tc_links_revision_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_chain_classic(int target, bool chain_tc_old)
+{
+	LIBBPF_OPTS(bpf_tc_opts, tc_opts, .handle = 1, .priority = 1);
+	LIBBPF_OPTS(bpf_tc_hook, tc_hook, .ifindex = loopback);
+	bool hook_created = false, tc_attached = false;
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	__u32 pid1, pid2, pid3;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	if (chain_tc_old) {
+		tc_hook.attach_point = target == BPF_TCX_INGRESS ?
+				       BPF_TC_INGRESS : BPF_TC_EGRESS;
+		err = bpf_tc_hook_create(&tc_hook);
+		if (err == 0)
+			hook_created = true;
+		err = err == -EEXIST ? 0 : err;
+		if (!ASSERT_OK(err, "bpf_tc_hook_create"))
+			goto cleanup;
+
+		tc_opts.prog_fd = bpf_program__fd(skel->progs.tc3);
+		err = bpf_tc_attach(&tc_hook, &tc_opts);
+		if (!ASSERT_OK(err, "bpf_tc_attach"))
+			goto cleanup;
+		tc_attached = true;
+	}
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, chain_tc_old, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__detach(skel->links.tc2);
+	if (!ASSERT_OK(err, "prog_detach"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, chain_tc_old, "seen_tc3");
+cleanup:
+	if (tc_attached) {
+		tc_opts.flags = tc_opts.prog_fd = tc_opts.prog_id = 0;
+		err = bpf_tc_detach(&tc_hook, &tc_opts);
+		ASSERT_OK(err, "bpf_tc_detach");
+	}
+	if (hook_created) {
+		tc_hook.attach_point = BPF_TC_INGRESS | BPF_TC_EGRESS;
+		bpf_tc_hook_destroy(&tc_hook);
+	}
+	assert_mprog_count(target, 1);
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_chain_classic(void)
+{
+	test_tc_chain_classic(BPF_TCX_INGRESS, false);
+	test_tc_chain_classic(BPF_TCX_EGRESS, false);
+	test_tc_chain_classic(BPF_TCX_INGRESS, true);
+	test_tc_chain_classic(BPF_TCX_EGRESS, true);
+}
+
+static void test_tc_links_replace_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, pid3, lid1, lid2;
+	__u32 prog_ids[4], link_ids[4];
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	optl.expected_revision = 1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_id = pid1;
+	optl.expected_revision = 2;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_REPLACE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+	optl.expected_revision = 3;
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 2);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_REPLACE | BPF_F_LINK;
+	optl.relative_fd = bpf_link__fd(skel->links.tc2);
+	optl.expected_revision = 3;
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 2);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_REPLACE | BPF_F_LINK | BPF_F_AFTER;
+	optl.relative_id = lid2;
+	optl.expected_revision = 0;
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 2);
+
+	err = bpf_link__update_program(skel->links.tc2, skel->progs.tc3);
+	if (!ASSERT_OK(err, "link_update"))
+		goto cleanup;
+
+	assert_mprog_count(target, 2);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 4, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid3, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__detach(skel->links.tc2);
+	if (!ASSERT_OK(err, "link_detach"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+	skel->bss->seen_tc3 = false;
+
+	err = bpf_link__update_program(skel->links.tc1, skel->progs.tc1);
+	if (!ASSERT_OK(err, "link_update_self"))
+		goto cleanup;
+
+	assert_mprog_count(target, 1);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 1, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], 0, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], 0, "link_ids[1]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, false, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_replace(void)
+{
+	test_tc_links_replace_target(BPF_TCX_INGRESS);
+	test_tc_links_replace_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_invalid_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, lid1;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+
+	assert_mprog_count(target, 0);
+
+	optl.flags = BPF_F_BEFORE | BPF_F_AFTER;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_ID;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER | BPF_F_ID;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_ID;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_LINK;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_ID;
+	optl.relative_id = pid2;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_ID;
+	optl.relative_id = 42;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 0);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER | BPF_F_LINK;
+	optl.relative_fd = bpf_program__fd(skel->progs.tc1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = ~0;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = lid1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_ID;
+	optl.relative_id = pid1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_ERR_PTR(link, "link_attach_should_fail")) {
+		bpf_link__destroy(link);
+		goto cleanup;
+	}
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE | BPF_F_LINK | BPF_F_ID;
+	optl.relative_id = lid1;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	assert_mprog_count(target, 2);
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_invalid(void)
+{
+	test_tc_links_invalid_target(BPF_TCX_INGRESS);
+	test_tc_links_invalid_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_prepend_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid2, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid2, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid1, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid1, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_BEFORE;
+
+	link = bpf_program__attach_tcx(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid4, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid4, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid3, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid3, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid2, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], lid2, "link_ids[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid1, "prog_ids[3]");
+	ASSERT_EQ(optq.link_ids[3], lid1, "link_ids[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_prepend(void)
+{
+	test_tc_links_prepend_target(BPF_TCX_INGRESS);
+	test_tc_links_prepend_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_append_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl,
+		.ifindex = loopback,
+	);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 prog_ids[5], link_ids[5];
+	__u32 pid1, pid2, pid3, pid4;
+	__u32 lid1, lid2, lid3, lid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	lid1 = id_from_link_fd(bpf_link__fd(skel->links.tc1));
+
+	assert_mprog_count(target, 1);
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER;
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	lid2 = id_from_link_fd(bpf_link__fd(skel->links.tc2));
+
+	assert_mprog_count(target, 2);
+
+	optq.prog_ids = prog_ids;
+	optq.link_ids = link_ids;
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 2, "count");
+	ASSERT_EQ(optq.revision, 3, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], 0, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], 0, "link_ids[2]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, false, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, false, "seen_tc4");
+
+	skel->bss->seen_tc1 = false;
+	skel->bss->seen_tc2 = false;
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER;
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc3 = link;
+
+	lid3 = id_from_link_fd(bpf_link__fd(skel->links.tc3));
+
+	LIBBPF_OPTS_CLEAR(optl);
+	optl.ifindex = loopback;
+	optl.flags = BPF_F_AFTER;
+
+	link = bpf_program__attach_tcx(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc4 = link;
+
+	lid4 = id_from_link_fd(bpf_link__fd(skel->links.tc4));
+
+	assert_mprog_count(target, 4);
+
+	memset(prog_ids, 0, sizeof(prog_ids));
+	memset(link_ids, 0, sizeof(link_ids));
+	optq.count = ARRAY_SIZE(prog_ids);
+
+	err = bpf_prog_query_opts(loopback, target, &optq);
+	if (!ASSERT_OK(err, "prog_query"))
+		goto cleanup;
+
+	ASSERT_EQ(optq.count, 4, "count");
+	ASSERT_EQ(optq.revision, 5, "revision");
+	ASSERT_EQ(optq.prog_ids[0], pid1, "prog_ids[0]");
+	ASSERT_EQ(optq.link_ids[0], lid1, "link_ids[0]");
+	ASSERT_EQ(optq.prog_ids[1], pid2, "prog_ids[1]");
+	ASSERT_EQ(optq.link_ids[1], lid2, "link_ids[1]");
+	ASSERT_EQ(optq.prog_ids[2], pid3, "prog_ids[2]");
+	ASSERT_EQ(optq.link_ids[2], lid3, "link_ids[2]");
+	ASSERT_EQ(optq.prog_ids[3], pid4, "prog_ids[3]");
+	ASSERT_EQ(optq.link_ids[3], lid4, "link_ids[3]");
+	ASSERT_EQ(optq.prog_ids[4], 0, "prog_ids[4]");
+	ASSERT_EQ(optq.link_ids[4], 0, "link_ids[4]");
+
+	ASSERT_OK(system(ping_cmd), ping_cmd);
+
+	ASSERT_EQ(skel->bss->seen_tc1, true, "seen_tc1");
+	ASSERT_EQ(skel->bss->seen_tc2, true, "seen_tc2");
+	ASSERT_EQ(skel->bss->seen_tc3, true, "seen_tc3");
+	ASSERT_EQ(skel->bss->seen_tc4, true, "seen_tc4");
+cleanup:
+	test_tc_link__destroy(skel);
+	assert_mprog_count(target, 0);
+}
+
+void serial_test_tc_links_append(void)
+{
+	test_tc_links_append_target(BPF_TCX_INGRESS);
+	test_tc_links_append_target(BPF_TCX_EGRESS);
+}
+
+static void test_tc_links_dev_cleanup_target(int target)
+{
+	LIBBPF_OPTS(bpf_tcx_opts, optl);
+	LIBBPF_OPTS(bpf_prog_query_opts, optq);
+	__u32 pid1, pid2, pid3, pid4;
+	struct test_tc_link *skel;
+	struct bpf_link *link;
+	int err, ifindex;
+
+	ASSERT_OK(system("ip link add dev tcx_opts1 type veth peer name tcx_opts2"), "add veth");
+	ifindex = if_nametoindex("tcx_opts1");
+	ASSERT_NEQ(ifindex, 0, "non_zero_ifindex");
+	optl.ifindex = ifindex;
+
+	skel = test_tc_link__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc1, target),
+		  0, "tc1_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc2, target),
+		  0, "tc2_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc3, target),
+		  0, "tc3_attach_type");
+	ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc4, target),
+		  0, "tc4_attach_type");
+
+	err = test_tc_link__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	pid1 = id_from_prog_fd(bpf_program__fd(skel->progs.tc1));
+	pid2 = id_from_prog_fd(bpf_program__fd(skel->progs.tc2));
+	pid3 = id_from_prog_fd(bpf_program__fd(skel->progs.tc3));
+	pid4 = id_from_prog_fd(bpf_program__fd(skel->progs.tc4));
+
+	ASSERT_NEQ(pid1, pid2, "prog_ids_1_2");
+	ASSERT_NEQ(pid3, pid4, "prog_ids_3_4");
+	ASSERT_NEQ(pid2, pid3, "prog_ids_2_3");
+
+	assert_mprog_count(target, 0);
+
+	link = bpf_program__attach_tcx(skel->progs.tc1, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc1 = link;
+
+	assert_mprog_count_ifindex(ifindex, target, 1);
+
+	link = bpf_program__attach_tcx(skel->progs.tc2, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc2 = link;
+
+	assert_mprog_count_ifindex(ifindex, target, 2);
+
+	link = bpf_program__attach_tcx(skel->progs.tc3, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc3 = link;
+
+	assert_mprog_count_ifindex(ifindex, target, 3);
+
+	link = bpf_program__attach_tcx(skel->progs.tc4, &optl);
+	if (!ASSERT_OK_PTR(link, "link_attach"))
+		goto cleanup;
+
+	skel->links.tc4 = link;
+
+	assert_mprog_count_ifindex(ifindex, target, 4);
+
+	ASSERT_OK(system("ip link del dev tcx_opts1"), "del veth");
+	ASSERT_EQ(if_nametoindex("tcx_opts1"), 0, "dev1_removed");
+	ASSERT_EQ(if_nametoindex("tcx_opts2"), 0, "dev2_removed");
+
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc1)), 0, "tc1_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc2)), 0, "tc2_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc3)), 0, "tc3_ifindex");
+	ASSERT_EQ(ifindex_from_link_fd(bpf_link__fd(skel->links.tc4)), 0, "tc4_ifindex");
+
+	test_tc_link__destroy(skel);
+	return;
+cleanup:
+	test_tc_link__destroy(skel);
+
+	ASSERT_OK(system("ip link del dev tcx_opts1"), "del veth");
+	ASSERT_EQ(if_nametoindex("tcx_opts1"), 0, "dev1_removed");
+	ASSERT_EQ(if_nametoindex("tcx_opts2"), 0, "dev2_removed");
+}
+
+void serial_test_tc_links_dev_cleanup(void)
+{
+	test_tc_links_dev_cleanup_target(BPF_TCX_INGRESS);
+	test_tc_links_dev_cleanup_target(BPF_TCX_EGRESS);
+}
Signature · 1 line
2.34.1