Skip to contentSkip to conversation

bpf: add BPF_F_SEALED flag to seal BPF links at creation

PATCH In review David Windsor 5 participants 10 messages Aug 3 – Aug 11

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpf: add BPF_F_SEALED flag to seal BPF links at creation

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

The sealed state is tracked by a new bool field on struct bpf_link.

---
 include/linux/bpf.h            |  2 ++
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           | 41 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 42 insertions(+), 3 deletions(-)
include/linux/bpf.h +2 -0
index 7bfc28673124..a9600a1483ec 100644
@@ -1934,6 +1934,8 @@ struct bpf_link {
 	 * link's semantics is determined by target attach hook
 	 */
 	bool sleepable;
+	/* set once by BPF_F_SEALED; blocks update/detach, pins link until reboot */
+	bool sealed;
 };
 
 struct bpf_link_ops {
include/uapi/linux/bpf.h +1 -0
index ffd96e8b920b..8cb30d18fec7 100644
@@ -1251,6 +1251,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_SEALED		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
kernel/bpf/syscall.c +38 -3

80 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +1 -0
index ffd96e8b920b..8cb30d18fec7 100644
@@ -1251,6 +1251,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_SEALED		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the

base-commit: 28e911d61d66b92a3bded8b54622ed3cd2795bf6

Signature · 1 line
2.53.0
#1 lore raw
Quoted 10 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

The sealed state is tracked by a new bool field on struct bpf_link.

Quoted 8 lines
---
 include/linux/bpf.h            |  2 ++
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           | 41 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 42 insertions(+), 3 deletions(-)

There is deliberately no selftest for this, as running any such selftest would necessarily create a sealed link, which cannot be cleaned up.

#2 lore raw
Quoted 20 lines · David Windsor
Quoted 10 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

The sealed state is tracked by a new bool field on struct bpf_link.

Quoted 8 lines
---
 include/linux/bpf.h            |  2 ++
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           | 41 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 42 insertions(+), 3 deletions(-)
Quoted 3 lines

There is deliberately no selftest for this, as running any such selftest would necessarily create a sealed link, which cannot be cleaned up.

Let's wait for more comments, esp. on whether this is useful or the right path forward, but I'd still prefer adding selftests to verify various aspects of the behavior if we proceed.

I think the point about a user being able to create many sealed links is definitely valid too, so it might make sense to keep this gated behind more privileged capabilities (if not already).

For tests, we might have to force decrement the link refcount (after verifying it stays elevated after close(2)) from the test. It might involve some creative use of a kfunc in bpf_testmod that can do the cleanup for us to avoid wedging the system on repeated invocations of the test. It wouldn't be available on a real system. It depends on whether we deem it important enough that leaking the link on VMs running tests is important enough. You'd probably have to run it thousands of time to exhaust memory unless the VM is provisioned with a tiny amount of memory.

Anyway, let's get into all that after seeing how others feel about the concept.

#3 lore raw
In reply to David Windsor
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The BPF_F_SEALED flag is applied via a file descriptor lookup after the FD has been exposed to…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The BPF_F_SEALED flag is applied via a file descriptor lookup after the FD has been exposed to user space, creating a TOCTOU race that allows cross-object sabotage and unauthorized link sealing.
- [High] The implementation of BPF_F_SEALED introduces a permanent memory leak, enabling a Denial of Service (DoS) via kernel resource exhaustion.
- [Medium] Concurrent executions of link_seal_fd on the same link can leak multiple self-references due to a missing atomic read-modify-write.
--

--- Patch [1]: [PATCH bpf-next] bpf: add BPF_F_SEALED flag to seal BPF links at creation --- commit f9d98fb378be461fdf9509562339b45a07b8d912
Author: David Windsor <[email protected]>

bpf: add BPF_F_SEALED flag to seal BPF links at creation

This commit introduces a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates a permanently sealed link. A sealed link cannot have its program replaced, cannot be detached, and holds an extra self-reference so it persists until reboot.

Quoted 4 lines
kernel/bpf/syscall.c +0 -0
index 94091130bcc50..e8bd57d5c907a 100644

[ ... ]

Quoted 4 lines
kernel/bpf/syscall.c +0 -0
@@ -5776,17 +5777,41 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 

kernel/bpf/syscall.c static int bpf_map_do_batch(const union bpf_attr *attr, · line 5789

Quoted 10 lines
kernel/bpf/syscall.c +10 -0
+/* Seal the just-created link: take a self-reference that is never released. */
+static void link_seal_fd(int fd)
+{
+	struct bpf_link *link;
+
+	link = bpf_link_get_from_fd(fd);
+	if (IS_ERR(link))
+		return;
+
+	if (!READ_ONCE(link->sealed)) {

[Severity: Medium]
Is this flag check safe against concurrent execution?

If the file descriptor is raced via dup2(), multiple threads might enter link_seal_fd() for the same victim link concurrently. Since the READ_ONCE() and WRITE_ONCE() are not atomic, could multiple threads observe false and increment the link reference count multiple times?

kernel/bpf/syscall.c static int bpf_map_do_batch(const union bpf_attr *attr, · line 5791

Quoted 2 lines
kernel/bpf/syscall.c +2 -0
+		bpf_link_inc(link);
+		WRITE_ONCE(link->sealed, true);

[Severity: High]
Does taking a self-reference here without any release path allow users to exhaust kernel memory?

Since link_detach() explicitly returns -EPERM for sealed links, a user could repeatedly create sealed links and close the fds, leaking the bpf_link and bpf_prog indefinitely. Is there an administrative way to reclaim this memory?

Quoted 4 lines
kernel/bpf/syscall.c +4 -0
+	}
+
+	bpf_link_put_direct(link);
+}

[ ... ]

kernel/bpf/syscall.c static int link_create(union bpf_attr *attr, bpfptr_t uattr) · line 5910

Quoted 7 lines
kernel/bpf/syscall.c +3 -0
@@ -5880,6 +5905,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 out:
 	if (ret < 0)
 		bpf_prog_put(prog);
+out_seal:
+	if (ret >= 0 && seal)
+		link_seal_fd(ret);

[Severity: High]
Does looking up the link from the file descriptor here create a race condition?

By the time link_create() reaches this point, attach functions called earlier have already installed the fd into the process's file descriptor table.

Could a concurrent thread use dup2() to overwrite this fd with one pointing to a different victim bpf_link before link_seal_fd() executes? This would cause the system to permanently seal the victim link instead of the newly created one.

Quoted 2 lines
kernel/bpf/syscall.c +0 -0
 	return ret;
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#4 lore raw
In reply to David Windsor
Quoted 8 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

The sealed state is tracked by a new bool field on struct bpf_link.

Why do you want BPF_F_SEALED? What's your case for it?

I think 'sealed' should be set before allocating the FD to avoid the race issues reported by Sashiko.

Thanks,
Leon

Quoted 1 line

[...]

#5 lore raw
Quoted 8 lines · Leon Hwang
Quoted 7 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

Quoted 8 lines
Quoted 1 line

The sealed state is tracked by a new bool field on struct bpf_link.

Why do you want BPF_F_SEALED? What's your case for it?

I think 'sealed' should be set before allocating the FD to avoid the race issues reported by Sashiko.

Agreed, then we would not need READ_ONCE() on the sealed bit either since the it stays immutable through the lifetime of the link.

Quoted 4 lines

Thanks,
Leon

Quoted 1 line

[...]

#6 lore raw
In reply to Leon Hwang
Quoted 9 lines · Leon Hwang
Quoted 7 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

Quoted 5 lines
Quoted 1 line

The sealed state is tracked by a new bool field on struct bpf_link.

Why do you want BPF_F_SEALED? What's your case for it?

For the BPF LSM use case, mainly. There is some language in the commitmsg about programs not being able to be unloaded until reboot, but I will be more explicit in v2.

There was a presentation ("Securing BPF LSMs against tampering") [1] at LSFMM+BPF about it.

[1]: https://lwn.net/Articles/1082111/

Quoted 3 lines

I think 'sealed' should be set before allocating the FD to avoid the race issues reported by Sashiko.

Will do this in v2.

Quoted 4 lines

Thanks,
Leon

Quoted 1 line

[...]

#7 lore raw
Quoted 17 lines · David Windsor
Quoted 9 lines · Leon Hwang
Quoted 7 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

Quoted 5 lines
Quoted 1 line

The sealed state is tracked by a new bool field on struct bpf_link.

Why do you want BPF_F_SEALED? What's your case for it?

Quoted 8 lines

For the BPF LSM use case, mainly. There is some language in the commitmsg about programs not being able to be unloaded until reboot, but I will be more explicit in v2.

There was a presentation ("Securing BPF LSMs against tampering") [1] at LSFMM+BPF about it.

[1]: https://lwn.net/Articles/1082111/

I had to skim the articles to recall that it was me proposing something like this :)

Anyways, as Kumar mentioned, a) link has to be crated as sealed before FD is exposed, it's straightforward, and b) I do think that it would be too easy to have syzbots of the world to accidentally or not create tons of sealed links, so I'd require CAP_SYS_ADMIN for this, which I assume won't be a problem for intended use cases?

But also, yeah, I wonder what Christian and other BPF LSM users think about this.

pw-bot: cr

Quoted 11 lines
Quoted 4 lines

I think 'sealed' should be set before allocating the FD to avoid the race issues reported by Sashiko.

Will do this in v2.

Quoted 4 lines

Thanks,
Leon

Quoted 1 line

[...]

#8 lore raw
Quoted 4 lines · Andrii Nakryiko

I had to skim the articles to recall that it was me proposing something like this :)

I wasn't actually there, but read about it on lwn. Happy to add your Suggested-By if that makes sense here.

Quoted 6 lines

Anyways, as Kumar mentioned, a) link has to be crated as sealed before FD is exposed, it's straightforward, and b) I do think that it would be too easy to have syzbots of the world to accidentally or not create tons of sealed links, so I'd require CAP_SYS_ADMIN for this, which I assume won't be a problem for intended use cases?

CAP_SYS_ADMIN makes sense.

Quoted 3 lines

But also, yeah, I wonder what Christian and other BPF LSM users think about this.

I'll send v2 with the above changes.

The overall intent here is to follow up with another series for map sealing. Currently, we're passing BPF_F_SEALED as a flag for BPF_LINK_CREATE, but I wonder if we should decompose BPF_F_SEALED into eg BPF_F_LINK_SEALED and BPF_F_MAPS_SEALED?

Link sealing is useful on its own, though, for preserving bpf-lsm programs around even after all user fd's have been closed.

#9 lore raw
In reply to David Windsor

+Cc Christian since he was involved in the discussion at LSF/MM/BPF.

Quoted 145 lines · David Windsor

Introduce a new BPF_F_SEALED flag for BPF_LINK_CREATE that creates the link permanently sealed. A sealed link can never have its program replaced via BPF_LINK_UPDATE, can never be detached via BPF_LINK_DETACH, and holds an extra self-reference that is never released, so the link and its program attachment persist until the machine reboots, even after user space closes every fd referring to it. There is no way to unseal a link.

The sealed state is tracked by a new bool field on struct bpf_link.

---
 include/linux/bpf.h            |  2 ++
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           | 41 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 42 insertions(+), 3 deletions(-)
include/linux/bpf.h +2 -0
index 7bfc28673124..a9600a1483ec 100644
@@ -1934,6 +1934,8 @@ struct bpf_link {
 	 * link's semantics is determined by target attach hook
 	 */
 	bool sleepable;
+	/* set once by BPF_F_SEALED; blocks update/detach, pins link until reboot */
+	bool sealed;
 };
 
 struct bpf_link_ops {
include/uapi/linux/bpf.h +1 -0
index ffd96e8b920b..8cb30d18fec7 100644
@@ -1251,6 +1251,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_SEALED		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
kernel/bpf/syscall.c +38 -3
index 94091130bcc5..e8bd57d5c907 100644
@@ -3411,6 +3411,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
 		seq_printf(m, "link_type:\t<%u>\n", type);
 	}
 	seq_printf(m, "link_id:\t%u\n", link->id);
+	seq_printf(m, "sealed:\t%d\n", READ_ONCE(link->sealed) ? 1 : 0);
 
 	rcu_read_lock();
 	prog = READ_ONCE(link->prog);
@@ -5776,17 +5777,41 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 
+/* Seal the just-created link: take a self-reference that is never released. */
+static void link_seal_fd(int fd)
+{
+	struct bpf_link *link;
+
+	link = bpf_link_get_from_fd(fd);
+	if (IS_ERR(link))
+		return;
+
+	if (!READ_ONCE(link->sealed)) {
+		bpf_link_inc(link);
+		WRITE_ONCE(link->sealed, true);
+	}
+
+	bpf_link_put_direct(link);
+}
+
 #define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd
 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 {
 	struct bpf_prog *prog;
+	bool seal;
 	int ret;
 
 	if (CHECK_ATTR(BPF_LINK_CREATE))
 		return -EINVAL;
 
-	if (attr->link_create.attach_type == BPF_STRUCT_OPS)
-		return bpf_struct_ops_link_create(attr);
+	/* Strip BPF_F_SEALED before per-type flag validation. */
+	seal = attr->link_create.flags & BPF_F_SEALED;
+	attr->link_create.flags &= ~BPF_F_SEALED;
+
+	if (attr->link_create.attach_type == BPF_STRUCT_OPS) {
+		ret = bpf_struct_ops_link_create(attr);
+		goto out_seal;
+	}
 
 	prog = bpf_prog_get(attr->link_create.prog_fd);
 	if (IS_ERR(prog))
@@ -5880,6 +5905,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 out:
 	if (ret < 0)
 		bpf_prog_put(prog);
+out_seal:
+	if (ret >= 0 && seal)
+		link_seal_fd(ret);
 	return ret;
 }
 
@@ -5932,6 +5960,11 @@ static int link_update(union bpf_attr *attr)
 	if (IS_ERR(link))
 		return PTR_ERR(link);
 
+	if (READ_ONCE(link->sealed)) {
+		ret = -EPERM;
+		goto out_put_link;
+	}
+
 	if (link->ops->update_map) {
 		ret = link_update_map(link, attr);
 		goto out_put_link;
@@ -5984,7 +6017,9 @@ static int link_detach(union bpf_attr *attr)
 	if (IS_ERR(link))
 		return PTR_ERR(link);
 
-	if (link->ops->detach)
+	if (READ_ONCE(link->sealed))
+		ret = -EPERM;
+	else if (link->ops->detach)
 		ret = link->ops->detach(link);
 	else
 		ret = -EOPNOTSUPP;
tools/include/uapi/linux/bpf.h +1 -0
index ffd96e8b920b..8cb30d18fec7 100644
@@ -1251,6 +1251,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_SEALED		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the

base-commit: 28e911d61d66b92a3bded8b54622ed3cd2795bf6