Skip to contentSkip to conversation

Global subprogs in RCU/{preempt,irq}-disabled sections

PATCH Superseded Kumar Kartikeya Dwivedi 1 participant 5 messages Mar 1, 2025

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v2 cover Global subprogs in RCU/{preempt,irq}-disabled sections

Small change to allow non-sleepable global subprogs in RCU, preempt-disabled, and irq-disabled sections. For now, we don't lift the limitation for locks as it requires more analysis, and will do this one resilient spin locks land.

This surfaced a bug where sleepable global subprogs were allowed in RCU read sections, that has been fixed. Tests have been added to cover various cases.

Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Rename subprog_info[i].sleepable to might_sleep, which more accurately reflects the nature of the bit. 'sleepable' means whether a given context is allowed to, while might_sleep captures if it does.
* Disallow extensions that might sleep to attach to targets that don't sleep, since they'd be permitted to be called in atomic contexts. (Eduard)
* Add tests for mixing non-sleepable and sleepable global function calls, and extensions attaching to non-sleepable global functions. (Eduard)
* Rename changes_pkt_data -> summarization

Kumar Kartikeya Dwivedi (3):
bpf: Summarize sleepable global subprogs selftests/bpf: Test sleepable global subprogs in atomic contexts selftests/bpf: Add tests for extending sleepable global subprogs

 include/linux/bpf.h                           |   1 +
 include/linux/bpf_verifier.h                  |   1 +
 kernel/bpf/verifier.c                         |  62 ++++++--
 .../bpf/prog_tests/changes_pkt_data.c         | 107 -------------
 .../selftests/bpf/prog_tests/rcu_read_lock.c  |   3 +
 .../selftests/bpf/prog_tests/spin_lock.c      |   3 +
 .../selftests/bpf/prog_tests/summarization.c  | 143 ++++++++++++++++++
 tools/testing/selftests/bpf/progs/irq.c       |  71 ++++++++-
 .../selftests/bpf/progs/preempt_lock.c        |  68 ++++++++-
 .../selftests/bpf/progs/rcu_read_lock.c       |  58 +++++++
 .../{changes_pkt_data.c => summarization.c}   |  38 ++++-
 ...ta_freplace.c => summarization_freplace.c} |  13 ++
 .../selftests/bpf/progs/test_spin_lock_fail.c |  69 +++++++++
 13 files changed, 513 insertions(+), 124 deletions(-)
 delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
 rename tools/testing/selftests/bpf/progs/{changes_pkt_data.c => summarization.c} (52%)
 rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (66%)

base-commit: 0b9363131daf4227d5ae11ee677acdcfff06e938

Signature · 1 line
2.43.5
author maintainer
#1 lore raw

PATCH v2 1/3 bpf: Summarize sleepable global subprogs

The verifier currently does not permit global subprog calls when a lock is held, preemption is disabled, or when IRQs are disabled. This is because we don't know whether the global subprog calls sleepable functions or not.

In case of locks, there's an additional reason: functions called by the global subprog may hold additional locks etc. The verifier won't know while verifying the global subprog whether it was called in context where a spin lock is already held by the program.

Perform summarization of the sleepable nature of a global subprog just like changes_pkt_data and then allow calls to global subprogs for non-sleepable ones from atomic context.

While making this change, I noticed that RCU read sections had no protection against sleepable global subprog calls, include it in the checks and fix this while we're at it.

Care needs to be taken to not allow global subprog calls when regular bpf_spin_lock is held. When resilient spin locks is held, we want to potentially have this check relaxed, but not for now.

Also make sure extensions freplacing global functions cannot do so in case the target is non-sleepable, but the extension is. The other combination is ok.

Tests are included in the next patch to handle all special conditions.

Fixes 9bb00b2895cb ("bpf: Add kfunc bpf_rcu_read_lock/unlock()")
---
 include/linux/bpf.h          |  1 +
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/verifier.c        | 62 ++++++++++++++++++++++++++++--------
 3 files changed, 50 insertions(+), 14 deletions(-)
include/linux/bpf.h +1 -0
index aec102868b93..4c4028d865ee 100644
@@ -1531,6 +1531,7 @@ struct bpf_prog_aux {
 	bool jits_use_priv_stack;
 	bool priv_stack_requested;
 	bool changes_pkt_data;
+	bool might_sleep;
 	u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
 	struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
 	struct bpf_arena *arena;
include/linux/bpf_verifier.h +1 -0
index bbd013c38ff9..d338f2a96bba 100644
@@ -667,6 +667,7 @@ struct bpf_subprog_info {
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
+	bool might_sleep: 1;
 
 	enum priv_stack_mode priv_stack_mode;
 	u8 arg_cnt;
kernel/bpf/verifier.c +48 -14

122 lines not on this page · raw message

Signature · 1 line
2.43.5
author maintainer
#2 lore raw

PATCH v2 2/3 selftests/bpf: Test sleepable global subprogs in atomic contexts

In reply to Kumar Kartikeya Dwivedi

Add tests for rejecting sleepable and accepting non-sleepable global function calls in atomic contexts. For spin locks, we still reject all global function calls. Once resilient spin locks land, we will carefully lift in cases where we deem it safe.

---
 .../selftests/bpf/prog_tests/rcu_read_lock.c  |  3 +
 .../selftests/bpf/prog_tests/spin_lock.c      |  3 +
 tools/testing/selftests/bpf/progs/irq.c       | 71 ++++++++++++++++++-
 .../selftests/bpf/progs/preempt_lock.c        | 68 +++++++++++++++++-
 .../selftests/bpf/progs/rcu_read_lock.c       | 58 +++++++++++++++
 .../selftests/bpf/progs/test_spin_lock_fail.c | 69 ++++++++++++++++++
 6 files changed, 270 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +3 -0
index ebe0c12b5536..c9f855e5da24 100644
@@ -81,6 +81,9 @@ static const char * const inproper_region_tests[] = {
 	"nested_rcu_region",
 	"rcu_read_lock_global_subprog_lock",
 	"rcu_read_lock_global_subprog_unlock",
+	"rcu_read_lock_sleepable_helper_global_subprog",
+	"rcu_read_lock_sleepable_kfunc_global_subprog",
+	"rcu_read_lock_sleepable_global_subprog_indirect",
 };
 
 static void test_inproper_region(void)
tools/testing/selftests/bpf/prog_tests/spin_lock.c +3 -0
index 2b0068742ef9..e3ea5dc2f697 100644
@@ -50,6 +50,9 @@ static struct {
 	{ "lock_id_mismatch_innermapval_mapval", "bpf_spin_unlock of different lock" },
 	{ "lock_global_subprog_call1", "global function calls are not allowed while holding a lock" },
 	{ "lock_global_subprog_call2", "global function calls are not allowed while holding a lock" },
+	{ "lock_global_sleepable_helper_subprog", "global function calls are not allowed while holding a lock" },
+	{ "lock_global_sleepable_kfunc_subprog", "global function calls are not allowed while holding a lock" },
+	{ "lock_global_sleepable_subprog_indirect", "global function calls are not allowed while holding a lock" },
 };
 
 static int match_regex(const char *pattern, const char *string)
tools/testing/selftests/bpf/progs/irq.c +70 -1

81 lines not on this page · raw message

tools/testing/selftests/bpf/progs/preempt_lock.c +67 -1

78 lines not on this page · raw message

tools/testing/selftests/bpf/progs/rcu_read_lock.c +58 -0

61 lines not on this page · raw message

tools/testing/selftests/bpf/progs/test_spin_lock_fail.c +69 -0

73 lines not on this page · raw message

Signature · 1 line
2.43.5
author maintainer
#3 lore raw

PATCH v2 3/3 selftests/bpf: Add tests for extending sleepable global subprogs

In reply to Kumar Kartikeya Dwivedi

Add tests for freplace behavior with the combination of sleepable and non-sleepable global subprogs. The changes_pkt_data selftest did all the hardwork, so simply rename it and include new support for more summarization tests for might_sleep bit.

Sleepable extensions don't work yet, so add support but disable it for now, allow support to be tested once it's enabled (and ensure we will complain then).

---
 .../bpf/prog_tests/changes_pkt_data.c         | 107 -------------
 .../selftests/bpf/prog_tests/summarization.c  | 143 ++++++++++++++++++
 .../{changes_pkt_data.c => summarization.c}   |  38 ++++-
 ...ta_freplace.c => summarization_freplace.c} |  13 ++
 4 files changed, 193 insertions(+), 108 deletions(-)
 delete mode 100644 tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/summarization.c
 rename tools/testing/selftests/bpf/progs/{changes_pkt_data.c => summarization.c} (52%)
 rename tools/testing/selftests/bpf/progs/{changes_pkt_data_freplace.c => summarization_freplace.c} (66%)
tools/testing/selftests/bpf/prog_tests/changes_pkt_data.c deleted +0 -107
deleted file mode 100644
index 7526de379081..000000000000
@@ -1,107 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include "bpf/libbpf.h"
-#include "changes_pkt_data_freplace.skel.h"
-#include "changes_pkt_data.skel.h"
-#include <test_progs.h>
-
-static void print_verifier_log(const char *log)
-{
-	if (env.verbosity >= VERBOSE_VERY)
-		fprintf(stdout, "VERIFIER LOG:\n=============\n%s=============\n", log);
-}
-
-static void test_aux(const char *main_prog_name,
-		     const char *to_be_replaced,
-		     const char *replacement,
-		     bool expect_load)
-{
-	struct changes_pkt_data_freplace *freplace = NULL;
-	struct bpf_program *freplace_prog = NULL;
-	struct bpf_program *main_prog = NULL;
-	LIBBPF_OPTS(bpf_object_open_opts, opts);
-	struct changes_pkt_data *main = NULL;
-	char log[16*1024];
-	int err;
-
-	opts.kernel_log_buf = log;
-	opts.kernel_log_size = sizeof(log);
-	if (env.verbosity >= VERBOSE_SUPER)
-		opts.kernel_log_level = 1 | 2 | 4;
-	main = changes_pkt_data__open_opts(&opts);
-	if (!ASSERT_OK_PTR(main, "changes_pkt_data__open"))
-		goto out;
-	main_prog = bpf_object__find_program_by_name(main->obj, main_prog_name);
-	if (!ASSERT_OK_PTR(main_prog, "main_prog"))
-		goto out;
-	bpf_program__set_autoload(main_prog, true);
-	err = changes_pkt_data__load(main);
-	print_verifier_log(log);
-	if (!ASSERT_OK(err, "changes_pkt_data__load"))
-		goto out;
-	freplace = changes_pkt_data_freplace__open_opts(&opts);
-	if (!ASSERT_OK_PTR(freplace, "changes_pkt_data_freplace__open"))
-		goto out;
-	freplace_prog = bpf_object__find_program_by_name(freplace->obj, replacement);
-	if (!ASSERT_OK_PTR(freplace_prog, "freplace_prog"))
-		goto out;
-	bpf_program__set_autoload(freplace_prog, true);
-	bpf_program__set_autoattach(freplace_prog, true);
-	bpf_program__set_attach_target(freplace_prog,
-				       bpf_program__fd(main_prog),
-				       to_be_replaced);
-	err = changes_pkt_data_freplace__load(freplace);
-	print_verifier_log(log);
-	if (expect_load) {
-		ASSERT_OK(err, "changes_pkt_data_freplace__load");
-	} else {
-		ASSERT_ERR(err, "changes_pkt_data_freplace__load");
-		ASSERT_HAS_SUBSTR(log, "Extension program changes packet data", "error log");
-	}
-
-out:
-	changes_pkt_data_freplace__destroy(freplace);
-	changes_pkt_data__destroy(main);
-}
-
-/* There are two global subprograms in both changes_pkt_data.skel.h:
- * - one changes packet data;
- * - another does not.
- * It is ok to freplace subprograms that change packet data with those
- * that either do or do not. It is only ok to freplace subprograms
- * that do not change packet data with those that do not as well.
- * The below tests check outcomes for each combination of such freplace.
- * Also test a case when main subprogram itself is replaced and is a single
- * subprogram in a program.
- */
-void test_changes_pkt_data_freplace(void)
-{
-	struct {
-		const char *main;
-		const char *to_be_replaced;
-		bool changes;
-	} mains[] = {
-		{ "main_with_subprogs",   "changes_pkt_data",         true },
-		{ "main_with_subprogs",   "does_not_change_pkt_data", false },
-		{ "main_changes",         "main_changes",             true },
-		{ "main_does_not_change", "main_does_not_change",     false },
-	};
-	struct {
-		const char *func;
-		bool changes;
-	} replacements[] = {
-		{ "changes_pkt_data",         true },
-		{ "does_not_change_pkt_data", false }
-	};
-	char buf[64];
-
-	for (int i = 0; i < ARRAY_SIZE(mains); ++i) {
-		for (int j = 0; j < ARRAY_SIZE(replacements); ++j) {
-			snprintf(buf, sizeof(buf), "%s_with_%s",
-				 mains[i].to_be_replaced, replacements[j].func);
-			if (!test__start_subtest(buf))
-				continue;
-			test_aux(mains[i].main, mains[i].to_be_replaced, replacements[j].func,
-				 mains[i].changes || !replacements[j].changes);
-		}
-	}
-}
tools/testing/selftests/bpf/prog_tests/summarization.c new file +143 -0

143 lines not on this page · raw message

tools/testing/selftests/bpf/progs/summarization.c renamed from tools/testing/selftests/bpf/progs/changes_pkt_data.c +37 -1
similarity index 52%
rename from tools/testing/selftests/bpf/progs/changes_pkt_data.c
rename to tools/testing/selftests/bpf/progs/summarization.c
index 43cada48b28a..730342f7b37c 100644
@@ -16,7 +16,7 @@ long does_not_change_pkt_data(struct __sk_buff *sk)
 }
 
 SEC("?tc")
-int main_with_subprogs(struct __sk_buff *sk)
+int main_changes_with_subprogs(struct __sk_buff *sk)
 {
 	changes_pkt_data(sk);
 	does_not_change_pkt_data(sk);
@@ -36,4 +36,40 @@ int main_does_not_change(struct __sk_buff *sk)
 	return 0;
 }
 
+__noinline
+long might_sleep(int i)
+{
+	bpf_copy_from_user(&i, sizeof(i), NULL);
+	return i;
+}
+
+__noinline __weak
+long does_not_sleep(int i)
+{
+	return 0;
+}
+
+SEC("?syscall")
+int main_might_sleep_with_subprogs(void *ctx)
+{
+	might_sleep(0);
+	does_not_sleep(0);
+	return 0;
+}
+
+SEC("?syscall")
+int main_might_sleep(void *ctx)
+{
+	int i;
+
+	bpf_copy_from_user(&i, sizeof(i), NULL);
+	return i;
+}
+
+SEC("?syscall")
+int main_does_not_sleep(void *sk)
+{
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/summarization_freplace.c renamed from tools/testing/selftests/bpf/progs/changes_pkt_data_freplace.c +13 -0
similarity index 66%
rename from tools/testing/selftests/bpf/progs/changes_pkt_data_freplace.c
rename to tools/testing/selftests/bpf/progs/summarization_freplace.c
index f9a622705f1b..c813b1278138 100644
@@ -15,4 +15,17 @@ long does_not_change_pkt_data(struct __sk_buff *sk)
 	return 0;
 }
 
+SEC("?freplace")
+long might_sleep(int i)
+{
+	bpf_copy_from_user(&i, sizeof(i), NULL);
+	return i;
+}
+
+SEC("?freplace")
+long does_not_sleep(int i)
+{
+	return 0;
+}
+
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.5
author maintainer
#4 lore raw

Re: [PATCH bpf-next v2 3/3] selftests/bpf: Add tests for extending sleepable global subprogs

Quoted 23 lines · Kumar Kartikeya Dwivedi

Add tests for freplace behavior with the combination of sleepable and non-sleepable global subprogs. The changes_pkt_data selftest did all the hardwork, so simply rename it and include new support for more summarization tests for might_sleep bit.

Sleepable extensions don't work yet, so add support but disable it for now, allow support to be tested once it's enabled (and ensure we will complain then).

---
[...]

tools/testing/selftests/bpf/prog_tests/summarization.c new file +9 -0
+void test_summarization_freplace(void)
+{
+       struct {
+               const char *main;
+               const char *to_be_replaced;
+               bool has_side_effect;
+       } mains[2][4] = {
+               {
+                       { "main_changes_with_subprogs",         "changes_pkt_data",         true },

tools/testing/selftests/bpf/prog_tests/summarization.c line 110

Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/summarization.c new file +10 -0
+                       { "main_changes_with_subprogs",         "does_not_change_pkt_data", false },
+                       { "main_changes",                       "main_changes",             true },
+                       { "main_does_not_change",               "main_does_not_change",     false },
+               },
+               {
+                       { "main_might_sleep_with_subprogs",     "might_sleep",              true },
+                       { "main_might_sleep_with_subprogs",     "does_not_sleep",           false },
+                       { "main_might_sleep",                   "might_sleep",              true },
+                       { "main_does_not_sleep",                "does_not_sleep",           false },
+               },

Sigh, I forgot to regen with git format-patch with the hunks fixing this, it is obviously incorrect.
Let me resend again.