Skip to contentSkip to conversation

Introduce bpf_preempt_{disable,enable}

PATCH Applied Kumar Kartikeya Dwivedi 3 participants 5 messages Apr 24, 2024

Subsystems (4)
lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v2 cover Introduce bpf_preempt_{disable,enable}

This set introduces two kfuncs, bpf_preempt_disable and bpf_preempt_enable, which are wrappers around preempt_disable and preempt_enable in the kernel. These functions allow a BPF program to have code sections where preemption is disabled. There are multiple use cases that are served by such a feature, a few are listed below:

1. Writing safe per-CPU alogrithms/data structures that work correctly across different contexts.
2. Writing safe per-CPU allocators similar to bpf_memalloc on top of array/arena memory blobs.
3. Writing locking algorithms in BPF programs natively.

Note that local_irq_disable/enable equivalent is also needed for proper IRQ context protection, but that is a more involved change and will be sent later.

While bpf_preempt_{disable,enable} is not sufficient for all of these usage scenarios on its own, it is still necessary.

The same effect as these kfuncs can in some sense be already achieved using the bpf_spin_lock or rcu_read_lock APIs, therefore from the standpoint of kernel functionality exposure in the verifier, this is well understood territory.

Note that these helpers do allow calling kernel helpers and kfuncs from within the non-preemptible region (unless sleepable). Otherwise, any locks built using the preemption helpers will be as limited as existing bpf_spin_lock.

Nesting is allowed by keeping a counter for tracking remaining enables required to be performed. Similar approach can be applied to rcu_read_locks in a follow up.

Changelog
=========
v1: https://lore.kernel.org/bpf/[email protected]

* Move kfunc BTF ID declerations above css task kfunc for !CONFIG_CGROUPS config (Alexei)
* Add test case for global function call in non-preemptible region (Jiri)

Kumar Kartikeya Dwivedi (2):
bpf: Introduce bpf_preempt_[disable,enable] kfuncs selftests/bpf: Add tests for preempt kfuncs

 include/linux/bpf_verifier.h                  |   1 +
 kernel/bpf/helpers.c                          |  12 ++
 kernel/bpf/verifier.c                         |  71 ++++++++-
 .../selftests/bpf/prog_tests/preempt_lock.c   |   9 ++
 .../selftests/bpf/progs/preempt_lock.c        | 135 ++++++++++++++++++
 5 files changed, 226 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/preempt_lock.c
 create mode 100644 tools/testing/selftests/bpf/progs/preempt_lock.c

base-commit: 6e10b6350a67d398c795ac0b93a7bb7103633fe4

Signature · 1 line
2.43.0
author maintainer
#1 lore raw

PATCH v2 1/2 bpf: Introduce bpf_preempt_[disable,enable] kfuncs

Introduce two new BPF kfuncs, bpf_preempt_disable and bpf_preempt_enable. These kfuncs allow disabling preemption in BPF programs. Nesting is allowed, since the intended use cases includes building native BPF spin locks without kernel helper involvement. Apart from that, this can be used to per-CPU data structures for cases where programs (or userspace) may preempt one or the other. Currently, while per-CPU access is stable, whether it will be consistent is not guaranteed, as only migration is disabled for BPF programs.

Global functions are disallowed from being called, but support for them will be added as a follow up not just preempt kfuncs, but rcu_read_lock kfuncs as well. Static subprog calls are permitted. Sleepable helpers and kfuncs are disallowed in non-preemptible regions.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/helpers.c         | 12 ++++++
 kernel/bpf/verifier.c        | 71 +++++++++++++++++++++++++++++++++++-
 3 files changed, 82 insertions(+), 2 deletions(-)
include/linux/bpf_verifier.h +1 -0
index 9db35530c878..50aa87f8d77f 100644
@@ -421,6 +421,7 @@ struct bpf_verifier_state {
 	struct bpf_active_lock active_lock;
 	bool speculative;
 	bool active_rcu_lock;
+	u32 active_preempt_lock;
 	/* If this state was ever pointed-to by other state's loop_entry field
 	 * this flag would be set to true. Used to avoid freeing such states
 	 * while they are still in use.
kernel/bpf/helpers.c +12 -0
index 047a21b7e4ba..84997e95ab78 100644
@@ -2742,6 +2742,16 @@ __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
 	return __bpf_async_set_callback(async, callback_fn, aux, flags, BPF_ASYNC_TYPE_WQ);
 }
 
+__bpf_kfunc void bpf_preempt_disable(void)
+{
+	preempt_disable();
+}
+
+__bpf_kfunc void bpf_preempt_enable(void)
+{
+	preempt_enable();
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(generic_btf_ids)
@@ -2822,6 +2832,8 @@ BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
 BTF_ID_FLAGS(func, bpf_wq_init)
 BTF_ID_FLAGS(func, bpf_wq_set_callback_impl)
 BTF_ID_FLAGS(func, bpf_wq_start)
+BTF_ID_FLAGS(func, bpf_preempt_disable)
+BTF_ID_FLAGS(func, bpf_preempt_enable)
 BTF_KFUNCS_END(common_btf_ids)
 
 static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +69 -2

146 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH v2 2/2 selftests/bpf: Add tests for preempt kfuncs

In reply to Kumar Kartikeya Dwivedi

Add tests for nested cases, nested count preservation upon different subprog calls that disable/enable preemption, and test sleepable helper call in non-preemptible regions.

182/1   preempt_lock/preempt_lock_missing_1:OK
182/2   preempt_lock/preempt_lock_missing_2:OK
182/3   preempt_lock/preempt_lock_missing_3:OK
182/4   preempt_lock/preempt_lock_missing_3_minus_2:OK
182/5   preempt_lock/preempt_lock_missing_1_subprog:OK
182/6   preempt_lock/preempt_lock_missing_2_subprog:OK
182/7   preempt_lock/preempt_lock_missing_2_minus_1_subprog:OK
182/8   preempt_lock/preempt_balance:OK
182/9   preempt_lock/preempt_balance_subprog_test:OK
182/10  preempt_lock/preempt_global_subprog_test:OK
182/11  preempt_lock/preempt_sleepable_helper:OK
182     preempt_lock:OK

Summary: 1/11 PASSED, 0 SKIPPED, 0 FAILED

---
 .../selftests/bpf/prog_tests/preempt_lock.c   |   9 ++
 .../selftests/bpf/progs/preempt_lock.c        | 135 ++++++++++++++++++
 2 files changed, 144 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/preempt_lock.c
 create mode 100644 tools/testing/selftests/bpf/progs/preempt_lock.c
tools/testing/selftests/bpf/prog_tests/preempt_lock.c new file +9 -0
new file mode 100644
index 000000000000..02917c672441
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <network_helpers.h>
+#include <preempt_lock.skel.h>
+
+void test_preempt_lock(void)
+{
+	RUN_TESTS(preempt_lock);
+}
tools/testing/selftests/bpf/progs/preempt_lock.c new file +135 -0

135 lines not on this page · raw message

Signature · 1 line
2.43.0
Acked-by on the v2 series
#3 lore raw
In reply to Kumar Kartikeya Dwivedi
Quoted 33 lines · Kumar Kartikeya Dwivedi

This set introduces two kfuncs, bpf_preempt_disable and bpf_preempt_enable, which are wrappers around preempt_disable and preempt_enable in the kernel. These functions allow a BPF program to have code sections where preemption is disabled. There are multiple use cases that are served by such a feature, a few are listed below:

1. Writing safe per-CPU alogrithms/data structures that work correctly across different contexts.
2. Writing safe per-CPU allocators similar to bpf_memalloc on top of array/arena memory blobs.
3. Writing locking algorithms in BPF programs natively.

Note that local_irq_disable/enable equivalent is also needed for proper IRQ context protection, but that is a more involved change and will be sent later.

While bpf_preempt_{disable,enable} is not sufficient for all of these usage scenarios on its own, it is still necessary.

The same effect as these kfuncs can in some sense be already achieved using the bpf_spin_lock or rcu_read_lock APIs, therefore from the standpoint of kernel functionality exposure in the verifier, this is well understood territory.

Note that these helpers do allow calling kernel helpers and kfuncs from within the non-preemptible region (unless sleepable). Otherwise, any locks built using the preemption helpers will be as limited as existing bpf_spin_lock.

Nesting is allowed by keeping a counter for tracking remaining enables required to be performed. Similar approach can be applied to rcu_read_locks in a follow up.

Quoted 8 lines

Changelog
=========
v1: https://lore.kernel.org/bpf/[email protected]

* Move kfunc BTF ID declerations above css task kfunc for !CONFIG_CGROUPS config (Alexei)
* Add test case for global function call in non-preemptible region (Jiri)

jirka

Quoted 20 lines

Kumar Kartikeya Dwivedi (2):
bpf: Introduce bpf_preempt_[disable,enable] kfuncs selftests/bpf: Add tests for preempt kfuncs

 include/linux/bpf_verifier.h                  |   1 +
 kernel/bpf/helpers.c                          |  12 ++
 kernel/bpf/verifier.c                         |  71 ++++++++-
 .../selftests/bpf/prog_tests/preempt_lock.c   |   9 ++
 .../selftests/bpf/progs/preempt_lock.c        | 135 ++++++++++++++++++
 5 files changed, 226 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/preempt_lock.c
 create mode 100644 tools/testing/selftests/bpf/progs/preempt_lock.c

base-commit: 6e10b6350a67d398c795ac0b93a7bb7103633fe4

Signature · 1 line
2.43.0
#4 lore raw
In reply to Kumar Kartikeya Dwivedi
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 6 lines · On Wed, 24 Apr 2024 03:13:13 +0000 you wrote:

This set introduces two kfuncs, bpf_preempt_disable and bpf_preempt_enable, which are wrappers around preempt_disable and preempt_enable in the kernel. These functions allow a BPF program to have code sections where preemption is disabled. There are multiple use cases that are served by such a feature, a few are listed below:

Quoted 7 lines

1. Writing safe per-CPU alogrithms/data structures that work correctly across different contexts.
2. Writing safe per-CPU allocators similar to bpf_memalloc on top of array/arena memory blobs.
3. Writing locking algorithms in BPF programs natively.

[...]

Here is the summary with links:
- [bpf-next,v2,1/2] bpf: Introduce bpf_preempt_[disable,enable] kfuncs

    https://git.kernel.org/bpf/bpf-next/c/fc7566ad0a82
  - [bpf-next,v2,2/2] selftests/bpf: Add tests for preempt kfuncs
    https://git.kernel.org/bpf/bpf-next/c/3134396f1cba

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html