libarena: Use compiler load-acquire/store-release in bpf_atomic.h
Subsystems (3)
PATCH v2 1/1 libarena: Use compiler load-acquire/store-release in bpf_atomic.h
Teach libarena's BPF atomic primitives to use compiler builtins for load-acquire and store-release when Clang advertises __BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the existing barrier-based fallback.
Notably, as BPF programs begin running on arm64, it is better to use the more appropriate variants since we can no longer rely on x86 TSO ordering.
Commit 880442305a39 ("bpf: Introduce load-acquire and store-release instructions") introduced support, hence kernels from 6.15 onwards are needed when compiling with compilers supporting these instructions. We have relatively recent kernel version requirements in libarena anyway, and have not cut first release, hence declare such a dependency.
---
Changelog:
v1: https://lore.kernel.org/bpf/[email protected]/
Changes in v2:
- Use __BPF_FEATURE_LOAD_ACQ_STORE_REL directly for guard.
- Use __unqual_typeof
--- .../bpf/libarena/include/bpf_atomic.h | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+)
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +20 -0
@@ -86,6 +86,25 @@ extern bool CONFIG_X86_64 __kconfig __weak;/* Control dependency provides LOAD->STORE, provide LOAD->LOAD */ #define smp_acquire__after_ctrl_dep () ({ smp_rmb(); }) +#if defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) +/* +* Clang advertises this feature when it can lower acquire/release atomic +* builtins to BPF_LOAD_ACQ/BPF_STORE_REL. Older compilers keep using the +* barrier-based fallback below. The generated instructions require kernel +* verifier/JIT support added in Linux 6.15; compile for an older BPF CPU to +* keep using the fallback when targeting older kernels. +*/ +#define smp_load_acquire (p) \ + ({ \ + __unqual_typeof(*(p)) ___p1 = __atomic_load_n((p), __ATOMIC_ACQUIRE); \ + (typeof(*(p)))___p1; \ + }) + +#define smp_store_release (p, val) \ + do { \ + __atomic_store_n((p), (val), __ATOMIC_RELEASE); \ + } while (0) +#else #define smp_load_acquire (p) \ ({ \ __unqual_typeof(*(p)) __v = READ_ONCE(*(p)); \
@@ -102,6 +121,7 @@ extern bool CONFIG_X86_64 __kconfig __weak;barrier (); \WRITE_ONCE (*(p), val); \ }) +#endif #define smp_cond_load_relaxed_label (p, cond_expr, label) \ ({ \
base-commit: a23a71823352e2d792dcaae25f1ebb744acbfc0b
Signature · 1 line
2.53.0-Meta
Kumar Kartikeya Dwivedi
maintainerQuoted 43 lines · Puranjay Mohan
Teach libarena's BPF atomic primitives to use compiler builtins for load-acquire and store-release when Clang advertises
__BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the existing barrier-based fallback.Notably, as BPF programs begin running on arm64, it is better to use the more appropriate variants since we can no longer rely on x86 TSO ordering.
Commit
880442305a39("bpf: Introduce load-acquire and store-release instructions") introduced support, hence kernels from 6.15 onwards are needed when compiling with compilers supporting these instructions. We have relatively recent kernel version requirements in libarena anyway, and have not cut first release, hence declare such a dependency.Signed-off-by Puranjay Mohan <[email protected]>---
Changelog:
v1: https://lore.kernel.org/bpf/[email protected]/
Changes in v2:
- Use__BPF_FEATURE_LOAD_ACQ_STORE_RELdirectly for guard.
- Use__unqual_typeof--- .../bpf/libarena/include/bpf_atomic.h | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+)tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +8
-0@@ -86,6 +86,25 @@ extern bool CONFIG_X86_64 __kconfig __weak;/* Control dependency provides LOAD->STORE, provide LOAD->LOAD */ #define smp_acquire__after_ctrl_dep () ({ smp_rmb(); }) +#if defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) +/* +* Clang advertises this feature when it can lower acquire/release atomic +* builtins to BPF_LOAD_ACQ/BPF_STORE_REL. Older compilers keep using the +* barrier-based fallback below. The generated instructions require kernel +* verifier/JIT support added in Linux 6.15; compile for an older BPF CPU to +* keep using the fallback when targeting older kernels. +*/
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h extern bool CONFIG_X86_64 __kconfig __weak; · line 106
Quoted 10 lines
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +10
-0+#define smp_load_acquire (p) \ + ({ \ + __unqual_typeof(*(p)) ___p1 = __atomic_load_n((p), __ATOMIC_ACQUIRE); \ + (typeof(*(p)))___p1; \ + }) + +#define smp_store_release (p, val) \ + do { \ + __atomic_store_n((p), (val), __ATOMIC_RELEASE); \ + } while (0)
Let's use consistent ({ }) for multi-line macros. Also, per your own suggestion (and the bot's), wouldn't you want __unqual_typeof() to stash the value for the store case as well?
pw-bot: cr
Quoted 14 lines
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +2
-0+#else #define smp_load_acquire (p) \ ({ \ __unqual_typeof(*(p)) __v = READ_ONCE(*(p)); \@@ -102,6 +121,7 @@ extern bool CONFIG_X86_64 __kconfig __weak;barrier (); \WRITE_ONCE (*(p), val); \ }) +#endif #define smp_cond_load_relaxed_label (p, cond_expr, label) \ ({ \base-commit:
a23a71823352e2d792dcaae25f1ebb744acbfc0b
Quoted 51 lines · Kumar Kartikeya Dwivedi
Quoted 49 lines · Puranjay Mohan
Teach libarena's BPF atomic primitives to use compiler builtins for load-acquire and store-release when Clang advertises
__BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the existing barrier-based fallback.Notably, as BPF programs begin running on arm64, it is better to use the more appropriate variants since we can no longer rely on x86 TSO ordering.
Commit
880442305a39("bpf: Introduce load-acquire and store-release instructions") introduced support, hence kernels from 6.15 onwards are needed when compiling with compilers supporting these instructions. We have relatively recent kernel version requirements in libarena anyway, and have not cut first release, hence declare such a dependency.Signed-off-by Puranjay Mohan <[email protected]>---
Changelog:
v1: https://lore.kernel.org/bpf/[email protected]/
Changes in v2:
- Use__BPF_FEATURE_LOAD_ACQ_STORE_RELdirectly for guard.
- Use__unqual_typeof--- .../bpf/libarena/include/bpf_atomic.h | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+)tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +14
-0@@ -86,6 +86,25 @@ extern bool CONFIG_X86_64 __kconfig __weak;/* Control dependency provides LOAD->STORE, provide LOAD->LOAD */ #define smp_acquire__after_ctrl_dep () ({ smp_rmb(); }) +#if defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) +/* +* Clang advertises this feature when it can lower acquire/release atomic +* builtins to BPF_LOAD_ACQ/BPF_STORE_REL. Older compilers keep using the +* barrier-based fallback below. The generated instructions require kernel +* verifier/JIT support added in Linux 6.15; compile for an older BPF CPU to +* keep using the fallback when targeting older kernels. +*/ +#define smp_load_acquire (p) \ + ({ \ + __unqual_typeof(*(p)) ___p1 = __atomic_load_n((p), __ATOMIC_ACQUIRE); \ + (typeof(*(p)))___p1; \ + }) +
Quoted 8 lines
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h extern bool CONFIG_X86_64 __kconfig __weak; · line 106
Quoted 4 lines
tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +4
-0+#define smp_store_release (p, val) \ + do { \ + __atomic_store_n((p), (val), __ATOMIC_RELEASE); \ + } while (0)Let's use consistent ({ }) for multi-line macros. Also, per your own suggestion (and the bot's), wouldn't you want
__unqual_typeof()to stash the value for the store case as well?
I will do the ({}) change but I don't think __unqual_typeof() is needed for store case. The load stashes into a temp because it produces a return value whose type callers observe. My earlier version that I sent off list was using __atomic_store() which takes a pointer to the value but this one doesn't.