libbpf: fix offsetof() and container_of() to work with CO-RE
Andrii Nakryiko
maintainerPATCH 1/1 libbpf: fix offsetof() and container_of() to work with CO-RE
It seems like __builtin_offset() doesn't preserve CO-RE field relocations properly. So if offsetof() macro is defined through __builtin_offset(), CO-RE-enabled BPF code using container_of() will be subtly and silently broken.
To avoid this problem, redefine offsetof() and container_of() in the form that works with CO-RE relocations more reliably.
--- tools/lib/bpf/bpf_helpers.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
tools/lib/bpf/bpf_helpers.h +10 -5
@@ -77,16 +77,21 @@/* * Helper macros to manipulate data structures */ -#ifndef offsetof -#define offsetof (TYPE ,MEMBER ) ((unsigned long)&((TYPE *)0)->MEMBER) -#endif -#ifndef container_of + +/* offsetof() definition that uses __builtin_offset() might not preserve field +* offset CO-RE relocation properly, so force-redefine offsetof() using +* old-school approach which works with CO-RE correctly +*/ +#undef offsetof +#define offsetof (type, member) ((unsigned long)&((type *)0)->member) + +/* redefined container_of() to ensure we use the above offsetof() macro */ +#undef container_of#define container_of (ptr, type, member) \ ({ \ void *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member))); \ }) -#endif /* * Compiler (optimization) barrier.
Signature · 1 line
2.34.1
Quoted 4 lines · Andrii Nakryiko
It seems like
__builtin_offset()doesn't preserve CO-RE field relocations properly. So ifoffsetof()macro is defined through__builtin_offset(), CO-RE-enabled BPF code usingcontainer_of()will be subtly and silently broken.
This is true. See 63fe3fd393dc("libbpf: Do not use __builtin_offsetof for offsetof"). At some point, we used __builtin_offset() and found CO-RE relocation won't work, so the above commit switched back to
use ((unsigned long)&((TYPE *)0)->MEMBER).
Quoted 3 lines
To avoid this problem, redefine
offsetof()andcontainer_of()in the form that works with CO-RE relocations more reliably.
I am okay with the change to forcefully define offsetof() and container_of() since this is critical for correct CO-RE relocations.
Quoted 4 lines
Fixes 5fbc220862fc ("tools/libpf: Add offsetof/container_of macro in bpf_helpers.h")Reported-by Lennart Poettering <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>
Ack with a nit below.
Quoted 12 lines
--- tools/lib/bpf/bpf_helpers.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)tools/lib/bpf/bpf_helpers.h +0
-0@@ -77,16 +77,21 @@/* * Helper macros to manipulate data structures */
tools/lib/bpf/bpf_helpers.h line 85
Quoted 10 lines
tools/lib/bpf/bpf_helpers.h +6
-4-#ifndef offsetof -#define offsetof (TYPE ,MEMBER ) ((unsigned long)&((TYPE *)0)->MEMBER) -#endif -#ifndef container_of + +/* offsetof() definition that uses __builtin_offset() might not preserve field +* offset CO-RE relocation properly, so force-redefine offsetof() using +* old-school approach which works with CO-RE correctly +*/ +#undef offsetof
I am not sure whether the above 'undef' is good or bad. In my opinion, I would just remove the above 'undef'. If user defines offset as __builtin_offset, the compiler will issue a warning and user should remove that macro or undef them.
Otherwise, user may get impression that their __builtin_offset is working but actually it is not. the same for container_of.
Quoted 13 lines
tools/lib/bpf/bpf_helpers.h +4
-1+#define offsetof (type, member) ((unsigned long)&((type *)0)->member) + +/* redefined container_of() to ensure we use the above offsetof() macro */ +#undef container_of#define container_of (ptr, type, member) \ ({ \ void *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member))); \ }) -#endif /* * Compiler (optimization) barrier.
Quoted 50 lines · Yonghong Song
Quoted 4 lines · Andrii Nakryiko
It seems like
__builtin_offset()doesn't preserve CO-RE field relocations properly. So ifoffsetof()macro is defined through__builtin_offset(), CO-RE-enabled BPF code usingcontainer_of()will be subtly and silently broken.This is true. See 63fe3fd393dc("libbpf: Do not use
__builtin_offsetoffor offsetof"). At some point, we used__builtin_offset()and found CO-RE relocation won't work, so the above commit switched back to use ((unsigned long)&((TYPE *)0)->MEMBER).Quoted 3 lines
To avoid this problem, redefine
offsetof()andcontainer_of()in the form that works with CO-RE relocations more reliably.I am okay with the change to forcefully define
offsetof()andcontainer_of()since this is critical for correct CO-RE relocations.Quoted 4 lines
Fixes 5fbc220862fc ("tools/libpf: Add offsetof/container_of macro in bpf_helpers.h")Reported-by Lennart Poettering <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Ack with a nit below.
Acked-by Yonghong Song <[email protected]>Quoted 19 lines
--- tools/lib/bpf/bpf_helpers.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)tools/lib/bpf/bpf_helpers.h +3
-4@@ -77,16 +77,21 @@/* * Helper macros to manipulate data structures */ -#ifndef offsetof -#define offsetof (TYPE ,MEMBER ) ((unsigned long)&((TYPE *)0)->MEMBER) -#endif -#ifndef container_of + +/* offsetof() definition that uses __builtin_offset() might not preserve field +* offset CO-RE relocation properly, so force-redefine offsetof() using
Quoted 8 lines
tools/lib/bpf/bpf_helpers.h line 85
Quoted 3 lines
tools/lib/bpf/bpf_helpers.h +3
-0+* old-school approach which works with CO-RE correctly +*/ +#undef offsetofI am not sure whether the above 'undef' is good or bad. In my opinion, I would just remove the above 'undef'. If user defines offset as
__builtin_offset, the compiler will issue a warning and user should remove that macro or undef them.
If we don't #undef, we are almost guaranteed to get a compilation warning. See [0], where I repro this problem based on Lennart's original code.
[0] https://github.com/anakryiko/libbpf-bootstrap/commit/2bad3e7f48e4e4eea1a083620f21eba59aa75b1a
Quoted 3 lines
Otherwise, user may get impression that their
__builtin_offsetis working but actually it is not. the same forcontainer_of.
Can we just fix __builtin_offset() to generate/preserve field offset CO-RE relocation?
Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/bpf_helpers.h +4
-1+#define offsetof (type, member) ((unsigned long)&((type *)0)->member) + +/* redefined container_of() to ensure we use the above offsetof() macro */ +#undef container_of#define container_of (ptr, type, member) \ ({ \ void *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member))); \ }) -#endif /* * Compiler (optimization) barrier.
Quoted 60 lines · Andrii Nakryiko
Quoted 50 lines · Yonghong Song
Quoted 4 lines · Andrii Nakryiko
It seems like
__builtin_offset()doesn't preserve CO-RE field relocations properly. So ifoffsetof()macro is defined through__builtin_offset(), CO-RE-enabled BPF code usingcontainer_of()will be subtly and silently broken.This is true. See 63fe3fd393dc("libbpf: Do not use
__builtin_offsetoffor offsetof"). At some point, we used__builtin_offset()and found CO-RE relocation won't work, so the above commit switched back to use ((unsigned long)&((TYPE *)0)->MEMBER).Quoted 3 lines
To avoid this problem, redefine
offsetof()andcontainer_of()in the form that works with CO-RE relocations more reliably.I am okay with the change to forcefully define
offsetof()andcontainer_of()since this is critical for correct CO-RE relocations.Quoted 4 lines
Fixes 5fbc220862fc ("tools/libpf: Add offsetof/container_of macro in bpf_helpers.h")Reported-by Lennart Poettering <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>Ack with a nit below.
Acked-by Yonghong Song <[email protected]>Quoted 19 lines
--- tools/lib/bpf/bpf_helpers.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)tools/lib/bpf/bpf_helpers.h +3
-4@@ -77,16 +77,21 @@/* * Helper macros to manipulate data structures */ -#ifndef offsetof -#define offsetof (TYPE ,MEMBER ) ((unsigned long)&((TYPE *)0)->MEMBER) -#endif -#ifndef container_of + +/* offsetof() definition that uses __builtin_offset() might not preserve field +* offset CO-RE relocation properly, so force-redefine offsetof() using Quoted 8 lines
tools/lib/bpf/bpf_helpers.h line 85
Quoted 3 lines
tools/lib/bpf/bpf_helpers.h +3
-0+* old-school approach which works with CO-RE correctly +*/ +#undef offsetofI am not sure whether the above 'undef' is good or bad. In my opinion, I would just remove the above 'undef'. If user defines offset as
__builtin_offset, the compiler will issue a warning and user should remove that macro or undef them.
Quoted 6 lines
If we don't #undef, we are almost guaranteed to get a compilation warning. See [0], where I repro this problem based on Lennart's original code.
[0] https://github.com/anakryiko/libbpf-bootstrap/commit/2bad3e7f48e4e4eea1a083620f21eba59aa75b1a
Okay, I see. They didn't use vmlinux.h...
Quoted 6 lines
Quoted 3 lines
Otherwise, user may get impression that their
__builtin_offsetis working but actually it is not. the same forcontainer_of.Can we just fix
__builtin_offset()to generate/preserve field offset CO-RE relocation?
I checked with llvm17 source code. The __builtin_offset() is handled in two places:
- clang/lib/Sema/SemaExpr.cpp: func BuildBuiltinOffsetOf
- clang/lib/CodeGen/CGExprScalar.cpp: func VisitOffsetOfExpr
As expected, the above two functions process __builtin_offset() without any arch specific handling.
I guess we could try to add CO-RE relocation to them but not sure whether upstream likes it or not.
But the same issue exists in earlier clang versions. So I am okay with the current patch with "#undef offsetof" and "#undef container_of" although clang headers does not have macro for "container_of".
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
tools/lib/bpf/bpf_helpers.h +4
-1+#define offsetof (type, member) ((unsigned long)&((type *)0)->member) + +/* redefined container_of() to ensure we use the above offsetof() macro */ +#undef container_of#define container_of (ptr, type, member) \ ({ \ void *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member))); \ }) -#endif /* * Compiler (optimization) barrier.
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Mon, 8 May 2023 23:55:02 -0700 you wrote:
It seems like
__builtin_offset()doesn't preserve CO-RE field relocations properly. So ifoffsetof()macro is defined through__builtin_offset(), CO-RE-enabled BPF code usingcontainer_of()will be subtly and silently broken.To avoid this problem, redefine
offsetof()andcontainer_of()in the form that works with CO-RE relocations more reliably.[...]
Here is the summary with links:
- [bpf-next] libbpf: fix offsetof() and container_of() to work with CO-RE
https://git.kernel.org/bpf/bpf-next/c/bdeeed3498c7
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html