Fix bpf_perf_event_data ABI breakage
Subsystems (5)
cover Fix bpf_perf_event_data ABI breakage
libbpf CI noticed that my recent changes broke bpf_perf_event_data ABI on s390 [1]. Testing shows that they introduced a similar breakage on arm64. The problem is that we are not allowed to extend user_pt_regs, since it's used by bpf_perf_event_data.
This series fixes these problems by removing the new members and introducing user_pt_regs_v2 instead.
[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduce user_pt_regs_v2
arm64/bpf: Introduce struct user_pt_regs_v2
arch/arm64/include/asm/ptrace.h | 1 + arch/arm64/include/uapi/asm/ptrace.h | 7 +++++++ arch/s390/include/asm/ptrace.h | 1 + arch/s390/include/uapi/asm/ptrace.h | 10 ++++++++-- tools/lib/bpf/bpf_tracing.h | 10 ++++++---- 5 files changed, 23 insertions(+), 6 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/2 s390/bpf: Introduce user_pt_regs_v2
Extending user_pt_regs breaks struct bpf_perf_event_data ABI, so instead of exposing orig_gpr2 through it, create its copy with orig_gpr2 at the end and use it in libbpf.
The existing members are copy-pasted, so now there are 3 copies in total. It might be tempting to add a user_pt_regs member to user_pt_regs_v2 instead, however, there is no guarantee that then user_pt_regs_v2.orig_gpr2 would be at the same offset as pt_regs.orig_gpr2.
--- arch/s390/include/asm/ptrace.h | 1 + arch/s390/include/uapi/asm/ptrace.h | 10 ++++++++-- tools/lib/bpf/bpf_tracing.h | 4 ++-- 3 files changed, 11 insertions(+), 4 deletions(-)
arch/s390/include/asm/ptrace.h +1 -0
@@ -79,6 +79,7 @@ enum { struct pt_regs { union { user_pt_regs user_regs; + user_pt_regs_v2 user_regs_v2; struct { unsigned long args[1]; psw_t psw;
arch/s390/include/uapi/asm/ptrace.h +8 -2
@@ -288,16 +288,22 @@ typedef struct { } s390_regs;/* -* The user_pt_regs structure exports the beginning of +* The user_pt_regs and user_pt_regs_v2 structures export the beginning of * the in-kernel pt_regs structure to user space. */ typedef struct {unsigned long args[1 ];psw_t psw;unsigned long gprs[NUM_GPRS ]; -unsigned long orig_gpr2; }user_pt_regs ; +typedef struct { +unsigned long args[1 ]; +psw_t psw; +unsigned long gprs[NUM_GPRS ]; +unsigned long orig_gpr2; +}user_pt_regs_v2 ; +/* * Now for the user space program event recording (trace) definitions. * The following structures are used only for the ptrace interface, don't
tools/lib/bpf/bpf_tracing.h +2 -2
@@ -114,8 +114,8 @@#elif defined(bpf_target_s390) -/* s390 provides user_pt_regs instead of struct pt_regs to userspace */ -#define __PT_REGS_CAST (x) ((const user_pt_regs *)(x)) +/* s390 provides user_pt_regs_v2 instead of struct pt_regs to userspace */ +#define __PT_REGS_CAST (x) ((const user_pt_regs_v2 *)(x))#define __PT_PARM1_REG gprs[2]#define __PT_PARM1_REG_SYSCALL orig_gpr2#define __PT_PARM2_REG gprs[3]
Signature · 1 line
2.34.1
PATCH 2/2 arm64/bpf: Introduce struct user_pt_regs_v2
In reply to Ilya LeoshkevichExtending struct user_pt_regs breaks struct bpf_perf_event_data ABI, so instead of exposing orig_x0 through it, create its copy with orig_x0 at the end and use it in libbpf.
The existing members are copy-pasted, so now there are 3 copies in total. It might be tempting to add a user_pt_regs member to user_pt_regs_v2 instead, however, there is no guarantee that then user_pt_regs_v2.orig_x0 would be at the same offset as pt_regs.orig_gpr2.
--- arch/arm64/include/asm/ptrace.h | 1 + arch/arm64/include/uapi/asm/ptrace.h | 7 +++++++ tools/lib/bpf/bpf_tracing.h | 6 ++++-- 3 files changed, 12 insertions(+), 2 deletions(-)
arch/arm64/include/asm/ptrace.h +1 -0
@@ -178,6 +178,7 @@ static inline unsigned long pstate_to_compat_psr(const unsigned long pstate) struct pt_regs { union { struct user_pt_regs user_regs; + struct user_pt_regs_v2 user_regs_v2; struct { u64 regs[31]; u64 sp;
arch/arm64/include/uapi/asm/ptrace.h +7 -0
@@ -90,6 +90,13 @@ struct user_pt_regs {__u64 sp;__u64 pc;__u64 pstate; +}; + +struct user_pt_regs_v2 { +__u64 regs[31 ]; +__u64 sp; +__u64 pc; +__u64 pstate;__u64 orig_x0; };
tools/lib/bpf/bpf_tracing.h +4 -2
@@ -143,8 +143,10 @@#elif defined(bpf_target_arm64) -/* arm64 provides struct user_pt_regs instead of struct pt_regs to userspace */ -#define __PT_REGS_CAST (x) ((const struct user_pt_regs *)(x)) +/* +* arm64 provides struct user_pt_regs_v2 instead of struct pt_regs to userspace +*/ +#define __PT_REGS_CAST (x) ((const struct user_pt_regs_v2 *)(x))#define __PT_PARM1_REG regs[0]#define __PT_PARM1_REG_SYSCALL orig_x0#define __PT_PARM2_REG regs[1]
Signature · 1 line
2.34.1
Quoted 6 lines · Ilya Leoshkevich
libbpf CI noticed that my recent changes broke
bpf_perf_event_dataABI on s390 [1]. Testing shows that they introduced a similar breakage on arm64. The problem is that we are not allowed to extenduser_pt_regs, since it's used bybpf_perf_event_data.
Quoted 8 lines
This series fixes these problems by removing the new members and introducing
user_pt_regs_v2instead.[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduceuser_pt_regs_v2
arm64/bpf: Introduce structuser_pt_regs_v2
Given it is bpf_perf_event_data and thus bpf_user_pt_regs_t definitions that are set in stone now, wouldn't it be better to instead just change
typedef user_pt_regs bpf_user_pt_regs_t; (s390x) typedef struct user_pt_regs bpf_user_pt_regs_t; (arm64)
to just define that fixed layout instead of reusing user_ptr_regs?
This whole v2 business looks really ugly.
Quoted 11 lines
arch/arm64/include/asm/ptrace.h | 1 + arch/arm64/include/uapi/asm/ptrace.h | 7 +++++++ arch/s390/include/asm/ptrace.h | 1 + arch/s390/include/uapi/asm/ptrace.h | 10 ++++++++-- tools/lib/bpf/bpf_tracing.h | 10 ++++++---- 5 files changed, 23 insertions(+), 6 deletions(-)--
2.34.1
Quoted 24 lines · Andrii Nakryiko
Quoted 9 lines · Ilya Leoshkevich
libbpf CI noticed that my recent changes broke
bpf_perf_event_dataABI
on s390 [1]. Testing shows that they introduced a similar breakage on
arm64. The problem is that we are not allowed to extenduser_pt_regs,
since it's used bybpf_perf_event_data.Quoted 8 lines
This series fixes these problems by removing the new members and introducing
user_pt_regs_v2instead.[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduceuser_pt_regs_v2
arm64/bpf: Introduce structuser_pt_regs_v2Given it is
bpf_perf_event_dataand thusbpf_user_pt_regs_tdefinitions that are set in stone now, wouldn't it be better to instead just change
Quoted 6 lines
typedef
user_pt_regsbpf_user_pt_regs_t; (s390x) typedef structuser_pt_regsbpf_user_pt_regs_t; (arm64)to just define that fixed layout instead of reusing
user_ptr_regs?This whole v2 business looks really ugly.
Wouldn't it break compilation of code like this?
bpf_perf_event_data data;
user_pt_regs *regs = &data.regs;Additionaly, after this I'm no longer sure I haven't missed any other places where user_pt_regs might be used. For example, arm64 seems to be using it not only for BPF, but also for ptrace?
static int gpr_get(struct task_struct *target,
const struct user_regset *regset, struct membuf to)
{
struct user_pt_regs *uregs = &task_pt_regs(target)->user_regs;
return membuf_write(&to, uregs, sizeof(*uregs));
}and then in e.g. gdb:
static voidaarch64_fill_gregset (struct regcache *regcache, void *buf) {
struct user_pt_regs *regset = (struct user_pt_regs *) buf;
...
I'm also not a big fan of the _v2 solution, but it looked the safest to me. At least for s390, a viable alternative that Vasily proposed would be to go ahead with replacing args[1] with orig_gpr2 and then also backporting the patch, so that the new libbpf would still work on the old stable kernels. But this won't work for arm64.
Quoted 29 lines · Ilya Leoshkevich
Quoted 27 lines · Andrii Nakryiko
Quoted 9 lines · Ilya Leoshkevich
libbpf CI noticed that my recent changes broke
bpf_perf_event_dataABI
on s390 [1]. Testing shows that they introduced a similar breakage on
arm64. The problem is that we are not allowed to extenduser_pt_regs,
since it's used bybpf_perf_event_data.Quoted 8 lines
This series fixes these problems by removing the new members and introducing
user_pt_regs_v2instead.[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduceuser_pt_regs_v2
arm64/bpf: Introduce structuser_pt_regs_v2Given it is
bpf_perf_event_dataand thusbpf_user_pt_regs_tdefinitions that are set in stone now, wouldn't it be better to instead just changetypedef
user_pt_regsbpf_user_pt_regs_t; (s390x) typedef structuser_pt_regsbpf_user_pt_regs_t; (arm64)
Quoted 8 lines
Quoted 3 lines
to just define that fixed layout instead of reusing
user_ptr_regs?This whole v2 business looks really ugly.
Wouldn't it break compilation of code like this?
bpf_perf_event_data data; user_pt_regs *regs = &data.regs;
why would it break? user_pt_regs gained extra fields at the end, so whoever works with the assumption of an old definition of user_pt_regs *through pointer* should be totally fine. The problem with bpf_perf_event_data is that user_pt_regs are embedded in the struct directly, so adding anything to it changes bpf_perf_event_data layout.
I, of course, can't know if this breaks any other use case (including ones you mentioned below), but using user_pt_regs_v2 will probably not work with CO-RE, because older kernels won't have such type defined (and thus relocations will fail).
I'm not sure the origins of the need for user_pt_regs (as opposed to using pt_regs directly, like x86-64 does), but with CO-RE and vmlinux.h it would be more reliable and straightforward to just stick to kernel-internal struct pt_regs everywhere. And for non-CO-RE macros maybe just using an offset within struct pt_regs (i.e., offsetofend(gprs)) would do it?
Quoted 26 lines
Additionaly, after this I'm no longer sure I haven't missed any other places where
user_pt_regsmight be used. For example, arm64 seems to be using it not only for BPF, but also for ptrace?static int gpr_get(struct
task_struct*target,
const structuser_regset*regset, struct membuf to){ struct user_pt_regs *uregs = &task_pt_regs(target)->user_regs; return membuf_write(&to, uregs, sizeof(*uregs)); }and then in e.g. gdb:
static void
aarch64_fill_gregset(struct regcache *regcache, void *buf) {
structuser_pt_regs*regset = (structuser_pt_regs*) buf;
...I'm also not a big fan of the
_v2solution, but it looked the safest to me. At least for s390, a viable alternative that Vasily proposed would be to go ahead with replacing args[1] withorig_gpr2and then also backporting the patch, so that the new libbpf would still work on the old stable kernels. But this won't work for arm64.
Heiko Carstens
maintainerQuoted 6 lines · Andrii Nakryiko
I'm not sure the origins of the need for
user_pt_regs(as opposed to usingpt_regsdirectly, like x86-64 does), but with CO-RE andvmlinux.hit would be more reliable and straightforward to just stick to kernel-internal structpt_regseverywhere. And for non-CO-RE macros maybe just using an offset within structpt_regs(i.e., offsetofend(gprs)) would do it?
user_pt_regs was introduced on s390 because struct pt_regs is _not_ stable. Only the first n entries (aka user_pt_regs) are supposed to be stable.
We could of course reduce struct pt_regs to the bare minimum, which seems to be the current user_pt_regs plus orig_gpr2; which semantically would match more or less what x86 has.
Then move pt_regs to uapi, so it is clear that it cannot be changed anymore, and have additional data in a separate structure on the stack, which has pt_regs at the beginning, and access this additional data with container_of & friends.
I guess that could work, even though this requires to keep user_pt_regs "for historical reasons".
Quoted 17 lines · Heiko Carstens
Quoted 6 lines · Andrii Nakryiko
I'm not sure the origins of the need for
user_pt_regs(as opposed to usingpt_regsdirectly, like x86-64 does), but with CO-RE andvmlinux.hit would be more reliable and straightforward to just stick to kernel-internal structpt_regseverywhere. And for non-CO-RE macros maybe just using an offset within structpt_regs(i.e., offsetofend(gprs)) would do it?
user_pt_regswas introduced on s390 because structpt_regsis _not_ stable. Only the first n entries (akauser_pt_regs) are supposed to be stable.We could of course reduce struct
pt_regsto the bare minimum, which seems to be the currentuser_pt_regsplusorig_gpr2; which semantically would match more or less what x86 has.
Quoted 7 lines
Then move
pt_regsto uapi, so it is clear that it cannot be changed anymore, and have additional data in a separate structure on the stack, which haspt_regsat the beginning, and access this additional data withcontainer_of& friends.I guess that could work, even though this requires to keep
user_pt_regs"for historical reasons".
It's just surprising and constraining that UAPI struct can be extended by adding new fields at the end :( I'll let you guys decide it for s390 and arm64. From libbpf's side, we have somewhat hacky ways to work around that, it seems.
Quoted 45 lines · Andrii Nakryiko
Quoted 36 lines · Ilya Leoshkevich
Quoted 34 lines · Andrii Nakryiko
Quoted 15 lines · Ilya Leoshkevich
libbpf CI noticed that my recent changes broke
bpf_perf_event_data
ABI
on s390 [1]. Testing shows that they introduced a similar breakage
on
arm64. The problem is that we are not allowed to extenduser_pt_regs,
since it's used bybpf_perf_event_data.This series fixes these problems by removing the new members and
introducinguser_pt_regs_v2instead.Quoted 5 lines
[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduceuser_pt_regs_v2
arm64/bpf: Introduce structuser_pt_regs_v2Given it is
bpf_perf_event_dataand thusbpf_user_pt_regs_tdefinitions that are set in stone now, wouldn't it be better to instead just changetypedef
user_pt_regsbpf_user_pt_regs_t; (s390x) typedef structuser_pt_regsbpf_user_pt_regs_t; (arm64)to just define that fixed layout instead of reusing
user_ptr_regs?Quoted 6 lines
Quoted 1 line
This whole v2 business looks really ugly.
Wouldn't it break compilation of code like this?
bpf_perf_event_datadata;
user_pt_regs*regs = &data.regs;
Quoted 7 lines
why would it break?
user_pt_regsgained extra fields at the end, so whoever works with the assumption of an old definition ofuser_pt_regs
*through pointer* should be totally fine. The problem withbpf_perf_event_datais thatuser_pt_regsare embedded in the struct directly, so adding anything to it changesbpf_perf_event_datalayout.
I meant only building from source, at runtime it should be fine. At compile time, the compiler should at least warn that pointer types don't match.
Quoted 6 lines
I, of course, can't know if this breaks any other use case (including ones you mentioned below), but using
user_pt_regs_v2will probably not
work with CO-RE, because older kernels won't have such type defined (and thus relocations will fail).
Quoted 7 lines
I'm not sure the origins of the need for
user_pt_regs(as opposed to usingpt_regsdirectly, like x86-64 does), but with CO-RE andvmlinux.hit would be more reliable and straightforward to just stick to kernel-internal structpt_regseverywhere. And for non-CO-RE macros
maybe just using an offset within structpt_regs(i.e., offsetofend(gprs)) would do it?
offsetofend sounds like a nice compromise. I'll give it a try, thanks.
Quoted 31 lines
Quoted 31 lines
Additionaly, after this I'm no longer sure I haven't missed any other
places whereuser_pt_regsmight be used. For example, arm64 seems to be
using it not only for BPF, but also for ptrace?static int
gpr_get(struct task_struct *target, const struct user_regset *regset, struct membuf to)
{
structuser_pt_regs*uregs = &task_pt_regs(target)-Quoted 1 line
user_regs;return
membuf_write(&to, uregs, sizeof(*uregs)); }and then in e.g. gdb:
static void
aarch64_fill_gregset(struct regcache *regcache, void *buf) {
structuser_pt_regs*regset = (structuser_pt_regs*) buf;
...I'm also not a big fan of the
_v2solution, but it looked the safest
to me. At least for s390, a viable alternative that Vasily proposed would be to go ahead with replacing args[1] withorig_gpr2and then also backporting the patch, so that the new libbpf would still work on
the old stable kernels. But this won't work for arm64.
Quoted 55 lines · Ilya Leoshkevich
Quoted 45 lines · Andrii Nakryiko
Quoted 36 lines · Ilya Leoshkevich
Quoted 34 lines · Andrii Nakryiko
Quoted 15 lines · Ilya Leoshkevich
libbpf CI noticed that my recent changes broke
bpf_perf_event_data
ABI
on s390 [1]. Testing shows that they introduced a similar breakage
on
arm64. The problem is that we are not allowed to extenduser_pt_regs,
since it's used bybpf_perf_event_data.This series fixes these problems by removing the new members and
introducinguser_pt_regs_v2instead.Quoted 5 lines
[1] https://github.com/libbpf/libbpf/runs/5079938810
Ilya Leoshkevich (2):
s390/bpf: Introduceuser_pt_regs_v2
arm64/bpf: Introduce structuser_pt_regs_v2Given it is
bpf_perf_event_dataand thusbpf_user_pt_regs_tdefinitions that are set in stone now, wouldn't it be better to instead just changetypedef
user_pt_regsbpf_user_pt_regs_t; (s390x) typedef structuser_pt_regsbpf_user_pt_regs_t; (arm64)to just define that fixed layout instead of reusing
user_ptr_regs?Quoted 6 lines
Quoted 1 line
This whole v2 business looks really ugly.
Wouldn't it break compilation of code like this?
bpf_perf_event_data data; user_pt_regs *regs = &data.regs;Quoted 7 lines
why would it break?
user_pt_regsgained extra fields at the end, so whoever works with the assumption of an old definition ofuser_pt_regs
*through pointer* should be totally fine. The problem withbpf_perf_event_datais thatuser_pt_regsare embedded in the struct directly, so adding anything to it changesbpf_perf_event_datalayout.
Quoted 3 lines
I meant only building from source, at runtime it should be fine. At compile time, the compiler should at least warn that pointer types don't match.
Oh, you meant that cast would be necessary. Well, strictly speaking code like in your example is broken, it should use the type specified in struct bpf_perf_event_data: bpf_user_pt_regs_t. But the fix to satisfy compilation is trivial as well, so doesn't matter much.
Quoted 15 lines
Quoted 7 lines
I, of course, can't know if this breaks any other use case (including ones you mentioned below), but using
user_pt_regs_v2will probably not
work with CO-RE, because older kernels won't have such type defined (and thus relocations will fail).Quoted 7 lines
I'm not sure the origins of the need for
user_pt_regs(as opposed to usingpt_regsdirectly, like x86-64 does), but with CO-RE andvmlinux.hit would be more reliable and straightforward to just stick to kernel-internal structpt_regseverywhere. And for non-CO-RE macros
maybe just using an offset within structpt_regs(i.e., offsetofend(gprs)) would do it?
Quoted 1 line
offsetofend sounds like a nice compromise. I'll give it a try, thanks.
It's kind of dangerous as well, let's maybe leave a comment in pt_regs that this orig_gpr2 location is assumed by libbpf's tracing macros so shouldn't be willy-nilly moved
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Additionaly, after this I'm no longer sure I haven't missed any other
places whereuser_pt_regsmight be used. For example, arm64 seems to be
using it not only for BPF, but also for ptrace?static int gpr_get(struct
task_struct*target,
const structuser_regset*regset, struct membuf to)
{
structuser_pt_regs*uregs = &task_pt_regs(target)-Quoted 1 line
user_regs;return membuf_write(&to, uregs, sizeof(*uregs)); }and then in e.g. gdb:
static void
aarch64_fill_gregset(struct regcache *regcache, void *buf) {
structuser_pt_regs*regset = (structuser_pt_regs*) buf;
...I'm also not a big fan of the
_v2solution, but it looked the safest
to me. At least for s390, a viable alternative that Vasily proposed would be to go ahead with replacing args[1] withorig_gpr2and then also backporting the patch, so that the new libbpf would still work on
the old stable kernels. But this won't work for arm64.