bpf kernel code leaks internal error codes to userspace
Hi!
It seems that the bpf code in the kernel sometimes leaks kernel-internal error codes, i.e. those from include/linux/errno.h into userspace (as opposed to those from include/uapi/asm-generic/errno.h which are public userspace facing API).
According to the comments from that internal header file: "These should never be seen by user programs."
Specifically, this is about ENOTSUPP, which userspace simply cannot handle, there's no error 524 defined in glibc or anywhere else.
We ran into this in systemd recently:
https://github.com/systemd/systemd/issues/32170#issuecomment-2076928761
(a google search reveals others were hit by this too)
We commited a work-around for this for now:
https://github.com/systemd/systemd/pull/33067
But it really sucks to work around this in userspace, this is a kernel internal definition after all, conflicting with userspace (where ENOTSUPP is just an alias for EOPNOTSUPP), hence not really fixable.
ENOSUPP is kinda useless anyway, since EOPNOTSUPP is pretty much equally expressive, and something userspace can actually handle.
Various kernel subsystems have been fixed over the years in similar situations. For example:
or
or
https://patchwork.ozlabs.org/project/linux-mtd/patch/[email protected]/
I think BPF should really fix that, too.
Or if you insist on leaking internals to userspace then please move the definition to the public "uapi" headers, but yikes you are in for some pain then given the conflicting userspace definitions.
Lennart
Quoted 38 lines · Lennart Poettering
Hi!
It seems that the bpf code in the kernel sometimes leaks kernel-internal error codes, i.e. those from
include/linux/errno.hinto userspace (as opposed to those frominclude/uapi/asm-generic/errno.hwhich are public userspace facing API).According to the comments from that internal header file: "These should never be seen by user programs."
Specifically, this is about ENOTSUPP, which userspace simply cannot handle, there's no error 524 defined in glibc or anywhere else.
We ran into this in systemd recently:
https://github.com/systemd/systemd/issues/32170#issuecomment-2076928761
(a google search reveals others were hit by this too)
We commited a work-around for this for now:
https://github.com/systemd/systemd/pull/33067
But it really sucks to work around this in userspace, this is a kernel internal definition after all, conflicting with userspace (where ENOTSUPP is just an alias for EOPNOTSUPP), hence not really fixable.
ENOSUPP is kinda useless anyway, since EOPNOTSUPP is pretty much equally expressive, and something userspace can actually handle.
Various kernel subsystems have been fixed over the years in similar situations. For example:
or
Quoted 7 lines
or
https://patchwork.ozlabs.org/project/linux-mtd/patch/[email protected]/
I think BPF should really fix that, too.
hm, I don't think we can change that, user space already depends on those values and we'd break it with new value
it's unfortunate, but I don't think we can do much about that, other than enforcing EOPNOTSUPP for new code
jirka
Quoted 7 lines
Or if you insist on leaking internals to userspace then please move the definition to the public "uapi" headers, but yikes you are in for some pain then given the conflicting userspace definitions.
Lennart
Quoted 38 lines · Jiri Olsa
Quoted 38 lines
It seems that the bpf code in the kernel sometimes leaks kernel-internal error codes, i.e. those from
include/linux/errno.hinto userspace (as opposed to those frominclude/uapi/asm-generic/errno.hwhich are public userspace facing API).According to the comments from that internal header file: "These should never be seen by user programs."
Specifically, this is about ENOTSUPP, which userspace simply cannot handle, there's no error 524 defined in glibc or anywhere else.
We ran into this in systemd recently:
https://github.com/systemd/systemd/issues/32170#issuecomment-2076928761
(a google search reveals others were hit by this too)
We commited a work-around for this for now:
https://github.com/systemd/systemd/pull/33067
But it really sucks to work around this in userspace, this is a kernel internal definition after all, conflicting with userspace (where ENOTSUPP is just an alias for EOPNOTSUPP), hence not really fixable.
ENOSUPP is kinda useless anyway, since EOPNOTSUPP is pretty much equally expressive, and something userspace can actually handle.
Various kernel subsystems have been fixed over the years in similar situations. For example:
or
Quoted 8 lines
Quoted 5 lines
or
https://patchwork.ozlabs.org/project/linux-mtd/patch/[email protected]/
I think BPF should really fix that, too.
hm, I don't think we can change that, user space already depends on those values and we'd break it with new value
Are you sure about that? To be able to handle this situation that userspace program whose existance you are indicating would have had to go the extra mile to literally handle error code 524 that is not known to userspace otherwise and handle it. If somebody goes the extra mile to do that, what makes you think that they didn't just handle it as equivalent to regular EOPNOSTUPP? In systemd at least that's what we are doing.
Also: if various other subsystems (I linked examples from wireless, media, mtd above) just fixed this, why not bpf, why is it special in this regard?
AFAICS the man pages of most syscalls just list errnos that *can* be returned, but usually doesn't list the precise conditions or makes guarantees about it. This specific error is not listed at all on any man page for the bpf() syscall, hence are you really sure this is actually as set in stone as you think it is?
I mean, bpf() is still a bit of bleeding edge tech, and people playing around with this probably tend to have quite new toolchains even, not old, hard to fix code?
Quoted 2 lines
it's unfortunate, but I don't think we can do much about that, other than enforcing EOPNOTSUPP for new code
I took the liberty to CC Linus on this:
Linus, what's the policy if some subsystem by mistake is leaking internal kernel error codes (such as ENOTSUP) to userspace? Leave it be as is (i.e. error number not defined in include/uapi/, not documented, but still returned), or fix it to the closest matching public error code (which is probably EOPNOTSUPP in this case) accepting a – mild, I would say – compat break?
[BTW, wouldn't it make sense to add a BUG_ON or so on syscalls that return an error number > 133 or so? This kind of issue is quite a recurring theme, see the patches above, and such a BUG_ON would probably catch 95% of all cases like this.]
Lennart
Quoted 3 lines · Lennart Poettering
Linus, what's the policy if some subsystem by mistake is leaking internal kernel error codes (such as ENOTSUP) to userspace?
Try to fix it, and stop leaking silly garbage in the hope that nobody cared.
And in most cases, people don't actually care, because they don't know to even test for non-standard errno values.
In fact it is _very_ rare that some actual program - outside of some test suite - cares about the errno values outside of the really special ones
So things like "EAGAIN" and "EINTR" have real semantic meaning and people test for them and do something explicitly different. Sometimes a few others too, but those are the two obvious ones.
The rest are *almost* always entirely interchangeable and only result in different error messages, not different _behavior_.
But then occasionally if it turns out that user space then breaks because some random case actually tested for a _particular_ error, and we have to put the silly garbage back.
Congrats, you have now invented a new ABI that you will support until the end of time (or until user space stops caring, whichever comes first).
That said, while I think ENOTSUPP should not be encouraged because it's so non-standard, I think the ship on ENOTSUPP has long since sailed.
Yes, yes, the comment may say "These should never be seen by user programs." but that comment is historically about ERESTARTSYS and friends, that are supposed to be turned into EINTR (and a restart etc depending on process flags and signal details).
And nobody reads comments anyway, and perhaps more importantly, they by definition have no semantic meaning, so...
We have a ton of code that returns ENOTSUPP - not in the least limited to bpf - and while we have a checkpatch rule for it, it's more of a "don't add new ones". Even that one is likely not really relevant.
And yes, some of them may get translated, but the very first one I randomly looked at was a proc_handler thing for some driver, and read -> ...->read_iter() -> proc_sys_read() -> proc_sys_call_handler() -> ...->proc_handler() most definitely does not.
ENOTSUPP in particular is very understandable, because the "standard" error names are garbage that nobody would ever use. EOPNOTSUPP? Never heard of it.
The standard errno for that is EINVAL, and people actively avoid it because it's _so_ common, so everybody goes "I want to make it clear that this is somethign else" and then they pick some other random name that sounds likely. And ENOTSUPP is right there and sounds very likely indeed.
If we really cared, we should have made them have a very different naming syntax, I'm afraid. Something explicitly inconvenient to make people not want to use it. Like __INTERNAL_ONLY_ENOTSUPP.
Linus
Quoted 49 lines · Lennart Poettering
Quoted 38 lines · Jiri Olsa
Quoted 38 lines
It seems that the bpf code in the kernel sometimes leaks kernel-internal error codes, i.e. those from
include/linux/errno.hinto userspace (as opposed to those frominclude/uapi/asm-generic/errno.hwhich are public userspace facing API).According to the comments from that internal header file: "These should never be seen by user programs."
Specifically, this is about ENOTSUPP, which userspace simply cannot handle, there's no error 524 defined in glibc or anywhere else.
We ran into this in systemd recently:
https://github.com/systemd/systemd/issues/32170#issuecomment-2076928761
(a google search reveals others were hit by this too)
We commited a work-around for this for now:
https://github.com/systemd/systemd/pull/33067
But it really sucks to work around this in userspace, this is a kernel internal definition after all, conflicting with userspace (where ENOTSUPP is just an alias for EOPNOTSUPP), hence not really fixable.
ENOSUPP is kinda useless anyway, since EOPNOTSUPP is pretty much equally expressive, and something userspace can actually handle.
Various kernel subsystems have been fixed over the years in similar situations. For example:
or
Quoted 8 lines
Quoted 5 lines
or
https://patchwork.ozlabs.org/project/linux-mtd/patch/[email protected]/
I think BPF should really fix that, too.
hm, I don't think we can change that, user space already depends on those values and we'd break it with new value
Quoted 7 lines
Are you sure about that? To be able to handle this situation that userspace program whose existance you are indicating would have had to go the extra mile to literally handle error code 524 that is not known to userspace otherwise and handle it. If somebody goes the extra mile to do that, what makes you think that they didn't just handle it as equivalent to regular EOPNOSTUPP? In systemd at least that's what we are doing.
cilium/ebpf [1] library is checking return values just for ENOTSUPP(524) on multiple places, libbpf has one place to check on that value for program type detection AFAICS
jirka
[1] https://github.com/cilium/ebpf/
Quoted 33 lines
Also: if various other subsystems (I linked examples from wireless, media, mtd above) just fixed this, why not bpf, why is it special in this regard?
AFAICS the man pages of most syscalls just list errnos that *can* be returned, but usually doesn't list the precise conditions or makes guarantees about it. This specific error is not listed at all on any man page for the
bpf()syscall, hence are you really sure this is actually as set in stone as you think it is?I mean,
bpf()is still a bit of bleeding edge tech, and people playing around with this probably tend to have quite new toolchains even, not old, hard to fix code?Quoted 2 lines
it's unfortunate, but I don't think we can do much about that, other than enforcing EOPNOTSUPP for new code
I took the liberty to CC Linus on this:
Linus, what's the policy if some subsystem by mistake is leaking internal kernel error codes (such as ENOTSUP) to userspace? Leave it be as is (i.e. error number not defined in
include/uapi/, not documented, but still returned), or fix it to the closest matching public error code (which is probably EOPNOTSUPP in this case) accepting a – mild, I would say – compat break?[BTW, wouldn't it make sense to add a
BUG_ONor so on syscalls that return an error number > 133 or so? This kind of issue is quite a recurring theme, see the patches above, and such aBUG_ONwould probably catch 95% of all cases like this.]Lennart
Quoted 59 lines · Jiri Olsa
Quoted 49 lines · Lennart Poettering
Quoted 38 lines · Jiri Olsa
Quoted 38 lines
It seems that the bpf code in the kernel sometimes leaks kernel-internal error codes, i.e. those from
include/linux/errno.hinto userspace (as opposed to those frominclude/uapi/asm-generic/errno.hwhich are public userspace facing API).According to the comments from that internal header file: "These should never be seen by user programs."
Specifically, this is about ENOTSUPP, which userspace simply cannot handle, there's no error 524 defined in glibc or anywhere else.
We ran into this in systemd recently:
https://github.com/systemd/systemd/issues/32170#issuecomment-2076928761
(a google search reveals others were hit by this too)
We commited a work-around for this for now:
https://github.com/systemd/systemd/pull/33067
But it really sucks to work around this in userspace, this is a kernel internal definition after all, conflicting with userspace (where ENOTSUPP is just an alias for EOPNOTSUPP), hence not really fixable.
ENOSUPP is kinda useless anyway, since EOPNOTSUPP is pretty much equally expressive, and something userspace can actually handle.
Various kernel subsystems have been fixed over the years in similar situations. For example:
or
Quoted 8 lines
Quoted 5 lines
or
https://patchwork.ozlabs.org/project/linux-mtd/patch/[email protected]/
I think BPF should really fix that, too.
hm, I don't think we can change that, user space already depends on those values and we'd break it with new value
Quoted 7 lines
Are you sure about that? To be able to handle this situation that userspace program whose existance you are indicating would have had to go the extra mile to literally handle error code 524 that is not known to userspace otherwise and handle it. If somebody goes the extra mile to do that, what makes you think that they didn't just handle it as equivalent to regular EOPNOSTUPP? In systemd at least that's what we are doing.
Quoted 8 lines
cilium/ebpf [1] library is checking return values just for ENOTSUPP(524) on multiple places, libbpf has one place to check on that value for program type detection AFAICS
jirka
fwiw both libraries can be changed to check for both error codes ENOTSUPP and EOPNOTSUPP,
but, as noted earlier, the problem is not limited to bpf. cilium library mentions
commit cb9a19fe4aa5 ("uprobes: Introduce prepare_uprobe()") back from 2012.
Where installing uprobe would return ENOTSUPP to user space.
On bpf side we've switched to EOPNOTSUPP for any new code long ago, but didn't adjust old code, because the damage was done before bpf existed.
$ git grep EOPNOTSUPP kernel/bpf|wc -l 56 $ git grep ENOTSUPP kernel/bpf|wc -l 61