Return EADDRNOTAVAIL when func matches several symbols during kprobe creation
Subsystems (3)
v5 cover Return EADDRNOTAVAIL when func matches several symbols during kprobe creation
Hi.
In the kernel source code, it exists different functions which share the same name but which have, of course, different addresses as they can be defined in different modules:
# Kernel was compiled with CONFIG_NTFS_FS and CONFIG_NTFS3_FS as built-in.
root@vm-amd64:~# grep ntfs_file_write_iter /proc/kallsyms
ffffffff814ce3c0 t __pfx_ntfs_file_write_iter
ffffffff814ce3d0 t ntfs_file_write_iter
ffffffff814fc8a0 t __pfx_ntfs_file_write_iter
ffffffff814fc8b0 t ntfs_file_write_iter
This can be source of troubles when you create a PMU kprobe for such a function,
as it will only install one for the first address (e.g. 0xffffffff814ce3d0 in
the above).
This could lead to some troubles were BPF based tools does not report any event
because the second function is not called:
root@vm-amd64:/mnt# mount | grep /mnt
/foo.img on /mnt type ntfs3 (rw,relatime,uid=0,gid=0,iocharset=utf8)
# ig is a tool which installs a PMU kprobe on ntfs_file_write_iter().
root@vm-amd64:/mnt# ig trace fsslower -m 0 -f ntfs3 --host &> /tmp/foo &
[1] 207
root@vm-amd64:/mnt# dd if=./foo of=./bar count=3
3+0 records in
3+0 records out
1536 bytes (1.5 kB, 1.5 KiB) copied, 0.00543323 s, 283 kB/s
root@vm-amd64:/mnt# fg
ig trace fsslower -m 0 -f ntfs3 --host &> /tmp/foo
^Croot@vm-amd64:/mnt# more /tmp/foo
RUNTIME.CONTAINERNAME RUNTIME.CONTAIN… PID COMM
T BYTES OFFSET LAT FILE
214 dd
R 512 0 766 foo
214 dd
R 512 512 9 foo
214 dd
As you can see in the above, only read events are reported and no write because
the kprobe is installed for the old ntfs_file_write_iter() and not the ntfs3
one.
The same behavior occurs with sysfs kprobe:
root@vm-amd64:/# echo 'p:probe/ntfs_file_write_iter ntfs_file_write_iter' > /sys/kernel/tracing/kprobe_events
root@vm-amd64:/# cat /sys/kernel/tracing/kprobe_events
p:probe/ntfs_file_write_iter ntfs_file_write_iter
root@vm-amd64:/# mount | grep /mnt
/foo.img on /mnt type ntfs3 (rw,relatime,uid=0,gid=0,iocharset=utf8)
root@vm-amd64:/# perf record -e probe:ntfs_file_write_iter &
[1] 210
root@vm-amd64:/# cd /mnt/
root@vm-amd64:/mnt# dd if=./foo of=./bar count=3
3+0 records in
3+0 records out
1536 bytes (1.5 kB, 1.5 KiB) copied, 0.00234793 s, 654 kB/s
root@vm-amd64:/mnt# cd -
/
root@vm-amd64:/# fg
perf record -e probe:ntfs_file_write_iter
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.056 MB perf.data ]root@vm-amd64:/# perf report Error: The perf.data data has no samples! # To display the perf.data header info, please use --header/--header-only optio> #
In this contribution, I modified the functions creating sysfs and PMU kprobes to test if the function name given as argument matches several symbols. In this case, these functions return EADDRNOTAVAIL to indicate the user to use addr and offs to remove this ambiguity.
So, when the above BPF tool is run, the following error message is printed:
root@vm-amd64:~# ig trace fsslower -m 0 -f ntfs3 --host &> /tmp/foo & [1] 228 root@vm-amd64:~# more /tmp/foo RUNTIME.CONTAINERNAME RUNTIME.CONTAIN… PID COMM T BYTES OFFSET LAT FILE Error: running gadget: running gadget: installing tracer: attaching kprobe: crea ting perf_kprobe PMU (arch-specific fallback for "ntfs_file_write_iter"): token ntfs_file_write_iter: opening perf event: cannot assign requested address And the same with sysfs kprobe: root@vm-amd64:/# echo 'p:probe/ntfs_file_write_iter ntfs_file_write_iter' > /sys/kernel/tracing/kprobe_events -bash: echo: write error: Cannot assign requested address Note that, this does not influence perf as it installs kprobes as offset on _text: root@vm-amd64:/# perf probe --add ntfs_file_write_iter Added new events: probe:ntfs_file_write_iter (on ntfs_file_write_iter) probe:ntfs_file_write_iter (on ntfs_file_write_iter) ... root@vm-amd64:/# cat /sys/kernel/tracing/kprobe_events p:probe/ntfs_file_write_iter _text+5039088 p:probe/ntfs_file_write_iter _text+5228752
Note that, this contribution is the conclusion of a previous RFC which intended to install a PMU kprobe for all matching symbols [1, 2].
If you see any way to improve this contribution, particularly if you have an idea to add tests or documentation for this behavior, please share your feedback.
Changes since:
v1:
* Use EADDRNOTAVAIL instead of adding a new error code.
* Correct also this behavior for sysfs kprobe.
v2:
* Count the number of symbols corresponding to function name and return EADDRNOTAVAIL if higher than 1.
* Return ENOENT if above count is 0, as it would be returned later by while registering the kprobe.
v3:
* Check symbol does not contain ':' before testing its uniqueness.
* Add a selftest to check this is not possible to install a kprobe for a non unique symbol.
v5:
* No changes, just add linux-stable as recipient.
Francis Laniel (2):
tracing/kprobes: Return EADDRNOTAVAIL when func matches several
symbols
selftests/ftrace: Add new test case which checks non unique symbol
kernel/trace/trace_kprobe.c | 63 +++++++++++++++++++ kernel/trace/trace_probe.h | 1 + .../test.d/kprobe/kprobe_non_uniq_symbol.tc | 13 ++++ 3 files changed, 77 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
Best regards and thank you in advance.
---
[1]: https://lore.kernel.org/lkml/[email protected]/
[2]: https://lore.kernel.org/lkml/[email protected]/
--
2.34.1
PATCH v5 1/2 tracing/kprobes: Return EADDRNOTAVAIL when func matches several symbols
Previously to this commit, if func matches several symbols, a kprobe, being either sysfs or PMU, would only be installed for the first matching address. This could lead to some misunderstanding when some BPF code was never called because it was attached to a function which was indeed not called, because the effectively called one has no kprobes attached.
So, this commit returns EADDRNOTAVAIL when func matches several symbols. This way, user needs to use address to remove the ambiguity.
--- kernel/trace/trace_kprobe.c | 63 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace_probe.h | 1 + 2 files changed, 64 insertions(+)
kernel/trace/trace_kprobe.c +63 -0
@@ -705,6 +705,25 @@ static struct notifier_block trace_kprobe_module_nb = { .priority = 1 /* Invoked after kprobe module callback */ }; +static int count_symbols (void *data,unsigned long unused) +{ +unsigned int *count = data; + + (*count)++; + +return 0 ; +} + +static unsigned int number_of_same_symbols (char *func_name) +{ +unsigned int count; + + count =0 ; +kallsyms_on_each_match_symbol (count_symbols, func_name, &count); + +return count; +} +static int __trace_kprobe_create (int argc,const char *argv[]) {/*
@@ -836,6 +855,31 @@ static int __trace_kprobe_create(int argc, const char *argv[]) } } +if (symbol && !strchr (symbol,':' )) { +unsigned int count; + + count =number_of_same_symbols (symbol); +if (count >1 ) { +/* +* Users should use ADDR to remove the ambiguity of +* using KSYM only. +*/ +trace_probe_log_err (0 ,NON_UNIQ_SYMBOL ); + ret = -EADDRNOTAVAIL ; + + goto error; + }else if (count ==0 ) { +/* +* We can return ENOENT earlier than when register the +* kprobe. +*/ +trace_probe_log_err (0 ,BAD_PROBE_ADDR ); + ret = -ENOENT ; + + goto error; + } + } +trace_probe_log_set_index (0 );if (event) { ret = traceprobe_parse_event_name(&event, &group, gbuf,
@@ -1695,6 +1739,7 @@ static int unregister_kprobe_event(struct trace_kprobe *tk) }#ifdef CONFIG_PERF_EVENTS +/* create a trace_kprobe, but don't add it to global lists */ struct trace_event_call * create_local_trace_kprobe(char *func, void *addr, unsigned long offs,
@@ -1705,6 +1750,24 @@ create_local_trace_kprobe(char *func, void *addr, unsigned long offs,int ret;char *event; +if (func) { +unsigned int count; + + count =number_of_same_symbols (func); +if (count >1 ) +/* +* Users should use addr to remove the ambiguity of +* using func only. +*/ +return ERR_PTR (-EADDRNOTAVAIL ); +else if (count ==0 ) +/* +* We can return ENOENT earlier than when register the +* kprobe. +*/ +return ERR_PTR (-ENOENT ); + } +/* * local trace_kprobes are not added to dyn_event, so they are never * searched in find_trace_kprobe(). Therefore, there is no concern of
kernel/trace/trace_probe.h +1 -0
@@ -450,6 +450,7 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,C (BAD_MAXACT ,"Invalid maxactive number" ), \C (MAXACT_TOO_BIG ,"Maxactive is too big" ), \C (BAD_PROBE_ADDR ,"Invalid probed address or symbol" ), \ +C (NON_UNIQ_SYMBOL ,"The symbol is not unique" ), \C (BAD_RETPROBE ,"Retprobe address must be an function entry" ), \C (NO_TRACEPOINT ,"Tracepoint is not found" ), \ C(BAD_ADDR_SUFFIX, "Invalid probed address suffix"), \
--
2.34.1
kernel test robot
build botRe: [PATCH v5 1/2] tracing/kprobes: Return EADDRNOTAVAIL when func matches several symbols
FYI: kernel test robot notices the stable kernel rule is not satisfied.
Hi,
Thanks for your patch.
FYI: kernel test robot notices the stable kernel rule is not satisfied.
The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html#option-1
Rule: add the tag "Cc: [email protected]" in the sign-off area to have the patch automatically included in the stable tree.
Subject: [PATCH v5 1/2] tracing/kprobes: Return EADDRNOTAVAIL when func matches several symbols
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v5 2/2 selftests/ftrace: Add new test case which checks non unique symbol
In reply to Francis LanielIf name_show() is non unique, this test will try to install a kprobe on this function which should fail returning EADDRNOTAVAIL.
On kernel where name_show() is not unique, this test is skipped.
--- .../ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc new file +13 -0
@@ -0,0 +1,13 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: Test failure of registering kprobe on non unique symbol +# requires: kprobe_events + +SYMBOL='name_show' + +# We skip this test on kernel where SYMBOL is unique or does not exist. +if [ "$(grep -c -E "[[:alnum:]]+ t ${SYMBOL}" /proc/kallsyms)" -le '1' ]; then + exit_unsupported +fi + +! echo "p:test_non_unique ${SYMBOL}" > kprobe_events
--
2.34.1
Greg KH
maintainerRe: [PATCH v5 2/2] selftests/ftrace: Add new test case which checks non unique symbol
Quoted 9 lines · Francis Laniel
If
name_show()is non unique, this test will try to install a kprobe on this function which should fail returning EADDRNOTAVAIL.
On kernel wherename_show()is not unique, this test is skipped.Signed-off-by Francis Laniel <[email protected]>--- .../ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
<formletter>
This is not the correct way to submit patches for inclusion in the stable kernel tree. Please read:
https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.
</formletter>
Steven Rostedt
maintainerQuoted 7 lines · On Wed, 18 Oct 2023 17:40:28 +0300 Francis Laniel…
Changes since:
v1:
* Use EADDRNOTAVAIL instead of adding a new error code.
* Correct also this behavior for sysfs kprobe.
v2:
* Count the number of symbols corresponding to function name and return EADDRNOTAVAIL if higher than 1.
Quoted 8 lines
* Return ENOENT if above count is 0, as it would be returned later by while registering the kprobe.
v3:
* Check symbol does not contain ':' before testing its uniqueness.
* Add a selftest to check this is not possible to install a kprobe for a non unique symbol.
v5:
* No changes, just add linux-stable as recipient.
So why is this adding stable? (and as Greg's form letter states, that's not how you do that)
I don't see this as a fix but a new feature.
-- Steve
Hi!
Quoted 20 lines · Le mercredi 18 octobre 2023, 20:00:42 EEST Steven Rostedt a écrit :
On Wed, 18 Oct 2023 17:40:28 +0300
Quoted 17 lines · Francis Laniel
Changes since:
v1:
* Use EADDRNOTAVAIL instead of adding a new error code.
* Correct also this behavior for sysfs kprobe.v2:
* Count the number of symbols corresponding to function name and return EADDRNOTAVAIL if higher than 1.
* Return ENOENT if above count is 0, as it would be returned later by while
registering the kprobe.v3:
* Check symbol does not contain ':' before testing its uniqueness.
* Add a selftest to check this is not possible to install a kprobe for a non unique symbol.
Quoted 5 lines
Quoted 2 lines
v5:
* No changes, just add linux-stable as recipient.So why is this adding stable? (and as Greg's form letter states, that's not how you do that)
Oops! Really sorry for this, I will correct everything for the next version!
Quoted 2 lines
I don't see this as a fix but a new feature.
You mean I should add a "Fix:" in the commit description?
Quoted 2 lines
-- Steve
Best regards.
Masami Hiramatsu (Google)
maintainerQuoted 19 lines · On Wed, 18 Oct 2023 13:00:42 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 7 lines · On Wed, 18 Oct 2023 17:40:28 +0300 Francis Laniel…
Changes since:
v1:
* Use EADDRNOTAVAIL instead of adding a new error code.
* Correct also this behavior for sysfs kprobe.
v2:
* Count the number of symbols corresponding to function name and return EADDRNOTAVAIL if higher than 1.Quoted 8 lines
* Return ENOENT if above count is 0, as it would be returned later by while registering the kprobe.
v3:
* Check symbol does not contain ':' before testing its uniqueness.
* Add a selftest to check this is not possible to install a kprobe for a non unique symbol.
v5:
* No changes, just add linux-stable as recipient.
Quoted 4 lines
So why is this adding stable? (and as Greg's form letter states, that's not how you do that)
I don't see this as a fix but a new feature.
I asked him to make this a fix since the current kprobe event' behavior is somewhat strange. It puts the probe on only the "first symbol" if user specifies a symbol name which has multiple instances. In this case, the actual probe address can not be solved by name. User must specify the probe address by unique name + offset. Unless, it can put a probe on unexpected address, especially if it specifies non-unique symbol + offset, the address may NOT be the instruction boundary.
To avoid this issue, it should check the given symbol is unique.
Thank you,
Quoted 2 lines
-- Steve
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Steven Rostedt
maintainerQuoted 5 lines · On Thu, 19 Oct 2023 21:18:43 +0900 Masami Hiramatsu (Google)…
Quoted 4 lines
So why is this adding stable? (and as Greg's form letter states, that's not how you do that)
I don't see this as a fix but a new feature.
Quoted 9 lines
I asked him to make this a fix since the current kprobe event' behavior is somewhat strange. It puts the probe on only the "first symbol" if user specifies a symbol name which has multiple instances. In this case, the actual probe address can not be solved by name. User must specify the probe address by unique name + offset. Unless, it can put a probe on unexpected address, especially if it specifies non-unique symbol + offset, the address may NOT be the instruction boundary.
To avoid this issue, it should check the given symbol is unique.
OK, so what is broken is that when you add a probe to a function that has multiple names, it will attach to the first one and not necessarily the one you want.
The change log needs to be more explicit in what the "bug" is. It does state this in a round about way, but it is written in a way that it doesn't stand out.
Previously to this commit, if func matches several symbols, a kprobe, being either sysfs or PMU, would only be installed for the first matching address. This could lead to some misunderstanding when some BPF code was never called because it was attached to a function which was indeed not called, because the effectively called one has no kprobes attached.
So, this commit returns EADDRNOTAVAIL when func matches several symbols. This way, user needs to use address to remove the ambiguity.
What it should say is:
When a kprobe is attached to a function that's name is not unique (is static and shares the name with other functions in the kernel), the kprobe is attached to the first function it finds. This is a bug as the function that it is attaching to is not necessarily the one that the user wants to attach to.
Instead of blindly picking a function to attach to what is ambiguous, error with EADDRNOTAVAIL to let the user know that this function is not unique, and that the user must use another unique function with an address offset to get to the function they want to attach to.
And yes, it should have:
which is how to mark something for stable, and
To the commit that caused the bug.
-- Steve
Masami Hiramatsu (Google)
maintainerQuoted 45 lines · On Thu, 19 Oct 2023 09:51:04 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 5 lines · On Thu, 19 Oct 2023 21:18:43 +0900 Masami Hiramatsu (Google)…
Quoted 4 lines
So why is this adding stable? (and as Greg's form letter states, that's not how you do that)
I don't see this as a fix but a new feature.
Quoted 9 lines
I asked him to make this a fix since the current kprobe event' behavior is somewhat strange. It puts the probe on only the "first symbol" if user specifies a symbol name which has multiple instances. In this case, the actual probe address can not be solved by name. User must specify the probe address by unique name + offset. Unless, it can put a probe on unexpected address, especially if it specifies non-unique symbol + offset, the address may NOT be the instruction boundary.
To avoid this issue, it should check the given symbol is unique.OK, so what is broken is that when you add a probe to a function that has multiple names, it will attach to the first one and not necessarily the one you want.
The change log needs to be more explicit in what the "bug" is. It does state this in a round about way, but it is written in a way that it doesn't stand out.
Previously to this commit, if func matches several symbols, a kprobe, being either sysfs or PMU, would only be installed for the first matching address. This could lead to some misunderstanding when some BPF code was never called because it was attached to a function which was indeed not called, because the effectively called one has no kprobes attached.
So, this commit returns EADDRNOTAVAIL when func matches several symbols. This way, user needs to use address to remove the ambiguity.
What it should say is:
When a kprobe is attached to a function that's name is not unique (is static and shares the name with other functions in the kernel), the kprobe is attached to the first function it finds. This is a bug as the function that it is attaching to is not necessarily the one that the user wants to attach to.
Quoted 5 lines
Instead of blindly picking a function to attach to what is ambiguous, error with EADDRNOTAVAIL to let the user know that this function is not unique, and that the user must use another unique function with an address offset to get to the function they want to attach to.
Great!, yes this looks good to me too.
Quoted 9 lines
And yes, it should have:
which is how to mark something for stable, and
Fixes ...To the commit that caused the bug.
Yes, this should be the first one.
Thank you,
Quoted 2 lines
-- Steve
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Hi!
Quoted 65 lines · Le jeudi 19 octobre 2023, 18:07:08 EEST Masami Hiramatsu a écrit :
On Thu, 19 Oct 2023 09:51:04 -0400
Quoted 48 lines · Steven Rostedt
On Thu, 19 Oct 2023 21:18:43 +0900
Quoted 6 lines · Masami Hiramatsu (Google)
Quoted 5 lines
So why is this adding stable? (and as Greg's form letter states, that's not
how you do that)I don't see this as a fix but a new feature.
Quoted 10 lines
I asked him to make this a fix since the current kprobe event' behavior is
somewhat strange. It puts the probe on only the "first symbol" if user specifies a symbol name which has multiple instances. In this case, the actual probe address can not be solved by name. User must specify the probe address by unique name + offset. Unless, it can put a probe on unexpected address, especially if it specifies non-unique symbol + offset,
the address may NOT be the instruction boundary.
To avoid this issue, it should check the given symbol is unique.OK, so what is broken is that when you add a probe to a function that has multiple names, it will attach to the first one and not necessarily the one
you want.The change log needs to be more explicit in what the "bug" is. It does state this in a round about way, but it is written in a way that it doesn't
stand out.Previously to this commit, if func matches several symbols, a kprobe, being either sysfs or PMU, would only be installed for the first matching address. This could lead to some misunderstanding when some BPF code was never called because it was attached to a function which was indeed not called, because the effectively called one has no kprobes attached.
So, this commit returns EADDRNOTAVAIL when func matches several symbols. This way, user needs to use address to remove the ambiguity.
What it should say is:
When a kprobe is attached to a function that's name is not unique (is static and shares the name with other functions in the kernel), the kprobe is attached to the first function it finds. This is a bug as the
function that it is attaching to is not necessarily the one that the user wants to attach to.Quoted 5 lines
Instead of blindly picking a function to attach to what is ambiguous, error with EADDRNOTAVAIL to let the user know that this function is not
unique, and that the user must use another unique function with an address offset to get to the function they want to attach to.Great!, yes this looks good to me too.
Quoted 6 lines
And yes, it should have:
which is how to mark something for stable, and
Quoted 7 lines
Quoted 3 lines
Fixes ...To the commit that caused the bug.
Yes, this should be the first one.
Fixes 413d37d1eb69 ("tracing: Add kprobe-based event tracer")
Thank you! I should have thought about it nonetheless but I will take more care in the future!
Quoted 3 lines
Thank you,
Quoted 1 line
-- Steve
Best regards.
Hi!
Quoted 45 lines · Le jeudi 19 octobre 2023, 16:51:04 EEST Steven Rostedt a écrit :
On Thu, 19 Oct 2023 21:18:43 +0900
Quoted 6 lines · Masami Hiramatsu (Google)
Quoted 5 lines
So why is this adding stable? (and as Greg's form letter states, that's not
how you do that)I don't see this as a fix but a new feature.
Quoted 8 lines
I asked him to make this a fix since the current kprobe event' behavior is somewhat strange. It puts the probe on only the "first symbol" if user specifies a symbol name which has multiple instances. In this case, the actual probe address can not be solved by name. User must specify the probe address by unique name + offset. Unless, it can put a probe on unexpected address, especially if it specifies non-unique symbol + offset, the address may NOT be the instruction boundary.
To avoid this issue, it should check the given symbol is unique.OK, so what is broken is that when you add a probe to a function that has multiple names, it will attach to the first one and not necessarily the one you want.
The change log needs to be more explicit in what the "bug" is. It does state this in a round about way, but it is written in a way that it doesn't stand out.
Previously to this commit, if func matches several symbols, a kprobe, being either sysfs or PMU, would only be installed for the first matching address. This could lead to some misunderstanding when some BPF code was never called because it was attached to a function which was indeed not called, because the effectively called one has no kprobes attached.
So, this commit returns EADDRNOTAVAIL when func matches several symbols. This way, user needs to use address to remove the ambiguity.
What it should say is:
When a kprobe is attached to a function that's name is not unique (is static and shares the name with other functions in the kernel), the kprobe is attached to the first function it finds. This is a bug as the function that it is attaching to is not necessarily the one that the user wants to attach to.
Quoted 4 lines
Instead of blindly picking a function to attach to what is ambiguous, error with EADDRNOTAVAIL to let the user know that this function is not unique, and that the user must use another unique function with an address offset to get to the function they want to attach to.
Thank you for the suggestion!
I updated the commit message and I am about to send v6!
Quoted 5 lines
And yes, it should have:
which is how to mark something for stable, and
I will for sure remember about it for future contributions! Thank you!
Quoted 5 lines
Fixes ...To the commit that caused the bug.
-- Steve
Best regards.