Skip to contentSkip to conversation

tracing: Add a way to filter function addresses to function names

PATCH In review Steven Rostedt 3 participants 9 messages Dec 19, 2022 – Dec 20, 2022

Subsystems (3)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Add a way to filter function addresses to function names

There's been several times where an event records a function address in its field and I needed to filter on that address for a specific function name. It required looking up the function in kallsyms, finding its size, and doing a compare of "field >= function_start && field < function_end".

But this would change from boot to boot and is unreliable in scripts. Also, it is useful to have this at boot up, where the addresses will not be known. For example, on the boot command line:

trace_trigger="initcall_finish.traceoff if func.function == acpi_init"

To implement this, add a ".function" prefix, that will check that the field is of size long, and the only operations allowed (so far) are "==" and "!=".

Changes since v1: https://lore.kernel.org/linux-trace-kernel/[email protected]/

- Fix commit log "initcall_finish.function" to "func.function" (Ross Zwisler)

- Fixed processing of address (Ross Zwisler)

- Added selftest (Masami Hiramatsu)

- Just use "trigger" instead of "strsep(&trigger, "")" (Zheng Yejian)

Steven Rostedt (Google) (2):
tracing: Add a way to filter function addresses to function names tracing/selftests: Add test for event filtering on function name

----

 Documentation/trace/events.rst                     | 12 +++
 kernel/trace/trace_events.c                        |  2 +-
 kernel/trace/trace_events_filter.c                 | 93 +++++++++++++++++++++-
 .../ftrace/test.d/filter/event-filter-function.tc  | 58 ++++++++++++++
 4 files changed, 163 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc
author maintainer
#1 lore raw

PATCH 1/2 tracing: Add a way to filter function addresses to function names

From: "Steven Rostedt (Google)" <[email protected]>

There's been several times where an event records a function address in its field and I needed to filter on that address for a specific function name. It required looking up the function in kallsyms, finding its size, and doing a compare of "field >= function_start && field < function_end".

But this would change from boot to boot and is unreliable in scripts. Also, it is useful to have this at boot up, where the addresses will not be known. For example, on the boot command line:

trace_trigger="initcall_finish.traceoff if func.function == acpi_init"

To implement this, add a ".function" prefix, that will check that the field is of size long, and the only operations allowed (so far) are "==" and "!=".

---
 Documentation/trace/events.rst     | 12 ++++
 kernel/trace/trace_events.c        |  2 +-
 kernel/trace/trace_events_filter.c | 93 +++++++++++++++++++++++++++++-
 3 files changed, 105 insertions(+), 2 deletions(-)
Documentation/trace/events.rst +12 -0
index c47f381d0c00..d0fd5c7220b7 100644
@@ -207,6 +207,18 @@ field name::
 As the kernel will have to know how to retrieve the memory that the pointer
 is at from user space.
 
+You can convert any long type to a function address and search by function name::
+
+  call_site.function == security_prepare_creds
+
+The above will filter when the field "call_site" falls on the address within
+"security_prepare_creds". That is, it will compare the value of "call_site" and
+the filter will return true if it is greater than or equal to the start of
+the function "security_prepare_creds" and less than the end of that function.
+
+The ".function" postfix can only be attached to values of size long, and can only
+be compared with "==" or "!=".
+
 5.2 Setting filters
 -------------------
 
kernel/trace/trace_events.c +1 -1
index 33e0b4f8ebe6..b07931ad97de 100644
@@ -2822,7 +2822,7 @@ static __init int setup_trace_triggers(char *str)
 		if (!trigger)
 			break;
 		bootup_triggers[i].event = strsep(&trigger, ".");
-		bootup_triggers[i].trigger = strsep(&trigger, ".");
+		bootup_triggers[i].trigger = trigger;
 		if (!bootup_triggers[i].trigger)
 			break;
 	}
kernel/trace/trace_events_filter.c +92 -1

143 lines not on this page · raw message

Signature · 1 line
2.35.1
Reviewed-by on v1 1/2
#2 lore raw
Quoted 11 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

There's been several times where an event records a function address in its field and I needed to filter on that address for a specific function name. It required looking up the function in kallsyms, finding its size, and doing a compare of "field >= function_start && field < function_end".

But this would change from boot to boot and is unreliable in scripts. Also, it is useful to have this at boot up, where the addresses will not be known. For example, on the boot command line:

Quoted 7 lines

trace_trigger="initcall_finish.traceoff if func.function == acpi_init"

To implement this, add a ".function" prefix, that will check that the field is of size long, and the only operations allowed (so far) are "==" and "!=".

author maintainer
#3 lore raw

PATCH 2/2 tracing/selftests: Add test for event filtering on function name

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

---
 .../test.d/filter/event-filter-function.tc    | 58 +++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc
tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc new file +58 -0
new file mode 100644
index 000000000000..e2ff3bf4df80
@@ -0,0 +1,58 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event filter function - test event filtering on functions
+# requires: set_event events/kmem/kmem_cache_free/filter
+# flags: instance
+
+fail() { #msg
+    echo $1
+    exit_fail
+}
+
+echo "Test event filter function name"
+echo 0 > tracing_on
+echo 0 > events/enable
+echo > trace
+echo 'call_site.function == exit_mmap' > events/kmem/kmem_cache_free/filter
+echo 1 > events/kmem/kmem_cache_free/enable
+echo 1 > tracing_on
+ls > /dev/null
+echo 0 > events/kmem/kmem_cache_free/enable
+
+hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l`
+misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l`
+
+if [ $hitcnt -eq 0 ]; then
+	exit_fail
+fi
+
+if [ $misscnt -gt 0 ]; then
+	exit_fail
+fi
+
+address=`grep ' exit_mmap$' /proc/kallsyms | cut -d' ' -f1`
+
+echo "Test event filter function address"
+echo 0 > tracing_on
+echo 0 > events/enable
+echo > trace
+echo "call_site.function == 0x$address" > events/kmem/kmem_cache_free/filter
+echo 1 > events/kmem/kmem_cache_free/enable
+echo 1 > tracing_on
+sleep 1
+echo 0 > events/kmem/kmem_cache_free/enable
+
+hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l`
+misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l`
+
+if [ $hitcnt -eq 0 ]; then
+	exit_fail
+fi
+
+if [ $misscnt -gt 0 ]; then
+	exit_fail
+fi
+
+reset_events_filter
+
+exit 0
Signature · 1 line
2.35.1
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

Quoted 11 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

Quoted 5 lines
maintainer
#5 lore raw

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

Quoted 12 lines · Ross Zwisler
Quoted 11 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

Quoted 7 lines
Quoted 5 lines

Thank you both. I will apply this after rc1 comes out.

thanks,
-- Shuah

author maintainer
#6 lore raw

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

Quoted 19 lines · On Mon, 19 Dec 2022 15:11:39 -0700 Shuah Khan <[email protected]> wrote:
Quoted 18 lines · Ross Zwisler
Quoted 16 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

Quoted 3 lines
Quoted 1 line

Thank you both. I will apply this after rc1 comes out.

It's dependent on the first patch.

-- Steve

maintainer Acked-by
#7 lore raw

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

Quoted 22 lines · Steven Rostedt
Quoted 19 lines · On Mon, 19 Dec 2022 15:11:39 -0700 Shuah Khan <[email protected]> wrote:
Quoted 18 lines · Ross Zwisler
Quoted 16 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Thank you both. I will apply this after rc1 comes out.

It's dependent on the first patch.

In which case,

thanks,
-- Shuah

author maintainer
#8 lore raw

Re: [PATCH 2/2] tracing/selftests: Add test for event filtering on function name

Quoted 25 lines · On Mon, 19 Dec 2022 17:20:39 -0700 Shuah Khan <[email protected]> wrote:
Quoted 24 lines · Steven Rostedt
Quoted 21 lines · On Mon, 19 Dec 2022 15:11:39 -0700 Shuah Khan <[email protected]> wrote:
Quoted 19 lines · Ross Zwisler
Quoted 11 lines · Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.

This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.

Quoted 5 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Thank you both. I will apply this after rc1 comes out.

It's dependent on the first patch.

In which case,

Thanks Shuah!

-- Steve