Skip to contentSkip to conversation

tracing: Fix bogus parsing of probes and set_ftrace_filter

RFC PATCH In review Steven Rostedt 3 participants 11 messages Feb 6, 2018 – Feb 8, 2018

Subsystems (4)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Fix bogus parsing of probes and set_ftrace_filter

Al Viro pointed out a couple of bugs with the parsing of the removing of function probes and the parsing of set_ftrace_filter strings.

This patch series implements his suggested fixes and adds selftests that would have caught the bugs in the first place.

Steven Rostedt (VMware) (6):
ftrace: Remove incorrect setting of glob search field
tracing: Fix parsing of globs with a wildcard at the beginning selftests/ftrace: Have reset_ftrace_filter handle modules selftests/ftrace: Have reset_ftrace_filter handle multiple instances selftests/ftrace: Add some missing glob checks selftests/ftrace: Add more tests for removing of function probes

----

 kernel/trace/ftrace.c                              |  1 -
 kernel/trace/trace_events_filter.c                 |  9 +++---
 .../ftrace/test.d/ftrace/func-filter-glob.tc       |  6 ++++
 .../ftrace/test.d/ftrace/func_set_ftrace_file.tc   | 37 ++++++++++++++++++++++
 tools/testing/selftests/ftrace/test.d/functions    | 10 ++++--
 5 files changed, 54 insertions(+), 9 deletions(-)
author maintainer
#1 lore raw

RFC PATCH 1/6 ftrace: Remove incorrect setting of glob search field

From: "Steven Rostedt (VMware)" <[email protected]>

__unregister_ftrace_function_probe() will incorrectly parse the glob filter because it resets the search variable that was setup by filter_parse_regex().

Al Viro reported this:

After that call of filter_parse_regex() we could have func_g.search not equal to glob only if glob started with '!' or '*'. In the former case we would've buggered off with -EINVAL (not = 1). In the latter we would've set func_g.search equal to glob + 1, calculated the length of that thing in func_g.len and proceeded to reset func_g.search back to glob.

Suppose the glob is e.g. *foo*. We end up with
func_g.type = MATCH_MIDDLE_ONLY;
func_g.len = 3;
func_g.search = "*foo";
Feeding that to ftrace_match_record() will not do anything sane - we will be looking for names containing "*foo" (->len is ignored for that one).

Cc Dmitry Safonov <[email protected]>
Fixes 3ba009297149f ("ftrace: Introduce ftrace_glob structure")
---
 kernel/trace/ftrace.c | 1 -
 1 file changed, 1 deletion(-)
kernel/trace/ftrace.c +0 -1
index dabd9d167d42..eac9ce2c57a2 100644
@@ -4456,7 +4456,6 @@ unregister_ftrace_function_probe_func(char *glob, struct trace_array *tr,
 		func_g.type = filter_parse_regex(glob, strlen(glob),
 						 &func_g.search, &not);
 		func_g.len = strlen(func_g.search);
-		func_g.search = glob;
 
 		/* we do not support '!' for function probes */
 		if (WARN_ON(not))
Signature · 1 line
2.15.1
Reviewed-by on v1 1/6
#2 lore raw

Re: [RFC][PATCH 1/6] ftrace: Remove incorrect setting of glob search field

2018-02-06 22:43 GMT+00:00 Steven Rostedt <[email protected]>:

Quoted 22 lines

From: "Steven Rostedt (VMware)" <[email protected]>

__unregister_ftrace_function_probe() will incorrectly parse the glob filter because it resets the search variable that was setup by filter_parse_regex().

Al Viro reported this:

After that call of filter_parse_regex() we could have func_g.search not equal to glob only if glob started with '!' or '*'. In the former case we would've buggered off with -EINVAL (not = 1). In the latter we would've set func_g.search equal to glob + 1, calculated the length of that thing in func_g.len and proceeded to reset func_g.search back to glob.

Suppose the glob is e.g. *foo*. We end up with
func_g.type = MATCH_MIDDLE_ONLY;
func_g.len = 3;
func_g.search = "*foo";
Feeding that to ftrace_match_record() will not do anything sane - we will be looking for names containing "*foo" (->len is ignored for that one).

Quoted 7 lines
Cc Dmitry Safonov <[email protected]>
Fixes 3ba009297149f ("ftrace: Introduce ftrace_glob structure")

Thank you, Steven.

Signature · 1 line
             Dmitry
maintainer Reviewed-by on v1 1/6
#3 lore raw

Re: [RFC][PATCH 1/6] ftrace: Remove incorrect setting of glob search field

In reply to Steven Rostedt
Quoted 22 lines · On Tue, 06 Feb 2018 17:43:39 -0500 Steven Rostedt <[email protected]> wrote:

From: "Steven Rostedt (VMware)" <[email protected]>

__unregister_ftrace_function_probe() will incorrectly parse the glob filter because it resets the search variable that was setup by filter_parse_regex().

Al Viro reported this:

After that call of filter_parse_regex() we could have func_g.search not equal to glob only if glob started with '!' or '*'. In the former case we would've buggered off with -EINVAL (not = 1). In the latter we would've set func_g.search equal to glob + 1, calculated the length of that thing in func_g.len and proceeded to reset func_g.search back to glob.

Suppose the glob is e.g. *foo*. We end up with
func_g.type = MATCH_MIDDLE_ONLY;
func_g.len = 3;
func_g.search = "*foo";
Feeding that to ftrace_match_record() will not do anything sane - we will be looking for names containing "*foo" (->len is ignored for that one).

Quoted 7 lines
Cc Dmitry Safonov <[email protected]>
Fixes 3ba009297149f ("ftrace: Introduce ftrace_glob structure")

Thank you!

Quoted 20 lines
---
 kernel/trace/ftrace.c | 1 -
 1 file changed, 1 deletion(-)
kernel/trace/ftrace.c +0 -1
index dabd9d167d42..eac9ce2c57a2 100644
@@ -4456,7 +4456,6 @@ unregister_ftrace_function_probe_func(char *glob, struct trace_array *tr,
 		func_g.type = filter_parse_regex(glob, strlen(glob),
 						 &func_g.search, &not);
 		func_g.len = strlen(func_g.search);
-		func_g.search = glob;
 
 		/* we do not support '!' for function probes */
 		if (WARN_ON(not))
Signature · 1 line
2.15.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#4 lore raw

RFC PATCH 2/6 tracing: Fix parsing of globs with a wildcard at the beginning

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Al Viro reported:

For substring - sure, but what about something like "*a*b" and "a*b"? AFAICS, filter_parse_regex() ends up with identical results in both cases - MATCH_GLOB and *search = "a*b". And no way for the caller to tell one from another.

Testing this with the following:

 # cd /sys/kernel/tracing
 # echo '*raw*lock' > set_ftrace_filter
 bash: echo: write error: Invalid argument

With this patch:

 # echo '*raw*lock' > set_ftrace_filter
 # cat set_ftrace_filter
_raw_read_trylock
_raw_write_trylock
_raw_read_unlock
_raw_spin_unlock
_raw_write_unlock
_raw_spin_trylock
_raw_spin_lock
_raw_write_lock
_raw_read_lock

Al recommended not setting the search buffer to skip the first '*' unless we know we are not using MATCH_GLOB. This implements his suggested logic.

Cc Masami Hiramatsu <[email protected]>
Fixes 60f1d5e3bac44 ("ftrace: Support full glob matching")
---
 kernel/trace/trace_events_filter.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
kernel/trace/trace_events_filter.c +4 -5
index 61e7f0678d33..a764aec3c9a1 100644
@@ -400,7 +400,6 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not)
 	for (i = 0; i < len; i++) {
 		if (buff[i] == '*') {
 			if (!i) {
-				*search = buff + 1;
 				type = MATCH_END_ONLY;
 			} else if (i == len - 1) {
 				if (type == MATCH_END_ONLY)
@@ -410,14 +409,14 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not)
 				buff[i] = 0;
 				break;
 			} else {	/* pattern continues, use full glob */
-				type = MATCH_GLOB;
-				break;
+				return MATCH_GLOB;
 			}
 		} else if (strchr("[?\\", buff[i])) {
-			type = MATCH_GLOB;
-			break;
+			return MATCH_GLOB;
 		}
 	}
+	if (buff[0] == '*')
+		*search = buff + 1;
 
 	return type;
 }
Signature · 1 line
2.15.1
maintainer Reviewed-by on v1 2/6
#5 lore raw

Re: [RFC][PATCH 2/6] tracing: Fix parsing of globs with a wildcard at the beginning

Quoted 32 lines · On Tue, 06 Feb 2018 17:43:40 -0500 Steven Rostedt <[email protected]> wrote:

From: "Steven Rostedt (VMware)" <[email protected]>

Al Viro reported:

For substring - sure, but what about something like "*a*b" and "a*b"? AFAICS, filter_parse_regex() ends up with identical results in both cases - MATCH_GLOB and *search = "a*b". And no way for the caller to tell one from another.

Testing this with the following:

 # cd /sys/kernel/tracing
 # echo '*raw*lock' > set_ftrace_filter
 bash: echo: write error: Invalid argument

With this patch:

 # echo '*raw*lock' > set_ftrace_filter
 # cat set_ftrace_filter
_raw_read_trylock
_raw_write_trylock
_raw_read_unlock
_raw_spin_unlock
_raw_write_unlock
_raw_spin_trylock
_raw_spin_lock
_raw_write_lock
_raw_read_lock

Al recommended not setting the search buffer to skip the first '*' unless we know we are not using MATCH_GLOB. This implements his suggested logic.

Quoted 8 lines
Cc Masami Hiramatsu <[email protected]>
Fixes 60f1d5e3bac44 ("ftrace: Support full glob matching")

This looks good to me.

Thank you!

Quoted 39 lines
---
 kernel/trace/trace_events_filter.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
kernel/trace/trace_events_filter.c +4 -5
index 61e7f0678d33..a764aec3c9a1 100644
@@ -400,7 +400,6 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not)
 	for (i = 0; i < len; i++) {
 		if (buff[i] == '*') {
 			if (!i) {
-				*search = buff + 1;
 				type = MATCH_END_ONLY;
 			} else if (i == len - 1) {
 				if (type == MATCH_END_ONLY)
@@ -410,14 +409,14 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not)
 				buff[i] = 0;
 				break;
 			} else {	/* pattern continues, use full glob */
-				type = MATCH_GLOB;
-				break;
+				return MATCH_GLOB;
 			}
 		} else if (strchr("[?\\", buff[i])) {
-			type = MATCH_GLOB;
-			break;
+			return MATCH_GLOB;
 		}
 	}
+	if (buff[0] == '*')
+		*search = buff + 1;
 
 	return type;
 }
Signature · 1 line
2.15.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#6 lore raw

RFC PATCH 3/6 selftests/ftrace: Have reset_ftrace_filter handle modules

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If a function probe in set_ftrace_filter belongs to a module, it will contain the module name. Like:

wmi_query_block [wmi]:traceoff:unlimited

But writing:

'!wmi_query_block [wmi]:traceoff' > set_ftrace_filter

will cause an error. We still need to write:

'!wmi_query_block:traceoff' > set_ftrace_filter

---
 tools/testing/selftests/ftrace/test.d/functions | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
tools/testing/selftests/ftrace/test.d/functions +4 -3
index f2019b37370d..e7c4c7b752a2 100644
@@ -37,17 +37,18 @@ reset_ftrace_filter() { # reset all triggers in set_ftrace_filter
 	if [ "$tr" = "" ]; then
 	    continue
 	fi
+	name=`echo $t | cut -d: -f1 | cut -d' ' -f1`
 	if [ $tr = "enable_event" -o $tr = "disable_event" ]; then
-	    tr=`echo $t | cut -d: -f1-4`
+	    tr=`echo $t | cut -d: -f2-4`
 	    limit=`echo $t | cut -d: -f5`
 	else
-	    tr=`echo $t | cut -d: -f1-2`
+	    tr=`echo $t | cut -d: -f2`
 	    limit=`echo $t | cut -d: -f3`
 	fi
 	if [ "$limit" != "unlimited" ]; then
 	    tr="$tr:$limit"
 	fi
-	echo "!$tr" > set_ftrace_filter
+	echo "!$name:$tr" > set_ftrace_filter
     done
 }
 
Signature · 1 line
2.15.1
author maintainer
#7 lore raw

RFC PATCH 4/6 selftests/ftrace: Have reset_ftrace_filter handle multiple instances

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If a probe is attached to a static function that is in multiple files with the same name, removing it by name will remove all instances:

 # grep jump_label_unlock set_ftrace_filter
jump_label_unlock:traceoff:unlimited
jump_label_unlock:traceoff:unlimited
 # echo '!jump_label_unlock:traceoff' >> set_ftrace_filter
 # grep jump_label_unlock set_ftrace_filter
 #

But the loop in reset_ftrace_filter will try to remove multiple instances multiple times. If this happens the second time will error and cause the test to fail.

At each iteration of the loop, check to see if the probe being removed still exists.

---
 tools/testing/selftests/ftrace/test.d/functions | 3 +++
 1 file changed, 3 insertions(+)
tools/testing/selftests/ftrace/test.d/functions +3 -0
index e7c4c7b752a2..df3dd7fe5f9b 100644
@@ -37,6 +37,9 @@ reset_ftrace_filter() { # reset all triggers in set_ftrace_filter
 	if [ "$tr" = "" ]; then
 	    continue
 	fi
+	if ! grep -q "$t" set_ftrace_filter; then
+		continue;
+	fi
 	name=`echo $t | cut -d: -f1 | cut -d' ' -f1`
 	if [ $tr = "enable_event" -o $tr = "disable_event" ]; then
 	    tr=`echo $t | cut -d: -f2-4`
Signature · 1 line
2.15.1
author maintainer
#8 lore raw

RFC PATCH 5/6 selftests/ftrace: Add some missing glob checks

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Al Viro discovered a bug in the glob ftrace filtering code where "*a*b" is treated the same as "a*b", and functions that would be selected by "*a*b" but not "a*b" are not selected with "*a*b".

Add tests for patterns "*a*b" and "a*b*" to the glob selftest.

---
 tools/testing/selftests/ftrace/test.d/ftrace/func-filter-glob.tc | 6 ++++++
 1 file changed, 6 insertions(+)
tools/testing/selftests/ftrace/test.d/ftrace/func-filter-glob.tc +6 -0
index 589d52b211b7..27a54a17da65 100644
@@ -29,6 +29,12 @@ ftrace_filter_check '*schedule*' '^.*schedule.*$'
 # filter by *, end match
 ftrace_filter_check 'schedule*' '^schedule.*$'
 
+# filter by *mid*end
+ftrace_filter_check '*aw*lock' '.*aw.*lock$'
+
+# filter by start*mid*
+ftrace_filter_check 'mutex*try*' '^mutex.*try.*'
+
 # Advanced full-glob matching feature is recently supported.
 # Skip the tests if we are sure the kernel does not support it.
 if grep -q 'accepts: .* glob-matching-pattern' README ; then
Signature · 1 line
2.15.1
maintainer Acked-by on v1 5/6
#9 lore raw

Re: [RFC][PATCH 5/6] selftests/ftrace: Add some missing glob checks

Quoted 7 lines · On Tue, 06 Feb 2018 17:43:43 -0500 Steven Rostedt <[email protected]> wrote:

From: "Steven Rostedt (VMware)" <[email protected]>

Al Viro discovered a bug in the glob ftrace filtering code where "*a*b" is treated the same as "a*b", and functions that would be selected by "*a*b" but not "a*b" are not selected with "*a*b".

Add tests for patterns "*a*b" and "a*b*" to the glob selftest.

Yeah, this is what we should have...

Thank you!

Quoted 30 lines
---
 tools/testing/selftests/ftrace/test.d/ftrace/func-filter-glob.tc | 6 ++++++
 1 file changed, 6 insertions(+)
tools/testing/selftests/ftrace/test.d/ftrace/func-filter-glob.tc +6 -0
index 589d52b211b7..27a54a17da65 100644
@@ -29,6 +29,12 @@ ftrace_filter_check '*schedule*' '^.*schedule.*$'
 # filter by *, end match
 ftrace_filter_check 'schedule*' '^schedule.*$'
 
+# filter by *mid*end
+ftrace_filter_check '*aw*lock' '.*aw.*lock$'
+
+# filter by start*mid*
+ftrace_filter_check 'mutex*try*' '^mutex.*try.*'
+
 # Advanced full-glob matching feature is recently supported.
 # Skip the tests if we are sure the kernel does not support it.
 if grep -q 'accepts: .* glob-matching-pattern' README ; then
Signature · 1 line
2.15.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#10 lore raw

RFC PATCH 6/6 selftests/ftrace: Add more tests for removing of function probes

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Al Viro discovered a bug in the removing of function probes where if it had a '*' at the beginning, it would fail to find any matches. That is, because it reset the glob search string to the the initial string with a "MATCH_END" type, instead of skipping the wildcard "*" it included it, where it would not match any functions because "*" was being treated as a normal character and not a wildcard one.

---
 .../ftrace/test.d/ftrace/func_set_ftrace_file.tc   | 37 ++++++++++++++++++++++
 1 file changed, 37 insertions(+)
tools/testing/selftests/ftrace/test.d/ftrace/func_set_ftrace_file.tc +37 -0
index 0f3f92622e33..68e7a48f5828 100644
@@ -128,6 +128,43 @@ if check_set_ftrace_filter "$FUNC1" "$FUNC2" ; then
     fail "Expected $FUNC1 and $FUNC2"
 fi
 
+test_actual() { # Compares $TMPDIR/expected with set_ftrace_filter
+    cat set_ftrace_filter | grep -v '#' | cut -d' ' -f1 | cut -d':' -f1 | sort -u > $TMPDIR/actual
+    DIFF=`diff $TMPDIR/actual $TMPDIR/expected`
+    test -z "$DIFF"
+}
+
+# Set traceoff trigger for all fuctions with "lock" in their name
+cat available_filter_functions | cut -d' ' -f1 |  grep 'lock' | sort -u > $TMPDIR/expected
+echo '*lock*:traceoff' > set_ftrace_filter
+test_actual
+
+# now remove all with 'try' in it, and end with lock
+grep -v 'try.*lock$' $TMPDIR/expected > $TMPDIR/expected2
+mv $TMPDIR/expected2 $TMPDIR/expected
+echo '!*try*lock:traceoff' >> set_ftrace_filter
+test_actual
+
+# remove all that start with "m" and end with "lock"
+grep -v '^m.*lock$' $TMPDIR/expected > $TMPDIR/expected2
+mv $TMPDIR/expected2 $TMPDIR/expected
+echo '!m*lock:traceoff' >> set_ftrace_filter
+test_actual
+
+# remove all that start with "c" and have "unlock"
+grep -v '^c.*unlock' $TMPDIR/expected > $TMPDIR/expected2
+mv $TMPDIR/expected2 $TMPDIR/expected
+echo '!c*unlock*:traceoff' >> set_ftrace_filter
+test_actual
+
+# clear all the rest
+> $TMPDIR/expected
+echo '!*:traceoff' >> set_ftrace_filter
+test_actual
+
+rm $TMPDIR/expected
+rm $TMPDIR/actual
+
 do_reset
 
 exit 0
Signature · 1 line
2.15.1