Skip to contentSkip to conversation

tracing: bug fixes for tracing/filters

PATCH In review Li Zefan 5 participants 21 messages Apr 11, 2009 – Apr 13, 2009 Tracing

lore
Share & tools
lore

Li Zefan

author
#0 lore raw

cover tracing: bug fixes for tracing/filters

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

Maybe 3rd to 6th patches can be regarded as small enhancements instead of bug fixes? But they are behavioural changes.

[PATCH 1/7] tracing/filters: NUL-terminate user input filter [PATCH 2/7] tracing/filters: fix NULL pointer dereference
[PATCH 3/7] tracing/filters: allow user input integer to be oct or hex [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string [PATCH 5/7] tracing/filters: disallow newline as delimeter
[PATCH 6/7] tracing/filters: return proper error code when writing filter file [PATCH 7/7] tracing/filters: make filter preds RCU safe

 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |   16 +++--
 kernel/trace/trace_events_filter.c  |  131 ++++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 121 insertions(+), 42 deletions(-)

---

author
#1 lore raw

PATCH 1/7 tracing/filters: NUL-terminate user input filter

Make sure messages from user space are NUL-terminated strings, otherwise we could dump random memory while reading filter file.

Try this:

 # echo 'parent_comm ==' > events/sched/sched_process_fork/filter
 # cat events/sched/sched_process_fork/filter
 parent_comm == �
---
 kernel/trace/trace_events.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/trace/trace_events.c +2 -0
index 64ec4d2..054bc18 100644
@@ -503,6 +503,7 @@ event_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 
 	if (copy_from_user(&buf, ubuf, cnt))
 		return -EFAULT;
+	buf[cnt] = '\0';
 
 	pred = kzalloc(sizeof(*pred), GFP_KERNEL);
 	if (!pred)
@@ -569,6 +570,7 @@ subsystem_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 
 	if (copy_from_user(&buf, ubuf, cnt))
 		return -EFAULT;
+	buf[cnt] = '\0';
 
 	pred = kzalloc(sizeof(*pred), GFP_KERNEL);
 	if (!pred)
Signature · 1 line
1.5.4.rc3
author
#2 lore raw

PATCH 2/7 tracing/filters: fix NULL pointer dereference

In reply to Li Zefan

Try this, and you'll see NULL pointer dereference bug:

  # echo -n 'parent_comm ==' > sched/sched_process_fork/filter

Because we passed NULL ptr to simple_strtoull().

---
 kernel/trace/trace_events_filter.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
kernel/trace/trace_events_filter.c +5 -0
index 026be41..9d2162f 100644
@@ -410,6 +410,11 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 		}
 	}
 
+	if (!val_str) {
+		pred->field_name = NULL;
+		return -EINVAL;
+	}
+
 	pred->field_name = kstrdup(pred->field_name, GFP_KERNEL);
 	if (!pred->field_name)
 		return -ENOMEM;
Signature · 1 line
1.5.4.rc3
author
#4 lore raw

PATCH 3/7 tracing/filters: allow user input integer to be oct or hex

In reply to Li Zefan

Before patching:

 # echo 'parent_pid == 0x10' > events/sched/sched_process_fork/filter
 # cat sched/sched_process_fork/filter
 parent_pid == 0

After patching:

 # cat sched/sched_process_fork/filter
 parent_pid == 16

Also check the input more strictly.

---
 kernel/trace/trace_events_filter.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
kernel/trace/trace_events_filter.c +3 -2
index 9d2162f..49b3ef5 100644
@@ -419,12 +419,13 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 	if (!pred->field_name)
 		return -ENOMEM;
 
-	pred->val = simple_strtoull(val_str, &tmp, 10);
+	pred->val = simple_strtoull(val_str, &tmp, 0);
 	if (tmp == val_str) {
 		pred->str_val = kstrdup(val_str, GFP_KERNEL);
 		if (!pred->str_val)
 			return -ENOMEM;
-	}
+	} else if (*tmp != '\0')
+		return -EINVAL;
 
 	return 0;
 }
Signature · 1 line
1.5.4.rc3
author
#6 lore raw

PATCH 4/7 tracing/filters: allow user to specify a filter val to be string

In reply to Li Zefan

Suppose we would like to trace all tasks named '123', but this will fail:

 # echo 'parent_comm == 123' > events/sched/sched_process_fork/filter
 bash: echo: write error: Invalid argument

With this patch, we allow it by:

 # echo 'parent_comm == \123' > events/sched/sched_process_fork/filter
 # cat events/sched/sched_process_fork/filter
 parent_comm == 123
---
 kernel/trace/trace_events_filter.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)
kernel/trace/trace_events_filter.c +10 -1
index 49b3ef5..2bf4481 100644
@@ -351,12 +351,19 @@ oom:
 	return -ENOMEM;
 }
 
+/*
+ * The filter format can be
+ *   - 0, which means remove all filter preds
+ *   - [||/&&] <field> ==/!= [\]<val>
+ *
+ * Note: '\' prevent a string type value beginning with a digit to
+ *       be treated as a number
+ */
 int filter_parse(char **pbuf, struct filter_pred *pred)
 {
 	char *tmp, *tok, *val_str = NULL;
 	int tok_n = 0;
 
-	/* field ==/!= number, or/and field ==/!= number, number */
 	while ((tok = strsep(pbuf, " \n"))) {
 		if (tok_n == 0) {
 			if (!strcmp(tok, "0")) {
@@ -421,6 +428,8 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 
 	pred->val = simple_strtoull(val_str, &tmp, 0);
 	if (tmp == val_str) {
+		if (*val_str == '\\')
+			val_str++;
 		pred->str_val = kstrdup(val_str, GFP_KERNEL);
 		if (!pred->str_val)
 			return -ENOMEM;
Signature · 1 line
1.5.4.rc3
author
#11 lore raw

PATCH 5/7 tracing/filters: disallow newline as delimeter

In reply to Li Zefan

I guess because user input is often ended with '\n' (like "echo xxx"), thus '\n' is used as a delimeter besides ' ' ? But we can just strip tailing spaces.

One of the effects of this patch, is fixing this inconsistency:

 # echo -n 'parent_comm ==' > filter
 bash: echo: write error: Invalid argument
 # echo 'parent_comm ==' > filter
 # cat filter
 parent_comm ==
---
 kernel/trace/trace_events_filter.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
kernel/trace/trace_events_filter.c +3 -1
index 2bf4481..8e3a095 100644
@@ -364,7 +364,9 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 	char *tmp, *tok, *val_str = NULL;
 	int tok_n = 0;
 
-	while ((tok = strsep(pbuf, " \n"))) {
+	strstrip(*pbuf);
+
+	while ((tok = strsep(pbuf, " "))) {
 		if (tok_n == 0) {
 			if (!strcmp(tok, "0")) {
 				pred->clear = 1;
Signature · 1 line
1.5.4.rc3
author
#12 lore raw

PATCH 6/7 tracing/filters: return proper error code when writing filter file

In reply to Li Zefan

- propogate return value of filter_add_pred() to the user
- return -ENOSPC but not -ENOMEM or -EINVAL when the filter array is full

---
 kernel/trace/trace_events.c        |   10 ++++++----
 kernel/trace/trace_events_filter.c |    4 ++--
 2 files changed, 8 insertions(+), 6 deletions(-)
kernel/trace/trace_events.c +6 -4
index 054bc18..576f4fa 100644
@@ -521,9 +521,10 @@ event_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 		return cnt;
 	}
 
-	if (filter_add_pred(call, pred)) {
+	err = filter_add_pred(call, pred);
+	if (err < 0) {
 		filter_free_pred(pred);
-		return -EINVAL;
+		return err;
 	}
 
 	*ppos += cnt;
@@ -588,10 +589,11 @@ subsystem_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 		return cnt;
 	}
 
-	if (filter_add_subsystem_pred(system, pred)) {
+	err = filter_add_subsystem_pred(system, pred);
+	if (err < 0) {
 		filter_free_subsystem_preds(system);
 		filter_free_pred(pred);
-		return -EINVAL;
+		return err;
 	}
 
 	*ppos += cnt;
kernel/trace/trace_events_filter.c +2 -2
index 8e3a095..f16a921 100644
@@ -215,7 +215,7 @@ static int __filter_add_pred(struct ftrace_event_call *call,
 		}
 	}
 
-	return -ENOMEM;
+	return -ENOSPC;
 }
 
 static int is_string_field(const char *type)
@@ -319,7 +319,7 @@ int filter_add_subsystem_pred(struct event_subsystem *system,
 	}
 
 	if (i == MAX_FILTER_PRED)
-		return -EINVAL;
+		return -ENOSPC;
 
 	events_for_each(call) {
 		int err;
Signature · 1 line
1.5.4.rc3
author
#14 lore raw

PATCH 7/7 tracing/filters: make filter preds RCU safe

In reply to Li Zefan

I noticed ftrace_event_call->preds and ->preds[*] are not protected by any lock, and I can manage to trigger kernel oops.

This patch adds filter_mutex to protect concurrent access to event_subsystem->preds/preds[i] or ftrace_event_call->preds/preds[i], and make the fast-path filter_match_preds() RCU safe.

---
 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |    4 +-
 kernel/trace/trace_events_filter.c  |  104 +++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 91 insertions(+), 33 deletions(-)
kernel/trace/trace.h +4 -2
index e685ac2..ae05fbb 100644
@@ -847,13 +847,15 @@ struct filter_pred {
 int trace_define_field(struct ftrace_event_call *call, char *type,
 		       char *name, int offset, int size);
 extern void filter_free_pred(struct filter_pred *pred);
-extern void filter_print_preds(struct filter_pred **preds,
+extern void filter_print_preds(struct ftrace_event_call *call,
 			       struct trace_seq *s);
 extern int filter_parse(char **pbuf, struct filter_pred *pred);
 extern int filter_add_pred(struct ftrace_event_call *call,
 			   struct filter_pred *pred);
 extern void filter_free_preds(struct ftrace_event_call *call);
-extern int filter_match_preds(struct ftrace_event_call *call, void *rec);
+extern int filter_match_preds(struct filter_pred **preds, void *rec);
+extern void filter_print_subsystem_preds(struct event_subsystem *system,
+					 struct trace_seq *s);
 extern void filter_free_subsystem_preds(struct event_subsystem *system);
 extern int filter_add_subsystem_pred(struct event_subsystem *system,
 				     struct filter_pred *pred);
kernel/trace/trace_events.c +2 -2
index 576f4fa..ca1a2b0 100644
@@ -481,7 +481,7 @@ event_filter_read(struct file *filp, char __user *ubuf, size_t cnt,
 
 	trace_seq_init(s);
 
-	filter_print_preds(call->preds, s);
+	filter_print_preds(call, s);
 	r = simple_read_from_buffer(ubuf, cnt, ppos, s->buffer, s->len);
 
 	kfree(s);
@@ -549,7 +549,7 @@ subsystem_filter_read(struct file *filp, char __user *ubuf, size_t cnt,
 
 	trace_seq_init(s);
 
-	filter_print_preds(system->preds, s);
+	filter_print_subsystem_preds(system, s);
 	r = simple_read_from_buffer(ubuf, cnt, ppos, s->buffer, s->len);
 
 	kfree(s);
kernel/trace/trace_events_filter.c +76 -28

185 lines not on this page · raw message

kernel/trace/trace_events_stage_3.h +9 -1
index 9d2fa78..973941d 100644
@@ -207,8 +207,10 @@ static void ftrace_raw_event_##call(proto)				\
 	struct ftrace_event_call *call = &event_##call;			\
 	struct ring_buffer_event *event;				\
 	struct ftrace_raw_##call *entry;				\
+	struct filter_pred **preds;					\
 	unsigned long irq_flags;					\
 	int pc;								\
+	int matched = 1;						\
 									\
 	local_save_flags(irq_flags);					\
 	pc = preempt_count();						\
@@ -222,7 +224,13 @@ static void ftrace_raw_event_##call(proto)				\
 									\
 	assign;								\
 									\
-	if (call->preds && !filter_match_preds(call, entry))		\
+	rcu_read_lock();						\
+	preds = rcu_dereference(call->preds);				\
+	if (preds)							\
+		matched = filter_match_preds(preds, entry);		\
+	rcu_read_unlock();						\
+									\
+	if (!matched)							\
 		ring_buffer_event_discard(event);			\
 									\
 	trace_nowake_buffer_unlock_commit(event, irq_flags, pc);	\
Signature · 1 line
1.5.4.rc3
#15 lore raw
In reply to Li Zefan

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

Tom

Quoted 18 lines

Maybe 3rd to 6th patches can be regarded as small enhancements instead of bug fixes? But they are behavioural changes.

[PATCH 1/7] tracing/filters: NUL-terminate user input filter [PATCH 2/7] tracing/filters: fix NULL pointer dereference
[PATCH 3/7] tracing/filters: allow user input integer to be oct or hex [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string [PATCH 5/7] tracing/filters: disallow newline as delimeter
[PATCH 6/7] tracing/filters: return proper error code when writing filter file [PATCH 7/7] tracing/filters: make filter preds RCU safe

 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |   16 +++--
 kernel/trace/trace_events_filter.c  |  131 ++++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 121 insertions(+), 42 deletions(-)

---

author
#16 lore raw
Quoted 7 lines · Tom Zanussi

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

Quoted 6 lines

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Hm, I didn't notice it, since I just started looking into tracing/filters 2 days ago.

Quoted 11 lines
Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

I'll keep an eye on this issue.

#7 lore raw

Re: [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string

In reply to Li Zefan

Hi Li,

Quoted 9 lines · Li Zefan

Suppose we would like to trace all tasks named '123', but this will fail:

 # echo 'parent_comm == 123' > events/sched/sched_process_fork/filter
 bash: echo: write error: Invalid argument

With this patch, we allow it by:

 # echo 'parent_comm == \123' > events/sched/sched_process_fork/filter
 # cat events/sched/sched_process_fork/filter
 parent_comm == 123

Well, IMHO, it would be rather better to just echo 'parent_comm == 123' and let it answer depending of which filter_pred_*() callback we have for the concerned field.

The culprit is this part in filter_parse():

pred->val = simple_strtoull(val_str, &tmp, 10);

if (tmp == val_str) {
	pred->str_val = kstrdup(val_str, GFP_KERNEL);
	if (!pred->str_val)
		return -ENOMEM;
}

The idea would be to not anymore base the check on simple_strtoull to guess whether this is a number or a string, making it act subsequently to this assumption, which is not the good assumption we must base our parsing yet.

Instead, we could let filter_parse only do the job of extracting the tokens and then fill the whole pred struct without yet bothering about the type of the value.

Thereafter we may defer the real value type checking on filter_add_pred() depending on the type of the concerned field:

if (is_string_field()) {
	add it as a string value;
} else {

do the check with simple_strtoull
looks good? Then go to the number size switch....
...
}

You see?

I think it would be a saner basis of parsing.

Thanks,
Frederic.

Quoted 43 lines
---
 kernel/trace/trace_events_filter.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)
kernel/trace/trace_events_filter.c +10 -1
index 49b3ef5..2bf4481 100644
@@ -351,12 +351,19 @@ oom:
 	return -ENOMEM;
 }
 
+/*
+ * The filter format can be
+ *   - 0, which means remove all filter preds
+ *   - [||/&&] <field> ==/!= [\]<val>
+ *
+ * Note: '\' prevent a string type value beginning with a digit to
+ *       be treated as a number
+ */
 int filter_parse(char **pbuf, struct filter_pred *pred)
 {
 	char *tmp, *tok, *val_str = NULL;
 	int tok_n = 0;
 
-	/* field ==/!= number, or/and field ==/!= number, number */
 	while ((tok = strsep(pbuf, " \n"))) {
 		if (tok_n == 0) {
 			if (!strcmp(tok, "0")) {
@@ -421,6 +428,8 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 
 	pred->val = simple_strtoull(val_str, &tmp, 0);
 	if (tmp == val_str) {
+		if (*val_str == '\\')
+			val_str++;
 		pred->str_val = kstrdup(val_str, GFP_KERNEL);
 		if (!pred->str_val)
 			return -ENOMEM;
Signature · 1 line
1.5.4.rc3
#17 lore raw
In reply to Tom Zanussi
Quoted 18 lines · Tom Zanussi

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Quoted 7 lines

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

Tom

It would be sort of reinventing rcu :-)
Well, Paul proposed something recently, hmm I should double check this discussion.

Anyway, the fixes from Li (other than 4 and 7 for which we have comments) look very good!

Frederic.

Quoted 20 lines
Quoted 19 lines

Maybe 3rd to 6th patches can be regarded as small enhancements instead of bug fixes? But they are behavioural changes.

[PATCH 1/7] tracing/filters: NUL-terminate user input filter [PATCH 2/7] tracing/filters: fix NULL pointer dereference
[PATCH 3/7] tracing/filters: allow user input integer to be oct or hex [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string [PATCH 5/7] tracing/filters: disallow newline as delimeter
[PATCH 6/7] tracing/filters: return proper error code when writing filter file [PATCH 7/7] tracing/filters: make filter preds RCU safe

 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |   16 +++--
 kernel/trace/trace_events_filter.c  |  131 ++++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 121 insertions(+), 42 deletions(-)

---

#18 lore raw
Quoted 25 lines · Frederic Weisbecker
Quoted 24 lines · Tom Zanussi

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

Quoted 5 lines
Quoted 1 line

Tom

It would be sort of reinventing rcu :-)
Well, Paul proposed something recently, hmm I should double check this discussion.

I am actually thinking about doing this. Please do take a look at:

	http://lkml.org/lkml/2009/4/6/332
	http://lkml.org/lkml/2009/4/6/496

What I need from you is to tell me whether or not the proposed placement of rcu_idle(), rcu_idle_start(), and rcu_idle_end() makes sense from a tracing viewpoint.

Thanx, Paul

Quoted 33 lines

Anyway, the fixes from Li (other than 4 and 7 for which we have comments) look very good!

Frederic.

Quoted 13 lines
Quoted 13 lines

Maybe 3rd to 6th patches can be regarded as small enhancements instead of bug fixes? But they are behavioural changes.

[PATCH 1/7] tracing/filters: NUL-terminate user input filter [PATCH 2/7] tracing/filters: fix NULL pointer dereference
[PATCH 3/7] tracing/filters: allow user input integer to be oct or hex [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string [PATCH 5/7] tracing/filters: disallow newline as delimeter
[PATCH 6/7] tracing/filters: return proper error code when writing filter file [PATCH 7/7] tracing/filters: make filter preds RCU safe

Quoted 7 lines
Quoted 6 lines
 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |   16 +++--
 kernel/trace/trace_events_filter.c  |  131 ++++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 121 insertions(+), 42 deletions(-)

---

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#19 lore raw
In reply to Frederic Weisbecker
Quoted 25 lines · Frederic Weisbecker
Quoted 24 lines · Tom Zanussi

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

Quoted 8 lines
Quoted 1 line

Tom

It would be sort of reinventing rcu :-)
Well, Paul proposed something recently, hmm I should double check this discussion.

Well, if so, the resemblance would be only superficial e.g. the difference between a cheap parlor trick and David Copperfield. ;-) Mine would be much simpler and local to the filter code. Also, I think I'm overstating the 'keeping the fields intact for awhile' part - it should be possible to not deal with that at all at the cost of a small possibility of a false positive or negative while switching filters.

I'll run it up the flagpole and see if it flies, and hopefully post a patch for it, sometime this weekend...

Tom

Quoted 28 lines

Anyway, the fixes from Li (other than 4 and 7 for which we have comments) look very good!

Frederic.

Quoted 20 lines
Quoted 19 lines

Maybe 3rd to 6th patches can be regarded as small enhancements instead of bug fixes? But they are behavioural changes.

[PATCH 1/7] tracing/filters: NUL-terminate user input filter [PATCH 2/7] tracing/filters: fix NULL pointer dereference
[PATCH 3/7] tracing/filters: allow user input integer to be oct or hex [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string [PATCH 5/7] tracing/filters: disallow newline as delimeter
[PATCH 6/7] tracing/filters: return proper error code when writing filter file [PATCH 7/7] tracing/filters: make filter preds RCU safe

 kernel/trace/trace.h                |    6 +-
 kernel/trace/trace_events.c         |   16 +++--
 kernel/trace/trace_events_filter.c  |  131 ++++++++++++++++++++++++++---------
 kernel/trace/trace_events_stage_3.h |   10 +++-
 4 files changed, 121 insertions(+), 42 deletions(-)

---

#20 lore raw
In reply to Frederic Weisbecker
Quoted 29 lines · Frederic Weisbecker
Quoted 18 lines · Tom Zanussi

Hi,

Quoted 3 lines · Li Zefan

This patchset fixes some bugs in tracing/filters. Most of the change goes to the last patch, and others are small ones.

At first glance, they look good and fix real problems - thanks for fixing them. Re patch 7, there's been some discussion about using rcu for this. See:

http://lkml.org/lkml/2009/4/5/46

Quoted 1 line

From that discussion, it seems some non-trivial changes to rcu would be

needed for this. I'm playing around with a different idea now to hopefully avoid the need for that, or the other approach mentioned, of temporarily stopping tracing while removing/changing the filters.

Quoted 7 lines

Basically my thought is to avoid the problem by not allocating or destroying the preds when removing filters but instead switch out the pred->fns with a nop version while keeping the fields intact for awhile. I think that will work for removing filters, but I still need to think about how it would (or would not) work for replacing them.

Tom

Quoted 7 lines

It would be sort of reinventing rcu :-)

Well, Paul proposed something recently, hmm I should double check this discussion.

Anyway, the fixes from Li (other than 4 and 7 for which we have comments) look very good!

Ok, i've applied 1,2,3,6 to tip/tracing/urgent (most of them are fixes needed for current mainline).

5 had dependency on 4 which is being discussed. 7 is being discussed as well, with Tom having promised a patch.

Thanks,

Ingo

#8 lore raw

Re: [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string

In reply to Frederic Weisbecker
Quoted 42 lines · Frederic Weisbecker

Hi Li,

Quoted 9 lines · Li Zefan

Suppose we would like to trace all tasks named '123', but this will fail:

 # echo 'parent_comm == 123' > events/sched/sched_process_fork/filter
 bash: echo: write error: Invalid argument

With this patch, we allow it by:

 # echo 'parent_comm == \123' > events/sched/sched_process_fork/filter
 # cat events/sched/sched_process_fork/filter
 parent_comm == 123

Well, IMHO, it would be rather better to just echo 'parent_comm == 123' and let it answer depending of which filter_pred_*() callback we have for the concerned field.

The culprit is this part in filter_parse():

pred->val = simple_strtoull(val_str, &tmp, 10);

if (tmp == val_str) {
	pred->str_val = kstrdup(val_str, GFP_KERNEL);
	if (!pred->str_val)
		return -ENOMEM;
}

The idea would be to not anymore base the check on simple_strtoull to guess whether this is a number or a string, making it act subsequently to this assumption, which is not the good assumption we must base our parsing yet.

Instead, we could let filter_parse only do the job of extracting the tokens and then fill the whole pred struct without yet bothering about the type of the value.

Thereafter we may defer the real value type checking on filter_add_pred() depending on the type of the concerned field:

if (is_string_field()) {
	add it as a string value;
} else {
Quoted 8 lines

do the check with simple_strtoull
looks good? Then go to the number size switch....
...
}

You see?

I think it would be a saner basis of parsing.

Agreed. The user input itself is unambigously parse-able - and we should not work around parsing limitations by complicating the user input.

The failure situation that Li found should obviously be fixed.

Ingo

author
#3 lore raw

[tip:tracing/urgent] tracing/filters: fix NULL pointer dereference

In reply to Li Zefan
Commit-ID:  bcabd91c271e50eebc0cb9220ac92700332b452e
Gitweb:     http://git.kernel.org/tip/bcabd91c271e50eebc0cb9220ac92700332b452e
Author:     Li Zefan <[email protected]>
AuthorDate: Sat, 11 Apr 2009 15:52:35 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 12 Apr 2009 11:59:28 +0200

tracing/filters: fix NULL pointer dereference

Try this, and you'll see NULL pointer dereference bug:

  # echo -n 'parent_comm ==' > sched/sched_process_fork/filter

Because we passed NULL ptr to simple_strtoull().

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_events_filter.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
kernel/trace/trace_events_filter.c +5 -0
index 026be41..9d2162f 100644
@@ -410,6 +410,11 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 		}
 	}
 
+	if (!val_str) {
+		pred->field_name = NULL;
+		return -EINVAL;
+	}
+
 	pred->field_name = kstrdup(pred->field_name, GFP_KERNEL);
 	if (!pred->field_name)
 		return -ENOMEM;
author
#5 lore raw

[tip:tracing/urgent] tracing/filters: allow user input integer to be oct or hex

In reply to Li Zefan
Commit-ID:  a3e0ab050774117d4a6173087c8bf3888662a83f
Gitweb:     http://git.kernel.org/tip/a3e0ab050774117d4a6173087c8bf3888662a83f
Author:     Li Zefan <[email protected]>
AuthorDate: Sat, 11 Apr 2009 15:52:51 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 12 Apr 2009 11:59:28 +0200

tracing/filters: allow user input integer to be oct or hex

Before patch:

 # echo 'parent_pid == 0x10' > events/sched/sched_process_fork/filter
 # cat sched/sched_process_fork/filter
 parent_pid == 0

After patch:

 # cat sched/sched_process_fork/filter
 parent_pid == 16

Also check the input more strictly.

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_events_filter.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
kernel/trace/trace_events_filter.c +3 -2
index 9d2162f..49b3ef5 100644
@@ -419,12 +419,13 @@ int filter_parse(char **pbuf, struct filter_pred *pred)
 	if (!pred->field_name)
 		return -ENOMEM;
 
-	pred->val = simple_strtoull(val_str, &tmp, 10);
+	pred->val = simple_strtoull(val_str, &tmp, 0);
 	if (tmp == val_str) {
 		pred->str_val = kstrdup(val_str, GFP_KERNEL);
 		if (!pred->str_val)
 			return -ENOMEM;
-	}
+	} else if (*tmp != '\0')
+		return -EINVAL;
 
 	return 0;
 }
author
#13 lore raw

[tip:tracing/urgent] tracing/filters: return proper error code when writing filter file

In reply to Li Zefan
Commit-ID:  44e9c8b7adc52079f0535f9de0c2c2477831389b
Gitweb:     http://git.kernel.org/tip/44e9c8b7adc52079f0535f9de0c2c2477831389b
Author:     Li Zefan <[email protected]>
AuthorDate: Sat, 11 Apr 2009 15:55:28 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 12 Apr 2009 11:59:29 +0200

tracing/filters: return proper error code when writing filter file

- propagate return value of filter_add_pred() to the user

- return -ENOSPC but not -ENOMEM or -EINVAL when the filter array is full

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_events.c        |   10 ++++++----
 kernel/trace/trace_events_filter.c |    4 ++--
 2 files changed, 8 insertions(+), 6 deletions(-)
kernel/trace/trace_events.c +6 -4
index 054bc18..576f4fa 100644
@@ -521,9 +521,10 @@ event_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 		return cnt;
 	}
 
-	if (filter_add_pred(call, pred)) {
+	err = filter_add_pred(call, pred);
+	if (err < 0) {
 		filter_free_pred(pred);
-		return -EINVAL;
+		return err;
 	}
 
 	*ppos += cnt;
@@ -588,10 +589,11 @@ subsystem_filter_write(struct file *filp, const char __user *ubuf, size_t cnt,
 		return cnt;
 	}
 
-	if (filter_add_subsystem_pred(system, pred)) {
+	err = filter_add_subsystem_pred(system, pred);
+	if (err < 0) {
 		filter_free_subsystem_preds(system);
 		filter_free_pred(pred);
-		return -EINVAL;
+		return err;
 	}
 
 	*ppos += cnt;
kernel/trace/trace_events_filter.c +2 -2
index 49b3ef5..e03cbf1 100644
@@ -215,7 +215,7 @@ static int __filter_add_pred(struct ftrace_event_call *call,
 		}
 	}
 
-	return -ENOMEM;
+	return -ENOSPC;
 }
 
 static int is_string_field(const char *type)
@@ -319,7 +319,7 @@ int filter_add_subsystem_pred(struct event_subsystem *system,
 	}
 
 	if (i == MAX_FILTER_PRED)
-		return -EINVAL;
+		return -ENOSPC;
 
 	events_for_each(call) {
 		int err;
author
#9 lore raw

Re: [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string

In reply to Frederic Weisbecker
Quoted 28 lines

Well, IMHO, it would be rather better to just echo 'parent_comm == 123' and let it answer depending of which filter_pred_*() callback we have for the concerned field.

The culprit is this part in filter_parse():

pred->val = simple_strtoull(val_str, &tmp, 10);

if (tmp == val_str) {
	pred->str_val = kstrdup(val_str, GFP_KERNEL);
	if (!pred->str_val)
		return -ENOMEM;
}

The idea would be to not anymore base the check on simple_strtoull to guess whether this is a number or a string, making it act subsequently to this assumption, which is not the good assumption we must base our parsing yet.

Instead, we could let filter_parse only do the job of extracting the tokens and then fill the whole pred struct without yet bothering about the type of the value.

Thereafter we may defer the real value type checking on filter_add_pred() depending on the type of the concerned field:

if (is_string_field()) {
	add it as a string value;
} else {
Quoted 7 lines

do the check with simple_strtoull
looks good? Then go to the number size switch....
...
}

You see?

Right! Actually I thought about this, then I found one issue, suppose event foo and event bar both have a field named fb but one is string and one is integer. Now do this:

  # echo 'fb == 123' > events/foo-bar/filter

This will set both filters, but not only the integer one.

But now I think this hardly happen in real-life, and it's not a big issue if it does happen. So I agree with you on this issue.

Thanks.

Quoted 2 lines

I think it would be a saner basis of parsing.

#10 lore raw

Re: [PATCH 4/7] tracing/filters: allow user to specify a filter val to be string

Quoted 40 lines · Li Zefan
Quoted 28 lines

Well, IMHO, it would be rather better to just echo 'parent_comm == 123' and let it answer depending of which filter_pred_*() callback we have for the concerned field.

The culprit is this part in filter_parse():

pred->val = simple_strtoull(val_str, &tmp, 10);

if (tmp == val_str) {
	pred->str_val = kstrdup(val_str, GFP_KERNEL);
	if (!pred->str_val)
		return -ENOMEM;
}

The idea would be to not anymore base the check on simple_strtoull to guess whether this is a number or a string, making it act subsequently to this assumption, which is not the good assumption we must base our parsing yet.

Instead, we could let filter_parse only do the job of extracting the tokens and then fill the whole pred struct without yet bothering about the type of the value.

Thereafter we may defer the real value type checking on filter_add_pred() depending on the type of the concerned field:

if (is_string_field()) {
	add it as a string value;
} else {
Quoted 7 lines

do the check with simple_strtoull
looks good? Then go to the number size switch....
...
}

You see?

Right! Actually I thought about this, then I found one issue, suppose event foo and event bar both have a field named fb but one is string and one is integer. Now do this:

Quoted 6 lines
  # echo 'fb == 123' > events/foo-bar/filter

This will set both filters, but not only the integer one.

But now I think this hardly happen in real-life, and it's not a big issue if it does happen. So I agree with you on this issue.

Yeah, good point. I think we should avoid such field name aliasing. Even without the filter assignment ambiguity problem it would be confusing to users to have two same-name but different-type fields in two separate events.

Ingo