Skip to contentSkip to conversation

Optimize pid filters and add --no-filter option

PATCH Superseded Slavomir Kaslev 4 participants 10 messages Apr 15, 2019 – Apr 17, 2019

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Optimize pid filters and add --no-filter option

This patchset optimizes how pid filters are expressed and makes it less likely that we overflow ftrace filters' size limit of one page.

Changes since v2:

Append exclude rules with &&

Changes since v1:

Add missing tags
Fix append_filter_pid_range() callers to pass valid range as [min,max]

Slavomir Kaslev (2):
trace-cmd: Optimize how pid filters are expressed
trace-cmd: Add --no-filter option to not filter recording processes

 tracecmd/trace-record.c | 131 ++++++++++++++++++++++++++++------------
 tracecmd/trace-usage.c  |   1 +
 2 files changed, 92 insertions(+), 40 deletions(-)
Signature · 1 line
2.19.1
#1 lore raw

PATCH v3 1/2 trace-cmd: Optimize how pid filters are expressed

Express pid filters as allowed/disallowed filter ranges

(pid>=100&&pid<=103)

instead of specifying them per pid

(pid==100||pid==101||pid==102||pid==103)

This makes the size of the resulting filter smaller (and faster) and avoids overflowing the filter size limit of one page which we can hit on bigger machines (say >160 CPUs).

---
 tracecmd/trace-record.c | 117 +++++++++++++++++++++++++++-------------
 1 file changed, 80 insertions(+), 37 deletions(-)
tracecmd/trace-record.c +80 -37
index 76ca92d..dae0396 100644
@@ -950,10 +950,63 @@ static void update_ftrace_pids(int reset)
 static void update_event_filters(struct buffer_instance *instance);
 static void update_pid_event_filters(struct buffer_instance *instance);
 
+static void append_filter_pid_range(char **filter, int *curr_len,
+				    const char *field,
+				    int start_pid, int end_pid, bool exclude)
+{
+	const char *op = "", *op1, *op2, *op3;
+	int len;
+
+	if (*filter && **filter)
+		op = exclude ? "&&" : "||";
+
+	/* Handle thus case explicitly so that we get `pid==3` instead of
+	 * `pid>=3&&pid<=3` for singleton ranges
+	 */
+	if (start_pid == end_pid) {
+#define FMT	"%s(%s%s%d)"
+		len = snprintf(NULL, 0, FMT, op,
+			       field, exclude ? "!=" : "==", start_pid);
+		*filter = realloc(*filter, *curr_len + len + 1);
+		if (!*filter)
+			die("realloc");
+
+		len = snprintf(*filter + *curr_len, len + 1, FMT, op,
+			       field, exclude ? "!=" : "==", start_pid);
+		*curr_len += len;
+
+		return;
+#undef FMT
+	}
+
+	if (exclude) {
+		op1 = "<";
+		op2 = "||";
+		op3 = ">";
+	} else {
+		op1 = ">=";
+		op2 = "&&";
+		op3 = "<=";
+	}
+
+#define FMT	"%s(%s%s%d%s%s%s%d)"
+	len = snprintf(NULL, 0, FMT, op,
+		       field, op1, start_pid, op2,
+		       field, op3, end_pid);
+	*filter = realloc(*filter, *curr_len + len + 1);
+	if (!*filter)
+		die("realloc");
+
+	len = snprintf(*filter + *curr_len, len + 1, FMT, op,
+		       field, op1, start_pid, op2,
+		       field, op3, end_pid);
+	*curr_len += len;
+}
+
 /**
  * make_pid_filter - create a filter string to all pids against @field
  * @curr_filter: Append to a previous filter (may realloc). Can be NULL
- * @field: The fild to compare the pids against
+ * @field: The field to compare the pids against
  *
  * Creates a new string or appends to an existing one if @curr_filter
  * is not NULL. The new string will contain a filter with all pids
@@ -963,54 +1016,44 @@ static void update_pid_event_filters(struct buffer_instance *instance);
  */
 static char *make_pid_filter(char *curr_filter, const char *field)
 {
+	int curr_len = 0, last_exclude = -1;
+	int start_pid = -1, last_pid = -1;
+	char *filter = NULL, *save;
 	struct filter_pids *p;
-	char *filter;
-	char *orit;
-	char *match;
-	char *str;
-	int curr_len = 0;
-	int len;
 
 	/* Use the new method if possible */
 	if (have_set_event_pid)
 		return NULL;
 
-	len = len_filter_pids + (strlen(field) + strlen("(==)||")) * nr_filter_pids;
-
-	if (curr_filter) {
-		curr_len = strlen(curr_filter);
-		filter = realloc(curr_filter, curr_len + len + strlen("(&&())"));
-		if (!filter)
-			die("realloc");
-		memmove(filter+1, curr_filter, curr_len);
-		filter[0] = '(';
-		strcat(filter, ")&&(");
-		curr_len = strlen(filter);
-	} else
-		filter = malloc(len);
-	if (!filter)
-		die("Failed to allocate pid filter");
-
-	/* Last '||' that is not used will cover the \0 */
-	str = filter + curr_len;
+	if (!filter_pids)
+		return curr_filter;
 
 	for (p = filter_pids; p; p = p->next) {
-		if (p->exclude) {
-			match = "!=";
-			orit = "&&";
-		} else {
-			match = "==";
-			orit = "||";
+		/* PIDs are inserted in `filter_pids` from the front and that's
+		 * why we expect them in descending order here.
+		 */
+		if (p->pid == last_pid - 1 && p->exclude == last_exclude) {
+			last_pid = p->pid;
+			continue;
 		}
-		if (p == filter_pids)
-			orit = "";
 
-		len = sprintf(str, "%s(%s%s%d)", orit, field, match, p->pid);
-		str += len;
+		if (start_pid != -1)
+			append_filter_pid_range(&filter, &curr_len, field,
+						last_pid, start_pid,
+						last_exclude);
+
+		start_pid = last_pid = p->pid;
+		last_exclude = p->exclude;
+
 	}
+	append_filter_pid_range(&filter, &curr_len, field,
+				last_pid, start_pid, last_exclude);
 
-	if (curr_len)
-		sprintf(str, ")");
+	if (curr_filter) {
+		save = filter;
+		asprintf(&filter, "(%s)&&(%s)", curr_filter, filter);
+		free(save);
+	}
 
 	return filter;
 }
Signature · 1 line
2.19.1
#2 lore raw

Re: [PATCH v3 1/2] trace-cmd: Optimize how pid filters are expressed

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1019

Quoted 3 lines · On Tue, 16 Apr 2019 02:00:15 +0300 Slavomir Kaslev <[email protected]> wrote:

static char *make_pid_filter(char *curr_filter, const char *field) {

tracecmd/trace-record.c +1 -0
+	int curr_len = 0, last_exclude = -1;

Small nit. Usually when adding multiple variables on one line like this, the variables should be related. Because curr_len and last_exclude are not related, it is best to keep them separate.

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1020

Quoted 1 line
tracecmd/trace-record.c +1 -0
+	int start_pid = -1, last_pid = -1;

start_pid and last_pid are fine on the same line or separate.

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1021

Quoted 1 line
tracecmd/trace-record.c +1 -0
+	char *filter = NULL, *save;

Note, It's better to keep variables separate. It makes it easier to see them and also when code changes and you get the "variable x is not used" warnings from gcc, it's easier to remove them. In emacs it brings you to the warning and places the cursor on the problem line. Then a simple "delete line" works well.

save could also be moved below.

Quoted 31 lines
tracecmd/trace-record.c +1 -24
 	struct filter_pids *p;
-	char *filter;
-	char *orit;
-	char *match;
-	char *str;
-	int curr_len = 0;
-	int len;
 
 	/* Use the new method if possible */
 	if (have_set_event_pid)
 		return NULL;
 
-	len = len_filter_pids + (strlen(field) + strlen("(==)||")) * nr_filter_pids;
-
-	if (curr_filter) {
-		curr_len = strlen(curr_filter);
-		filter = realloc(curr_filter, curr_len + len + strlen("(&&())"));
-		if (!filter)
-			die("realloc");
-		memmove(filter+1, curr_filter, curr_len);
-		filter[0] = '(';
-		strcat(filter, ")&&(");
-		curr_len = strlen(filter);
-	} else
-		filter = malloc(len);
-	if (!filter)
-		die("Failed to allocate pid filter");
-
-	/* Last '||' that is not used will cover the \0 */
-	str = filter + curr_len;
+	if (!filter_pids)

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1032

Quoted 10 lines
tracecmd/trace-record.c +2 -6
+		return curr_filter;
 
 	for (p = filter_pids; p; p = p->next) {
-		if (p->exclude) {
-			match = "!=";
-			orit = "&&";
-		} else {
-			match = "==";
-			orit = "||";
+		/* PIDs are inserted in `filter_pids` from the front and that's

Comment notation should be:

		/*
		 * PIDs are inserted in ...

Only the Networking folks are allowed to do it that way in the kernel ;-)

Quoted 16 lines
tracecmd/trace-record.c +10 -4
+		 * why we expect them in descending order here.
+		 */
+		if (p->pid == last_pid - 1 && p->exclude == last_exclude) {
+			last_pid = p->pid;
+			continue;
 		}
-		if (p == filter_pids)
-			orit = "";
 
-		len = sprintf(str, "%s(%s%s%d)", orit, field, match, p->pid);
-		str += len;
+		if (start_pid != -1)
+			append_filter_pid_range(&filter, &curr_len, field,
+						last_pid, start_pid,
+						last_exclude);
+

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1052

Quoted 10 lines
tracecmd/trace-record.c +6 -2
+		start_pid = last_pid = p->pid;
+		last_exclude = p->exclude;
+
 	}
+	append_filter_pid_range(&filter, &curr_len, field,
+				last_pid, start_pid, last_exclude);
 
-	if (curr_len)
-		sprintf(str, ")");
+	if (curr_filter) {

As save is only used here, we could have:

		char *save = filter;

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1059

Quoted 7 lines
tracecmd/trace-record.c +4 -0
+		save = filter;
+		asprintf(&filter, "(%s)&&(%s)", curr_filter, filter);
+		free(save);
+	}
 
 	return filter;
 }

-- Steve

#3 lore raw

Re: [PATCH v3 1/2] trace-cmd: Optimize how pid filters are expressed

Quoted 10 lines · Steven Rostedt

tracecmd/trace-record.c static void update_pid_event_filters(struct buffer_instance *instance); · line 1019

Quoted 3 lines · On Tue, 16 Apr 2019 02:00:15 +0300 Slavomir Kaslev <[email protected]> wrote:

static char *make_pid_filter(char *curr_filter, const char *field) {

tracecmd/trace-record.c +1 -0
+	int curr_len = 0, last_exclude = -1;

Small nit. Usually when adding multiple variables on one line like this, the variables should be related. Because curr_len and last_exclude are not related, it is best to keep them separate.

Sent v4 addressing those and added a section about --no-filter to trace-cmd record's man page.

Cheers,

-- Slavi

#4 lore raw

PATCH v3 2/2 trace-cmd: Add --no-filter option to not filter recording processes

In reply to Slavomir Kaslev

Add --no-filter option which doesn't install filters for the trace-cmd recording processes pids.

---
 tracecmd/trace-record.c | 14 +++++++++++---
 tracecmd/trace-usage.c  |  1 +
 2 files changed, 12 insertions(+), 3 deletions(-)
tracecmd/trace-record.c +11 -3
index dae0396..05a313b 100644
@@ -86,6 +86,7 @@ static int do_ptrace;
 
 static int filter_task;
 static int filter_pid = -1;
+static bool no_filter = false;
 
 static int local_cpu_count;
 
@@ -1063,6 +1064,9 @@ static void update_task_filter(void)
 	struct buffer_instance *instance;
 	int pid = getpid();
 
+	if (no_filter)
+		return;
+
 	if (filter_task)
 		add_filter_pid(pid, 0);
 
@@ -4377,9 +4381,9 @@ void update_first_instance(struct buffer_instance *instance, int topt)
 }
 
 enum {
-
-	OPT_quiet		= 246,
-	OPT_debug		= 247,
+	OPT_quiet		= 245,
+	OPT_debug		= 246,
+	OPT_no_filter		= 247,
 	OPT_max_graph_depth	= 248,
 	OPT_tsoffset		= 249,
 	OPT_bycomm		= 250,
@@ -4603,6 +4607,7 @@ static void parse_record_options(int argc,
 			{"by-comm", no_argument, NULL, OPT_bycomm},
 			{"ts-offset", required_argument, NULL, OPT_tsoffset},
 			{"max-graph-depth", required_argument, NULL, OPT_max_graph_depth},
+			{"no-filter", no_argument, NULL, OPT_no_filter},
 			{"debug", no_argument, NULL, OPT_debug},
 			{"quiet", no_argument, NULL, OPT_quiet},
 			{"help", no_argument, NULL, '?'},
@@ -4877,6 +4882,9 @@ static void parse_record_options(int argc,
 			if (!ctx->max_graph_depth)
 				die("Could not allocate option");
 			break;
+		case OPT_no_filter:
+			no_filter = true;
+			break;
 		case OPT_debug:
 			debug = 1;
 			break;
tracecmd/trace-usage.c +1 -0
index 9ea1906..29a7081 100644
@@ -56,6 +56,7 @@ static struct usage_help usage_help[] = {
 		"          --func-stack perform a stack trace for function tracer\n"
 		"             (use with caution)\n"
 		"          --max-graph-depth limit function_graph depth\n"
+		"          --no-filter do not set any event filters\n"
 	},
 	{
 		"start",
Signature · 1 line
2.19.1
#5 lore raw

Re: [PATCH v3 2/2] trace-cmd: Add --no-filter option to not filter recording processes

Quoted 5 lines · On Tue, 16 Apr 2019 02:00:16 +0300 Slavomir Kaslev <[email protected]> wrote:

Add --no-filter option which doesn't install filters for the trace-cmd recording processes pids.

I'll apply this, but we should also update the man page.

-- Steve

Quoted 70 lines
---
 tracecmd/trace-record.c | 14 +++++++++++---
 tracecmd/trace-usage.c  |  1 +
 2 files changed, 12 insertions(+), 3 deletions(-)
tracecmd/trace-record.c +11 -3
index dae0396..05a313b 100644
@@ -86,6 +86,7 @@ static int do_ptrace;
 
 static int filter_task;
 static int filter_pid = -1;
+static bool no_filter = false;
 
 static int local_cpu_count;
 
@@ -1063,6 +1064,9 @@ static void update_task_filter(void)
 	struct buffer_instance *instance;
 	int pid = getpid();
 
+	if (no_filter)
+		return;
+
 	if (filter_task)
 		add_filter_pid(pid, 0);
 
@@ -4377,9 +4381,9 @@ void update_first_instance(struct buffer_instance *instance, int topt)
 }
 
 enum {
-
-	OPT_quiet		= 246,
-	OPT_debug		= 247,
+	OPT_quiet		= 245,
+	OPT_debug		= 246,
+	OPT_no_filter		= 247,
 	OPT_max_graph_depth	= 248,
 	OPT_tsoffset		= 249,
 	OPT_bycomm		= 250,
@@ -4603,6 +4607,7 @@ static void parse_record_options(int argc,
 			{"by-comm", no_argument, NULL, OPT_bycomm},
 			{"ts-offset", required_argument, NULL, OPT_tsoffset},
 			{"max-graph-depth", required_argument, NULL, OPT_max_graph_depth},
+			{"no-filter", no_argument, NULL, OPT_no_filter},
 			{"debug", no_argument, NULL, OPT_debug},
 			{"quiet", no_argument, NULL, OPT_quiet},
 			{"help", no_argument, NULL, '?'},
@@ -4877,6 +4882,9 @@ static void parse_record_options(int argc,
 			if (!ctx->max_graph_depth)
 				die("Could not allocate option");
 			break;
+		case OPT_no_filter:
+			no_filter = true;
+			break;
 		case OPT_debug:
 			debug = 1;
 			break;
tracecmd/trace-usage.c +1 -0
index 9ea1906..29a7081 100644
@@ -56,6 +56,7 @@ static struct usage_help usage_help[] = {
 		"          --func-stack perform a stack trace for function tracer\n"
 		"             (use with caution)\n"
 		"          --max-graph-depth limit function_graph depth\n"
+		"          --no-filter do not set any event filters\n"
 	},
 	{
 		"start",
Acked-by on the v3 series
#6 lore raw
In reply to Slavomir Kaslev
Quoted 13 lines · On Tue, Apr 16, 2019 at 02:00:14AM +0300 Slavomir Kaslev wrote:

This patchset optimizes how pid filters are expressed and makes it less likely that we overflow ftrace filters' size limit of one page.

Changes since v2:

Append exclude rules with &&

Changes since v1:

Add missing tags
Fix append_filter_pid_range() callers to pass valid range as [min,max]

Quoted 11 lines

Slavomir Kaslev (2):
trace-cmd: Optimize how pid filters are expressed
trace-cmd: Add --no-filter option to not filter recording processes

 tracecmd/trace-record.c | 131 ++++++++++++++++++++++++++++------------
 tracecmd/trace-usage.c  |   1 +
 2 files changed, 92 insertions(+), 40 deletions(-)
Signature · 1 line
2.19.1

This version is also working for my use case. The logic in the filter does show the recording thread pids in the sched switch event, at least when switching to/from a non-excluded process. But I think that's desired.

The --no-filter option works as expected.

Thanks,
Phil

#7 lore raw
Quoted 10 lines · On Tue, 16 Apr 2019 09:22:54 -0400 Phil Auld <[email protected]> wrote:

This version is also working for my use case. The logic in the filter does show the recording thread pids in the sched switch event, at least when switching to/from a non-excluded process. But I think that's desired.

The --no-filter option works as expected.

Could we get a Tested-by: from you for this series?

Thanks!

-- Steve

Tested-by
#8 lore raw
Quoted 8 lines · On Tue, Apr 16, 2019 at 05:39:09PM -0400 Steven Rostedt wrote:
Quoted 5 lines · On Tue, 16 Apr 2019 09:22:54 -0400 Phil Auld <[email protected]> wrote:

This version is also working for my use case. The logic in the filter does show the recording thread pids in the sched switch event, at least when switching to/from a non-excluded process. But I think that's desired.

Quoted 7 lines
Quoted 5 lines

The --no-filter option works as expected.

Could we get a Tested-by: from you for this series?

The second one for sure. I did it both with and without and saw the difference.

I was not comfortable with that for the first because I really only tested it in my setup on one machine (160 cpus). I didn't do really thorough testing. If that's enough for you then sure :)

Cheers,
Phil

Quoted 4 lines

Thanks!

-- Steve

#9 lore raw
Quoted 10 lines · On Tue, 16 Apr 2019 20:29:10 -0400 Phil Auld <[email protected]> wrote:
Quoted 1 line

Could we get a Tested-by: from you for this series?

The second one for sure. I did it both with and without and saw the difference.

I was not comfortable with that for the first because I really only tested it in my setup on one machine (160 cpus). I didn't do really thorough testing. If that's enough for you then sure :)

I'll just put it on the second one then.

Thanks!

-- Steve