Skip to contentSkip to conversation

perf record: mmap output file - v5

PATCH In review David Ahern 6 participants 42 messages Nov 12, 2013 – Nov 19, 2013 perf events

lore
Share & tools
lore
#0 lore raw

cover perf record: mmap output file - v5

Next round or mmap based output. This round addresses re-use of mmap-pages argument parsing, adds -O shortcut for --out-pages, allows -O0 to use write instead of mmap, and handles out of space failures.

David Ahern (5):
perf record: Fix segfault with --no-mmap-pages
perf tool: Round mmap pages to power 2 - v2
perf tool: Refactor mmap_pages parsing
perf record: mmap output file - v5
perf record: Handle out of space failures writing data with mmap

 tools/perf/Documentation/perf-record.txt |   7 ++
 tools/perf/builtin-record.c              | 174 +++++++++++++++++++++++++++++++
 tools/perf/util/evlist.c                 |  75 +++++++++----
 tools/perf/util/evlist.h                 |   3 +
 4 files changed, 240 insertions(+), 19 deletions(-)
Signature · 1 line
1.8.3.4 (Apple Git-47)
#3 lore raw

PATCH 2/5 perf tool: Round mmap pages to power 2 - v2

Currently perf requires the -m / --mmap_pages option to be a power of 2. To be more user friendly perf should automatically round this up to the next power of 2.

Currently:

  $ perf record -m 3 -a -- sleep 1
  --mmap_pages/-m value must be a power of two.sleep: Terminated

With patch:

  $ perf record -m 3 -a -- sleep 1
  rounding mmap pages size to 16384 (4 pages)
  ...

v2: Add bytes units to rounding message per Ingo's request. Other
suggestions (e.g., prefixing INFO) should be addressed by wrapping pr_info to catch all instances.

---
 tools/perf/util/evlist.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
tools/perf/util/evlist.c +6 -12
index 3960560f873a..fb4727d3df85 100644
@@ -710,7 +710,6 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 {
 	unsigned int *mmap_pages = opt->value;
 	unsigned long pages, val;
-	size_t size;
 	static struct parse_tag tags[] = {
 		{ .tag  = 'B', .mult = 1       },
 		{ .tag  = 'K', .mult = 1 << 10 },
@@ -726,11 +725,6 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 	if (val != (unsigned long) -1) {
 		/* we got file size value */
 		pages = PERF_ALIGN(val, page_size) / page_size;
-		if (pages < (1UL << 31) && !is_power_of_2(pages)) {
-			pages = next_pow2(pages);
-			pr_info("rounding mmap pages size to %lu (%lu pages)\n",
-				pages * page_size, pages);
-		}
 	} else {
 		/* we got pages count value */
 		char *eptr;
@@ -741,14 +735,14 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		}
 	}
 
-	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
-		pr_err("--mmap_pages/-m value too big\n");
-		return -1;
+	if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+		pages = next_pow2(pages);
+		pr_info("rounding mmap pages size to %lu bytes (%lu pages)\n",
+			pages * page_size, pages);
 	}
 
-	size = perf_evlist__mmap_size(pages);
-	if (!size) {
-		pr_err("--mmap_pages/-m value must be a power of two.");
+	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
+		pr_err("--mmap_pages/-m value too big\n");
 		return -1;
 	}
 
Signature · 1 line
1.8.3.4 (Apple Git-47)
#5 lore raw

PATCH 3/5 perf tool: Refactor mmap_pages parsing

In reply to David Ahern

Logic will be re-used for the out-pages argument for mmap based writes in perf-record.

Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Adrian Hunter <[email protected]>
---
 tools/perf/util/evlist.c | 39 ++++++++++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 11 deletions(-)
tools/perf/util/evlist.c +28 -11
index fb4727d3df85..cb19044601bb 100644
@@ -705,10 +705,9 @@ static size_t perf_evlist__mmap_size(unsigned long pages)
 	return (pages + 1) * page_size;
 }
 
-int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
-				  int unset __maybe_unused)
+static long parse_pages_arg(const char *str, unsigned long min,
+			    unsigned long max)
 {
-	unsigned int *mmap_pages = opt->value;
 	unsigned long pages, val;
 	static struct parse_tag tags[] = {
 		{ .tag  = 'B', .mult = 1       },
@@ -719,7 +718,7 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 	};
 
 	if (str == NULL)
-		return -1;
+		return -EINVAL;
 
 	val = parse_tag_value(str, tags);
 	if (val != (unsigned long) -1) {
@@ -729,20 +728,38 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		/* we got pages count value */
 		char *eptr;
 		pages = strtoul(str, &eptr, 10);
-		if (*eptr != '\0') {
-			pr_err("failed to parse --mmap_pages/-m value\n");
-			return -1;
-		}
+		if (*eptr != '\0')
+			return -EINVAL;
 	}
 
-	if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+	if ((pages == 0) && (min == 0)) {
+		/* leave number of pages at 0 */
+	} else if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+		/* round pages up to next power of 2 */
 		pages = next_pow2(pages);
 		pr_info("rounding mmap pages size to %lu bytes (%lu pages)\n",
 			pages * page_size, pages);
 	}
 
-	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
-		pr_err("--mmap_pages/-m value too big\n");
+	if (pages > max)
+		return -EINVAL;
+
+	return pages;
+}
+
+int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
+				  int unset __maybe_unused)
+{
+	unsigned int *mmap_pages = opt->value;
+	unsigned long max = UINT_MAX;
+	long pages;
+
+	if (max < SIZE_MAX / page_size)
+		max = SIZE_MAX / page_size;
+
+	pages = parse_pages_arg(str, 1, max);
+	if (pages < 0) {
+		pr_err("Invalid argument for --mmap_pages/-m\n");
 		return -1;
 	}
 
Signature · 1 line
1.8.3.4 (Apple Git-47)
#39 lore raw

PATCH 5/5 perf record: Handle out of space failures writing data with mmap

In reply to David Ahern

If the filesystem where a file is written using mmap fills perf record gets a SIGBUS and terminated. Handle the SIGBUS by using longjmp to bounce out of the memcpy and fail the write.

Cc Ingo Molnar <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 tools/perf/builtin-record.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
tools/perf/builtin-record.c +16 -6
index 1a4fa5df215b..48d6535d144f 100644
@@ -29,9 +29,11 @@
 #include <unistd.h>
 #include <sched.h>
 #include <sys/mman.h>
+#include <setjmp.h>
 
 /* output file mmap'ed N chunks at a time */
 #define MMAP_OUTPUT_SIZE   (64*1024*1024)
+sigjmp_buf mmap_jmp;
 
 #ifndef HAVE_ON_EXIT_SUPPORT
 #ifndef ATEXIT_MAX
@@ -141,6 +143,7 @@ static int do_mmap_output(struct perf_record *rec, void *buf, size_t size)
 {
 	u64 remaining;
 	off_t offset;
+	volatile size_t total_len = 0;
 
 	if (rec->mmap.addr == NULL) {
 next_segment:
@@ -157,20 +160,23 @@ next_segment:
 	 * space write what we can then go back and create the
 	 * next segment
 	 */
-	if (size > remaining) {
-		memcpy(rec->mmap.addr + rec->mmap.offset, buf, remaining);
+	if (setjmp(mmap_jmp) != 0) {
+		pr_err("mmap copy failed.\n");
+		return -1;
+	}
+	if (size-total_len > remaining) {
+		memcpy(rec->mmap.addr + rec->mmap.offset, buf+total_len, remaining);
 		rec->bytes_written += remaining;
 
-		size -= remaining;
-		buf  += remaining;
+		total_len += remaining;
 
 		munmap(rec->mmap.addr, rec->mmap.out_size);
 		goto next_segment;
 	}
 
 	/* more data to copy and it fits in the current segment */
-	if (size) {
-		memcpy(rec->mmap.addr + rec->mmap.offset, buf, size);
+	if (size - total_len) {
+		memcpy(rec->mmap.addr + rec->mmap.offset, buf+total_len, size-total_len);
 		rec->bytes_written += size;
 		rec->mmap.offset += size;
 	}
@@ -272,6 +278,9 @@ static void sig_handler(int sig)
 	if (sig == SIGCHLD)
 		child_finished = 1;
 
+	if (sig == SIGBUS)
+		longjmp(mmap_jmp, 1);
+
 	done = 1;
 	signr = sig;
 }
@@ -532,6 +541,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	signal(SIGINT, sig_handler);
 	signal(SIGUSR1, sig_handler);
 	signal(SIGTERM, sig_handler);
+	signal(SIGBUS, sig_handler);
 
 	session = perf_session__new(file, false, NULL);
 	if (session == NULL) {
Signature · 1 line
1.8.3.4 (Apple Git-47)
#7 lore raw

PATCH 4/5 perf record: mmap output file - v5

In reply to David Ahern

When recording raw_syscalls for the entire system, e.g.,
perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1

you end up with a negative feedback loop as perf itself calls write() fairly often. This patch handles the problem by mmap'ing the file in chunks of 64M at a time and copies events from the event buffers to the file avoiding write system calls.

Before (with write syscall):

perf record -o /tmp/perf.data -e raw_syscalls:*,sched:sched_switch -a -- sleep 1 [ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 81.843 MB /tmp/perf.data (~3575786 samples) ]

After (using mmap):

perf record -o /tmp/perf.data -e raw_syscalls:*,sched:sched_switch -a -- sleep 1 [ perf record: Woken up 31 times to write data ]
[ perf record: Captured and wrote 8.203 MB /tmp/perf.data (~358388 samples) ]

In addition to perf-trace benefits using mmap lowers the overhead of perf-record. For example,

perf stat -i -- perf record -g -o /tmp/perf.data openssl speed aes

shows a drop in time, CPU cycles, and instructions all drop by more than a factor of 3. Jiri also ran a test that showed a big improvement.

v5: Addressed misc comments from Jiri, Adrian and Arnaldo. Added -O shortcut
for --out-pages. Added -O 0 as a means to fall back to write

v4: Refactoring per Ingo's comments

v3: Removed use of bytes_at_mmap_start at the stat() that set it
Added user option to control the size of the mmap for writing file.

v2: Removed msync call before munmap per Jiri's suggestion

Cc Ingo Molnar <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 tools/perf/Documentation/perf-record.txt |   7 ++
 tools/perf/builtin-record.c              | 164 +++++++++++++++++++++++++++++++
 tools/perf/util/evlist.c                 |  23 +++++
 tools/perf/util/evlist.h                 |   3 +
 4 files changed, 197 insertions(+)
tools/perf/Documentation/perf-record.txt +7 -0
index 052f7c4dc00c..7c67dad9e341 100644
@@ -201,6 +201,13 @@ abort events and some memory events in precise mode on modern Intel CPUs.
 --transaction::
 Record transaction flags for transaction related events.
 
+-O::
+--out-pages=::
+Number of pages to mmap for writing data to file or size specification
+with appended unit character - B/K/M/G. The size is rounded up to have nearest
+pages power of two value. 0 falls back to write instead of mmap. Default size
+is 64M.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +164 -0

218 lines not on this page · raw message

tools/perf/util/evlist.c +23 -0
index cb19044601bb..3d1f7faa30d7 100644
@@ -767,6 +767,29 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 	return 0;
 }
 
+int perf_evlist__parse_out_pages(const struct option *opt, const char *str,
+				  int unset __maybe_unused)
+{
+	unsigned int *out_pages = opt->value;
+	unsigned long max = UINT_MAX;
+	long pages;
+
+	if (max < SIZE_MAX / page_size)
+		max = SIZE_MAX / page_size;
+
+	pages = parse_pages_arg(str, 0, max);
+	if (pages < 0) {
+		pr_err("Invalid argument for --out-pages/-O\n");
+		return -1;
+	}
+
+	if (pages == 0)
+		pr_debug("Reverting to write instead of mmap for output file\n");
+
+	*out_pages = pages;
+	return 0;
+}
+
 /**
  * perf_evlist__mmap - Create mmaps to receive events.
  * @evlist: list of events
tools/perf/util/evlist.h +3 -0
index ecaa582f40e2..749488147276 100644
@@ -107,6 +107,9 @@ int perf_evlist__prepare_workload(struct perf_evlist *evlist,
 				  bool want_signal);
 int perf_evlist__start_workload(struct perf_evlist *evlist);
 
+int perf_evlist__parse_out_pages(const struct option *opt,
+				  const char *str, int unset);
+
 int perf_evlist__parse_mmap_pages(const struct option *opt,
 				  const char *str,
 				  int unset);
Signature · 1 line
1.8.3.4 (Apple Git-47)
#1 lore raw

PATCH 1/5 perf record: Fix segfault with --no-mmap-pages

In reply to David Ahern

Adrian reported a segfault when using --no-out-pages:

$ tools/perf/perf record -vv --no-out-pages uname
Segmentation fault (core dumped)

The same occurs with --no-mmap-pages. Fix by checking that str is non-NULL before parsing it.

---
 tools/perf/util/evlist.c | 3 +++
 1 file changed, 3 insertions(+)
tools/perf/util/evlist.c +3 -0
index 99dc58e5dcc3..3960560f873a 100644
@@ -719,6 +719,9 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		{ .tag  = 0 },
 	};
 
+	if (str == NULL)
+		return -1;
+
 	val = parse_tag_value(str, tags);
 	if (val != (unsigned long) -1) {
 		/* we got file size value */
Signature · 1 line
1.8.3.4 (Apple Git-47)
maintainer
#8 lore raw
In reply to David Ahern
Quoted 7 lines · David Ahern

When recording raw_syscalls for the entire system, e.g.,
perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1

you end up with a negative feedback loop as perf itself calls write() fairly often. This patch handles the problem by mmap'ing the file in chunks of 64M at a time and copies events from the event buffers to the file avoiding write system calls.

You know this completely fails the moment you trace faults, because every new access to one of those pages (to mark it dirty) will trigger a fault. And we'll take a bunch more faults -- one for each page -- than we ever did write() syscalls.

Anyway the over all performance improvements still make it worth it. But the above seems like a false argument in favour of this.

#9 lore raw

Em Tue, Nov 12, 2013 at 03:57:07PM +0100, Peter Zijlstra escreveu:

Quoted 9 lines · David Ahern
Quoted 7 lines

When recording raw_syscalls for the entire system, e.g.,
perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1

you end up with a negative feedback loop as perf itself calls write() fairly often. This patch handles the problem by mmap'ing the file in chunks of 64M at a time and copies events from the event buffers to the file avoiding write system calls.

Quoted 4 lines

You know this completely fails the moment you trace faults, because every new access to one of those pages (to mark it dirty) will trigger a fault. And we'll take a bunch more faults -- one for each page -- than we ever did write() syscalls.

So we should provide a neon lettered warning when doing that, no? :-)

Quoted 2 lines

Anyway the over all performance improvements still make it worth it. But the above seems like a false argument in favour of this.

maintainer
#10 lore raw
Quoted 10 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 12, 2013 at 03:57:07PM +0100, Peter Zijlstra escreveu:

Quoted 9 lines · David Ahern
Quoted 7 lines

When recording raw_syscalls for the entire system, e.g.,
perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1

you end up with a negative feedback loop as perf itself calls write() fairly often. This patch handles the problem by mmap'ing the file in chunks of 64M at a time and copies events from the event buffers to the file avoiding write system calls.

Quoted 6 lines
Quoted 4 lines

You know this completely fails the moment you trace faults, because every new access to one of those pages (to mark it dirty) will trigger a fault. And we'll take a bunch more faults -- one for each page -- than we ever did write() syscalls.

So we should provide a neon lettered warning when doing that, no? :-)

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

#11 lore raw
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 12, 2013 at 03:57:07PM +0100, Peter Zijlstra escreveu:

Quoted 13 lines · David Ahern
Quoted 7 lines

When recording raw_syscalls for the entire system, e.g.,
perf record -e raw_syscalls:*,sched:sched_switch -a -- sleep 1

you end up with a negative feedback loop as perf itself calls write() fairly often. This patch handles the problem by mmap'ing the file in chunks of 64M at a time and copies events from the event buffers to the file avoiding write system calls.

You know this completely fails the moment you trace faults, because every new access to one of those pages (to mark it dirty) will trigger a fault. And we'll take a bunch more faults -- one for each page -- than we ever did write() syscalls.

Quoted 5 lines
Quoted 1 line

So we should provide a neon lettered warning when doing that, no? :-)

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

David

#12 lore raw
Quoted 2 lines · David Ahern
Quoted 2 lines

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

Which could be the case with call-graph recording, right?

Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Thanks,

Ingo

#40 lore raw

Re: [PATCH 5/5] perf record: Handle out of space failures writing data with mmap

In reply to David Ahern
Quoted 71 lines · David Ahern

If the filesystem where a file is written using mmap fills perf record gets a SIGBUS and terminated. Handle the SIGBUS by using longjmp to bounce out of the memcpy and fail the write.

Cc Ingo Molnar <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 tools/perf/builtin-record.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
tools/perf/builtin-record.c +12 -6
index 1a4fa5df215b..48d6535d144f 100644
@@ -29,9 +29,11 @@
 #include <unistd.h>
 #include <sched.h>
 #include <sys/mman.h>
+#include <setjmp.h>
 
 /* output file mmap'ed N chunks at a time */
 #define MMAP_OUTPUT_SIZE   (64*1024*1024)
+sigjmp_buf mmap_jmp;
 
 #ifndef HAVE_ON_EXIT_SUPPORT
 #ifndef ATEXIT_MAX
@@ -141,6 +143,7 @@ static int do_mmap_output(struct perf_record *rec, void *buf, size_t size)
 {
 	u64 remaining;
 	off_t offset;
+	volatile size_t total_len = 0;
 
 	if (rec->mmap.addr == NULL) {
 next_segment:
@@ -157,20 +160,23 @@ next_segment:
 	 * space write what we can then go back and create the
 	 * next segment
 	 */
-	if (size > remaining) {
-		memcpy(rec->mmap.addr + rec->mmap.offset, buf, remaining);
+	if (setjmp(mmap_jmp) != 0) {
+		pr_err("mmap copy failed.\n");
+		return -1;
+	}
+	if (size-total_len > remaining) {
+		memcpy(rec->mmap.addr + rec->mmap.offset, buf+total_len, remaining);
 		rec->bytes_written += remaining;
 
-		size -= remaining;
-		buf  += remaining;
+		total_len += remaining;
 
 		munmap(rec->mmap.addr, rec->mmap.out_size);
 		goto next_segment;
 	}
 
 	/* more data to copy and it fits in the current segment */
-	if (size) {
-		memcpy(rec->mmap.addr + rec->mmap.offset, buf, size);
+	if (size - total_len) {
+		memcpy(rec->mmap.addr + rec->mmap.offset, buf+total_len, size-total_len);
 		rec->bytes_written += size;
 		rec->mmap.offset += size;
 	}

tools/perf/builtin-record.c static void sig_handler(int sig) · line 282

Quoted 5 lines
tools/perf/builtin-record.c +2 -0
@@ -272,6 +278,9 @@ static void sig_handler(int sig)
 	if (sig == SIGCHLD)
 		child_finished = 1;
 
+	if (sig == SIGBUS)
+		longjmp(mmap_jmp, 1);

So this isn't very robust, because it assumes that all sources of SIGBUS are due to that memcpy() hitting -ENOSPC...

There are several failure modes:

- If mmap_jmp is not set yet and we get a SIGBUS is some other place, then the longjmp() result will be undefined.

- If mmap_jmp environment is set, but we've returned from do_mmap_output() already, then the result will be undefined - likely a non-obvious crash.

So at minimum we need a flag that tells us whether the jump environment is valid or not - i.e. whether we are executing inside the protected region or not - and only do the longjmp() if that flag is set.

Is there really no other way to handle the -ENOSPC case robustly? I guess not because the memcpy() really needs memory to write to, but I thought I'd ask ...

Thanks,

Ingo

#4 lore raw

[tip:perf/urgent] perf evlist: Round mmap pages to power 2 - v2

In reply to David Ahern
perf evlist: Round mmap pages to power 2 - v2
Commit-ID:  9639837e95db90d056f4683c911717921519320e
Gitweb:     http://git.kernel.org/tip/9639837e95db90d056f4683c911717921519320e
Author:     David Ahern <[email protected]>
AuthorDate: Tue, 12 Nov 2013 07:46:54 -0700
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 12 Nov 2013 16:31:53 -0300

perf evlist: Round mmap pages to power 2 - v2

Currently perf requires the -m / --mmap_pages option to be a power of 2.

To be more user friendly perf should automatically round this up to the next power of 2.

Currently:

  $ perf record -m 3 -a -- sleep 1
  --mmap_pages/-m value must be a power of two.sleep: Terminated

With patch:

  $ perf record -m 3 -a -- sleep 1
  rounding mmap pages size to 16384 (4 pages)
  ...

v2: Add bytes units to rounding message per Ingo's request. Other
suggestions (e.g., prefixing INFO) should be addressed by wrapping pr_info to catch all instances.

---
 tools/perf/util/evlist.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
tools/perf/util/evlist.c +6 -12
index 3960560..fb4727d 100644
@@ -710,7 +710,6 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 {
 	unsigned int *mmap_pages = opt->value;
 	unsigned long pages, val;
-	size_t size;
 	static struct parse_tag tags[] = {
 		{ .tag  = 'B', .mult = 1       },
 		{ .tag  = 'K', .mult = 1 << 10 },
@@ -726,11 +725,6 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 	if (val != (unsigned long) -1) {
 		/* we got file size value */
 		pages = PERF_ALIGN(val, page_size) / page_size;
-		if (pages < (1UL << 31) && !is_power_of_2(pages)) {
-			pages = next_pow2(pages);
-			pr_info("rounding mmap pages size to %lu (%lu pages)\n",
-				pages * page_size, pages);
-		}
 	} else {
 		/* we got pages count value */
 		char *eptr;
@@ -741,14 +735,14 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		}
 	}
 
-	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
-		pr_err("--mmap_pages/-m value too big\n");
-		return -1;
+	if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+		pages = next_pow2(pages);
+		pr_info("rounding mmap pages size to %lu bytes (%lu pages)\n",
+			pages * page_size, pages);
 	}
 
-	size = perf_evlist__mmap_size(pages);
-	if (!size) {
-		pr_err("--mmap_pages/-m value must be a power of two.");
+	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
+		pr_err("--mmap_pages/-m value too big\n");
 		return -1;
 	}
 
#6 lore raw

[tip:perf/urgent] perf evlist: Refactor mmap_pages parsing

In reply to David Ahern
perf evlist: Refactor mmap_pages parsing
Commit-ID:  33c2dcfdfe7f114cc656bcb4c839f5939d5e60ba
Gitweb:     http://git.kernel.org/tip/33c2dcfdfe7f114cc656bcb4c839f5939d5e60ba
Author:     David Ahern <[email protected]>
AuthorDate: Tue, 12 Nov 2013 07:46:55 -0700
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 12 Nov 2013 16:33:22 -0300

perf evlist: Refactor mmap_pages parsing

Logic will be re-used for the out-pages argument for mmap based writes in perf-record.

---
 tools/perf/util/evlist.c | 39 ++++++++++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 11 deletions(-)
tools/perf/util/evlist.c +28 -11
index fb4727d..cb19044 100644
@@ -705,10 +705,9 @@ static size_t perf_evlist__mmap_size(unsigned long pages)
 	return (pages + 1) * page_size;
 }
 
-int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
-				  int unset __maybe_unused)
+static long parse_pages_arg(const char *str, unsigned long min,
+			    unsigned long max)
 {
-	unsigned int *mmap_pages = opt->value;
 	unsigned long pages, val;
 	static struct parse_tag tags[] = {
 		{ .tag  = 'B', .mult = 1       },
@@ -719,7 +718,7 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 	};
 
 	if (str == NULL)
-		return -1;
+		return -EINVAL;
 
 	val = parse_tag_value(str, tags);
 	if (val != (unsigned long) -1) {
@@ -729,20 +728,38 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		/* we got pages count value */
 		char *eptr;
 		pages = strtoul(str, &eptr, 10);
-		if (*eptr != '\0') {
-			pr_err("failed to parse --mmap_pages/-m value\n");
-			return -1;
-		}
+		if (*eptr != '\0')
+			return -EINVAL;
 	}
 
-	if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+	if ((pages == 0) && (min == 0)) {
+		/* leave number of pages at 0 */
+	} else if (pages < (1UL << 31) && !is_power_of_2(pages)) {
+		/* round pages up to next power of 2 */
 		pages = next_pow2(pages);
 		pr_info("rounding mmap pages size to %lu bytes (%lu pages)\n",
 			pages * page_size, pages);
 	}
 
-	if (pages > UINT_MAX || pages > SIZE_MAX / page_size) {
-		pr_err("--mmap_pages/-m value too big\n");
+	if (pages > max)
+		return -EINVAL;
+
+	return pages;
+}
+
+int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
+				  int unset __maybe_unused)
+{
+	unsigned int *mmap_pages = opt->value;
+	unsigned long max = UINT_MAX;
+	long pages;
+
+	if (max < SIZE_MAX / page_size)
+		max = SIZE_MAX / page_size;
+
+	pages = parse_pages_arg(str, 1, max);
+	if (pages < 0) {
+		pr_err("Invalid argument for --mmap_pages/-m\n");
 		return -1;
 	}
 
#2 lore raw

[tip:perf/urgent] perf record: Fix segfault with --no-mmap-pages

In reply to David Ahern
perf record: Fix segfault with --no-mmap-pages
Commit-ID:  8973504be70b2986a2081eeff7d9a4210dec295d
Gitweb:     http://git.kernel.org/tip/8973504be70b2986a2081eeff7d9a4210dec295d
Author:     David Ahern <[email protected]>
AuthorDate: Tue, 12 Nov 2013 07:46:53 -0700
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 12 Nov 2013 16:30:54 -0300

perf record: Fix segfault with --no-mmap-pages

Adrian reported a segfault when using --no-out-pages:

$ tools/perf/perf record -vv --no-out-pages uname
Segmentation fault (core dumped)

The same occurs with --no-mmap-pages. Fix by checking that str is non-NULL before parsing it.

---
 tools/perf/util/evlist.c | 3 +++
 1 file changed, 3 insertions(+)
tools/perf/util/evlist.c +3 -0
index 99dc58e..3960560 100644
@@ -719,6 +719,9 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 		{ .tag  = 0 },
 	};
 
+	if (str == NULL)
+		return -1;
+
 	val = parse_tag_value(str, tags);
 	if (val != (unsigned long) -1) {
 		/* we got file size value */
maintainer
#13 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · David Ahern
Quoted 2 lines

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

Which could be the case with call-graph recording, right?

Not typically, I think we're limiting call graphs to 127 u64, which is ~1k. Maybe you can blow the single page if you also do a large top-of-stack copy for dwarf/unwind nonsense.

Quoted 5 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

You can always screw yourself over using this stuff, no exceptions.

#14 lore raw
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Ingo Molnar

* David Ahern <[email protected]> wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

Which could be the case with call-graph recording, right?

Not typically, I think we're limiting call graphs to 127 u64, which is ~1k. Maybe you can blow the single page if you also do a large top-of-stack copy for dwarf/unwind nonsense.

What I meant was dwarf style call graph recording:

tools/perf/builtin-record.c: const unsigned long default_stack_dump_size = 8192;

Doesn't that mean 8K+ events?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

That ought to be a far less common occurance than tracing page faults though.

Quoted 1 line

You can always screw yourself over using this stuff, no exceptions.

Granted, as the many notrace markings demonstrate this stuff really wants to observe itself observing itself all the time! :)

Thanks,

Ingo

maintainer
#15 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · David Ahern
Quoted 2 lines

Dunno.. it _should_ all work. Try it and see what it does.. Once the events are bigger than a page things might get 'interesting' though.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Which could be the case with call-graph recording, right?

Not typically, I think we're limiting call graphs to 127 u64, which is ~1k. Maybe you can blow the single page if you also do a large top-of-stack copy for dwarf/unwind nonsense.

What I meant was dwarf style call graph recording:

Ah, one of those things I've never yet used ;-)

Quoted 3 lines

tools/perf/builtin-record.c: const unsigned long default_stack_dump_size = 8192;

Doesn't that mean 8K+ events?

Oh yes.. that's ideal to trigger this.

#16 lore raw
In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

Quoted 7 lines

That ought to be a far less common occurance than tracing page faults though.

Quoted 1 line

You can always screw yourself over using this stuff, no exceptions.

Granted, as the many notrace markings demonstrate this stuff really wants to observe itself observing itself all the time! :)

perf can have options for output -- write(), mmap(), splice() -- and some logic of when not to use it:

1. don't use write for system wide tracing

2. don't use mmap for page fault tracing

3. don't use splice for .... (need a splice implementation)

and allow the user to override.

David

#41 lore raw

Re: [PATCH 5/5] perf record: Handle out of space failures writing data with mmap

In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar

So this isn't very robust, because it assumes that all sources of SIGBUS are due to that memcpy() hitting -ENOSPC...

There are several failure modes:

- If mmap_jmp is not set yet and we get a SIGBUS is some other place, then the longjmp() result will be undefined.

Quoted 7 lines

- If mmap_jmp environment is set, but we've returned from do_mmap_output() already, then the result will be undefined - likely a non-obvious crash.

So at minimum we need a flag that tells us whether the jump environment is valid or not - i.e. whether we are executing inside the protected region or not - and only do the longjmp() if that flag is set.

Right I meant to add that -- a flag to know when the jmp should be used. Got distracted.

Quoted 4 lines

Is there really no other way to handle the -ENOSPC case robustly? I guess not because the memcpy() really needs memory to write to, but I thought I'd ask ...

You need some means to interrupt memcpy and bounce out of it. longjmp is the only option I know of.

David

#17 lore raw
In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 5 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

You can always screw yourself over using this stuff, no exceptions.

What now? Can we add the mmap path as an option? Leave write by default and users can select mmap if they want? e.g., have out_size default to 0 and 'perf trace record' can set --out-pages to 64M. It bypasses all system calls at the expense of more page faults.

David

maintainer
#18 lore raw
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Quoted 7 lines
Quoted 5 lines

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

You can always screw yourself over using this stuff, no exceptions.

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

#19 lore raw
Quoted 20 lines · Peter Zijlstra
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Quoted 7 lines
Quoted 5 lines

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

You can always screw yourself over using this stuff, no exceptions.

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

Quoted 5 lines

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

Thanks,

Ingo

maintainer
#20 lore raw

Hi Ingo,

Quoted 25 lines · Ingo Molnar
Quoted 23 lines · Peter Zijlstra
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 3 lines

one option here is not allow page faults and system wide system calls. system wide tracing needs mmap; page faults for a task can use write(). I left that option in case something like this came up.

So maybe splice() sounds like the right long term solution after all? :-/

Quoted 7 lines
Quoted 5 lines

Right until you put a tracepoint (kprobe) somewhere in whatever function is used to transfer a single page into/from a splice pipe.

You can always screw yourself over using this stuff, no exceptions.

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Quoted 6 lines
Quoted 2 lines

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

We now use 4 khz sampling frequency by default, just FYI. :)

Thanks,
Namhyung

#21 lore raw
Quoted 9 lines · Namhyung Kim
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

We now use 4 khz sampling frequency by default, just FYI. :)

I think Peter is asking about:

    perf record -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

And as expected it is a massive feedback spiraling out of control.

David

maintainer
#22 lore raw

Hi David,

Quoted 21 lines · David Ahern
Quoted 20 lines · Namhyung Kim
Quoted 18 lines
Quoted 8 lines
Quoted 2 lines

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

Quoted 5 lines

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

Quoted 7 lines
Quoted 1 line

We now use 4 khz sampling frequency by default, just FYI. :)

I think Peter is asking about:

    perf record -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

And as expected it is a massive feedback spiraling out of control.

Ah, I missed that part - just blindly answered about the freq - thinking he's talking about the default freq of perf record/top.

Anyway, for above case, I guess it won't affect much as stack usually is in memory so no page fault will occur even recording with mmap unless the system suffers from a high memory pressure, right?

But I agree that copying 8KB for each sample seems too large.

Signature · 2 lines
Thanks,
Namhyung
maintainer
#23 lore raw
In reply to David Ahern
Quoted 17 lines · David Ahern
Quoted 16 lines · Namhyung Kim
Quoted 15 lines
Quoted 11 lines
Quoted 1 line

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

Quoted 6 lines
Quoted 1 line

We now use 4 khz sampling frequency by default, just FYI. :)

I think Peter is asking about:

    perf record -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

And as expected it is a massive feedback spiraling out of control.

How about adding an option to exclude the perf tools from recording for system-wide (or cpu-wide) session?

This way, we can prevent the feedback loops for page-fault or syscall events you mentioned IMHO.

Thanks,
Namhyung

maintainer
#24 lore raw
In reply to David Ahern
Quoted 13 lines · David Ahern
Quoted 12 lines · Namhyung Kim
Quoted 12 lines
Quoted 11 lines
Quoted 1 line

What now? Can we add the mmap path as an option?

I'd say an option is always a possibility, but someone please try what happens if you use stupid large events (dwarf stack copies) on PERF_COUNT_SW_PAGE_FAULTS (.period=1) while recording with mmap().

The other option is to simply disallow PERF_SAMPLE_STACK_USER for that event.

Personally I think 8k copies for every event are way stupid anyway, that's a metric ton of data at a huge cost.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

We now use 4 khz sampling frequency by default, just FYI. :)

I think Peter is asking about:

    perf record -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

I think it should be

  perf record -e cycles -F 4000 -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

(at least to generate the feedback spiral more efficiently..)

Well, I know that we don't support this now. But wouldn't it make sense to support this kind of thing?

Thanks,
Namhyung

#25 lore raw
Quoted 5 lines · Namhyung Kim

I think it should be

   perf record -e cycles -F 4000 -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

(at least to generate the feedback spiral more efficiently..)

you don't need the cycles. faults by itself works. Each event contains > 2 pages of data in the sample. With mmap-based output a single sample (1 page fault in any process) generates 2-3 page faults by perf which cause 2-3 >8k samples to be generated, which generates faults, ....

David

maintainer
#26 lore raw
Quoted 5 lines · David Ahern
Quoted 4 lines · Namhyung Kim

I think it should be

   perf record -e cycles -F 4000 -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1
Quoted 7 lines
Quoted 1 line

(at least to generate the feedback spiral more efficiently..)

you don't need the cycles. faults by itself works. Each event contains 2 pages of data in the sample. With mmap-based output a single sample (1 page fault in any process) generates 2-3 page faults by perf which cause 2-3 >8k samples to be generated, which generates faults,
....

But after perf touches all pages in ring-buffer and stack, it won't generate page-faults for itself anymore, right?

Hmm.. thinking it again, perf has all ring-buffer pages in memory when mmap() called, right? If so why not doing something like MAP_POPULATE so that it doesn't need to generate minor-faults?

Thanks,
Namhyung

#27 lore raw
Quoted 14 lines · Namhyung Kim
Quoted 5 lines · David Ahern
Quoted 4 lines · Namhyung Kim

I think it should be

    perf record -e cycles -F 4000 -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1
Quoted 7 lines
Quoted 1 line

(at least to generate the feedback spiral more efficiently..)

you don't need the cycles. faults by itself works. Each event contains 2 pages of data in the sample. With mmap-based output a single sample (1 page fault in any process) generates 2-3 page faults by perf which cause 2-3 >8k samples to be generated, which generates faults,
....

Quoted 6 lines

But after perf touches all pages in ring-buffer and stack, it won't generate page-faults for itself anymore, right?

Hmm.. thinking it again, perf has all ring-buffer pages in memory when mmap() called, right? If so why not doing something like MAP_POPULATE so that it doesn't need to generate minor-faults?

This is mmap'ed output, not the ring buffers or its stack. As the output file grows, new pages are needed and those are allocated on access via page faults. The ftruncate only extends the file size, it does not allocate pages at that time.

David

maintainer
#28 lore raw
Quoted 29 lines · David Ahern
Quoted 20 lines · Namhyung Kim
Quoted 8 lines · David Ahern
Quoted 6 lines · Namhyung Kim

I think it should be

    perf record -e cycles -F 4000 -e faults -c 1 --call-graph dwarf,8192

-a -- sleep 1

Quoted 8 lines
Quoted 1 line

(at least to generate the feedback spiral more efficiently..)

you don't need the cycles. faults by itself works. Each event contains 2 pages of data in the sample. With mmap-based output a single sample (1 page fault in any process) generates 2-3 page faults by perf which cause 2-3 >8k samples to be generated, which generates faults,
....

Quoted 6 lines

But after perf touches all pages in ring-buffer and stack, it won't generate page-faults for itself anymore, right?

Hmm.. thinking it again, perf has all ring-buffer pages in memory when mmap() called, right? If so why not doing something like MAP_POPULATE so that it doesn't need to generate minor-faults?

Quoted 4 lines

This is mmap'ed output, not the ring buffers or its stack. As the output file grows, new pages are needed and those are allocated on access via page faults. The ftruncate only extends the file size, it does not allocate pages at that time.

Argh, I was completely confused! ;-)

Please feel free to ignore what I said here..

Thanks,
Namhyung

#38 lore raw
In reply to Namhyung Kim
Quoted 5 lines · Namhyung Kim
Quoted 3 lines

Well, with 1 khz sampling of a single threaded workload it's 8MB per second - that's 80 MB for 10 seconds profiling - not the end of the world.

We now use 4 khz sampling frequency by default, just FYI. :)

Yes, but if someone samples with 1 khz that's still plenty enough for a long enough run.

With 4k sampling you coul do 2.5 seconds of profiling and still have 80 MB of data :-)

Thanks,

Ingo

#29 lore raw
In reply to David Ahern
Quoted 4 lines · David Ahern

This is mmap'ed output, not the ring buffers or its stack. As the output file grows, new pages are needed and those are allocated on access via page faults. The ftruncate only extends the file size, it does not allocate pages at that time.

Hm, doesn't MAP_POPULATE prefault pages in this case as well? Prefaulting would avoid the most obvious page fault driven feedback loops and it would probably be faster as well, because it avoids all the pagefaults ...

Thanks,

Ingo

maintainer
#30 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 4 lines · David Ahern

This is mmap'ed output, not the ring buffers or its stack. As the output file grows, new pages are needed and those are allocated on access via page faults. The ftruncate only extends the file size, it does not allocate pages at that time.

Quoted 4 lines

Hm, doesn't MAP_POPULATE prefault pages in this case as well? Prefaulting would avoid the most obvious page fault driven feedback loops and it would probably be faster as well, because it avoids all the pagefaults ...

MAP_POPULATE does indeed seem to write pre-fault for writable maps (mm/mlock.c:__mlock_vma_pages_range).

And that does indeed seem to side-step the perf sw pagefault event, but that is arguably a perf bug.

If we were to fix this, MAP_POPULATE would simply generate a metric ton of events, we'd have to write out with yet more MAP_POPULATE.

maintainer
#31 lore raw
Quoted 2 lines · Peter Zijlstra

And that does indeed seem to side-step the perf sw pagefault event, but that is arguably a perf bug.

To clarify; mm/memory.c:handle_mm_fault() is where the VM counts its generic PGFAULT event, but our perf sw event is in the arch fault handler.

So they count different but related things.

maintainer
#37 lore raw
In reply to David Ahern
Quoted 4 lines · David Ahern

I think Peter is asking about:

    perf record -e faults -c 1 --call-graph dwarf,8192 -a -- sleep 1

And as expected it is a massive feedback spiraling out of control.

I was and good to know, thanks for testing.

#32 lore raw
In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

And that does indeed seem to side-step the perf sw pagefault event, but that is arguably a perf bug.

To clarify; mm/memory.c:handle_mm_fault() is where the VM counts its generic PGFAULT event, but our perf sw event is in the arch fault handler.

So they count different but related things.

I think that assymetry was intended: we didn't want to count 'synchronous' pagefaults like get_user_pages() or mlock() bringing in pages, only asynchronous/real ones, or so.

Thanks,

Ingo

maintainer
#33 lore raw
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

And that does indeed seem to side-step the perf sw pagefault event, but that is arguably a perf bug.

To clarify; mm/memory.c:handle_mm_fault() is where the VM counts its generic PGFAULT event, but our perf sw event is in the arch fault handler.

Quoted 5 lines
Quoted 1 line

So they count different but related things.

I think that assymetry was intended: we didn't want to count 'synchronous' pagefaults like get_user_pages() or mlock() bringing in pages, only asynchronous/real ones, or so.

OK, I couldn't remember.

Anyway, I don't want to hold up this patch set, and the speedup in the 'normal' case is nice.

The only reason I reacted was because the changelog mentioned avoiding a feedback loop -- so I obviously had to point out that it didn't do such a thing, it only changed the details of the loop.

I'm fairly certain this particular problem is unavoidable, no matter what the mechanism used, you can always create feedback.

At some point people will just have to know wtf they're doing. Also, I don't think anything really bad happens, at worst we'll just drop a bunch of events because the output cannot keep up for obvious reasons.

#34 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

And that does indeed seem to side-step the perf sw pagefault event, but that is arguably a perf bug.

To clarify; mm/memory.c:handle_mm_fault() is where the VM counts its generic PGFAULT event, but our perf sw event is in the arch fault handler.

Quoted 5 lines
Quoted 1 line

So they count different but related things.

I think that assymetry was intended: we didn't want to count 'synchronous' pagefaults like get_user_pages() or mlock() bringing in pages, only asynchronous/real ones, or so.

Quoted 8 lines

OK, I couldn't remember.

Anyway, I don't want to hold up this patch set, and the speedup in the 'normal' case is nice.

The only reason I reacted was because the changelog mentioned avoiding a feedback loop -- so I obviously had to point out that it didn't do such a thing, it only changed the details of the loop.

So with MAP_POPULATE the 'feedback window' is moved entirely into the kernel (to within a single syscall) and is also reduced significantly, compared to a write() loop.

While you are right that it is not an elimination of the problem - yet it is still a significant reduction of its cross section surface in practice.

Quoted 2 lines

I'm fairly certain this particular problem is unavoidable, no matter what the mechanism used, you can always create feedback.

Well, we could exclude the profiling task itself from profiling events (just like ftrace and core bits of perf does it out of necessity), but I intentionally wanted to avoid that, to make sure we are honest and to make sure people don't tolerate profiling overhead that disturbs other workloads.

Thanks,

Ingo

#35 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

The only reason I reacted was because the changelog mentioned avoiding a feedback loop -- so I obviously had to point out that it didn't do such a thing, it only changed the details of the loop.

So with MAP_POPULATE the 'feedback window' is moved entirely into the kernel (to within a single syscall) and is also reduced significantly, compared to a write() loop.

As I understand it we have to use MAP_SHARED, not MAP_PRIVATE for files. So MAP_POPULATE does not work here. (And I tried to verify -- MAP_PRIVATE | MAP_POPULATE drops the feedback loop, but the file is 0's after the header).

Quoted 8 lines
Quoted 2 lines

I'm fairly certain this particular problem is unavoidable, no matter what the mechanism used, you can always create feedback.

Well, we could exclude the profiling task itself from profiling events (just like ftrace and core bits of perf does it out of necessity), but I intentionally wanted to avoid that, to make sure we are honest and to make sure people don't tolerate profiling overhead that disturbs other workloads.

Samples generated by perf itself need to be observable -- e.g. process scheduling I want to see the time consumed by the data collector itself and there are times when 'perf trace -- perf ...' is useful.

perf just needs options to do the right thing and stay out of its own way. Having a restriction that you can't do system wide collection of systems calls AND faults does not seem all that limiting.

David

#36 lore raw
Quoted 10 lines · David Ahern
Quoted 5 lines

Well, we could exclude the profiling task itself from profiling events (just like ftrace and core bits of perf does it out of necessity), but I intentionally wanted to avoid that, to make sure we are honest and to make sure people don't tolerate profiling overhead that disturbs other workloads.

Samples generated by perf itself need to be observable -- e.g. process scheduling I want to see the time consumed by the data collector itself and there are times when 'perf trace -- perf ...' is useful.

Absolutely agreed - a measurement instrument affects the measurement, and we must not try to hide that.

Still we can try to make the disturbance smaller and more managable.

For example if I have enough RAM it should be possible to run perf record with a 1 GB ring-buffer, and in that case as long as the perf.data is smaller than 1 GB there should be no writeout or any other IO activity until the measurement ends.

Thanks,

Ingo