Skip to contentSkip to conversation

Improvements for kallsyms__parse

PATCH In review Ian Rogers 2 participants 6 messages Apr 30, 2020 – May 1, 2020 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

cover Improvements for kallsyms__parse

kallsyms__parse is called 4 times during perf record startup. Add a benchmark to measure its performance. Transition it to using the api io.h buffered reading, improving performance by ~8% or saving ~5% from perf record start up time.

Ian Rogers (2):
perf bench: add kallsyms parsing
lib kallsyms: parse using io api

 tools/lib/api/io.h          |  3 ++
 tools/lib/symbol/kallsyms.c | 81 +++++++++++++++++++------------------
 tools/perf/bench/Build      |  1 +
 tools/perf/bench/bench.h    |  1 +
 tools/perf/builtin-bench.c  |  1 +
 5 files changed, 47 insertions(+), 40 deletions(-)
Signature · 1 line
2.26.2.526.g744177e7f7-goog
author reviewer
#1 lore raw

PATCH 1/1 perf bench: add kallsyms parsing

Add a benchmark for kallsyms parsing. Example output:

Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 103.971 ms (+- 0.121 ms)

---
 tools/perf/bench/Build            |  1 +
 tools/perf/bench/bench.h          |  1 +
 tools/perf/bench/kallsyms-parse.c | 75 +++++++++++++++++++++++++++++++
 tools/perf/builtin-bench.c        |  1 +
 4 files changed, 78 insertions(+)
 create mode 100644 tools/perf/bench/kallsyms-parse.c
tools/perf/bench/Build +1 -0
index 042827385c87..768e408757a0 100644
@@ -9,6 +9,7 @@ perf-y += futex-lock-pi.o
 perf-y += epoll-wait.o
 perf-y += epoll-ctl.o
 perf-y += synthesize.o
+perf-y += kallsyms-parse.o
 
 perf-$(CONFIG_X86_64) += mem-memcpy-x86-64-lib.o
 perf-$(CONFIG_X86_64) += mem-memcpy-x86-64-asm.o
tools/perf/bench/bench.h +1 -0
index 4d669c803237..61cae4966cae 100644
@@ -44,6 +44,7 @@ int bench_futex_lock_pi(int argc, const char **argv);
 int bench_epoll_wait(int argc, const char **argv);
 int bench_epoll_ctl(int argc, const char **argv);
 int bench_synthesize(int argc, const char **argv);
+int bench_kallsyms_parse(int argc, const char **argv);
 
 #define BENCH_FORMAT_DEFAULT_STR	"default"
 #define BENCH_FORMAT_DEFAULT		0
tools/perf/bench/kallsyms-parse.c new file +75 -0

75 lines not on this page · raw message

tools/perf/builtin-bench.c +1 -0
index 11c79a8d85d6..083273209c88 100644
@@ -78,6 +78,7 @@ static struct bench epoll_benchmarks[] = {
 
 static struct bench internals_benchmarks[] = {
 	{ "synthesize", "Benchmark perf event synthesis",	bench_synthesize	},
+	{ "kallsyms-parse", "Benchmark kallsyms parsing",	bench_kallsyms_parse	},
 	{ NULL,		NULL,					NULL			}
 };
 
Signature · 1 line
2.26.2.526.g744177e7f7-goog
author reviewer
#2 lore raw

PATCH 2/2 lib kallsyms: parse using io api

In reply to Ian Rogers

Perf record will call kallsyms__parse 4 times during startup and process megabytes of data. This changes kallsyms__parse to use the io library rather than fgets to improve performance of the user code by over 8%.

Before:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 103.988 ms (+- 0.203 ms)
After:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 95.571 ms (+- 0.006 ms)

For a workload like:

$ perf record /bin/true
Run under 'perf record -e cycles:u -g' the time goes from:
Before
30.10%     1.67%  perf     perf                [.] kallsyms__parse
After
25.55%    20.04%  perf     perf                [.] kallsyms__parse
So a little under 5% of the start-up time is removed. A lot of what
remains is on the kernel side, but caching kallsyms within perf would
at least impact memory footprint.
---
 tools/lib/api/io.h          |  3 ++
 tools/lib/symbol/kallsyms.c | 81 +++++++++++++++++++------------------
 2 files changed, 44 insertions(+), 40 deletions(-)
tools/lib/api/io.h +3 -0
index b7e55b5f8a4a..777c20f6b604 100644
@@ -7,6 +7,9 @@
 #ifndef __API_IO__
 #define __API_IO__
 
+#include <stdlib.h>
+#include <unistd.h>
+
 struct io {
 	/* File descriptor being read/ */
 	int fd;
tools/lib/symbol/kallsyms.c +41 -40

109 lines not on this page · raw message

Signature · 1 line
2.26.2.526.g744177e7f7-goog
#3 lore raw

Re: [PATCH 2/2] lib kallsyms: parse using io api

Quoted 12 lines · Ian Rogers

Perf record will call kallsyms__parse 4 times during startup and process megabytes of data. This changes kallsyms__parse to use the io library rather than fgets to improve performance of the user code by over 8%.

Before:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 103.988 ms (+- 0.203 ms)
After:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 95.571 ms (+- 0.006 ms)

For a workload like:

Quoted 9 lines
$ perf record /bin/true
Run under 'perf record -e cycles:u -g' the time goes from:
Before
30.10%     1.67%  perf     perf                [.] kallsyms__parse
After
25.55%    20.04%  perf     perf                [.] kallsyms__parse
So a little under 5% of the start-up time is removed. A lot of what
remains is on the kernel side, but caching kallsyms within perf would
at least impact memory footprint.

with your change I'm getting following warnings:

$ sudo ./perf record -a
Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.
Quoted 12 lines
---
 tools/lib/api/io.h          |  3 ++
 tools/lib/symbol/kallsyms.c | 81 +++++++++++++++++++------------------
 2 files changed, 44 insertions(+), 40 deletions(-)
tools/lib/api/io.h +0 -0
index b7e55b5f8a4a..777c20f6b604 100644

tools/lib/api/io.h line 11

Quoted 5 lines
tools/lib/api/io.h +2 -0
@@ -7,6 +7,9 @@
 #ifndef __API_IO__
 #define __API_IO__
 
+#include <stdlib.h>
+#include <unistd.h>

was this missing?

jirka

Quoted 4 lines
tools/lib/api/io.h +1 -0
+
 struct io {
 	/* File descriptor being read/ */
 	int fd;

SNIP

author reviewer
#4 lore raw

Re: [PATCH 2/2] lib kallsyms: parse using io api

Quoted 24 lines · Jiri Olsa
Quoted 12 lines · Ian Rogers

Perf record will call kallsyms__parse 4 times during startup and process megabytes of data. This changes kallsyms__parse to use the io library rather than fgets to improve performance of the user code by over 8%.

Before:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 103.988 ms (+- 0.203 ms)
After:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 95.571 ms (+- 0.006 ms)

For a workload like:

Quoted 9 lines
$ perf record /bin/true
Run under 'perf record -e cycles:u -g' the time goes from:
Before
30.10%     1.67%  perf     perf                [.] kallsyms__parse
After
25.55%    20.04%  perf     perf                [.] kallsyms__parse
So a little under 5% of the start-up time is removed. A lot of what
remains is on the kernel side, but caching kallsyms within perf would
at least impact memory footprint.
Quoted 6 lines

with your change I'm getting following warnings:

$ sudo ./perf record -a
Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.

I'll investigate, sorry in advance for sending this out too early.

Quoted 15 lines
Quoted 15 lines
---
 tools/lib/api/io.h          |  3 ++
 tools/lib/symbol/kallsyms.c | 81 +++++++++++++++++++------------------
 2 files changed, 44 insertions(+), 40 deletions(-)
tools/lib/api/io.h +0 -0
index b7e55b5f8a4a..777c20f6b604 100644
@@ -7,6 +7,9 @@
 #ifndef __API_IO__
 #define __API_IO__
 
Quoted 4 lines

tools/lib/api/io.h line 11

Quoted 2 lines
tools/lib/api/io.h +2 -0
+#include <stdlib.h>
+#include <unistd.h>

was this missing?

Yes, they were getting picked up by a transitive #include in synthesize-events.c, but given the call to read and use of size_t are here it makes sense for the #includes to be here.

Thanks,
Ian

Quoted 9 lines

jirka

Quoted 4 lines
tools/lib/api/io.h +1 -0
+
 struct io {
      /* File descriptor being read/ */
      int fd;

SNIP

author reviewer
#5 lore raw

Re: [PATCH 2/2] lib kallsyms: parse using io api

Quoted 26 lines · Ian Rogers
Quoted 24 lines · Jiri Olsa
Quoted 21 lines · Ian Rogers

Perf record will call kallsyms__parse 4 times during startup and process megabytes of data. This changes kallsyms__parse to use the io library rather than fgets to improve performance of the user code by over 8%.

Before:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 103.988 ms (+- 0.203 ms)
After:
Running 'internals/kallsyms-parse' benchmark:
Average kallsyms__parse took: 95.571 ms (+- 0.006 ms)

For a workload like:

$ perf record /bin/true
Run under 'perf record -e cycles:u -g' the time goes from:
Before
30.10%     1.67%  perf     perf                [.] kallsyms__parse
After
25.55%    20.04%  perf     perf                [.] kallsyms__parse
So a little under 5% of the start-up time is removed. A lot of what
remains is on the kernel side, but caching kallsyms within perf would
at least impact memory footprint.
Quoted 8 lines
Quoted 6 lines

with your change I'm getting following warnings:

$ sudo ./perf record -a
Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.

I'll investigate, sorry in advance for sending this out too early.

It was an issue with the result in the case of success, it should be the last result of the function pointer and not 0. Sorry about not spotting that. Fixed in:
https://lore.kernel.org/lkml/[email protected]/T/#md59bc11419ca5adc8ab29fa3460288329e8dca91

Thanks,
Ian

Quoted 36 lines
Quoted 15 lines
Quoted 15 lines
---
 tools/lib/api/io.h          |  3 ++
 tools/lib/symbol/kallsyms.c | 81 +++++++++++++++++++------------------
 2 files changed, 44 insertions(+), 40 deletions(-)
tools/lib/api/io.h +0 -0
index b7e55b5f8a4a..777c20f6b604 100644
@@ -7,6 +7,9 @@
 #ifndef __API_IO__
 #define __API_IO__
 
Quoted 4 lines

tools/lib/api/io.h line 11

Quoted 2 lines
tools/lib/api/io.h +2 -0
+#include <stdlib.h>
+#include <unistd.h>

was this missing?

Yes, they were getting picked up by a transitive #include in synthesize-events.c, but given the call to read and use of size_t are here it makes sense for the #includes to be here.

Thanks,
Ian

Quoted 9 lines

jirka

Quoted 4 lines
tools/lib/api/io.h +1 -0
+
 struct io {
      /* File descriptor being read/ */
      int fd;

SNIP