perf report: Add 'tgid' sort key
Namhyung Kim
maintainerPATCH 1/1 perf report: Add 'tgid' sort key
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloop
$ perf report --stdio -s tgid,pid -H
...
#
# Overhead Tgid:Command / Pid:Command
# ........... ..........................
#
100.00% 2018407:perf
50.34% 2018407:perf
49.66% 2018409:perf--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)
tools/perf/Documentation/perf-report.txt +1 -0
@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sample
tools/perf/util/hist.h +1 -0
@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,
tools/perf/util/sort.c +35 -0
@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t
@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),
tools/perf/util/sort.h +1 -0
@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK,
Signature · 1 line
2.48.1.502.g6dc24dfdaf-goog
Quoted 20 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloop$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perf
Quoted 2 lines
Suggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>
Thanks,
Ian
Arnaldo Carvalho de Melo
maintainerQuoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloop
Unrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#
I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next.. 9de1ed6fa3b73cb1 (HEAD -> perf-tools-next) perf report: Add 'tgid' sort key 23e98ede2a353530 perf trace: Add --summary-mode optione6d6104625a3790b perf tools: Get rid of now-unused rb_resort.h 173ec14e72ef4ed7 perf trace: Convert syscall_stats to hashmap 66edfb5d404e743d perf trace: Allocate syscall stats only if summary is on ca6637e1ea08e6f4 perf parse-events filter: Use evsel__find_pmu() bd1ac4a678f7f2c8 perf bench evlist-open-close: Reduce scope of 2 variables cd59081880e89df8 perf test: Add direct off-cpu test56cbd794c0c46ba9 perf record --off-cpu: Add --off-cpu-thresh option 28d9b19c5455556f perf record --off-cpu: Dump the remaining samples in BPF's stack trace map 2bc05b02743b50a7 perf script: Display off-cpu samples correctly bfa457a621596947 perf record --off-cpu: Disable perf_event's callchain collection eca732cc42d20266 perf evsel: Assemble offcpu samples74ce50e40c569e90 perf record --off-cpu: Dump off-cpu samples in BPF e75f8ce63bfa6cb9 perf record --off-cpu: Preparation of off-cpu BPF program 0ffab9d26971c91c perf record --off-cpu: Parse off-cpu eventefc3fe2070853b7d perf evsel: Expose evsel__is_offcpu_event() for future use ⬢ [acme@toolbox perf-tools-next]$
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
- Arnaldo
Quoted 110 lines
$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perfSuggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)tools/perf/Documentation/perf-report.txt +1
-0@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sampletools/perf/util/hist.h +1
-0@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,tools/perf/util/sort.c +35
-0@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),tools/perf/util/sort.h +1
-0@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK,Signature · 1 line
2.48.1.502.g6dc24dfdaf-goog
Arnaldo Carvalho de Melo
maintainerQuoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$
Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread 0x7fffdf24c6c0 (LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread 0x7fffdf24c6c0 (LWP 580622) exited]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file /usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)",
file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95,
function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95
#7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538
#8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176
#9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238#10 0x00000000006388af in perf_session__destroy_kernel_maps (session=0xf97f60) at util/session.c:105 #11 0x0000000000638df0 in perf_session__delete (session=0xf97f60) at util/session.c:248
#12 0x0000000000431f18 in __cmd_record (rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #13 0x00000000004351fb in cmd_record (argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#14 0x00000000004bd4d4 in run_builtin (p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #15 0x00000000004bd77b in handle_internal_command (argc=6, argv=0x7fffffffde60) at perf.c:404
#16 0x00000000004bd8d4 in run_argv (argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#17 0x00000000004bdc1d in main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)
Quoted 112 lines
- Arnaldo
Quoted 110 lines
$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perfSuggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)tools/perf/Documentation/perf-report.txt +1
-0@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sampletools/perf/util/hist.h +1
-0@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,tools/perf/util/sort.c +35
-0@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),tools/perf/util/sort.h +1
-0@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK,Signature · 1 line
2.48.1.502.g6dc24dfdaf-goog
Ian Rogers
reviewerQuoted 78 lines · Arnaldo Carvalho de Melo
Quoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread
0x7fffdf24c6c0(LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread0x7fffdf24c6c0(LWP 580622) exited]
perf:util/maps.c:95:check_invariants: Assertion `map__end(prev) <=map__end(map)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file/usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238
Quoted 9 lines
#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #130x00000000004351fbincmd_record(argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#140x00000000004bd4d4inrun_builtin(p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #150x00000000004bd77binhandle_internal_command(argc=6, argv=0x7fffffffde60) at perf.c:404
#160x00000000004bd8d4inrun_argv(argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#170x00000000004bdc1din main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)
So my guess would be that something modified a map and broke the invariants of the maps_by_addresss/maps_by_name. It should be possible to add more check_invariants to work out where this happens.
Thanks,
Ian
Quoted 112 lines
Quoted 112 lines
- Arnaldo
Quoted 110 lines
$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perfSuggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)tools/perf/Documentation/perf-report.txt +1
-0@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sampletools/perf/util/hist.h +1
-0@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,tools/perf/util/sort.c +35
-0@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),tools/perf/util/sort.h +1
-1@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK, --2.48.1.502.g6dc24dfdaf-goog
Ian Rogers
reviewerQuoted 91 lines · Ian Rogers
Quoted 78 lines · Arnaldo Carvalho de Melo
Quoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread
0x7fffdf24c6c0(LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread0x7fffdf24c6c0(LWP 580622) exited]
perf:util/maps.c:95:check_invariants: Assertion `map__end(prev) <=map__end(map)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file/usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238Quoted 9 lines
#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #130x00000000004351fbincmd_record(argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#140x00000000004bd4d4inrun_builtin(p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #150x00000000004bd77binhandle_internal_command(argc=6, argv=0x7fffffffde60) at perf.c:404
#160x00000000004bd8d4inrun_argv(argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#170x00000000004bdc1din main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)
Quoted 6 lines
So my guess would be that something modified a map and broke the invariants of the maps_by_addresss/maps_by_name. It should be possible to add more
check_invariantsto work out where this happens.Thanks,
Ian
I also suspect this is a regression. If you could bisect to find the cause then the fix is probably to not modify a map but clone it, change it and then reinsert it into the maps - the insert is called maps__fixup_overlap_and_insert so that maps don't overlap one another like the invariant check is detecting. Fwiw, in the older rbtree code, invariant breakages like this would be silently ignored, so we may have a latent bug :-(
Thanks,
Ian
Quoted 112 lines
Quoted 112 lines
Quoted 112 lines
- Arnaldo
Quoted 110 lines
$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perfSuggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)tools/perf/Documentation/perf-report.txt +1
-0@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sampletools/perf/util/hist.h +1
-0@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,tools/perf/util/sort.c +35
-0@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),tools/perf/util/sort.h +1
-1@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK, --2.48.1.502.g6dc24dfdaf-goog
Namhyung Kim
maintainerQuoted 99 lines · Ian Rogers
Quoted 91 lines · Ian Rogers
Quoted 78 lines · Arnaldo Carvalho de Melo
Quoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread
0x7fffdf24c6c0(LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread0x7fffdf24c6c0(LWP 580622) exited]
perf:util/maps.c:95:check_invariants: Assertion `map__end(prev) <=map__end(map)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file/usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238Quoted 9 lines
#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #130x00000000004351fbincmd_record(argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#140x00000000004bd4d4inrun_builtin(p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #150x00000000004bd77binhandle_internal_command(argc=6, argv=0x7fffffffde60) at perf.c:404
#160x00000000004bd8d4inrun_argv(argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#170x00000000004bdc1din main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)Quoted 6 lines
So my guess would be that something modified a map and broke the invariants of the maps_by_addresss/maps_by_name. It should be possible to add more
check_invariantsto work out where this happens.Thanks,
Ian
Quoted 7 lines
I also suspect this is a regression. If you could bisect to find the cause then the fix is probably to not modify a map but clone it, change it and then reinsert it into the maps - the insert is called
maps__fixup_overlap_and_insertso that maps don't overlap one another like the invariant check is detecting. Fwiw, in the older rbtree code, invariant breakages like this would be silently ignored, so we may have a latent bug :-(
Sorry, I also cannot reproduce it on my machine. But I think it's unrelated to this change since you saw it during record. It'd be nice if you could bisect.
And I think I forgot to call thread__put() for the leader thread. :) Will update in v2.
Thanks,
Namhyung
Quoted 89 lines · Ian Rogers
Quoted 87 lines · Ian Rogers
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread
0x7fffdf24c6c0(LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread0x7fffdf24c6c0(LWP 580622) exited]
perf:util/maps.c:95:check_invariants: Assertion `map__end(prev) <=map__end(map)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file/usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #130x00000000004351fbincmd_record(argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#140x00000000004bd4d4inrun_builtin(p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #150x00000000004bd77binhandle_internal_command(argc=6, argv=0x7fffffffde60) at perf.c:404
#160x00000000004bd8d4inrun_argv(argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#170x00000000004bdc1din main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)
Quoted 5 lines
Quoted 3 lines
So my guess would be that something modified a map and broke the invariants of the maps_by_addresss/maps_by_name. It should be possible to add more
check_invariantsto work out where this happens.I also suspect this is a regression. If you could bisect to find the
I bisected it to:
⬢ [acme@toolbox perf-tools-next]$ git bisect good 876e80cf83d10585df6ee1e353cfbf562f9a930e is the first bad commit commit 876e80cf83d10585df6ee1e353cfbf562f9a930e
Author: Namhyung Kim <[email protected]> Date: Wed Dec 18 14:04:53 2024 -0800
perf tools: Fixup end address of modules
In machine__create_module(), it reads /proc/modules to get a list of modules in the system. The file shows the start address (of text) and the size of the module so it uses the info to reconstruct system memory maps for symbol resolution.
But module memory consists of multiple segments and they can be scaterred. Currently perf tools assume they are contiguous and see some overlaps. This can confuse the tool when it finds a map containing a given address.
As we mostly care about the function symbols in the text segment, it can fixup the size or end address of modules when there's an overlap. We can use maps__fixup_end() which updates the end address using the start address of the next map.
Ideally it should be able to track other segments (like data/rodata), but that would require some changes in /proc/modules IMHO.
tools/perf/util/machine.c | 2 ++ 1 file changed, 2 insertions(+)
⬢ [acme@toolbox perf-tools-next]$
If we simply revert this it gets back working:
⬢ [acme@toolbox perf-tools-next]$ git revert 876e80cf83d10585df6ee1e353cfbf562f9a930e Auto-merging tools/perf/util/machine.c
[perf-tools-next 1ab31115859a0944] Revert "perf tools: Fixup end address of modules"
1 file changed, 2 deletions(-) # rm -rf build dir, rebuild it
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.403 MB perf.data (7960 samples) ] root@number:~#
No time today to try to dig deeper, so just reporting the bisection result.
- Arnaldo
Quoted 122 lines
cause then the fix is probably to not modify a map but clone it, change it and then reinsert it into the maps - the insert is called
maps__fixup_overlap_and_insertso that maps don't overlap one another like the invariant check is detecting. Fwiw, in the older rbtree code, invariant breakages like this would be silently ignored, so we may have a latent bug :-(Thanks,
IanQuoted 112 lines
Quoted 112 lines
Quoted 112 lines
- Arnaldo
Quoted 110 lines
$ perf report --stdio -s tgid,pid -H ... # # Overhead Tgid:Command / Pid:Command # ........... .......................... # 100.00% 2018407:perf 50.34% 2018407:perf 49.66% 2018409:perfSuggested-by Stephane Eranian <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 35 ++++++++++++++++++++++++ tools/perf/util/sort.h | 1 + 4 files changed, 38 insertions(+)tools/perf/Documentation/perf-report.txt +1
-0@@ -79,6 +79,7 @@ OPTIONS - comm: command (name) of the task which can be read via /proc/<pid>/comm - pid: command and tid of the task + - tgid: command and tgid of the task - dso: name of library or module executed at the time of sample - dso_size: size of library or module executed at the time of sample - symbol: name of function executed at the time of sampletools/perf/util/hist.h +1
-0@@ -38,6 +38,7 @@ enum hist_column {HISTC_TIME ,HISTC_DSO ,HISTC_THREAD , +HISTC_TGID ,HISTC_COMM ,HISTC_CGROUP_ID , HISTC_CGROUP,tools/perf/util/sort.c +35
-0@@ -141,6 +141,40 @@ struct sort_entry sort_thread = { .se_width_idx = HISTC_THREAD, }; +/* --sort tgid */ + +static int64_t +sort__tgid_cmp (struct hist_entry *left,struct hist_entry *right) +{ +return thread__pid (right->thread) -thread__pid (left->thread); +} + +static int hist_entry__tgid_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +int tgid =thread__pid (he->thread); +const char *comm =NULL ; + +if (thread__pid (he->thread) ==thread__tid (he->thread)) { + comm =thread__comm_str (he->thread); + }else { +struct maps *maps =thread__maps (he->thread); +struct thread *leader =machine__find_thread (maps__machine (maps), + tgid, tgid); +if (leader) + comm =thread__comm_str (leader); + } + width =max (7U , width) -8 ; +return repsep_snprintf (bf, size,"%7d:%-*.*s" , tgid, width, width, comm ?:"" ); +} + +struct sort_entry sort_tgid = { + .se_header =" Tgid:Command" , + .se_cmp = sort__tgid_cmp, + .se_snprintf = hist_entry__tgid_snprintf, + .se_width_idx =HISTC_TGID , +}; +/* --sort simd */ static int64_t@@ -2501,6 +2535,7 @@ static void sort_dimension_add_dynamic_header(struct sort_dimension *sd)static struct sort_dimension common_sort_dimensions[] = {DIM (SORT_PID ,"pid" , sort_thread), +DIM (SORT_TGID ,"tgid" , sort_tgid),DIM (SORT_COMM ,"comm" , sort_comm),DIM (SORT_DSO ,"dso" , sort_dso),DIM (SORT_SYM ,"symbol" , sort_sym),tools/perf/util/sort.h +1
-1@@ -72,6 +72,7 @@ enum sort_type {SORT_ANNOTATE_DATA_TYPE_OFFSET ,SORT_SYM_OFFSET ,SORT_ANNOTATE_DATA_TYPE_CACHELINE , +SORT_TGID ,/* branch stack specific sort keys */ __SORT_BRANCH_STACK, --2.48.1.502.g6dc24dfdaf-goog
Arnaldo Carvalho de Melo
maintainerQuoted 151 lines · Arnaldo Carvalho de Melo
Quoted 89 lines · Ian Rogers
Quoted 87 lines · Ian Rogers
Quoted 86 lines · Arnaldo Carvalho de Melo
Quoted 47 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Namhyung Kim
Sometimes we need to analyze the data in process level but current sort keys only work on thread level. Let's add 'tgid' sort key for that as 'pid' is already taken for thread.
This will look mostly the same, but it only uses tgid instead of tid. Here's an example of a process with two threads (thloop).
$ perf record -- perf test -w thloopUnrelated, but when building perf with DEBUG=1 and trying to test the above I noticed:
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.404 MB perf.data (7968 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~# perf record -- perf test -w offcpu [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.040 MB perf.data (23 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. Aborted (core dumped) root@number:~#I have:
⬢ [acme@toolbox perf-tools-next]$ git log --oneline perf-tools-next/perf-tools-next..
9de1ed6fa3b73cb1(HEAD -> perf-tools-next) perf report: Add 'tgid' sort key23e98ede2a353530perf trace: Add --summary-mode optione6d6104625a3790bperf tools: Get rid of now-unusedrb_resort.h173ec14e72ef4ed7perf trace: Convertsyscall_statsto hashmap66edfb5d404e743dperf trace: Allocate syscall stats only if summary is onca6637e1ea08e6f4perf parse-events filter: Useevsel__find_pmu()bd1ac4a678f7f2c8perf bench evlist-open-close: Reduce scope of 2 variablescd59081880e89df8perf test: Add direct off-cpu test56cbd794c0c46ba9perf record --off-cpu: Add --off-cpu-thresh option28d9b19c5455556fperf record --off-cpu: Dump the remaining samples in BPF's stack trace map2bc05b02743b50a7perf script: Display off-cpu samples correctlybfa457a621596947perf record --off-cpu: Disable perf_event's callchain collectioneca732cc42d20266perf evsel: Assemble offcpu samples74ce50e40c569e90perf record --off-cpu: Dump off-cpu samples in BPFe75f8ce63bfa6cb9perf record --off-cpu: Preparation of off-cpu BPF program0ffab9d26971c91cperf record --off-cpu: Parse off-cpu eventefc3fe2070853b7dperf evsel: Exposeevsel__is_offcpu_event()for future use ⬢ [acme@toolbox perf-tools-next]$Quoted 3 lines
locally, that is the stuff I've been testing lately, doubt it is related to these patches, I'll investigate later, have to go AFK, so FWIW as a heads up.
Had time to extract this, now going really AFK:
[New Thread
0x7fffdf24c6c0(LWP 580622)]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.403 MB perf.data (7948 samples) ] [Thread0x7fffdf24c6c0(LWP 580622) exited]
perf:util/maps.c:95:check_invariants: Assertion `map__end(prev) <=map__end(map)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
Downloading 4.06 K source file/usr/src/debug/glibc-2.39-37.fc40.x86_64/nptl/pthread_kill.c__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev) <= map__end(map)", file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888 #130x00000000004351fbincmd_record(argc=4, argv=0x7fffffffde60) at builtin-record.c:4286
#140x00000000004bd4d4inrun_builtin(p=0xecddc0 <commands+288>, argc=6, argv=0x7fffffffde60) at perf.c:351 #150x00000000004bd77binhandle_internal_command(argc=6, argv=0x7fffffffde60) at perf.c:404
#160x00000000004bd8d4inrun_argv(argcp=0x7fffffffdc4c, argv=0x7fffffffdc40) at perf.c:448
#170x00000000004bdc1din main (argc=6, argv=0x7fffffffde60) at perf.c:556
(gdb)Quoted 5 lines
Quoted 3 lines
So my guess would be that something modified a map and broke the invariants of the maps_by_addresss/maps_by_name. It should be possible to add more
check_invariantsto work out where this happens.I also suspect this is a regression. If you could bisect to find the
I bisected it to:
⬢ [acme@toolbox perf-tools-next]$ git bisect good
876e80cf83d10585df6ee1e353cfbf562f9a930eis the first bad commit commit876e80cf83d10585df6ee1e353cfbf562f9a930eAuthor: Namhyung Kim <[email protected]> Date: Wed Dec 18 14:04:53 2024 -0800perf tools: Fixup end address of modules
In
machine__create_module(), it reads/proc/modulesto get a list of modules in the system. The file shows the start address (of text) and the size of the module so it uses the info to reconstruct system memory maps for symbol resolution.But module memory consists of multiple segments and they can be scaterred. Currently perf tools assume they are contiguous and see some overlaps. This can confuse the tool when it finds a map containing a given address.
As we mostly care about the function symbols in the text segment, it can fixup the size or end address of modules when there's an overlap. We can use
maps__fixup_end()which updates the end address using the start address of the next map.Ideally it should be able to track other segments (like data/rodata), but that would require some changes in
/proc/modulesIMHO.Reported-by Blake Jones <[email protected]>Signed-off-by Namhyung Kim <[email protected]>Acked-by Ian Rogers <[email protected]>Cc Adrian Hunter <[email protected]>Cc Daniel Gomez <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jiri Olsa <[email protected]>Cc Kan Liang <[email protected]>Cc Luis Chamberlain <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Petr Pavlu <[email protected]>Cc Sami Tolvanen <[email protected]>Signed-off-by Arnaldo Carvalho de Melo <[email protected]>tools/perf/util/machine.c | 2 ++ 1 file changed, 2 insertions(+)⬢ [acme@toolbox perf-tools-next]$
If we simply revert this it gets back working:
⬢ [acme@toolbox perf-tools-next]$ git revert
876e80cf83d10585df6ee1e353cfbf562f9a930eAuto-mergingtools/perf/util/machine.c
[perf-tools-next1ab31115859a0944] Revert "perf tools: Fixup end address of modules"1 file changed, 2 deletions(-) # rm -rf build dir, rebuild it
Quoted 7 lines
root@number:~# perf record -- perf test -w thloop [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.403 MB perf.data (7960 samples) ] root@number:~#No time today to try to dig deeper, so just reporting the bisection result.
So, using 'perf probe' + 'perf trace' to trace 'perf record':
root@number:~# perf probe -x ~/bin/perf maps__insert maps 'map->dso->name:string' Target program is compiled without optimization. Skipping prologue. Probe on address 0x634e4e to force probing at the function entry.
Added new event:
probe_perf:maps_insert (on maps__insert in /home/acme/bin/perf with maps name=map->dso->name:string)
You can now use it in all perf tools, such as:
perf record -e probe_perf:maps_insert -aR sleep 1
root@number:~# perf probe -x ~/bin/perf maps__fixup_end maps Target program is compiled without optimization. Skipping prologue. Probe on address 0x636c14 to force probing at the function entry.
Added new event:
probe_perf:maps_fixup_end (on maps__fixup_end in /home/acme/bin/perf with maps)
You can now use it in all perf tools, such as:
perf record -e probe_perf:maps_fixup_end -aR sleep 1
root@number:~# perf probe -l
probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps)
probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name)
root@number:~# perf trace -e probe_perf:maps* perf record sleep 1
0.000 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[kernel.kallsyms]")
0.040 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[overlay]")
0.053 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[rfcomm]")
<SNIP>
2.736 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[nvme_auth]")
2.757 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[video]")
2.773 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[wmi]")
2.789 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[pinctrl_alderlake]")
2.804 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[ip6_tables]")
2.821 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[ip_tables]")
2.838 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[fuse]")
66.799 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "[kernel.kallsyms]")
192.465 perf/1346602 probe_perf:maps_fixup_end(__probe_ip: 6515745, maps: 155203024)
[ perf record: Woken up 1 times to write data ]
1327.967 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_e8932b6bae2b9745_restrict_filesystems")
1328.015 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_40ddf486530245f5_sd_devices")
1328.025 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
<SNIP>
1328.296 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
1328.308 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_ingress")
1328.320 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_egress")
1328.331 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_6deef7357e7b4530_sd_fw_ingress")
1328.343 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_prog_be31ae23198a0378_sd_devices")
1328.354 perf/1346602 probe_perf:maps_insert(__probe_ip: 6508126, maps: 155203024, name: "bpf_trampoline_6442522522")
[ perf record: Captured and wrote 0.036 MB perf.data (19 samples) ]
perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
root@number:~#So the call to maps_fixup_end() will set maps->end_broken to false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:
#6 0x0000000000633d52 in check_invariants (maps=0xf967c0) at util/maps.c:95
95 assert(map__end(prev) <= map__end(map));
(gdb) p prev->dso->name
$1 = 0xfc47ab "bpf_trampoline_6442522522"
(gdb) p map->dso->name
$2 = 0xfe5dcb "bpf_prog_40ddf486530245f5_sd_devices"
(gdb) p /x prev->start
$3 = 0xffffffffc0147640
(gdb) p /x map->start
$4 = 0xffffffffc014774c
(gdb) p /x prev->end
$5 = 0xffffffffc0148640
(gdb) p /x map->end
$6 = 0xffffffffc014788b
(gdb) p prev->start < map->start
$7 = 1
(gdb) p prev->end < map->start
$8 = 0
(gdb)
If we ask for backtraces we see where maps are added after fixup_end:
59.088 perf/1448114 probe_perf:maps_insert(__probe_ip: 6508126, maps: 267654608, name: "[kernel.kallsyms]")
maps__insert (/home/acme/bin/perf)
machine__update_kernel_mmap (/home/acme/bin/perf)
machine__create_kernel_maps (/home/acme/bin/perf)
perf_session__create_kernel_maps (/home/acme/bin/perf)
__perf_session__new (/home/acme/bin/perf)
perf_session__new (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
185.548 perf/1448114 probe_perf:maps_fixup_end(__probe_ip: 6515745, maps: 267654608)
maps__fixup_end (/home/acme/bin/perf)
machine__create_kernel_maps (/home/acme/bin/perf)
perf_session__create_kernel_maps (/home/acme/bin/perf)
__perf_session__new (/home/acme/bin/perf)
perf_session__new (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)sleep: missing operand
Try 'sleep --help' for more information.
[ perf record: Woken up 1 times to write data ]
320.675 perf/1448114 probe_perf:maps_insert(__probe_ip: 6508126, maps: 267654608, name: "bpf_prog_e8932b6bae2b9745_restrict_filesystems")
maps__insert (/home/acme/bin/perf)
machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf)
perf_event__process_ksymbol (/home/acme/bin/perf)
machines__deliver_event (/home/acme/bin/perf)
perf_session__deliver_event (/home/acme/bin/perf)
perf_session__process_event (/home/acme/bin/perf)
process_simple (/home/acme/bin/perf)
Now looking at machine__process_ksymbol_register()...
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"
So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#
root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#
So a PERF_RECORD_KSYMBOL processing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register() does:
713 map__set_start(map, event->ksymbol.addr);
714 map__set_end(map, map__start(map) + event->ksymbol.len);
715 err = maps__insert(machine__kernel_maps(machine), map);
And:
(gdb) p /x event->ksymbol.addr
$2 = 0xffffffffc0147a2c
(gdb) p event->ksymbol.len
$3 = 306
Thread 1 "perf" hit Breakpoint 1, machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
688 {
(gdb) bt
#0 machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688
#1 0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779
#2 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296
#3 0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392,
file_path=0xf96850 "perf.data") at util/session.c:1334
#4 0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data")
at util/session.c:1367
#5 0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626
#6 0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203
#7 0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132
#8 0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181
#9 0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226#10 0x000000000063e988 in perf_session__process_events (session=0xf95f40) at util/session.c:2390 #11 0x000000000042d98b in process_buildids (rec=0xec8ce0 <record>) at builtin-record.c:1475
#12 0x000000000042e963 in record__finish_output (rec=0xec8ce0 <record>) at builtin-record.c:1798 #13 0x0000000000431c46 in __cmd_record (rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841 #14 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#15 0x00000000004bcf65 in run_builtin (p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351 #16 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#17 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#18 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
So, this one liner "refixes" the "modules" ends when processing the records to find the build ids, unsure if it is the best solution tho:
tools/perf/builtin-record.c +2 -0
@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)if (!rec->no_buildid) {process_buildids (rec); +maps__fixup_end (machine__kernel_maps (&rec->session->machines.host)); +if (rec->buildid_all)perf_session__dsos_hit_all (rec->session); }
Namhyung Kim
maintainerHi Arnaldo,
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713
map__set_start(map, event->ksymbol.addr);
714map__set_end(map, map__start(map) + event->ksymbol.len);
715 err =maps__insert(machine__kernel_maps(machine), map);
Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306
Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522)
+---------------------------------------+
| |
| +------------------------+ |
| | (bpf_prog_40ddf486...) | <----+---- adding this
| | | |
| | | |
| c0147a2c |
| |
c0147640 c0148640And it failed to add bpf_prog_40ddf486... in check_invariants() because the end address is smaller than the previous map.
Quoted 28 lines
Thread 1 "perf" hit Breakpoint 1,
machine__process_ksymbol_register(machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) atutil/machine.c:688
688 {
(gdb) bt#0 machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688 #1 0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779 #2 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296 #3 0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1334 #4 0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1367 #5 0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626 #6 0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203 #7 0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132 #8 0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181 #9 0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226#10
0x000000000063e988inperf_session__process_events(session=0xf95f40) atutil/session.c:2390#110x000000000042d98binprocess_buildids(rec=0xec8ce0 <record>) at builtin-record.c:1475
#120x000000000042e963inrecord__finish_output(rec=0xec8ce0 <record>) at builtin-record.c:1798 #130x0000000000431c46in__cmd_record(rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841 #140x000000000043513fincmd_record(argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#150x00000000004bcf65inrun_builtin(p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351 #160x00000000004bd20cinhandle_internal_command(argc=3, argv=0x7fffffffde80) at perf.c:404
#170x00000000004bd365inrun_argv(argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#180x00000000004bd6aein main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
Quoted 2 lines
So, this one liner "refixes" the "modules" ends when processing the records to find the build ids, unsure if it is the best solution tho:
I think it "fixes" the problem by not clearing maps->ends_broken during the sample processing. So check_invariants() will not check the end addresses of overlapping bpf_trampoline and bpf_prog.
I'm curious how other commands (like perf report) are affected. I think the original concern was the output of perf buildid-list -m.
Thanks,
Namhyung
Quoted 16 lines
tools/perf/builtin-record.c +2
-0@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)if (!rec->no_buildid) {process_buildids (rec); +maps__fixup_end (machine__kernel_maps (&rec->session->machines.host)); +if (rec->buildid_all)perf_session__dsos_hit_all (rec->session); }
Arnaldo Carvalho de Melo
maintainerQuoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713
map__set_start(map, event->ksymbol.addr);
714map__set_end(map, map__start(map) + event->ksymbol.len);
715 err =maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | | c0147640 c0148640
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.
No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared ends_broken, then, later, when the checks_invariant is finally called at perf record exit time:
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
#4
0x00007ffff6e3681ein__assert_fail_base(fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev)<=map__end(map)",file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888
is when we detect the problem, but I see what you mean, I'm trying to figure out why this isn't caught here:
machine__process_ksymbol_register() ->
int maps__insert(struct maps *maps, struct map *map)
{
int ret;
down_write(maps__lock(maps));
ret = __maps__insert(maps, map);
check_invariants(maps);
up_write(maps__lock(maps));
return ret;
}Some more tracing:
root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l Removed event: probe_perf:check_invariants Removed event: probe_perf:maps_fixup_end Removed event: probe_perf:maps_insert probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted) probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#
And then:
root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep
<SNIP>
316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec)
316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
check_invariants (/home/acme/bin/perf)
maps__insert (/home/acme/bin/perf)
machine__process_ksymbol_register (/home/acme/bin/perf)
machine__process_ksymbol (/home/acme/bin/perf)
perf_event__process_ksymbol (/home/acme/bin/perf)
machines__deliver_event (/home/acme/bin/perf)
perf_session__deliver_event (/home/acme/bin/perf)
perf_session__process_event (/home/acme/bin/perf)
process_simple (/home/acme/bin/perf)
reader__read_event (/home/acme/bin/perf)
reader__process_events (/home/acme/bin/perf)
__perf_session__process_events (/home/acme/bin/perf)
perf_session__process_events (/home/acme/bin/perf)
process_buildids (/home/acme/bin/perf)
record__finish_output (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)
handle_internal_command (/home/acme/bin/perf)
run_argv (/home/acme/bin/perf)
main (/home/acme/bin/perf)
__libc_start_call_main (/usr/lib64/libc.so.6)
__libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
_start (/home/acme/bin/perf)
316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2)
316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
check_invariants (/home/acme/bin/perf)
maps__insert (/home/acme/bin/perf)
machine__process_ksymbol_register (/home/acme/bin/perf)
machine__process_ksymbol (/home/acme/bin/perf)
perf_event__process_ksymbol (/home/acme/bin/perf)
machines__deliver_event (/home/acme/bin/perf)
perf_session__deliver_event (/home/acme/bin/perf)
perf_session__process_event (/home/acme/bin/perf)
process_simple (/home/acme/bin/perf)
reader__read_event (/home/acme/bin/perf)
reader__process_events (/home/acme/bin/perf)
__perf_session__process_events (/home/acme/bin/perf)
perf_session__process_events (/home/acme/bin/perf)
process_buildids (/home/acme/bin/perf)
record__finish_output (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)
handle_internal_command (/home/acme/bin/perf)
run_argv (/home/acme/bin/perf)
main (/home/acme/bin/perf)
__libc_start_call_main (/usr/lib64/libc.so.6)
__libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
_start (/home/acme/bin/perf)
316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640)
316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0)
check_invariants (/home/acme/bin/perf)
maps__insert (/home/acme/bin/perf)
machine__process_ksymbol_register (/home/acme/bin/perf)
machine__process_ksymbol (/home/acme/bin/perf)
perf_event__process_ksymbol (/home/acme/bin/perf)
machines__deliver_event (/home/acme/bin/perf)
perf_session__deliver_event (/home/acme/bin/perf)
perf_session__process_event (/home/acme/bin/perf)
process_simple (/home/acme/bin/perf)
reader__read_event (/home/acme/bin/perf)
reader__process_events (/home/acme/bin/perf)
__perf_session__process_events (/home/acme/bin/perf)
perf_session__process_events (/home/acme/bin/perf)
process_buildids (/home/acme/bin/perf)
record__finish_output (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)
handle_internal_command (/home/acme/bin/perf)
run_argv (/home/acme/bin/perf)
main (/home/acme/bin/perf)
__libc_start_call_main (/usr/lib64/libc.so.6)
__libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
_start (/home/acme/bin/perf)
316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1)
check_invariants (/home/acme/bin/perf)
__maps__insert_sorted (/home/acme/bin/perf)
__maps__fixup_overlap_and_insert (/home/acme/bin/perf)
maps__fixup_overlap_and_insert (/home/acme/bin/perf)
thread__insert_map (/home/acme/bin/perf)
machine__process_mmap2_event (/home/acme/bin/perf)
perf_event__process_mmap2 (/home/acme/bin/perf)
build_id__process_mmap2 (/home/acme/bin/perf)
machines__deliver_event (/home/acme/bin/perf)
perf_session__deliver_event (/home/acme/bin/perf)
ordered_events__deliver_event (/home/acme/bin/perf)
do_flush (/home/acme/bin/perf)
__ordered_events__flush (/home/acme/bin/perf)
ordered_events__flush (/home/acme/bin/perf)
__perf_session__process_events (/home/acme/bin/perf)
perf_session__process_events (/home/acme/bin/perf)
process_buildids (/home/acme/bin/perf)
record__finish_output (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)
handle_internal_command (/home/acme/bin/perf)
run_argv (/home/acme/bin/perf)
main (/home/acme/bin/perf)
__libc_start_call_main (/usr/lib64/libc.so.6)
__libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
_start (/home/acme/bin/perf)
<SNIP>
[ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ]
1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1)
check_invariants (/home/acme/bin/perf)
maps__remove (/home/acme/bin/perf)
machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed.
)
machines__destroy_kernel_maps (/home/acme/bin/perf)
perf_session__destroy_kernel_maps (/home/acme/bin/perf)
perf_session__delete (/home/acme/bin/perf)
__cmd_record (/home/acme/bin/perf)
cmd_record (/home/acme/bin/perf)
run_builtin (/home/acme/bin/perf)
handle_internal_command (/home/acme/bin/perf)
run_argv (/home/acme/bin/perf)
main (/home/acme/bin/perf)
__libc_start_call_main (/usr/lib64/libc.so.6)
__libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6)
_start (/home/acme/bin/perf)
root@number:~#check_invariants() doesn't check the ends because the maps_byh_address_sorted is not set, I'll soon disappear into a call, but the above should help as a checkpoint, I'll be back.
- Arnaldo
Quoted 26 lines
Quoted 25 lines
Thread 1 "perf" hit Breakpoint 1,
machine__process_ksymbol_register(machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) atutil/machine.c:688
688 {
(gdb) bt#0 machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688 #1 0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779 #2 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296 #3 0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1334 #4 0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1367 #5 0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626 #6 0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203 #7 0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132 #8 0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181 #9 0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226#10
0x000000000063e988inperf_session__process_events(session=0xf95f40) atutil/session.c:2390#110x000000000042d98binprocess_buildids(rec=0xec8ce0 <record>) at builtin-record.c:1475
#120x000000000042e963inrecord__finish_output(rec=0xec8ce0 <record>) at builtin-record.c:1798 #130x0000000000431c46in__cmd_record(rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841 #140x000000000043513fincmd_record(argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#150x00000000004bcf65inrun_builtin(p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351 #160x00000000004bd20cinhandle_internal_command(argc=3, argv=0x7fffffffde80) at perf.c:404
#170x00000000004bd365inrun_argv(argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#180x00000000004bd6aein main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
Quoted 6 lines
Quoted 2 lines
So, this one liner "refixes" the "modules" ends when processing the records to find the build ids, unsure if it is the best solution tho:
I think it "fixes" the problem by not clearing
maps->ends_brokenduring the sample processing. Socheck_invariants()will not check the end addresses of overlappingbpf_trampolineandbpf_prog.
You mean my one-liner?
I meant "refixes" as in maps__fixup_end() will fixup the overlapping of the bpf_trampoline and bpf_prog and will re-clear maps->ends_broken (needlessly, it was already cleared by the first call to maps__fixup_end() after loading modules, at the start of the session).
Then check_invariants() _will_, check again, because maps->ends_broken is cleared (was cleared twice even), the end addresses and there will not be any overlapping, no?
- Arnaldo
Quoted 22 lines
I'm curious how other commands (like perf report) are affected. I think the original concern was the output of
perf buildid-list -m.Thanks,
NamhyungQuoted 16 lines
tools/perf/builtin-record.c +2
-0@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)if (!rec->no_buildid) {process_buildids (rec); +maps__fixup_end (machine__kernel_maps (&rec->session->machines.host)); +if (rec->buildid_all)perf_session__dsos_hit_all (rec->session); }
Arnaldo Carvalho de Melo
maintainerQuoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713
map__set_start(map, event->ksymbol.addr);
714map__set_end(map, map__start(map) + event->ksymbol.len);
715 err =maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | | c0147640 c0148640
Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:
Nope, __maps__insert() should notice that the ends are broken and set it:
if (nr_maps == 1) {
/* If there's just 1 entry then maps are sorted. */
maps__set_maps_by_address_sorted(maps, true);
maps__set_maps_by_name_sorted(maps, maps_by_name != NULL);
} else {
/* Sorted if maps were already sorted and this map starts after the last one. */
maps__set_maps_by_address_sorted(maps,
maps__maps_by_address_sorted(maps) &&
map__end(maps_by_address[nr_maps - 2]) <= map__start(new));
maps__set_maps_by_name_sorted(maps, false);
}
if (map__end(new) < map__start(new))
RC_CHK_ACCESS(maps)->ends_broken = true;humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus maps_by_address_sorted is set to false, ends_broken continues false and at the end maps_by_address_sorted is set to true and the last check_invariants triggerrs the asserts...
Quoted 244 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
#4
0x00007ffff6e3681ein__assert_fail_base(fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev)<=map__end(map)",file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888is when we detect the problem, but I see what you mean, I'm trying to figure out why this isn't caught here:
machine__process_ksymbol_register()->int maps__insert(struct maps *maps, struct map *map) { int ret; down_write(maps__lock(maps)); ret = __maps__insert(maps, map); check_invariants(maps); up_write(maps__lock(maps)); return ret; }Some more tracing:
root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l Removed event: probe_perf:check_invariants Removed event: probe_perf:maps_fixup_end Removed event: probe_perf:maps_insert probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted) probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#And then:
root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep <SNIP> 316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) 316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1) check_invariants (/home/acme/bin/perf) __maps__insert_sorted (/home/acme/bin/perf) __maps__fixup_overlap_and_insert (/home/acme/bin/perf) maps__fixup_overlap_and_insert (/home/acme/bin/perf) thread__insert_map (/home/acme/bin/perf) machine__process_mmap2_event (/home/acme/bin/perf) perf_event__process_mmap2 (/home/acme/bin/perf) build_id__process_mmap2 (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) ordered_events__deliver_event (/home/acme/bin/perf) do_flush (/home/acme/bin/perf) __ordered_events__flush (/home/acme/bin/perf) ordered_events__flush (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) <SNIP> [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] 1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1) check_invariants (/home/acme/bin/perf) maps__remove (/home/acme/bin/perf) machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. ) machines__destroy_kernel_maps (/home/acme/bin/perf) perf_session__destroy_kernel_maps (/home/acme/bin/perf) perf_session__delete (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) root@number:~#
check_invariants()doesn't check the ends because themaps_byh_address_sortedis not set, I'll soon disappear into a call, but the above should help as a checkpoint, I'll be back.- Arnaldo
Quoted 26 lines
Quoted 25 lines
Thread 1 "perf" hit Breakpoint 1,
machine__process_ksymbol_register(machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) atutil/machine.c:688
688 {
(gdb) bt#0 machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688 #1 0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779 #2 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296 #3 0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1334 #4 0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1367 #5 0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626 #6 0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203 #7 0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132 #8 0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181 #9 0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226#10
0x000000000063e988inperf_session__process_events(session=0xf95f40) atutil/session.c:2390#110x000000000042d98binprocess_buildids(rec=0xec8ce0 <record>) at builtin-record.c:1475
#120x000000000042e963inrecord__finish_output(rec=0xec8ce0 <record>) at builtin-record.c:1798 #130x0000000000431c46in__cmd_record(rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841 #140x000000000043513fincmd_record(argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#150x00000000004bcf65inrun_builtin(p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351 #160x00000000004bd20cinhandle_internal_command(argc=3, argv=0x7fffffffde80) at perf.c:404
#170x00000000004bd365inrun_argv(argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#180x00000000004bd6aein main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)Quoted 6 lines
Quoted 2 lines
So, this one liner "refixes" the "modules" ends when processing the records to find the build ids, unsure if it is the best solution tho:
I think it "fixes" the problem by not clearing
maps->ends_brokenduring the sample processing. Socheck_invariants()will not check the end addresses of overlappingbpf_trampolineandbpf_prog.You mean my one-liner?
I meant "refixes" as in
maps__fixup_end()will fixup the overlapping of thebpf_trampolineandbpf_progand will re-clearmaps->ends_broken(needlessly, it was already cleared by the first call tomaps__fixup_end()after loading modules, at the start of the session).Then
check_invariants()_will_, check again, becausemaps->ends_brokenis cleared (was cleared twice even), the end addresses and there will not be any overlapping, no?- Arnaldo
Quoted 22 lines
I'm curious how other commands (like perf report) are affected. I think the original concern was the output of
perf buildid-list -m.Thanks,
NamhyungQuoted 16 lines
tools/perf/builtin-record.c +2
-0@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)if (!rec->no_buildid) {process_buildids (rec); +maps__fixup_end (machine__kernel_maps (&rec->session->machines.host)); +if (rec->buildid_all)perf_session__dsos_hit_all (rec->session); }
Namhyung Kim
maintainerQuoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713
map__set_start(map, event->ksymbol.addr);
714map__set_end(map, map__start(map) + event->ksymbol.len);
715 err =maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | | c0147640 c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;
Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...
Right, probably it needs to set the ends_broken when the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.
Thanks,
Namhyung
Quoted 245 lines
Quoted 245 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
#4
0x00007ffff6e3681ein__assert_fail_base(fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bef08 "map__end(prev)<=map__end(map)",file=file@entry=0x7bedf8 "util/maps.c", line=line@entry=95, function=function@entry=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bef08 "map__end(prev) <= map__end(map)", file=0x7bedf8 "util/maps.c", line=95, function=0x7bf1c0 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x00000000006347a1 in check_invariants (maps=0xf987e0) at util/maps.c:95 #7 0x0000000000635ae2 in maps__remove (maps=0xf987e0, map=0xf98a80) at util/maps.c:538 #8 0x000000000062afd2 in machine__destroy_kernel_maps (machine=0xf98178) at util/machine.c:1176 #9 0x000000000062b32b in machines__destroy_kernel_maps (machines=0xf98178) at util/machine.c:1238Quoted 3 lines
#10
0x00000000006388afinperf_session__destroy_kernel_maps(session=0xf97f60) atutil/session.c:105#110x0000000000638df0inperf_session__delete(session=0xf97f60) atutil/session.c:248
#120x0000000000431f18in__cmd_record(rec=0xecace0 <record>, argc=4, argv=0x7fffffffde60) at builtin-record.c:2888is when we detect the problem, but I see what you mean, I'm trying to figure out why this isn't caught here:
machine__process_ksymbol_register()->int maps__insert(struct maps *maps, struct map *map) { int ret; down_write(maps__lock(maps)); ret = __maps__insert(maps, map); check_invariants(maps); up_write(maps__lock(maps)); return ret; }Some more tracing:
root@number:~# perf probe -d probe_perf:* ; perf probe -qx ~/bin/perf check_invariants maps 'maps->maps_by_address_sorted' ; perf probe -qx ~/bin/perf maps__insert maps 'map->dso->name:string' 'map->start' 'map->end' ; perf probe -qx ~/bin/perf maps__fixup_end maps ; perf probe -l Removed event: probe_perf:check_invariants Removed event: probe_perf:maps_fixup_end Removed event: probe_perf:maps_insert probe_perf:check_invariants (on check_invariants:1@util/maps.c in /home/acme/bin/perf with maps maps_by_address_sorted) probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#And then:
root@number:~# perf trace --lib -e probe_perf:maps_*,probe_perf:check_invariants/max-stack=32/ perf record sleep <SNIP> 316.283 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 316.284 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.296 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 316.298 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.310 perf/1882053 probe_perf:maps_insert((634e64) maps=0x342785d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) 316.311 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x0) check_invariants (/home/acme/bin/perf) maps__insert (/home/acme/bin/perf) machine__process_ksymbol_register (/home/acme/bin/perf) machine__process_ksymbol (/home/acme/bin/perf) perf_event__process_ksymbol (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) perf_session__process_event (/home/acme/bin/perf) process_simple (/home/acme/bin/perf) reader__read_event (/home/acme/bin/perf) reader__process_events (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) 316.369 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342a6950 maps_by_address_sorted=0x1) check_invariants (/home/acme/bin/perf) __maps__insert_sorted (/home/acme/bin/perf) __maps__fixup_overlap_and_insert (/home/acme/bin/perf) maps__fixup_overlap_and_insert (/home/acme/bin/perf) thread__insert_map (/home/acme/bin/perf) machine__process_mmap2_event (/home/acme/bin/perf) perf_event__process_mmap2 (/home/acme/bin/perf) build_id__process_mmap2 (/home/acme/bin/perf) machines__deliver_event (/home/acme/bin/perf) perf_session__deliver_event (/home/acme/bin/perf) ordered_events__deliver_event (/home/acme/bin/perf) do_flush (/home/acme/bin/perf) __ordered_events__flush (/home/acme/bin/perf) ordered_events__flush (/home/acme/bin/perf) __perf_session__process_events (/home/acme/bin/perf) perf_session__process_events (/home/acme/bin/perf) process_buildids (/home/acme/bin/perf) record__finish_output (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) <SNIP> [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] 1195.433 perf/1882053 probe_perf:check_invariants((633b0f) maps=0x342785d0 maps_by_address_sorted=0x1) check_invariants (/home/acme/bin/perf) maps__remove (/home/acme/bin/perf) machine__destroy_kernel_maps (/home/acme/bin/perfperf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. ) machines__destroy_kernel_maps (/home/acme/bin/perf) perf_session__destroy_kernel_maps (/home/acme/bin/perf) perf_session__delete (/home/acme/bin/perf) __cmd_record (/home/acme/bin/perf) cmd_record (/home/acme/bin/perf) run_builtin (/home/acme/bin/perf) handle_internal_command (/home/acme/bin/perf) run_argv (/home/acme/bin/perf) main (/home/acme/bin/perf) __libc_start_call_main (/usr/lib64/libc.so.6) __libc_start_main@@GLIBC_2.34 (/usr/lib64/libc.so.6) _start (/home/acme/bin/perf) root@number:~#
check_invariants()doesn't check the ends because themaps_byh_address_sortedis not set, I'll soon disappear into a call, but the above should help as a checkpoint, I'll be back.- Arnaldo
Quoted 26 lines
Quoted 25 lines
Thread 1 "perf" hit Breakpoint 1,
machine__process_ksymbol_register(machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) atutil/machine.c:688
688 {
(gdb) bt#0 machine__process_ksymbol_register (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:688 #1 0x00000000006294ca in machine__process_ksymbol (machine=0xf96158, event=0x7ffff7fb9aa0, sample=0x7fffffffa860) at util/machine.c:779 #2 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec8ce0 <record>, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, machine=0xf96158) at util/event.c:296 #3 0x000000000063b6e4 in machines__deliver_event (machines=0xf96158, evlist=0xf521f0, event=0x7ffff7fb9aa0, sample=0x7fffffffa860, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1334 #4 0x000000000063b8c9 in perf_session__deliver_event (session=0xf95f40, event=0x7ffff7fb9aa0, tool=0xec8ce0 <record>, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1367 #5 0x000000000063c6bd in perf_session__process_event (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:1626 #6 0x000000000063de3d in process_simple (session=0xf95f40, event=0x7ffff7fb9aa0, file_offset=31392, file_path=0xf96850 "perf.data") at util/session.c:2203 #7 0x000000000063daf4 in reader__read_event (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2132 #8 0x000000000063dcee in reader__process_events (rd=0x7fffffffafa0, session=0xf95f40, prog=0x7fffffffaf70) at util/session.c:2181 #9 0x000000000063df8b in __perf_session__process_events (session=0xf95f40) at util/session.c:2226#10
0x000000000063e988inperf_session__process_events(session=0xf95f40) atutil/session.c:2390#110x000000000042d98binprocess_buildids(rec=0xec8ce0 <record>) at builtin-record.c:1475
#120x000000000042e963inrecord__finish_output(rec=0xec8ce0 <record>) at builtin-record.c:1798 #130x0000000000431c46in__cmd_record(rec=0xec8ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841 #140x000000000043513fincmd_record(argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#150x00000000004bcf65inrun_builtin(p=0xecbd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351 #160x00000000004bd20cinhandle_internal_command(argc=3, argv=0x7fffffffde80) at perf.c:404
#170x00000000004bd365inrun_argv(argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#180x00000000004bd6aein main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)Quoted 6 lines
Quoted 2 lines
So, this one liner "refixes" the "modules" ends when processing the records to find the build ids, unsure if it is the best solution tho:
I think it "fixes" the problem by not clearing
maps->ends_brokenduring the sample processing. Socheck_invariants()will not check the end addresses of overlappingbpf_trampolineandbpf_prog.You mean my one-liner?
I meant "refixes" as in
maps__fixup_end()will fixup the overlapping of thebpf_trampolineandbpf_progand will re-clearmaps->ends_broken(needlessly, it was already cleared by the first call tomaps__fixup_end()after loading modules, at the start of the session).Then
check_invariants()_will_, check again, becausemaps->ends_brokenis cleared (was cleared twice even), the end addresses and there will not be any overlapping, no?- Arnaldo
Quoted 22 lines
I'm curious how other commands (like perf report) are affected. I think the original concern was the output of
perf buildid-list -m.Thanks,
NamhyungQuoted 16 lines
tools/perf/builtin-record.c +2
-0@@ -1797,6 +1797,8 @@ record__finish_output(struct record *rec)if (!rec->no_buildid) {process_buildids (rec); +maps__fixup_end (machine__kernel_maps (&rec->session->machines.host)); +if (rec->buildid_all)perf_session__dsos_hit_all (rec->session); }
Ian Rogers
reviewerQuoted 98 lines · Namhyung Kim
Quoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713 map__set_start(map, event->ksymbol.addr); 714 map__set_end(map, map__start(map) + event->ksymbol.len); 715 err = maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | | c0147640 c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...
Quoted 3 lines
Right, probably it needs to set the
ends_brokenwhen the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.
Ugh, I get git blamed for ends_broken and I was wondering what the heck it is:
https://lore.kernel.org/all/[email protected]/
My memory is that when the rb-tree was built the maps put in it could be broken and ends_broken was to capture we were in this state as the sorting would get broken, invariants be off, etc.. The rb-tree constructing code would then call maps__fixup_end. Having the caller call maps__fixup_end seems error prone, as does the whole "ends_broken" thing - remember I was in the code to fix memory leaks so modifying the maps API wasn't front of mind. I added ends_broken, the original rb-tree had no notion of it, because I was trying to get the invariants right for the testing I could do and ends_broken was the pragmatic thing to do for odd cases like kernel modules before maps__fixup_end is called.
The maps API has evolved and we have a pretty robust, but possibly not fast, maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd make maps__insert uphold the invariants and not have ends_broken. I'm worried that making ends_broken more load bearing isn't the right thing to do, we may even be able to not have the variable for the "ifndef NDEBUG" case, which making it load bearing would completely defeat.
So I think the fix here should be to understand the maps construction code for the modules, try to work out why maps__fixup_end wasn't called, perhaps migrate the code to maps__fixup_overlap_and_insert or add a missed maps__fixup_end call.
Given the blame I kind of feel responsible for this, but the real issue is adding the invariant checks has caught a latent bug that the rb-tree code would have just ignored and possibly been broken as a consequence. I lack bandwidth and a reproduction so thank you for digging into this.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 126 lines · Ian Rogers
Quoted 98 lines · Namhyung Kim
Quoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713 map__set_start(map, event->ksymbol.addr); 714 map__set_end(map, map__start(map) + event->ksymbol.len); 715 err = maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | |
c0147640c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...Quoted 3 lines
Right, probably it needs to set the
ends_brokenwhen the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.Ugh, I get git blamed for
ends_brokenand I was wondering what the heck it is:
https://lore.kernel.org/all/[email protected]/
My memory is that when the rb-tree was built the maps put in it could be broken andends_brokenwas to capture we were in this state as the sorting would get broken, invariants be off, etc.. The rb-tree constructing code would then callmaps__fixup_end. Having the caller callmaps__fixup_endseems error prone, as does the whole "ends_broken" thing - remember I was in the code to fix memory leaks so modifying the maps API wasn't front of mind. I addedends_broken, the original rb-tree had no notion of it, because I was trying to get the invariants right for the testing I could do andends_brokenwas the pragmatic thing to do for odd cases like kernel modules beforemaps__fixup_endis called.The maps API has evolved and we have a pretty robust, but possibly not fast,
maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd makemaps__insertuphold the invariants and not haveends_broken. I'm worried that makingends_brokenmore load bearing isn't the right thing to do, we may even be able to not have the variable for the "ifndef NDEBUG" case, which making it load bearing would completely defeat.
Quoted 4 lines
So I think the fix here should be to understand the maps construction code for the modules, try to work out why
maps__fixup_endwasn't called, perhaps migrate the code tomaps__fixup_overlap_and_insertor add a missedmaps__fixup_endcall.
IIUC module size in /proc/modules are wrong due to the reason in the commit 876e80cf83d10585 ("perf tools: Fixup end address of modules") and it called maps__fixup_end() for that.
But the problem is some BPF maps processed at real-time during the build-id processing at the end of perf record. One map is inside of another and check_invariants() didn't expect such maps and crashed.
Maybe we can fix maps__insert() to check such condition and fix it everytime. But it means it needs to sort the maps which would add big overhead we had before. So I just wanted to set the flag quickly and to fix the end address when it calls maps__find() or similar later.
Thanks,
Namhyung
Quoted 9 lines
Given the blame I kind of feel responsible for this, but the real issue is adding the invariant checks has caught a latent bug that the rb-tree code would have just ignored and possibly been broken as a consequence. I lack bandwidth and a reproduction so thank you for digging into this.
Thanks,
Ian
Ian Rogers
reviewerQuoted 133 lines · Namhyung Kim
Quoted 126 lines · Ian Rogers
Quoted 98 lines · Namhyung Kim
Quoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713 map__set_start(map, event->ksymbol.addr); 714 map__set_end(map, map__start(map) + event->ksymbol.len); 715 err = maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | |
c0147640c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...Quoted 3 lines
Right, probably it needs to set the
ends_brokenwhen the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.Ugh, I get git blamed for
ends_brokenand I was wondering what the heck it is:
https://lore.kernel.org/all/[email protected]/
My memory is that when the rb-tree was built the maps put in it could be broken andends_brokenwas to capture we were in this state as the sorting would get broken, invariants be off, etc.. The rb-tree constructing code would then callmaps__fixup_end. Having the caller callmaps__fixup_endseems error prone, as does the whole "ends_broken" thing - remember I was in the code to fix memory leaks so modifying the maps API wasn't front of mind. I addedends_broken, the original rb-tree had no notion of it, because I was trying to get the invariants right for the testing I could do andends_brokenwas the pragmatic thing to do for odd cases like kernel modules beforemaps__fixup_endis called.The maps API has evolved and we have a pretty robust, but possibly not fast,
maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd makemaps__insertuphold the invariants and not haveends_broken. I'm worried that makingends_brokenmore load bearing isn't the right thing to do, we may even be able to not have the variable for the "ifndef NDEBUG" case, which making it load bearing would completely defeat.Quoted 4 lines
So I think the fix here should be to understand the maps construction code for the modules, try to work out why
maps__fixup_endwasn't called, perhaps migrate the code tomaps__fixup_overlap_and_insertor add a missedmaps__fixup_endcall.
Quoted 7 lines
IIUC module size in
/proc/modulesare wrong due to the reason in the commit876e80cf83d10585("perf tools: Fixup end address of modules") and it calledmaps__fixup_end()for that.But the problem is some BPF maps processed at real-time during the build-id processing at the end of perf record. One map is inside of another and
check_invariants()didn't expect such maps and crashed.
I thought the real-time processing had to use maps__fixup_overlap_and_insert (rather than maps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?
Thanks,
Ian
Quoted 17 lines
Maybe we can fix
maps__insert()to check such condition and fix it everytime. But it means it needs to sort the maps which would add big overhead we had before. So I just wanted to set the flag quickly and to fix the end address when it callsmaps__find()or similar later.Thanks,
NamhyungQuoted 9 lines
Given the blame I kind of feel responsible for this, but the real issue is adding the invariant checks has caught a latent bug that the rb-tree code would have just ignored and possibly been broken as a consequence. I lack bandwidth and a reproduction so thank you for digging into this.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 138 lines · Ian Rogers
Quoted 137 lines · Namhyung Kim
Quoted 126 lines · Ian Rogers
Quoted 98 lines · Namhyung Kim
Quoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713 map__set_start(map, event->ksymbol.addr); 714 map__set_end(map, map__start(map) + event->ksymbol.len); 715 err = maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | |
c0147640c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...Quoted 3 lines
Right, probably it needs to set the
ends_brokenwhen the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.Ugh, I get git blamed for
ends_brokenand I was wondering what the heck it is:
https://lore.kernel.org/all/[email protected]/
My memory is that when the rb-tree was built the maps put in it could be broken andends_brokenwas to capture we were in this state as the sorting would get broken, invariants be off, etc.. The rb-tree constructing code would then callmaps__fixup_end. Having the caller callmaps__fixup_endseems error prone, as does the whole "ends_broken" thing - remember I was in the code to fix memory leaks so modifying the maps API wasn't front of mind. I addedends_broken, the original rb-tree had no notion of it, because I was trying to get the invariants right for the testing I could do andends_brokenwas the pragmatic thing to do for odd cases like kernel modules beforemaps__fixup_endis called.The maps API has evolved and we have a pretty robust, but possibly not fast,
maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd makemaps__insertuphold the invariants and not haveends_broken. I'm worried that makingends_brokenmore load bearing isn't the right thing to do, we may even be able to not have the variable for the "ifndef NDEBUG" case, which making it load bearing would completely defeat.Quoted 4 lines
So I think the fix here should be to understand the maps construction code for the modules, try to work out why
maps__fixup_endwasn't called, perhaps migrate the code tomaps__fixup_overlap_and_insertor add a missedmaps__fixup_endcall.IIUC module size in
/proc/modulesare wrong due to the reason in the commit876e80cf83d10585("perf tools: Fixup end address of modules") and it calledmaps__fixup_end()for that.
Quoted 8 lines
Quoted 3 lines
But the problem is some BPF maps processed at real-time during the build-id processing at the end of perf record. One map is inside of another and
check_invariants()didn't expect such maps and crashed.I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?
Make sense. How about this?
Thanks,
Namhyung
---8<---
tools/perf/util/machine.c +3 -3
@@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,map__set_start (map, event->ksymbol.addr);map__set_end (map,map__start (map) + event->ksymbol.len); - err =maps__insert (machine__kernel_maps (machine), map); + err =maps__fixup_overlap_and_insert (machine__kernel_maps (machine), map);if (err) { err = -ENOMEM ; goto out;
@@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)if (modules__parse (modules, machine, machine__create_module))return -1 ; -maps__fixup_end (machine__kernel_maps (machine)); -if (!machine__set_modules_path (machine))return 0 ;
@@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine) } } +maps__fixup_end (machine__kernel_maps (machine)); + out_put:dso__put (kernel);return ret;
Ian Rogers
reviewerQuoted 144 lines · Namhyung Kim
Quoted 142 lines · Ian Rogers
Quoted 140 lines · Namhyung Kim
Quoted 126 lines · Ian Rogers
Quoted 98 lines · Namhyung Kim
Quoted 89 lines · Arnaldo Carvalho de Melo
Quoted 62 lines · Arnaldo Carvalho de Melo
Quoted 61 lines · Namhyung Kim
Quoted 40 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Arnaldo Carvalho de Melo
So the call to
maps_fixup_end()will setmaps->end_brokento false, since it fixed up the map ends, etc, but then we insert more maps with broken ends:#6
0x0000000000633d52incheck_invariants(maps=0xf967c0) atutil/maps.c:95
95assert(map__end(prev) <= map__end(map));
(gdb) pprev->dso->name
$1 =0xfc47ab"bpf_trampoline_6442522522"So the above map is created overlapping a previously existing map:
root@number:~# perf probe -l probe_perf:maps_fixup_end (on maps__fixup_end:1@util/maps.c in /home/acme/bin/perf with maps) probe_perf:maps_insert (on maps__insert:1@util/maps.c in /home/acme/bin/perf with maps name start end) root@number:~#root@number:~# perf trace --lib -e probe_perf:maps* perf record sleep <SNIP> 319.791 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc0160788 end=0xffffffffc01607c8) 319.810 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01647b8 end=0xffffffffc01647f8) 319.822 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_egress" start=0xffffffffc016482c end=0xffffffffc016486c) 319.834 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_6deef7357e7b4530_sd_fw_ingress" start=0xffffffffc01648ac end=0xffffffffc01648ec) 319.845 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_prog_be31ae23198a0378_sd_devices" start=0xffffffffc0186388 end=0xffffffffc01864b2) 319.857 perf/1732173 probe_perf:maps_insert((634e5e) maps=0x2d9715d0 name="bpf_trampoline_6442522522" start=0xffffffffc0147640 end=0xffffffffc0148640) [ perf record: Captured and wrote 0.035 MB perf.data (7 samples) ] perf: util/maps.c:95: check_invariants: Assertion `map__end(prev) <= map__end(map)' failed. root@number:~#So a
PERF_RECORD_KSYMBOLprocessing will add a map for "bpf_trampoline_6442522522" that has its start after before the "bpf_prog_40ddf486530245f5_sd_devices" start, ok, but ends after "bpf_prog_40ddf486530245f5_sd_devices", overlapping it.
machine__process_ksymbol_register()does:713 map__set_start(map, event->ksymbol.addr); 714 map__set_end(map, map__start(map) + event->ksymbol.len); 715 err = maps__insert(machine__kernel_maps(machine), map);Quoted 6 lines
And:
(gdb) p /x
event->ksymbol.addr
$2 =0xffffffffc0147a2c
(gdb) pevent->ksymbol.len
$3 = 306Hmm.. so I think the situation is like below.
(bpf_trampoline_6442522522) +---------------------------------------+ | | | +------------------------+ | | | (bpf_prog_40ddf486...) | <----+---- adding this | | | | | | | | | c0147a2c | | |
c0147640c0148640Quoted 7 lines
Quoted 2 lines
And it failed to add
bpf_prog_40ddf486... incheck_invariants()because the end address is smaller than the previous map.No, it didn't fail to add, it managed to do it which left the kernel maps in a broken state, with overlappings while it had a cleared
ends_broken, then, later, when thechecks_invariantis finally called at perf record exit time:Nope,
__maps__insert()should notice that the ends are broken and set it:if (nr_maps == 1) { /* If there's just 1 entry then maps are sorted. */ maps__set_maps_by_address_sorted(maps, true); maps__set_maps_by_name_sorted(maps, maps_by_name != NULL); } else { /* Sorted if maps were already sorted and this map starts after the last one. */ maps__set_maps_by_address_sorted(maps, maps__maps_by_address_sorted(maps) && map__end(maps_by_address[nr_maps - 2]) <= map__start(new)); maps__set_maps_by_name_sorted(maps, false); } if (map__end(new) < map__start(new)) RC_CHK_ACCESS(maps)->ends_broken = true;Quoted 6 lines
humm, RC_CHK_ACCESS(maps)->ends_broken should be set for the case we have and I think it isn't being... Then the bpf trampoline map that is the last entry to be added is before the last entry and thus
maps_by_address_sortedis set to false,ends_brokencontinues false and at the endmaps_by_address_sortedis set to true and the lastcheck_invariantstriggerrs the asserts...Quoted 3 lines
Right, probably it needs to set the
ends_brokenwhen the end address of the new map is smaller than the previous (but the start address is bigger) and fixup the end address when it sorts the maps by address.Ugh, I get git blamed for
ends_brokenand I was wondering what the heck it is:
https://lore.kernel.org/all/[email protected]/
My memory is that when the rb-tree was built the maps put in it could be broken andends_brokenwas to capture we were in this state as the sorting would get broken, invariants be off, etc.. The rb-tree constructing code would then callmaps__fixup_end. Having the caller callmaps__fixup_endseems error prone, as does the whole "ends_broken" thing - remember I was in the code to fix memory leaks so modifying the maps API wasn't front of mind. I addedends_broken, the original rb-tree had no notion of it, because I was trying to get the invariants right for the testing I could do andends_brokenwas the pragmatic thing to do for odd cases like kernel modules beforemaps__fixup_endis called.The maps API has evolved and we have a pretty robust, but possibly not fast,
maps__fixup_overlap_and_insert:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/maps.h?h=perf-tools-next#n69
I think ideally we'd makemaps__insertuphold the invariants and not haveends_broken. I'm worried that makingends_brokenmore load bearing isn't the right thing to do, we may even be able to not have the variable for the "ifndef NDEBUG" case, which making it load bearing would completely defeat.Quoted 4 lines
So I think the fix here should be to understand the maps construction code for the modules, try to work out why
maps__fixup_endwasn't called, perhaps migrate the code tomaps__fixup_overlap_and_insertor add a missedmaps__fixup_endcall.IIUC module size in
/proc/modulesare wrong due to the reason in the commit876e80cf83d10585("perf tools: Fixup end address of modules") and it calledmaps__fixup_end()for that.But the problem is some BPF maps processed at real-time during the build-id processing at the end of perf record. One map is inside of another and
check_invariants()didn't expect such maps and crashed.
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does maps__fixup_end need to get pushed later?
Thanks,
Ian
Quoted 36 lines
Thanks,
Namhyung---8<---
tools/perf/util/machine.c +3
-3@@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,map__set_start (map, event->ksymbol.addr);map__set_end (map,map__start (map) + event->ksymbol.len); - err =maps__insert (machine__kernel_maps (machine), map); + err =maps__fixup_overlap_and_insert (machine__kernel_maps (machine), map);if (err) { err = -ENOMEM ; goto out;@@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)if (modules__parse (modules, machine, machine__create_module))return -1 ; -maps__fixup_end (machine__kernel_maps (machine)); -if (!machine__set_modules_path (machine))return 0 ;@@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine) } } +maps__fixup_end (machine__kernel_maps (machine)); + out_put:dso__put (kernel);return ret;
Namhyung Kim
maintainerQuoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Thanks,
Namhyung
Quoted 40 lines
Thanks,
IanQuoted 36 lines
Thanks,
Namhyung---8<---
tools/perf/util/machine.c +3
-3@@ -717,7 +717,7 @@ static int machine__process_ksymbol_register(struct machine *machine,map__set_start (map, event->ksymbol.addr);map__set_end (map,map__start (map) + event->ksymbol.len); - err =maps__insert (machine__kernel_maps (machine), map); + err =maps__fixup_overlap_and_insert (machine__kernel_maps (machine), map);if (err) { err = -ENOMEM ; goto out;@@ -1459,8 +1459,6 @@ static int machine__create_modules(struct machine *machine)if (modules__parse (modules, machine, machine__create_module))return -1 ; -maps__fixup_end (machine__kernel_maps (machine)); -if (!machine__set_modules_path (machine))return 0 ;@@ -1554,6 +1552,8 @@ int machine__create_kernel_maps(struct machine *machine) } } +maps__fixup_end (machine__kernel_maps (machine)); + out_put:dso__put (kernel);return ret;
Arnaldo Carvalho de Melo
maintainerQuoted 11 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Quoted 7 lines
Quoted 2 lines
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?
Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
Arnaldo Carvalho de Melo
maintainerQuoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert() does:
if (dso && dso__kernel(dso)) {
struct kmap *kmap = map__kmap(new);
if (kmap)
kmap->kmaps = maps;
else
pr_err("Internal error: kernel dso with non kernel map\n");
}while maps__fixup_overlap_and_insert() doesn't.
It calls __maps__insert_sorted() that probably should do what __maps__insert() does?
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }
Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?
Ok, so I did the following patch but this case fails:
@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)
*/
map__put(maps_by_address[i]);
maps_by_address[i] = map__get(new);
+ map__set_kmap(new, maps);
check_invariants(maps);
return err;
}With:
perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
As:
106 /*
107 * Maps by name maps should be in maps_by_address, so
108 * the reference count should be higher.
109 */
110 assert(refcount_read(map__refcnt(map)) > 1);
Since it is just replacing the map in the maps_by_address and not touching on the maps_by_name? Thus the refcount is just 1:
[ perf record: Woken up 1 times to write data ]
perf: util/maps.c:110: check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.
Thread 1 "perf" received signal SIGABRT, Aborted.__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bcdd0 "refcount_read(map__refcnt(map)) > 1",
file=file@entry=0x7bcc53 "util/maps.c", line=line@entry=110, function=function@entry=0x7bd010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bcdd0 "refcount_read(map__refcnt(map)) > 1", file=0x7bcc53 "util/maps.c", line=110,
function=0x7bd010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6 0x0000000000633e3b in check_invariants (maps=0xf947c0) at util/maps.c:110
#7 0x00000000006362b2 in __maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:932
#8 0x000000000063636c in maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:954
#9 0x000000000062920a in machine__process_ksymbol_register (machine=0xf94158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715
#10 0x00000000006294ca in machine__process_ksymbol (machine=0xf94158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779
#11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec6ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf94158) at util/event.c:296
#12 0x000000000063b76c in machines__deliver_event (machines=0xf94158, evlist=0xf501f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec6ce0 <record>, file_offset=35752,
file_path=0xf94850 "perf.data") at util/session.c:1334
#13 0x000000000063b951 in perf_session__deliver_event (session=0xf93f40, event=0x7ffff7fbaba8, tool=0xec6ce0 <record>, file_offset=35752, file_path=0xf94850 "perf.data")
at util/session.c:1367#14 0x000000000063c745 in perf_session__process_event (session=0xf93f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf94850 "perf.data") at util/session.c:1626 #15 0x000000000063dec5 in process_simple (session=0xf93f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf94850 "perf.data") at util/session.c:2203 #16 0x000000000063db7c in reader__read_event (rd=0x7fffffffafa0, session=0xf93f40, prog=0x7fffffffaf70) at util/session.c:2132
#17 0x000000000063dd76 in reader__process_events (rd=0x7fffffffafa0, session=0xf93f40, prog=0x7fffffffaf70) at util/session.c:2181
#18 0x000000000063e013 in __perf_session__process_events (session=0xf93f40) at util/session.c:2226
#19 0x000000000063ea10 in perf_session__process_events (session=0xf93f40) at util/session.c:2390
#20 0x000000000042d98b in process_buildids (rec=0xec6ce0 <record>) at builtin-record.c:1475
#21 0x000000000042e963 in record__finish_output (rec=0xec6ce0 <record>) at builtin-record.c:1798
#22 0x0000000000431c46 in __cmd_record (rec=0xec6ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#23 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#24 0x00000000004bcf65 in run_builtin (p=0xec9d60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#25 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#26 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#27 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
#6 0x0000000000633e3b in check_invariants (maps=0xf947c0) at util/maps.c:110
110 assert(refcount_read(map__refcnt(map)) > 1);
(gdb) p map
$2 = (struct map *) 0xfe6060
(gdb) p map->dso
$3 = (struct dso *) 0xfe5ea0
(gdb) p map->dso->name
$4 = 0xfe602b "bpf_prog_6deef7357e7b4530_sd_fw_ingress"
(gdb) p refcount_read(map__refcnt(map))
$5 = 1
(gdb) fr 7
#7 0x00000000006362b2 in __maps__fixup_overlap_and_insert (maps=0xf947c0, new=0xfc27e0) at util/maps.c:932
932 check_invariants(maps);
(gdb) p next
$6 = (struct map *) 0xfe4bc0
(gdb) p next->dso
$7 = (struct dso *) 0xfe4a00
(gdb) p next->dso->name
$8 = 0xfe4b8b "bpf_prog_6deef7357e7b4530_sd_fw_egress"
(gdb)
(gdb) p new->dso->name
$9 = 0xfc27ab "bpf_trampoline_6442522521"
(gdb) p /x map__start(next)
$12 = 0xffffffffc012b158
(gdb) p /x map__start(new)
$13 = 0xffffffffc0129640
(gdb) p /x map__end(next)
$14 = 0xffffffffc012b198
(gdb) p /x map__end(new)
$15 = 0xffffffffc012a640
(gdb)
So its again that case of overlapping maps...
Ah, the patch below is on top of Namhyungs.
- Arnaldo
tools/perf/util/maps.c +27 -8
@@ -428,11 +428,29 @@ static unsigned int maps__by_name_index(const struct maps *maps, const struct mareturn -1 ; } +static void map__set_kmap (struct map *map,struct maps *maps) +{ +struct dso *dso; + +if (map ==NULL ) +return ; + + dso =map__dso (map); + +if (dso &&dso__kernel (dso)) { +struct kmap *kmap =map__kmap (map); + +if (kmap) + kmap->kmaps = maps; +else +pr_err ("Internal error: kernel dso with non kernel map\n" ); + } +} +static int __maps__insert (struct maps *maps,struct map *new) {struct map **maps_by_address =maps__maps_by_address (maps);struct map **maps_by_name =maps__maps_by_name (maps); -const struct dso *dso =map__dso (new);unsigned int nr_maps =maps__nr_maps (maps);unsigned int nr_allocate =RC_CHK_ACCESS (maps)->nr_maps_allocated;
@@ -483,14 +501,9 @@ static int __maps__insert(struct maps *maps, struct map *new) }if (map__end (new) <map__start (new))RC_CHK_ACCESS (maps)->ends_broken = true; -if (dso &&dso__kernel (dso)) { -struct kmap *kmap =map__kmap (new); -if (kmap) - kmap->kmaps = maps; -else -pr_err ("Internal error: kernel dso with non kernel map\n" ); - } +map__set_kmap (new, maps); +return 0 ; }
@@ -784,7 +797,12 @@ static int __maps__insert_sorted(struct maps *maps, unsigned int first_after_ind maps_by_name[nr_maps +1 ] =map__get (new2); }RC_CHK_ACCESS (maps)->nr_maps = nr_maps + to_add; +maps__set_maps_by_name_sorted (maps, false); + +map__set_kmap (new1, maps); +map__set_kmap (new2, maps); +check_invariants (maps);return 0 ; }
@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +map__set_kmap (new, maps);check_invariants (maps);return err; }
Namhyung Kim
maintainerQuoted 68 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
(quoted diff) +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new) */ map__put(maps_by_address[i]); maps_by_address[i] = map__get(new); + map__set_kmap(new, maps); check_invariants(maps); return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
Quoted 8 lines
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:
Sounds like it. Can you add this on top?
Thanks,
Namhyung
---8<---
tools/perf/util/maps.c +18 -1
@@ -797,7 +797,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new) {int err =0 ;FILE *fp =debug_file (); -unsigned int i; +unsigned int i, ni;if (!maps__maps_by_address_sorted (maps))__maps__sort_by_address (maps);
@@ -808,6 +808,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ for (i =first_ending_after (maps, new); i <maps__nr_maps (maps); ) {struct map **maps_by_address =maps__maps_by_address (maps); +struct map **maps_by_name =maps__maps_by_name (maps);struct map *pos = maps_by_address[i];struct map *before =NULL , *after =NULL ;
@@ -827,6 +828,9 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)map__fprintf (pos, fp); } +if (maps_by_name) + ni =maps__by_name_index (maps, pos); +/* * Now check if we need to create new maps for areas not * overlapped by the new map:
@@ -871,6 +875,11 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)if (before) {map__put (maps_by_address[i]); maps_by_address[i] = before; +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (before); + } +/* Maps are still ordered, go to next one. */ i++;if (after) {
@@ -892,6 +901,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + } err =__maps__insert_sorted (maps, i +1 , after,NULL );map__put (after);check_invariants (maps);
@@ -910,6 +923,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }
Arnaldo Carvalho de Melo
maintainerQuoted 75 lines · Namhyung Kim
Quoted 74 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
tools/perf/util/maps.c +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +map__set_kmap (new, maps);check_invariants (maps);return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);
Quoted 4 lines
Quoted 2 lines
Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:Sounds like it. Can you add this on top?
Trying, but somehow its not applying cleanly, checking:
⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch
patching file tools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to file tools/perf/util/maps.c.rej ⬢ [acme@toolbox perf-tools]$
⬢ [acme@toolbox perf-tools]$ git log --oneline -54a9f5d76130b707f (HEAD -> perf-tools) wip: acmed5ba0f5af35937c7 wip: namhyung42367eca7604e16e (perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup 293f324ce96d7001 tools: Unify top-level quiet infrastructure9fae5884bb0e3480 (tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs ⬢ [acme@toolbox perf-tools]$
Quoted 70 lines
Thanks,
Namhyung---8<---
tools/perf/util/maps.c +18
-1@@ -797,7 +797,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new) {int err =0 ;FILE *fp =debug_file (); -unsigned int i; +unsigned int i, ni;if (!maps__maps_by_address_sorted (maps))__maps__sort_by_address (maps);@@ -808,6 +808,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ for (i =first_ending_after (maps, new); i <maps__nr_maps (maps); ) {struct map **maps_by_address =maps__maps_by_address (maps); +struct map **maps_by_name =maps__maps_by_name (maps);struct map *pos = maps_by_address[i];struct map *before =NULL , *after =NULL ;@@ -827,6 +828,9 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)map__fprintf (pos, fp); } +if (maps_by_name) + ni =maps__by_name_index (maps, pos); +/* * Now check if we need to create new maps for areas not * overlapped by the new map: @@ -871,6 +875,11 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)if (before) {map__put (maps_by_address[i]); maps_by_address[i] = before; +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (before); + } +/* Maps are still ordered, go to next one. */ i++;if (after) {@@ -892,6 +901,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + } err =__maps__insert_sorted (maps, i +1 , after,NULL );map__put (after);check_invariants (maps);@@ -910,6 +923,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }
Arnaldo Carvalho de Melo
maintainerQuoted 94 lines · Arnaldo Carvalho de Melo
Quoted 75 lines · Namhyung Kim
Quoted 74 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
(quoted diff) +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new) */ map__put(maps_by_address[i]); maps_by_address[i] = map__get(new); + map__set_kmap(new, maps); check_invariants(maps); return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);Quoted 4 lines
Quoted 2 lines
Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:Sounds like it. Can you add this on top?
Trying, but somehow its not applying cleanly, checking:
⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch
patching filetools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to filetools/perf/util/maps.c.rej⬢ [acme@toolbox perf-tools]$
Quoted 7 lines
⬢ [acme@toolbox perf-tools]$ git log --oneline -5
4a9f5d76130b707f(HEAD -> perf-tools) wip: acmed5ba0f5af35937c7wip: namhyung42367eca7604e16e(perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup293f324ce96d7001tools: Unify top-level quiet infrastructure9fae5884bb0e3480(tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs ⬢ [acme@toolbox perf-tools]$
⬢ [acme@toolbox perf-tools]$ cat tools/perf/util/maps.c.rej
perf/util/maps.c +4 -0
@@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }
⬢ [acme@toolbox perf-tools]$
Fixing this up by hand
Arnaldo Carvalho de Melo
maintainerQuoted 119 lines · Arnaldo Carvalho de Melo
Quoted 94 lines · Arnaldo Carvalho de Melo
Quoted 75 lines · Namhyung Kim
Quoted 74 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
perf/util/maps.c +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +map__set_kmap (new, maps);check_invariants (maps);return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);Quoted 4 lines
Quoted 2 lines
Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:Sounds like it. Can you add this on top?
Trying, but somehow its not applying cleanly, checking:
⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch
patching filetools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to filetools/perf/util/maps.c.rej⬢ [acme@toolbox perf-tools]$Quoted 7 lines
⬢ [acme@toolbox perf-tools]$ git log --oneline -5
4a9f5d76130b707f(HEAD -> perf-tools) wip: acmed5ba0f5af35937c7wip: namhyung42367eca7604e16e(perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup293f324ce96d7001tools: Unify top-level quiet infrastructure9fae5884bb0e3480(tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs ⬢ [acme@toolbox perf-tools]$⬢ [acme@toolbox perf-tools]$ cat
tools/perf/util/maps.c.rejperf/util/maps.c +4
-0@@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }⬢ [acme@toolbox perf-tools]$
Quoted 1 line
Fixing this up by hand
I see, I had tried this after sending that patch:
map__set_kmap(new, maps);
Before check_invariants(), but that doesn't make sense, I should've dropped that, doing it now.
- Arnaldo
Arnaldo Carvalho de Melo
maintainerQuoted 120 lines · Arnaldo Carvalho de Melo
Quoted 118 lines · Arnaldo Carvalho de Melo
Quoted 94 lines · Arnaldo Carvalho de Melo
Quoted 75 lines · Namhyung Kim
Quoted 74 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert(rather thanmaps__insert) as mmap events only give us VMA data and two mmaps may have been merged. Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
(quoted diff) +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new) */ map__put(maps_by_address[i]); maps_by_address[i] = map__get(new); + map__set_kmap(new, maps); check_invariants(maps); return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);Quoted 4 lines
Quoted 2 lines
Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:Sounds like it. Can you add this on top?
Trying, but somehow its not applying cleanly, checking:
⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch
patching filetools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to filetools/perf/util/maps.c.rej⬢ [acme@toolbox perf-tools]$Quoted 7 lines
⬢ [acme@toolbox perf-tools]$ git log --oneline -5
4a9f5d76130b707f(HEAD -> perf-tools) wip: acmed5ba0f5af35937c7wip: namhyung42367eca7604e16e(perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup293f324ce96d7001tools: Unify top-level quiet infrastructure9fae5884bb0e3480(tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs ⬢ [acme@toolbox perf-tools]$⬢ [acme@toolbox perf-tools]$ cat
tools/perf/util/maps.c.rejperf/util/maps.c +4
-0@@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }⬢ [acme@toolbox perf-tools]$
Quoted 8 lines
Quoted 1 line
Fixing this up by hand
I see, I had tried this after sending that patch:
map__set_kmap(new, maps);Before
check_invariants(), but that doesn't make sense, I should've dropped that, doing it now.
Nope, it still triggers:
930 } else {
931 struct map *next = NULL;
932
933 if (i + 1 < maps__nr_maps(maps))
934 next = maps_by_address[i + 1];
935
936 if (!next || map__start(next) >= map__end(new)) {
937 /*
938 * Replace existing mapping and end knowing
939 * there aren't later overlapping or any
940 * mappings.
941 */
942 map__put(maps_by_address[i]);
943 maps_by_address[i] = map__get(new);
944 if (maps_by_name) {
945 map__put(maps_by_name[ni]);
946 maps_by_name[ni] = map__get(new);
947 }
948 check_invariants(maps);
949 return err;
950 }
951 __maps__remove(maps, pos);
952 check_invariants(maps);
953 /*
954 * Maps are ordered but no need to increase `i` as the
955 * later maps were moved down.
956 */957 }
perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.
Thread 1 "perf" received signal SIGABRT, Aborted.__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78
#2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007ffff6e36902 in __GI_abort () at abort.c:79
#4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)",
file=file@entry=0x7bfc53 "util/maps.c", line=line@entry=80, function=function@entry=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96
#5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", file=0x7bfc53 "util/maps.c", line=80,
function=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105
#6 0x0000000000633c74 in check_invariants (maps=0xf977c0) at util/maps.c:80
#7 0x00000000006363a6 in __maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:948
#8 0x0000000000636460 in maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:970
#9 0x000000000062920a in machine__process_ksymbol_register (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715
#10 0x00000000006294ca in machine__process_ksymbol (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779
#11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec9ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf97158) at util/event.c:296
#12 0x000000000063b860 in machines__deliver_event (machines=0xf97158, evlist=0xf531f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec9ce0 <record>, file_offset=35752,
file_path=0xf97850 "perf.data") at util/session.c:1334
#13 0x000000000063ba45 in perf_session__deliver_event (session=0xf96f40, event=0x7ffff7fbaba8, tool=0xec9ce0 <record>, file_offset=35752, file_path=0xf97850 "perf.data")
at util/session.c:1367#14 0x000000000063c839 in perf_session__process_event (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:1626 #15 0x000000000063dfb9 in process_simple (session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:2203 #16 0x000000000063dc70 in reader__read_event (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2132
#17 0x000000000063de6a in reader__process_events (rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) at util/session.c:2181
#18 0x000000000063e107 in __perf_session__process_events (session=0xf96f40) at util/session.c:2226
#19 0x000000000063eb04 in perf_session__process_events (session=0xf96f40) at util/session.c:2390
#20 0x000000000042d98b in process_buildids (rec=0xec9ce0 <record>) at builtin-record.c:1475
#21 0x000000000042e963 in record__finish_output (rec=0xec9ce0 <record>) at builtin-record.c:1798
#22 0x0000000000431c46 in __cmd_record (rec=0xec9ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#23 0x000000000043513f in cmd_record (argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#24 0x00000000004bcf65 in run_builtin (p=0xeccd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#25 0x00000000004bd20c in handle_internal_command (argc=3, argv=0x7fffffffde80) at perf.c:404
#26 0x00000000004bd365 in run_argv (argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#27 0x00000000004bd6ae in main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)
humm, it seems that thing I removed may be needed after all...
Yeah, please check if adding this, on top of your latest (second) patch, makese sense:
tools/perf/util/maps.c +1 -0
@@ -945,6 +945,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)map__put (maps_by_name[ni]); maps_by_name[ni] =map__get (new); } +map__set_kmap (new, maps);check_invariants (maps);return err; }
⬢ [acme@toolbox perf-tools]$
With your two patches and my two patches finally I get:
root@number:~# perf record sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.035 MB perf.data (14 samples) ] root@number:~#
on a perf-tools/perf-tools build with DEBUG=1
- Arnaldo
Namhyung Kim
maintainerQuoted 222 lines · Arnaldo Carvalho de Melo
Quoted 120 lines · Arnaldo Carvalho de Melo
Quoted 118 lines · Arnaldo Carvalho de Melo
Quoted 94 lines · Arnaldo Carvalho de Melo
Quoted 75 lines · Namhyung Kim
Quoted 74 lines · Arnaldo Carvalho de Melo
Quoted 45 lines · Arnaldo Carvalho de Melo
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 15 lines · Arnaldo Carvalho de Melo
Quoted 13 lines · Namhyung Kim
Quoted 2 lines · Ian Rogers
Quoted 1 line · Namhyung Kim
On Mon, Feb 24, 2025 at 10:18:37AM -0800, Ian Rogers wrote:
[SNIP]
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
I thought the real-time processing had to use
maps__fixup_overlap_and_insert (rather than maps__insert) as mmap
events only give us VMA data and two mmaps may have been merged.
Shouldn't doing this change be the simplest fix?Make sense. How about this?
Lgtm, I have no way to test the issue. Why does
maps__fixup_endneed to get pushed later?Quoted 6 lines
Quoted 4 lines
I just noticed it would add extra kernel maps after modules. I think it should fixup end address of the kernel maps after adding all maps first.
Arnaldo, can you please test this?
Trying it now.
Now we have something different:
Quoted 7 lines
root@number:~# perf record sleep sleep: missing operand Try 'sleep --help' for more information. [ perf record: Woken up 1 times to write data ] perf: util/maps.c:80: check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed. Aborted (core dumped) root@number:~#
__maps__insert()does:if (dso && dso__kernel(dso)) { struct kmap *kmap = map__kmap(new); if (kmap) kmap->kmaps = maps; else pr_err("Internal error: kernel dso with non kernel map\n"); }Quoted 4 lines
while
maps__fixup_overlap_and_insert()doesn't.It calls
__maps__insert_sorted()that probably should do what__maps__insert()does?Ok, so I did the following patch but this case fails:
tools/perf/util/maps.c +1
-0@@ -910,6 +928,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +map__set_kmap (new, maps);check_invariants (maps);return err; }With:
perf:
util/maps.c:110:check_invariants: Assertion `refcount_read(map__refcnt(map)) > 1' failed.As:
106 /*
107 * Maps by name maps should be inmaps_by_address, so
108 * the reference count should be higher.
109 */
110assert(refcount_read(map__refcnt(map)) > 1);Quoted 4 lines
Quoted 2 lines
Since it is just replacing the map in the
maps_by_addressand not touching on themaps_by_name? Thus the refcount is just 1:Sounds like it. Can you add this on top?
Trying, but somehow its not applying cleanly, checking:
⬢ [acme@toolbox perf-tools]$ patch -p1 < ~/wb/1.patch
patching filetools/perf/util/maps.c
Hunk #1 succeeded at 815 (offset 18 lines).
Hunk #2 succeeded at 826 (offset 18 lines).
Hunk #3 succeeded at 846 (offset 18 lines).
Hunk #4 succeeded at 893 (offset 18 lines).
Hunk #5 succeeded at 919 (offset 18 lines).
Hunk #6 FAILED at 923.
1 out of 6 hunks FAILED -- saving rejects to filetools/perf/util/maps.c.rej⬢ [acme@toolbox perf-tools]$Quoted 7 lines
⬢ [acme@toolbox perf-tools]$ git log --oneline -5
4a9f5d76130b707f(HEAD -> perf-tools) wip: acmed5ba0f5af35937c7wip: namhyung42367eca7604e16e(perf-tools/tmp.perf-tools, perf-tools/perf-tools) tools: Remove redundant quiet setup293f324ce96d7001tools: Unify top-level quiet infrastructure9fae5884bb0e3480(tag: perf-tools-fixes-for-v6.14-2025-01-30) perf cpumap: Fix die and cluster IDs ⬢ [acme@toolbox perf-tools]$⬢ [acme@toolbox perf-tools]$ cat
tools/perf/util/maps.c.rejperf/util/maps.c +4
-0@@ -923,6 +936,10 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)*/ map__put (maps_by_address[i]); maps_by_address[i] =map__get (new); +if (maps_by_name) { +map__put (maps_by_name[ni]); + maps_by_name[ni] =map__get (new); + }check_invariants (maps);return err; }⬢ [acme@toolbox perf-tools]$
Quoted 8 lines
Quoted 1 line
Fixing this up by hand
I see, I had tried this after sending that patch:
map__set_kmap(new, maps);Before
check_invariants(), but that doesn't make sense, I should've dropped that, doing it now.Nope, it still triggers:
930 } else {
931 struct map *next = NULL;
932
933 if (i + 1 <maps__nr_maps(maps))
934 next = maps_by_address[i + 1];
935
936 if (!next ||map__start(next)>=map__end(new)) {
937 /*938 * Replace existing mapping and end knowing 939 * there aren't later overlapping or any 940 * mappings. 941 */ 942 map__put(maps_by_address[i]); 943 maps_by_address[i] = map__get(new); 944 if (maps_by_name) { 945 map__put(maps_by_name[ni]); 946 maps_by_name[ni] = map__get(new); 947 } 948 check_invariants(maps); 949 return err; 950 } 951 __maps__remove(maps, pos); 952 check_invariants(maps); 953 /* 954 * Maps are ordered but no need to increase `i` as the 955 * later maps were moved down. 956 */957 }
perf:
util/maps.c:80:check_invariants: Assertion `RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)' failed.Thread 1 "perf" received signal SIGABRT, Aborted.
__pthread_kill_implementation(threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 returnINTERNAL_SYSCALL_ERROR_P(ret) ?INTERNAL_SYSCALL_ERRNO(ret) : 0;
(gdb) bt#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff6ea80a3 in __pthread_kill_internal (threadid=<optimized out>, signo=6) at pthread_kill.c:78 #2 0x00007ffff6e4ef1e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff6e36902 in __GI_abort () at abort.c:79 #4 0x00007ffff6e3681e in __assert_fail_base (fmt=0x7ffff6fc3bb8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", file=file@entry=0x7bfc53 "util/maps.c", line=line@entry=80, function=function@entry=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:96 #5 0x00007ffff6e47047 in __assert_fail (assertion=0x7bfd08 "RC_CHK_EQUAL(map__kmap(map)->kmaps, maps)", file=0x7bfc53 "util/maps.c", line=80, function=0x7c0010 <__PRETTY_FUNCTION__.6> "check_invariants") at assert.c:105 #6 0x0000000000633c74 in check_invariants (maps=0xf977c0) at util/maps.c:80 #7 0x00000000006363a6 in __maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:948 #8 0x0000000000636460 in maps__fixup_overlap_and_insert (maps=0xf977c0, new=0xfc57e0) at util/maps.c:970 #9 0x000000000062920a in machine__process_ksymbol_register (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:715 #10 0x00000000006294ca in machine__process_ksymbol (machine=0xf97158, event=0x7ffff7fbaba8, sample=0x7fffffffa860) at util/machine.c:779 #11 0x00000000005ce2fd in perf_event__process_ksymbol (tool=0xec9ce0 <record>, event=0x7ffff7fbaba8, sample=0x7fffffffa860, machine=0xf97158) at util/event.c:296 #12 0x000000000063b860 in machines__deliver_event (machines=0xf97158, evlist=0xf531f0, event=0x7ffff7fbaba8, sample=0x7fffffffa860, tool=0xec9ce0 <record>, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:1334 #13 0x000000000063ba45 in perf_session__deliver_event (session=0xf96f40, event=0x7ffff7fbaba8, tool=0xec9ce0 <record>, file_offset=35752, file_path=0xf97850 "perf.data") at util/session.c:1367#14
0x000000000063c839inperf_session__process_event(session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") atutil/session.c:1626#150x000000000063dfb9inprocess_simple(session=0xf96f40, event=0x7ffff7fbaba8, file_offset=35752, file_path=0xf97850 "perf.data") atutil/session.c:2203#160x000000000063dc70inreader__read_event(rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) atutil/session.c:2132
#170x000000000063de6ainreader__process_events(rd=0x7fffffffafa0, session=0xf96f40, prog=0x7fffffffaf70) atutil/session.c:2181
#180x000000000063e107in__perf_session__process_events(session=0xf96f40) atutil/session.c:2226
#190x000000000063eb04inperf_session__process_events(session=0xf96f40) atutil/session.c:2390
#200x000000000042d98binprocess_buildids(rec=0xec9ce0 <record>) at builtin-record.c:1475
#210x000000000042e963inrecord__finish_output(rec=0xec9ce0 <record>) at builtin-record.c:1798
#220x0000000000431c46in__cmd_record(rec=0xec9ce0 <record>, argc=2, argv=0x7fffffffde80) at builtin-record.c:2841
#230x000000000043513fincmd_record(argc=2, argv=0x7fffffffde80) at builtin-record.c:4260
#240x00000000004bcf65inrun_builtin(p=0xeccd60 <commands+288>, argc=3, argv=0x7fffffffde80) at perf.c:351
#250x00000000004bd20cinhandle_internal_command(argc=3, argv=0x7fffffffde80) at perf.c:404
#260x00000000004bd365inrun_argv(argcp=0x7fffffffdc6c, argv=0x7fffffffdc60) at perf.c:448
#270x00000000004bd6aein main (argc=3, argv=0x7fffffffde80) at perf.c:556
(gdb)humm, it seems that thing I removed may be needed after all...
Yeah, please check if adding this, on top of your latest (second) patch, makese sense:
tools/perf/util/maps.c +1
-0@@ -945,6 +945,7 @@ static int __maps__fixup_overlap_and_insert(struct maps *maps, struct map *new)map__put (maps_by_name[ni]); maps_by_name[ni] =map__get (new); } +map__set_kmap (new, maps);check_invariants (maps);return err; }⬢ [acme@toolbox perf-tools]$
Quoted 8 lines
With your two patches and my two patches finally I get:
root@number:~# perf record sleep 1 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.035 MB perf.data (14 samples) ] root@number:~#on a perf-tools/perf-tools build with DEBUG=1
Great! Can you please send the final version as a formal patch?
Thanks,
Namhyung