BUG: perf top enters loop synthesizing events for existing threads
Heads up,
While testing some unrelated patches I noticed, in one of my
machines, that 'perf top', as root, is entering some sort of loop and ends up consuming a lot of memory, making the system unresponsive, bisected it down to:
[acme@zoo linux]$ git bisect good ca6c41c59b964d362823e80442e9e32c31106b29 is the first bad commit commit ca6c41c59b964d362823e80442e9e32c31106b29
Author: David Ahern <[email protected]> Date: Mon Mar 30 14:35:58 2015 -0600
perf tools: Fix ppid for synthesized fork events
363b785f38 added synthesized fork events and set a thread's parent id to itself. Since we are already processing /proc/<pid>/status the ppid can be determined properly. Make it so.
:040000 040000 24d479e4afbe95aaba882850473a870b6b3c7e87cc86fe0e51b5e8a66e8a23f934fd0c4ff0f8dcd6 M tools
[acme@zoo linux]$
I also noticed that if I set /proc/sys/kernel/perf_event_paranoid to -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...
I might need to revert it for this cycle...
- Arnaldo
Em Wed, Apr 08, 2015 at 10:42:51AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 31 lines
Heads up,
While testing some unrelated patches I noticed, in one of my
machines, that 'perf top', as root, is entering some sort of loop and ends up consuming a lot of memory, making the system unresponsive, bisected it down to:[acme@zoo linux]$ git bisect good
ca6c41c59b964d362823e80442e9e32c31106b29is the first bad commit commitca6c41c59b964d362823e80442e9e32c31106b29Author: David Ahern <[email protected]> Date: Mon Mar 30 14:35:58 2015 -0600perf tools: Fix ppid for synthesized fork events
363b785f38added synthesized fork events and set a thread's parent id to itself. Since we are already processing /proc/<pid>/status the ppid can be determined properly. Make it so.Signed-off-by David Ahern <[email protected]>Acked-by Don Zickus <[email protected]>Acked-by Jiri Olsa <[email protected]>Cc Joe Mario <[email protected]>Signed-off-by Arnaldo Carvalho de Melo <[email protected]>:040000 040000
24d479e4afbe95aaba882850473a870b6b3c7e87cc86fe0e51b5e8a66e8a23f934fd0c4ff0f8dcd6M tools
[acme@zoo linux]$
Quoted 7 lines
I also noticed that if I set
/proc/sys/kernel/perf_event_paranoidto -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...I might need to revert it for this cycle...
Just reverting it makes the symptoms go away, to works as before. Trying to understand the problem now.
- Arnaldo
Quoted 5 lines · Arnaldo Carvalho de Melo
Quoted 5 lines
I also noticed that if I set
/proc/sys/kernel/perf_event_paranoidto -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...
odd. because this has nothing to do with perf_events; it is just walking /proc and for the ppid adds a strstr and atoi(str). Something else is at play. I'll take a look.
David
Em Wed, Apr 08, 2015 at 07:48:23AM -0600, David Ahern escreveu:
Quoted 10 lines · Arnaldo Carvalho de Melo
Quoted 5 lines
Quoted 5 lines
I also noticed that if I set
/proc/sys/kernel/perf_event_paranoidto -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...odd. because this has nothing to do with
perf_events; it is just walking /proc and for the ppid adds a strstr and atoi(str). Something else is at play. I'll take a look.
You are correcly setting the pid values, that will have effects when using findnew, i.e. threads will be added to the rbtree, which causes allocations, etc I.e. it is probably triggering a dormant bug
- Arnaldo
Em Wed, Apr 08, 2015 at 11:14:27AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 8 lines
Em Wed, Apr 08, 2015 at 07:48:23AM -0600, David Ahern escreveu:
Quoted 7 lines · Arnaldo Carvalho de Melo
Quoted 5 lines
Quoted 5 lines
I also noticed that if I set
/proc/sys/kernel/perf_event_paranoidto -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...
Quoted 7 lines
Quoted 3 lines
odd. because this has nothing to do with
perf_events; it is just walking /proc and for the ppid adds a strstr and atoi(str). Something else is at play. I'll take a look.You are correcly setting the pid values, that will have effects when using findnew, i.e. threads will be added to the rbtree, which causes allocations, etc I.e. it is probably triggering a dormant bug
Sitting there, will stop chrome to see if the problem is triggered by it...
Program received signal SIGTERM, Terminated. 0x00000000004abc39 in rb_next (node=0x16202d20) at ../../lib/rbtree.c:457
457 while (node->rb_left)
(gdb) bt
#0 0x00000000004abc39 in rb_next (node=0x16202d20) at ../../lib/rbtree.c:457
#1 0x00000000004ccd27 in map_groups__fixup_overlappings (mg=0x12745740, map=0x16676210, fp=0x7ffff5a711c0 <_IO_2_1_stderr_>) at util/map.c:643
#2 0x00000000004d457d in thread__insert_map (thread=0x12745680, map=0x16676210) at util/thread.c:177
#3 0x00000000004c8de3 in machine__process_mmap2_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1204
#4 0x00000000004c92c3 in machine__process_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1329
#5 0x0000000000494a81 in perf_event__process (tool=0x0, event=0xa59410, sample=0x870160 <synth_sample>, machine=0xa28570) at util/event.c:778
#6 0x0000000000493606 in perf_event__synthesize_mmap_events (tool=0x0, event=0xa59410, pid=2513, tgid=2513, process=0x494a42 <perf_event__process>, machine=0xa28570, mmap_data=false) at util/event.c:302
#7 0x0000000000493b7b in __event__synthesize_thread (comm_event=0xa2a820, mmap_event=0xa59410, fork_event=0xa2a6a0, pid=2513, full=1, process=0x494a42 <perf_event__process>, tool=0x0, machine=0xa28570, mmap_data=false)
at util/event.c:429#8 0x0000000000493fe8 in perf_event__synthesize_threads (tool=0x0, process=0x494a42 <perf_event__process>, machine=0xa28570, mmap_data=false) at util/event.c:550
#9 0x00000000004ca82e in __machine__synthesize_threads (machine=0xa28570, tool=0x0, target=0x7fffffffb008, threads=0x94e490, process=0x494a42 <perf_event__process>, data_mmap=false) at util/machine.c:1781 #10 0x000000000043ae4c in machine__synthesize_threads (machine=0xa28570, target=0x7fffffffb008, threads=0x94e490, data_mmap=false) at util/machine.h:218
#11 0x000000000043d744 in __cmd_top (top=0x7fffffffaf80) at builtin-top.c:952
#12 0x000000000043ea4f in cmd_top (argc=0, argv=0x7fffffffe3d0, prefix=0x0) at builtin-top.c:1267
#13 0x0000000000489132 in run_builtin (p=0x86f128 <commands+264>, argc=1, argv=0x7fffffffe3d0) at perf.c:370
#14 0x0000000000489391 in handle_internal_command (argc=1, argv=0x7fffffffe3d0) at perf.c:429
#15 0x00000000004894dd in run_argv (argcp=0x7fffffffe22c, argv=0x7fffffffe220) at perf.c:473
#16 0x000000000048983e in main (argc=1, argv=0x7fffffffe3d0) at perf.c:588
(gdb)
(gdb) fr 3
#3 0x00000000004c8de3 in machine__process_mmap2_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1204
1204 thread__insert_map(thread, map);
(gdb) print *thread
$1 = {{rb_node = {__rb_parent_color = 37177040, rb_right = 0x0, rb_left = 0x0}, node = {next = 0x23746d0, prev = 0x0}}, mg = 0x12745740, pid_ = 2513, tid = 2513, ppid = 2480, cpu = -1, refcnt = 2, shortname = "\000\000",
comm_set = true, dead = false, comm_list = {next = 0x16f948b0, prev = 0x12745710}, comm_len = 0, db_id = 0, priv = 0x0, ts = 0x0}
(gdb) print event->header
$4 = {type = 10, misc = 2, size = 104}
(gdb)
[acme@zoo linux]$ ps ax|grep 2513
2513 ? S 0:00 /opt/google/chrome/chrome --type=gpu-broker
[acme@zoo linux]$ grep xp /proc/2513/maps
7fac6ac94000-7fac6ac9c000 r-xp 00000000 fd:01 1187628 /usr/lib64/libpciaccess.so.0.11.1 7fac6ae9e000-7fac6aeaa000 r-xp 00000000 fd:01 1184221 /usr/lib64/libdrm_radeon.so.1.0.1 7fac6b0ab000-7fac6b0b1000 r-xp 00000000 fd:01 1184218 /usr/lib64/libdrm_nouveau.so.2.0.0 7fac6b2b2000-7fac6b2d2000 r-xp 00000000 fd:01 1184215 /usr/lib64/libdrm_intel.so.1.0.0 7fac6b4d3000-7fac6b9ef000 r-xp 00000000 fd:01 1442035 /usr/lib64/dri/i965_dri.so 7fac6bc1d000-7fac6bc2d000 r-xp 00000000 fd:01 1192079 /usr/lib64/libudev.so.1.4.0 7fac73553000-7fac735ca000 r-xp 00000000 fd:01 1185534 /usr/lib64/libfreebl3.so 7fac737d0000-7fac737d3000 r-xp 00000000 fd:01 1187499 /usr/lib64/libkeyutils.so.1.5 7fac739d4000-7fac739e1000 r-xp 00000000 fd:01 1189618 /usr/lib64/libkrb5support.so.0.1 7fac73be2000-7fac73be7000 r-xp 00000000 fd:01 1188148 /usr/lib64/libXxf86vm.so.1.0.0 7fac73de8000-7fac73dff000 r-xp 00000000 fd:01 1188964 /usr/lib64/libxcb-glx.so.0.0.0 7fac74002000-7fac74004000 r-xp 00000000 fd:01 1188124 /usr/lib64/libXau.so.6.0.0 7fac74206000-7fac74211000 r-xp 00000000 fd:01 1179773 /usr/lib64/libdrm.so.2.4.0 7fac74412000-7fac74437000 r-xp 00000000 fd:01 1190780 /usr/lib64/libglapi.so.0.0.0 7fac7463c000-7fac7464a000 r-xp 00000000 fd:01 1188942 /usr/lib64/libwayland-server.so.0.1.0 7fac7484c000-7fac74857000 r-xp 00000000 fd:01 1188939 /usr/lib64/libwayland-client.so.0.1.0 7fac74a59000-7fac74a66000 r-xp 00000000 fd:01 1187278 /usr/lib64/libgbm.so.1.0.0 7fac74c67000-7fac74c6a000 r-xp 00000000 fd:01 1188970 /usr/lib64/libxcb-shape.so.0.0.0 7fac74e6b000-7fac74e71000 r-xp 00000000 fd:01 1188976 /usr/lib64/libxcb-xfixes.so.0.0.0 7fac75072000-7fac75076000 r-xp 00000000 fd:01 1188963 /usr/lib64/libxcb-dri2.so.0.0.0 7fac75277000-7fac75278000 r-xp 00000000 fd:01 1188121 /usr/lib64/libX11-xcb.so.1.0.0 7fac75479000-7fac75493000 r-xp 00000000 fd:01 1188433 /usr/lib64/libgraphite2.so.3.0.1 7fac75695000-7fac756b9000 r-xp 00000000 fd:01 1184295 /usr/lib64/liblzma.so.5.0.99 7fac758ba000-7fac7591f000 r-xp 00000000 fd:01 1190568 /usr/lib64/libpcre.so.1.2.1 7fac75b20000-7fac75b22000 r-xp 00000000 fd:01 1188135 /usr/lib64/libXinerama.so.1.0.0 7fac75d23000-7fac75d43000 r-xp 00000000 fd:01 1188169 /usr/lib64/libatk-1.0.so.0.21009.1 7fac75f46000-7fac75f4e000 r-xp 00000000 fd:01 1188257 /usr/lib64/libcrypt-2.18.so 7fac7617d000-7fac7618d000 r-xp 00000000 fd:01 1188177 /usr/lib64/libavahi-client.so.3.2.9 7fac7638e000-7fac7639a000 r-xp 00000000 fd:01 1188178 /usr/lib64/libavahi-common.so.3.5.3 7fac7659b000-7fac76758000 r-xp 00000000 fd:01 1180383 /usr/lib64/libcrypto.so.1.0.1e 7fac76983000-7fac769e5000 r-xp 00000000 fd:01 1185907 /usr/lib64/libssl.so.1.0.1e 7fac76bef000-7fac76bf2000 r-xp 00000000 fd:01 1185817 /usr/lib64/libcom_err.so.2.1 7fac76df3000-7fac76e25000 r-xp 00000000 fd:01 1187372 /usr/lib64/libk5crypto.so.3.1 7fac77028000-7fac770f8000 r-xp 00000000 fd:01 1187503 /usr/lib64/libkrb5.so.3.3 7fac77308000-7fac7734f000 r-xp 00000000 fd:01 1189531 /usr/lib64/libgssapi_krb5.so.2.2 7fac77552000-7fac775e2000 r-xp 00000000 fd:01 1190785 /usr/lib64/libGL.so.1.2.0 7fac777e6000-7fac77806000 r-xp 00000000 fd:01 1188983 /usr/lib64/libxcb.so.1.1.0 7fac77a07000-7fac77a10000 r-xp 00000000 fd:01 1188967 /usr/lib64/libxcb-render.so.0.0.0 7fac77c11000-7fac77c13000 r-xp 00000000 fd:01 1188971 /usr/lib64/libxcb-shm.so.0.0.0 7fac77e14000-7fac77e3c000 r-xp 00000000 fd:01 1190790 /usr/lib64/libEGL.so.1.0.0 7fac7803e000-7fac780e0000 r-xp 00000000 fd:01 1187639 /usr/lib64/libpixman-1.so.0.30.0 7fac782e8000-7fac7833c000 r-xp 00000000 fd:01 1187448 /usr/lib64/libharfbuzz.so.0.927.0 7fac7853e000-7fac7853f000 r-xp 00000000 fd:01 1188489 /usr/lib64/libgthread-2.0.so.0.3800.2 7fac78740000-7fac78754000 r-xp 00000000 fd:01 1187618 /usr/lib64/libpangoft2-1.0.so.0.3600.1 7fac78956000-7fac78988000 r-xp 00000000 fd:01 1184251 /usr/lib64/libpng16.so.16.6.0 7fac78b89000-7fac78b9f000 r-xp 00000000 fd:01 1189224 /usr/lib64/libresolv-2.18.so 7fac78da3000-7fac78dc4000 r-xp 00000000 fd:01 1187969 /usr/lib64/libselinux.so.1 7fac78fc7000-7fac78fdc000 r-xp 00000000 fd:01 1188999 /usr/lib64/libz.so.1.2.8 7fac791dd000-7fac79203000 r-xp 00000000 fd:01 1188266 /usr/lib64/libdbus-glib-1.so.2.2.2 7fac79405000-7fac79408000 r-xp 00000000 fd:01 1188412 /usr/lib64/libgmodule-2.0.so.0.3800.2 7fac79609000-7fac7960c000 r-xp 00000000 fd:01 1189435 /usr/lib64/libplds4.so 7fac7980d000-7fac79814000 r-xp 00000000 fd:01 1188331 /usr/lib64/libffi.so.6.0.1 7fac79a15000-7fac79bc9000 r-xp 00000000 fd:01 1187054 /usr/lib64/libc-2.18.so 7fac79dd3000-7fac79de8000 r-xp 00000000 fd:01 1183174 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 7fac79fe9000-7fac7a0d2000 r-xp 00000000 fd:01 1183177 /usr/lib64/libstdc++.so.6.0.19 7fac7a2f1000-7fac7a2f3000 r-xp 00000000 fd:01 1190263 /usr/lib64/libXss.so.1.0.0 7fac7a4f5000-7fac7a518000 r-xp 00000000 fd:01 1187313 /usr/lib64/libgdk_pixbuf-2.0.so.0.3000.3 7fac7a719000-7fac7a7d5000 r-xp 00000000 fd:01 1185350 /usr/lib64/libgdk-x11-2.0.so.0.2400.27 7fac7a9da000-7fac7ae52000 r-xp 00000000 fd:01 1185352 /usr/lib64/libgtk-x11-2.0.so.0.2400.27 7fac7b060000-7fac7b0a6000 r-xp 00000000 fd:01 1185836 /usr/lib64/libdbus-1.so.3.7.12 7fac7b2a8000-7fac7b2c0000 r-xp 00000000 fd:01 1188678 /usr/lib64/libpthread-2.18.so 7fac7b4c5000-7fac7b539000 r-xp 00000000 fd:01 1185342 /usr/lib64/libcups.so.2 7fac7b742000-7fac7b769000 r-xp 00000000 fd:01 1188325 /usr/lib64/libexpat.so.1.6.0 7fac7b769000-7fac7b969000 ---p 00027000 fd:01 1188325 /usr/lib64/libexpat.so.1.6.0 7fac7b969000-7fac7b96b000 r--p 00027000 fd:01 1188325 /usr/lib64/libexpat.so.1.6.0 7fac7b96b000-7fac7b96c000 rw-p 00029000 fd:01 1188325 /usr/lib64/libexpat.so.1.6.0 7fac7b96c000-7fac7b975000 r-xp 00000000 fd:01 1188139 /usr/lib64/libXrandr.so.2.2.0 7fac7bb76000-7fac7bb7b000 r-xp 00000000 fd:01 1188142 /usr/lib64/libXtst.so.6.1.0 7fac7bd7c000-7fac7bd7e000 r-xp 00000000 fd:01 1188127 /usr/lib64/libXdamage.so.1.1.0 7fac7bf7f000-7fac7c084000 r-xp 00000000 fd:01 1188374 /usr/lib64/libm-2.18.so 7fac7c286000-7fac7c367000 r-xp 00000000 fd:01 1188164 /usr/lib64/libasound.so.2.0.0 7fac7c56f000-7fac7c571000 r-xp 00000000 fd:01 1188125 /usr/lib64/libXcomposite.so.1.0.0 7fac7c772000-7fac7c77b000 r-xp 00000000 fd:01 1188140 /usr/lib64/libXrender.so.1.3.0 7fac7c97c000-7fac7c981000 r-xp 00000000 fd:01 1188131 /usr/lib64/libXfixes.so.3.1.0 7fac7cb82000-7fac7cb93000 r-xp 00000000 fd:01 1188130 /usr/lib64/libXext.so.6.4.0 7fac7cd94000-7fac7cd9e000 r-xp 00000000 fd:01 1188126 /usr/lib64/libXcursor.so.1.0.2 7fac7cf9f000-7fac7cfae000 r-xp 00000000 fd:01 1183298 /usr/lib64/libXi.so.6.1.0 7fac7d1af000-7fac7d2e7000 r-xp 00000000 fd:01 1188122 /usr/lib64/libX11.so.6.3.0 7fac7d4ed000-7fac7d535000 r-xp 00000000 fd:01 1187616 /usr/lib64/libpango-1.0.so.0.3600.1 7fac7d738000-7fac7d859000 r-xp 00000000 fd:01 1187140 /usr/lib64/libcairo.so.2.11400.0 7fac7da5f000-7fac7da6b000 r-xp 00000000 fd:01 1187617 /usr/lib64/libpangocairo-1.0.so.0.3600.1 7fac7dc6c000-7fac7dd0b000 r-xp 00000000 fd:01 1187649 /usr/lib64/libfreetype.so.6.10.2 7fac7df11000-7fac7df4c000 r-xp 00000000 fd:01 1187277 /usr/lib64/libfontconfig.so.1.8.0 7fac7e14e000-7fac7e2ac000 r-xp 00000000 fd:01 1188399 /usr/lib64/libgio-2.0.so.0.3800.2 7fac7e4b3000-7fac7e4e3000 r-xp 00000000 fd:01 1188370 /usr/lib64/libgconf-2.so.4.1.5 7fac7e6e5000-7fac7e71f000 r-xp 00000000 fd:01 1189378 /usr/lib64/libnspr4.so 7fac7e923000-7fac7e927000 r-xp 00000000 fd:01 1189431 /usr/lib64/libplc4.so 7fac7eb28000-7fac7eb4c000 r-xp 00000000 fd:01 1192057 /usr/lib64/libsmime3.so 7fac7ed4f000-7fac7ed74000 r-xp 00000000 fd:01 1189436 /usr/lib64/libnssutil3.so 7fac7ef7b000-7fac7f099000 r-xp 00000000 fd:01 1191883 /usr/lib64/libnss3.so 7fac7f2a2000-7fac7f3cb000 r-xp 00000000 fd:01 1188407 /usr/lib64/libglib-2.0.so.0.3800.2 7fac7f5ce000-7fac7f61d000 r-xp 00000000 fd:01 1188425 /usr/lib64/libgobject-2.0.so.0.3800.2 7fac7f81f000-7fac7f822000 r-xp 00000000 fd:01 1188290 /usr/lib64/libdl-2.18.so 7fac7fa23000-7fac7fa2a000 r-xp 00000000 fd:01 1189264 /usr/lib64/librt-2.18.so 7fac7fc2b000-7fac7fc4b000 r-xp 00000000 fd:01 1187898 /usr/lib64/ld-2.18.so 7fac7fe48000-7fac7fe49000 rwxp 00000000 00:00 0 7fac7fe4d000-7fac84fa0000 r-xp 00000000 fd:01 917553 /opt/google/chrome/chrome 7ffdd8dea000-7ffdd8dec000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
[acme@zoo linux]$
Em Wed, Apr 08, 2015 at 11:34:37AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 13 lines
Em Wed, Apr 08, 2015 at 11:14:27AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 12 lines
Em Wed, Apr 08, 2015 at 07:48:23AM -0600, David Ahern escreveu:
Quoted 10 lines · Arnaldo Carvalho de Melo
Quoted 5 lines
Quoted 5 lines
I also noticed that if I set
/proc/sys/kernel/perf_event_paranoidto -1 to run it as !root, then the problem "goes away", which I think probably is explained by, as !root, not being able to parse some of the /proc files for existing threads and thus not triggering the bug, still investigating...odd. because this has nothing to do with
perf_events; it is just walking /proc and for the ppid adds a strstr and atoi(str). Something else is at play. I'll take a look.
Quoted 5 lines
Quoted 3 lines
You are correcly setting the pid values, that will have effects when using findnew, i.e. threads will be added to the rbtree, which causes allocations, etc I.e. it is probably triggering a dormant bug
Sitting there, will stop chrome to see if the problem is triggered by it...
Yeah, stopping chrome gets the system to a state that doesn't make it trigger the bug, why it works when your patch is reverted still a puzzle...
- Arnaldo