Skip to contentSkip to conversation

perf: perf sched lat segfault - bisected

DISCUSSION Mike Galbraith 4 participants 16 messages Oct 8, 2013 perf events

lore
Share & tools
lore
#0 lore raw

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

The bisect winner:

marge:/usr/local/src/kernel/linux-3.x.git # git bisect good 5c5e854bc760a2e2c878df3cfcf2afa4febcd511 is the first bad commit commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511

Author: Stephane Eranian <[email protected]>
Date:   Wed Aug 21 12:10:25 2013 +0200

perf tools: Add attr->mmap2 support

This patch adds support for the new PERF_RECORD_MMAP2 record type exposed by the kernel. This is an extended PERF_RECORD_MMAP record.

It adds for each file-backed mapping the device major, minor number and the inode number and generation.

This triplet uniquely identifies the source of a file-backed mapping. It can be used to detect identical virtual mappings between processes, for instance.

The patch will prefer MMAP2 over MMAP.

[ Cope with 314add6 "Change machine__findnew_thread() to set thread pid",
fix 'perf test' regression test entry affected,
use perf_missing_features.mmap2 to fallback to not using .mmap2 in older kernels, so that new tools can work with kernels where this feature is not present ]

:040000 040000 2d600afbdd11b323bbcd00a4c6b50ee82a604acc d2a81503d08aeea994e203245358ae939eb5e787 M tools

git bisect start
# bad: [8b5ede69d24db939f52b47effff2f6fe1e83e08b] powerpc/irq: Don't switch to irq stack from softirq stack
git bisect bad 8b5ede69d24db939f52b47effff2f6fe1e83e08b
# good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
git bisect good 6e4664525b1db28f8c4e1130957f70a94c19213e
# good: [57d730924d5cc2c3e280af16a9306587c3a511db] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 57d730924d5cc2c3e280af16a9306587c3a511db
# good: [d8cacd3a259bf522ea5e6c4c60eba67ba22f599c] Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good d8cacd3a259bf522ea5e6c4c60eba67ba22f599c
# good: [b55ee2816ed6d8f8a00d4badab0e3642ffbac19f] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good b55ee2816ed6d8f8a00d4badab0e3642ffbac19f
# bad: [d45004f9947e521fc9e812bc7937ed6e0328793b] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
git bisect bad d45004f9947e521fc9e812bc7937ed6e0328793b
# bad: [9bf12df31f282e845b3dfaac1e5d5376a041da22] Merge git://git.kvack.org/~bcrl/aio-next
git bisect bad 9bf12df31f282e845b3dfaac1e5d5376a041da22
# bad: [26935fb06ee88f1188789807687c03041f3c70d9] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 26935fb06ee88f1188789807687c03041f3c70d9
# bad: [5223161dc0f5e44fbf3d5e42d23697b6796cdf4e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/cooloney/linux-leds
git bisect bad 5223161dc0f5e44fbf3d5e42d23697b6796cdf4e
# bad: [02b9735c12892e04d3e101b06e4c6d64a814f566] Merge tag 'pm+acpi-fixes-3.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad 02b9735c12892e04d3e101b06e4c6d64a814f566
# good: [0df03a30c333d67ee9b4c37f32d423624f48fe05] Merge branch 'pm-cpufreq'
git bisect good 0df03a30c333d67ee9b4c37f32d423624f48fe05
# bad: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
git bisect bad 5c5e854bc760a2e2c878df3cfcf2afa4febcd511
# good: [41817815666e4a6f302dad1fea47cbe64cc45e40] perf trace: Check if MAP_32BIT is defined
git bisect good 41817815666e4a6f302dad1fea47cbe64cc45e40
# good: [ccf53eac2097fb47bc40875ffb22c2d10fa8c46c] perf trace: Handle perf.data files with no tracepoints
git bisect good ccf53eac2097fb47bc40875ffb22c2d10fa8c46c
# good: [e71aa28312b208a14cd87fa61e941ac8c85072f4] perf kvm: Fix sample_type manipulation
git bisect good e71aa28312b208a14cd87fa61e941ac8c85072f4

# first bad commit: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
#1 lore raw
Quoted 6 lines · Mike Galbraith

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

I don't understand the test case here.
If I do this, perf sched lat complains of no trace records and no kernel crash. Could you provide the full cmdline?

Quoted 73 lines

The bisect winner:

marge:/usr/local/src/kernel/linux-3.x.git # git bisect good 5c5e854bc760a2e2c878df3cfcf2afa4febcd511 is the first bad commit commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511

Author: Stephane Eranian <[email protected]>
Date:   Wed Aug 21 12:10:25 2013 +0200

perf tools: Add attr->mmap2 support

This patch adds support for the new PERF_RECORD_MMAP2 record type exposed by the kernel. This is an extended PERF_RECORD_MMAP record.

It adds for each file-backed mapping the device major, minor number and the inode number and generation.

This triplet uniquely identifies the source of a file-backed mapping. It can be used to detect identical virtual mappings between processes, for instance.

The patch will prefer MMAP2 over MMAP.

[ Cope with 314add6 "Change machine__findnew_thread() to set thread pid",
fix 'perf test' regression test entry affected,
use perf_missing_features.mmap2 to fallback to not using .mmap2 in older kernels, so that new tools can work with kernels where this feature is not present ]

:040000 040000 2d600afbdd11b323bbcd00a4c6b50ee82a604acc d2a81503d08aeea994e203245358ae939eb5e787 M tools

git bisect start
# bad: [8b5ede69d24db939f52b47effff2f6fe1e83e08b] powerpc/irq: Don't switch to irq stack from softirq stack
git bisect bad 8b5ede69d24db939f52b47effff2f6fe1e83e08b
# good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
git bisect good 6e4664525b1db28f8c4e1130957f70a94c19213e
# good: [57d730924d5cc2c3e280af16a9306587c3a511db] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 57d730924d5cc2c3e280af16a9306587c3a511db
# good: [d8cacd3a259bf522ea5e6c4c60eba67ba22f599c] Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good d8cacd3a259bf522ea5e6c4c60eba67ba22f599c
# good: [b55ee2816ed6d8f8a00d4badab0e3642ffbac19f] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good b55ee2816ed6d8f8a00d4badab0e3642ffbac19f
# bad: [d45004f9947e521fc9e812bc7937ed6e0328793b] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
git bisect bad d45004f9947e521fc9e812bc7937ed6e0328793b
# bad: [9bf12df31f282e845b3dfaac1e5d5376a041da22] Merge git://git.kvack.org/~bcrl/aio-next
git bisect bad 9bf12df31f282e845b3dfaac1e5d5376a041da22
# bad: [26935fb06ee88f1188789807687c03041f3c70d9] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 26935fb06ee88f1188789807687c03041f3c70d9
# bad: [5223161dc0f5e44fbf3d5e42d23697b6796cdf4e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/cooloney/linux-leds
git bisect bad 5223161dc0f5e44fbf3d5e42d23697b6796cdf4e
# bad: [02b9735c12892e04d3e101b06e4c6d64a814f566] Merge tag 'pm+acpi-fixes-3.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad 02b9735c12892e04d3e101b06e4c6d64a814f566
# good: [0df03a30c333d67ee9b4c37f32d423624f48fe05] Merge branch 'pm-cpufreq'
git bisect good 0df03a30c333d67ee9b4c37f32d423624f48fe05
# bad: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
git bisect bad 5c5e854bc760a2e2c878df3cfcf2afa4febcd511
# good: [41817815666e4a6f302dad1fea47cbe64cc45e40] perf trace: Check if MAP_32BIT is defined
git bisect good 41817815666e4a6f302dad1fea47cbe64cc45e40
# good: [ccf53eac2097fb47bc40875ffb22c2d10fa8c46c] perf trace: Handle perf.data files with no tracepoints
git bisect good ccf53eac2097fb47bc40875ffb22c2d10fa8c46c
# good: [e71aa28312b208a14cd87fa61e941ac8c85072f4] perf kvm: Fix sample_type manipulation
git bisect good e71aa28312b208a14cd87fa61e941ac8c85072f4

# first bad commit: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
#2 lore raw
Quoted 10 lines · Stephane Eranian
Quoted 6 lines · Mike Galbraith

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

I don't understand the test case here.
If I do this, perf sched lat complains of no trace records and no kernel crash. Could you provide the full cmdline?

Hm. That was it, just like so.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.763 MB perf.data (~33348 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched lat Segmentation fault
marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

-Mike

#3 lore raw
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Stephane Eranian
Quoted 6 lines · Mike Galbraith

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

I don't understand the test case here.
If I do this, perf sched lat complains of no trace records and no kernel crash. Could you provide the full cmdline?

Quoted 8 lines

Hm. That was it, just like so.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.763 MB perf.data (~33348 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched lat Segmentation fault
marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

commit 72f3bdf7fd1003373b454bf923d3a196bbc36600

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:37 2013 +0200
    Revert "perf tools: Add attr->mmap2 support"
    
    This reverts commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511.

commit a8af8531d08d3edb37a7d67d1a4dc2e1f2f4da58

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:23 2013 +0200

Revert "perf trace: Add mmap2 handler"

This reverts commit 384c671e33a7ddf905f2c60b433b9883f0e5a605.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.647 MB perf.data (~28289 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # perf sched lat marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

(emitted this)

---------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Average delay ms | Maximum delay ms | Maximum delay at |
---------------------------------------------------------------------------------------------------------------

  kworker/3:2:383       |      0.073 ms |        1 | avg:    0.020 ms | max:    0.020 ms | max at: 7197.856052 s
  sleep:24717           |      1.163 ms |        3 | avg:    0.011 ms | max:    0.022 ms | max at: 7196.884445 s
  kworker/2:1:36        |      0.097 ms |        1 | avg:    0.007 ms | max:    0.007 ms | max at: 7197.796023 s
  haveged:3547          |      0.023 ms |        2 | avg:    0.006 ms | max:    0.009 ms | max at: 7197.796345 s
  rcu_sched:9           |      0.126 ms |       14 | avg:    0.006 ms | max:    0.021 ms | max at: 7196.897051 s
  ntpd:4185             |      0.034 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.746409 s
  kded4:4594            |      0.058 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.422130 s
  evolution:15686       |      0.065 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7196.915715 s
  pickup:21579          |      0.062 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.329185 s
  JS Watchdog:20053     |      0.020 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.623752 s
  perf:24716            |      3.330 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.885690 s
  plasma-desktop:4685   |      1.632 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7196.933762 s
  master:4307           |      0.104 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7197.329136 s
  Xorg:4331             |      0.485 ms |        4 | avg:    0.003 ms | max:    0.005 ms | max at: 7197.010801 s
  ksoftirqd/2:18        |      0.014 ms |        1 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.897039 s
  alsa-sink:5282        |      0.000 ms |      200 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.905149 s
  migration/3:22        |      0.000 ms |        1 | avg:    0.002 ms | max:    0.002 ms | max at: 7196.884423 s

-----------------------------------------------------------------------------------------
TOTAL: | 7.285 ms | 237 |
---------------------------------------------------

#4 lore raw

Mike,

Ok, I missed the sched record, let me track this down. Note that there was a second commit to fix this one by Arnaldo. Do you also have it in your kernel?

51131c4 perf: Fix up MMAP2 buffer space reservation

I will try to reproduce your problem ASAP.

Quoted 71 lines · Mike Galbraith
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Stephane Eranian
Quoted 6 lines · Mike Galbraith

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

I don't understand the test case here.
If I do this, perf sched lat complains of no trace records and no kernel crash. Could you provide the full cmdline?

Quoted 8 lines

Hm. That was it, just like so.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.763 MB perf.data (~33348 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched lat Segmentation fault
marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

commit 72f3bdf7fd1003373b454bf923d3a196bbc36600

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:37 2013 +0200
    Revert "perf tools: Add attr->mmap2 support"

    This reverts commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511.

commit a8af8531d08d3edb37a7d67d1a4dc2e1f2f4da58

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:23 2013 +0200

Revert "perf trace: Add mmap2 handler"

This reverts commit 384c671e33a7ddf905f2c60b433b9883f0e5a605.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.647 MB perf.data (~28289 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # perf sched lat marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

(emitted this)

---------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Average delay ms | Maximum delay ms | Maximum delay at |
---------------------------------------------------------------------------------------------------------------

  kworker/3:2:383       |      0.073 ms |        1 | avg:    0.020 ms | max:    0.020 ms | max at: 7197.856052 s
  sleep:24717           |      1.163 ms |        3 | avg:    0.011 ms | max:    0.022 ms | max at: 7196.884445 s
  kworker/2:1:36        |      0.097 ms |        1 | avg:    0.007 ms | max:    0.007 ms | max at: 7197.796023 s
  haveged:3547          |      0.023 ms |        2 | avg:    0.006 ms | max:    0.009 ms | max at: 7197.796345 s
  rcu_sched:9           |      0.126 ms |       14 | avg:    0.006 ms | max:    0.021 ms | max at: 7196.897051 s
  ntpd:4185             |      0.034 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.746409 s
  kded4:4594            |      0.058 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.422130 s
  evolution:15686       |      0.065 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7196.915715 s
  pickup:21579          |      0.062 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.329185 s
  JS Watchdog:20053     |      0.020 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.623752 s
  perf:24716            |      3.330 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.885690 s
  plasma-desktop:4685   |      1.632 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7196.933762 s
  master:4307           |      0.104 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7197.329136 s
  Xorg:4331             |      0.485 ms |        4 | avg:    0.003 ms | max:    0.005 ms | max at: 7197.010801 s
  ksoftirqd/2:18        |      0.014 ms |        1 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.897039 s
  alsa-sink:5282        |      0.000 ms |      200 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.905149 s
  migration/3:22        |      0.000 ms |        1 | avg:    0.002 ms | max:    0.002 ms | max at: 7196.884423 s

-----------------------------------------------------------------------------------------
TOTAL: | 7.285 ms | 237 |
---------------------------------------------------

#5 lore raw

Mike,

I found the problem in perf. The tool->mmap2 pointer is NULL for sched tool. Let me send you a patch to fix this.

Quoted 83 lines · Stephane Eranian

Mike,

Ok, I missed the sched record, let me track this down. Note that there was a second commit to fix this one by Arnaldo. Do you also have it in your kernel?

51131c4 perf: Fix up MMAP2 buffer space reservation

I will try to reproduce your problem ASAP.

Quoted 71 lines · Mike Galbraith
Quoted 12 lines · Mike Galbraith
Quoted 10 lines · Stephane Eranian
Quoted 6 lines · Mike Galbraith

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

I don't understand the test case here.
If I do this, perf sched lat complains of no trace records and no kernel crash. Could you provide the full cmdline?

Quoted 8 lines

Hm. That was it, just like so.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.763 MB perf.data (~33348 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched lat Segmentation fault
marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

commit 72f3bdf7fd1003373b454bf923d3a196bbc36600

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:37 2013 +0200
    Revert "perf tools: Add attr->mmap2 support"

    This reverts commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511.

commit a8af8531d08d3edb37a7d67d1a4dc2e1f2f4da58

Author: Mike Galbraith <[email protected]>
Date:   Tue Oct 8 12:18:23 2013 +0200

Revert "perf trace: Add mmap2 handler"

This reverts commit 384c671e33a7ddf905f2c60b433b9883f0e5a605.

marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # ./perf sched record -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.647 MB perf.data (~28289 samples) ] marge:/usr/local/src/kernel/linux-3.x.git/tools/perf # perf sched lat marge:/usr/local/src/kernel/linux-3.x.git/tools/perf #

(emitted this)

---------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Average delay ms | Maximum delay ms | Maximum delay at |
---------------------------------------------------------------------------------------------------------------

  kworker/3:2:383       |      0.073 ms |        1 | avg:    0.020 ms | max:    0.020 ms | max at: 7197.856052 s
  sleep:24717           |      1.163 ms |        3 | avg:    0.011 ms | max:    0.022 ms | max at: 7196.884445 s
  kworker/2:1:36        |      0.097 ms |        1 | avg:    0.007 ms | max:    0.007 ms | max at: 7197.796023 s
  haveged:3547          |      0.023 ms |        2 | avg:    0.006 ms | max:    0.009 ms | max at: 7197.796345 s
  rcu_sched:9           |      0.126 ms |       14 | avg:    0.006 ms | max:    0.021 ms | max at: 7196.897051 s
  ntpd:4185             |      0.034 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.746409 s
  kded4:4594            |      0.058 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.422130 s
  evolution:15686       |      0.065 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7196.915715 s
  pickup:21579          |      0.062 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.329185 s
  JS Watchdog:20053     |      0.020 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.623752 s
  perf:24716            |      3.330 ms |        1 | avg:    0.004 ms | max:    0.004 ms | max at: 7197.885690 s
  plasma-desktop:4685   |      1.632 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7196.933762 s
  master:4307           |      0.104 ms |        2 | avg:    0.004 ms | max:    0.005 ms | max at: 7197.329136 s
  Xorg:4331             |      0.485 ms |        4 | avg:    0.003 ms | max:    0.005 ms | max at: 7197.010801 s
  ksoftirqd/2:18        |      0.014 ms |        1 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.897039 s
  alsa-sink:5282        |      0.000 ms |      200 | avg:    0.003 ms | max:    0.003 ms | max at: 7196.905149 s
  migration/3:22        |      0.000 ms |        1 | avg:    0.002 ms | max:    0.002 ms | max at: 7196.884423 s

-----------------------------------------------------------------------------------------
TOTAL: | 7.285 ms | 237 |
---------------------------------------------------

#6 lore raw
Quoted 4 lines · Stephane Eranian

Mike,

I found the problem in perf. The tool->mmap2 pointer is NULL for sched tool. Let me send you a patch to fix this.

Ah, no need. It's probably something like so, which fixed it.

tools/perf/builtin-sched.c +1 -0
index d8c51b2f263f..a60d14dd18e7 100644
@@ -1658,6 +1658,7 @@ static struct perf_sched sched = {
                .comm            = perf_event__process_comm,
                .lost            = perf_event__process_lost,
                .fork            = perf_sched__process_fork_event,
+               .mmap2           = perf_event__process_mmap2,
                .ordered_samples = true,
        },
        .cmp_pid              = LIST_HEAD_INIT(sched.cmp_pid),
#7 lore raw

Mike,

I posted the fix now. It was somewhere else. But you got the idea. thanks for reporting the problem.

Quoted 22 lines · Mike Galbraith
Quoted 4 lines · Stephane Eranian

Mike,

I found the problem in perf. The tool->mmap2 pointer is NULL for sched tool. Let me send you a patch to fix this.

Ah, no need. It's probably something like so, which fixed it.

tools/perf/builtin-sched.c +1 -0
index d8c51b2f263f..a60d14dd18e7 100644
@@ -1658,6 +1658,7 @@ static struct perf_sched sched = {
                .comm            = perf_event__process_comm,
                .lost            = perf_event__process_lost,
                .fork            = perf_sched__process_fork_event,
+               .mmap2           = perf_event__process_mmap2,
                .ordered_samples = true,
        },
        .cmp_pid              = LIST_HEAD_INIT(sched.cmp_pid),
#8 lore raw
Quoted 3 lines · Stephane Eranian

Mike,

I posted the fix now.

I tested it just in case, works fine.

-Mike

#9 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Stephane Eranian

Mike,

I posted the fix now.

I tested it just in case, works fine.

Thanks.

Quoted 2 lines

-Mike

#10 lore raw
In reply to Stephane Eranian
Quoted 7 lines · Stephane Eranian

Mike,

Ok, I missed the sched record, let me track this down. Note that there was a second commit to fix this one by Arnaldo. Do you also have it in your kernel?

51131c4 perf: Fix up MMAP2 buffer space reservation

Not with that commit id, but..

commit d008d5258e9c1a1b7ee6547b8d444323aef331b3

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue Sep 10 10:24:05 2013 -0300

perf: Fix up MMAP2 buffer space reservation

-Mike

#11 lore raw
In reply to Mike Galbraith

Em Tue, Oct 08, 2013 at 10:41:55AM +0200, Mike Galbraith escreveu:

Quoted 7 lines

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

The bisect winner:

Can you try with today's tip/perf/urgent branch?

- Arnaldo

Quoted 70 lines

marge:/usr/local/src/kernel/linux-3.x.git # git bisect good 5c5e854bc760a2e2c878df3cfcf2afa4febcd511 is the first bad commit commit 5c5e854bc760a2e2c878df3cfcf2afa4febcd511

Author: Stephane Eranian <[email protected]>
Date:   Wed Aug 21 12:10:25 2013 +0200

perf tools: Add attr->mmap2 support

This patch adds support for the new PERF_RECORD_MMAP2 record type exposed by the kernel. This is an extended PERF_RECORD_MMAP record.

It adds for each file-backed mapping the device major, minor number and the inode number and generation.

This triplet uniquely identifies the source of a file-backed mapping. It can be used to detect identical virtual mappings between processes, for instance.

The patch will prefer MMAP2 over MMAP.

[ Cope with 314add6 "Change machine__findnew_thread() to set thread pid",
fix 'perf test' regression test entry affected,
use perf_missing_features.mmap2 to fallback to not using .mmap2 in older kernels, so that new tools can work with kernels where this feature is not present ]

:040000 040000 2d600afbdd11b323bbcd00a4c6b50ee82a604acc d2a81503d08aeea994e203245358ae939eb5e787 M tools

git bisect start
# bad: [8b5ede69d24db939f52b47effff2f6fe1e83e08b] powerpc/irq: Don't switch to irq stack from softirq stack
git bisect bad 8b5ede69d24db939f52b47effff2f6fe1e83e08b
# good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
git bisect good 6e4664525b1db28f8c4e1130957f70a94c19213e
# good: [57d730924d5cc2c3e280af16a9306587c3a511db] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 57d730924d5cc2c3e280af16a9306587c3a511db
# good: [d8cacd3a259bf522ea5e6c4c60eba67ba22f599c] Merge tag 'virtio-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good d8cacd3a259bf522ea5e6c4c60eba67ba22f599c
# good: [b55ee2816ed6d8f8a00d4badab0e3642ffbac19f] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good b55ee2816ed6d8f8a00d4badab0e3642ffbac19f
# bad: [d45004f9947e521fc9e812bc7937ed6e0328793b] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
git bisect bad d45004f9947e521fc9e812bc7937ed6e0328793b
# bad: [9bf12df31f282e845b3dfaac1e5d5376a041da22] Merge git://git.kvack.org/~bcrl/aio-next
git bisect bad 9bf12df31f282e845b3dfaac1e5d5376a041da22
# bad: [26935fb06ee88f1188789807687c03041f3c70d9] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 26935fb06ee88f1188789807687c03041f3c70d9
# bad: [5223161dc0f5e44fbf3d5e42d23697b6796cdf4e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/cooloney/linux-leds
git bisect bad 5223161dc0f5e44fbf3d5e42d23697b6796cdf4e
# bad: [02b9735c12892e04d3e101b06e4c6d64a814f566] Merge tag 'pm+acpi-fixes-3.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect bad 02b9735c12892e04d3e101b06e4c6d64a814f566
# good: [0df03a30c333d67ee9b4c37f32d423624f48fe05] Merge branch 'pm-cpufreq'
git bisect good 0df03a30c333d67ee9b4c37f32d423624f48fe05
# bad: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
git bisect bad 5c5e854bc760a2e2c878df3cfcf2afa4febcd511
# good: [41817815666e4a6f302dad1fea47cbe64cc45e40] perf trace: Check if MAP_32BIT is defined
git bisect good 41817815666e4a6f302dad1fea47cbe64cc45e40
# good: [ccf53eac2097fb47bc40875ffb22c2d10fa8c46c] perf trace: Handle perf.data files with no tracepoints
git bisect good ccf53eac2097fb47bc40875ffb22c2d10fa8c46c
# good: [e71aa28312b208a14cd87fa61e941ac8c85072f4] perf kvm: Fix sample_type manipulation
git bisect good e71aa28312b208a14cd87fa61e941ac8c85072f4

# first bad commit: [5c5e854bc760a2e2c878df3cfcf2afa4febcd511] perf tools: Add attr->mmap2 support
#12 lore raw
Quoted 10 lines · Arnaldo Carvalho de Melo

Em Tue, Oct 08, 2013 at 10:41:55AM +0200, Mike Galbraith escreveu:

Quoted 7 lines

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

The bisect winner:

Can you try with today's tip/perf/urgent branch?

Works fine.

-Mike

#13 lore raw
Quoted 8 lines · Mike Galbraith
Quoted 7 lines · Arnaldo Carvalho de Melo

Em Tue, Oct 08, 2013 at 10:41:55AM +0200, Mike Galbraith escreveu:

Quoted 6 lines

Box: Q6600

Reproducer:
perf sched record -- sleep 1
perf sched lat (boom)

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

The bisect winner:

Can you try with today's tip/perf/urgent branch?

Works fine.

(except replay segfaults, but lord knows when that last worked)

#14 lore raw
Quoted 2 lines · Mike Galbraith

(except replay segfaults, but lord knows when that last worked)

Fairly recently actually. I ran it early summer adding my timehist command to builtin-sched.c. I'll take a look later today.

David

#15 lore raw
Quoted 6 lines · David Ahern
Quoted 2 lines · Mike Galbraith

(except replay segfaults, but lord knows when that last worked)

Fairly recently actually. I ran it early summer adding my timehist command to builtin-sched.c. I'll take a look later today.

pr_err() blows up in self_open_counters() when you hit 1024 fds. Too many threads to play back after recording make in perf directory. Playing a static load back works fine.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffedbc8700 (LWP 5270)]
buffered_vfprintf (s=0x7ffff301f700, format=0x65b9e0 "Error: sys_perf_event_open() syscall returned with %d (%s)\n", args=0x7fffedbc7a90) at vfprintf.c:2265
2265 result = INTUSE(_IO_vfprintf) (hp, format, args);
(gdb) where

#0  buffered_vfprintf (s=0x7ffff301f700, format=0x65b9e0 "Error: sys_perf_event_open() syscall returned with %d (%s)\n", args=0x7fffedbc7a90) at vfprintf.c:2265
#1  0x00007ffff2cd9dbe in _IO_vfprintf_internal (s=0x7ffff301f700, format=0x65b9e0 "Error: sys_perf_event_open() syscall returned with %d (%s)\n", ap=0x7fffedbc7a90) at vfprintf.c:1291
#2  0x00007ffff2d84148 in ___vfprintf_chk (fp=0x7ffff301f700, flag=1, format=<optimized out>, ap=<optimized out>) at vfprintf_chk.c:35
#3  0x000000000048d4f8 in vfprintf (__ap=0x7fffedbc7a90, __fmt=<optimized out>, __stream=<optimized out>) at /usr/include/bits/stdio2.h:128
#4  eprintf (level=<optimized out>, fmt=<optimized out>) at util/debug.c:29
#5  0x000000000043aa10 in self_open_counters () at builtin-sched.c:441
#6  thread_func (ctx=<optimized out>) at builtin-sched.c:476
#7  0x00007ffff79b1f05 in start_thread (arg=0x7fffedbc8700) at pthread_create.c:301
#8  0x00007ffff2d7010d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

(gdb)