Skip to contentSkip to conversation

perf: cannot read counts in per-process mode in 3.17-rcX

BUG In review Stephane Eranian 4 participants 8 messages Sep 8, 2014 – Sep 16, 2014 perf events

lore
Share & tools
lore
#0 lore raw

Hi,

It seems something is seriously broken with perf_events in 3.17-rcX. I have tried rc3, rc4. No way to get any counts out using perf stat in per-process mode. I am trying on Intel and the PMU is correctly detected:

$ perf stat -e cycles ls
      <not counted> cycles

It is not a permission problem. It is a read problem!

$ strace perf stat -e cycles ls

perf_event_open(0x27d7e20, 2261, -1, -1, 0x8 /* PERF_FLAG_??? */) = 3
write(6, "\0", 1) = 1
close(6) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2261 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2261, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn() = 2261
read(3, "", 24) = 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Looks like read() on the perf returns 0 (nothing read!).

It seems to work fine in system-wide mode. So I bet there were some recent changes in the way the events are stored when the process terminates.

I think the perf tool is not to blame here. I see the same problem with my libpfm4 toy examples.

Can you reproduce the problem on your system?

#1 lore raw
Quoted 13 lines · Stephane Eranian

Hi,

It seems something is seriously broken with perf_events in 3.17-rcX. I have tried rc3, rc4. No way to get any counts out using perf stat in per-process mode. I am trying on Intel and the PMU is correctly detected:

$ perf stat -e cycles ls
      <not counted> cycles

It is not a permission problem. It is a read problem!

$ strace perf stat -e cycles ls
Quoted 9 lines

perf_event_open(0x27d7e20, 2261, -1, -1, 0x8 /* PERF_FLAG_??? */) = 3
write(6, "\0", 1) = 1
close(6) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2261 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2261, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn() = 2261
read(3, "", 24) = 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ouch thats me.. sry :-\

the PERF_EVENT_STATE_EXIT check should not go to the read path.. could you please test attached patch?

jirka

---

kernel/events/core.c +1 -2
index d8cb4d2..6d1c9ce 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
 	 * error state (i.e. because it was pinned but it couldn't be
 	 * scheduled on to the CPU at some point).
 	 */
-	if ((event->state == PERF_EVENT_STATE_ERROR) ||
-	    (event->state == PERF_EVENT_STATE_EXIT))
+	if (event->state == PERF_EVENT_STATE_ERROR)
 		return 0;
 
 	if (count < event->read_size)
author Acked-by
#2 lore raw
Quoted 25 lines · Jiri Olsa
Quoted 13 lines · Stephane Eranian

Hi,

It seems something is seriously broken with perf_events in 3.17-rcX. I have tried rc3, rc4. No way to get any counts out using perf stat in per-process mode. I am trying on Intel and the PMU is correctly detected:

$ perf stat -e cycles ls
      <not counted> cycles

It is not a permission problem. It is a read problem!

$ strace perf stat -e cycles ls
Quoted 9 lines

perf_event_open(0x27d7e20, 2261, -1, -1, 0x8 /* PERF_FLAG_??? */) = 3
write(6, "\0", 1) = 1
close(6) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2261 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2261, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn() = 2261
read(3, "", 24) = 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Quoted 7 lines

ouch thats me.. sry :-\

the PERF_EVENT_STATE_EXIT check should not go to the read path.. could you please test attached patch?

jirka

Works for me again now. Thanks for the quick fix. Please push it upstream.

Quoted 17 lines

---

kernel/events/core.c +1 -2
index d8cb4d2..6d1c9ce 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
         * error state (i.e. because it was pinned but it couldn't be
         * scheduled on to the CPU at some point).
         */
-       if ((event->state == PERF_EVENT_STATE_ERROR) ||
-           (event->state == PERF_EVENT_STATE_EXIT))
+       if (event->state == PERF_EVENT_STATE_ERROR)
                return 0;
 
        if (count < event->read_size)
#3 lore raw

PATCH 1/1 perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

Quoted 28 lines · Stephane Eranian
Quoted 27 lines · Jiri Olsa
Quoted 13 lines · Stephane Eranian

Hi,

It seems something is seriously broken with perf_events in 3.17-rcX. I have tried rc3, rc4. No way to get any counts out using perf stat in per-process mode. I am trying on Intel and the PMU is correctly detected:

$ perf stat -e cycles ls
      <not counted> cycles

It is not a permission problem. It is a read problem!

$ strace perf stat -e cycles ls
Quoted 9 lines

perf_event_open(0x27d7e20, 2261, -1, -1, 0x8 /* PERF_FLAG_??? */) = 3
write(6, "\0", 1) = 1
close(6) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2261 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2261, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn() = 2261
read(3, "", 24) = 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ouch thats me.. sry :-\

Quoted 8 lines
Quoted 5 lines

the PERF_EVENT_STATE_EXIT check should not go to the read path.. could you please test attached patch?

jirka

Works for me again now. Thanks for the quick fix. Please push it upstream.

attached, thanks
jirka

---
Revert PERF_EVENT_STATE_EXIT check on read syscall path. It breaks standard way to read counter, which is to open the counter, wait for the monitored process to die and read the counter.

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/events/core.c +1 -2
index d8cb4d21a346..6d1c9ce1643e 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
 	 * error state (i.e. because it was pinned but it couldn't be
 	 * scheduled on to the CPU at some point).
 	 */
-	if ((event->state == PERF_EVENT_STATE_ERROR) ||
-	    (event->state == PERF_EVENT_STATE_EXIT))
+	if (event->state == PERF_EVENT_STATE_ERROR)
 		return 0;
 
 	if (count < event->read_size)
Signature · 1 line
1.8.3.1
maintainer
#4 lore raw

Re: [PATCH] perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

Quoted 5 lines · Jiri Olsa

Revert PERF_EVENT_STATE_EXIT check on read syscall path. It breaks standard way to read counter, which is to open the counter, wait for the monitored process to die and read the counter.

Quoted 9 lines
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>

Thanks!

Quoted 21 lines
---
 kernel/events/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/events/core.c +1 -2
index d8cb4d21a346..6d1c9ce1643e 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
 	 * error state (i.e. because it was pinned but it couldn't be
 	 * scheduled on to the CPU at some point).
 	 */
-	if ((event->state == PERF_EVENT_STATE_ERROR) ||
-	    (event->state == PERF_EVENT_STATE_EXIT))
+	if (event->state == PERF_EVENT_STATE_ERROR)
 		return 0;
 
 	if (count < event->read_size)
Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH] perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

Peter,

Quoted 16 lines · Peter Zijlstra
Quoted 5 lines · Jiri Olsa

Revert PERF_EVENT_STATE_EXIT check on read syscall path. It breaks standard way to read counter, which is to open the counter, wait for the monitored process to die and read the counter.

Quoted 9 lines
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
Quoted 1 line

Thanks!

Still don't see this fix in tip.git.

Quoted 21 lines
Quoted 21 lines
---
 kernel/events/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/events/core.c +1 -3
index d8cb4d21a346..6d1c9ce1643e 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
       * error state (i.e. because it was pinned but it couldn't be
       * scheduled on to the CPU at some point).
       */
-     if ((event->state == PERF_EVENT_STATE_ERROR) ||
-         (event->state == PERF_EVENT_STATE_EXIT))
+     if (event->state == PERF_EVENT_STATE_ERROR)
              return 0;
 
      if (count < event->read_size)
--

1.8.3.1

#6 lore raw

Re: [PATCH] perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

Quoted 22 lines · Stephane Eranian

Peter,

Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Jiri Olsa

Revert PERF_EVENT_STATE_EXIT check on read syscall path. It breaks standard way to read counter, which is to open the counter, wait for the monitored process to die and read the counter.

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
Quoted 3 lines
Quoted 1 line

Thanks!

Still don't see this fix in tip.git.

Still not there. This is a problem.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
 kernel/events/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/events/core.c +1 -3
index d8cb4d21a346..6d1c9ce1643e 100644
@@ -3600,8 +3600,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
       * error state (i.e. because it was pinned but it couldn't be
       * scheduled on to the CPU at some point).
       */
-     if ((event->state == PERF_EVENT_STATE_ERROR) ||
-         (event->state == PERF_EVENT_STATE_EXIT))
+     if (event->state == PERF_EVENT_STATE_ERROR)
              return 0;
 
      if (count < event->read_size)
--

1.8.3.1

#7 lore raw

[tip:perf/core] perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

In reply to Jiri Olsa
perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path
Commit-ID:  c88f2096136416b261bd3647cc260935f6e95805
Gitweb:     http://git.kernel.org/tip/c88f2096136416b261bd3647cc260935f6e95805
Author:     Jiri Olsa <[email protected]>
AuthorDate: Mon, 8 Sep 2014 16:31:07 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 16 Sep 2014 10:30:36 +0200

perf: Do not check PERF_EVENT_STATE_EXIT on syscall read path

Revert PERF_EVENT_STATE_EXIT check on read syscall path. It breaks standard way to read counter, which is to open the counter, wait for the monitored process to die and read the counter.

---
 kernel/events/core.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/events/core.c +1 -2
index f917dec..733c616 100644
@@ -3601,8 +3601,7 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
 	 * error state (i.e. because it was pinned but it couldn't be
 	 * scheduled on to the CPU at some point).
 	 */
-	if ((event->state == PERF_EVENT_STATE_ERROR) ||
-	    (event->state == PERF_EVENT_STATE_EXIT))
+	if (event->state == PERF_EVENT_STATE_ERROR)
 		return 0;
 
 	if (count < event->read_size)