Skip to contentSkip to conversation

perf: expose thread context switch out event type to user space

PATCH Superseded Alexey Budankov 3 participants 7 messages Mar 5, 2018 perf events

lore
Share & tools
lore
#0 lore raw

cover perf: expose thread context switch out event type to user space

Here is a series of small patches that implement exposing type of context-switch-out event as a part of PERF_RECORD_SWITCH[_CPU_WIDE] record.

Introduced types of context-switch-out events assumed to be:
a) preempt: task->state == TASK_RUNNING
misc &= PERF_RECORD_MISC_SWITCH_OUT

b) yield: !preempt - using new bit PERF_RECORD_MISC_SWITCH_OUT_YIELD:
misc &= PERF_RECORD_MISC_SWITCH_OUT|PERF_RECORD_MISC_SWITCH_OUT_YIELD

Perf tool report and script commands output has been extended to decode new yield bit and the updated output looks like in the examples below.

The documentation has been updated to mention yield switch out events and its decoding symbols in perf script output.

The changes have been manually tested on Fedora 27 with the patched kernel:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core

perf report -D -i system-wide.perf:

0x1b9c50 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 20 30 00 01 1e 00 00 01 1e 00 00  ..... 0.........
.  0010:  00 00 00 00 00 00 00 00 85 ae d4 e3 3e 0e 00 00  ............>...
.  0020:  54 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  T...............

5 15663273127557 0x1b9c50 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT next pid/tid: 7681/7681

0x2646c0 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 60 30 00 00 00 00 00 00 00 00 00  .....`0.........
.  0010:  00 1e 00 00 00 1e 00 00 29 1e d5 e3 3e 0e 00 00  ........)...>...
.  0020:  56 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00  V...............

7 15663273156137 0x2646c0 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0

perf script --show-switch-events -F +misc -I -i system-wide.perf:

amplxe-perf  7681 [005] S     15663.273151: PERF_RECORD_SWITCH_CPU_WIDE OUT  next pid/tid:    39/39   
migration/5    39 [005]       15663.273152: PERF_RECORD_SWITCH_CPU_WIDE IN   prev pid/tid:  7681/7681 
amplxe-perf  7680 [007] K     15663.273153:          1                                                                                                                                       context-switch: 

aaa488 schedule ([kernel.kallsyms]) 1a9f50 __poll_nocancel (inlined)

amplxe-perf 7680 [007] Sy 15663.273156: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0
migration/5 39 [005] K 15663.273157:

---
Alexey Budankov (3):
perf/core: store context switch out type into Perf trace perf report: extend raw dump (-D) out with switch out event type perf script: extend misc field decoding with switch out event type

 include/uapi/linux/perf_event.h          |  5 +++++
 kernel/events/core.c                     |  4 +++-
 tools/include/uapi/linux/perf_event.h    |  5 +++++
 tools/perf/Documentation/perf-script.txt | 17 +++++++++--------
 tools/perf/builtin-script.c              |  5 ++++-
 tools/perf/util/event.c                  |  4 +++-
 6 files changed, 29 insertions(+), 11 deletions(-)
#1 lore raw

PATCH 1/3 perf/core: store context switch out type into Perf trace

Store thread context-switch-out event type into Perf trace as a part of PERF_RECORD_SWITCH[_CPU_WIDE] records.

Introduced types of switch-out events assumed to be
a) preempt: task->state == TASK_RUNNING and b) yield: !preempt;

New yield event type is encoded using special PERF_RECORD_MISC_SWITCH_OUT_YIELD bit extending PERF_RECORD_MISC_SWITCH_OUT meaning traditional preemption switch out event:

misc &= PERF_RECORD_MISC_SWITCH_OUT | PERF_RECORD_MISC_SWITCH_OUT_YIELD

---
 include/uapi/linux/perf_event.h | 5 +++++
 kernel/events/core.c            | 4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)
include/uapi/linux/perf_event.h +5 -0
index 6f873503552d..0339c829cda5 100644
@@ -654,6 +654,11 @@ struct perf_event_mmap_page {
  * perf_event_attr::precise_ip.
  */
 #define PERF_RECORD_MISC_EXACT_IP		(1 << 14)
+/*
+ * Indicates that thread explicitly yielded cpu due to
+ * a call of some synchronization API e.g. futex system call
+ */
+#define PERF_RECORD_MISC_SWITCH_OUT_YIELD	(1 << 14)
 /*
  * Reserve the last bit to indicate some extended misc field
  */
kernel/events/core.c +3 -1
index 57898102847f..1faa6dde090c 100644
@@ -7216,6 +7216,8 @@ static void perf_event_switch(struct task_struct *task,
 			      struct task_struct *next_prev, bool sched_in)
 {
 	struct perf_switch_event switch_event;
+	__u16 switch_type = sched_in ? 0 : PERF_RECORD_MISC_SWITCH_OUT |
+		(task->state == TASK_RUNNING ? 0 : PERF_RECORD_MISC_SWITCH_OUT_YIELD);
 
 	/* N.B. caller checks nr_switch_events != 0 */
 
@@ -7225,7 +7227,7 @@ static void perf_event_switch(struct task_struct *task,
 		.event_id	= {
 			.header = {
 				/* .type */
-				.misc = sched_in ? 0 : PERF_RECORD_MISC_SWITCH_OUT,
+				.misc = switch_type,
 				/* .size */
 			},
 			/* .next_prev_pid */
#2 lore raw

PATCH 2/3 perf report: extend raw dump (-D) out with switch out event type

In reply to Alexey Budankov

Print additional 'yield' tag for PERF_RECORD_SWITCH[_CPU_WIDE] OUT records when event header misc field contains PERF_RECORD_MISC_SWITCH_OUT_YIELD bit set designating synchronization context switch out event:

perf report -D -i system-wide.perf:

...

0x1b9c50 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 20 30 00 01 1e 00 00 01 1e 00 00  ..... 0.........
.  0010:  00 00 00 00 00 00 00 00 85 ae d4 e3 3e 0e 00 00  ............>...
.  0020:  54 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  T...............

5 15663273127557 0x1b9c50 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT next pid/tid: 7681/7681

0x2646c0 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 60 30 00 00 00 00 00 00 00 00 00  .....`0.........
.  0010:  00 1e 00 00 00 1e 00 00 29 1e d5 e3 3e 0e 00 00  ........)...>...
.  0020:  56 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00  V...............

7 15663273156137 0x2646c0 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0

...

---
 tools/include/uapi/linux/perf_event.h | 5 +++++
 tools/perf/util/event.c               | 4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)
tools/include/uapi/linux/perf_event.h +5 -0
index 6f873503552d..0339c829cda5 100644
@@ -654,6 +654,11 @@ struct perf_event_mmap_page {
  * perf_event_attr::precise_ip.
  */
 #define PERF_RECORD_MISC_EXACT_IP		(1 << 14)
+/*
+ * Indicates that thread explicitly yielded cpu due to
+ * a call of some synchronization API e.g. futex system call
+ */
+#define PERF_RECORD_MISC_SWITCH_OUT_YIELD	(1 << 14)
 /*
  * Reserve the last bit to indicate some extended misc field
  */
tools/perf/util/event.c +3 -1
index f0a6cbd033cc..af5a85d56446 100644
@@ -1421,7 +1421,9 @@ size_t perf_event__fprintf_itrace_start(union perf_event *event, FILE *fp)
 size_t perf_event__fprintf_switch(union perf_event *event, FILE *fp)
 {
 	bool out = event->header.misc & PERF_RECORD_MISC_SWITCH_OUT;
-	const char *in_out = out ? "OUT" : "IN ";
+	const char *in_out = !out ? "IN " :
+		!(event->header.misc & PERF_RECORD_MISC_SWITCH_OUT_YIELD) ?
+				"OUT" : "OUT yield";
 
 	if (event->header.type == PERF_RECORD_SWITCH)
 		return fprintf(fp, " %s\n", in_out);
#3 lore raw

PATCH 3/3 perf script: extend misc field deconding with switch out event type

In reply to Alexey Budankov

Append 'y' sign to 'S' tag designating the type of context switch out event so 'S' means preemption context switch and 'Sy' means synchronization context switch. Documentation is extended to cover new presentation changes.

perf script --show-switch-events -F +misc -I -i system-wide.perf:

amplxe-perf  7681 [005] S     15663.273151: PERF_RECORD_SWITCH_CPU_WIDE OUT  next pid/tid:    39/39   
migration/5    39 [005]       15663.273152: PERF_RECORD_SWITCH_CPU_WIDE IN   prev pid/tid:  7681/7681 
amplxe-perf  7680 [007] K     15663.273153:          1                                                                                                                                       context-switch: 

aaa488 schedule ([kernel.kallsyms]) 1a9f50 __poll_nocancel (inlined)

amplxe-perf 7680 [007] Sy 15663.273156: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0
migration/5 39 [005] K 15663.273157:

---
 tools/perf/Documentation/perf-script.txt | 17 +++++++++--------
 tools/perf/builtin-script.c              |  5 ++++-
 2 files changed, 13 insertions(+), 9 deletions(-)
tools/perf/Documentation/perf-script.txt +9 -8
index 36ec0257f8d3..80e94cbbf520 100644
@@ -228,14 +228,15 @@ OPTIONS
 	For sample events it's possible to display misc field with -F +misc option,
 	following letters are displayed for each bit:
 
-	  PERF_RECORD_MISC_KERNEL        K
-	  PERF_RECORD_MISC_USER          U
-	  PERF_RECORD_MISC_HYPERVISOR    H
-	  PERF_RECORD_MISC_GUEST_KERNEL  G
-	  PERF_RECORD_MISC_GUEST_USER    g
-	  PERF_RECORD_MISC_MMAP_DATA*    M
-	  PERF_RECORD_MISC_COMM_EXEC     E
-	  PERF_RECORD_MISC_SWITCH_OUT    S
+	  PERF_RECORD_MISC_KERNEL               K
+	  PERF_RECORD_MISC_USER                 U
+	  PERF_RECORD_MISC_HYPERVISOR           H
+	  PERF_RECORD_MISC_GUEST_KERNEL         G
+	  PERF_RECORD_MISC_GUEST_USER           g
+	  PERF_RECORD_MISC_MMAP_DATA*           M
+	  PERF_RECORD_MISC_COMM_EXEC            E
+	  PERF_RECORD_MISC_SWITCH_OUT           S
+	  PERF_RECORD_MISC_SWITCH_OUT_YIELD     Sy
 
 	  $ perf script -F +misc ...
 	   sched-messaging  1414 K     28690.636582:       4590 cycles ...
tools/perf/builtin-script.c +4 -1
index cce926aeb0c0..06d7fe73886f 100644
@@ -657,8 +657,11 @@ static int perf_sample__fprintf_start(struct perf_sample *sample,
 			break;
 		case PERF_RECORD_SWITCH:
 		case PERF_RECORD_SWITCH_CPU_WIDE:
-			if (has(SWITCH_OUT))
+			if (has(SWITCH_OUT)) {
 				ret += fprintf(fp, "S");
+				if (sample->misc & PERF_RECORD_MISC_SWITCH_OUT_YIELD)
+					ret += fprintf(fp, "y");
+			}
 		default:
 			break;
 		}
#4 lore raw
In reply to Alexey Budankov

Em Mon, Mar 05, 2018 at 02:35:02PM +0300, Alexey Budankov escreveu:

Quoted 9 lines

Here is a series of small patches that implement exposing type of context-switch-out event as a part of PERF_RECORD_SWITCH[_CPU_WIDE] record.

Introduced types of context-switch-out events assumed to be:
a) preempt: task->state == TASK_RUNNING
misc &= PERF_RECORD_MISC_SWITCH_OUT

b) yield: !preempt - using new bit PERF_RECORD_MISC_SWITCH_OUT_YIELD:

misc &= PERF_RECORD_MISC_SWITCH_OUT|PERF_RECORD_MISC_SWITCH_OUT_YIELD

Quoted 3 lines

Perf tool report and script commands output has been extended to decode new yield bit and the updated output looks like in the examples below.

I'm just waiting for the current reviewers to be satisfied with this, but I think this is a great addition and 'perf trace' is another tool that should jump into this, showing forced context switches together with syscalls.

- Arnaldo

Quoted 50 lines

The documentation has been updated to mention yield switch out events and its decoding symbols in perf script output.

The changes have been manually tested on Fedora 27 with the patched kernel:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core

perf report -D -i system-wide.perf:

0x1b9c50 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 20 30 00 01 1e 00 00 01 1e 00 00  ..... 0.........
.  0010:  00 00 00 00 00 00 00 00 85 ae d4 e3 3e 0e 00 00  ............>...
.  0020:  54 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  T...............

5 15663273127557 0x1b9c50 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT next pid/tid: 7681/7681

0x2646c0 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 60 30 00 00 00 00 00 00 00 00 00  .....`0.........
.  0010:  00 1e 00 00 00 1e 00 00 29 1e d5 e3 3e 0e 00 00  ........)...>...
.  0020:  56 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00  V...............

7 15663273156137 0x2646c0 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0

perf script --show-switch-events -F +misc -I -i system-wide.perf:

amplxe-perf  7681 [005] S     15663.273151: PERF_RECORD_SWITCH_CPU_WIDE OUT  next pid/tid:    39/39   
migration/5    39 [005]       15663.273152: PERF_RECORD_SWITCH_CPU_WIDE IN   prev pid/tid:  7681/7681 
amplxe-perf  7680 [007] K     15663.273153:          1                                                                                                                                       context-switch: 

aaa488 schedule ([kernel.kallsyms]) 1a9f50 __poll_nocancel (inlined)

amplxe-perf 7680 [007] Sy 15663.273156: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0
migration/5 39 [005] K 15663.273157:

---
Alexey Budankov (3):
perf/core: store context switch out type into Perf trace perf report: extend raw dump (-D) out with switch out event type perf script: extend misc field decoding with switch out event type

 include/uapi/linux/perf_event.h          |  5 +++++
 kernel/events/core.c                     |  4 +++-
 tools/include/uapi/linux/perf_event.h    |  5 +++++
 tools/perf/Documentation/perf-script.txt | 17 +++++++++--------
 tools/perf/builtin-script.c              |  5 ++++-
 tools/perf/util/event.c                  |  4 +++-
 6 files changed, 29 insertions(+), 11 deletions(-)
#5 lore raw

Hi Arnaldo,

Quoted 12 lines · Arnaldo Carvalho de Melo

Em Mon, Mar 05, 2018 at 02:35:02PM +0300, Alexey Budankov escreveu:

Quoted 9 lines

Here is a series of small patches that implement exposing type of context-switch-out event as a part of PERF_RECORD_SWITCH[_CPU_WIDE] record.

Introduced types of context-switch-out events assumed to be:
a) preempt: task->state == TASK_RUNNING
misc &= PERF_RECORD_MISC_SWITCH_OUT

b) yield: !preempt - using new bit PERF_RECORD_MISC_SWITCH_OUT_YIELD:

misc &= PERF_RECORD_MISC_SWITCH_OUT|PERF_RECORD_MISC_SWITCH_OUT_YIELD

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Perf tool report and script commands output has been extended to decode new yield bit and the updated output looks like in the examples below.

I'm just waiting for the current reviewers to be satisfied with this, but I think this is a great addition and 'perf trace' is another tool that should jump into this, showing forced context switches together with syscalls.

It's great to know there is a value in that change for other Perf tools.

Extending perf trace (strace inspired) tool in that respect might makes sense. I anticipate possible tracing overhead needs to be expected and probably handled somehow.

But, anyway, yep, per-thread syscall traces enriched with typed context switch boundaries may be the great extension, in comparison to the original strace tool.

IMHO, some simple summary metrics like amount of preempt or yield context switches (per-thread or per-process) could bring even more value into perf trace tool functionality.

BR,
Alexey

Quoted 54 lines

- Arnaldo

Quoted 50 lines

The documentation has been updated to mention yield switch out events and its decoding symbols in perf script output.

The changes have been manually tested on Fedora 27 with the patched kernel:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core

perf report -D -i system-wide.perf:

0x1b9c50 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 20 30 00 01 1e 00 00 01 1e 00 00  ..... 0.........
.  0010:  00 00 00 00 00 00 00 00 85 ae d4 e3 3e 0e 00 00  ............>...
.  0020:  54 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00  T...............

5 15663273127557 0x1b9c50 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT next pid/tid: 7681/7681

0x2646c0 [0x30]: event: 15
.
. ... raw event: size 48 bytes

.  0000:  0f 00 00 00 00 60 30 00 00 00 00 00 00 00 00 00  .....`0.........
.  0010:  00 1e 00 00 00 1e 00 00 29 1e d5 e3 3e 0e 00 00  ........)...>...
.  0020:  56 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00  V...............

7 15663273156137 0x2646c0 [0x30]: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0

perf script --show-switch-events -F +misc -I -i system-wide.perf:

amplxe-perf  7681 [005] S     15663.273151: PERF_RECORD_SWITCH_CPU_WIDE OUT  next pid/tid:    39/39   
migration/5    39 [005]       15663.273152: PERF_RECORD_SWITCH_CPU_WIDE IN   prev pid/tid:  7681/7681 
amplxe-perf  7680 [007] K     15663.273153:          1                                                                                                                                       context-switch: 

aaa488 schedule ([kernel.kallsyms]) 1a9f50 __poll_nocancel (inlined)

amplxe-perf 7680 [007] Sy 15663.273156: PERF_RECORD_SWITCH_CPU_WIDE OUT yield next pid/tid: 0/0
migration/5 39 [005] K 15663.273157:

---
Alexey Budankov (3):
perf/core: store context switch out type into Perf trace perf report: extend raw dump (-D) out with switch out event type perf script: extend misc field decoding with switch out event type

 include/uapi/linux/perf_event.h          |  5 +++++
 kernel/events/core.c                     |  4 +++-
 tools/include/uapi/linux/perf_event.h    |  5 +++++
 tools/perf/Documentation/perf-script.txt | 17 +++++++++--------
 tools/perf/builtin-script.c              |  5 ++++-
 tools/perf/util/event.c                  |  4 +++-
 6 files changed, 29 insertions(+), 11 deletions(-)