Skip to contentSkip to conversation

perf: Add PERF_RECORD_SWITCH to indicate context switches

RFC PATCH Superseded Adrian Hunter 6 participants 16 messages Jun 9, 2015 – Jun 12, 2015 perf events

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Adrian Hunter

author reviewer
#0 lore raw

RFC PATCH 1/1 perf: Add PERF_RECORD_SWITCH to indicate context switches

There are already two events for context switches, namely the tracepoint sched:sched_switch and the software event context_switches. Unfortunately neither are suitable for use by non-privileged users for the purpose of synchronizing hardware trace data (e.g. Intel PT) to the context switch.

Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.

On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.

Now many distributions do default perf_event_paranoid to 1 making context_switches a contender, except it has another problem (which is also shared with sched:sched_switch) which is that it happens before perf schedules events out instead of after perf schedules events in. Whereas a privileged user can see all the events anyway, a non-privileged user only sees events for their own processes, in other words they see when their process was scheduled out not when it was scheduled in. That presents two problems to use the event: 1. the information comes too late, so tools have to look ahead in the event stream to find out what the current state is 2. if they are unlucky tracing might have stopped before the context-switches event is recorded.

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

This implementation has a quirk which is that, if possible, it scrounges the event data from the ID sample instead of deriving it twice.

Also I haven't tested this patch yet, so it is RFC at the moment.

---
 include/uapi/linux/perf_event.h |  15 +++++-
 kernel/events/core.c            | 108 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 122 insertions(+), 1 deletion(-)
include/uapi/linux/perf_event.h +14 -1
index 31b10b02db75..f5403660c9f8 100644
@@ -332,7 +332,8 @@ struct perf_event_attr {
 				mmap2          :  1, /* include mmap with inode data     */
 				comm_exec      :  1, /* flag comm events that are due to an exec */
 				use_clockid    :  1, /* use @clockid for time fields */
-				__reserved_1   : 38;
+				context_switch :  1, /* context switch data */
+				__reserved_1   : 37;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -812,6 +813,18 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_ITRACE_START		= 12,
 
+	/*
+	 *
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u32				pid, tid;
+	 *	u64				time;
+	 * 	struct sample_id		sample_id;
+	 * };
+ 	 */
+	PERF_RECORD_SWITCH			= 13,
+
 	PERF_RECORD_MAX,			/* non-ABI */
 };
 
kernel/events/core.c +108 -0

144 lines not on this page · raw message

Signature · 1 line
1.9.1
maintainer
#1 lore raw
Quoted 6 lines · Adrian Hunter

Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.

On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.

So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think sched_switch() is one of those because it explicitly exposes timing information on other tasks.

Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

Typically the switch even provides prev and next and thereby is complete, but since we're limiting it to the one specific task, we'll not have the sched_out data.

include/uapi/linux/perf_event.h enum perf_event_type { · line 822

Quoted 11 lines
include/uapi/linux/perf_event.h +7 -0
@@ -812,6 +813,18 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_ITRACE_START		= 12,
 
+	/*
+	 *
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u32				pid, tid;
+	 *	u64				time;

all 3 are already part of sample_id.

Quoted 4 lines
include/uapi/linux/perf_event.h +4 -0
+	 * 	struct sample_id		sample_id;
+	 * };
+ 	 */
+	PERF_RECORD_SWITCH			= 13,
#2 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Adrian Hunter

Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.

Quoted 7 lines
Quoted 2 lines

On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.

So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think sched_switch() is one of those because it explicitly exposes timing information on other tasks.

It's trivial for a running program to measure when it gets context switched by looking at timing. I don't think this event provides anything new over that.

-Andi

maintainer
#3 lore raw
Quoted 9 lines · Andi Kleen
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Adrian Hunter

Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.

On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.

Quoted 7 lines
Quoted 4 lines

So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think sched_switch() is one of those because it explicitly exposes timing information on other tasks.

It's trivial for a running program to measure when it gets context switched by looking at timing. I don't think this event provides anything new over that.

The _existing_ ones expose timing of _other_ tasks, the proposed one does not.

#4 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

Yes and with the option of collecting callchains for sched_out.

David

author reviewer
#5 lore raw
Quoted 11 lines · David Ahern
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

Yes and with the option of collecting callchains for sched_out.

So what do you want that is different different from PERF_COUNT_SW_CONTEXT_SWITCHES? And why?

#6 lore raw
Quoted 8 lines · Adrian Hunter
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

Yes and with the option of collecting callchains for sched_out.

So what do you want that is different different from PERF_COUNT_SW_CONTEXT_SWITCHES? And why?

My 'perf sched timehist' command can use either sched_switch and context switches. As you have pointed out you need to collect them system wide even if you only care about a subset of tasks. If you are going to add a sched_in there is good symmetry by also having a sched_out that is used and generated in a consistent manner.

author reviewer
#7 lore raw
Quoted 18 lines · David Ahern
Quoted 8 lines · Adrian Hunter
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

Yes and with the option of collecting callchains for sched_out.

So what do you want that is different different from PERF_COUNT_SW_CONTEXT_SWITCHES? And why?

Quoted 5 lines

My 'perf sched timehist' command can use either sched_switch and context switches. As you have pointed out you need to collect them system wide even if you only care about a subset of tasks. If you are going to add a sched_in there is good symmetry by also having a sched_out that is used and generated in a consistent manner.

Yes I will certainly add sched out, but the new thing is not a sample so it does not have callchains. But presumably you could use PERF_COUNT_SW_CONTEXT_SWITCHES as well?

author reviewer
#8 lore raw
In reply to Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 6 lines · Adrian Hunter

Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.

On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.

So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think sched_switch() is one of those because it explicitly exposes timing information on other tasks.

Quoted 8 lines
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.

Quoted 4 lines

Typically the switch even provides prev and next and thereby is complete, but since we're limiting it to the one specific task, we'll not have the sched_out data.

That makes sense for completeness, but as I wrote, it would be interesting to know what someone might actually use that for.

Quoted 5 lines
Quoted 5 lines
include/uapi/linux/perf_event.h +0 -0
@@ -812,6 +813,18 @@ enum perf_event_type {
 	 */
 	PERF_RECORD_ITRACE_START		= 12,
 
Quoted 9 lines

include/uapi/linux/perf_event.h enum perf_event_type { · line 822

Quoted 7 lines
include/uapi/linux/perf_event.h +7 -0
+	/*
+	 *
+	 *
+	 * struct {
+	 *	struct perf_event_header	header;
+	 *	u32				pid, tid;
+	 *	u64				time;

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

It doesn't affect my case either way because perf tools always sets sample_id_all if it can.

Quoted 7 lines
Quoted 5 lines
include/uapi/linux/perf_event.h +4 -0
+	 * 	struct sample_id		sample_id;
+	 * };
+ 	 */
+	PERF_RECORD_SWITCH			= 13,
maintainer
#9 lore raw
Quoted 9 lines · Adrian Hunter
Quoted 2 lines · Peter Zijlstra

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.

I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.

	---- A ----] .... [---- B -----

If you do not have the sched_out time, you cannot correct for that.

Quoted 7 lines
Quoted 1 line

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

I think we recently had this discussion:

lkml.kernel.org/r/[email protected]

The patch we ended up merging:

f38b0dbb491a ("perf/x86/intel: Introduce PERF_RECORD_LOST_SAMPLES")

Does indeed require sample_id.

#10 lore raw

Em Fri, Jun 12, 2015 at 02:09:38PM +0200, Peter Zijlstra escreveu:

Quoted 11 lines · Adrian Hunter
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.

Quoted 15 lines

I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.

	---- A ----] .... [---- B -----

If you do not have the sched_out time, you cannot correct for that.

Quoted 7 lines
Quoted 1 line

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

The fact that there is this duplication is because sample_id_all came after those events, but this new one being proposed doesn't have to do it :-)

Quoted 9 lines

I think we recently had this discussion:

lkml.kernel.org/r/[email protected]

The patch we ended up merging:

f38b0dbb491a ("perf/x86/intel: Introduce PERF_RECORD_LOST_SAMPLES")

Does indeed require sample_id.

author reviewer
#11 lore raw
Quoted 19 lines · Arnaldo Carvalho de Melo

Em Fri, Jun 12, 2015 at 02:09:38PM +0200, Peter Zijlstra escreveu:

Quoted 18 lines · Adrian Hunter
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.

I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.

	---- A ----] .... [---- B -----

If you do not have the sched_out time, you cannot correct for that.

Thanks for the example.

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

Quoted 3 lines

The fact that there is this duplication is because sample_id_all came after those events, but this new one being proposed doesn't have to do it :-)

Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.

Quoted 12 lines
Quoted 10 lines

I think we recently had this discussion:

lkml.kernel.org/r/[email protected]

The patch we ended up merging:

f38b0dbb491a ("perf/x86/intel: Introduce PERF_RECORD_LOST_SAMPLES")

Does indeed require sample_id.

#12 lore raw
Quoted 8 lines · Adrian Hunter
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

Quoted 6 lines
Quoted 3 lines

The fact that there is this duplication is because sample_id_all came after those events, but this new one being proposed doesn't have to do it :-)

Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.

Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.

Pawel

#13 lore raw
Quoted 13 lines · Pawel Moll
Quoted 12 lines · Adrian Hunter
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

all 3 are already part of sample_id.

You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.

The fact that there is this duplication is because sample_id_all came after those events, but this new one being proposed doesn't have to do it :-)

Quoted 6 lines
Quoted 2 lines

Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.

Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.

Of course, cpu is a part of sample_id as well. I'm sorted out then :-)

Pawel

#14 lore raw
In reply to Pawel Moll
Quoted 6 lines · Pawel Moll
Quoted 2 lines

Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.

Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.

yes that is essential.

#15 lore raw
In reply to Adrian Hunter
Quoted 6 lines · Adrian Hunter
Quoted 6 lines
Quoted 5 lines

This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.

Quoted 7 lines
Quoted 2 lines

Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.

That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.

One use case is wanting to analyze a set of processes -- how long they run when scheduled, where they are when scheduled out, scheduling delay on wakeups, time between scehd in. I wrote the timehist tool in 2010 it has been really helpful understanding what is happening on each cpu and characteristics of a set of processes (e.g. ping ponging between tasks sending messages to each other).

In this case it is not necessarily scheduler behavior (though it does it enter the picture to a degree), but rather behavior of a process or set of tasks.