perf: Add PERF_RECORD_SWITCH to indicate context switches
Adrian Hunter
reviewerRFC PATCH 1/1 perf: Add PERF_RECORD_SWITCH to indicate context switches
There are already two events for context switches, namely the tracepoint sched:sched_switch and the software event context_switches. Unfortunately neither are suitable for use by non-privileged users for the purpose of synchronizing hardware trace data (e.g. Intel PT) to the context switch.
Tracepoints are no good at all for non-privileged users because they need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= -1.
On the other hand, kernel software events need either CAP_SYS_ADMIN or /proc/sys/kernel/perf_event_paranoid <= 1.
Now many distributions do default perf_event_paranoid to 1 making context_switches a contender, except it has another problem (which is also shared with sched:sched_switch) which is that it happens before perf schedules events out instead of after perf schedules events in. Whereas a privileged user can see all the events anyway, a non-privileged user only sees events for their own processes, in other words they see when their process was scheduled out not when it was scheduled in. That presents two problems to use the event: 1. the information comes too late, so tools have to look ahead in the event stream to find out what the current state is 2. if they are unlucky tracing might have stopped before the context-switches event is recorded.
This new PERF_RECORD_SWITCH event does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.
This implementation has a quirk which is that, if possible, it scrounges the event data from the ID sample instead of deriving it twice.
Also I haven't tested this patch yet, so it is RFC at the moment.
--- include/uapi/linux/perf_event.h | 15 +++++- kernel/events/core.c | 108 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+), 1 deletion(-)
include/uapi/linux/perf_event.h +14 -1
@@ -332,7 +332,8 @@ struct perf_event_attr { mmap2 : 1, /* include mmap with inode data */ comm_exec : 1, /* flag comm events that are due to an exec */ use_clockid : 1, /* use @clockid for time fields */ - __reserved_1 : 38; + context_switch : 1, /* context switch data */ + __reserved_1 : 37;union {__u32 wakeup_events;/* wakeup every n events */
@@ -812,6 +813,18 @@ enum perf_event_type {*/ PERF_RECORD_ITRACE_START =12 , +/* +* +* +* struct { +* struct perf_event_header header; +* u32 pid, tid; +* u64 time; +* struct sample_id sample_id; +* }; +*/ +PERF_RECORD_SWITCH =13 , + PERF_RECORD_MAX, /* non-ABI */ };
kernel/events/core.c +108 -0
144 lines not on this page · raw message
Signature · 1 line
1.9.1
Peter Zijlstra
maintainerQuoted 6 lines · Adrian Hunter
Tracepoints are no good at all for non-privileged users because they need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= -1.On the other hand, kernel software events need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= 1.
So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think sched_switch() is one of those because it explicitly exposes timing information on other tasks.
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.
Right, so the one wee problem I have is that this only provides sched_in data, I imagine people might be interested in sched_out as well.
Typically the switch even provides prev and next and thereby is complete, but since we're limiting it to the one specific task, we'll not have the sched_out data.
include/uapi/linux/perf_event.h enum perf_event_type { · line 822
Quoted 11 lines
include/uapi/linux/perf_event.h +7
-0@@ -812,6 +813,18 @@ enum perf_event_type {*/ PERF_RECORD_ITRACE_START = 12, +/* +* +* +* struct { +* struct perf_event_header header; +* u32 pid, tid; +* u64 time;
all 3 are already part of sample_id.
Quoted 4 lines
include/uapi/linux/perf_event.h +4
-0+* struct sample_id sample_id; +* }; +*/ + PERF_RECORD_SWITCH = 13,
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Adrian Hunter
Tracepoints are no good at all for non-privileged users because they need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= -1.
Quoted 7 lines
Quoted 2 lines
On the other hand, kernel software events need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= 1.So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think
sched_switch()is one of those because it explicitly exposes timing information on other tasks.
It's trivial for a running program to measure when it gets context switched by looking at timing. I don't think this event provides anything new over that.
-Andi
Peter Zijlstra
maintainerQuoted 9 lines · Andi Kleen
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Adrian Hunter
Tracepoints are no good at all for non-privileged users because they need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= -1.On the other hand, kernel software events need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= 1.
Quoted 7 lines
Quoted 4 lines
So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think
sched_switch()is one of those because it explicitly exposes timing information on other tasks.It's trivial for a running program to measure when it gets context switched by looking at timing. I don't think this event provides anything new over that.
The _existing_ ones expose timing of _other_ tasks, the proposed one does not.
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.
Yes and with the option of collecting callchains for sched_out.
David
Adrian Hunter
reviewerQuoted 11 lines · David Ahern
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.Yes and with the option of collecting callchains for
sched_out.
So what do you want that is different different from PERF_COUNT_SW_CONTEXT_SWITCHES? And why?
Quoted 8 lines · Adrian Hunter
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.Yes and with the option of collecting callchains for
sched_out.So what do you want that is different different from
PERF_COUNT_SW_CONTEXT_SWITCHES? And why?
My 'perf sched timehist' command can use either sched_switch and context switches. As you have pointed out you need to collect them system wide even if you only care about a subset of tasks. If you are going to add a sched_in there is good symmetry by also having a sched_out that is used and generated in a consistent manner.
Adrian Hunter
reviewerQuoted 18 lines · David Ahern
Quoted 8 lines · Adrian Hunter
Quoted 7 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.Yes and with the option of collecting callchains for
sched_out.So what do you want that is different different from
PERF_COUNT_SW_CONTEXT_SWITCHES? And why?
Quoted 5 lines
My 'perf sched timehist' command can use either
sched_switchand context switches. As you have pointed out you need to collect them system wide even if you only care about a subset of tasks. If you are going to add asched_inthere is good symmetry by also having asched_outthat is used and generated in a consistent manner.
Yes I will certainly add sched out, but the new thing is not a sample so it does not have callchains. But presumably you could use PERF_COUNT_SW_CONTEXT_SWITCHES as well?
Adrian Hunter
reviewerQuoted 13 lines · Peter Zijlstra
Quoted 6 lines · Adrian Hunter
Tracepoints are no good at all for non-privileged users because they need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= -1.On the other hand, kernel software events need either
CAP_SYS_ADMINor/proc/sys/kernel/perf_event_paranoid<= 1.So while I think it makes sense to allow some tracepoint outside of that priv level, IOW have a per tracepoint priv level filter thingy, I don't think
sched_switch()is one of those because it explicitly exposes timing information on other tasks.
Quoted 8 lines
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.
That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.
Quoted 4 lines
Typically the switch even provides prev and next and thereby is complete, but since we're limiting it to the one specific task, we'll not have the
sched_outdata.
That makes sense for completeness, but as I wrote, it would be interesting to know what someone might actually use that for.
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/perf_event.h +0
-0@@ -812,6 +813,18 @@ enum perf_event_type {*/ PERF_RECORD_ITRACE_START = 12,
Quoted 9 lines
include/uapi/linux/perf_event.h enum perf_event_type { · line 822
Quoted 7 lines
include/uapi/linux/perf_event.h +7
-0+/* +* +* +* struct { +* struct perf_event_header header; +* u32 pid, tid; +* u64 time; all 3 are already part of
sample_id.
You have to decide whether you expect to be able to use an event without sample_id. MMAP and MMAP2 both have pid, tid which are in sample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are in sample_id etc. So it currently looks like we expect to be able to use an event without requiring sample_id.
It doesn't affect my case either way because perf tools always sets sample_id_all if it can.
Quoted 7 lines
Quoted 5 lines
include/uapi/linux/perf_event.h +4
-0+* struct sample_id sample_id; +* }; +*/ + PERF_RECORD_SWITCH = 13,
Peter Zijlstra
maintainerQuoted 9 lines · Adrian Hunter
Quoted 2 lines · Peter Zijlstra
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.
I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.
---- A ----] .... [---- B -----
If you do not have the sched_out time, you cannot correct for that.
Quoted 7 lines
Quoted 1 line
all 3 are already part of
sample_id.You have to decide whether you expect to be able to use an event without
sample_id. MMAP and MMAP2 both have pid, tid which are insample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are insample_idetc. So it currently looks like we expect to be able to use an event without requiringsample_id.
I think we recently had this discussion:
lkml.kernel.org/r/[email protected]
The patch we ended up merging:
f38b0dbb491a ("perf/x86/intel: Introduce PERF_RECORD_LOST_SAMPLES")
Does indeed require sample_id.
Arnaldo Carvalho de Melo
maintainerEm Fri, Jun 12, 2015 at 02:09:38PM +0200, Peter Zijlstra escreveu:
Quoted 11 lines · Adrian Hunter
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.
Quoted 15 lines
I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.
---- A ----] .... [---- B -----If you do not have the
sched_outtime, you cannot correct for that.Quoted 7 lines
Quoted 1 line
all 3 are already part of
sample_id.You have to decide whether you expect to be able to use an event without
sample_id. MMAP and MMAP2 both have pid, tid which are insample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are insample_idetc. So it currently looks like we expect to be able to use an event without requiringsample_id.
The fact that there is this duplication is because sample_id_all came after those events, but this new one being proposed doesn't have to do it :-)
Quoted 9 lines
I think we recently had this discussion:
lkml.kernel.org/r/[email protected]
The patch we ended up merging:
f38b0dbb491a("perf/x86/intel: IntroducePERF_RECORD_LOST_SAMPLES")Does indeed require
sample_id.
Adrian Hunter
reviewerQuoted 19 lines · Arnaldo Carvalho de Melo
Em Fri, Jun 12, 2015 at 02:09:38PM +0200, Peter Zijlstra escreveu:
Quoted 18 lines · Adrian Hunter
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.
I was more thinking about it being used to track event duration inside a task. Say you want measure the time between event A and event B but got scheduled out in between.
---- A ----] .... [---- B -----If you do not have the
sched_outtime, you cannot correct for that.
Thanks for the example.
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
all 3 are already part of
sample_id.You have to decide whether you expect to be able to use an event without
sample_id. MMAP and MMAP2 both have pid, tid which are insample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are insample_idetc. So it currently looks like we expect to be able to use an event without requiringsample_id.
Quoted 3 lines
The fact that there is this duplication is because
sample_id_allcame after those events, but this new one being proposed doesn't have to do it :-)
Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.
Quoted 12 lines
Quoted 10 lines
I think we recently had this discussion:
lkml.kernel.org/r/[email protected]
The patch we ended up merging:
f38b0dbb491a("perf/x86/intel: IntroducePERF_RECORD_LOST_SAMPLES")Does indeed require
sample_id.
Quoted 8 lines · Adrian Hunter
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
all 3 are already part of
sample_id.You have to decide whether you expect to be able to use an event without
sample_id. MMAP and MMAP2 both have pid, tid which are insample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are insample_idetc. So it currently looks like we expect to be able to use an event without requiringsample_id.
Quoted 6 lines
Quoted 3 lines
The fact that there is this duplication is because
sample_id_allcame after those events, but this new one being proposed doesn't have to do it :-)Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.
Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.
Pawel
Quoted 13 lines · Pawel Moll
Quoted 12 lines · Adrian Hunter
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
all 3 are already part of
sample_id.You have to decide whether you expect to be able to use an event without
sample_id. MMAP and MMAP2 both have pid, tid which are insample_id, LOST has id, EXIT and FORK have time, all of the THROTTLE/UNTHROTTLE members are insample_idetc. So it currently looks like we expect to be able to use an event without requiringsample_id.The fact that there is this duplication is because
sample_id_allcame after those events, but this new one being proposed doesn't have to do it :-)
Quoted 6 lines
Quoted 2 lines
Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.
Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.
Of course, cpu is a part of sample_id as well. I'm sorted out then :-)
Pawel
Quoted 6 lines · Pawel Moll
Quoted 2 lines
Thanks, that's clear then. There will just need to be a flag to indicate whether it is scheduling in or out.
Just a thought: wouldn't it be good to know what CPU have we been scheduled from/to? This kind of information would be especially valuable in heterogeneous systems.
yes that is essential.
Quoted 6 lines · Adrian Hunter
Quoted 6 lines
Quoted 5 lines
This new
PERF_RECORD_SWITCHevent does not have those problems and it also has a couple of other small advantages. It is easier to use because it is an auxiliary event (like mmap, comm and task events) which can be enabled by setting a single bit. It is smaller than sched:sched_switch and easier to parse.
Quoted 7 lines
Quoted 2 lines
Right, so the one wee problem I have is that this only provides
sched_indata, I imagine people might be interested insched_outas well.That is not a problem although it would be interesting to know the use-case. To me it seemed unreasonable to expect to analyze scheduler behaviour without admin-level privileges since it is inherently an administrative activity.
One use case is wanting to analyze a set of processes -- how long they run when scheduled, where they are when scheduled out, scheduling delay on wakeups, time between scehd in. I wrote the timehist tool in 2010 it has been really helpful understanding what is happening on each cpu and characteristics of a set of processes (e.g. ping ponging between tasks sending messages to each other).
In this case it is not necessarily scheduler behavior (though it does it enter the picture to a degree), but rather behavior of a process or set of tasks.