sched,perf: unify tracers in sched and move perf on top of TP
Subsystems (12)
RFC PATCH 1/1 sched,perf: unify tracers in sched and move perf on top of TP
Hello, all.
This patchset does the following two things.
* 0001-0007: Unify the three tracers (tracepoints, perf_events and preempt/sched notifiers) in scheduler.
* 0008-0012: Move perf hooks in sched on top of tracepoints if TPs are enabled.
It probably should be two separate patchsets with the first part on top of sched/core which then gets pulled into perf branch, on top of which the second part is applied but in the spirit of RFC I'm posting it as a single chunk on top of mainline+perf/urgent.
This patchset contains the following 12 patches.
0001-sched-drop-cpu-argument-from-sched_in-preempt-notifi.patch 0002-sched-rename-preempt_notifiers-to-sched_notifiers-an.patch 0003-perf-add-perf_event_task_migrate.patch 0004-perf-add-rq-to-perf_event_task_sched_out.patch 0005-perf-move-perf_event_task_sched_in-next-to-fire_sche.patch 0006-sched-relocate-fire_sched_notifiers_out-and-trace_sc.patch 0007-sched-coalesce-event-notifiers.patch 0008-sched-add-switch_in-and-tick-tracepoints.patch 0009-perf-factor-out-perf_event_switch_clones.patch 0010-perf-make-nr_events-an-int-and-add-perf_online_mutex.patch 0011-perf-prepare-to-move-sched-perf-functions-on-top-of-.patch 0012-perf-move-sched-perf-functions-on-top-of-tracepoints.patch
0001-0006 move tracers in sched around and massage them here and there to bring the related ones together and make them take the same arguments. The only change worth noting would be 0005 where perf_event_task_sched_in() is moved after finish_lock_switch() which costs the function a pair of local_irq_disable/enable() when switching a task context in when there's a context to switch in (the unlikely path).
0007 implements SCHED_EVENT[_RQ]() macros which can define all or any subset of the three at once and uses them to replace all trace calls.
0008-0011 prepare to move sched perf functions on top of matching TPs and make them enabled on demand.
0012 makes all sched perf functions TP probe functions and enable/disable them on-demand.
The whole patchset is more or less straight forward except that the on-demand (or rather off-demand?) disabling of perf functions in 0012 is somewhat complicated because the callback which can determine when to turn it off is called under rq lock while unregistering requires a thread context. As rq lock is depended upon by workqueue, tasklet and timer, none of those can be used directly. It's worked around by having a periodic timer polling for completion.
This patchset is available in the following git branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git sched-trace
The tree is on top of linus#master + sched/urgent for now. Diffstat follows.
arch/ia64/kvm/Kconfig | 2 arch/powerpc/kvm/Kconfig | 2 arch/s390/kvm/Kconfig | 2 arch/x86/kvm/Kconfig | 2 include/linux/kvm_host.h | 4 include/linux/perf_event.h | 39 ++++- include/linux/preempt.h | 48 ------- include/linux/sched.h | 52 +++++++ include/trace/events/sched.h | 14 ++ init/Kconfig | 2 kernel/perf_event.c | 288 +++++++++++++++++++++++++++++++++++++------ kernel/sched.c | 175 ++++++++++++++------------ virt/kvm/kvm_main.c | 28 +--- 13 files changed, 459 insertions(+), 199 deletions(-)
Thanks.
--
tejun
PATCH 1/12 sched: drop @cpu argument from sched_in preempt notifier
@cpu parameter is superflous. Drop it. This will help unifying notifiers in sched.
--- include/linux/preempt.h | 3 +-- kernel/sched.c | 2 +- virt/kvm/kvm_main.c | 4 ++-- 3 files changed, 4 insertions(+), 5 deletions(-)
include/linux/preempt.h +1 -2
@@ -101,7 +101,6 @@ struct preempt_notifier;* preempt_ops - notifiers called when a task is preempted and rescheduled * @sched_in: we're about to be rescheduled: * notifier: struct preempt_notifier for the task being scheduled -* cpu: cpu we're scheduled on * @sched_out: we've just been preempted * notifier: struct preempt_notifier for the task being preempted * next: the task that's kicking us out
@@ -112,7 +111,7 @@ struct preempt_notifier;* difference is intentional and depended upon by its users. */ struct preempt_ops { -void (*sched_in)(struct preempt_notifier *notifier,int cpu); +void (*sched_in)(struct preempt_notifier *notifier);void (*sched_out)(struct preempt_notifier *notifier,struct task_struct *next); };
kernel/sched.c +1 -1
@@ -2718,7 +2718,7 @@ static void fire_sched_in_preempt_notifiers(struct task_struct *curr)struct hlist_node *node;hlist_for_each_entry (notifier, node, &curr->preempt_notifiers, link) - notifier->ops->sched_in (notifier,raw_smp_processor_id ()); + notifier->ops->sched_in (notifier); } static void
virt/kvm/kvm_main.c +2 -2
@@ -2164,11 +2164,11 @@ struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn) return container_of(pn, struct kvm_vcpu, preempt_notifier); } -static void kvm_sched_in (struct preempt_notifier *pn,int cpu) +static void kvm_sched_in (struct preempt_notifier *pn) {struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); -kvm_arch_vcpu_load (vcpu, cpu); +kvm_arch_vcpu_load (vcpu,raw_smp_processor_id ()); } static void kvm_sched_out(struct preempt_notifier *pn,
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: [PATCH 01/12] sched: drop @cpu argument from sched_in preempt notifier
Quoted 2 lines · Tejun Heo
@cpu parameter is superflous. Drop it. This will help unifying notifiers in sched.
This seems sensible enough.
PATCH 2/12 sched: rename preempt_notifiers to sched_notifiers and refactor implementation
In reply to Tejun HeoRename preempt_notifiers to sched_notifiers and move it to sched.h. Also, refactor implementation in sched.c such that adding new callbacks is easier.
This patch does not introduce any functional change and in fact generates the same binary at least with my configuration (x86_64 SMP, kvm and some debug options).
--- arch/ia64/kvm/Kconfig | 2 +- arch/powerpc/kvm/Kconfig | 2 +- arch/s390/kvm/Kconfig | 2 +- arch/x86/kvm/Kconfig | 2 +- include/linux/kvm_host.h | 4 +- include/linux/preempt.h | 47 ------------------ include/linux/sched.h | 52 +++++++++++++++++++- init/Kconfig | 2 +- kernel/sched.c | 119 +++++++++++++++++++++------------------------- virt/kvm/kvm_main.c | 26 +++++----- 10 files changed, 123 insertions(+), 135 deletions(-)
arch/ia64/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM && MODULES && EXPERIMENTAL # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES select HAVE_KVM_IRQCHIP select KVM_APIC_ARCHITECTURE
arch/powerpc/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM bool - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES select KVM_MMIO
arch/s390/kvm/Kconfig +1 -1
@@ -18,7 +18,7 @@ if VIRTUALIZATION config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM && EXPERIMENTAL - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select ANON_INODES ---help--- Support hosting paravirtualized guest machines using the SIE
arch/x86/kvm/Kconfig +1 -1
@@ -22,7 +22,7 @@ config KVM depends on HAVE_KVM # for device assignment: depends on PCI - select PREEMPT_NOTIFIERS + select SCHED_NOTIFIERS select MMU_NOTIFIER select ANON_INODES select HAVE_KVM_IRQCHIP
include/linux/kvm_host.h +2 -2
@@ -75,8 +75,8 @@ int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,struct kvm_vcpu {struct kvm *kvm; -#ifdef CONFIG_PREEMPT_NOTIFIERS -struct preempt_notifier preempt_notifier; +#ifdef CONFIG_SCHED_NOTIFIERS +struct sched_notifier sched_notifier;#endif int vcpu_id;struct mutex mutex;
include/linux/preempt.h +0 -47
51 lines not on this page · raw message
include/linux/sched.h +49 -3
64 lines not on this page · raw message
init/Kconfig +1 -1
@@ -1242,7 +1242,7 @@ config STOP_MACHINE source "block/Kconfig" -config PREEMPT_NOTIFIERS +config SCHED_NOTIFIERS bool config PADATA
kernel/sched.c +55 -64
155 lines not on this page · raw message
virt/kvm/kvm_main.c +12 -14
@@ -77,7 +77,7 @@ static atomic_t hardware_enable_failed;struct kmem_cache *kvm_vcpu_cache;EXPORT_SYMBOL_GPL (kvm_vcpu_cache); -static __read_mostly struct preempt_ops kvm_preempt_ops; +static __read_mostly struct sched_notifier_ops kvm_sched_notifier_ops;struct dentry *kvm_debugfs_dir;
@@ -111,7 +111,7 @@ void vcpu_load(struct kvm_vcpu *vcpu)mutex_lock (&vcpu->mutex); cpu =get_cpu (); -preempt_notifier_register (&vcpu->preempt_notifier); +sched_notifier_register (&vcpu->sched_notifier);kvm_arch_vcpu_load (vcpu, cpu);put_cpu (); }
@@ -120,7 +120,7 @@ void vcpu_put(struct kvm_vcpu *vcpu) {preempt_disable ();kvm_arch_vcpu_put (vcpu); -preempt_notifier_unregister (&vcpu->preempt_notifier); +sched_notifier_unregister (&vcpu->sched_notifier);preempt_enable ();mutex_unlock (&vcpu->mutex); }
@@ -1315,7 +1315,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)if (IS_ERR (vcpu))return PTR_ERR (vcpu); -preempt_notifier_init (&vcpu->preempt_notifier, &kvm_preempt_ops); +sched_notifier_init (&vcpu->sched_notifier, &kvm_sched_notifier_ops); r =kvm_arch_vcpu_setup (vcpu);if (r)
@@ -2158,23 +2158,21 @@ static struct sys_device kvm_sysdev = {struct page *bad_page;pfn_t bad_pfn; -static inline -struct kvm_vcpu *preempt_notifier_to_vcpu (struct preempt_notifier *pn) +static inline struct kvm_vcpu *sched_notifier_to_vcpu (struct sched_notifier *sn) { - return container_of(pn, struct kvm_vcpu, preempt_notifier); + return container_of(sn, struct kvm_vcpu, sched_notifier); } -static void kvm_sched_in (struct preempt_notifier *pn) +static void kvm_sched_in (struct sched_notifier *sn) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_load (vcpu,raw_smp_processor_id ()); } -static void kvm_sched_out (struct preempt_notifier *pn, -struct task_struct *next) +static void kvm_sched_out (struct sched_notifier *sn,struct task_struct *next) { -struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); +struct kvm_vcpu *vcpu =sched_notifier_to_vcpu (sn);kvm_arch_vcpu_put (vcpu); }
@@ -2247,8 +2245,8 @@ int kvm_init(void *opaque, unsigned int vcpu_size, goto out_free; } - kvm_preempt_ops.sched_in = kvm_sched_in; - kvm_preempt_ops.sched_out = kvm_sched_out; + kvm_sched_notifier_ops.in = kvm_sched_in; + kvm_sched_notifier_ops.out = kvm_sched_out;kvm_init_debug ();
Signature · 1 line
1.6.4.2
PATCH 3/12 perf: add perf_event_task_migrate()
In reply to Tejun HeoInstead of calling perf_sw_event() directly from set_task_cpu(), implement perf_event_task_migrate() which takes the same arguments as trace_sched_migrate_task() and invokes perf_sw_event() is the task is really migrating (cur_cpu != new_cpu). This will help unifying notifiers in sched.
--- include/linux/perf_event.h | 3 +++ kernel/perf_event.c | 11 +++++++++++ kernel/sched.c | 5 ++--- 3 files changed, 16 insertions(+), 3 deletions(-)
include/linux/perf_event.h +3 -0
@@ -754,6 +754,7 @@ extern int perf_max_events;extern const struct pmu *hw_perf_event_init (struct perf_event *event); +extern void perf_event_task_migrate (struct task_struct *task,int new_cpu);extern void perf_event_task_sched_in (struct task_struct *task);extern void perf_event_task_sched_out (struct task_struct *task,struct task_struct *next);extern void perf_event_task_tick (struct task_struct *task);
@@ -949,6 +950,8 @@ extern void perf_event_enable(struct perf_event *event);extern void perf_event_disable (struct perf_event *event);#else static inline void +perf_event_task_migrate (struct task_struct *task,int new_cpu) { } +static inline void perf_event_task_sched_in (struct task_struct *task) { } static inline void perf_event_task_sched_out(struct task_struct *task,
kernel/perf_event.c +11 -0
@@ -1148,6 +1148,17 @@ static void perf_event_sync_stat(struct perf_event_context *ctx, }/* +* Called from scheduler set_task_cpu() to notify migration events. +* If the task is moving to a different cpu, generate a migration sw +* event. +*/ +void perf_event_task_migrate (struct task_struct *task,int new_cpu) +{ +if (task_cpu (task) != new_cpu) +perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); +} + +/* * Called from scheduler to remove the events of the current task, * with interrupts disabled. *
kernel/sched.c +2 -3
@@ -2084,11 +2084,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)#endif trace_sched_migrate_task (p, new_cpu); +perf_event_task_migrate (p, new_cpu); -if (task_cpu (p) != new_cpu) { +if (task_cpu (p) != new_cpu) p->se.nr_migrations++; -perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); - }__set_task_cpu (p, new_cpu); }
Signature · 1 line
1.6.4.2
Re: [PATCH 03/12] perf: add perf_event_task_migrate()
Quoted 45 lines · Tejun Heo
Instead of calling
perf_sw_event()directly fromset_task_cpu(), implementperf_event_task_migrate()which takes the same arguments astrace_sched_migrate_task()and invokesperf_sw_event()is the task is really migrating (cur_cpu!=new_cpu). This will help unifying notifiers in sched.Signed-off-by Tejun Heo <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Paul Mackerras <[email protected]>Cc Ingo Molnar <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>--- include/linux/perf_event.h | 3 +++ kernel/perf_event.c | 11 +++++++++++ kernel/sched.c | 5 ++--- 3 files changed, 16 insertions(+), 3 deletions(-)include/linux/perf_event.h +3
-0@@ -754,6 +754,7 @@ extern int perf_max_events;extern const struct pmu *hw_perf_event_init (struct perf_event *event); +extern void perf_event_task_migrate (struct task_struct *task,int new_cpu);extern void perf_event_task_sched_in (struct task_struct *task);extern void perf_event_task_sched_out (struct task_struct *task,struct task_struct *next);extern void perf_event_task_tick (struct task_struct *task);@@ -949,6 +950,8 @@ extern void perf_event_enable(struct perf_event *event);extern void perf_event_disable (struct perf_event *event);#else static inline void +perf_event_task_migrate (struct task_struct *task,int new_cpu) { } +static inline void perf_event_task_sched_in (struct task_struct *task) { } static inline void perf_event_task_sched_out(struct task_struct *task,kernel/perf_event.c +0
-0@@ -1148,6 +1148,17 @@ static void perf_event_sync_stat(struct perf_event_context *ctx, }
kernel/perf_event.c static void perf_event_sync_stat(struct perf_event_context *ctx, · line 1159
Quoted 10 lines
kernel/perf_event.c +9
-0/* +* Called from scheduler set_task_cpu() to notify migration events. +* If the task is moving to a different cpu, generate a migration sw +* event. +*/ +void perf_event_task_migrate (struct task_struct *task,int new_cpu) +{ +if (task_cpu (task) != new_cpu) +perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); +}
This needs to be static and inline (I haven't seem external users in this patchset).
And we want it to be inlined because we save the caller address and the frame pointer from perf_sw_event(), and a new level of call is not wanted here.
Quoted 10 lines
kernel/perf_event.c +2
-0+ +/* * Called from scheduler to remove the events of the current task, * with interrupts disabled. * kernel/sched.c +0
-0
kernel/sched.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 2089
Quoted 7 lines
kernel/sched.c +2
-1@@ -2084,11 +2084,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)#endif trace_sched_migrate_task (p, new_cpu); +perf_event_task_migrate (p, new_cpu); -if (task_cpu (p) != new_cpu) { +if (task_cpu (p) != new_cpu)
In fact why not moving both tracing calls under this check. This is going to fix the migrate trace event that gets called even on "spurious" migrations, and you avoid the duplicate check in the perf callback.
Thanks.
Quoted 14 lines
kernel/sched.c +0
-2p->se.nr_migrations++; -perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); - }__set_task_cpu (p, new_cpu); }Signature · 7 lines
1.6.4.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 03/12] perf: add perf_event_task_migrate()
Hello,
Quoted 5 lines · Frederic Weisbecker
Quoted 5 lines
/*
+ * Called from schedulerset_task_cpu()to notify migration events.
+ * If the task is moving to a different cpu, generate a migration sw
+ * event.
+ */
Quoted 8 lines
kernel/perf_event.c static void perf_event_sync_stat(struct perf_event_context *ctx, · line 1159
Quoted 5 lines
kernel/perf_event.c +5
-0+void perf_event_task_migrate (struct task_struct *task,int new_cpu) +{ +if (task_cpu (task) != new_cpu) +perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); +}This needs to be static and inline (I haven't seem external users in this patchset).
Hmm... after the last patch, this one becomes a TP probe function which can't be inlined.
Quoted 2 lines
And we want it to be inlined because we save the caller address and the frame pointer from
perf_sw_event(), and a new level of call is not wanted here.
Oh I see. So, to use it with TP, I would need increase the @skip parameter to perf_fetch_caller_regs() somehow, right? Also, it probably would be a good idea to add a comment w/ big fat warning to perf_sw_event().
Quoted 6 lines
Quoted 6 lines
kernel/sched.c +1
-0@@ -2084,11 +2084,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)#endif trace_sched_migrate_task (p, new_cpu); +perf_event_task_migrate (p, new_cpu);
Quoted 7 lines
kernel/sched.c void set_task_cpu(struct task_struct *p, unsigned int new_cpu) · line 2089
Quoted 2 lines
kernel/sched.c +1
-1-if (task_cpu (p) != new_cpu) { +if (task_cpu (p) != new_cpu)In fact why not moving both tracing calls under this check. This is going to fix the migrate trace event that gets called even on "spurious" migrations, and you avoid the duplicate check in the perf callback.
Yeah, that would be my preferred choice too. I just didn't know that could be changed. Cool. I'll.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 03/12] perf: add perf_event_task_migrate()
In reply to Frederic WeisbeckerQuoted 5 lines · Frederic Weisbecker
In fact why not moving both tracing calls under this check. This is going to fix the migrate trace event that gets called even on "spurious" migrations, and you avoid the duplicate check in the perf callback.
I kept the tracepoint out of that because I wanted to see how often it attempts silly migrations :-)
Re: [PATCH 03/12] perf: add perf_event_task_migrate()
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
In fact why not moving both tracing calls under this check. This is going to fix the migrate trace event that gets called even on "spurious" migrations, and you avoid the duplicate check in the perf callback.
I kept the tracepoint out of that because I wanted to see how often it attempts silly migrations :-)
Do you still need to keep it or is it okay to move it under the silliness check?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 03/12] perf: add perf_event_task_migrate()
Quoted 8 lines · Tejun Heo
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
In fact why not moving both tracing calls under this check. This is going to fix the migrate trace event that gets called even on "spurious" migrations, and you avoid the duplicate check in the perf callback.
Quoted 5 lines
Quoted 2 lines
I kept the tracepoint out of that because I wanted to see how often it attempts silly migrations :-)
Do you still need to keep it or is it okay to move it under the silliness check?
I guess you can move it, I often end up adding tons of trace_printk() anyway when hunting funnies, might as well add one here when its relevant.
Re: [PATCH 03/12] perf: add perf_event_task_migrate()
Quoted 3 lines · Peter Zijlstra
I guess you can move it, I often end up adding tons of
trace_printk()anyway when hunting funnies, might as well add one here when its relevant.
Great, will do so.
Thanks.
Signature · 1 line
tejun
PATCH 4/12 perf: add @rq to perf_event_task_sched_out()
In reply to Tejun HeoAdd @rq to perf_event_task_sched_out() so that its argument matches those of trace_sched_switch(). This will help unifying notifiers in sched.
--- include/linux/perf_event.h | 5 +++-- kernel/perf_event.c | 4 ++-- kernel/sched.c | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-)
include/linux/perf_event.h +3 -2
@@ -756,7 +756,8 @@ extern const struct pmu *hw_perf_event_init(struct perf_event *event);extern void perf_event_task_migrate (struct task_struct *task,int new_cpu);extern void perf_event_task_sched_in (struct task_struct *task); -extern void perf_event_task_sched_out (struct task_struct *task,struct task_struct *next); +extern void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, +struct task_struct *next);extern void perf_event_task_tick (struct task_struct *task);extern int perf_event_init_task (struct task_struct *child);extern void perf_event_exit_task (struct task_struct *child);
@@ -954,7 +955,7 @@ perf_event_task_migrate(struct task_struct *task, int new_cpu) { }static inline void perf_event_task_sched_in (struct task_struct *task) { }static inline void -perf_event_task_sched_out (struct task_struct *task, +perf_event_task_sched_out (struct rq *rq,struct task_struct *task,struct task_struct *next) { }static inline void perf_event_task_tick (struct task_struct *task) { }
kernel/perf_event.c +2 -2
@@ -1169,8 +1169,8 @@ void perf_event_task_migrate(struct task_struct *task, int new_cpu)* accessing the event control register. If a NMI hits, then it will * not restart the event. */ -void perf_event_task_sched_out (struct task_struct *task, -struct task_struct *next) +void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, +struct task_struct *next) {struct perf_cpu_context *cpuctx = &__get_cpu_var (perf_cpu_context);struct perf_event_context *ctx = task->perf_event_ctxp;
kernel/sched.c +1 -1
@@ -3720,7 +3720,7 @@ need_resched_nonpreemptible:if (likely (prev != next)) {sched_info_switch (prev, next); -perf_event_task_sched_out (prev, next); +perf_event_task_sched_out (rq, prev, next); rq->nr_switches++; rq->curr = next;
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerRe: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.
The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
Re: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
The comment in include/trace/events/sched.h says...
* (NOTE: the 'rq' argument is not used by generic trace events,
* but used by the latency tracer plugin. )
So, I left it there. It would be great if all those @rq params can be removed tho. The latency tracer thing needs it, right?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Quoted 11 lines · Tejun Heo
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
Quoted 7 lines
The comment in
include/trace/events/sched.hsays...* (NOTE: the 'rq' argument is not used by generic trace events,
* but used by the latency tracer plugin. )So, I left it there. It would be great if all those @rq params can be removed tho. The latency tracer thing needs it, right?
I had a quick look but couldn't actually find anything.
Steven Rostedt
maintainerRe: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
In reply to Tejun HeoQuoted 11 lines · Tejun Heo
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
Quoted 7 lines
The comment in
include/trace/events/sched.hsays...* (NOTE: the 'rq' argument is not used by generic trace events,
* but used by the latency tracer plugin. )So, I left it there. It would be great if all those @rq params can be removed tho. The latency tracer thing needs it, right?
I believe the rq was used by the original latency tracer code that was in the -rt patch set before ftrace. It is most likely there for historical purposes. I guess it would be fine to just remove it.
-- Steve
Peter Zijlstra
maintainerRe: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Quoted 17 lines · Steven Rostedt
Quoted 16 lines · Tejun Heo
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
The comment in
include/trace/events/sched.hsays...* (NOTE: the 'rq' argument is not used by generic trace events,
* but used by the latency tracer plugin. )
Quoted 6 lines
Quoted 2 lines
So, I left it there. It would be great if all those @rq params can be removed tho. The latency tracer thing needs it, right?
I believe the rq was used by the original latency tracer code that was in the -rt patch set before ftrace. It is most likely there for historical purposes. I guess it would be fine to just remove it.
I'm building the below, so far so good... ;-)
--- include/trace/events/sched.h | 32 ++++++++++---------------------- kernel/sched.c | 8 ++++---- kernel/trace/trace_sched_wakeup.c | 5 ++--- 3 files changed, 16 insertions(+), 29 deletions(-)
include/trace/events/sched.h +10 -22
@@ -51,15 +51,12 @@ TRACE_EVENT(sched_kthread_stop_ret,/* * Tracepoint for waiting on task to unschedule: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ TRACE_EVENT(sched_wait_task, - TP_PROTO(struct rq *rq, struct task_struct *p), + TP_PROTO(struct task_struct *p), - TP_ARGS(rq, p), + TP_ARGS(p), TP_STRUCT__entry( __array( char, comm, TASK_COMM_LEN )
@@ -79,15 +76,12 @@ TRACE_EVENT(sched_wait_task,/* * Tracepoint for waking up a task: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ DECLARE_EVENT_CLASS(sched_wakeup_template, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), + TP_PROTO(struct task_struct *p, int success), - TP_ARGS(rq, p, success), + TP_ARGS(p, success), TP_STRUCT__entry( __array( char, comm, TASK_COMM_LEN )
@@ -111,31 +105,25 @@ DECLARE_EVENT_CLASS(sched_wakeup_template, );DEFINE_EVENT (sched_wakeup_template, sched_wakeup, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), -TP_ARGS (rq, p, success)); + TP_PROTO(struct task_struct *p, int success), +TP_ARGS (p, success));/* * Tracepoint for waking up a new task: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ DEFINE_EVENT (sched_wakeup_template, sched_wakeup_new, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), -TP_ARGS (rq, p, success)); + TP_PROTO(struct task_struct *p, int success), +TP_ARGS (p, success));/* * Tracepoint for task switches, performed by the scheduler: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ TRACE_EVENT(sched_switch, - TP_PROTO(struct rq *rq, struct task_struct *prev, + TP_PROTO(struct task_struct *prev, struct task_struct *next), - TP_ARGS(rq, prev, next), + TP_ARGS(prev, next), TP_STRUCT__entry( __array( char, prev_comm, TASK_COMM_LEN )
kernel/sched.c +4 -4
@@ -2135,7 +2135,7 @@ unsigned long wait_task_inactive(struct task_struct *p, long match_state)* just go back and repeat. */ rq =task_rq_lock (p, &flags); -trace_sched_wait_task (rq, p); +trace_sched_wait_task (p); running =task_running (rq, p); on_rq = p->se.on_rq; ncsw =0 ;
@@ -2406,7 +2406,7 @@ out_activate: success =1 ; out_running: -trace_sched_wakeup (rq, p, success); +trace_sched_wakeup (p, success);check_preempt_curr (rq, p, wake_flags); p->state =TASK_RUNNING ;
@@ -2580,7 +2580,7 @@ void wake_up_new_task(struct task_struct *p, unsigned long clone_flags) rq =task_rq_lock (p, &flags);activate_task (rq, p,0 ); -trace_sched_wakeup_new (rq, p,1 ); +trace_sched_wakeup_new (p,1 );check_preempt_curr (rq, p,WF_FORK );#ifdef CONFIG_SMP if (p->sched_class->task_woken)
@@ -2800,7 +2800,7 @@ context_switch(struct rq *rq, struct task_struct *prev,struct mm_struct *mm, *oldmm;prepare_task_switch (rq, prev, next); -trace_sched_switch (rq, prev, next); +trace_sched_switch (prev, next); mm = next->mm; oldmm = prev->active_mm;/*
kernel/trace/trace_sched_wakeup.c +2 -3
@@ -107,8 +107,7 @@ static void probe_wakeup_migrate_task(struct task_struct *task, int cpu) } static void notrace -probe_wakeup_sched_switch (struct rq *rq,struct task_struct *prev, -struct task_struct *next) +probe_wakeup_sched_switch (struct task_struct *prev,struct task_struct *next) {struct trace_array_cpu *data;cycle_t T0 ,T1 , delta;
@@ -200,7 +199,7 @@ static void wakeup_reset(struct trace_array *tr) }static void -probe_wakeup (struct rq *rq,struct task_struct *p,int success) +probe_wakeup (struct task_struct *p,int success) {struct trace_array_cpu *data;int cpu =smp_processor_id ();
Re: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Quoted 39 lines · Peter Zijlstra
Quoted 17 lines · Steven Rostedt
Quoted 16 lines · Tejun Heo
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
Add @rq to
perf_event_task_sched_out()so that its argument matches those oftrace_sched_switch(). This will help unifying notifiers in sched.The alternative is dropping rq from the trace events, its not like anybody outside sched.o can do anything sensible with it anyway.
The comment in
include/trace/events/sched.hsays...* (NOTE: the 'rq' argument is not used by generic trace events,
* but used by the latency tracer plugin. )Quoted 6 lines
Quoted 2 lines
So, I left it there. It would be great if all those @rq params can be removed tho. The latency tracer thing needs it, right?
I believe the rq was used by the original latency tracer code that was in the -rt patch set before ftrace. It is most likely there for historical purposes. I guess it would be fine to just remove it.
I'm building the below, so far so good... ;-)
--- include/trace/events/sched.h | 32 ++++++++++---------------------- kernel/sched.c | 8 ++++---- kernel/trace/trace_sched_wakeup.c | 5 ++--- 3 files changed, 16 insertions(+), 29 deletions(-)include/trace/events/sched.h +0
-0@@ -51,15 +51,12 @@ TRACE_EVENT(sched_kthread_stop_ret,
include/trace/events/sched.h TRACE_EVENT(sched_kthread_stop_ret, · line 57
Quoted 10 lines
include/trace/events/sched.h +1
-4/* * Tracepoint for waiting on task to unschedule: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ TRACE_EVENT(sched_wait_task, - TP_PROTO(struct rq *rq, struct task_struct *p), + TP_PROTO(struct task_struct *p),
-- Steve
Re: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
Quoted 7 lines · Steven Rostedt
Quoted 5 lines
Quoted 3 lines
I believe the rq was used by the original latency tracer code that was in the -rt patch set before ftrace. It is most likely there for historical purposes. I guess it would be fine to just remove it.
I'm building the below, so far so good... ;-)
Acked-by Steven Rostedt <[email protected]>
Cool, then I can remove SCHED_EVENT_RQ() thing then.
Thanks.
Signature · 1 line
tejun
Re: [PATCH 04/12] perf: add @rq to perf_event_task_sched_out()
In reply to Steven RostedtQuoted 7 lines · Steven Rostedt
Quoted 5 lines
--- include/trace/events/sched.h | 32 ++++++++++---------------------- kernel/sched.c | 8 ++++---- kernel/trace/trace_sched_wakeup.c | 5 ++--- 3 files changed, 16 insertions(+), 29 deletions(-)Acked-by Steven Rostedt <[email protected]>
Peter, can I include this patch into my series? If so, can you please post a patch w/ description?
Thanks.
Signature · 1 line
tejun
[tip:sched/core] sched: Remove rq argument to the tracepoints
In reply to Peter ZijlstraCommit-ID: 27a9da6538ee18046d7bff8e36a9f783542c54c3 Gitweb: http://git.kernel.org/tip/27a9da6538ee18046d7bff8e36a9f783542c54c3 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 4 May 2010 20:36:56 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 7 May 2010 11:28:17 +0200
sched: Remove rq argument to the tracepoints
struct rq isn't visible outside of sched.o so its near useless to expose the pointer, also there are no users of it, so remove it.
LKML-Reference: <1272997616.1642.207.camel@laptop>
--- include/trace/events/sched.h | 32 ++++++++++---------------------- kernel/sched.c | 8 ++++---- kernel/trace/ftrace.c | 3 +-- kernel/trace/trace_sched_switch.c | 5 ++--- kernel/trace/trace_sched_wakeup.c | 5 ++--- 5 files changed, 19 insertions(+), 34 deletions(-)
include/trace/events/sched.h +10 -22
@@ -51,15 +51,12 @@ TRACE_EVENT(sched_kthread_stop_ret,/* * Tracepoint for waiting on task to unschedule: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ TRACE_EVENT(sched_wait_task, - TP_PROTO(struct rq *rq, struct task_struct *p), + TP_PROTO(struct task_struct *p), - TP_ARGS(rq, p), + TP_ARGS(p), TP_STRUCT__entry( __array( char, comm, TASK_COMM_LEN )
@@ -79,15 +76,12 @@ TRACE_EVENT(sched_wait_task,/* * Tracepoint for waking up a task: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ DECLARE_EVENT_CLASS(sched_wakeup_template, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), + TP_PROTO(struct task_struct *p, int success), - TP_ARGS(rq, p, success), + TP_ARGS(p, success), TP_STRUCT__entry( __array( char, comm, TASK_COMM_LEN )
@@ -111,31 +105,25 @@ DECLARE_EVENT_CLASS(sched_wakeup_template, );DEFINE_EVENT (sched_wakeup_template, sched_wakeup, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), -TP_ARGS (rq, p, success)); + TP_PROTO(struct task_struct *p, int success), +TP_ARGS (p, success));/* * Tracepoint for waking up a new task: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ DEFINE_EVENT (sched_wakeup_template, sched_wakeup_new, - TP_PROTO(struct rq *rq, struct task_struct *p, int success), -TP_ARGS (rq, p, success)); + TP_PROTO(struct task_struct *p, int success), +TP_ARGS (p, success));/* * Tracepoint for task switches, performed by the scheduler: -* -* (NOTE: the 'rq' argument is not used by generic trace events, -* but used by the latency tracer plugin. ) */ TRACE_EVENT(sched_switch, - TP_PROTO(struct rq *rq, struct task_struct *prev, + TP_PROTO(struct task_struct *prev, struct task_struct *next), - TP_ARGS(rq, prev, next), + TP_ARGS(prev, next), TP_STRUCT__entry( __array( char, prev_comm, TASK_COMM_LEN )
kernel/sched.c +4 -4
@@ -2168,7 +2168,7 @@ unsigned long wait_task_inactive(struct task_struct *p, long match_state)* just go back and repeat. */ rq =task_rq_lock (p, &flags); -trace_sched_wait_task (rq, p); +trace_sched_wait_task (p); running =task_running (rq, p); on_rq = p->se.on_rq; ncsw =0 ;
@@ -2439,7 +2439,7 @@ out_activate: success =1 ; out_running: -trace_sched_wakeup (rq, p, success); +trace_sched_wakeup (p, success);check_preempt_curr (rq, p, wake_flags); p->state =TASK_RUNNING ;
@@ -2613,7 +2613,7 @@ void wake_up_new_task(struct task_struct *p, unsigned long clone_flags) rq =task_rq_lock (p, &flags);activate_task (rq, p,0 ); -trace_sched_wakeup_new (rq, p,1 ); +trace_sched_wakeup_new (p,1 );check_preempt_curr (rq, p,WF_FORK );#ifdef CONFIG_SMP if (p->sched_class->task_woken)
@@ -2833,7 +2833,7 @@ context_switch(struct rq *rq, struct task_struct *prev,struct mm_struct *mm, *oldmm;prepare_task_switch (rq, prev, next); -trace_sched_switch (rq, prev, next); +trace_sched_switch (prev, next); mm = next->mm; oldmm = prev->active_mm;/*
kernel/trace/ftrace.c +1 -2
@@ -3212,8 +3212,7 @@ free: }static void -ftrace_graph_probe_sched_switch (struct rq *__rq,struct task_struct *prev, -struct task_struct *next) +ftrace_graph_probe_sched_switch (struct task_struct *prev,struct task_struct *next) {unsigned long long timestamp;int index;
kernel/trace/trace_sched_switch.c +2 -3
@@ -50,8 +50,7 @@ tracing_sched_switch_trace(struct trace_array *tr, }static void -probe_sched_switch (struct rq *__rq,struct task_struct *prev, -struct task_struct *next) +probe_sched_switch (struct task_struct *prev,struct task_struct *next) {struct trace_array_cpu *data;unsigned long flags;
@@ -109,7 +108,7 @@ tracing_sched_wakeup_trace(struct trace_array *tr, }static void -probe_sched_wakeup (struct rq *__rq,struct task_struct *wakee,int success) +probe_sched_wakeup (struct task_struct *wakee,int success) {struct trace_array_cpu *data;unsigned long flags;
kernel/trace/trace_sched_wakeup.c +2 -3
@@ -107,8 +107,7 @@ static void probe_wakeup_migrate_task(struct task_struct *task, int cpu) } static void notrace -probe_wakeup_sched_switch (struct rq *rq,struct task_struct *prev, -struct task_struct *next) +probe_wakeup_sched_switch (struct task_struct *prev,struct task_struct *next) {struct trace_array_cpu *data;cycle_t T0 ,T1 , delta;
@@ -200,7 +199,7 @@ static void wakeup_reset(struct trace_array *tr) }static void -probe_wakeup (struct rq *rq,struct task_struct *p,int success) +probe_wakeup (struct task_struct *p,int success) {struct trace_array_cpu *data;int cpu =smp_processor_id ();
PATCH 5/12 perf: move perf_event_task_sched_in() next to fire_sched_notifiers_in()
In reply to Tejun HeoMove perf_event_task_sched_in() after finish_lock_switch() right before fire_sched_notifiers_in(). This costs an extra pair of irq enable/disable when switching in a perf task context but allows unifying event functions in a more flexible position (irq enabled, rq lock released).
--- kernel/perf_event.c | 9 +++++++-- kernel/sched.c | 8 +------- 2 files changed, 8 insertions(+), 9 deletions(-)
kernel/perf_event.c +7 -2
@@ -1369,14 +1369,17 @@ static void task_ctx_sched_in(struct task_struct *task,*/ void perf_event_task_sched_in (struct task_struct *task) { -struct perf_cpu_context *cpuctx = &__get_cpu_var (perf_cpu_context);struct perf_event_context *ctx = task->perf_event_ctxp; +struct perf_cpu_context *cpuctx;if (likely (!ctx))return ; +local_irq_disable (); + + cpuctx = &__get_cpu_var (perf_cpu_context);if (cpuctx->task_ctx == ctx) -return ; + goto out_enable;/* * We want to keep the following priority order:
@@ -1390,6 +1393,8 @@ void perf_event_task_sched_in(struct task_struct *task)ctx_sched_in (ctx, cpuctx,EVENT_FLEXIBLE ); cpuctx->task_ctx = ctx; +out_enable: +local_irq_enable (); }#define MAX_INTERRUPTS (~0ULL)
kernel/sched.c +1 -7
@@ -2794,15 +2794,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)*/ prev_state = prev->state;finish_arch_switch (prev); -#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW -local_irq_disable (); -#endif /* __ARCH_WANT_INTERRUPTS_ON_CTXSW */ -perf_event_task_sched_in (current); -#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW -local_irq_enable (); -#endif /* __ARCH_WANT_INTERRUPTS_ON_CTXSW */ finish_lock_switch (rq, prev); +perf_event_task_sched_in (current);fire_sched_notifiers_in (current);if (mm)mmdrop (mm);
Signature · 1 line
1.6.4.2
PATCH 6/12 sched: relocate fire_sched_notifiers_out() and trace_sched_switch()
In reply to Tejun HeoRelocate fire_sched_notifiers_out() from prepare_task_switch() and trace_sched_switch() from context_switch() to schedule() so that trace_sched_switch(), perf_event_task_sched_out() and fire_sched_notifiers_out() are colocated.
Both are now called before switch counts and rq->curr are updated and in addition trace_sched_switch() is called before prepare_lock/arch_switch(); however, this shouldn't make any visible different for either.
This will help unifying notifiers in sched.
--- kernel/sched.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/sched.c +2 -2
@@ -2753,7 +2753,6 @@ static inline void prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { -fire_sched_notifiers_out (prev, next);prepare_lock_switch (rq, next);prepare_arch_switch (next); }
@@ -2882,7 +2881,6 @@ context_switch(struct rq *rq, struct task_struct *prev,struct mm_struct *mm, *oldmm;prepare_task_switch (rq, prev, next); -trace_sched_switch (rq, prev, next); mm = next->mm; oldmm = prev->active_mm;/*
@@ -3714,7 +3712,9 @@ need_resched_nonpreemptible:if (likely (prev != next)) {sched_info_switch (prev, next); +trace_sched_switch (rq, prev, next);perf_event_task_sched_out (rq, prev, next); +fire_sched_notifiers_out (prev, next); rq->nr_switches++; rq->curr = next;
Signature · 1 line
1.6.4.2
PATCH 7/12 sched: coalesce event notifiers
In reply to Tejun Heosched currently hosts three different event notification mechanisms - tracepoints, perf_event functions and sched_notifiers. The previous patches modified and moved them around so that they are colocated where applicable and share most of the arguments. This patch introduces and uses SCHED_EVENT() and SCHED_EVENT_RQ() to coalesce calls to different mechanisms.
--- kernel/sched.c | 51 ++++++++++++++++++++++++++++++++++++++++----------- 1 files changed, 40 insertions(+), 11 deletions(-)
kernel/sched.c +40 -11
@@ -1439,6 +1439,39 @@ static inline void fire_sched_notifiers_in(struct task_struct *p)fire_sched_notifiers (in, p); } +/* +* Sched is watched by three different mechanisms - tracepoint, +* perf_event and sched_notifiers. SCHED_EVENT*() can be used to +* define all or any part of them at once so that code clutter is kept +* to minimum and optimizations can be applied according to different +* config options. +* +* In SCHED_EVENT(), the first three arguments specify the name of +* tracepoint, perf_event and sched_notifier. NONE can be used to +* omit any subset of the three. The last argument is event arguments +* wrapped inside SE_ARGS() macro. +* +* SCHED_EVENT_RQ() is identical except that @rq argument will be +* added to tracepoint and perf calls. +*/ +#define trace_sched_NONE(args...) do { } while (0) +#define perf_event_task_NONE(args...) do { } while (0) +#define fire_sched_notifiers_NONE(args...) do { } while (0) + +#define SE_ARGS(args...) args + +#define SCHED_EVENT (TP ,PERF ,SN , args) do { \ + trace_sched_##TP(args); \ + perf_event_task_##PERF(args); \ + fire_sched_notifiers_##SN(args); \ +} while (0) + +#define SCHED_EVENT_RQ (TP ,PERF ,SN , rq, args) do { \ + trace_sched_##TP(rq, args); \ + perf_event_task_##PERF(rq, args); \ + fire_sched_notifiers_##SN(args); \ +} while (0) +static inline void inc_cpu_load (struct rq *rq,unsigned long load) {update_load_add (&rq->load, load);
@@ -2083,8 +2116,7 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu) !(task_thread_info(p)->preempt_count & PREEMPT_ACTIVE));#endif -trace_sched_migrate_task (p, new_cpu); -perf_event_task_migrate (p, new_cpu); +SCHED_EVENT (migrate_task, migrate,NONE ,SE_ARGS (p, new_cpu));if (task_cpu (p) != new_cpu) p->se.nr_migrations++;
@@ -2223,7 +2255,7 @@ unsigned long wait_task_inactive(struct task_struct *p, long match_state)* just go back and repeat. */ rq =task_rq_lock (p, &flags); -trace_sched_wait_task (rq, p); +SCHED_EVENT_RQ (wait_task,NONE ,NONE , rq,SE_ARGS (p)); running =task_running (rq, p); on_rq = p->se.on_rq; ncsw =0 ;
@@ -2515,7 +2547,7 @@ out_activate: } out_running: -trace_sched_wakeup (rq, p, success); +SCHED_EVENT_RQ (wakeup,NONE ,NONE , rq,SE_ARGS (p, success));check_preempt_curr (rq, p, wake_flags); p->state =TASK_RUNNING ;
@@ -2726,7 +2758,7 @@ void wake_up_new_task(struct task_struct *p, unsigned long clone_flags) p->state =TASK_RUNNING ;update_rq_clock (rq);activate_task (rq, p,0 ); -trace_sched_wakeup_new (rq, p,1 ); +SCHED_EVENT_RQ (wakeup_new,NONE ,NONE , rq,SE_ARGS (p,1 ));check_preempt_curr (rq, p,WF_FORK );#ifdef CONFIG_SMP if (p->sched_class->task_woken)
@@ -2795,8 +2827,7 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)finish_arch_switch (prev);finish_lock_switch (rq, prev); -perf_event_task_sched_in (current); -fire_sched_notifiers_in (current); +SCHED_EVENT (NONE , sched_in, in,SE_ARGS (current));if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) {
@@ -3498,7 +3529,7 @@ void scheduler_tick(void) curr->sched_class->task_tick (rq, curr,0 );raw_spin_unlock (&rq->lock); -perf_event_task_tick (curr); +SCHED_EVENT (NONE , tick,NONE ,SE_ARGS (curr));#ifdef CONFIG_SMP rq->idle_at_tick =idle_cpu (cpu);
@@ -3712,9 +3743,7 @@ need_resched_nonpreemptible:if (likely (prev != next)) {sched_info_switch (prev, next); -trace_sched_switch (rq, prev, next); -perf_event_task_sched_out (rq, prev, next); -fire_sched_notifiers_out (prev, next); +SCHED_EVENT_RQ (switch, sched_out, out, rq,SE_ARGS (prev, next)); rq->nr_switches++; rq->curr = next;
Signature · 1 line
1.6.4.2
PATCH 8/12 sched: add switch_in and tick tracepoints
In reply to Tejun HeoDefine and add sched_switch_in and sched_tick tracepoints. Both are colocated with perf event functions. These will be used to make perf use tracepoints.
--- include/trace/events/sched.h | 14 ++++++++++++++ kernel/sched.c | 4 ++-- 2 files changed, 16 insertions(+), 2 deletions(-)
include/trace/events/sched.h +14 -0
@@ -220,6 +220,20 @@ DECLARE_EVENT_CLASS(sched_process_template, );/* +* Tracepoint for task switch in: +*/ +DEFINE_EVENT (sched_process_template, sched_switch_in, + TP_PROTO(struct task_struct *p), +TP_ARGS (p)); + +/* +* Tracepoint for scheduler tick: +*/ +DEFINE_EVENT (sched_process_template, sched_tick, + TP_PROTO(struct task_struct *p), +TP_ARGS (p)); + +/* * Tracepoint for freeing a task: */ DEFINE_EVENT(sched_process_template, sched_process_free,
kernel/sched.c +2 -2
@@ -2827,7 +2827,7 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)finish_arch_switch (prev);finish_lock_switch (rq, prev); -SCHED_EVENT (NONE , sched_in, in,SE_ARGS (current)); +SCHED_EVENT (switch_in, sched_in, in,SE_ARGS (current));if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) {
@@ -3529,7 +3529,7 @@ void scheduler_tick(void) curr->sched_class->task_tick (rq, curr,0 );raw_spin_unlock (&rq->lock); -SCHED_EVENT (NONE , tick,NONE ,SE_ARGS (curr)); +SCHED_EVENT (tick, tick,NONE ,SE_ARGS (curr));#ifdef CONFIG_SMP rq->idle_at_tick =idle_cpu (cpu);
Signature · 1 line
1.6.4.2
PATCH 9/12 perf: factor out perf_event_switch_clones()
In reply to Tejun HeoFactor out perf_event_switch_clones() from perf_event_task_sched_out(). This is to ease future changes and doesn't cause any functional difference.
--- kernel/perf_event.c | 62 +++++++++++++++++++++++++++++--------------------- 1 files changed, 36 insertions(+), 26 deletions(-)
kernel/perf_event.c +36 -26
@@ -1158,30 +1158,14 @@ void perf_event_task_migrate(struct task_struct *task, int new_cpu)perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 ); } -/* -* Called from scheduler to remove the events of the current task, -* with interrupts disabled. -* -* We stop each event and update the event value in event->count. -* -* This does not protect us against NMI, but disable() -* sets the disabled bit in the control field of event _before_ -* accessing the event control register. If a NMI hits, then it will -* not restart the event. -*/ -void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, -struct task_struct *next) +static bool perf_event_switch_clones (struct perf_cpu_context *cpuctx, +struct perf_event_context *ctx, +struct task_struct *task, +struct task_struct *next) { -struct perf_cpu_context *cpuctx = &__get_cpu_var (perf_cpu_context); -struct perf_event_context *ctx = task->perf_event_ctxp;struct perf_event_context *next_ctx;struct perf_event_context *parent; -int do_switch =1 ; - -perf_sw_event (PERF_COUNT_SW_CONTEXT_SWITCHES ,1 ,1 ,NULL ,0 ); - -if (likely (!ctx || !cpuctx->task_ctx)) -return ; +bool switched = false;rcu_read_lock (); parent =rcu_dereference (ctx->parent_ctx);
@@ -1208,7 +1192,7 @@ void perf_event_task_sched_out(struct rq *rq, struct task_struct *task, next->perf_event_ctxp = ctx; ctx->task = next; next_ctx->task = task; - do_switch =0 ; + switched = true;perf_event_sync_stat (ctx, next_ctx); }
@@ -1217,10 +1201,36 @@ void perf_event_task_sched_out(struct rq *rq, struct task_struct *task, }rcu_read_unlock (); -if (do_switch) { -ctx_sched_out (ctx, cpuctx,EVENT_ALL ); - cpuctx->task_ctx =NULL ; - } +return switched; +} + +/* +* Called from scheduler to remove the events of the current task, +* with interrupts disabled. +* +* We stop each event and update the event value in event->count. +* +* This does not protect us against NMI, but disable() +* sets the disabled bit in the control field of event _before_ +* accessing the event control register. If a NMI hits, then it will +* not restart the event. +*/ +void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, +struct task_struct *next) +{ +struct perf_cpu_context *cpuctx = &__get_cpu_var (perf_cpu_context); +struct perf_event_context *ctx = task->perf_event_ctxp; + +perf_sw_event (PERF_COUNT_SW_CONTEXT_SWITCHES ,1 ,1 ,NULL ,0 ); + +if (likely (!ctx || !cpuctx->task_ctx)) +return ; + +if (perf_event_switch_clones (cpuctx, ctx, task, next)) +return ; + +ctx_sched_out (ctx, cpuctx,EVENT_ALL ); + cpuctx->task_ctx =NULL ; } static void task_ctx_sched_out(struct perf_event_context *ctx,
Signature · 1 line
1.6.4.2
PATCH 10/12 perf: make nr_events an int and add perf_online_mutex to protect it
In reply to Tejun HeoChange nr_events from atomic_t to int and add perf_online_mutex to protect it. Helpers to manipulate the count, perf_inc_nr_events() and perf_dec_nr_events(), are added. The former may fail although it doesn't in the current code.
This will be used to make sched perf functions called via tracepoints.
--- kernel/perf_event.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 files changed, 36 insertions(+), 4 deletions(-)
kernel/perf_event.c +36 -4
@@ -43,7 +43,7 @@ int perf_max_events __read_mostly = 1;static int perf_reserved_percpu__read_mostly ;static int perf_overcommit__read_mostly =1 ; -static atomic_t nr_events__read_mostly ; +static int nr_events__read_mostly ;static atomic_t nr_mmap_events__read_mostly ;static atomic_t nr_comm_events__read_mostly ;static atomic_t nr_task_events__read_mostly ;
@@ -72,6 +72,26 @@ static atomic64_t perf_event_id;static DEFINE_SPINLOCK (perf_resource_lock);/* +* Lock to protect nr_events and master enable: +*/ +static DEFINE_MUTEX (perf_online_mutex); + +static int perf_inc_nr_events (void ) +{ +mutex_lock (&perf_online_mutex); + nr_events++; +mutex_unlock (&perf_online_mutex); +return 0 ; +} + +static void perf_dec_nr_events (void ) +{ +mutex_lock (&perf_online_mutex); + nr_events--; +mutex_unlock (&perf_online_mutex); +} + +/* * Architecture provided APIs - weak aliases: */ extern __weak const struct pmu *hw_perf_event_init (struct perf_event *event)
@@ -1589,7 +1609,13 @@ void perf_event_task_tick(struct task_struct *curr)struct perf_event_context *ctx;int rotate =0 ; -if (!atomic_read (&nr_events)) +/* +* nr_events is incremented before events are enabled and +* decremented after they have been pulled out, so it is +* guaranteed to read as non-zero with any enabled event on +* this cpu. +*/ +if (!nr_events)return ; cpuctx = &__get_cpu_var (perf_cpu_context);
@@ -1857,7 +1883,7 @@ static void free_event(struct perf_event *event)perf_pending_sync (event);if (!event->parent) { -atomic_dec (&nr_events); +perf_dec_nr_events ();if (event->attr.mmap)atomic_dec (&nr_mmap_events);if (event->attr.comm)
@@ -4569,6 +4595,12 @@ perf_event_alloc(struct perf_event_attr *attr,if (!event)return ERR_PTR (-ENOMEM ); + err =perf_inc_nr_events (); +if (err) { +kfree (event); +return ERR_PTR (err); + } +/* * Single events are their own group leaders, with an * empty sibling list:
@@ -4657,6 +4689,7 @@ done:if (err) {if (event->ns)put_pid_ns (event->ns); +perf_dec_nr_events ();kfree (event);return ERR_PTR (err); }
@@ -4664,7 +4697,6 @@ done: event->pmu = pmu;if (!event->parent) { -atomic_inc (&nr_events);if (event->attr.mmap)atomic_inc (&nr_mmap_events);if (event->attr.comm)
Signature · 1 line
1.6.4.2
PATCH 11/12 perf: prepare to move sched perf functions on top of tracepoints
In reply to Tejun Heo* Move prototypes of perf_event_task_{migrate|sched_in|sched_out|tick}() into a separate #ifdef block and append _fn to function names and define macros to redirect calls.
* Define PE_STATIC which currently is empty so that these functions can be made static depending on config option.
* Define no-op perf_task_sched_out_done() and call it from the end of perf_task_sched_out().
Other than renaming the functions, this function doesn't introduce any visible change. This is to prepare for moving these functions on top of tracepoints.
--- include/linux/perf_event.h | 43 +++++++++++++++++++++++++++++-------------- kernel/perf_event.c | 30 ++++++++++++++++++++---------- 2 files changed, 49 insertions(+), 24 deletions(-)
include/linux/perf_event.h +29 -14
@@ -747,6 +747,35 @@ struct perf_output_handle {#ifdef CONFIG_PERF_EVENTS +extern void perf_event_task_migrate_fn (struct task_struct *task,int new_cpu); +extern void perf_event_task_sched_in_fn (struct task_struct *task); +extern void perf_event_task_sched_out_fn (struct rq *rq, +struct task_struct *task, +struct task_struct *next); +extern void perf_event_task_tick_fn (struct task_struct *task); + +#define perf_event_task_migrate (t, c) perf_event_task_migrate_fn((t), (c)) +#define perf_event_task_sched_in (t) perf_event_task_sched_in_fn((t)) +#define perf_event_task_sched_out (r, t, n) \ + perf_event_task_sched_out_fn((r), (t), (n)) +#define perf_event_task_tick (t) perf_event_task_tick_fn((t)) + +#else + +static inline void +perf_event_task_migrate (struct task_struct *task,int new_cpu) { } +static inline void +perf_event_task_sched_in (struct task_struct *task) { } +static inline void +perf_event_task_sched_out (struct rq *rq,struct task_struct *task, +struct task_struct *next) { } +static inline void +perf_event_task_tick (struct task_struct *task) { } + +#endif + +#ifdef CONFIG_PERF_EVENTS +/* * Set by architecture code: */
@@ -754,11 +783,6 @@ extern int perf_max_events;extern const struct pmu *hw_perf_event_init (struct perf_event *event); -extern void perf_event_task_migrate (struct task_struct *task,int new_cpu); -extern void perf_event_task_sched_in (struct task_struct *task); -extern void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, -struct task_struct *next); -extern void perf_event_task_tick (struct task_struct *task);extern int perf_event_init_task (struct task_struct *child);extern void perf_event_exit_task (struct task_struct *child);extern void perf_event_free_task (struct task_struct *task);
@@ -950,15 +974,6 @@ extern void perf_swevent_put_recursion_context(int rctx);extern void perf_event_enable (struct perf_event *event);extern void perf_event_disable (struct perf_event *event);#else -static inline void -perf_event_task_migrate (struct task_struct *task,int new_cpu) { } -static inline void -perf_event_task_sched_in (struct task_struct *task) { } -static inline void -perf_event_task_sched_out (struct rq *rq,struct task_struct *task, -struct task_struct *next) { } -static inline void -perf_event_task_tick (struct task_struct *task) { }static inline int perf_event_init_task (struct task_struct *child) {return 0 ; }static inline void perf_event_exit_task (struct task_struct *child) { }static inline void perf_event_free_task (struct task_struct *task) { }
kernel/perf_event.c +20 -10
@@ -76,6 +76,8 @@ static DEFINE_SPINLOCK(perf_resource_lock);*/ static DEFINE_MUTEX (perf_online_mutex); +#define PE_STATIC +static int perf_inc_nr_events (void ) {mutex_lock (&perf_online_mutex);
@@ -91,6 +93,10 @@ static void perf_dec_nr_events(void)mutex_unlock (&perf_online_mutex); } +static void perf_task_sched_out_done (struct perf_event_context *ctx) +{ +} +/* * Architecture provided APIs - weak aliases: */
@@ -189,7 +195,7 @@ perf_lock_task_context(struct task_struct *task, unsigned long *flags)/* * If this context is a clone of another, it might * get swapped for another underneath us by -* perf_event_task_sched_out, though the +* perf_event_task_sched_out_fn, though the * rcu_read_lock() protects us from any context * getting freed. Lock the context and check if it * got swapped before we could get the lock, and retry
@@ -582,7 +588,8 @@ static void __perf_event_disable(void *info)* goes to exit will block in sync_child_event. * When called from perf_pending_event it's OK because event->ctx * is the current context on this CPU and preemption is disabled, -* hence we can't get into perf_event_task_sched_out for this context. +* hence we can't get into perf_event_task_sched_out_fn for this +* context. */ void perf_event_disable (struct perf_event *event) {
@@ -1172,7 +1179,7 @@ static void perf_event_sync_stat(struct perf_event_context *ctx,* If the task is moving to a different cpu, generate a migration sw * event. */ -void perf_event_task_migrate (struct task_struct *task,int new_cpu) +PE_STATIC void perf_event_task_migrate_fn(struct task_struct *task, int new_cpu) {if (task_cpu (task) != new_cpu)perf_sw_event (PERF_COUNT_SW_CPU_MIGRATIONS ,1 ,1 ,NULL ,0 );
@@ -1235,8 +1242,9 @@ static bool perf_event_switch_clones(struct perf_cpu_context *cpuctx,* accessing the event control register. If a NMI hits, then it will * not restart the event. */ -void perf_event_task_sched_out (struct rq *rq,struct task_struct *task, -struct task_struct *next) +PE_STATIC void perf_event_task_sched_out_fn(struct rq *rq, +struct task_struct *task, +struct task_struct *next) {struct perf_cpu_context *cpuctx = &__get_cpu_var (perf_cpu_context);struct perf_event_context *ctx = task->perf_event_ctxp;
@@ -1244,13 +1252,15 @@ void perf_event_task_sched_out(struct rq *rq, struct task_struct *task,perf_sw_event (PERF_COUNT_SW_CONTEXT_SWITCHES ,1 ,1 ,NULL ,0 );if (likely (!ctx || !cpuctx->task_ctx)) -return ; + goto out;if (perf_event_switch_clones (cpuctx, ctx, task, next)) -return ; + goto out;ctx_sched_out (ctx, cpuctx,EVENT_ALL ); cpuctx->task_ctx =NULL ; +out: +perf_task_sched_out_done (cpuctx->task_ctx); } static void task_ctx_sched_out(struct perf_event_context *ctx,
@@ -1397,7 +1407,7 @@ static void task_ctx_sched_in(struct task_struct *task,* accessing the event control register. If a NMI hits, then it will * keep the event running. */ -void perf_event_task_sched_in (struct task_struct *task) +PE_STATIC void perf_event_task_sched_in_fn(struct task_struct *task) {struct perf_event_context *ctx = task->perf_event_ctxp;struct perf_cpu_context *cpuctx;
@@ -1603,7 +1613,7 @@ static void rotate_ctx(struct perf_event_context *ctx)raw_spin_unlock (&ctx->lock); } -void perf_event_task_tick (struct task_struct *curr) +PE_STATIC void perf_event_task_tick_fn(struct task_struct *curr) {struct perf_cpu_context *cpuctx;struct perf_event_context *ctx;
@@ -1705,7 +1715,7 @@ static void perf_event_enable_on_exec(struct task_struct *task)raw_spin_unlock (&ctx->lock); -perf_event_task_sched_in (task); +perf_event_task_sched_in_fn (task); out:local_irq_restore (flags); }
Signature · 1 line
1.6.4.2
PATCH 12/12 perf: move sched perf functions on top of tracepoints
In reply to Tejun HeoNow that all sched perf functions are colocated with tracepoints, those perf functions can be moved on top of tracepoints instead of being called directly. After this patch, if both perf and tracepoints are enabled, the four sched perf macros become noop and the backend functions are defined static and registered as trace point probes on demand.
The enable part is relatively simple. Perf functions are registered as tp probes. sched_in is registered the last so that contexts don't get scheduled in without all the functions active.
Disable is a bit more involved. First, all probes other than sched_out are unregistered and drained and online cpus are recorded in a cpumask. With zero nr_events, sched_out always switches out task context and records that there's no task context for the cpu. A periodic timer is setup to watch the cpumask and when it sees that all cpus have switched out their contexts, the sched_out probe is unregistered.
The timer trick is necessary because unregistering a probe requires thread context while neither workqueue nor tasklet can be directly used from sched_out which is called under rq lock.
This results in reduced overhead when both tracepoints and perf are enabled and opens up possibilities for further optimization. Although the sched functions are the frequently called ones, other perf functions can also be converted to use TPs in similar manner.
--- include/linux/perf_event.h | 2 +- kernel/perf_event.c | 152 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 152 insertions(+), 2 deletions(-)
include/linux/perf_event.h +1 -1
@@ -745,7 +745,7 @@ struct perf_output_handle {int locked; }; -#ifdef CONFIG_PERF_EVENTS +#if defined(CONFIG_PERF_EVENTS ) && !defined(CONFIG_TRACEPOINTS )extern void perf_event_task_migrate_fn (struct task_struct *task,int new_cpu);extern void perf_event_task_sched_in_fn (struct task_struct *task);
kernel/perf_event.c +151 -1
188 lines not on this page · raw message
Signature · 1 line
1.6.4.2
Peter Zijlstra
maintainerQuoted 5 lines · Tejun Heo
This patchset does the following two things.
* 0001-0007: Unify the three tracers (tracepoints,
perf_eventsand preempt/sched notifiers) in scheduler.
Right, so I really don't like the SCHED_EVENT() thing much, esp the 3 different function postfixes.
Quoted 2 lines
* 0008-0012: Move perf hooks in sched on top of tracepoints if TPs are enabled.
This seems to add a terrible amount of overhead for no particular reason.
I know Ingo was pushing to consolidate things, but imho this cure is worse than the ailment.
Hello,
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines
* 0001-0007: Unify the three tracers (tracepoints,
perf_eventsand preempt/sched notifiers) in scheduler.Right, so I really don't like the
SCHED_EVENT()thing much, esp the 3 different function postfixes.
Yeap, it's not the prettiest thing. I thought about renaming all of them so that they share the same postfix but then again I need a way to tell the script which to enable which is the easiest with specifying postfixes as macro argument. Any better ideas?
Quoted 5 lines
Quoted 2 lines
* 0008-0012: Move perf hooks in sched on top of tracepoints if TPs are enabled.
This seems to add a terrible amount of overhead for no particular reason.
Hmm... What overhead?
What matters more here is avoiding the overhead when perf or whatever tracing mechanism is disabled. When both perf and TPs are compiled in but not enabled, moving perf hooks on top of TPs means that the sched core code doesn't have to call into extra functions for perf and TPs can be nooped. ie. less overhead. Also, even when perf is enabled, there isn't any inherent extra runtime overhead other than during enabling/disabling which again is a much colder path.
The only place where noticeable overhead is added is the extra pair of irq enable/disable that I added for sched_out hook. After glancing through what perf does during context switch, I thought that overhead wouldn't be anything noticeable but if it is, they can definitely be separated. The only purpose of that change was colocating sched notifiers and perf hooks.
Also, if a few more perf hooks are converted to use the same mechanism, it would be possible to put perf properly on top of TPs other than init/exit code paths which will be cleaner for the rest of the kernel and there won't be any extra runtime overhead.
The code will be much cleaner if perf depends on TPs. With perf hooks completely removed, there won't be much point in SCHED_EVENT and the messy ifdefs in perf can also go away. Would this be a possibility?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 6 lines · Tejun Heo
Hello,
Quoted 3 lines · Peter Zijlstra
Quoted 2 lines
* 0001-0007: Unify the three tracers (tracepoints,
perf_eventsand preempt/sched notifiers) in scheduler.
Quoted 7 lines
Quoted 2 lines
Right, so I really don't like the
SCHED_EVENT()thing much, esp the 3 different function postfixes.Yeap, it's not the prettiest thing. I thought about renaming all of them so that they share the same postfix but then again I need a way to tell the script which to enable which is the easiest with specifying postfixes as macro argument. Any better ideas?
Add more NOP functions for the missing bits I guess, but that gets cumbersome too.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
* 0008-0012: Move perf hooks in sched on top of tracepoints if TPs are enabled.
This seems to add a terrible amount of overhead for no particular reason.
Hmm... What overhead?
Well, the perf_{inc,dec}_nr_events() stuff, that's massively painful. If that is the price to pay for using tracepoints then I'd rather not.
Also, the whole magic dance to keep !CONFIG_TRACEPOINTS working doesn't make the code any saner.
Quoted 7 lines
What matters more here is avoiding the overhead when perf or whatever tracing mechanism is disabled. When both perf and TPs are compiled in but not enabled, moving perf hooks on top of TPs means that the sched core code doesn't have to call into extra functions for perf and TPs can be nooped. ie. less overhead. Also, even when perf is enabled, there isn't any inherent extra runtime overhead other than during enabling/disabling which again is a much colder path.
Well, in the current code we get the call overhead, but all perf functions bail on !nr_events. We could invert that and have an inline function check nr_events and only then do the call.
Quoted 6 lines
The only place where noticeable overhead is added is the extra pair of irq enable/disable that I added for
sched_outhook. After glancing through what perf does during context switch, I thought that overhead wouldn't be anything noticeable but if it is, they can definitely be separated. The only purpose of that change was colocating sched notifiers and perf hooks.
Right, recent Intel chips seem to do much better at IRQ disable than say P4 and IA64, and poking at the PMU MSRs is way more painful, not sure how it would balance out for PowerPC (or anything other) though.
Quoted 8 lines
Also, if a few more perf hooks are converted to use the same mechanism, it would be possible to put perf properly on top of TPs other than init/exit code paths which will be cleaner for the rest of the kernel and there won't be any extra runtime overhead.
The code will be much cleaner if perf depends on TPs. With perf hooks completely removed, there won't be much point in
SCHED_EVENTand the messy ifdefs in perf can also go away. Would this be a possibility?
Well, I already utterly hate that x86 can't build with ! CONFIG_PERF_EVENTS, also requiring CONFIG_TRACEPOINTS is going the wrong way.
I've always explicitly avoided depending on tracepoints, and I'd very much like to keep it that way.
Hello,
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines
Yeap, it's not the prettiest thing. I thought about renaming all of them so that they share the same postfix but then again I need a way to tell the script which to enable which is the easiest with specifying postfixes as macro argument. Any better ideas?
Add more NOP functions for the missing bits I guess, but that gets cumbersome too.
Hmm... yeah, we'll need NOP functions for every missing hook. Might as well just spell each one out.
Quoted 4 lines
Quoted 1 line
Hmm... What overhead?
Well, the perf_{inc,dec}_nr_events() stuff, that's massively painful. If that is the price to pay for using tracepoints then I'd rather not.
Ah, okay, you mean the code complexity overhead. Eh well, if you wanna do it on-demand, there's certain amount you need to pay. As long as the runtime overhead on hotpath isn't increased, I don't think it's too bad tho.
Quoted 2 lines
Also, the whole magic dance to keep !CONFIG_TRACEPOINTS working doesn't make the code any saner.
Things would have been much prettier if PERF could simply depend on TRACEPOINTS. More on this later.
Quoted 3 lines
Well, in the current code we get the call overhead, but all perf functions bail on !nr_events. We could invert that and have an inline function check
nr_eventsand only then do the call.
The thing is that unifying hooks not only saves the extra check now but also makes future optimizations much easier. ie. Ingo was talking about run time patching TPs so that disabled ones just become noops. It's far better to have all the tracing stuff on single hooking mechanism for optimization efforts like that.
Quoted 3 lines
Right, recent Intel chips seem to do much better at IRQ disable than say P4 and IA64, and poking at the PMU MSRs is way more painful, not sure how it would balance out for PowerPC (or anything other) though.
As I wrote above, here, the problem is mostly cosmetic. It's just that kvm hook needs irq on while perf hook can save one irq flipping if called with irq off. If we call them separately, there's no problem. If we wanna colocate them, it makes sense to pull perf hook out of irq disabled section. This one doesn't really make that much of a difference.
Quoted 6 lines
Well, I already utterly hate that x86 can't build with !
CONFIG_PERF_EVENTS, also requiringCONFIG_TRACEPOINTSis going the wrong way.I've always explicitly avoided depending on tracepoints, and I'd very much like to keep it that way.
I was wondering the other way around - ie. the possibility to make perf optional and maybe even as a module which depends on TPs, which would be nicer than the current situation and make the code less cluttered too.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 5 lines · Tejun Heo
Quoted 5 lines
Well, I already utterly hate that x86 can't build with !
CONFIG_PERF_EVENTS, also requiringCONFIG_TRACEPOINTSis going the wrong way.
Quoted 7 lines
Quoted 2 lines
I've always explicitly avoided depending on tracepoints, and I'd very much like to keep it that way.
I was wondering the other way around - ie. the possibility to make perf optional and maybe even as a module which depends on TPs, which would be nicer than the current situation and make the code less cluttered too.
I really really hate making perf rely on tracepoints.
Hello,
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines
I was wondering the other way around - ie. the possibility to make perf optional and maybe even as a module which depends on TPs, which would be nicer than the current situation and make the code less cluttered too.
I really really hate making perf rely on tracepoints.
Hmmm.... may I ask why? Unifying hooking mechanism seems like a good idea to me and it's not like it's gonna add any runtime overhead although it does complicate init/exit but well that's something you have to pay if you wanna do things dynamically and sans the ifdef stuff it's like a couple hundred lines of isolated code.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 11 lines · Tejun Heo
Hello,
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines
I was wondering the other way around - ie. the possibility to make perf optional and maybe even as a module which depends on TPs, which would be nicer than the current situation and make the code less cluttered too.
I really really hate making perf rely on tracepoints.
Hmmm.... may I ask why?
Because it will bloat the kernel for no reason. And I don't like the endless indirections either.
Like said, I already really dislike x86 won't even build without CONFIG_PERF_EVENTS, adding a hard requirement on TRACEEVENTS will only make the whole thing even worse.
Sure, most distro configs will include both perf and tracepoints, but I don't want to burden everybody who wants to use perf with the tracepoint overhead.
As it stands I'd argue to simply drop this whole idea. The SCHED_EVENT() thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf and sched_notifiers rely on tracepoints.
I don't see the few direct functions calls we have now as a big problem and it sure as hell is a lot easier to read than the whole tracepoint indirection mess.
If you care about the call overhead of the perf hooks, we can do the same immediate value optimization to them as would be done to the tracepoints.
Hello,
Quoted 3 lines · Peter Zijlstra
As it stands I'd argue to simply drop this whole idea. The
SCHED_EVENT()thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf andsched_notifiersrely on tracepoints.
Hmmm... okay. Well, this thing is born out of Ingo's suggestion that for more sched notifiers to be added notification mechanisms need to be consolidated. As long as I can get those two notifiers needed for cmwq, it's okay with me. Ingo, what do you think?
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello,
Quoted 3 lines · Peter Zijlstra
As it stands I'd argue to simply drop this whole idea. The
SCHED_EVENT()thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf andsched_notifiersrely on tracepoints.Hmmm... okay. Well, this thing is born out of Ingo's suggestion that for more sched notifiers to be added notification mechanisms need to be consolidated. As long as I can get those two notifiers needed for cmwq, it's okay with me. Ingo, what do you think?
I'd much rather see the *_EVENT() APIs used - but enhanced to address Peter's objections. One change would be to add a DEFINE_EVENT_ABI() variant, which would be called via trace_abi_*() calls. That way we always know they are 'hardwired' events in the extreme.
That then would allow the software events to be consolidated.
Peter?
Ingo
Peter Zijlstra
maintainerQuoted 14 lines · Ingo Molnar
Quoted 11 lines · Tejun Heo
Hello,
Quoted 3 lines · Peter Zijlstra
As it stands I'd argue to simply drop this whole idea. The
SCHED_EVENT()thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf andsched_notifiersrely on tracepoints.Hmmm... okay. Well, this thing is born out of Ingo's suggestion that for more sched notifiers to be added notification mechanisms need to be consolidated. As long as I can get those two notifiers needed for cmwq, it's okay with me. Ingo, what do you think?
Quoted 8 lines
I'd much rather see the
*_EVENT()APIs used - but enhanced to address Peter's objections. One change would be to add aDEFINE_EVENT_ABI()variant, which would be called via trace_abi_*() calls. That way we always know they are 'hardwired' events in the extreme.That then would allow the software events to be consolidated.
Peter?
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that _EVENT() junk.
Also, we don't need ABI muck for this.
Peter Zijlstra
maintainerQuoted 3 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.
And again, I oppose mandating CONFIG_TRACEEVENT.
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.And again, I oppose mandating
CONFIG_TRACEEVENT.
And me too. But you don't need CONFIG_EVENT_TRACING for that. TRACE_EVENT() with !CONFIG_EVENT_TRACING only produces tracepoints if CONFIG_TRACEPOINTS.
In fact, a first progress that would handle these compromizes would be to have CONFIG_PERF_EVENT_SW.
For now perf_event_task_sched_in and perf_event_task_sched_out can stay as is because they are perf core utils.
But all the rest (faults, migrations, etc..) could be tracepoints builtin only if CONFIG_PERF_EVENT_SW.
Which means CONFIG_PERF_EVENT_SW depends on CONFIG_TRACEPOINTS.
But nobody is forced to build CONFIG_PERF_EVENT_SW, breakpoints don't need it.
Quoted 18 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.And again, I oppose mandating
CONFIG_TRACEEVENT.And me too. But you don't need
CONFIG_EVENT_TRACINGfor that.TRACE_EVENT()with !CONFIG_EVENT_TRACING only produces tracepoints ifCONFIG_TRACEPOINTS.In fact, a first progress that would handle these compromizes would be to have
CONFIG_PERF_EVENT_SW.For now
perf_event_task_sched_inandperf_event_task_sched_outcan stay as is because they are perf core utils.
Quoted 6 lines
But all the rest (faults, migrations, etc..) could be tracepoints builtin only if
CONFIG_PERF_EVENT_SW. Which meansCONFIG_PERF_EVENT_SWdepends onCONFIG_TRACEPOINTS.But nobody is forced to build
CONFIG_PERF_EVENT_SW, breakpoints don't need it.
Yep. Also, most of the default distro kernels will have 3 sets of facilities:
- preempt notifiers
- tracepoints
- sw events
which is crazy. We can just standardize on using the tracepoint interface definition methods - they are properly typed, widespread and well-known enough to be perfect for this.
( They are also under intense optimization - the jump-tracepoints patch makes
them probably even cheaper than preempt notifiers, in the off case. )
So lets get over this and consolidate our crazy hookery, and stand behind a single facility.
I'm also all for slimming down the trace events facilities by not requiring the /debug/tracing/ bits to be present.
Ingo
Hello,
Quoted 2 lines · Ingo Molnar
( They are also under intense optimization - the jump-tracepoints patch makes
them probably even cheaper than preempt notifiers, in the off case. )
I mostly agree on other points but TPs and sched_notifiers are inherently different in how they are enabled/disabled. sched_notifiers are enabled/disabled per-task and at least w/ cmwq, there will always be some tasks with active sched_notifiers so code level optimizations aren't really useful.
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello,
Quoted 2 lines · Ingo Molnar
( They are also under intense optimization - the jump-tracepoints patch makes
them probably even cheaper than preempt notifiers, in the off case. )I mostly agree on other points but TPs and
sched_notifiersare inherently different in how they are enabled/disabled.sched_notifiersare enabled/disabled per-task and at least w/ cmwq, there will always be some tasks with activesched_notifiersso code level optimizations aren't really useful.
Note that preempt notifiers used to modify scheduling behavior/semantics i'm really against. Please get that functionality into the scheduler - especially if it's essentially always-on.
Preempt notifiers are for things like KVM which want to do extended context saving. They are _NOT_ 'hooks' into the scheduler.
Ingo
Hello, Ingo.
Quoted 3 lines · Ingo Molnar
Note that preempt notifiers used to modify scheduling behavior/semantics i'm really against. Please get that functionality into the scheduler - especially if it's essentially always-on.
Okay. Well, cmwq's usage of it isn't that hairy. It's used for two things.
1. to count the number of currently running workers which I think is fine for such hooks.
2. to wake up another worker. I don't think this falls in the category of "behavior/semantics" change. It just does try_to_wake_up() with the only caveat being it's called under the rq lock so requires a different API. There's nothing which alters the scheduler behavior in any pervasive way. It just looks at the counter and wakes up another task if the condition is right *before* beginning any of real scheduling work.
But, anyways, making a dedicated fixed function calls certainly isn't a problem and might as well be slightly cheaper as it's always gonna be there anyway. If that's preferred, I'll happily take that path.
Thanks.
Signature · 1 line
tejun
Quoted 7 lines · Ingo Molnar
Yep. Also, most of the default distro kernels will have 3 sets of facilities:
- preempt notifiers
- tracepoints
- sw events
Quoted 8 lines
which is crazy. We can just standardize on using the tracepoint interface definition methods - they are properly typed, widespread and well-known enough to be perfect for this.
( They are also under intense optimization - the jump-tracepoints patch makes
them probably even cheaper than preempt notifiers, in the off case. )
What about the on case? kvm has preempt notifiers enabled all the time, and relies on their being fast.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 2 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.
Direct calls into code were fine 10 years ago, but since then we got:
- preempt notifiers
- sw events
- tracepoints
Which add up to a lot more than just a plain call into code.
Also, with the jump-optimizations we will have tracepoints that are _cheaper_ than a plain function call.
Quoted 1 line
Also, we don't need ABI muck for this.
we already have an ABIs in place here - this would just properly unify and enumerate it.
Ingo
Peter Zijlstra
maintainerQuoted 8 lines · Ingo Molnar
Quoted 2 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.Direct calls into code were fine 10 years ago, but since then we got:
- preempt notifiers
Are per task an no good for driving perf.
Quoted 2 lines
- sw events
- tracepoints
Are unrelated to the core perf scheduler calls.
Quoted 4 lines
Which add up to a lot more than just a plain call into code.
Also, with the jump-optimizations we will have tracepoints that are _cheaper_ than a plain function call.
Which can just as easily be used on the core perf hooks.
Quoted 4 lines
Quoted 1 line
Also, we don't need ABI muck for this.
we already have an ABIs in place here - this would just properly unify and enumerate it.
I'm not getting it, this is about in-kernel stuff, there are _NO_ in kernel ABIs.
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · Peter Zijlstra
Well, I'd much rather just see a direct call in the code than having to reverse engineer wth hangs onto that
_EVENT()junk.Direct calls into code were fine 10 years ago, but since then we got:
- preempt notifiers
Are per task an no good for driving perf.
No, but they could be expressed via tracepoints.
Quoted 4 lines
Quoted 2 lines
- sw events
- tracepointsAre unrelated to the core perf scheduler calls.
Still they pose 3 different types of callback regimes, all for a slightly different purpose.
Quoted 7 lines
Quoted 4 lines
Which add up to a lot more than just a plain call into code.
Also, with the jump-optimizations we will have tracepoints that are _cheaper_ than a plain function call.
Which can just as easily be used on the core perf hooks.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Also, we don't need ABI muck for this.
we already have an ABIs in place here - this would just properly unify and enumerate it.
I'm not getting it, this is about in-kernel stuff, there are _NO_ in kernel ABIs.
Yes - but the callbacks i mentioned do include ABIs, such as the sw event callbacks.
So my point is to have one coherent callback mechanism for it.
That said, i agree with you that turning the basic perf scheduling calls into events goes one step too far. But all the _other_ 3 types of callbacks (and we might as well add a fourth type as well: the butt-ugly hotplug event notifier chains) should be consolidated really.
Ingo
Hello,
Quoted 2 lines · Ingo Molnar
That said, i agree with you that turning the basic perf scheduling calls into events goes one step too far.
But if you put all those calls on top of TPs and apply static optimization to TPs (yeah, that can be applied directly to perf calls too but it's simpler to consolidate, IMHO), perf becomes zero-cost when disabled.
Thanks.
Signature · 1 line
tejun
Quoted 2 lines · Tejun Heo
(yeah, that can be applied directly to perf calls too but it's simpler to consolidate, IMHO)
Oh, also, if static text optimization is applied to perf calls directly, it would require about the same complexity overhead as doing it on top of TPs. It would still have to drain contexts before turning off those hooks, so the only advantage is the ability to build the kernel w/ PERF but w/o TPs.
Thanks.
Signature · 1 line
tejun
Quoted 4 lines · Peter Zijlstra
As it stands I'd argue to simply drop this whole idea. The
SCHED_EVENT()thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf andsched_notifiersrely on tracepoints.
We can make perf rely on sched_notifiers, I think it's exactly the same case as kvm - extra cpu state that needs switching at context switch time. We could also make fpu switching use sched_notifiers, but that's pushing it a bit.
Signature · 1 line
error compiling committee.c: too many arguments to function
Hello,
Quoted 9 lines · Avi Kivity
Quoted 3 lines · Peter Zijlstra
As it stands I'd argue to simply drop this whole idea. The
SCHED_EVENT()thing doesn't look like its worth the obfuscation and I'm very much opposed to making perf andsched_notifiersrely on tracepoints.We can make perf rely on
sched_notifiers, I think it's exactly the same case as kvm - extra cpu state that needs switching at context switch time. We could also make fpu switching usesched_notifiers, but that's pushing it a bit.
Oh, that was my first attempt too. The difference between TPs and sched_notifiers is the enable condition. TP is system-wide while sched_notifier is per-task. The thing is that for perf hooks, system-wide enable condition works much better. Also, w/ sched_notifiers, there's the problem that no one else but self can enable/disable it (which can be changed but doing it without causing call time overhead requires a bit of effort).
Thanks.
Signature · 1 line
tejun
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines
I was wondering the other way around - ie. the possibility to make perf optional and maybe even as a module which depends on TPs, which would be nicer than the current situation and make the code less cluttered too.
Quoted 7 lines
Quoted 1 line
I really really hate making perf rely on tracepoints.
Hmmm.... may I ask why? Unifying hooking mechanism seems like a good idea to me and it's not like it's gonna add any runtime overhead although it does complicate init/exit but well that's something you have to pay if you wanna do things dynamically and sans the ifdef stuff it's like a couple hundred lines of isolated code.
Don't forget where perf_events started out - as a way to count and record hardware events. So perf_events is very useful even in a kernel that has no tracing infrastructure configured in at all.
Paul.
Hello,
Quoted 3 lines · Paul Mackerras
Don't forget where
perf_eventsstarted out - as a way to count and record hardware events. Soperf_eventsis very useful even in a kernel that has no tracing infrastructure configured in at all.
Alright, but I don't think it's very relevant to worry about !CONFIG_TRACEPOINTS && CONFIG_PERF_EVENTS case as long as both can be compiled out. TPs are supposed to have very low to almost no runtime overhead and building them into the kernel shouldn't cost all that much either (I hear that it has been improved lately). Sure, perf can be used w/o TPs but I can't see what that configurability buys us.
Thanks.
Signature · 1 line
tejun