hw-breakpoints: Rewrite on top of perf events v5
Subsystems (7)
Ingo,
Please pull the tracing/hw-breakpoints branch that can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
tracing/hw-breakpoints
Changes in v5:
- Split-up the asm-generic/hw-breakpoint.h moving to linux/hw_breakpoint.h into a separate patch (Suggested by Prasad)
- Optimize the breakpoints restoring while switching from kvm guest to host. We only want to restore the state if we have active breakpoints to the host, otherwise we don't care about messed-up address registers (Suggested by Jan Kizka)
- Add asm/hw_breakpoint.h to Kbuild
- Fix bad breakpoint type in trace_selftest.c
- Fix unreleased non-pinned task-bound-only counters. We only released it in the first cpu. (Thanks to Paul Mackerras for reporting that)
Thanks,
Frederic
---
Frederic Weisbecker (4):
perf/core: Add a callback to perf events
hw-breakpoint: Move asm-generic/hw_breakpoint.h to linux/hw_breakpoint.h
hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
hw-breakpoints: Arbitrate access to pmu following registers constraints
Arjan van de Ven (1):
perf/core: Provide a kernel-internal interface to get to performance counters
Li Zefan (1):
ksym_tracer: Remove KSYM_SELFTEST_ENTRY
Paul Mundt (1):
x86/hw-breakpoints: Actually flush thread breakpoints in flush_thread().
arch/Kconfig | 3 + arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/debugreg.h | 11 +- arch/x86/include/asm/hw_breakpoint.h | 58 +++-- arch/x86/include/asm/processor.h | 12 +- arch/x86/kernel/hw_breakpoint.c | 391 ++++++++++++++-------- arch/x86/kernel/process.c | 9 +- arch/x86/kernel/process_32.c | 26 +-- arch/x86/kernel/process_64.c | 26 +-- arch/x86/kernel/ptrace.c | 182 +++++++---- arch/x86/kernel/smpboot.c | 3 - arch/x86/kvm/x86.c | 18 +- arch/x86/power/cpu.c | 6 - include/asm-generic/hw_breakpoint.h | 139 -------- include/linux/hw_breakpoint.h | 131 ++++++++ include/linux/perf_event.h | 37 ++- kernel/exit.c | 5 + kernel/hw_breakpoint.c | 602 +++++++++++++++++++++------------- kernel/perf_event.c | 136 ++++++++- kernel/trace/trace.h | 1 - kernel/trace/trace_entries.h | 6 +- kernel/trace/trace_ksym.c | 126 ++++---- kernel/trace/trace_selftest.c | 4 +- 23 files changed, 1194 insertions(+), 739 deletions(-)
PATCH 1/7 perf/core: Provide a kernel-internal interface to get to performance counters
From: Arjan van de Ven <[email protected]>
There are reasons for kernel code to ask for, and use, performance counters.
For example, in CPU freq governors this tends to be a good idea, but there are other examples possible as well of course.
This patch adds the needed bits to do enable this functionality; they have been tested in an experimental cpufreq driver that I'm working on, and the changes are all that I needed to access counters properly.
[[email protected]: added pid to perf_event_create_kernel_counter so that we can profile a particular task too
TODO: Have a better error reporting, don't just return NULL in fail case.]
v2: Remove the wrong comment about the fact
perf_event_create_kernel_counter must be called from a kernel thread.
LKML-Reference: <[email protected]>
--- include/linux/perf_event.h | 6 +++ kernel/perf_event.c | 75 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 80 insertions(+), 1 deletions(-)
include/linux/perf_event.h +6 -0
@@ -744,6 +744,12 @@ extern int hw_perf_group_sched_in(struct perf_event *group_leader,struct perf_cpu_context *cpuctx, struct perf_event_context *ctx, int cpu);extern void perf_event_update_userpage (struct perf_event *event); +extern int perf_event_release_kernel (struct perf_event *event); +extern struct perf_event * +perf_event_create_kernel_counter (struct perf_event_attr *attr, +int cpu, +pid_t pid); +extern u64 perf_event_read_value (struct perf_event *event);struct perf_sample_data {u64 type;
kernel/perf_event.c +74 -1
99 lines not on this page · raw message
Signature · 1 line
1.6.2.3
PATCH 2/7 x86/hw-breakpoints: Actually flush thread breakpoints in flush_thread().
In reply to Frederic WeisbeckerFrom: Paul Mundt <[email protected]>
flush_thread() tries to do a TIF_DEBUG check before calling in to flush_thread_hw_breakpoint() (which subsequently clears the thread flag), but for some reason, the x86 code is manually clearing TIF_DEBUG immediately before the test, so this path will never be taken.
This kills off the erroneous clear_tsk_thread_flag() and lets flush_thread_hw_breakpoint() actually get invoked.
Presumably folks were getting lucky with testing and the free_thread_info() -> free_thread_xstate() path was taking care of the flush there.
LKML-Reference: <[email protected]>
--- arch/x86/kernel/process.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
arch/x86/kernel/process.c +0 -2
@@ -107,8 +107,6 @@ void flush_thread(void) }#endif -clear_tsk_thread_flag (tsk,TIF_DEBUG ); -if (unlikely (test_tsk_thread_flag (tsk,TIF_DEBUG )))flush_thread_hw_breakpoint (tsk);memset (tsk->thread.tls_array,0 ,sizeof (tsk->thread.tls_array));
Signature · 1 line
1.6.2.3
PATCH 3/7 perf/core: Add a callback to perf events
In reply to Frederic WeisbeckerA simple callback in a perf event can be used for multiple purposes. For example it is useful for triggered based events like hardware breakpoints that need a callback to dispatch a triggered breakpoint event.
v2: Simplify a bit the callback attribution as suggested by Paul
Mackerras
--- include/linux/perf_event.h | 7 ++++++- kernel/perf_event.c | 14 ++++++++++---- 2 files changed, 16 insertions(+), 5 deletions(-)
include/linux/perf_event.h +6 -1
@@ -544,6 +544,8 @@ struct perf_pending_entry {void (*func)(struct perf_pending_entry *); }; +typedef void (*perf_callback_t )(struct perf_event *,void *); +/** * struct perf_event - performance event kernel representation: */
@@ -639,6 +641,8 @@ struct perf_event {struct event_filter *filter;#endif +perf_callback_t callback; +#endif /* CONFIG_PERF_EVENTS */ };
@@ -748,7 +752,8 @@ extern int perf_event_release_kernel(struct perf_event *event);extern struct perf_event *perf_event_create_kernel_counter (struct perf_event_attr *attr,int cpu, -pid_t pid); +pid_t pid, +perf_callback_t callback);extern u64 perf_event_read_value (struct perf_event *event);struct perf_sample_data {
kernel/perf_event.c +10 -4
@@ -4293,6 +4293,7 @@ perf_event_alloc(struct perf_event_attr *attr,struct perf_event_context *ctx, struct perf_event *group_leader, struct perf_event *parent_event, + perf_callback_t callback, gfp_t gfpflags) { const struct pmu *pmu;
@@ -4335,6 +4336,11 @@ perf_event_alloc(struct perf_event_attr *attr, event->state =PERF_EVENT_STATE_INACTIVE ; +if (!callback && parent_event) + callback = parent_event->callback; + + event->callback = callback; +if (attr->disabled) event->state =PERF_EVENT_STATE_OFF ;
@@ -4611,7 +4617,7 @@ SYSCALL_DEFINE5(perf_event_open, } event =perf_event_alloc (&attr, cpu, ctx, group_leader, -NULL ,GFP_KERNEL ); +NULL ,NULL ,GFP_KERNEL ); err =PTR_ERR (event);if (IS_ERR (event)) goto err_put_context;
@@ -4668,7 +4674,7 @@ err_put_context:*/ struct perf_event *perf_event_create_kernel_counter (struct perf_event_attr *attr,int cpu, -pid_t pid) +pid_t pid,perf_callback_t callback) {struct perf_event *event;struct perf_event_context *ctx;
@@ -4683,7 +4689,7 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,return NULL ; event =perf_event_alloc (attr, cpu, ctx,NULL , -NULL ,GFP_KERNEL ); +NULL , callback,GFP_KERNEL ); err =PTR_ERR (event);if (IS_ERR (event)) goto err_put_context;
@@ -4736,7 +4742,7 @@ inherit_event(struct perf_event *parent_event, child_event =perf_event_alloc (&parent_event->attr, parent_event->cpu, child_ctx, group_leader, parent_event, -GFP_KERNEL ); +NULL ,GFP_KERNEL );if (IS_ERR (child_event))return child_event;get_ctx (child_ctx);
Signature · 1 line
1.6.2.3
PATCH 4/7 hw-breakpoint: Move asm-generic/hw_breakpoint.h to linux/hw_breakpoint.h
In reply to Frederic WeisbeckerWe plan to make the breakpoints parameters generic among architectures. For that it's better to move the asm-generic header to a generic linux header.
---
arch/x86/include/asm/hw_breakpoint.h | 2 +-
include/{asm-generic => linux}/hw_breakpoint.h | 9 +++------
2 files changed, 4 insertions(+), 7 deletions(-)
rename include/{asm-generic => linux}/hw_breakpoint.h (96%)arch/x86/include/asm/hw_breakpoint.h +1 -1
@@ -12,7 +12,7 @@ struct arch_hw_breakpoint { };#include <linux/kdebug.h> -#include <asm-generic/hw_breakpoint.h> +#include <linux/hw_breakpoint.h> /* Available HW breakpoint length encodings */ #define HW_BREAKPOINT_LEN_1 0x40
include/linux/hw_breakpoint.h renamed from include/asm-generic/hw_breakpoint.h +3 -6
@@ -1,9 +1,6 @@ -#ifndef _ASM_GENERIC_HW_BREAKPOINT_H -#define _ASM_GENERIC_HW_BREAKPOINT_H +#ifndef _LINUX_HW_BREAKPOINT_H +#define _LINUX_HW_BREAKPOINT_H -#ifndef __ARCH_HW_BREAKPOINT_H -#error "Please don't include this file directly" -#endif #ifdef __KERNEL__#include <linux/list.h>
@@ -136,4 +133,4 @@ extern void unregister_kernel_hw_breakpoint(struct hw_breakpoint *bp);extern unsigned int hbp_kernel_pos;#endif /* __KERNEL__ */ -#endif /* _ASM_GENERIC_HW_BREAKPOINT_H */ +#endif /* _LINUX_HW_BREAKPOINT_H */
Signature · 1 line
1.6.2.3
PATCH 5/7 hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
In reply to Frederic WeisbeckerThis patch rebase the implementation of the breakpoints API on top of perf events instances.
Each breakpoints are now perf events that handle the register scheduling, thread/cpu attachment, etc..
The new layering is now made as follows:
ptrace kgdb ftrace perf syscall
\ | / /
\ | / /
/
Core breakpoint API /
/
| /
| /
Breakpoints perf events
|
|
Breakpoints PMU ---- Debug Register constraints handling
(Part of core breakpoint API)
|
|
Hardware debug registers
Reasons of this rewrite:
- Use the centralized/optimized pmu registers scheduling, implying an easier arch integration
- More powerful register handling: perf attributes (pinned/flexible events, exclusive/non-exclusive, tunable period, etc...)
Impact:
- New perf ABI: the hardware breakpoints counters
- Ptrace breakpoints setting remains tricky and still needs some per thread breakpoints references.
Todo (in the order):
- Support breakpoints perf counter events for perf tools (ie: implement perf_bpcounter_event())
- Support from perf tools
Changes in v2:
- Follow the perf "event " rename
- The ptrace regression have been fixed (ptrace breakpoint perf events weren't released when a task ended)
- Drop the struct hw_breakpoint and store generic fields in perf_event_attr.
- Separate core and arch specific headers, drop asm-generic/hw_breakpoint.h and create linux/hw_breakpoint.h
- Use new generic len/type for breakpoint
- Handle off case: when breakpoints api is not supported by an arch
Changes in v3:
- Fix broken CONFIG_KVM, we need to propagate the breakpoint api changes to kvm when we exit the guest and restore the bp registers to the host.
Changes in v4:
- Drop the hw_breakpoint_restore() stub as it is only used by KVM
- EXPORT_SYMBOL_GPL hw_breakpoint_restore() as KVM can be built as a module
- Restore the breakpoints unconditionally on kvm guest exit:
TIF_DEBUG_THREAD doesn't anymore cover every cases of running breakpoints and vcpu->arch.switch_db_regs might not always be set when the guest used debug registers.
(Waiting for a reliable optimization)
Changes in v5:
- Split-up the asm-generic/hw-breakpoint.h moving to linux/hw_breakpoint.h into a separate patch
- Optimize the breakpoints restoring while switching from kvm guest to host. We only want to restore the state if we have active breakpoints to the host, otherwise we don't care about messed-up address registers.
- Add asm/hw_breakpoint.h to Kbuild
- Fix bad breakpoint type in trace_selftest.c
--- arch/Kconfig | 3 + arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/debugreg.h | 11 +- arch/x86/include/asm/hw_breakpoint.h | 58 +++-- arch/x86/include/asm/processor.h | 12 +- arch/x86/kernel/hw_breakpoint.c | 391 ++++++++++++++++++++----------- arch/x86/kernel/process.c | 7 +- arch/x86/kernel/process_32.c | 26 +-- arch/x86/kernel/process_64.c | 26 +-- arch/x86/kernel/ptrace.c | 182 ++++++++++----- arch/x86/kernel/smpboot.c | 3 - arch/x86/kvm/x86.c | 18 +- arch/x86/power/cpu.c | 6 - include/linux/hw_breakpoint.h | 243 ++++++++++---------- include/linux/perf_event.h | 26 ++- kernel/exit.c | 5 + kernel/hw_breakpoint.c | 424 ++++++++++++++-------------------- kernel/perf_event.c | 53 ++++- kernel/trace/trace_entries.h | 6 +- kernel/trace/trace_ksym.c | 126 +++++------ kernel/trace/trace_selftest.c | 3 +- 21 files changed, 884 insertions(+), 746 deletions(-)
arch/Kconfig +3 -0
@@ -128,6 +128,9 @@ config HAVE_DEFAULT_NO_SPIN_MUTEXES config HAVE_HW_BREAKPOINT bool + depends on HAVE_PERF_EVENTS + select ANON_INODES + select PERF_EVENTS source "kernel/gcov/Kconfig"
arch/x86/include/asm/Kbuild +1 -0
@@ -10,6 +10,7 @@ header-y += ptrace-abi.h header-y += sigcontext32.h header-y += ucontext.h header-y += processor-flags.h +header-y += hw_breakpoint.h unifdef-y += e820.h unifdef-y += ist.h
arch/x86/include/asm/debugreg.h +5 -6
@@ -75,13 +75,8 @@*/ #ifdef __KERNEL__ -/* For process management */ -extern void flush_thread_hw_breakpoint (struct task_struct *tsk); -extern int copy_thread_hw_breakpoint (struct task_struct *tsk, -struct task_struct *child,unsigned long clone_flags); +DECLARE_PER_CPU(unsigned long, dr7); -/* For CPU management */ -extern void load_debug_registers (void );static inline void hw_breakpoint_disable (void ) {/* Zero the control register for HW Breakpoint */
@@ -94,6 +89,10 @@ static inline void hw_breakpoint_disable(void)set_debugreg (0UL ,3 ); } +#ifdef CONFIG_KVM +extern void hw_breakpoint_restore (void ); +#endif +#endif /* __KERNEL__ */ #endif /* _ASM_X86_DEBUGREG_H */
arch/x86/include/asm/hw_breakpoint.h +38 -20
88 lines not on this page · raw message
arch/x86/include/asm/processor.h +6 -6
@@ -423,6 +423,8 @@ extern unsigned int xstate_size;extern void free_thread_xstate (struct task_struct *);extern struct kmem_cache *task_xstate_cachep; +struct perf_event ; +struct thread_struct {/* Cached TLS descriptors: */ struct desc_struct tls_array[GDT_ENTRY_TLS_ENTRIES ];
@@ -444,12 +446,10 @@ struct thread_struct {unsigned long fs;#endif unsigned long gs; -/* Hardware debugging registers: */ -unsigned long debugreg[HBP_NUM ]; -unsigned long debugreg6; -unsigned long debugreg7; -/* Hardware breakpoint info */ -struct hw_breakpoint *hbp[HBP_NUM ]; +/* Save middle states of ptrace breakpoints */ +struct perf_event *ptrace_bps[HBP_NUM ]; +/* Debug status used for traps, single steps, etc... */ +unsigned long debugreg6;/* Fault info: */ unsigned long cr2;unsigned long trap_no;
arch/x86/kernel/hw_breakpoint.c +255 -136
549 lines not on this page · raw message
arch/x86/kernel/process.c +2 -5
@@ -10,6 +10,7 @@#include <linux/clockchips.h> #include <linux/random.h> #include <trace/events/power.h> +#include <linux/hw_breakpoint.h> #include <asm/system.h> #include <asm/apic.h> #include <asm/syscalls.h>
@@ -18,7 +19,6 @@#include <asm/i387.h> #include <asm/ds.h> #include <asm/debugreg.h> -#include <asm/hw_breakpoint.h> unsigned long idle_halt;EXPORT_SYMBOL (idle_halt);
@@ -47,8 +47,6 @@ void free_thread_xstate(struct task_struct *tsk)kmem_cache_free (task_xstate_cachep, tsk->thread.xstate); tsk->thread.xstate =NULL ; } -if (unlikely (test_tsk_thread_flag (tsk,TIF_DEBUG ))) -flush_thread_hw_breakpoint (tsk);WARN (tsk->thread.ds_ctx,"leaking DS context\n" ); }
@@ -107,8 +105,7 @@ void flush_thread(void) }#endif -if (unlikely (test_tsk_thread_flag (tsk,TIF_DEBUG ))) -flush_thread_hw_breakpoint (tsk); +flush_ptrace_hw_breakpoint (tsk);memset (tsk->thread.tls_array,0 ,sizeof (tsk->thread.tls_array));/* * Forget coprocessor state..
arch/x86/kernel/process_32.c +2 -24
@@ -59,7 +59,6 @@#include <asm/syscalls.h> #include <asm/ds.h> #include <asm/debugreg.h> -#include <asm/hw_breakpoint.h> asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
@@ -264,9 +263,8 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, p->thread.io_bitmap_ptr =NULL ; tsk = current; err = -ENOMEM ; -if (unlikely (test_tsk_thread_flag (tsk,TIF_DEBUG ))) -if (copy_thread_hw_breakpoint (tsk, p, clone_flags)) - goto out; + +memset (p->thread.ptrace_bps,0 ,sizeof (p->thread.ptrace_bps));if (unlikely (test_tsk_thread_flag (tsk,TIF_IO_BITMAP ))) { p->thread.io_bitmap_ptr = kmemdup(tsk->thread.io_bitmap_ptr,
@@ -287,13 +285,10 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, err =do_set_thread_area (p,-1 , (struct user_desc __user *)childregs->si,0 ); -out:if (err && p->thread.io_bitmap_ptr) {kfree (p->thread.io_bitmap_ptr); p->thread.io_bitmap_max =0 ; } -if (err) -flush_thread_hw_breakpoint (p);clear_tsk_thread_flag (p,TIF_DS_AREA_MSR ); p->thread.ds_ctx =NULL ;
@@ -437,23 +432,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)lazy_load_gs (next->gs);percpu_write (current_task, next_p); -/* -* There's a problem with moving the arch_install_thread_hw_breakpoint() -* call before current is updated. Suppose a kernel breakpoint is -* triggered in between the two, the hw-breakpoint handler will see that -* the 'current' task does not have TIF_DEBUG flag set and will think it -* is leftover from an old task (lazy switching) and will erase it. Then -* until the next context switch, no user-breakpoints will be installed. -* -* The real problem is that it's impossible to update both current and -* physical debug registers at the same instant, so there will always be -* a window in which they disagree and a breakpoint might get triggered. -* Since we use lazy switching, we are forced to assume that a -* disagreement means that current is correct and the exception is due -* to lazy debug register switching. -*/ -if (unlikely (test_tsk_thread_flag (next_p,TIF_DEBUG ))) -arch_install_thread_hw_breakpoint (next_p);return prev_p; }
arch/x86/kernel/process_64.c +1 -25
@@ -53,7 +53,6 @@#include <asm/syscalls.h> #include <asm/ds.h> #include <asm/debugreg.h> -#include <asm/hw_breakpoint.h> asmlinkage extern void ret_from_fork(void);
@@ -244,8 +243,6 @@ void release_thread(struct task_struct *dead_task)BUG (); } } -if (unlikely (dead_task->thread.debugreg7)) -flush_thread_hw_breakpoint (dead_task); }static inline void set_32bit_tls (struct task_struct *t,int tls,u32 addr)
@@ -309,9 +306,7 @@ int copy_thread(unsigned long clone_flags, unsigned long sp,savesegment (ds, p->thread.ds); err = -ENOMEM ; -if (unlikely (test_tsk_thread_flag (me,TIF_DEBUG ))) -if (copy_thread_hw_breakpoint (me, p, clone_flags)) - goto out; +memset (p->thread.ptrace_bps,0 ,sizeof (p->thread.ptrace_bps));if (unlikely (test_tsk_thread_flag (me,TIF_IO_BITMAP ))) { p->thread.io_bitmap_ptr =kmalloc (IO_BITMAP_BYTES ,GFP_KERNEL );
@@ -351,8 +346,6 @@ out:kfree (p->thread.io_bitmap_ptr); p->thread.io_bitmap_max =0 ; } -if (err) -flush_thread_hw_breakpoint (p);return err; }
@@ -508,23 +501,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)*/ if (preload_fpu)__math_state_restore (); -/* -* There's a problem with moving the arch_install_thread_hw_breakpoint() -* call before current is updated. Suppose a kernel breakpoint is -* triggered in between the two, the hw-breakpoint handler will see that -* the 'current' task does not have TIF_DEBUG flag set and will think it -* is leftover from an old task (lazy switching) and will erase it. Then -* until the next context switch, no user-breakpoints will be installed. -* -* The real problem is that it's impossible to update both current and -* physical debug registers at the same instant, so there will always be -* a window in which they disagree and a breakpoint might get triggered. -* Since we use lazy switching, we are forced to assume that a -* disagreement means that current is correct and the exception is due -* to lazy debug register switching. -*/ -if (unlikely (test_tsk_thread_flag (next_p,TIF_DEBUG ))) -arch_install_thread_hw_breakpoint (next_p);return prev_p; }
arch/x86/kernel/ptrace.c +125 -57
257 lines not on this page · raw message
arch/x86/kernel/smpboot.c +0 -3
@@ -64,7 +64,6 @@#include <asm/apic.h> #include <asm/setup.h> #include <asm/uv/uv.h> -#include <asm/debugreg.h> #include <linux/mc146818rtc.h> #include <asm/smpboot_hooks.h>
@@ -328,7 +327,6 @@ notrace static void __cpuinit start_secondary(void *unused) x86_cpuinit.setup_percpu_clockev ();wmb (); -load_debug_registers ();cpu_idle (); }
@@ -1269,7 +1267,6 @@ void cpu_disable_common(void)remove_cpu_from_maps (cpu);unlock_vector_lock ();fixup_irqs (); -hw_breakpoint_disable (); }int native_cpu_disable (void )
arch/x86/kvm/x86.c +10 -8
@@ -42,6 +42,7 @@#define CREATE_TRACE_POINTS #include "trace.h" +#include <asm/debugreg.h> #include <asm/uaccess.h> #include <asm/msr.h> #include <asm/desc.h>
@@ -3643,14 +3644,15 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)trace_kvm_entry (vcpu->vcpu_id); kvm_x86_ops->run (vcpu, kvm_run); -if (unlikely (vcpu->arch.switch_db_regs ||test_thread_flag (TIF_DEBUG ))) { -set_debugreg (current->thread.debugreg[0 ],0 ); -set_debugreg (current->thread.debugreg[1 ],1 ); -set_debugreg (current->thread.debugreg[2 ],2 ); -set_debugreg (current->thread.debugreg[3 ],3 ); -set_debugreg (current->thread.debugreg6,6 ); -set_debugreg (current->thread.debugreg7,7 ); - } +/* +* If the guest has used debug registers, at least dr7 +* will be disabled while returning to the host. +* If we don't have active breakpoints in the host, we don't +* care about the messed up debug address registers. But if +* we have some of them active, restore the old state. +*/ +if (__get_cpu_var (dr7) &DR_GLOBAL_ENABLE_MASK ) +hw_breakpoint_restore ();set_bit (KVM_REQ_KICK , &vcpu->requests);local_irq_enable ();
arch/x86/power/cpu.c +0 -6
@@ -105,7 +105,6 @@ static void __save_processor_state(struct saved_context *ctxt) ctxt->cr4 =read_cr4 (); ctxt->cr8 =read_cr8 ();#endif -hw_breakpoint_disable (); }/* Needed by apm.c */
@@ -144,11 +143,6 @@ static void fix_processor_context(void)#endif load_TR_desc ();/* This does ltr */ load_LDT (¤t->active_mm->context);/* This does lldt */ - -/* -* Now maybe reload the debug registers -*/ -load_debug_registers (); }/**
include/linux/hw_breakpoint.h +119 -124
255 lines not on this page · raw message
include/linux/perf_event.h +25 -1
@@ -18,6 +18,10 @@#include <linux/ioctl.h> #include <asm/byteorder.h> +#ifdef CONFIG_HAVE_HW_BREAKPOINT +#include <asm/hw_breakpoint.h> +#endif +/* * User-space ABI bits: */
@@ -31,6 +35,7 @@ enum perf_type_id {PERF_TYPE_TRACEPOINT =2 ,PERF_TYPE_HW_CACHE =3 ,PERF_TYPE_RAW =4 , +PERF_TYPE_BREAKPOINT =5 , PERF_TYPE_MAX, /* non-ABI */ };
@@ -207,6 +212,15 @@ struct perf_event_attr {__u32 wakeup_events;/* wakeup every n events */ __u32 wakeup_watermark;/* bytes before wakeup */ }; + +union { +struct {/* Hardware breakpoint info */ +__u64 bp_addr; +__u32 bp_type; +__u32 bp_len; + }; + }; +__u32 __reserved_2;__u64 __reserved_3;
@@ -476,6 +490,11 @@ struct hw_perf_event {atomic64_t count;struct hrtimer hrtimer; }; +#ifdef CONFIG_HAVE_HW_BREAKPOINT +union {/* breakpoint */ +struct arch_hw_breakpoint info; + }; +#endif };atomic64_t prev_count;u64 sample_period;
@@ -588,7 +607,7 @@ struct perf_event {u64 tstamp_running;u64 tstamp_stopped; -struct perf_event_attr attr; +struct perf_event_attr attr;struct hw_perf_event hw;struct perf_event_context *ctx;
@@ -643,6 +662,8 @@ struct perf_event {perf_callback_t callback; +perf_callback_t event_callback; +#endif /* CONFIG_PERF_EVENTS */ };
@@ -831,6 +852,7 @@ extern int sysctl_perf_event_sample_rate;extern void perf_event_init (void );extern void perf_tp_event (int event_id,u64 addr,u64 count,void *record,int entry_size); +extern void perf_bp_event (struct perf_event *event,void *data);#ifndef perf_misc_flags#define perf_misc_flags (regs) (user_mode(regs) ? PERF_RECORD_MISC_USER : \
@@ -865,6 +887,8 @@ static inline int perf_event_task_enable(void) { return -EINVAL; }static inline void perf_sw_event (u32 event_id,u64 nr,int nmi,struct pt_regs *regs,u64 addr) { } +static inline void +perf_bp_event (struct perf_event *event,void *data) { }static inline void perf_event_mmap (struct vm_area_struct *vma) { }static inline void perf_event_comm (struct task_struct *tsk) { }
kernel/exit.c +5 -0
@@ -49,6 +49,7 @@#include <linux/init_task.h> #include <linux/perf_event.h> #include <trace/events/sched.h> +#include <linux/hw_breakpoint.h> #include <asm/uaccess.h> #include <asm/unistd.h>
@@ -980,6 +981,10 @@ NORET_TYPE void do_exit(long code)proc_exit_connector (tsk);/* +* FIXME: do that only when needed, using sched_exit tracepoint +*/ +flush_ptrace_hw_breakpoint (tsk); +/* * Flush inherited counters to the parent - before the parent * gets woken up by child-exit notifications. */
kernel/hw_breakpoint.c +170 -254
515 lines not on this page · raw message
kernel/perf_event.c +52 -1
77 lines not on this page · raw message
kernel/trace/trace_entries.h +3 -3
@@ -372,11 +372,11 @@ FTRACE_ENTRY(ksym_trace, ksym_trace_entry, F_STRUCT( __field( unsigned long, ip ) __field( unsigned char, type ) - __array( char , ksym_name, KSYM_NAME_LEN ) __array( char , cmd, TASK_COMM_LEN ) + __field( unsigned long, addr ) ), - F_printk("ip: %pF type: %d ksym_name: %s cmd: %s", + F_printk("ip: %pF type: %d ksym_name: %pS cmd: %s", (void *)__entry->ip, (unsigned int )__entry->type, - __entry->ksym_name, __entry->cmd) + (void *)__entry->addr, __entry->cmd) );
kernel/trace/trace_ksym.c +60 -66
241 lines not on this page · raw message
kernel/trace/trace_selftest.c +2 -1
@@ -828,7 +828,8 @@ trace_selftest_startup_ksym(struct tracer *trace, struct trace_array *tr) ksym_selftest_dummy =0 ;/* Register the read-write tracing request */ - ret =process_new_ksym_entry (KSYM_SELFTEST_ENTRY ,HW_BREAKPOINT_RW , + ret =process_new_ksym_entry (KSYM_SELFTEST_ENTRY , +HW_BREAKPOINT_R |HW_BREAKPOINT_W , (unsigned long )(&ksym_selftest_dummy));if (ret <0 ) {
Signature · 1 line
1.6.2.3
Re: [PATCH 5/7] hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
arch/x86/kernel/smpboot.c notrace static void __cpuinit start_secondary(void *unused) · line 332
Quoted 9 lines
#include <
asm/smpboot_hooks.h>arch/x86/kernel/smpboot.c +0
-1@@ -328,7 +327,6 @@ notrace static void __cpuinit start_secondary(void *unused) x86_cpuinit.setup_percpu_clockev ();wmb (); -load_debug_registers ();cpu_idle (); }
So, will the breakpoint values be stored in per-cpu variables and be restored when a cpu (from the list of for_each_possible_cpu) eventually turns online due to cpu-hotplug or resume from hibernate (as originally done by load_debug_registers())?
A few more observations....
int reserve_bp_slot(struct perf_event *bp)
{
...
....
if (!bp->attr.pinned) {
/*
* If there are already flexible counters here,
* there is at least one slot reserved for all
* of them. Just join the party.
*
* Otherwise, check there is at least one free slot
*/
if (!slots.flexible && slots.pinned == HBP_NUM) {
ret = -ENOSPC;
goto end;
}
/* Flexible counters need to keep at least one slot */
} else if (slots.pinned + (!!slots.flexible) == HBP_NUM) {
ret = -ENOSPC;
goto end;
}..
...
}
It appears that you're reserving one slot for the non-pinned breakpoint requests, which I'm afraid wouldn't play well with PPC64 (having one DABR). Even with x86, I can't understand why we should allow the use of only 3 registers for ptrace requests (which are all pinned) on every system even if perf-events will never be used on them.
int __register_perf_hw_breakpoint(struct perf_event *bp) {
..
...
if (!bp->attr.disabled) ret = arch_validate_hwbkpt_settings(bp, bp->ctx->task); return ret; }
Can't the arch_validate_() check be done unconditionally?
struct perf_event * modify_user_hw_breakpoint(struct perf_event *bp,
unsigned long addr, int len,
int type, perf_callback_t triggered, struct task_struct *tsk, bool active)
{
/*
* FIXME: do it without unregistering
* - We don't want to lose our slot
* - If the new bp is incorrect, don't lose the older one
*/
unregister_hw_breakpoint(bp);
return register_user_hw_breakpoint(addr, len, type, triggered,
tsk, active);
}Given that modify_user_hw_breakpoint() is used by ptrace, there's a risk of breaking/changing ptrace's behaviour i.e. new ptrace failure scenarios (while modifying DR0-DR3) which may not be acceptable to user-space applications. I presume that you intend to address the FIXME during the next few iterations itself...
void arch_uninstall_hw_breakpoint(struct perf_event *bp) {
...
....
dr7 = &__get_cpu_var(dr7); *dr7 &= ~encode_dr7(i, info->len, info->type); set_debugreg(*dr7, 7); }
The &= ~encode_dr7 would unset the GE flag in DR7 (encoded as DR_GLOBAL_SLOWDOWN) which, I think is unintended.
struct pmu perf_ops_bp = {
.enable = arch_install_hw_breakpoint,
.disable = arch_uninstall_hw_breakpoint,
.read = hw_breakpoint_pmu_read,
.unthrottle = hw_breakpoint_pmu_unthrottle};
So the enable/disable points to those routines that clear the debug register off its breakpoint values but still don't seem to release the slot (even temporarily)....i.e. nr_bp_flexible is unchanged Will perf-events automatically re-use the disabled slot for breakpoint requests from other perf-event instances?
Thanks,
K.Prasad
Re: [PATCH 5/7] hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
Quoted 4 lines · K.Prasad
Quoted 4 lines
#include <
asm/smpboot_hooks.h>arch/x86/kernel/smpboot.c +0
-0@@ -328,7 +327,6 @@ notrace static void __cpuinit start_secondary(void *unused) x86_cpuinit.setup_percpu_clockev ();
Quoted 10 lines
arch/x86/kernel/smpboot.c notrace static void __cpuinit start_secondary(void *unused) · line 332
Quoted 5 lines
arch/x86/kernel/smpboot.c +0
-1wmb (); -load_debug_registers ();cpu_idle (); }So, will the breakpoint values be stored in per-cpu variables and be restored when a cpu (from the list of
for_each_possible_cpu) eventually turns online due to cpu-hotplug or resume from hibernate (as originally done byload_debug_registers())?
Yep. Perf has callbacks that handle that.
Quoted 25 lines
A few more observations....
int
reserve_bp_slot(struct perf_event *bp)
{
...
....if (!bp->attr.pinned) { /* * If there are already flexible counters here, * there is at least one slot reserved for all * of them. Just join the party. * * Otherwise, check there is at least one free slot */ if (!slots.flexible && slots.pinned == HBP_NUM) { ret = -ENOSPC; goto end; } /* Flexible counters need to keep at least one slot */ } else if (slots.pinned + (!!slots.flexible) == HBP_NUM) { ret = -ENOSPC; goto end; }
Quoted 7 lines
..
...
}It appears that you're reserving one slot for the non-pinned breakpoint requests, which I'm afraid wouldn't play well with PPC64 (having one DABR).
I don't understand what you mean. PPC64 has only one DABR, or...?
Quoted 4 lines
Even with x86, I can't understand why we should allow the use of only 3 registers for ptrace requests (which are all pinned) on every system even if perf-events will never be used on them.
Oh I see.
I'm not reserving any register for non-pinned events if none are registered.
The fact is non-pinned registers can be round-robined. So we need at least register for all of them.
So the constraint here is: if I want to register a non-pinned breakpoint, I need to check if we have a free register left, in this case, reserve it.
But if we don't have any non-pinned breakpoint in use, the four registers are all usable for ptrace or any other pinned breakpoints.
Quoted 11 lines
int
__register_perf_hw_breakpoint(struct perf_event *bp){
..
...if (!bp->attr.disabled) ret = arch_validate_hwbkpt_settings(bp, bp->ctx->task); return ret; }Can't the
arch_validate_()check be done unconditionally?
Perhaps, yeah. I did that because when ptrace writes in an address register, it creates a breakpoint stub, registered but disabled until it is activated through a write in dr7. This is to reserve the breakpoint until we activate it.
But now I'm not sure this is neede. If we register the breakpoint later, only when dr7 is set to enable it, even if we refuse the request, gdb handles it and reports the error.
Quoted 20 lines
struct
perf_event* modify_user_hw_breakpoint(structperf_event*bp,
unsigned long addr, int len,
int type,perf_callback_ttriggered, structtask_struct*tsk, bool active){ /* * FIXME: do it without unregistering * - We don't want to lose our slot * - If the new bp is incorrect, don't lose the older one */ unregister_hw_breakpoint(bp); return register_user_hw_breakpoint(addr, len, type, triggered, tsk, active); }
Quoted 5 lines
Given that
modify_user_hw_breakpoint()is used by ptrace, there's a risk of breaking/changing ptrace's behaviour i.e. new ptrace failure scenarios (while modifying DR0-DR3) which may not be acceptable to user-space applications. I presume that you intend to address the FIXME during the next few iterations itself...
Sure.
Quoted 5 lines
void
arch_uninstall_hw_breakpoint(struct perf_event *bp){
...
....
Quoted 8 lines
dr7 = &__get_cpu_var(dr7); *dr7 &= ~encode_dr7(i, info->len, info->type); set_debugreg(*dr7, 7); }The &= ~encode_dr7 would unset the GE flag in DR7 (encoded as
DR_GLOBAL_SLOWDOWN) which, I think is unintended.
Oh right. I'll fix this.
Quoted 10 lines
struct pmu
perf_ops_bp= {.enable = arch_install_hw_breakpoint, .disable = arch_uninstall_hw_breakpoint, .read = hw_breakpoint_pmu_read, .unthrottle = hw_breakpoint_pmu_unthrottle};
So the enable/disable points to those routines that clear the debug register off its breakpoint values but still don't seem to release the slot (even temporarily)....i.e.
nr_bp_flexibleis unchanged
nr_bp_flexible and others are just logical constructs that keep track of the breakpoints registered (ie: that can be scheduled in at any time).
But they have nothing related to the registers themselves. The registers are used by the enable/disable callback. On enable time, we seek a free register and bind to it, on disable, we release the register we were using. All that is handled by perf internally that schedules the events depending on the contexts.
Quoted 2 lines
Will perf-events automatically re-use the disabled slot for breakpoint requests from other perf-event instances?
Yeah.
We have a per cpu array[4] of perf events, those keep track of the breakpoints in use.
The below is called when a breakpoint is scheduled out by perf:
void arch_uninstall_hw_breakpoint(struct perf_event *bp)
{
[...]
for (i = 0; i < HBP_NUM; i++) {
struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
if (*slot == bp) {
*slot = NULL; <---- here, we release the slot
break;
}
}
[...]
}And when it is scheduled in:
int arch_install_hw_breakpoint(struct perf_event *bp)
{
for (i = 0; i < HBP_NUM; i++) {
struct perf_event **slot = &__get_cpu_var(bp_per_reg[i]);
if (!*slot) {
*slot = bp; <-- here we bind to a free slot
break;
}
}
}Thanks for your reviews!
Re: [PATCH 5/7] hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
Quoted 30 lines · Frederic Weisbecker
Quoted 29 lines · K.Prasad
A few more observations....
int
reserve_bp_slot(struct perf_event *bp)
{
...
....if (!bp->attr.pinned) { /* * If there are already flexible counters here, * there is at least one slot reserved for all * of them. Just join the party. * * Otherwise, check there is at least one free slot */ if (!slots.flexible && slots.pinned == HBP_NUM) { ret = -ENOSPC; goto end; } /* Flexible counters need to keep at least one slot */ } else if (slots.pinned + (!!slots.flexible) == HBP_NUM) { ret = -ENOSPC; goto end; }..
...
}
Quoted 6 lines
Quoted 3 lines
It appears that you're reserving one slot for the non-pinned breakpoint requests, which I'm afraid wouldn't play well with PPC64 (having one DABR).
I don't understand what you mean. PPC64 has only one DABR, or...?
Yes, PPC64 has just one DABR. And so this scheme will allow the first request (be it 'pinned' or 'unpinned') to use the debug register? Sounds fine.
Thanks,
K.Prasad
Re: [PATCH 5/7] hw-breakpoints: Rewrite the hw-breakpoints layer on top of perf events
Quoted 35 lines · K.Prasad
Quoted 34 lines · Frederic Weisbecker
Quoted 32 lines · K.Prasad
A few more observations....
int
reserve_bp_slot(struct perf_event *bp)
{
...
....if (!bp->attr.pinned) { /* * If there are already flexible counters here, * there is at least one slot reserved for all * of them. Just join the party. * * Otherwise, check there is at least one free slot */ if (!slots.flexible && slots.pinned == HBP_NUM) { ret = -ENOSPC; goto end; } /* Flexible counters need to keep at least one slot */ } else if (slots.pinned + (!!slots.flexible) == HBP_NUM) { ret = -ENOSPC; goto end; }..
...
}It appears that you're reserving one slot for the non-pinned breakpoint requests, which I'm afraid wouldn't play well with PPC64 (having one DABR).
Quoted 7 lines
Quoted 2 lines
I don't understand what you mean. PPC64 has only one DABR, or...?
Yes, PPC64 has just one DABR. And so this scheme will allow the first request (be it 'pinned' or 'unpinned') to use the debug register? Sounds fine.
After what you, Paul and Benjamin told me, I now doubt about the possibility of a generic core set of constraints. May be all that should move into x86 code because it will be hard to find robust and flexible enough generic constraints.
Anyway, we'll see how that evolves.
PATCH 6/7 hw-breakpoints: Arbitrate access to pmu following registers constraints
In reply to Frederic WeisbeckerAllow or refuse to build a counter using the breakpoints pmu following given constraints.
We keep track of the pmu users by using three per cpu variables:
- nr_cpu_bp_pinned stores the number of pinned cpu breakpoints counters in the given cpu
- nr_bp_flexible stores the number of non-pinned breakpoints counters in the given cpu.
- task_bp_pinned stores the number of pinned task breakpoints in a cpu
The latter is not a simple counter but gathers the number of tasks that have n pinned breakpoints.
Considering HBP_NUM the number of available breakpoint address registers:
task_bp_pinned[0] is the number of tasks having 1 breakpoint task_bp_pinned[1] is the number of tasks having 2 breakpoints [...]
task_bp_pinned[HBP_NUM - 1] is the number of tasks having the maximum number of registers (HBP_NUM).
When a breakpoint counter is created and wants an access to the pmu, we evaluate the following constraints:
== Non-pinned counter ==
- If attached to a single cpu, check:
(per_cpu(nr_bp_flexible, cpu) || (per_cpu(nr_cpu_bp_pinned, cpu)
+ max(per_cpu(task_bp_pinned, cpu)))) < HBP_NUM
-> If there are already non-pinned counters in this cpu, it
means there is already a free slot for them. Otherwise, we check that the maximum number of per task breakpoints (for this cpu) plus the number of per cpu breakpoint (for this cpu) doesn't cover every registers.
- If attached to every cpus, check:
(per_cpu(nr_bp_flexible, *) || (max(per_cpu(nr_cpu_bp_pinned, *))
+ max(per_cpu(task_bp_pinned, *)))) < HBP_NUM
-> This is roughly the same, except we check the number of per
cpu bp for every cpu and we keep the max one. Same for the per tasks breakpoints.
== Pinned counter ==
- If attached to a single cpu, check:
((per_cpu(nr_bp_flexible, cpu) > 1)
+ per_cpu(nr_cpu_bp_pinned, cpu)
+ max(per_cpu(task_bp_pinned, cpu))) < HBP_NUM
-> Same checks as before. But now the nr_bp_flexible, if any,
must keep one register at least (or flexible breakpoints will never be be fed).
- If attached to every cpus, check:
((per_cpu(nr_bp_flexible, *) > 1)
+ max(per_cpu(nr_cpu_bp_pinned, *))
+ max(per_cpu(task_bp_pinned, *))) < HBP_NUM
Changes in v2:
- Counter -> event rename
Changes in v5:
- Fix unreleased non-pinned task-bound-only counters. We only released it in the first cpu. (Thanks to Paul Mackerras for reporting that)
--- kernel/hw_breakpoint.c | 244 ++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 238 insertions(+), 6 deletions(-)
kernel/hw_breakpoint.c +238 -6
@@ -16,6 +16,8 @@* Copyright (C) 2007 Alan Stern * Copyright (C) IBM Corporation, 2009 * Copyright (C) 2009, Frederic Weisbecker <[email protected]> +* +* Thanks to Ingo Molnar for his many suggestions. */ /*
@@ -44,24 +46,254 @@#include <asm/debugreg.h> #endif -static atomic_t bp_slot; +/* +* Constraints data +*/ + +/* Number of pinned cpu breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, nr_cpu_bp_pinned); -int reserve_bp_slot (struct perf_event *bp) +/* Number of pinned task breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, task_bp_pinned[HBP_NUM]); + +/* Number of non-pinned cpu/task breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, nr_bp_flexible); + +/* Gather the number of total pinned and un-pinned bp in a cpuset */ +struct bp_busy_slots { +unsigned int pinned; +unsigned int flexible; +}; + +/* Serialize accesses to the above constraints */ +static DEFINE_MUTEX (nr_bp_mutex); + +/* +* Report the maximum number of pinned breakpoints a task +* have in this cpu +*/ +static unsigned int max_task_bp_pinned (int cpu) { -if (atomic_inc_return (&bp_slot) ==HBP_NUM ) { -atomic_dec (&bp_slot); +int i; +unsigned int *tsk_pinned =per_cpu (task_bp_pinned, cpu); -return -ENOSPC ; +for (i =HBP_NUM -1 ; i >=0 ; i--) { +if (tsk_pinned[i] >0 ) +return i +1 ; }return 0 ; } +/* +* Report the number of pinned/un-pinned breakpoints we have in +* a given cpu (cpu > -1) or in all of them (cpu = -1). +*/ +static void fetch_bp_busy_slots (struct bp_busy_slots *slots,int cpu) +{ +if (cpu >=0 ) { + slots->pinned =per_cpu (nr_cpu_bp_pinned, cpu); + slots->pinned +=max_task_bp_pinned (cpu); + slots->flexible =per_cpu (nr_bp_flexible, cpu); + +return ; + } + +for_each_online_cpu (cpu) { +unsigned int nr; + + nr =per_cpu (nr_cpu_bp_pinned, cpu); + nr +=max_task_bp_pinned (cpu); + +if (nr > slots->pinned) + slots->pinned = nr; + + nr =per_cpu (nr_bp_flexible, cpu); + +if (nr > slots->flexible) + slots->flexible = nr; + } +} + +/* +* Add a pinned breakpoint for the given task in our constraint table +*/ +static void toggle_bp_task_slot (struct task_struct *tsk,int cpu,bool enable) +{ +int count =0 ; +struct perf_event *bp; +struct perf_event_context *ctx = tsk->perf_event_ctxp; +unsigned int *task_bp_pinned; +struct list_head *list; +unsigned long flags; + +if (WARN_ONCE (!ctx,"No perf context for this task" )) +return ; + + list = &ctx->event_list; + +spin_lock_irqsave (&ctx->lock, flags); + +/* +* The current breakpoint counter is not included in the list +* at the open() callback time +*/ +list_for_each_entry (bp, list, event_entry) { +if (bp->attr.type ==PERF_TYPE_BREAKPOINT ) + count++; + } + +spin_unlock_irqrestore (&ctx->lock, flags); + +if (WARN_ONCE (count <0 ,"No breakpoint counter found in the counter list" )) +return ; + + task_bp_pinned =per_cpu (task_bp_pinned, cpu); +if (enable) { + task_bp_pinned[count]++; +if (count >0 ) + task_bp_pinned[count-1 ]--; + }else { + task_bp_pinned[count]--; +if (count >0 ) + task_bp_pinned[count-1 ]++; + } +} + +static void toggle_bp_flexible_slot (int cpu,bool enable) +{ +if (enable) +per_cpu (nr_bp_flexible, cpu)++; +else +per_cpu (nr_bp_flexible, cpu)--; +} + +/* +* Add/remove the given breakpoint in our constraint table +*/ +static void toggle_bp_slot (struct perf_event *bp,bool enable) +{ +int cpu = bp->cpu; +struct task_struct *tsk = bp->ctx->task; + +/* Flexible */ +if (!bp->attr.pinned) { +if (cpu >=0 ) { +toggle_bp_flexible_slot (cpu, enable); +return ; + } + +for_each_online_cpu (cpu) { +toggle_bp_flexible_slot (cpu, enable); +return ; + } + } +/* Pinned counter task profiling */ +if (tsk) { +if (cpu >=0 ) { +toggle_bp_task_slot (tsk, cpu, enable); +return ; + } + +for_each_online_cpu (cpu) +toggle_bp_task_slot (tsk, cpu, enable); +return ; + } + +/* Pinned counter cpu profiling */ +if (enable) +per_cpu (nr_cpu_bp_pinned, bp->cpu)++; +else +per_cpu (nr_cpu_bp_pinned, bp->cpu)--; +} + +/* +* Contraints to check before allowing this new breakpoint counter: +* +* == Non-pinned counter == +* +* - If attached to a single cpu, check: +* +* (per_cpu(nr_bp_flexible, cpu) || (per_cpu(nr_cpu_bp_pinned, cpu) +* + max(per_cpu(task_bp_pinned, cpu)))) < HBP_NUM +* +* -> If there are already non-pinned counters in this cpu, it means +* there is already a free slot for them. +* Otherwise, we check that the maximum number of per task +* breakpoints (for this cpu) plus the number of per cpu breakpoint +* (for this cpu) doesn't cover every registers. +* +* - If attached to every cpus, check: +* +* (per_cpu(nr_bp_flexible, *) || (max(per_cpu(nr_cpu_bp_pinned, *)) +* + max(per_cpu(task_bp_pinned, *)))) < HBP_NUM +* +* -> This is roughly the same, except we check the number of per cpu +* bp for every cpu and we keep the max one. Same for the per tasks +* breakpoints. +* +* +* == Pinned counter == +* +* - If attached to a single cpu, check: +* +* ((per_cpu(nr_bp_flexible, cpu) > 1) + per_cpu(nr_cpu_bp_pinned, cpu) +* + max(per_cpu(task_bp_pinned, cpu))) < HBP_NUM +* +* -> Same checks as before. But now the nr_bp_flexible, if any, must keep +* one register at least (or they will never be fed). +* +* - If attached to every cpus, check: +* +* ((per_cpu(nr_bp_flexible, *) > 1) + max(per_cpu(nr_cpu_bp_pinned, *)) +* + max(per_cpu(task_bp_pinned, *))) < HBP_NUM +*/ +int reserve_bp_slot (struct perf_event *bp) +{ +struct bp_busy_slots slots = {0 }; +int ret =0 ; + +mutex_lock (&nr_bp_mutex); + +fetch_bp_busy_slots (&slots, bp->cpu); + +if (!bp->attr.pinned) { +/* +* If there are already flexible counters here, +* there is at least one slot reserved for all +* of them. Just join the party. +* +* Otherwise, check there is at least one free slot +*/ +if (!slots.flexible && slots.pinned ==HBP_NUM ) { + ret = -ENOSPC ; + goto end; + } + +/* Flexible counters need to keep at least one slot */ + }else if (slots.pinned + (!!slots.flexible) ==HBP_NUM ) { + ret = -ENOSPC ; + goto end; + } + +toggle_bp_slot (bp, true); + +end: +mutex_unlock (&nr_bp_mutex); + +return ret; +} +void release_bp_slot (struct perf_event *bp) { -atomic_dec (&bp_slot); +mutex_lock (&nr_bp_mutex); + +toggle_bp_slot (bp, false); + +mutex_unlock (&nr_bp_mutex); } +int __register_perf_hw_breakpoint (struct perf_event *bp) {int ret;
Signature · 1 line
1.6.2.3
PATCH v6 6/7 hw-breakpoints: Arbitrate access to pmu following registers constraints
(Just did a small mistake in this patch. The release of non-pinned task-bound counter reference wall still wrong).
This is fixed in this patch (I've updated the branch directly).
---
From: Frederic Weisbecker <[email protected]>
Date: Thu, 10 Sep 2009 09:26:21 +0200
Subject: [PATCH 6/7] hw-breakpoints: Arbitrate access to pmu following registers constraints
Cc: Li Zefan <[email protected]>, Prasad <[email protected]>, Alan Stern <[email protected]>, Peter Zijlstra <[email protected]>, Arnaldo Carvalho de Melo <[email protected]>, Steven Rostedt <[email protected]>, Jan Kiszka <[email protected]>, Jiri Slaby <[email protected]>, Avi Kivity <[email protected]>, Paul Mackerras <[email protected]>, Mike Galbraith <[email protected]>, Masami Hiramatsu <[email protected]>, Paul Mundt <[email protected]>, Arjan van de Ven <[email protected]>
Allow or refuse to build a counter using the breakpoints pmu following given constraints.
We keep track of the pmu users by using three per cpu variables:
- nr_cpu_bp_pinned stores the number of pinned cpu breakpoints counters in the given cpu
- nr_bp_flexible stores the number of non-pinned breakpoints counters in the given cpu.
- task_bp_pinned stores the number of pinned task breakpoints in a cpu
The latter is not a simple counter but gathers the number of tasks that have n pinned breakpoints.
Considering HBP_NUM the number of available breakpoint address registers:
task_bp_pinned[0] is the number of tasks having 1 breakpoint task_bp_pinned[1] is the number of tasks having 2 breakpoints [...]
task_bp_pinned[HBP_NUM - 1] is the number of tasks having the maximum number of registers (HBP_NUM).
When a breakpoint counter is created and wants an access to the pmu, we evaluate the following constraints:
== Non-pinned counter ==
- If attached to a single cpu, check:
(per_cpu(nr_bp_flexible, cpu) || (per_cpu(nr_cpu_bp_pinned, cpu)
+ max(per_cpu(task_bp_pinned, cpu)))) < HBP_NUM
-> If there are already non-pinned counters in this cpu, it
means there is already a free slot for them. Otherwise, we check that the maximum number of per task breakpoints (for this cpu) plus the number of per cpu breakpoint (for this cpu) doesn't cover every registers.
- If attached to every cpus, check:
(per_cpu(nr_bp_flexible, *) || (max(per_cpu(nr_cpu_bp_pinned, *))
+ max(per_cpu(task_bp_pinned, *)))) < HBP_NUM
-> This is roughly the same, except we check the number of per
cpu bp for every cpu and we keep the max one. Same for the per tasks breakpoints.
== Pinned counter ==
- If attached to a single cpu, check:
((per_cpu(nr_bp_flexible, cpu) > 1)
+ per_cpu(nr_cpu_bp_pinned, cpu)
+ max(per_cpu(task_bp_pinned, cpu))) < HBP_NUM
-> Same checks as before. But now the nr_bp_flexible, if any,
must keep one register at least (or flexible breakpoints will never be be fed).
- If attached to every cpus, check:
((per_cpu(nr_bp_flexible, *) > 1)
+ max(per_cpu(nr_cpu_bp_pinned, *))
+ max(per_cpu(task_bp_pinned, *))) < HBP_NUM
Changes in v2:
- Counter -> event rename
Changes in v5:
- Fix unreleased non-pinned task-bound-only counters. We only released it in the first cpu. (Thanks to Paul Mackerras for reporting that)
--- kernel/hw_breakpoint.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 237 insertions(+), 6 deletions(-)
kernel/hw_breakpoint.c +237 -6
@@ -16,6 +16,8 @@* Copyright (C) 2007 Alan Stern * Copyright (C) IBM Corporation, 2009 * Copyright (C) 2009, Frederic Weisbecker <[email protected]> +* +* Thanks to Ingo Molnar for his many suggestions. */ /*
@@ -44,24 +46,253 @@#include <asm/debugreg.h> #endif -static atomic_t bp_slot; +/* +* Constraints data +*/ + +/* Number of pinned cpu breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, nr_cpu_bp_pinned); -int reserve_bp_slot (struct perf_event *bp) +/* Number of pinned task breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, task_bp_pinned[HBP_NUM]); + +/* Number of non-pinned cpu/task breakpoints in a cpu */ +static DEFINE_PER_CPU(unsigned int, nr_bp_flexible); + +/* Gather the number of total pinned and un-pinned bp in a cpuset */ +struct bp_busy_slots { +unsigned int pinned; +unsigned int flexible; +}; + +/* Serialize accesses to the above constraints */ +static DEFINE_MUTEX (nr_bp_mutex); + +/* +* Report the maximum number of pinned breakpoints a task +* have in this cpu +*/ +static unsigned int max_task_bp_pinned (int cpu) { -if (atomic_inc_return (&bp_slot) ==HBP_NUM ) { -atomic_dec (&bp_slot); +int i; +unsigned int *tsk_pinned =per_cpu (task_bp_pinned, cpu); -return -ENOSPC ; +for (i =HBP_NUM -1 ; i >=0 ; i--) { +if (tsk_pinned[i] >0 ) +return i +1 ; }return 0 ; } +/* +* Report the number of pinned/un-pinned breakpoints we have in +* a given cpu (cpu > -1) or in all of them (cpu = -1). +*/ +static void fetch_bp_busy_slots (struct bp_busy_slots *slots,int cpu) +{ +if (cpu >=0 ) { + slots->pinned =per_cpu (nr_cpu_bp_pinned, cpu); + slots->pinned +=max_task_bp_pinned (cpu); + slots->flexible =per_cpu (nr_bp_flexible, cpu); + +return ; + } + +for_each_online_cpu (cpu) { +unsigned int nr; + + nr =per_cpu (nr_cpu_bp_pinned, cpu); + nr +=max_task_bp_pinned (cpu); + +if (nr > slots->pinned) + slots->pinned = nr; + + nr =per_cpu (nr_bp_flexible, cpu); + +if (nr > slots->flexible) + slots->flexible = nr; + } +} + +/* +* Add a pinned breakpoint for the given task in our constraint table +*/ +static void toggle_bp_task_slot (struct task_struct *tsk,int cpu,bool enable) +{ +int count =0 ; +struct perf_event *bp; +struct perf_event_context *ctx = tsk->perf_event_ctxp; +unsigned int *task_bp_pinned; +struct list_head *list; +unsigned long flags; + +if (WARN_ONCE (!ctx,"No perf context for this task" )) +return ; + + list = &ctx->event_list; + +spin_lock_irqsave (&ctx->lock, flags); + +/* +* The current breakpoint counter is not included in the list +* at the open() callback time +*/ +list_for_each_entry (bp, list, event_entry) { +if (bp->attr.type ==PERF_TYPE_BREAKPOINT ) + count++; + } + +spin_unlock_irqrestore (&ctx->lock, flags); + +if (WARN_ONCE (count <0 ,"No breakpoint counter found in the counter list" )) +return ; + + task_bp_pinned =per_cpu (task_bp_pinned, cpu); +if (enable) { + task_bp_pinned[count]++; +if (count >0 ) + task_bp_pinned[count-1 ]--; + }else { + task_bp_pinned[count]--; +if (count >0 ) + task_bp_pinned[count-1 ]++; + } +} + +static void toggle_bp_flexible_slot (int cpu,bool enable) +{ +if (enable) +per_cpu (nr_bp_flexible, cpu)++; +else +per_cpu (nr_bp_flexible, cpu)--; +} + +/* +* Add/remove the given breakpoint in our constraint table +*/ +static void toggle_bp_slot (struct perf_event *bp,bool enable) +{ +int cpu = bp->cpu; +struct task_struct *tsk = bp->ctx->task; + +/* Flexible */ +if (!bp->attr.pinned) { +if (cpu >=0 ) { +toggle_bp_flexible_slot (cpu, enable); +return ; + } + +for_each_online_cpu (cpu) +toggle_bp_flexible_slot (cpu, enable); +return ; + } +/* Pinned counter task profiling */ +if (tsk) { +if (cpu >=0 ) { +toggle_bp_task_slot (tsk, cpu, enable); +return ; + } + +for_each_online_cpu (cpu) +toggle_bp_task_slot (tsk, cpu, enable); +return ; + } + +/* Pinned counter cpu profiling */ +if (enable) +per_cpu (nr_cpu_bp_pinned, bp->cpu)++; +else +per_cpu (nr_cpu_bp_pinned, bp->cpu)--; +} + +/* +* Contraints to check before allowing this new breakpoint counter: +* +* == Non-pinned counter == +* +* - If attached to a single cpu, check: +* +* (per_cpu(nr_bp_flexible, cpu) || (per_cpu(nr_cpu_bp_pinned, cpu) +* + max(per_cpu(task_bp_pinned, cpu)))) < HBP_NUM +* +* -> If there are already non-pinned counters in this cpu, it means +* there is already a free slot for them. +* Otherwise, we check that the maximum number of per task +* breakpoints (for this cpu) plus the number of per cpu breakpoint +* (for this cpu) doesn't cover every registers. +* +* - If attached to every cpus, check: +* +* (per_cpu(nr_bp_flexible, *) || (max(per_cpu(nr_cpu_bp_pinned, *)) +* + max(per_cpu(task_bp_pinned, *)))) < HBP_NUM +* +* -> This is roughly the same, except we check the number of per cpu +* bp for every cpu and we keep the max one. Same for the per tasks +* breakpoints. +* +* +* == Pinned counter == +* +* - If attached to a single cpu, check: +* +* ((per_cpu(nr_bp_flexible, cpu) > 1) + per_cpu(nr_cpu_bp_pinned, cpu) +* + max(per_cpu(task_bp_pinned, cpu))) < HBP_NUM +* +* -> Same checks as before. But now the nr_bp_flexible, if any, must keep +* one register at least (or they will never be fed). +* +* - If attached to every cpus, check: +* +* ((per_cpu(nr_bp_flexible, *) > 1) + max(per_cpu(nr_cpu_bp_pinned, *)) +* + max(per_cpu(task_bp_pinned, *))) < HBP_NUM +*/ +int reserve_bp_slot (struct perf_event *bp) +{ +struct bp_busy_slots slots = {0 }; +int ret =0 ; + +mutex_lock (&nr_bp_mutex); + +fetch_bp_busy_slots (&slots, bp->cpu); + +if (!bp->attr.pinned) { +/* +* If there are already flexible counters here, +* there is at least one slot reserved for all +* of them. Just join the party. +* +* Otherwise, check there is at least one free slot +*/ +if (!slots.flexible && slots.pinned ==HBP_NUM ) { + ret = -ENOSPC ; + goto end; + } + +/* Flexible counters need to keep at least one slot */ + }else if (slots.pinned + (!!slots.flexible) ==HBP_NUM ) { + ret = -ENOSPC ; + goto end; + } + +toggle_bp_slot (bp, true); + +end: +mutex_unlock (&nr_bp_mutex); + +return ret; +} +void release_bp_slot (struct perf_event *bp) { -atomic_dec (&bp_slot); +mutex_lock (&nr_bp_mutex); + +toggle_bp_slot (bp, false); + +mutex_unlock (&nr_bp_mutex); } +int __register_perf_hw_breakpoint (struct perf_event *bp) {int ret;
Signature · 1 line
1.6.2.3
PATCH 7/7 ksym_tracer: Remove KSYM_SELFTEST_ENTRY
In reply to Frederic WeisbeckerFrom: Li Zefan <[email protected]>
The macro used to be used in both trace_selftest.c and trace_ksym.c, but no longer, so remove it from header file.
--- kernel/trace/trace.h | 1 - kernel/trace/trace_selftest.c | 3 ++- 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace.h +0 -1
@@ -370,7 +370,6 @@ int register_tracer(struct tracer *type);void unregister_tracer (struct tracer *type);int is_tracing_stopped (void ); -#define KSYM_SELFTEST_ENTRY "ksym_selftest_dummy"extern int process_new_ksym_entry (char *ksymname,int op,unsigned long addr);extern unsigned long nsecs_to_usecs (unsigned long nsecs);
kernel/trace/trace_selftest.c +2 -1
@@ -828,7 +828,8 @@ trace_selftest_startup_ksym(struct tracer *trace, struct trace_array *tr) ksym_selftest_dummy =0 ;/* Register the read-write tracing request */ - ret =process_new_ksym_entry (KSYM_SELFTEST_ENTRY , + + ret =process_new_ksym_entry ("ksym_selftest_dummy" ,HW_BREAKPOINT_R |HW_BREAKPOINT_W , (unsigned long )(&ksym_selftest_dummy));
Signature · 1 line
1.6.2.3
On Fri, Nov 06, 2009 at 03:28:55AM +0100, Frederic Weisbecker wrote:
<snipped>
Quoted 16 lines
arch/Kconfig | 3 + arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/debugreg.h | 11 +- arch/x86/include/asm/hw_breakpoint.h | 58 +++-- arch/x86/include/asm/processor.h | 12 +- arch/x86/kernel/hw_breakpoint.c | 391 ++++++++++++++-------- arch/x86/kernel/process.c | 9 +- arch/x86/kernel/process_32.c | 26 +-- arch/x86/kernel/process_64.c | 26 +-- arch/x86/kernel/ptrace.c | 182 +++++++---- arch/x86/kernel/smpboot.c | 3 - arch/x86/kvm/x86.c | 18 +- arch/x86/power/cpu.c | 6 - include/asm-generic/hw_breakpoint.h | 139 -------- include/linux/hw_breakpoint.h | 131 ++++++++ include/linux/perf_event.h | 37 ++-
Quoted 8 lines
kernel/exit.c | 5 + kernel/hw_breakpoint.c | 602 +++++++++++++++++++++------------- kernel/perf_event.c | 136 ++++++++- kernel/trace/trace.h | 1 - kernel/trace/trace_entries.h | 6 +- kernel/trace/trace_ksym.c | 126 ++++---- kernel/trace/trace_selftest.c | 4 +- 23 files changed, 1194 insertions(+), 739 deletions(-)
Please make changes to samples/hw_breakpoint/data_breakpoint.c simultaneously....it breaks the build if CONFIG_SAMPLE_HW_BREAKPOINT is selected.
Thanks,
K.Prasad
Quoted 29 lines · K.Prasad
On Fri, Nov 06, 2009 at 03:28:55AM +0100, Frederic Weisbecker wrote:
<snipped>
Quoted 16 lines
arch/Kconfig | 3 + arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/debugreg.h | 11 +- arch/x86/include/asm/hw_breakpoint.h | 58 +++-- arch/x86/include/asm/processor.h | 12 +- arch/x86/kernel/hw_breakpoint.c | 391 ++++++++++++++-------- arch/x86/kernel/process.c | 9 +- arch/x86/kernel/process_32.c | 26 +-- arch/x86/kernel/process_64.c | 26 +-- arch/x86/kernel/ptrace.c | 182 +++++++---- arch/x86/kernel/smpboot.c | 3 - arch/x86/kvm/x86.c | 18 +- arch/x86/power/cpu.c | 6 - include/asm-generic/hw_breakpoint.h | 139 -------- include/linux/hw_breakpoint.h | 131 ++++++++ include/linux/perf_event.h | 37 ++-Quoted 8 lines
kernel/exit.c | 5 + kernel/hw_breakpoint.c | 602 +++++++++++++++++++++------------- kernel/perf_event.c | 136 ++++++++- kernel/trace/trace.h | 1 - kernel/trace/trace_entries.h | 6 +- kernel/trace/trace_ksym.c | 126 ++++---- kernel/trace/trace_selftest.c | 4 +- 23 files changed, 1194 insertions(+), 739 deletions(-)
Quoted 7 lines
Please make changes to
samples/hw_breakpoint/data_breakpoint.csimultaneously....it breaks the build ifCONFIG_SAMPLE_HW_BREAKPOINTis selected.Thanks,
K.Prasad
Sorry.
I've fixed the sample side in my latest iteration.