hw-breakpoints: Make the API generic + support for perfcounters
Subsystems (4)
cover hw-breakpoints: Make the API generic + support for perfcounters
Hi,
This patchset aims to make the hardware breakpoint API generic and also to provide the support for hardware breakpoints from perfcounters.
The hardware breakpoint API changes are a first shot of idea.
The support for perf counter has strange effects. It looks like only a part of the breakpoint events are actually recorded. The triggered callback receives well the events but if there are only few of them, none is passed in userspace (or userspace ignores them, dunno). Once I have some time, I'll take a deeper look inside the perf mmap syscall or whatever could be the origin.
Thanks.
Example which traces the BKL accesses by stressing reiserfs:
bkl=0x$(cat ../../System.map | grep kernel_flag | cut -d" " -f 1) ./perf record -f -g -e 5:$bkl -- dbench -t 20 20
./perf report -g -s s
# Samples: 36
#
# Overhead Symbol
# ........ ......
#
83.33% [k] lock_kernel
|
|--51.72%-- __pwrite64
|
|--27.59%-- 0x7f9d83a17b15
|
|--13.79%-- __open
|
--10.34%-- unlink 16.67% [k] unlock_kernel
|
|--80.00%-- 0x7f9d83a17b15
|
|--20.00%-- unlink
|
--20.00%-- __pwrite64Frederic Weisbecker (5):
hw-breakpoints: Make kernel breakpoints API truly generic
hw-breakpoints: Pull up the target symbol in a generic field
hw-breakpoints: Make user breakpoints API truly generic
perfcounter: Grow the event number to 64 bits
perfcounter: Add support for kernel hardware breakpoints
arch/x86/include/asm/hw_breakpoint.h | 9 ++- arch/x86/kernel/hw_breakpoint.c | 91 +++++++++++----- arch/x86/kernel/ptrace.c | 21 ++-- include/asm-generic/hw_breakpoint.h | 116 ++++++++------------- include/linux/perf_counter.h | 1 + kernel/hw_breakpoint.c | 173 ++++++++++++++++++++++++++++--- kernel/perf_counter.c | 41 +++++++- kernel/trace/trace.h | 5 +- kernel/trace/trace_ksym.c | 67 ++++++------ kernel/trace/trace_selftest.c | 2 +- samples/hw_breakpoint/data_breakpoint.c | 10 +-- 11 files changed, 368 insertions(+), 168 deletions(-)
RFC PATCH 1/5 hw-breakpoints: Make kernel breakpoints API truly generic
To define a kernel hardware breakpoint, one need to define the address, type and length of the breakpoint using arch specific operations and then register it using a core helper.
The first stage is truly not scalable with respect to the number of archictures, because for each of them that support hardware breakpoints, we would need a seperate specific field definition for the breakpoint.
However, the supported breakpoint functionalities may be very different between architectures.
Then this new API tries to compose with the following constraints:
- a given architecture may perhaps not support the triggering on one of the usual memory access (read-write/read/write/execute)
- a given architecture may perhaps not support the ability to trigger a breakpoint only on specific memory access size lower than the word size for this arch.
- a given architecture may perhaps not support breakpoints on addresses range.
The new API changes the following prototype for a kernel breakpoint registration:
int register_kernel_hw_breakpoint(struct hw_breakpoint *bp)
into:
int register_kernel_hw_breakpoint(struct hw_breakpoint *bp,
unsigned long addr,
int len, enum breakpoint_type type)
The choice of passing the breakpoint settings as parameters of the registration helper and not by adding generic fields into the breakpoint structure is motivated by the need of a very specific per arch representation of the breakpoint:
- the arch may only need an address, but could also need a couple for breakpoints in ranges.
- the type is subject to arch interpretation (values of debug registers)
- the length too.
Then, to get back these values from a generic breakpoint structure that have specific encodings into the arch fields, this API comes along with abstract accessors which implementation is arch specific:
- type hw_breakpoint_type(bp)
- addr hw_breakpoint_addr(bp)
However, open debates come along this RFC patch:
- the address could be a generic field in struct hw_breakpoint. If we are dealing with a range breakpoint, then we would just need to compute addr + length to get the end of the range.
- the length and type could also be generic fields of struct hw_breakpoint. It would then be up to the arch to get a translation between such generic values and per arch needs.
--- arch/x86/include/asm/hw_breakpoint.h | 8 +++ arch/x86/kernel/hw_breakpoint.c | 62 +++++++++++++++++++ include/asm-generic/hw_breakpoint.h | 99 ++++++++++--------------------- kernel/hw_breakpoint.c | 12 +++- kernel/trace/trace.h | 5 +- kernel/trace/trace_ksym.c | 53 ++++++++-------- kernel/trace/trace_selftest.c | 2 +- samples/hw_breakpoint/data_breakpoint.c | 6 +-- 8 files changed, 142 insertions(+), 105 deletions(-)
arch/x86/include/asm/hw_breakpoint.h +8 -0
@@ -43,6 +43,8 @@ extern unsigned int hbp_user_refcount[HBP_NUM];extern void arch_install_thread_hw_breakpoint (struct task_struct *tsk);extern void arch_uninstall_thread_hw_breakpoint (void );extern int arch_check_va_in_userspace (unsigned long va,u8 hbp_len); +extern int arch_fill_hw_breakpoint (struct hw_breakpoint *bp,unsigned long addr, +int len,enum breakpoint_type type);extern int arch_validate_hwbkpt_settings (struct hw_breakpoint *bp,struct task_struct *tsk);extern void arch_update_user_hw_breakpoint (int pos,struct task_struct *tsk);
@@ -50,6 +52,12 @@ extern void arch_flush_thread_hw_breakpoint(struct task_struct *tsk);extern void arch_update_kernel_hw_breakpoint (void *);extern int hw_breakpoint_exceptions_notify (struct notifier_block *unused,unsigned long val,void *data); +/* +* Typical accessors +*/ +extern enum breakpoint_type hw_breakpoint_type (struct hw_breakpoint *bp); +extern unsigned long hw_breakpoint_addr (struct hw_breakpoint *bp); +#endif /* __KERNEL__ */ #endif /* _I386_HW_BREAKPOINT_H */
arch/x86/kernel/hw_breakpoint.c +62 -0
74 lines not on this page · raw message
include/asm-generic/hw_breakpoint.h +31 -68
144 lines not on this page · raw message
kernel/hw_breakpoint.c +9 -3
@@ -297,15 +297,21 @@ EXPORT_SYMBOL_GPL(unregister_user_hw_breakpoint);/** * register_kernel_hw_breakpoint - register a hardware breakpoint for kernel space * @bp: the breakpoint structure to register -* -* @bp.info->name or @bp.info->address, @bp.info->len, @bp.info->type and +* @addr: the address where we want to set the breakpoint +* @len: length of the value in memory to break in +* @type: the type of the breakpoint (read/write/execute) * @bp->triggered must be set properly before invocation * */ -int register_kernel_hw_breakpoint (struct hw_breakpoint *bp) +int register_kernel_hw_breakpoint (struct hw_breakpoint *bp,unsigned long addr, +int len,enum breakpoint_type type) {int rc; + rc =arch_fill_hw_breakpoint (bp, addr, len, type); +if (rc) +return rc; + rc =arch_validate_hwbkpt_settings (bp,NULL );if (rc)return rc;
kernel/trace/trace.h +3 -2
@@ -213,12 +213,13 @@ struct syscall_trace_exit { };#define KSYM_SELFTEST_ENTRY "ksym_selftest_dummy" -extern int process_new_ksym_entry (char *ksymname,int op,unsigned long addr); +extern int process_new_ksym_entry (char *ksymname,enum breakpoint_type type, +unsigned long addr);struct ksym_trace_entry {struct trace_entry ent;unsigned long ip; -unsigned char type; +enum breakpoint_type type;char ksym_name[KSYM_NAME_LEN ];char cmd[TASK_COMM_LEN ]; };
kernel/trace/trace_ksym.c +27 -26
151 lines not on this page · raw message
kernel/trace/trace_selftest.c +1 -1
@@ -827,7 +827,7 @@ trace_selftest_startup_ksym(struct tracer *trace, struct trace_array *tr) ksym_selftest_dummy =0 ;/* Register the read-write tracing request */ - ret =process_new_ksym_entry (KSYM_SELFTEST_ENTRY ,HW_BREAKPOINT_RW , + ret =process_new_ksym_entry (KSYM_SELFTEST_ENTRY ,BREAK_RW , (unsigned long )(&ksym_selftest_dummy));if (ret <0 ) {
samples/hw_breakpoint/data_breakpoint.c +1 -5
@@ -51,14 +51,10 @@ static int __init hw_break_module_init(void)#ifdef CONFIG_X86 sample_hbp.info.name = ksym_name; - sample_hbp.info.type =HW_BREAKPOINT_WRITE ; - sample_hbp.info.len =HW_BREAKPOINT_LEN_4 ;#endif /* CONFIG_X86 */ - sample_hbp.triggered = (void *)sample_hbp_handler; - ret =register_kernel_hw_breakpoint (&sample_hbp); - + ret =register_kernel_hw_breakpoint (&sample_hbp,0 ,4 ,BREAK_W );if (ret <0 ) {printk (KERN_INFO "Breakpoint registration failed\n" );return ret;
Signature · 1 line
1.6.2.3
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
* Frederic Weisbecker ([email protected]) wrote:
[...]
Quoted 5 lines
kernel/hw_breakpoint.c +0
-0
kernel/hw_breakpoint.c EXPORT_SYMBOL_GPL(unregister_user_hw_breakpoint); · line 303
Quoted 9 lines
kernel/hw_breakpoint.c +3
-2@@ -297,15 +297,21 @@ EXPORT_SYMBOL_GPL(unregister_user_hw_breakpoint);/** * register_kernel_hw_breakpoint - register a hardware breakpoint for kernel space * @bp: the breakpoint structure to register -* -* @bp.info->name or @bp.info->address, @bp.info->len, @bp.info->type and +* @addr: the address where we want to set the breakpoint +* @len: length of the value in memory to break in +* @type: the type of the breakpoint (read/write/execute) * @bp->triggered must be set properly before invocation
Hi Frederic,
I think one of the great addition in this patchset is to allow using breakpoints from arch-agnostic code.
It becomes important to document the error values which can be returned by register_kernel_hw_breakpoint, so this will serve as guidelines for architecture-specific arch_fill_hw_breakpoint() implementation. This will become increasingly important, as this abstraction layer will basically be responsible for either:
- Finding the best support the architecture can provide for a given hw breakpoint.
- Failing with an explicit error value telling the in-kernel user why it failed (e.g. if it must use a fallback, or return the error to the user).
Maybe we should think of a more flexible breakpoint type mapping too, e.g.:
monitor _strictly_ execute operation on address 0x...
-> would fail if the architecture does not support execution access
monitoring
monitor (at least) execute operations on address 0x...
-> would be allowed to use a more general monitor (e.g. RWX) if the
architecture does not support "execute only" monitor.
(same for read and write)
Mathieu
Quoted 15 lines
kernel/hw_breakpoint.c +6
-1* */ -int register_kernel_hw_breakpoint (struct hw_breakpoint *bp) +int register_kernel_hw_breakpoint (struct hw_breakpoint *bp,unsigned long addr, +int len,enum breakpoint_type type) {int rc; + rc =arch_fill_hw_breakpoint (bp, addr, len, type); +if (rc) +return rc; + rc =arch_validate_hwbkpt_settings (bp,NULL );if (rc)return rc;
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
Quoted 22 lines · Mathieu Desnoyers
* Frederic Weisbecker ([email protected]) wrote:
[...]Quoted 5 lines
kernel/hw_breakpoint.c +0
-0kernel/hw_breakpoint.c EXPORT_SYMBOL_GPL(unregister_user_hw_breakpoint); · line 303
Quoted 9 lines
kernel/hw_breakpoint.c +3
-2@@ -297,15 +297,21 @@ EXPORT_SYMBOL_GPL(unregister_user_hw_breakpoint);/** * register_kernel_hw_breakpoint - register a hardware breakpoint for kernel space * @bp: the breakpoint structure to register -* -* @bp.info->name or @bp.info->address, @bp.info->len, @bp.info->type and +* @addr: the address where we want to set the breakpoint +* @len: length of the value in memory to break in +* @type: the type of the breakpoint (read/write/execute) * @bp->triggered must be set properly before invocation Hi Frederic,
I think one of the great addition in this patchset is to allow using breakpoints from arch-agnostic code.
Quoted 8 lines
It becomes important to document the error values which can be returned by
register_kernel_hw_breakpoint, so this will serve as guidelines for architecture-specificarch_fill_hw_breakpoint()implementation. This will become increasingly important, as this abstraction layer will basically be responsible for either:- Finding the best support the architecture can provide for a given hw breakpoint.
Indeed, and that's the biggest problem it has to face because supported hardware breakpoint features are very differents from one architecture to another.
Quoted 3 lines
- Failing with an explicit error value telling the in-kernel user why it failed (e.g. if it must use a fallback, or return the error to the user).
Yeah, nice point, I'll send another iteration which better documents the error return values, at least once I get a mostly agreed core implementation :-)
Quoted 10 lines
Maybe we should think of a more flexible breakpoint type mapping too, e.g.:
monitor _strictly_ execute operation on address 0x...
-> would fail if the architecture does not support execution access
monitoring
monitor (at least) execute operations on address 0x...
-> would be allowed to use a more general monitor (e.g. RWX) if the
architecture does not support "execute only" monitor.
Quoted 3 lines
(same for read and write)
Mathieu
Well, I'm not sure the problem mostly resides in the hardware implementation of strict exec breakpoint types. But I guess your point is not limiting to that. Yeah for example, x86 doesn't support read-only breakpoints. But I guess that can be simulated using software artifacts, for example using READ-WRITE breakpoints + the x86 decoder API, recently submitted by Masami, to find the nature of the current instruction.
Anyway, your point is indeed important: return common error values for unsupported breakpoint operations.
Thanks.
Quoted 19 lines
Quoted 16 lines
kernel/hw_breakpoint.c +6
-1* */ -int register_kernel_hw_breakpoint (struct hw_breakpoint *bp) +int register_kernel_hw_breakpoint (struct hw_breakpoint *bp,unsigned long addr, +int len,enum breakpoint_type type) {int rc; + rc =arch_fill_hw_breakpoint (bp, addr, len, type); +if (rc) +return rc; + rc =arch_validate_hwbkpt_settings (bp,NULL );if (rc)return rc;Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
Quoted 1 line · * Frederic Weisbecker ([email protected]) wrote:
On Mon, Jul 20, 2009 at 01:27:48PM -0400, Mathieu Desnoyers wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Maybe we should think of a more flexible breakpoint type mapping too, e.g.:
monitor _strictly_ execute operation on address 0x...
-> would fail if the architecture does not support execution access
monitoring
monitor (at least) execute operations on address 0x...
-> would be allowed to use a more general monitor (e.g. RWX) if the
architecture does not support "execute only" monitor.
Quoted 8 lines
Quoted 3 lines
(same for read and write)
Mathieu
Well, I'm not sure the problem mostly resides in the hardware implementation of strict exec breakpoint types. But I guess your point is not limiting to that. Yeah for example, x86 doesn't support read-only breakpoints.
Exactly. I used "execute" only as an example, but in the end, we could end up with a list looking like:
HW_WATCH_R (1 << 0)HW_WATCH_NOT_R (1 << 0)HW_WATCH_W (1 << 1)HW_WATCH_NOT_W (1 << 1)HW_WATCH_X (1 << 2)HW_WATCH_NOT_X (1 << 2)
So for instance, flags :
HW_WATCH_R|HW_WATCH_NOT_W|HW_WATCH_NOT_X
would specify that the architecture has to support a "read" watchpoint which does not trigger on write nor execute.
HW_WATCH_R
would specify that the architecture _must_ support "read" watchpoint, and we don't care about W or X.
HW_WATCH_R|HW_WATCH_W|HW_WATCH_X
Would ask for watching rwx on an address. The architecture would have to support all those three.
Some combinations would be invalid (e.g. HW_WATCH_R|HW_WATCH_NOT_R).
There might be better ways to express this, but at this it should show my point a bit more clearly.
Mathieu
Quoted 9 lines
But I guess that can be simulated using software artifacts, for example using READ-WRITE breakpoints + the x86 decoder API, recently submitted by Masami, to find the nature of the current instruction.
Anyway, your point is indeed important: return common error values for unsupported breakpoint operations.
Thanks.
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
Quoted 50 lines · Mathieu Desnoyers
Quoted 1 line · * Frederic Weisbecker ([email protected]) wrote:
On Mon, Jul 20, 2009 at 01:27:48PM -0400, Mathieu Desnoyers wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Maybe we should think of a more flexible breakpoint type mapping too, e.g.:
monitor _strictly_ execute operation on address 0x...
-> would fail if the architecture does not support execution access
monitoring
monitor (at least) execute operations on address 0x...
-> would be allowed to use a more general monitor (e.g. RWX) if the
architecture does not support "execute only" monitor.Quoted 8 lines
Quoted 3 lines
(same for read and write)
Mathieu
Well, I'm not sure the problem mostly resides in the hardware implementation of strict exec breakpoint types. But I guess your point is not limiting to that. Yeah for example, x86 doesn't support read-only breakpoints.
Exactly. I used "execute" only as an example, but in the end, we could end up with a list looking like:
HW_WATCH_R(1 << 0)HW_WATCH_NOT_R(1 << 0)HW_WATCH_W(1 << 1)HW_WATCH_NOT_W(1 << 1)HW_WATCH_X(1 << 2)HW_WATCH_NOT_X(1 << 2)So for instance, flags :
HW_WATCH_R|HW_WATCH_NOT_W|HW_WATCH_NOT_X
would specify that the architecture has to support a "read" watchpoint which does not trigger on write nor execute.
HW_WATCH_Rwould specify that the architecture _must_ support "read" watchpoint, and we don't care about W or X.
HW_WATCH_R|HW_WATCH_W|HW_WATCH_X
Would ask for watching rwx on an address. The architecture would have to support all those three.
Quoted 6 lines
Some combinations would be invalid (e.g. HW_WATCH_R|HW_WATCH_NOT_R).
There might be better ways to express this, but at this it should show my point a bit more clearly.
Mathieu
Ok, I see your point.
Before making a choice to generically express the modes supported by an arch, I need a good global view of what is supported by most archs, thing that I lack and which I should look at quickly :)
Thanks.
Quoted 14 lines
Quoted 10 lines
But I guess that can be simulated using software artifacts, for example using READ-WRITE breakpoints + the x86 decoder API, recently submitted by Masami, to find the nature of the current instruction.
Anyway, your point is indeed important: return common error values for unsupported breakpoint operations.
Thanks.
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
In reply to Frederic WeisbeckerQuoted 26 lines · Frederic Weisbecker
To define a kernel hardware breakpoint, one need to define the address, type and length of the breakpoint using arch specific operations and then register it using a core helper.
The first stage is truly not scalable with respect to the number of archictures, because for each of them that support hardware breakpoints, we would need a seperate specific field definition for the breakpoint.
However, the supported breakpoint functionalities may be very different between architectures.
Then this new API tries to compose with the following constraints:- a given architecture may perhaps not support the triggering on one of the usual memory access (read-write/read/write/execute)
- a given architecture may perhaps not support the ability to trigger a breakpoint only on specific memory access size lower than the word size for this arch.
- a given architecture may perhaps not support breakpoints on addresses range.
The new API changes the following prototype for a kernel breakpoint registration:
Quoted 7 lines
int
register_kernel_hw_breakpoint(struct hw_breakpoint *bp)into:
int register_kernel_hw_breakpoint(struct
hw_breakpoint*bp,
unsigned long addr,
int len, enumbreakpoint_typetype)
It is not clear how adding these new parameters to the interface would help it become generic, as opposed to moving them to 'struct hw_breakpoint'.
It would make the usage cumbersome of some architectures - say for instance the PPC64 which always has a breakpoint length of 8 bytes. So the user needs to specify either '8' always or '0' to indicate variable length not supported (but it is counter-intuitive..may be interpreted as zero-length).
Quoted 20 lines
The choice of passing the breakpoint settings as parameters of the registration helper and not by adding generic fields into the breakpoint structure is motivated by the need of a very specific per arch representation of the breakpoint:
- the arch may only need an address, but could also need a couple for breakpoints in ranges.
- the type is subject to arch interpretation (values of debug registers)
- the length too.Then, to get back these values from a generic breakpoint structure that have specific encodings into the arch fields, this API comes along with abstract accessors which implementation is arch specific:
- type
hw_breakpoint_type(bp)
- addrhw_breakpoint_addr(bp)However, open debates come along this RFC patch:
Quoted 8 lines
- the address could be a generic field in struct
hw_breakpoint. If we are dealing with a range breakpoint, then we would just need to compute addr + length to get the end of the range.- the length and type could also be generic fields of struct
hw_breakpoint. It would then be up to the arch to get a translation between such generic values and per arch needs.
While the issues have been enumerated above, the patchset only pushes the issue into a different domain i.e. make the user determine if a breakpoint type or len is supported in a given architecture vs the existing implementation in which the user determines if a constant pertaining to a given len/type is defined. But the accessor-routines hw_breakpoint_type() and hw_breakpoint_addr() make it much easier to use and is a good addition.
To make the usage much easier, I would see a combination of the following:
- Define constants/enums for length and type that are common to all architectures.
- Define accessor routines that help determine if a given type/len is supported on the host processor.
- Move fields such as address, len and type to generic breakpoint structure (if it still matters despite the two changes above).
Let me know what you think.
Thanks,
K.Prasad
Re: [RFC][PATCH 1/5] hw-breakpoints: Make kernel breakpoints API truly generic
Quoted 39 lines · K.Prasad
Quoted 26 lines · Frederic Weisbecker
To define a kernel hardware breakpoint, one need to define the address, type and length of the breakpoint using arch specific operations and then register it using a core helper.
The first stage is truly not scalable with respect to the number of archictures, because for each of them that support hardware breakpoints, we would need a seperate specific field definition for the breakpoint.
However, the supported breakpoint functionalities may be very different between architectures.
Then this new API tries to compose with the following constraints:- a given architecture may perhaps not support the triggering on one of the usual memory access (read-write/read/write/execute)
- a given architecture may perhaps not support the ability to trigger a breakpoint only on specific memory access size lower than the word size for this arch.
- a given architecture may perhaps not support breakpoints on addresses range.
The new API changes the following prototype for a kernel breakpoint registration:
Quoted 7 lines
int
register_kernel_hw_breakpoint(struct hw_breakpoint *bp)into:
int register_kernel_hw_breakpoint(struct
hw_breakpoint*bp,
unsigned long addr,
int len, enumbreakpoint_typetype)It is not clear how adding these new parameters to the interface would help it become generic, as opposed to moving them to 'struct
hw_breakpoint'.
Quoted 5 lines
It would make the usage cumbersome of some architectures - say for instance the PPC64 which always has a breakpoint length of 8 bytes. So the user needs to specify either '8' always or '0' to indicate variable length not supported (but it is counter-intuitive..may be interpreted as zero-length).
Hm, so may be we should only push the target and the type and set the most intuitive access length automatically? ie: 1 for x86 so that it triggers for every accesses, and 8 for ppc?
Also how is actually interpreted an 8 bytes access on ppc? Ie: is a single byte access actually interpreted through an 8 byte access because of memory access constraints?
Quoted 37 lines
Quoted 20 lines
The choice of passing the breakpoint settings as parameters of the registration helper and not by adding generic fields into the breakpoint structure is motivated by the need of a very specific per arch representation of the breakpoint:
- the arch may only need an address, but could also need a couple for breakpoints in ranges.
- the type is subject to arch interpretation (values of debug registers)
- the length too.Then, to get back these values from a generic breakpoint structure that have specific encodings into the arch fields, this API comes along with abstract accessors which implementation is arch specific:
- type
hw_breakpoint_type(bp)
- addrhw_breakpoint_addr(bp)However, open debates come along this RFC patch:
Quoted 8 lines
- the address could be a generic field in struct
hw_breakpoint. If we are dealing with a range breakpoint, then we would just need to compute addr + length to get the end of the range.- the length and type could also be generic fields of struct
hw_breakpoint. It would then be up to the arch to get a translation between such generic values and per arch needs.While the issues have been enumerated above, the patchset only pushes the issue into a different domain i.e. make the user determine if a breakpoint type or len is supported in a given architecture vs the existing implementation in which the user determines if a constant pertaining to a given len/type is defined. But the accessor-routines
hw_breakpoint_type()andhw_breakpoint_addr()make it much easier to use and is a good addition.
Quoted 5 lines
To make the usage much easier, I would see a combination of the following:
- Define constants/enums for length and type that are common to all architectures.
Indeed, in the hope that most basic types of breakpoints are supported by every architectures (x and rw).
Quoted 2 lines
- Define accessor routines that help determine if a given type/len is supported on the host processor.
Or may be we should forget about the length as stated above?
But indeed we can think about an arch routine that reports the supported types, that would provide a better granularity of error message than getting a "non implemented" error while registering a breakpoint.
Quoted 2 lines
- Move fields such as address, len and type to generic breakpoint structure (if it still matters despite the two changes above).
Ok, I guess that would indeed be better to do that than using abstract accessors. Let the arch interpret these generic informations by itself.
Thanks!
Quoted 10 lines
Let me know what you think.
Thanks,
K.Prasad--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
RFC PATCH 2/5 hw-breakpoints: Pull up the target symbol in a generic field
In reply to Frederic WeisbeckerThe target symbol of a breakpoint doesn't need to be arch specific. It can be resolved back from the core, then this patch pulls up this job to the core, avoiding the need to implement that from arch.
--- arch/x86/include/asm/hw_breakpoint.h | 1 - arch/x86/kernel/hw_breakpoint.c | 35 ++++-------------------------- include/asm-generic/hw_breakpoint.h | 1 + kernel/hw_breakpoint.c | 17 +++++++++++++++ kernel/trace/trace_ksym.c | 14 ++++++------ samples/hw_breakpoint/data_breakpoint.c | 4 +-- 6 files changed, 31 insertions(+), 41 deletions(-)
arch/x86/include/asm/hw_breakpoint.h +0 -1
@@ -5,7 +5,6 @@#define __ARCH_HW_BREAKPOINT_Hstruct arch_hw_breakpoint { -char *name;/* Contains name of the symbol to set bkpt */ unsigned long address;u8 len;u8 type;
arch/x86/kernel/hw_breakpoint.c +5 -30
@@ -195,28 +195,9 @@ static int arch_check_va_in_kernelspace(unsigned long va, u8 hbp_len) }/* -* Store a breakpoint's encoded address, length, and type. +* Pick breakpoint's generic address, length, and type and convert them +* into x86 specific debug register values. */ -static int arch_store_info (struct hw_breakpoint *bp,struct task_struct *tsk) -{ -/* -* User-space requests will always have the address field populated -* Symbol names from user-space are rejected -*/ -if (tsk && bp->info.name) -return -EINVAL ; -/* -* For kernel-addresses, either the address or symbol name can be -* specified. -*/ -if (bp->info.name) - bp->info.address = (unsigned long ) -kallsyms_lookup_name (bp->info.name); -if (bp->info.address) -return 0 ; -return -EINVAL ; -} -int arch_fill_hw_breakpoint (struct hw_breakpoint *bp,unsigned long addr,int len,enum breakpoint_type type) {
@@ -267,7 +248,6 @@ int arch_validate_hwbkpt_settings(struct hw_breakpoint *bp, struct task_struct *tsk) {unsigned int align; -int ret = -EINVAL ;switch (bp->info.type) {/*
@@ -279,14 +259,14 @@ int arch_validate_hwbkpt_settings(struct hw_breakpoint *bp,if ((!arch_check_va_in_userspace (bp->info.address, bp->info.len)) && bp->info.len !=HW_BREAKPOINT_LEN_EXECUTE ) -return ret; +return -EINVAL ;break ;case HW_BREAKPOINT_WRITE :break ;case HW_BREAKPOINT_RW :break ;default : -return ret; +return -EINVAL ; }switch (bp->info.len) {
@@ -305,14 +285,9 @@ int arch_validate_hwbkpt_settings(struct hw_breakpoint *bp,break ;#endif default : -return ret; +return -EINVAL ; } -if (bp->triggered) - ret =arch_store_info (bp, tsk); - -if (ret <0 ) -return ret;/* * Check that the low-order bits of the address are appropriate * for the alignment implied by len.
include/asm-generic/hw_breakpoint.h +1 -0
@@ -67,6 +67,7 @@ enum breakpoint_type {* */ struct hw_breakpoint { +char *name;/* Contains name of the symbol to set breakpoint */ void (*triggered)(struct hw_breakpoint *,struct pt_regs *);struct arch_hw_breakpoint info; };
kernel/hw_breakpoint.c +17 -0
@@ -225,6 +225,13 @@ int register_user_hw_breakpoint(struct task_struct *tsk,struct thread_struct *thread = &(tsk->thread);int i, rc = -ENOSPC ; +/* +* User-space requests will always have the address field populated +* Symbol names from user-space are rejected +*/ +if (bp->name || !bp->triggered) +return -EINVAL ; +spin_lock_bh (&hw_breakpoint_lock);for (i =0 ; i < hbp_kernel_pos; i++) {
@@ -308,6 +315,16 @@ int register_kernel_hw_breakpoint(struct hw_breakpoint *bp, unsigned long addr, {int rc; +/* +* For kernel-addresses, either the address or symbol name can be +* specified. +*/ +if (bp->name) + addr = (unsigned long )kallsyms_lookup_name (bp->name); + +if (!addr || !bp->triggered) +return -EINVAL ; + rc =arch_fill_hw_breakpoint (bp, addr, len, type);if (rc)return rc;
kernel/trace/trace_ksym.c +7 -7
@@ -96,7 +96,7 @@ void ksym_hbp_handler(struct hw_breakpoint *hbp, struct pt_regs *regs) entry =ring_buffer_event_data (event); entry->ip =instruction_pointer (regs); entry->type =hw_breakpoint_type (hbp); -strlcpy (entry->ksym_name, hbp->info.name,KSYM_SYMBOL_LEN ); +strlcpy (entry->ksym_name, hbp->name,KSYM_SYMBOL_LEN );strlcpy (entry->cmd, current->comm,TASK_COMM_LEN );#ifdef CONFIG_PROFILE_KSYM_TRACER
@@ -200,8 +200,8 @@ int process_new_ksym_entry(char *ksymname, enum breakpoint_type type,if (!entry->ksym_hbp) goto err; - entry->ksym_hbp->info.name =kstrdup (ksymname,GFP_KERNEL ); -if (!entry->ksym_hbp->info.name) + entry->ksym_hbp->name =kstrdup (ksymname,GFP_KERNEL ); +if (!entry->ksym_hbp->name) goto err; entry->ksym_addr = addr;
@@ -219,7 +219,7 @@ int process_new_ksym_entry(char *ksymname, enum breakpoint_type type,return 0 ; err:if (entry->ksym_hbp) -kfree (entry->ksym_hbp->info.name); +kfree (entry->ksym_hbp->name);kfree (entry->ksym_hbp);kfree (entry);return ret;
@@ -242,7 +242,7 @@ static ssize_t ksym_trace_filter_read(struct file *filp, char __user *ubuf,mutex_lock (&ksym_tracer_mutex);hlist_for_each_entry (entry, node, &ksym_filter_head, ksym_hlist) { - ret =trace_seq_printf (s,"%s:" , entry->ksym_hbp->info.name); + ret =trace_seq_printf (s,"%s:" , entry->ksym_hbp->name);if (hw_breakpoint_type (entry->ksym_hbp) ==BREAK_W ) ret =trace_seq_puts (s,"-w-\n" );else if (hw_breakpoint_type (entry->ksym_hbp) ==BREAK_RW )
@@ -310,7 +310,7 @@ static ssize_t ksym_trace_filter_write(struct file *file, ksym_filter_entry_count--;hlist_del_rcu (&(entry->ksym_hlist));synchronize_rcu (); -kfree (entry->ksym_hbp->info.name); +kfree (entry->ksym_hbp->name);kfree (entry->ksym_hbp);kfree (entry); goto out;
@@ -350,7 +350,7 @@ static void ksym_trace_reset(struct trace_array *tr) ksym_filter_entry_count--;hlist_del_rcu (&(entry->ksym_hlist));synchronize_rcu (); -kfree (entry->ksym_hbp->info.name); +kfree (entry->ksym_hbp->name);kfree (entry->ksym_hbp);kfree (entry); }
samples/hw_breakpoint/data_breakpoint.c +1 -3
@@ -49,9 +49,7 @@ static int __init hw_break_module_init(void) {int ret; -#ifdef CONFIG_X86 - sample_hbp.info.name = ksym_name; -#endif /* CONFIG_X86 */ + sample_hbp.name = ksym_name; sample_hbp.triggered = (void *)sample_hbp_handler; ret =register_kernel_hw_breakpoint (&sample_hbp,0 ,4 ,BREAK_W );
Signature · 1 line
1.6.2.3
RFC PATCH 3/5 hw-breakpoints: Make user breakpoints API truly generic
In reply to Frederic WeisbeckerFollowing the API changes on the kernel breakpoints API, the user breakpoints registration now follows the same pattern.
The target, access length and type and now given as parameters in the registration helpers to avoid per arch code in generic code.
However, to keep registering or modifying easily a user breakpoint from arch code with a prefilled arch breakpoint structure, we provide two new helpers:
- register_user_hw_breakpoint_filled(tsk, bp)
- modify_user_hw_breakpoint_filled(tsk, bp)
--- arch/x86/kernel/ptrace.c | 21 ++++++----- include/asm-generic/hw_breakpoint.h | 12 +++++- kernel/hw_breakpoint.c | 65 +++++++++++++++++++++++++++++----- 3 files changed, 76 insertions(+), 22 deletions(-)
arch/x86/kernel/ptrace.c +11 -10
@@ -525,17 +525,18 @@ restore:if (!bp) { rc = -ENOMEM ; bp =kzalloc (sizeof (struct hw_breakpoint ),GFP_KERNEL ); -if (bp) { - bp->info.address = thread->debugreg[i]; - bp->triggered = ptrace_triggered; - bp->info.len = len; - bp->info.type = type; - rc =register_user_hw_breakpoint (tsk, bp); -if (rc) -kfree (bp); - } +if (!bp) +break ; + + bp->info.address = thread->debugreg[i]; + bp->triggered = ptrace_triggered; + bp->info.len = len; + bp->info.type = type; + rc =register_user_hw_breakpoint_filled (tsk, bp); +if (rc) +kfree (bp); }else - rc =modify_user_hw_breakpoint (tsk, bp); + rc =modify_user_hw_breakpoint_filled (tsk, bp);if (rc)break ; }
include/asm-generic/hw_breakpoint.h +10 -2
@@ -83,10 +83,18 @@ struct hw_breakpoint {* 1-, 2-, and 4-byte lengths may be unavailable. */ +extern int register_user_hw_breakpoint_filled (struct task_struct *tsk, +struct hw_breakpoint *bp);extern int register_user_hw_breakpoint (struct task_struct *tsk, -struct hw_breakpoint *bp); +struct hw_breakpoint *bp, +unsigned long addr,int len, +enum breakpoint_type type); +extern int modify_user_hw_breakpoint_filled (struct task_struct *tsk, +struct hw_breakpoint *bp);extern int modify_user_hw_breakpoint (struct task_struct *tsk, -struct hw_breakpoint *bp); +struct hw_breakpoint *bp, +unsigned long addr, +int len,enum breakpoint_type type);extern void unregister_user_hw_breakpoint (struct task_struct *tsk,struct hw_breakpoint *bp);/*
kernel/hw_breakpoint.c +55 -10
92 lines not on this page · raw message
Signature · 1 line
1.6.2.3
RFC PATCH 4/5 perfcounter: Grow the event number to 64 bits
In reply to Frederic WeisbeckerThe event number are defined by the config attr of a counter. When it is passed while adding a hit, it is compared against the initial config attr, a 64 bits number, to retrieve the targetted counter.
But when we want to add a hit, this event is passed through 32 bits parameters, which makes loosing the real event number if its storage needs more (case of an address).
This patch changes the event parameter in some of the helpers which add the event hits.
--- kernel/perf_counter.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
kernel/perf_counter.c +3 -3
@@ -3435,7 +3435,7 @@ static int perf_swcounter_is_counting(struct perf_counter *counter)static int perf_swcounter_match (struct perf_counter *counter,enum perf_type_id type, -u32 event,struct pt_regs *regs) +u64 event,struct pt_regs *regs) {if (!perf_swcounter_is_counting (counter))return 0 ;
@@ -3467,7 +3467,7 @@ static void perf_swcounter_add(struct perf_counter *counter, u64 nr,static void perf_swcounter_ctx_event (struct perf_counter_context *ctx,enum perf_type_id type, -u32 event,u64 nr,int nmi, +u64 event,u64 nr,int nmi,struct perf_sample_data *data) {struct perf_counter *counter;
@@ -3497,7 +3497,7 @@ static int *perf_swcounter_recursion_context(struct perf_cpu_context *cpuctx)return &cpuctx->recursion[0 ]; } -static void do_perf_swcounter_event (enum perf_type_id type,u32 event, +static void do_perf_swcounter_event (enum perf_type_id type,u64 event,u64 nr,int nmi,struct perf_sample_data *data) {
Signature · 1 line
1.6.2.3
RFC PATCH 5/5 perfcounter: Add support for kernel hardware breakpoints
In reply to Frederic WeisbeckerThis adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Example which traces the BKL accesses by stressing reiserfs:
bkl=0x$(cat ../../System.map | grep kernel_flag | cut -d" " -f 1) ./perf record -f -g -e 5:$bkl -- dbench -t 20 20
./perf report -g -s s
# Samples: 36
#
# Overhead Symbol
# ........ ......
#
83.33% [k] lock_kernel
|
|--51.72%-- __pwrite64
|
|--27.59%-- 0x7f9d83a17b15
|
|--13.79%-- __open
|
--10.34%-- unlink 16.67% [k] unlock_kernel
|
|--80.00%-- 0x7f9d83a17b15
|
|--20.00%-- unlink
|
--20.00%-- __pwrite64For now you can only pass raw kernel addresses.
What is planned next:
- profile by symbol names
- profile with user breakpoints
--- include/asm-generic/hw_breakpoint.h | 4 ++ include/linux/perf_counter.h | 1 + kernel/hw_breakpoint.c | 79 +++++++++++++++++++++++++++++++++++ kernel/perf_counter.c | 35 +++++++++++++++ 4 files changed, 119 insertions(+), 0 deletions(-)
include/asm-generic/hw_breakpoint.h +4 -0
@@ -105,6 +105,10 @@ register_kernel_hw_breakpoint(struct hw_breakpoint *bp, unsigned long addr, int len, enum breakpoint_type type);extern void unregister_kernel_hw_breakpoint (struct hw_breakpoint *bp); +#ifdef CONFIG_PERF_COUNTERS +extern void bp_perf_triggered (struct hw_breakpoint *bp,struct pt_regs *regs); +#endif +extern unsigned int hbp_kernel_pos;#endif /* __KERNEL__ */
include/linux/perf_counter.h +1 -0
@@ -31,6 +31,7 @@ enum perf_type_id {PERF_TYPE_TRACEPOINT =2 ,PERF_TYPE_HW_CACHE =3 ,PERF_TYPE_RAW =4 , +PERF_TYPE_BREAKPOINT =5 , PERF_TYPE_MAX, /* non-ABI */ };
kernel/hw_breakpoint.c +79 -0
85 lines not on this page · raw message
kernel/perf_counter.c +35 -0
@@ -29,6 +29,7 @@#include <linux/perf_counter.h> #include <asm/irq_regs.h> +#include <asm/hw_breakpoint.h> /* * Each CPU has a list of per CPU counters:
@@ -3718,6 +3719,36 @@ static const struct pmu *tp_perf_counter_init(struct perf_counter *counter) }#endif +extern int hw_breakpoint_perf_init (unsigned long addr); +extern void hw_breakpoint_perf_exit (unsigned long addr); + +void bp_perf_triggered (struct hw_breakpoint *bp,struct pt_regs *regs) +{ +unsigned long target =hw_breakpoint_addr (bp); + +struct perf_sample_data data = { + .regs = regs, + .addr =instruction_pointer (regs), + }; + +do_perf_swcounter_event (PERF_TYPE_BREAKPOINT , target,1 ,0 , &data); +} + +static void bp_perf_counter_destroy (struct perf_counter *counter) +{ +hw_breakpoint_perf_exit (counter->attr.config); +} + +static const struct pmu *bp_perf_counter_init (struct perf_counter *counter) +{ +if (hw_breakpoint_perf_init ((unsigned long )counter->attr.config)) +return NULL ; + + counter->destroy = bp_perf_counter_destroy; + +return &perf_ops_generic; +} +atomic_t perf_swcounter_enabled[PERF_COUNT_SW_MAX ];static void sw_perf_counter_destroy (struct perf_counter *counter)
@@ -3857,6 +3888,10 @@ perf_counter_alloc(struct perf_counter_attr *attr, pmu =tp_perf_counter_init (counter);break ; +case PERF_TYPE_BREAKPOINT : + pmu =bp_perf_counter_init (counter); +break ; +default :break ; }
Signature · 1 line
1.6.2.3
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 7 lines · Frederic Weisbecker
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
kernel/perf_counter.c +1
-0+void bp_perf_triggered (struct hw_breakpoint *bp,struct pt_regs *regs)
kernel/perf_counter.c static const struct pmu *tp_perf_counter_init(struct perf_counter *counter) · line 3735
Quoted 10 lines
kernel/perf_counter.c +10
-0+{ +unsigned long target =hw_breakpoint_addr (bp); + +struct perf_sample_data data = { + .regs = regs, + .addr =instruction_pointer (regs), + }; + +do_perf_swcounter_event (PERF_TYPE_BREAKPOINT , target,1 ,0 , &data); +}
.addr would be an associated data address, like for pagefaults and cache-misses have, its not the RIP of the faulting ins.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
2009/7/20, Peter Zijlstra <[email protected]>:
Quoted 7 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 14 lines
kernel/perf_counter.c static const struct pmu *tp_perf_counter_init(struct perf_counter *counter) · line 3735
Quoted 11 lines
kernel/perf_counter.c +11
-0+void bp_perf_triggered (struct hw_breakpoint *bp,struct pt_regs *regs) +{ +unsigned long target =hw_breakpoint_addr (bp); + +struct perf_sample_data data = { + .regs = regs, + .addr =instruction_pointer (regs), + }; + +do_perf_swcounter_event (PERF_TYPE_BREAKPOINT , target,1 ,0 , &data); +}.addr would be an associated data address, like for pagefaults and cache-misses have, its not the RIP of the faulting ins.
Ah ok, then I'll set the actual data memory address that is breakpointed.
Thanks.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Frederic WeisbeckerQuoted 4 lines · Frederic Weisbecker
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
This seems to be IP based breakpoints. Are there plans for data based breakpoints as well? In that case we might want to think about the namespace issue, we cannot both call them breakpoint/bp etc.. ;-)
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
2009/7/20, Peter Zijlstra <[email protected]>:
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
This seems to be IP based breakpoints. Are there plans for data based breakpoints as well? In that case we might want to think about the namespace issue, we cannot both call them breakpoint/bp etc.. ;-)
Nop, by default these breakpoints trigger on READ/WRITE accesses, it's meant for data.
The example in the changelog profiles the bkl accesses, not by tracing lock_kernel() or so...but by tracing the kernel_flag spinlock itself.
So it's the opposite, we may start thinking about naming issues against possible future plans
for IP breakpoint :-)
But actually for the latter case, I would suggest Kprobe...
Damn, I forgot Masami in the Cc list...
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 16 lines · Frédéric Weisbecker
2009/7/20, Peter Zijlstra <[email protected]>:
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
This seems to be IP based breakpoints. Are there plans for data based breakpoints as well? In that case we might want to think about the namespace issue, we cannot both call them breakpoint/bp etc.. ;-)
Nop, by default these breakpoints trigger on READ/WRITE accesses, it's meant for data.
The example in the changelog profiles the bkl accesses, not by tracinglock_kernel()or so...but by tracing thekernel_flagspinlock itself.
Quoted 5 lines
So it's the opposite, we may start thinking about naming issues against possible future plans
for IP breakpoint :-)But actually for the latter case, I would suggest Kprobe...
Yeah, after kprobes-based event tracer is supported on ftrace, it is easy to set up IP breakpoints in the kernel. (maybe, I need to enhance it for perfcounter.)
Thank you,
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: [email protected]
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Frederic WeisbeckerQuoted 1 line · Frederic Weisbecker
+
do_perf_swcounter_event(PERF_TYPE_BREAKPOINT, target, 1, 0, &data);
These breakpoints run from IRQ context?
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
2009/7/20, Peter Zijlstra <[email protected]>:
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines
+ do_perf_swcounter_event(PERF_TYPE_BREAKPOINT, target, 1, 0,
&data);These breakpoints run from IRQ context?
Hmm, I guess but I'm not sure exactly :-/
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Frederic WeisbeckerQuoted 4 lines · Frederic Weisbecker
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Do the breakpoints work on virtual or physical addresses?
Also, do we have an 64bit architecture with split address-spaces?
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 1 line · Peter Zijlstra
Quoted 1 line
Is there a limit to these hardware breakpoints? If so, the software
Certainly there's a limit. They require hardware resources, after all.
Quoted 5 lines
Quoted 3 lines
counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Do the breakpoints work on virtual or physical addresses?
On x86, hardware breakpoints take virtual addresses. I don't know about other architectures.
Alan Stern
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Peter Zijlstra2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
+static const struct pmu *bp_perf_counter_init(struct perf_counter *counter) +{
+ if (hw_breakpoint_perf_init((unsigned long)counter->attr.config))
+ return NULL;
+
IIRC, hw_breakpoint_perf_init() calls register_kernel_breakpoint() which in turn returns -ENOSPC if we haven't any breakpoint room left.
It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 22 lines · Frédéric Weisbecker
2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
kernel/perf_counter.c +5
-0+static const struct pmu *bp_perf_counter_init (struct perf_counter *counter) +{ +if (hw_breakpoint_perf_init ((unsigned long )counter->attr.config)) +return NULL ; +
Quoted 5 lines
IIRC,
hw_breakpoint_perf_init()callsregister_kernel_breakpoint()which in turn returns-ENOSPCif we haven't any breakpoint room left.It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Ah, that's not the correct way of doing that. Suppose that you would register 4 breakpoint counter to one task, that would leave you unable to register a breakpoint counter on another task. Even though these breakpoints would never be scheduled simultaneously.
Also, regular perf counters would multiplex counters when over-committed on a hardware resource, allowing you to create more such breakpoints than you have actual hardware slots.
The way to do this is to create a breakpoint pmu which would simply fail the pmu->enable() method if there are insufficient hardware resources available.
Also, your init routine, the above hw_breakpoint_perf_init(), will have to verify that when the counter is part of a group, this and all other hw breakpoint counters in that group can, now, but also in the future, be scheduled simultaneously.
This means that there should be some arbitration towards other in-kernel hw breakpoint users, because if you allow all 4 hw breakpoints in a group and then let another hw breakpoint users have one, you can never schedule that group again.
[ which raises a fun point, Paulus do we handle groups having multiple
'hardware' pmu's in? ]
Now, for the actual counter implementation you can probably re-use the swcounter code, but you also need a pmu implementation.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frédéric Weisbecker
2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
kernel/perf_counter.c +5
-0+static const struct pmu *bp_perf_counter_init (struct perf_counter *counter) +{ +if (hw_breakpoint_perf_init ((unsigned long )counter->attr.config)) +return NULL ; +IIRC,
hw_breakpoint_perf_init()callsregister_kernel_breakpoint()which in turn returns-ENOSPCif we haven't any breakpoint room left.
Quoted 7 lines
Quoted 2 lines
It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Ah, that's not the correct way of doing that. Suppose that you would register 4 breakpoint counter to one task, that would leave you unable to register a breakpoint counter on another task. Even though these breakpoints would never be scheduled simultaneously.
Ah, but the breakpoint API deals with that.
We have two types of breakpoints: the kernel bp and the user bp. The kernel breakpoints are global points that don't deal with task scheduling, virtual registers, etc...
But the user breakpoints (eg: used with ptrace) are dealt with virtual debug registers in a way similar to perfcounter: each time we switch the current task on a cpu, the hardware register states are stored in the thread, and we load the virtual values into the hardware for the next thread.
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Quoted 3 lines
Also, regular perf counters would multiplex counters when over-committed on a hardware resource, allowing you to create more such breakpoints than you have actual hardware slots.
In the task level I talked above?
Quoted 3 lines
The way to do this is to create a breakpoint pmu which would simply fail the
pmu->enable()method if there are insufficient hardware resources available.
Now I wonder if the code that manages hardware debug breakpoint task switching and the code from perfcounter could be factorized in one common thing.
Quoted 4 lines
Also, your init routine, the above
hw_breakpoint_perf_init(), will have to verify that when the counter is part of a group, this and all other hw breakpoint counters in that group can, now, but also in the future, be scheduled simultaneously.
This is already dealt from the hardware breakpoint API.
We use only one breakpoint register for the user breakpoints, and the rest for kernel breakpoints. Also if no user breakpoint is registered, every registers can be used for kernek breakpoints.
Quoted 4 lines
This means that there should be some arbitration towards other in-kernel hw breakpoint users, because if you allow all 4 hw breakpoints in a group and then let another hw breakpoint users have one, you can never schedule that group again.
That's also why I think it's better to keep this virtual register management from inside the breakpoint API, so that it can deal with perfcounter, ptrace, etc... at the same.
What do you think?
Quoted 7 lines
[ which raises a fun point, Paulus do we handle groups having multiple
'hardware' pmu's in? ]Now, for the actual counter implementation you can probably re-use the swcounter code, but you also need a pmu implementation.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 42 lines · Frederic Weisbecker
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frédéric Weisbecker
2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
kernel/perf_counter.c +5
-0+static const struct pmu *bp_perf_counter_init (struct perf_counter *counter) +{ +if (hw_breakpoint_perf_init ((unsigned long )counter->attr.config)) +return NULL ; +IIRC,
hw_breakpoint_perf_init()callsregister_kernel_breakpoint()which in turn returns-ENOSPCif we haven't any breakpoint room left.Quoted 7 lines
Quoted 2 lines
It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Ah, that's not the correct way of doing that. Suppose that you would register 4 breakpoint counter to one task, that would leave you unable to register a breakpoint counter on another task. Even though these breakpoints would never be scheduled simultaneously.
Ah, but the breakpoint API deals with that.
We have two types of breakpoints: the kernel bp and the user bp. The kernel breakpoints are global points that don't deal with task scheduling, virtual registers, etc...
Quoted 5 lines
But the user breakpoints (eg: used with ptrace) are dealt with virtual debug registers in a way similar to perfcounter: each time we switch the current task on a cpu, the hardware register states are stored in the thread, and we load the virtual values into the hardware for the next thread.
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
Quoted 5 lines
Quoted 3 lines
Also, regular perf counters would multiplex counters when over-committed on a hardware resource, allowing you to create more such breakpoints than you have actual hardware slots.
In the task level I talked above?
For either cpu or task level.
Quoted 6 lines
Quoted 3 lines
The way to do this is to create a breakpoint pmu which would simply fail the
pmu->enable()method if there are insufficient hardware resources available.Now I wonder if the code that manages hardware debug breakpoint task switching and the code from perfcounter could be factorized in one common thing.
Dunno, its really not that hard to RR a list of counters/breakpoint.
Quoted 9 lines
Quoted 4 lines
Also, your init routine, the above
hw_breakpoint_perf_init(), will have to verify that when the counter is part of a group, this and all other hw breakpoint counters in that group can, now, but also in the future, be scheduled simultaneously.This is already dealt from the hardware breakpoint API.
We use only one breakpoint register for the user breakpoints, and the rest for kernel breakpoints. Also if no user breakpoint is registered, every registers can be used for kernek breakpoints.
This means that you can only ever allow 3 breakpoints into any one group and have to ensure that no other user can come in when they're not in active use -- the group is scheduled out.
That is, you have to reserve the max number of breakpoint in a group for exclusive use by perf counters.
Also, this 1 for userspace seems restrictive. I'd want to have all 4 from GDB if I'd knew my hardware was capable and I'd needed that many.
Quoted 10 lines
Quoted 4 lines
This means that there should be some arbitration towards other in-kernel hw breakpoint users, because if you allow all 4 hw breakpoints in a group and then let another hw breakpoint users have one, you can never schedule that group again.
That's also why I think it's better to keep this virtual register management from inside the breakpoint API, so that it can deal with perfcounter, ptrace, etc... at the same.
What do you think?
I think not. I think the breakpoint API should not do task state, or at least have an interface without this.
Having two multiplexing layers on top of one another is inefficient and error prone.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 49 lines · Peter Zijlstra
Quoted 42 lines · Frederic Weisbecker
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frédéric Weisbecker
2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
kernel/perf_counter.c +5
-0+static const struct pmu *bp_perf_counter_init (struct perf_counter *counter) +{ +if (hw_breakpoint_perf_init ((unsigned long )counter->attr.config)) +return NULL ; +IIRC,
hw_breakpoint_perf_init()callsregister_kernel_breakpoint()which in turn returns-ENOSPCif we haven't any breakpoint room left.Quoted 7 lines
Quoted 2 lines
It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Ah, that's not the correct way of doing that. Suppose that you would register 4 breakpoint counter to one task, that would leave you unable to register a breakpoint counter on another task. Even though these breakpoints would never be scheduled simultaneously.
Ah, but the breakpoint API deals with that.
We have two types of breakpoints: the kernel bp and the user bp. The kernel breakpoints are global points that don't deal with task scheduling, virtual registers, etc...Quoted 5 lines
But the user breakpoints (eg: used with ptrace) are dealt with virtual debug registers in a way similar to perfcounter: each time we switch the current task on a cpu, the hardware register states are stored in the thread, and we load the virtual values into the hardware for the next thread.
Quoted 4 lines
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
Ah ok.
Quoted 2 lines
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
That would forbid some kinds of profiling (explanations below).
Quoted 5 lines
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
I would actually propose to have a separate layer that manages the hardware registers <-> per thread virtual registers handling for things like breakpoint api and perfcounter.
I know a simple RR of registers is not that hard to write, but at least that can allow simultaneous use of perfcounter and other users of breakpoint API without having two different versions of register management.
Quoted 27 lines
Quoted 5 lines
Quoted 3 lines
Also, regular perf counters would multiplex counters when over-committed on a hardware resource, allowing you to create more such breakpoints than you have actual hardware slots.
In the task level I talked above?
For either cpu or task level.
Quoted 6 lines
Quoted 3 lines
The way to do this is to create a breakpoint pmu which would simply fail the
pmu->enable()method if there are insufficient hardware resources available.Now I wonder if the code that manages hardware debug breakpoint task switching and the code from perfcounter could be factorized in one common thing.
Dunno, its really not that hard to RR a list of counters/breakpoint.
Quoted 9 lines
Quoted 4 lines
Also, your init routine, the above
hw_breakpoint_perf_init(), will have to verify that when the counter is part of a group, this and all other hw breakpoint counters in that group can, now, but also in the future, be scheduled simultaneously.This is already dealt from the hardware breakpoint API.
We use only one breakpoint register for the user breakpoints, and the rest for kernel breakpoints. Also if no user breakpoint is registered, every registers can be used for kernek breakpoints.
Quoted 5 lines
This means that you can only ever allow 3 breakpoints into any one group and have to ensure that no other user can come in when they're not in active use -- the group is scheduled out.
That is, you have to reserve the max number of breakpoint in a group for exclusive use by perf counters.
Hmm, if we reserve all breakpoints registers for perfcounter exclusive use when it runs, that excludes any profiling of ptrace while doing a POKE_USR or gdb while using breakpoints.
That's why I think it would be better to make use of the hardware breakpoints from perfcounter using the bp API. Doing so allows concurrent users of bp while perf is using them. Then we have no restriction concerning the profiling of code that uses breakpoints.
Using a seperate hardware register <-> virtual register management layer would then solve the problem of two different ad hoc implementations to maintain and which impacts profiling performances.
Quoted 2 lines
Also, this 1 for userspace seems restrictive. I'd want to have all 4 from GDB if I'd knew my hardware was capable and I'd needed that many.
Actually I've made a mistake, you can use several user breakpoints, as many as the number of hardware breakpoints, minus the number of kernel bp currently set.
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
This means that there should be some arbitration towards other in-kernel hw breakpoint users, because if you allow all 4 hw breakpoints in a group and then let another hw breakpoint users have one, you can never schedule that group again.
That's also why I think it's better to keep this virtual register management from inside the breakpoint API, so that it can deal with perfcounter, ptrace, etc... at the same.
Quoted 7 lines
Quoted 1 line
What do you think?
I think not. I think the breakpoint API should not do task state, or at least have an interface without this.
Having two multiplexing layers on top of one another is inefficient and error prone.
And what do you think of the above idea?
Thanks.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 152 lines · * Frederic Weisbecker ([email protected]) wrote:
Quoted 49 lines · Peter Zijlstra
Quoted 42 lines · Frederic Weisbecker
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frédéric Weisbecker
2009/7/23 Peter Zijlstra <[email protected]>:
Quoted 6 lines · Frederic Weisbecker
Quoted 4 lines
This adds the support for kernel hardware breakpoints in perfcounter. It is added as a new type of software counter and can be defined by using the counter number 5 and by passsing the address of the breakpoint to set through the config attribute.
Quoted 6 lines
Is there a limit to these hardware breakpoints? If so, the software counter model is not sufficient, since we assume we can always schedule all software counters. However if you were to add more counters than you have hardware breakpoints you're hosed.
Hmm, indeed. But this patch handles this case:
(quoted diff) +5
-0+static const struct pmu *bp_perf_counter_init(struct perf_counter *counter) +{ + if (hw_breakpoint_perf_init((unsigned long)counter->attr.config)) + return NULL; +IIRC,
hw_breakpoint_perf_init()callsregister_kernel_breakpoint()which in turn returns-ENOSPCif we haven't any breakpoint room left.Quoted 7 lines
Quoted 2 lines
It seems we can only set 4 breakpoints simultaneously in x86, or something close to that.
Ah, that's not the correct way of doing that. Suppose that you would register 4 breakpoint counter to one task, that would leave you unable to register a breakpoint counter on another task. Even though these breakpoints would never be scheduled simultaneously.
Ah, but the breakpoint API deals with that.
We have two types of breakpoints: the kernel bp and the user bp. The kernel breakpoints are global points that don't deal with task scheduling, virtual registers, etc...Quoted 5 lines
But the user breakpoints (eg: used with ptrace) are dealt with virtual debug registers in a way similar to perfcounter: each time we switch the current task on a cpu, the hardware register states are stored in the thread, and we load the virtual values into the hardware for the next thread.
Quoted 4 lines
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
Ah ok.
Quoted 2 lines
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
That would forbid some kinds of profiling (explanations below).
Quoted 5 lines
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
I would actually propose to have a separate layer that manages the hardware registers <-> per thread virtual registers handling for things like breakpoint api and perfcounter.
I know a simple RR of registers is not that hard to write, but at least that can allow simultaneous use of perfcounter and other users of breakpoint API without having two different versions of register management.
Quoted 27 lines
Quoted 5 lines
Quoted 3 lines
Also, regular perf counters would multiplex counters when over-committed on a hardware resource, allowing you to create more such breakpoints than you have actual hardware slots.
In the task level I talked above?
For either cpu or task level.
Quoted 6 lines
Quoted 3 lines
The way to do this is to create a breakpoint pmu which would simply fail the
pmu->enable()method if there are insufficient hardware resources available.Now I wonder if the code that manages hardware debug breakpoint task switching and the code from perfcounter could be factorized in one common thing.
Dunno, its really not that hard to RR a list of counters/breakpoint.
Quoted 9 lines
Quoted 4 lines
Also, your init routine, the above
hw_breakpoint_perf_init(), will have to verify that when the counter is part of a group, this and all other hw breakpoint counters in that group can, now, but also in the future, be scheduled simultaneously.This is already dealt from the hardware breakpoint API.
We use only one breakpoint register for the user breakpoints, and the rest for kernel breakpoints. Also if no user breakpoint is registered, every registers can be used for kernek breakpoints.Quoted 5 lines
This means that you can only ever allow 3 breakpoints into any one group and have to ensure that no other user can come in when they're not in active use -- the group is scheduled out.
That is, you have to reserve the max number of breakpoint in a group for exclusive use by perf counters.Hmm, if we reserve all breakpoints registers for perfcounter exclusive use when it runs, that excludes any profiling of ptrace while doing a
POKE_USRor gdb while using breakpoints.That's why I think it would be better to make use of the hardware breakpoints from perfcounter using the bp API. Doing so allows concurrent users of bp while perf is using them. Then we have no restriction concerning the profiling of code that uses breakpoints.
Using a seperate hardware register <-> virtual register management layer would then solve the problem of two different ad hoc implementations to maintain and which impacts profiling performances.
Quoted 2 lines
Also, this 1 for userspace seems restrictive. I'd want to have all 4 from GDB if I'd knew my hardware was capable and I'd needed that many.
Actually I've made a mistake, you can use several user breakpoints, as many as the number of hardware breakpoints, minus the number of kernel bp currently set.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
This means that there should be some arbitration towards other in-kernel hw breakpoint users, because if you allow all 4 hw breakpoints in a group and then let another hw breakpoint users have one, you can never schedule that group again.
That's also why I think it's better to keep this virtual register management from inside the breakpoint API, so that it can deal with perfcounter, ptrace, etc... at the same.
What do you think?
Quoted 9 lines
Quoted 5 lines
I think not. I think the breakpoint API should not do task state, or at least have an interface without this.
Having two multiplexing layers on top of one another is inefficient and error prone.
And what do you think of the above idea?
I agree with your idea to split the hardware counter management from virtual per-process counters. IMO, this limited resource needs to be managed centrally at one location and because system-wide level performance counters do not need to flip the performance counters depending on the current task. We can easily think of an embedded system where providing system-wide performance counters would be important (for tracing for instance), but which would compile-out the per-task performance counters to save space.
Note that you will have to deal with some policy here, because you can have performance counter reservation asked from both the kernel (for either kernel and per-task watchpoints) and from userspace (for per-task watchpoints).
You have to think of scenarios like:
A userspace application asks for all hardware watchpoints, and then the kernel needs 1-2 of them. The application should probably be "kicked out" of 2 of those watchpoints _after_ they have been set up, otherwise this would allow a resource exhaustion to be performed by the application at the detriment of the kernel.
Having the ability for a performance counter read to fail when a virtual performance counter has been kicked-out by a more important user would allow the kernel to take control of the performance counters without letting userspace in its way.
Mathieu
Quoted 2 lines
Thanks.
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 5 lines · Mathieu Desnoyers
I agree with your idea to split the hardware counter management from virtual per-process counters. IMO, this limited resource needs to be managed centrally at one location and because system-wide level performance counters do not need to flip the performance counters depending on the current task.
System wide counters never care about the task state.
Its task counters we sometimes don't re-program counters for on context switch when both tasks have the same configuration, saving greatly on context switch costs.
Quoted 4 lines
We can easily think of an embedded system
where providing system-wide performance counters would be important (for tracing for instance), but which would compile-out the per-task performance counters to save space.
That doesn't make sense, the per task/global parts of perf counters are tightly interwoven and don't differ much.
Quoted 4 lines
Note that you will have to deal with some policy here, because you can have performance counter reservation asked from both the kernel (for either kernel and per-task watchpoints) and from userspace (for per-task watchpoints).
cpu counters can be both kernel and user
task counter can be both kernel and user.
Your above use of 'and' doesn't make sense.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Frederic WeisbeckerQuoted 23 lines · Frederic Weisbecker
Quoted 4 lines
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
Ah ok.
Quoted 2 lines
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
That would forbid some kinds of profiling (explanations below).
Quoted 5 lines
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
Quoted 8 lines
I would actually propose to have a separate layer that manages the hardware registers <-> per thread virtual registers handling for things like breakpoint api and perfcounter.
I know a simple RR of registers is not that hard to write, but at least that can allow simultaneous use of perfcounter and other users of breakpoint API without having two different versions of register management.
I simply cannot see how you would be able to multiplex userspace/debug breakpoints. I'd utterly hate it if I'd missed a breakpoint simply because someone else also wanted to make use of it.
I'd declare the system broken and useless.
Counters OTOH can be multiplexed because of their statistical nature, you can simply scale them back up based on their time share.
Therefore you'll have to deal with hard reservations anyway.
Also, you don't need to a-priory reserve all breakpoints, you'll simply need as many as the largest group (wrt breakpoints) has.
Quoted 12 lines
Quoted 10 lines
Quoted 4 lines
This is already dealt from the hardware breakpoint API.
We use only one breakpoint register for the user breakpoints, and the rest for kernel breakpoints. Also if no user breakpoint is registered, every registers can be used for kernek breakpoints.This means that you can only ever allow 3 breakpoints into any one group and have to ensure that no other user can come in when they're not in active use -- the group is scheduled out.
That is, you have to reserve the max number of breakpoint in a group for exclusive use by perf counters.
Quoted 8 lines
Hmm, if we reserve all breakpoints registers for perfcounter exclusive use when it runs, that excludes any profiling of ptrace while doing a
POKE_USRor gdb while using breakpoints.That's why I think it would be better to make use of the hardware breakpoints from perfcounter using the bp API. Doing so allows concurrent users of bp while perf is using them. Then we have no restriction concerning the profiling of code that uses breakpoints.
Like said above, you cannot do this.
Quoted 5 lines
Using a seperate hardware register <-> virtual register management layer would then solve the problem of two different ad hoc implementations to maintain and which impacts profiling performances.
Quoted 7 lines
Quoted 2 lines
Also, this 1 for userspace seems restrictive. I'd want to have all 4 from GDB if I'd knew my hardware was capable and I'd needed that many.
Actually I've made a mistake, you can use several user breakpoints, as many as the number of hardware breakpoints, minus the number of kernel bp currently set.
Better :-)
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
Quoted 4 lines
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
Ah ok.
Quoted 2 lines
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
That would forbid some kinds of profiling (explanations below).
Quoted 5 lines
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
Quoted 8 lines
I would actually propose to have a separate layer that manages the hardware registers <-> per thread virtual registers handling for things like breakpoint api and perfcounter.
I know a simple RR of registers is not that hard to write, but at least that can allow simultaneous use of perfcounter and other users of breakpoint API without having two different versions of register management.
I simply cannot see how you would be able to multiplex userspace/debug breakpoints. I'd utterly hate it if I'd missed a breakpoint simply because someone else also wanted to make use of it.
Quoted 7 lines
I'd declare the system broken and useless.
Counters OTOH can be multiplexed because of their statistical nature, you can simply scale them back up based on their time share.
Therefore you'll have to deal with hard reservations anyway.
I don't claim to have understood the requirements of perf-counters fully, but I sense the persistence of a few doubts about the register allocation mechanism of the hw-breakpoints API...thought it might be worthwhile to briefly explain.
A few notes about the same:
Kernel-space breakpoints: They are system-wide i.e. one kernel-space breakpoint request consumes one debug register on 'all' CPUs of the system.
User-space breakpoints: Requests are stored in per-thread data structures. Written onto the debug register only when scheduled-into the CPU, and are removed when another task using the debug registers is scheduled onto the CPU.
Debug register allocation mechanism: register request succeeds only when the availability of a debug register is guaranteed (for both user and kernel space). Allocation on a first-come-first-serve basis, no priorities for register requests and hence no pre-emption of requests.
In short the available debug registers are divided into:
# of debug registers = # of kernel-space requests + MAX(# of all user-space requests) + freely available debug registers.
Thus on an x86 system with 4 debug registers, if there exists 1 kernel-space request it consumes one debug register on all CPUs in the system. The remaining 3 debug registers can be consumed by user-space i.e. upto 3 user-space requests per-thread.
There is no restriction on how many debug registers can be consumed by the kernel- or user-space requests and it is entirely dependent on the number of free registers available for use then.
Thanks,
K.Prasad
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 24 lines · K.Prasad
I don't claim to have understood the requirements of perf-counters fully, but I sense the persistence of a few doubts about the register allocation mechanism of the hw-breakpoints API...thought it might be worthwhile to briefly explain.
A few notes about the same:
Kernel-space breakpoints: They are system-wide i.e. one kernel-space breakpoint request consumes one debug register on 'all' CPUs of the system.
User-space breakpoints: Requests are stored in per-thread data structures. Written onto the debug register only when scheduled-into the CPU, and are removed when another task using the debug registers is scheduled onto the CPU.
Debug register allocation mechanism: register request succeeds only when the availability of a debug register is guaranteed (for both user and kernel space). Allocation on a first-come-first-serve basis, no priorities for register requests and hence no pre-emption of requests.In short the available debug registers are divided into:
# of debug registers = # of kernel-space requests + MAX(# of all user-space requests) + freely available debug registers.
Quoted 8 lines
Thus on an x86 system with 4 debug registers, if there exists 1 kernel-space request it consumes one debug register on all CPUs in the system. The remaining 3 debug registers can be consumed by user-space i.e. upto 3 user-space requests per-thread.
There is no restriction on how many debug registers can be consumed by the kernel- or user-space requests and it is entirely dependent on the number of free registers available for use then.
Right, this is an utter miss-match for what perf counters would want.
Firstly, you seem to have this weird split of kernel/userspace breakpoints. Perf counters looks at things in a per-cpu fashion, so the all-cpus kernel breakpoint stuff is useless. Also, from perf counters' POV its perfectly reasonable to have a per-task kernel breakpoint.
Secondly, perf counters wants to schedule the per task breakpoints because we can optimize the context switch, saving lots of these MSR writes under some common scenarios.
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Like I said, please use the raw per-cpu breakpoint interface for perf counters and connect that with the minimally required reservation you need to make your other thing work.
You simply cannot put perf-counter breakpoints on top of whatever virt layer you created going by what you say it is.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 35 lines · Peter Zijlstra
Quoted 24 lines · K.Prasad
I don't claim to have understood the requirements of perf-counters fully, but I sense the persistence of a few doubts about the register allocation mechanism of the hw-breakpoints API...thought it might be worthwhile to briefly explain.
A few notes about the same:
Kernel-space breakpoints: They are system-wide i.e. one kernel-space breakpoint request consumes one debug register on 'all' CPUs of the system.
User-space breakpoints: Requests are stored in per-thread data structures. Written onto the debug register only when scheduled-into the CPU, and are removed when another task using the debug registers is scheduled onto the CPU.
Debug register allocation mechanism: register request succeeds only when the availability of a debug register is guaranteed (for both user and kernel space). Allocation on a first-come-first-serve basis, no priorities for register requests and hence no pre-emption of requests.In short the available debug registers are divided into:
# of debug registers = # of kernel-space requests + MAX(# of all user-space requests) + freely available debug registers.Quoted 8 lines
Thus on an x86 system with 4 debug registers, if there exists 1 kernel-space request it consumes one debug register on all CPUs in the system. The remaining 3 debug registers can be consumed by user-space i.e. upto 3 user-space requests per-thread.
There is no restriction on how many debug registers can be consumed by the kernel- or user-space requests and it is entirely dependent on the number of free registers available for use then.
Quoted 6 lines
Right, this is an utter miss-match for what perf counters would want.
Firstly, you seem to have this weird split of kernel/userspace breakpoints. Perf counters looks at things in a per-cpu fashion, so the all-cpus kernel breakpoint stuff is useless. Also, from perf counters' POV its perfectly reasonable to have a per-task kernel breakpoint.
Hmm, indeed. The breakpoint api lacks these services for perf counter, although targetting specific cpus seems a (somewhat) trivial add-on to me on the bp API.
Also I forgot the kernel-only profiling things.
Quoted 3 lines
Secondly, perf counters wants to schedule the per task breakpoints because we can optimize the context switch, saving lots of these MSR writes under some common scenarios.
Ok with that.
But if you have an efficient pmu scheduling, it may be still interesting to make it a seperate core entity. The bp API may find some profit in it.
The breakpoint registers also require specific treatments with several registers magic, all organized inside the arch API which also cares about cpu hotplug and kexec things.
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Again, I'm stuck in what you mean by multiplexing here :-)
Quoted 6 lines
Like I said, please use the raw per-cpu breakpoint interface for perf counters and connect that with the minimally required reservation you need to make your other thing work.
You simply cannot put perf-counter breakpoints on top of whatever virt layer you created going by what you say it is.
Well, indeed it seems perfcounter handles the counters to cover very specific needs, uncovered by the debug API we have.
I guess we can create a struct pmu for the breakpoints and plug it with some low level add-on in the breakpoint API.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Again, I'm stuck in what you mean by multiplexing here :-)
If you'd create say 16 breakpoint counters, we'd RR them over the 4 available hardware breakpoints (or less when others are taken by someone else).
Since its all statistics anyway, we can simply scale the event counts up by the time-share they received.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 5 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Quoted 9 lines
Quoted 1 line
Again, I'm stuck in what you mean by multiplexing here :-)
If you'd create say 16 breakpoint counters, we'd RR them over the 4 available hardware breakpoints (or less when others are taken by someone else).
Since its all statistics anyway, we can simply scale the event counts up by the time-share they received.
Although hw breakpoint and performance counters could be multiplexed to be used in RR for profiling, I fear using these for tracing does not necessarily fit well with these statistical artefacts.
Therefore I'd recommend leaving such bp RR as a feature separate from a low-level breakpoint API, because it seem only useful to profilers.
I think we all have our very precise use-case in mind, and this is what makes discussion so difficult. So, for my point of view (tracing), I'd like to be able to set performance counters and breakpoints both monitoring userspace and kernel. The userspace breakpoints would be exchanged when a different process is scheduled.
I see 2 complementary ways to collect data into the trace:
- triggering an interrupt each X hardware events, trace this interrupt.
- reading the performance counter value at tracepoints identifying execution context change (e.g. system call entry/exit, interrupt entry/exit...)
As you see, none of these 2 ways to gather trace data allow statistical RR. Therefore, having this feature in the low-level API does not seem to make sense for tracing.
So if other use-cases could be explained, maybe we can factor out the common interfaces.
Thanks,
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 3 lines · Mathieu Desnoyers
As you see, none of these 2 ways to gather trace data allow statistical RR. Therefore, having this feature in the low-level API does not seem to make sense for tracing.
I'm not asking for it to be available in the low level API. Quite contrary, I'm saying that the current API is too high level and pretty useless for perf counters.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Peter ZijlstraQuoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Quoted 9 lines
Quoted 1 line
Again, I'm stuck in what you mean by multiplexing here :-)
If you'd create say 16 breakpoint counters, we'd RR them over the 4 available hardware breakpoints (or less when others are taken by someone else).
Since its all statistics anyway, we can simply scale the event counts up by the time-share they received.
Aah, ok I understand now.
But I fear it may kill the accuracy of the breakpoints statistics. It's fine for a theoretical linear rate of breakpoint events.
But what happens if we are profiling something much more unstable with a rain of hits in a small window of memory between large timeframes? If we have a breakpoint inside this window of memory and this breakpoint is not plugged, waiting for its turn in the RR, we loose this rain of events.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Quoted 9 lines
Quoted 1 line
Again, I'm stuck in what you mean by multiplexing here :-)
If you'd create say 16 breakpoint counters, we'd RR them over the 4 available hardware breakpoints (or less when others are taken by someone else).
Since its all statistics anyway, we can simply scale the event counts up by the time-share they received.
Quoted 7 lines
Aah, ok I understand now.
But I fear it may kill the accuracy of the breakpoints statistics. It's fine for a theoretical linear rate of breakpoint events.
But what happens if we are profiling something much more unstable with a rain of hits in a small window of memory between large timeframes? If we have a breakpoint inside this window of memory and this breakpoint is not plugged, waiting for its turn in the RR, we loose this rain of events.
You don't have to overload the breakpoint set, but we will if you create more than there are hardware resources available.
If the RR period is independent of the application and significantly shorter than the sample duration, statistics will be usable.
Sure, they'll not be perfect, but we can improve our confidence interval by running longer. At some point it really doesn't matter anymore.
Take your example, of small bursts of events, due to the first assumption - the RR period being independent of the application - we're bound to see those busts in a proportional number of sample windows.
If the full sample duration is in the same order of the RR period, then yes, you'll have a fair chance of missing the burst -- don't do that then :-)
If you know you'll be running very very short, use attr.pinned, to request a counter that'll not be multiplexed.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 38 lines · Peter Zijlstra
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
Quoted 9 lines
Quoted 1 line
Again, I'm stuck in what you mean by multiplexing here :-)
If you'd create say 16 breakpoint counters, we'd RR them over the 4 available hardware breakpoints (or less when others are taken by someone else).
Since its all statistics anyway, we can simply scale the event counts up by the time-share they received.
Quoted 7 lines
Aah, ok I understand now.
But I fear it may kill the accuracy of the breakpoints statistics. It's fine for a theoretical linear rate of breakpoint events.
But what happens if we are profiling something much more unstable with a rain of hits in a small window of memory between large timeframes? If we have a breakpoint inside this window of memory and this breakpoint is not plugged, waiting for its turn in the RR, we loose this rain of events.You don't have to overload the breakpoint set, but we will if you create more than there are hardware resources available.
If the RR period is independent of the application and significantly shorter than the sample duration, statistics will be usable.
Sure, they'll not be perfect, but we can improve our confidence interval by running longer. At some point it really doesn't matter anymore.
Take your example, of small bursts of events, due to the first assumption - the RR period being independent of the application - we're bound to see those busts in a proportional number of sample windows.
Quoted 6 lines
If the full sample duration is in the same order of the RR period, then yes, you'll have a fair chance of missing the burst -- don't do that then :-)
If you know you'll be running very very short, use attr.pinned, to request a counter that'll not be multiplexed.
Ok, I'm convinced :-)
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Peter ZijlstraQuoted 35 lines · Peter Zijlstra
Quoted 24 lines · K.Prasad
I don't claim to have understood the requirements of perf-counters fully, but I sense the persistence of a few doubts about the register allocation mechanism of the hw-breakpoints API...thought it might be worthwhile to briefly explain.
A few notes about the same:
Kernel-space breakpoints: They are system-wide i.e. one kernel-space breakpoint request consumes one debug register on 'all' CPUs of the system.
User-space breakpoints: Requests are stored in per-thread data structures. Written onto the debug register only when scheduled-into the CPU, and are removed when another task using the debug registers is scheduled onto the CPU.
Debug register allocation mechanism: register request succeeds only when the availability of a debug register is guaranteed (for both user and kernel space). Allocation on a first-come-first-serve basis, no priorities for register requests and hence no pre-emption of requests.In short the available debug registers are divided into:
# of debug registers = # of kernel-space requests + MAX(# of all user-space requests) + freely available debug registers.Quoted 8 lines
Thus on an x86 system with 4 debug registers, if there exists 1 kernel-space request it consumes one debug register on all CPUs in the system. The remaining 3 debug registers can be consumed by user-space i.e. upto 3 user-space requests per-thread.
There is no restriction on how many debug registers can be consumed by the kernel- or user-space requests and it is entirely dependent on the number of free registers available for use then.
Quoted 8 lines
Right, this is an utter miss-match for what perf counters would want.
Firstly, you seem to have this weird split of kernel/userspace breakpoints. Perf counters looks at things in a per-cpu fashion, so the all-cpus kernel breakpoint stuff is useless. Also, from perf counters' POV its perfectly reasonable to have a per-task kernel breakpoint.
Although the existing implementation of hw-breakpoint API doesn't support per-task kernel-space breakpoints, it isn't very difficult to extend it to do so.
We could change the breakpoint infrastructure to something like this:
kernel-space breakpoints:
kernel-space addresses, system-wide i.e. on all CPUs, persist till explicit unregistration, consume 1 debug register always.
New per-task breakpoints (i.e. modified user-space breakpoints):
accepts kernel- or user-space addresses, enabled per-task, consumes 1 debug register (only when task is scheduled on the CPU), releases debug register when yielding the CPU.
Quoted 4 lines
Secondly, perf counters wants to schedule the per task breakpoints because we can optimize the context switch, saving lots of these MSR writes under some common scenarios.
perf counters can continue to schedule per-task breakpoints - enabling/disabling a breakpoint would require a call to the 'register'/'unregister' interface and since it is per-cpu it is light-weight when compared to system-wide breakpoints (that require IPIs for propagation).
The common breakpoints can be identified and exempted from yielding the debug registers (i.e. from the unregister-->register cycle) in the perf-counter code.
As a side note, I'm not sure if extra-polating (linearly?) the debug register's "hit counter" value is a good idea. While a function may cause several 'write' operations on a variable (say due to a loop statement) for once, it may not exhibit similar behaviour throughout the time-slice of the program's execution. Scaling the values may lead to incorrect results.
Quoted 3 lines
Thirdly, we can multiplex perf counters beyond their hardware maximum, something you simply cannot do for a debug interface.
I suppose that you are referring to the RR scheduling of breakpoints and scaling of results? It can be achieved in the manner as explained above.
Quoted 7 lines
Like I said, please use the raw per-cpu breakpoint interface for perf counters and connect that with the minimally required reservation you need to make your other thing work.
You simply cannot put perf-counter breakpoints on top of whatever virt layer you created going by what you say it is.
One of the design goals of the hw-breakpoint API is to provide a layer of arbitration between various consumers of the physical debug register. We should be able to extend the API to meet the demands of new users with unique requirements (if not supported already), and the description above broadly describe them for perf-counters.
If agreeable, I'll submit a patch that modifies the user-space breakpoints into a per-task one. Frederic may want to base the perf-counter integration code on the proposed interface.
Let me know what your thoughts on the same.
Thanks,
K.Prasad
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 12 lines · K.Prasad
Quoted 5 lines
Firstly, you seem to have this weird split of kernel/userspace breakpoints. Perf counters looks at things in a per-cpu fashion, so the all-cpus kernel breakpoint stuff is useless. Also, from perf counters' POV its perfectly reasonable to have a per-task kernel breakpoint.
Although the existing implementation of hw-breakpoint API doesn't support per-task kernel-space breakpoints, it isn't very difficult to extend it to do so.
We could change the breakpoint infrastructure to something like this:
Quoted 8 lines
kernel-space breakpoints:
kernel-space addresses, system-wide i.e. on all CPUs, persist till explicit unregistration, consume 1 debug register always.New per-task breakpoints (i.e. modified user-space breakpoints):
accepts kernel- or user-space addresses, enabled per-task, consumes 1 debug register (only when task is scheduled on the CPU), releases debug register when yielding the CPU.
That still doesn't provide per-cpu breakpoints.
Quoted 11 lines
Quoted 4 lines
Secondly, perf counters wants to schedule the per task breakpoints because we can optimize the context switch, saving lots of these MSR writes under some common scenarios.
perf counters can continue to schedule per-task breakpoints - enabling/disabling a breakpoint would require a call to the 'register'/'unregister' interface and since it is per-cpu it is light-weight when compared to system-wide breakpoints (that require IPIs for propagation).
Quoted 3 lines
The common breakpoints can be identified and exempted from yielding the debug registers (i.e. from the unregister-->register cycle) in the perf-counter code.
If you want to implement it that way.. looking for duplicates is bound to result in something O(n^2), but with n=4 that's manageable.
Again, you seem to be missing per-cpu breakpoints.
Quoted 5 lines
As a side note, I'm not sure if extra-polating (linearly?) the debug register's "hit counter" value is a good idea. While a function may cause several 'write' operations on a variable (say due to a loop statement) for once, it may not exhibit similar behaviour throughout the time-slice of the program's execution. Scaling the values may lead to incorrect results.
Sure, it won't be perfect, but if you assume the RR interval is decoupled from the task you can get statistically relevant information.
Quoted 8 lines
Quoted 7 lines
Like I said, please use the raw per-cpu breakpoint interface for perf counters and connect that with the minimally required reservation you need to make your other thing work.
You simply cannot put perf-counter breakpoints on top of whatever virt layer you created going by what you say it is.
Quoted 5 lines
One of the design goals of the hw-breakpoint API is to provide a layer of arbitration between various consumers of the physical debug register. We should be able to extend the API to meet the demands of new users with unique requirements (if not supported already), and the description above broadly describe them for perf-counters.
Sure, but currently it does too much.
All you need for perf counter support is a per-cpu interface, no per-task, no system-wide.
But you want to mix that up with your per-task interface, which will complicate matters.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 18 lines · Peter Zijlstra
Quoted 16 lines · K.Prasad
Quoted 5 lines
Firstly, you seem to have this weird split of kernel/userspace breakpoints. Perf counters looks at things in a per-cpu fashion, so the all-cpus kernel breakpoint stuff is useless. Also, from perf counters' POV its perfectly reasonable to have a per-task kernel breakpoint.
Although the existing implementation of hw-breakpoint API doesn't support per-task kernel-space breakpoints, it isn't very difficult to extend it to do so.
We could change the breakpoint infrastructure to something like this:
kernel-space breakpoints:
kernel-space addresses, system-wide i.e. on all CPUs, persist till explicit unregistration, consume 1 debug register always.
Quoted 7 lines
Quoted 4 lines
New per-task breakpoints (i.e. modified user-space breakpoints):
accepts kernel- or user-space addresses, enabled per-task, consumes 1 debug register (only when task is scheduled on the CPU), releases debug register when yielding the CPU.That still doesn't provide per-cpu breakpoints.
Yes, it doesn't provide a per-cpu only implementation. One can obtain the per-cpu data from the system-wide breakpoints by filtering it for a given CPU (agreed, it will associated overhead).
A true per-cpu breakpoint implementation that co-exists with system-wide and per-task breakpoints will be difficult. It might require the re-introduction of some old features and a few new ones (like switching between kernel and user-space breakpoints at syscall time) that were rejected earlier by the community.
Also, the reason for a per-cpu only breakpoint (user and kernel-space) isn't very obvious. While kernel variables can be read/written throughout the system and user-space variables are per-task, the need for obtaining per-cpu information isn't clear.
Quoted 43 lines
Quoted 11 lines
Quoted 4 lines
Secondly, perf counters wants to schedule the per task breakpoints because we can optimize the context switch, saving lots of these MSR writes under some common scenarios.
perf counters can continue to schedule per-task breakpoints - enabling/disabling a breakpoint would require a call to the 'register'/'unregister' interface and since it is per-cpu it is light-weight when compared to system-wide breakpoints (that require IPIs for propagation).
Quoted 3 lines
The common breakpoints can be identified and exempted from yielding the debug registers (i.e. from the unregister-->register cycle) in the perf-counter code.
If you want to implement it that way.. looking for duplicates is bound to result in something O(n^2), but with n=4 that's manageable.
Again, you seem to be missing per-cpu breakpoints.
Quoted 5 lines
As a side note, I'm not sure if extra-polating (linearly?) the debug register's "hit counter" value is a good idea. While a function may cause several 'write' operations on a variable (say due to a loop statement) for once, it may not exhibit similar behaviour throughout the time-slice of the program's execution. Scaling the values may lead to incorrect results.
Sure, it won't be perfect, but if you assume the RR interval is decoupled from the task you can get statistically relevant information.
Quoted 8 lines
Quoted 7 lines
Like I said, please use the raw per-cpu breakpoint interface for perf counters and connect that with the minimally required reservation you need to make your other thing work.
You simply cannot put perf-counter breakpoints on top of whatever virt layer you created going by what you say it is.
Quoted 5 lines
One of the design goals of the hw-breakpoint API is to provide a layer of arbitration between various consumers of the physical debug register. We should be able to extend the API to meet the demands of new users with unique requirements (if not supported already), and the description above broadly describe them for perf-counters.
Quoted 8 lines
Sure, but currently it does too much.
All you need for perf counter support is a per-cpu interface, no per-task, no system-wide.
But you want to mix that up with your per-task interface, which will complicate matters.
This is a little confusing. I'm trying to understand which of the questions below does perf-counter try to answer. i) and ii) is what I thought would be, and asking iii) doesn't make much sense. What do you think?
i) Tell me how many times kernel variable 'x' was updated when task 'a' was scheduled
ii) Tell me how many times kernel variable 'x' was updated on the system since I registered for the breakpoint
iii) Tell me how many times kernel variable 'x' was updated on CPU 'n'
Thanks,
K.Prasad
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 7 lines · K.Prasad
Quoted 2 lines
That still doesn't provide per-cpu breakpoints.
Yes, it doesn't provide a per-cpu only implementation. One can obtain the per-cpu data from the system-wide breakpoints by filtering it for a given CPU (agreed, it will associated overhead).
Quoted 5 lines
A true per-cpu breakpoint implementation that co-exists with system-wide and per-task breakpoints will be difficult. It might require the re-introduction of some old features and a few new ones (like switching between kernel and user-space breakpoints at syscall time) that were rejected earlier by the community.
I'm not clear on why you'd need to switch breakpoints on syscall entry. You can simply leave the kernel address breakpoint around in userspace, they're not able to poke at that address space anyway.
Quoted 4 lines
Also, the reason for a per-cpu only breakpoint (user and kernel-space) isn't very obvious. While kernel variables can be read/written throughout the system and user-space variables are per-task, the need for obtaining per-cpu information isn't clear.
Well, suppose you're monitoring a per-cpu variable, or interested in the effects of a workload confined to 1 cpu or node, there is no reason to have this breakpoint on all cpus.
Quoted 11 lines
This is a little confusing. I'm trying to understand which of the questions below does perf-counter try to answer. i) and ii) is what I thought would be, and asking iii) doesn't make much sense. What do you think?
i) Tell me how many times kernel variable 'x' was updated when task 'a' was scheduled
ii) Tell me how many times kernel variable 'x' was updated on the system since I registered for the breakpoint
iii) Tell me how many times kernel variable 'x' was updated on CPU 'n'
It's 1 and 3. perf-counters doesn't deal with system wide for the reasons given above.
If you want system wide, you can create a perf counter for each cpu.
But the larger the machine gets, the less likely it is that that is what you're after. Being per-cpu allows you to do things on a per-node/socket/etc.. basis which is impossible to do when you limit yourself to per task or system wide.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Em Wed, Jul 29, 2009 at 11:22:52AM +0200, Peter Zijlstra escreveu:
Quoted 19 lines · K.Prasad
Quoted 8 lines
Quoted 3 lines
That still doesn't provide per-cpu breakpoints.
Yes, it doesn't provide a per-cpu only implementation. One can obtain the per-cpu data from the system-wide breakpoints by filtering it for a given CPU (agreed, it will associated overhead).
Quoted 5 lines
A true per-cpu breakpoint implementation that co-exists with system-wide and per-task breakpoints will be difficult. It might require the re-introduction of some old features and a few new ones (like switching between kernel and user-space breakpoints at syscall time) that were rejected earlier by the community.
I'm not clear on why you'd need to switch breakpoints on syscall entry. You can simply leave the kernel address breakpoint around in userspace, they're not able to poke at that address space anyway.
Quoted 8 lines
Quoted 4 lines
Also, the reason for a per-cpu only breakpoint (user and kernel-space) isn't very obvious. While kernel variables can be read/written throughout the system and user-space variables are per-task, the need for obtaining per-cpu information isn't clear.
Well, suppose you're monitoring a per-cpu variable, or interested in the effects of a workload confined to 1 cpu or node, there is no reason to have this breakpoint on all cpus.
I was going to talk about exactly that, CPU isolation for one specific workload, I don't want that hits on tcp_v4_rcv on another CPU get counted, just the ones on that specific CPU.
The other CPUs can be given a break (pun intended :-P) by not having to process traps we're uninterested in.
- Arnaldo
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
In reply to Peter ZijlstraQuoted 29 lines · Peter Zijlstra
Quoted 27 lines · Frederic Weisbecker
Quoted 4 lines
Ah, but that is sub-optimal, perf counters doesn't actually change the state if both tasks have the same counter configuration. Yielding a great performance benefit on scheduling intensive workloads. Poking at these MSRs, esp. writing to them is very expensive.
Ah ok.
Quoted 2 lines
So I would suggest not using that feature of the breakpoint API for the perf counter integration.
That would forbid some kinds of profiling (explanations below).
Quoted 5 lines
Quoted 2 lines
However, this patchset only deals with kernel breakpoint for now (wide tracing).
Right, and that's all you would need for perf counter support, please don't use whatever task state handling you have in place.
I would actually propose to have a separate layer that manages the hardware registers <-> per thread virtual registers handling for things like breakpoint api and perfcounter.
Quoted 8 lines
Quoted 4 lines
I know a simple RR of registers is not that hard to write, but at least that can allow simultaneous use of perfcounter and other users of breakpoint API without having two different versions of register management.
I simply cannot see how you would be able to multiplex userspace/debug breakpoints. I'd utterly hate it if I'd missed a breakpoint simply because someone else also wanted to make use of it.
What I mean by multiplexing is that, say in x86, each task can have 4 breakpoints maximum. Once the task is scheduled out, its breakpoints are saved and the hardware debug registers are used for the next task. Once a task registers a breakpoint, it never looses it.
Quoted 9 lines
I'd declare the system broken and useless.
Counters OTOH can be multiplexed because of their statistical nature, you can simply scale them back up based on their time share.
Therefore you'll have to deal with hard reservations anyway.
Also, you don't need to a-priory reserve all breakpoints, you'll simply need as many as the largest group (wrt breakpoints) has.
I still don't understand why it is needed to reserve breakpoints for a group of monitored tasks. Once they have registered their breakpoints, the number of necessary hardware registers for these will be available every time the task is scheduled.
By nature, MAX_NR (4 in x86) breakpoints are available for every tasks, minus the number of wide kernel breakpoints in use.
I don't see the need of a reservation here (which is already done by the API), I feel a bit confused in this debate.
Re: [RFC][PATCH 5/5] perfcounter: Add support for kernel hardware breakpoints
Quoted 5 lines · Frederic Weisbecker
I still don't understand why it is needed to reserve breakpoints for a group of monitored tasks. Once they have registered their breakpoints, the number of necessary hardware registers for these will be available every time the task is scheduled.
Its possible to create >4 breakpoint counters. We time-share them.
The only constraint we have is that those within a group should be on the 'PMU' at the same time.
If you have no groups, a single available breakpoint slot should suffice for any configuration (although more is better), but if you get groups you have to guarantee your created group can run.