V2 kgdb regression fixes for 2.6.33
cover V2 kgdb regression fixes for 2.6.33
I would like to get acks from the respective parties to fix these reported regressions against kgdb in 2.6.33.
The constraints for the hw breakpoint API is possibly was definitely a dicey issue, but is hopefully resolved in this series. Even without the constraints patch it is possible to use hw breakpoints in the kernel debugger in the same manner that has existed since 2.6.26 (only kgdb gets to use hw breakpoints).
The regression are:
* hw breakpoints no longer work on x86 after the perf API merge
* softlockup watchdog can reboot the system while using the kernel debugger *** This has been in linux-next for several months waiting for acks
Dropped from the series was the clocksource patch, it was resolved separately.
I collected all the patches which could go into the tip branch or the kgdb for_linus branch at the following location depending on the status of the discussion that ensues.
git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb.git for_igno
Thanks,
Jason.
---
The following changes since commit be8cde8b24c9dca1e54598690115eee5b1476519:
Linus Torvalds (1):
Merge git://git.kernel.org/.../jejb/scsi-rc-fixes-2.6
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb.git for_ingo
Jason Wessel (3):
softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume x86,hw_breakpoint,kgdb: kgdb to use hw_breakpoint API perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger
arch/x86/kernel/hw_breakpoint.c | 5 +- arch/x86/kernel/kgdb.c | 251 ++++++++++++++++++++++++++++++--------- include/linux/hw_breakpoint.h | 2 + include/linux/sched.h | 4 + kernel/hw_breakpoint.c | 52 +++++++-- kernel/kgdb.c | 9 +- kernel/softlockup.c | 15 +++ 7 files changed, 264 insertions(+), 74 deletions(-)
PATCH 1/3 softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, sched_clock() gets the time from hardware such as the TSC on x86. In this configuration kgdb will report a softlock warning message on resuming or detaching from a debug session.
Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()
2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ... touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;
3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec
4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"
update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...
scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
clock" to "hardware time" = 180s) ... }5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }
note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"
Why is "touch_timestamp" 100 sec, instead of 180 sec?
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, the call trace of get_timestamp() is:
get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)
The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" value is too big for sched_clock_data, it will be ignored.
The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(). This allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.
--- include/linux/sched.h | 4 ++++ kernel/kgdb.c | 6 +++--- kernel/softlockup.c | 15 +++++++++++++++ 3 files changed, 22 insertions(+), 3 deletions(-)
include/linux/sched.h +4 -0
@@ -310,6 +310,7 @@ extern void sched_show_task(struct task_struct *p);#ifdef CONFIG_DETECT_SOFTLOCKUP extern void softlockup_tick (void );extern void touch_softlockup_watchdog (void ); +extern void touch_softlockup_watchdog_sync (void );extern void touch_all_softlockup_watchdogs (void ); extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write, void __user *buffer,
@@ -323,6 +324,9 @@ static inline void softlockup_tick(void)static inline void touch_softlockup_watchdog (void ) { } +static inline void touch_softlockup_watchdog_sync (void ) +{ +}static inline void touch_all_softlockup_watchdogs (void ) { }
kernel/kgdb.c +3 -3
@@ -596,7 +596,7 @@ static void kgdb_wait(struct pt_regs *regs)/* Signal the primary CPU that we are done: */ atomic_set (&cpu_in_kgdb[cpu],0 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags); }
@@ -1450,7 +1450,7 @@ acquirelock: (kgdb_info[cpu].task && kgdb_info[cpu].task->pid != kgdb_sstep_pid) && --sstep_tries) {atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
@@ -1550,7 +1550,7 @@ kgdb_restore: }/* Free kgdb_active */ atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
kernel/softlockup.c +15 -0
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock); static DEFINE_PER_CPU(unsigned long, softlockup_touch_ts); /* touch timestamp */ static DEFINE_PER_CPU(unsigned long, softlockup_print_ts); /* print timestamp */ static DEFINE_PER_CPU(struct task_struct *, softlockup_watchdog); +static DEFINE_PER_CPU(bool, softlock_touch_sync);static int __read_mostly did_panic;int __read_mostly softlockup_thresh =60 ;
@@ -79,6 +80,12 @@ void touch_softlockup_watchdog(void) }EXPORT_SYMBOL (touch_softlockup_watchdog); +void touch_softlockup_watchdog_sync (void ) +{ +__raw_get_cpu_var (softlock_touch_sync) = true; +__raw_get_cpu_var (softlockup_touch_ts) =0 ; +} +void touch_all_softlockup_watchdogs (void ) {int cpu;
@@ -118,6 +125,14 @@ void softlockup_tick(void) }if (touch_ts ==0 ) { +if (unlikely (per_cpu (softlock_touch_sync, this_cpu))) { +/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ +per_cpu (softlock_touch_sync, this_cpu) = false; +sched_clock_tick (); + }__touch_softlockup_watchdog ();return ; }
Signature · 1 line
1.6.4.rc1
Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
Quoted 4 lines · Jason Wessel
kernel/softlockup.c +0
-0@@ -118,6 +125,14 @@ void softlockup_tick(void) }if (touch_ts ==0 ) {
kernel/softlockup.c void softlockup_tick(void) · line 137
Quoted 10 lines
kernel/softlockup.c +8
-0+if (unlikely (per_cpu (softlock_touch_sync, this_cpu))) { +/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ +per_cpu (softlock_touch_sync, this_cpu) = false; +sched_clock_tick (); + }__touch_softlockup_watchdog ();return ;
Shouldnt just all of touch_softlockup_watchdog() gain this new sched_clock_tick() call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?
That would also make the patch much simpler i guess, as we'd only have the chunk above.
Ingo
Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Jason Wessel
kernel/softlockup.c +1
-0@@ -118,6 +125,14 @@ void softlockup_tick(void) }if (touch_ts ==0 ) { +if (unlikely (per_cpu (softlock_touch_sync, this_cpu))) {kernel/softlockup.c void softlockup_tick(void) · line 138
Quoted 10 lines
kernel/softlockup.c +7
-0+/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ +per_cpu (softlock_touch_sync, this_cpu) = false; +sched_clock_tick (); + }__touch_softlockup_watchdog ();return ;Shouldnt just all of
touch_softlockup_watchdog()gain this newsched_clock_tick()call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?
Quoted 3 lines
That would also make the patch much simpler i guess, as we'd only have the chunk above.
We have already been down that road, and it breaks other cases.
http://lkml.org/lkml/2009/7/28/204
Specifically the test case of:
echo 3 > /proc/sys/kernel/softlockup_thresh
And then some kernel code in a thread like:
local_irq_disable();
printk("Disable local irq for 11 seconds\n");
mdelay(11000);
local_irq_enable();I could consider calling sched_cpu_clock() before returning the kernel to normal execution, but that didn't look very safe to call from the exception context, which is why it was delayed until the next time the soft lockup code ran.
Resuming from a long sleep is a ugly problem, so I am open to short term and long term suggestions, including a polling time API (obviously we would prefer not to go down that rat hole :-)
Jason.
Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
Quoted 34 lines · Jason Wessel
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Jason Wessel
kernel/softlockup.c +1
-0@@ -118,6 +125,14 @@ void softlockup_tick(void) } if (touch_ts == 0) { + if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {kernel/softlockup.c void softlockup_tick(void) · line 138
Quoted 10 lines
kernel/softlockup.c +7
-0+ /* + * If the time stamp was touched atomically + * make sure the scheduler tick is up to date. + */ + per_cpu(softlock_touch_sync, this_cpu) = false; + sched_clock_tick(); + } __touch_softlockup_watchdog(); return;Shouldnt just all of
touch_softlockup_watchdog()gain this newsched_clock_tick()call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?Quoted 3 lines
That would also make the patch much simpler i guess, as we'd only have the chunk above.
We have already been down that road, and it breaks other cases.
http://lkml.org/lkml/2009/7/28/204
Specifically the test case of:
Quoted 7 lines
echo 3 >
/proc/sys/kernel/softlockup_threshAnd then some kernel code in a thread like:
local_irq_disable();
printk("Disable local irq for 11 seconds\n"); mdelay(11000);
local_irq_enable();
Hi Jason,
Maybe this problem was fixed by
commit baf48f6577e581a9adb8fe849dc80e24b21d171d - "softlock: fix false panic which can occur if softlockup_thresh is reduced".
Thanks,
Dongdong
Quoted 18 lines
I could consider calling
sched_cpu_clock()before returning the kernel to normal execution, but that didn't look very safe to call from the exception context, which is why it was delayed until the next time the soft lockup code ran.Resuming from a long sleep is a ugly problem, so I am open to short term and long term suggestions, including a polling time API (obviously we would prefer not to go down that rat hole :-)
Jason.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
Quoted 12 lines · Dongdong Deng
Quoted 8 lines · Jason Wessel
echo 3 >
/proc/sys/kernel/softlockup_threshAnd then some kernel code in a thread like:
local_irq_disable(); printk("Disable local irq for 11 seconds\n"); mdelay(11000); local_irq_enable();
Quoted 6 lines
Hi Jason,
Maybe this problem was fixed by
commitbaf48f6577e581a9adb8fe849dc80e24b21d171d- "softlock: fix false panic which can occur ifsoftlockup_threshis reduced".
That is not the same problem. The fix you referenced is a corner case
where you end up with the stack trace at the point in time you reduce the threshold. The only reason I reduce the threshold in the first place is just to shorten the amount of time it takes to observe the problem.
You can just change the numbers for the mdelay and use the default softlockup threshold values and still see the problem I reported.
Jason.
Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
Quoted 21 lines · Jason Wessel
Quoted 12 lines · Dongdong Deng
Quoted 8 lines · Jason Wessel
echo 3 >
/proc/sys/kernel/softlockup_threshAnd then some kernel code in a thread like:
local_irq_disable();
printk("Disable local irq for 11 seconds\n"); mdelay(11000);
local_irq_enable();Quoted 6 lines
Hi Jason,
Maybe this problem was fixed by
commitbaf48f6577e581a9adb8fe849dc80e24b21d171d- "softlock: fix false panic which can occur ifsoftlockup_threshis reduced".
Quoted 4 lines
That is not the same problem. The fix you referenced is a corner case where you end up with the stack trace at the point in time you reduce the threshold. The only reason I reduce the threshold in the first place is just to shorten the amount of time it takes to observe the problem.
Thanks for your nice explanation. :)
Dongdong
Quoted 5 lines
You can just change the numbers for the mdelay and use the default softlockup threshold values and still see the problem I reported.
Jason.
[tip:core/urgent] softlockup: Add sched_clock_tick() to avoid kernel warning on kgdb resume
In reply to Jason WesselCommit-ID: d6ad3e286d2c075a60b9f11075a2c55aeeeca2ad Gitweb: http://git.kernel.org/tip/d6ad3e286d2c075a60b9f11075a2c55aeeeca2ad Author: Jason Wessel <[email protected]> AuthorDate: Wed, 27 Jan 2010 16:25:22 -0600 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 1 Feb 2010 08:22:32 +0100
softlockup: Add sched_clock_tick() to avoid kernel warning on kgdb resume
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, sched_clock() gets the time from hardware such as the TSC on x86. In this configuration kgdb will report a softlock warning message on resuming or detaching from a debug session.
Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()
2) Debugger waits in kgdb_handle_exception() for 80 sec and on
exit the following is called ... touch_softlockup_watchdog() -->
__raw_get_cpu_var(touch_timestamp) = 0;3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec
4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"
update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...
scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
clock" to "hardware time" = 180s) ... }5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }
note: ***(A) reset "touch_timestamp" to
"get_timestamp(this_cpu)"
Why is "touch_timestamp" 100 sec, instead of 180 sec?
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, the call trace of get_timestamp() is:
get_timestamp(this_cpu)
-->cpu_clock(this_cpu)
-->sched_clock_cpu(this_cpu)
-->__update_sched_clock(sched_clock_data, now)
The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" value is too big for sched_clock_data, it will be ignored.
The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(). This allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.
LKML-Reference: <[email protected]>
--- include/linux/sched.h | 4 ++++ kernel/kgdb.c | 6 +++--- kernel/softlockup.c | 15 +++++++++++++++ 3 files changed, 22 insertions(+), 3 deletions(-)
include/linux/sched.h +4 -0
@@ -310,6 +310,7 @@ extern void sched_show_task(struct task_struct *p);#ifdef CONFIG_DETECT_SOFTLOCKUP extern void softlockup_tick (void );extern void touch_softlockup_watchdog (void ); +extern void touch_softlockup_watchdog_sync (void );extern void touch_all_softlockup_watchdogs (void ); extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write, void __user *buffer,
@@ -323,6 +324,9 @@ static inline void softlockup_tick(void)static inline void touch_softlockup_watchdog (void ) { } +static inline void touch_softlockup_watchdog_sync (void ) +{ +}static inline void touch_all_softlockup_watchdogs (void ) { }
kernel/kgdb.c +3 -3
@@ -596,7 +596,7 @@ static void kgdb_wait(struct pt_regs *regs)/* Signal the primary CPU that we are done: */ atomic_set (&cpu_in_kgdb[cpu],0 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags); }
@@ -1450,7 +1450,7 @@ acquirelock: (kgdb_info[cpu].task && kgdb_info[cpu].task->pid != kgdb_sstep_pid) && --sstep_tries) {atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
@@ -1550,7 +1550,7 @@ kgdb_restore: }/* Free kgdb_active */ atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
kernel/softlockup.c +15 -0
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock); static DEFINE_PER_CPU(unsigned long, softlockup_touch_ts); /* touch timestamp */ static DEFINE_PER_CPU(unsigned long, softlockup_print_ts); /* print timestamp */ static DEFINE_PER_CPU(struct task_struct *, softlockup_watchdog); +static DEFINE_PER_CPU(bool, softlock_touch_sync);static int __read_mostly did_panic;int __read_mostly softlockup_thresh =60 ;
@@ -79,6 +80,12 @@ void touch_softlockup_watchdog(void) }EXPORT_SYMBOL (touch_softlockup_watchdog); +void touch_softlockup_watchdog_sync (void ) +{ +__raw_get_cpu_var (softlock_touch_sync) = true; +__raw_get_cpu_var (softlockup_touch_ts) =0 ; +} +void touch_all_softlockup_watchdogs (void ) {int cpu;
@@ -118,6 +125,14 @@ void softlockup_tick(void) }if (touch_ts ==0 ) { +if (unlikely (per_cpu (softlock_touch_sync, this_cpu))) { +/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ +per_cpu (softlock_touch_sync, this_cpu) = false; +sched_clock_tick (); + }__touch_softlockup_watchdog ();return ; }
PATCH 2/3 x86,hw_breakpoint,kgdb: kgdb to use hw_breakpoint API
In reply to Jason WesselIn the 2.6.33 kernel, the hw_breakpoint API is now used for the performance event counters. The hw_breakpoint_handler() now consumes the hw breakpoints that were previously set by kgdb arch specific code. In order for kgdb to work in conjunction with this core API change, kgdb must use some of the low level functions of the hw_breakpoint API to install, uninstall, and receive call backs for hw breakpoints.
The kgdb core needs to call kgdb_disable_hw_debug anytime a slave cpu enters kgdb_wait() in order to keep all the hw breakpoints in sync as well as to prevent hitting a hw breakpoint while kgdb is active.
During the architecture specific initialization of kgdb, it will pre-allocate 4 disabled (struct perf event **) structures. Kgdb will use these to manage the capabilities for the 4 hw breakpoint registers. Right now the hw_breakpoint API does not have a way to ask how many breakpoints are available, on each CPU so it is possible that the install of a breakpoint might fail when kgdb restores the system to the run state. The intent of this patch is to first get the basic functionality of hw breakpoints working and leave it to the person debugging the kernel to understand what hw breakpoints are in use and what restrictions have been imposed as a result.
While atomic, the x86 specific kgdb code will call arch_uninstall_hw_breakpoint() and arch_install_hw_breakpoint() to manage the cpu specific hw breakpoints.
The arch specific hw_breakpoint_handler() was changed to restore the cpu specific dr7 instead of the dr7 that was locally saved, because the dr7 can be modified while in a call back to kgdb.
The net result of these changes allow kgdb to use the same pool of hw_breakpoints that are used by the perf event API, but neither knows about future reservations for the available hw breakpoint slots.
--- arch/x86/kernel/hw_breakpoint.c | 5 +- arch/x86/kernel/kgdb.c | 206 ++++++++++++++++++++++++++++----------- kernel/kgdb.c | 3 + 3 files changed, 152 insertions(+), 62 deletions(-)
arch/x86/kernel/hw_breakpoint.c +2 -3
@@ -466,7 +466,7 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args) {int i, cpu, rc =NOTIFY_STOP ;struct perf_event *bp; -unsigned long dr7, dr6; +unsigned long dr6;unsigned long *dr6_p;/* The DR6 value is pointed by args->err */
@@ -477,7 +477,6 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)if ((dr6 &DR_TRAP_BITS ) ==0 )return NOTIFY_DONE ; -get_debugreg (dr7,7 );/* Disable breakpoints during exception handling */ set_debugreg (0UL ,7 );/*
@@ -525,7 +524,7 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)if (dr6 & (~DR_TRAP_BITS )) rc =NOTIFY_DONE ; -set_debugreg (dr7,7 ); +set_debugreg (__get_cpu_var (cpu_dr7),7 );put_cpu ();return rc;
arch/x86/kernel/kgdb.c +147 -59
315 lines not on this page · raw message
kernel/kgdb.c +3 -0
@@ -583,6 +583,9 @@ static void kgdb_wait(struct pt_regs *regs)smp_wmb ();atomic_set (&cpu_in_kgdb[cpu],1 ); +/* Disable any cpu specific hw breakpoints */ +kgdb_disable_hw_debug (regs); +/* Wait till primary CPU is done with debugging */ while (atomic_read (&passive_cpu_wait[cpu]))cpu_relax ();
Signature · 1 line
1.6.4.rc1
PATCH 3/3 perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger
In reply to Jason WesselThe kernel debugger cannot use any mutex_lock() calls because it can start the kernel running from an invalid context.
The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.
Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.
--- arch/x86/kernel/kgdb.c | 49 +++++++++++++++++++++++++++++++++++++- include/linux/hw_breakpoint.h | 2 + kernel/hw_breakpoint.c | 52 +++++++++++++++++++++++++++++++++-------- 3 files changed, 92 insertions(+), 11 deletions(-)
arch/x86/kernel/kgdb.c +48 -1
@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)hw_breakpoint_restore (); } +static int hw_break_reserve_slot (int breakno) +{ +int cpu; +int cnt =0 ; +struct perf_event **pevent; + +for_each_online_cpu (cpu) { + cnt++; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +if (dbg_reserve_bp_slot (*pevent)) + goto fail; + } + +return 0 ; + +fail: +for_each_online_cpu (cpu) { + cnt--; +if (!cnt) +break ; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +dbg_release_bp_slot (*pevent); + } +return -1 ; +} + +static int hw_break_release_slot (int breakno) +{ +struct perf_event **pevent; +int ret; +int cpu; + +for_each_online_cpu (cpu) { + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); + ret =dbg_release_bp_slot (*pevent); + } +return ret; +} +static int kgdb_remove_hw_break (unsigned long addr,int len,enum kgdb_bptype bptype) {
@@ -250,6 +289,10 @@ kgdb_remove_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype)if (i ==4 )return -1 ; +if (hw_break_release_slot (i)) { +printk (KERN_ERR "Cannot remove hw breakpoint at %lx\n" , addr); +return -1 ; + } breakinfo[i].enabled =0 ;return 0 ;
@@ -313,9 +356,13 @@ kgdb_set_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype)break ;#endif default : -return -1 ; +return -1 ; } breakinfo[i].addr = addr; +if (hw_break_reserve_slot (i)) { + breakinfo[i].addr =0 ; +return -1 ; + } breakinfo[i].enabled =1 ;return 0 ;
include/linux/hw_breakpoint.h +2 -0
@@ -75,6 +75,8 @@ extern int __register_perf_hw_breakpoint(struct perf_event *bp);extern void unregister_hw_breakpoint (struct perf_event *bp);extern void unregister_wide_hw_breakpoint (struct perf_event **cpu_events); +extern int dbg_reserve_bp_slot (struct perf_event *bp); +extern int dbg_release_bp_slot (struct perf_event *bp);extern int reserve_bp_slot (struct perf_event *bp);extern void release_bp_slot (struct perf_event *bp);
kernel/hw_breakpoint.c +42 -10
80 lines not on this page · raw message
Signature · 1 line
1.6.4.rc1
Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger
Quoted 57 lines · Jason Wessel
The kernel debugger cannot use any
mutex_lock()calls because it can start the kernel running from an invalid context.The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.
Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.
CC Frederic Weisbecker <[email protected]>CC Ingo Molnar <[email protected]>CC K.Prasad <[email protected]>CC Peter Zijlstra <[email protected]>CC Alan Stern <[email protected]>Signed-off-by Jason Wessel <[email protected]>--- arch/x86/kernel/kgdb.c | 49 +++++++++++++++++++++++++++++++++++++- include/linux/hw_breakpoint.h | 2 + kernel/hw_breakpoint.c | 52 +++++++++++++++++++++++++++++++++-------- 3 files changed, 92 insertions(+), 11 deletions(-)arch/x86/kernel/kgdb.c +25
-0@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)hw_breakpoint_restore (); } +static int hw_break_reserve_slot (int breakno) +{ +int cpu; +int cnt =0 ; +struct perf_event **pevent; + +for_each_online_cpu (cpu) { + cnt++; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +if (dbg_reserve_bp_slot (*pevent)) + goto fail; + } + +return 0 ; + +fail: +for_each_online_cpu (cpu) { + cnt--; +if (!cnt) +break ; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +dbg_release_bp_slot (*pevent); + } +return -1 ; +}
arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void) · line 276
Quoted 10 lines
arch/x86/kernel/kgdb.c +10
-0+ +static int hw_break_release_slot (int breakno) +{ +struct perf_event **pevent; +int ret; +int cpu; + +for_each_online_cpu (cpu) { + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); + ret =dbg_release_bp_slot (*pevent);
So, you are missing some return errors there. Actually, a slot release shouldn't return an error.
Quoted 12 lines
arch/x86/kernel/kgdb.c +12
-0+/* +* Allow the kernel debugger to reserve breakpoint slots without +* taking a lock using the dbg_* variant of for the reserve and +* release breakpoint slots. +*/ +int dbg_reserve_bp_slot (struct perf_event *bp) +{ +if (mutex_is_locked (&nr_bp_mutex)) +return -1 ; + +return __reserve_bp_slot (bp); +}
arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void)
Quoted 10 lines
arch/x86/kernel/kgdb.c +10
-0+ +int dbg_release_bp_slot (struct perf_event *bp) +{ +if (mutex_is_locked (&nr_bp_mutex)) +return -1 ; + +__release_bp_slot (bp); + +return 0 ; +}
Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual release_bp_slot, may be toward the workqueues.
Quoted 6 lines
int
register_perf_hw_breakpoint(struct perf_event *bp){Signature · 1 line
1.6.4.rc1
Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kerneldebugger
Quoted 67 lines · Frederic Weisbecker
Quoted 65 lines · Jason Wessel
The kernel debugger cannot use any
mutex_lock()calls because it can start the kernel running from an invalid context.The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.
Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.
CC Frederic Weisbecker <[email protected]>CC Ingo Molnar <[email protected]>CC K.Prasad <[email protected]>CC Peter Zijlstra <[email protected]>CC Alan Stern <[email protected]>Signed-off-by Jason Wessel <[email protected]>--- arch/x86/kernel/kgdb.c | 49 +++++++++++++++++++++++++++++++++++++- include/linux/hw_breakpoint.h | 2 + kernel/hw_breakpoint.c | 52 +++++++++++++++++++++++++++++++++-------- 3 files changed, 92 insertions(+), 11 deletions(-)arch/x86/kernel/kgdb.c +33
-0@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)hw_breakpoint_restore (); } +static int hw_break_reserve_slot (int breakno) +{ +int cpu; +int cnt =0 ; +struct perf_event **pevent; + +for_each_online_cpu (cpu) { + cnt++; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +if (dbg_reserve_bp_slot (*pevent)) + goto fail; + } + +return 0 ; + +fail: +for_each_online_cpu (cpu) { + cnt--; +if (!cnt) +break ; + pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); +dbg_release_bp_slot (*pevent); + } +return -1 ; +} + +static int hw_break_release_slot (int breakno) +{ +struct perf_event **pevent; +int ret; +int cpu; + +for_each_online_cpu (cpu) {
Quoted 11 lines
arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void)
Quoted 3 lines
arch/x86/kernel/kgdb.c +2
-0+ pevent =per_cpu_ptr (breakinfo[breakno].pev, cpu); + ret =dbg_release_bp_slot (*pevent);So, you are missing some return errors there. Actually, a slot release shouldn't return an error.
This is a trick so to speak. Either all the slot releases will return
0 or -1 depending on if the mutex is available, so it is not really missed.
Certainly I can change this to just quit immediately on error.
Quoted 23 lines
Quoted 23 lines
arch/x86/kernel/kgdb.c +22
-0+/* +* Allow the kernel debugger to reserve breakpoint slots without +* taking a lock using the dbg_* variant of for the reserve and +* release breakpoint slots. +*/ +int dbg_reserve_bp_slot (struct perf_event *bp) +{ +if (mutex_is_locked (&nr_bp_mutex)) +return -1 ; + +return __reserve_bp_slot (bp); +} + +int dbg_release_bp_slot (struct perf_event *bp) +{ +if (mutex_is_locked (&nr_bp_mutex)) +return -1 ; + +__release_bp_slot (bp); + +return 0 ; +}
Quoted 12 lines
Quoted 1 line
Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual
release_bp_slot, may be toward the workqueues.
If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).
Quoted 10 lines
Quoted 8 lines
int
register_perf_hw_breakpoint(struct perf_event *bp){Signature · 1 line
1.6.4.rc1
Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kerneldebugger
Quoted 20 lines · Jason Wessel
Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines
arch/x86/kernel/kgdb.c +7
-0+static int hw_break_release_slot (int breakno) +{ +struct perf_event **pevent; +int ret; +int cpu; + +for_each_online_cpu (cpu) {Quoted 11 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu); + ret = dbg_release_bp_slot(*pevent);So, you are missing some return errors there. Actually, a slot release shouldn't return an error.
Quoted 3 lines
This is a trick so to speak. Either all the slot releases will return
0 or -1 depending on if the mutex is available, so it is not really missed.
Oh right, I forgot everything was freezed here :)
Quoted 9 lines
Quoted 8 lines
Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual
release_bp_slot, may be toward the workqueues.
Quoted 3 lines
If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).
Yeah truly it's a corner case, especially if the debugger can handle that later.
May be just add a comment so that future reviewers don't stick to this part.
Thanks!
Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken forkerneldebugger
Quoted 11 lines · Frederic Weisbecker
Quoted 3 lines
If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).Yeah truly it's a corner case, especially if the debugger can handle that later.
May be just add a comment so that future reviewers don't stick to this part.
If you approve, I'll add your ack.
It looks like this now:
+static int hw_break_release_slot(int breakno)
+{
+ struct perf_event **pevent;
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+ if (dbg_release_bp_slot(*pevent))
+ /*
+ * The debugger is responisble for handing the retry on
+ * remove failure.
+ */
+ return -1;
+ }
+ return 0;
+}
+
Thanks,
Jason.