Skip to contentSkip to conversation

V2 kgdb regression fixes for 2.6.33

PATCH In review Jason Wessel 4 participants 14 messages Jan 27, 2010 – Feb 1, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover V2 kgdb regression fixes for 2.6.33

I would like to get acks from the respective parties to fix these reported regressions against kgdb in 2.6.33.

The constraints for the hw breakpoint API is possibly was definitely a dicey issue, but is hopefully resolved in this series. Even without the constraints patch it is possible to use hw breakpoints in the kernel debugger in the same manner that has existed since 2.6.26 (only kgdb gets to use hw breakpoints).

The regression are:
* hw breakpoints no longer work on x86 after the perf API merge
* softlockup watchdog can reboot the system while using the kernel debugger *** This has been in linux-next for several months waiting for acks

Dropped from the series was the clocksource patch, it was resolved separately.

I collected all the patches which could go into the tip branch or the kgdb for_linus branch at the following location depending on the status of the discussion that ensues.

git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb.git for_igno

Thanks,
Jason.

---

The following changes since commit be8cde8b24c9dca1e54598690115eee5b1476519:
Linus Torvalds (1):

        Merge git://git.kernel.org/.../jejb/scsi-rc-fixes-2.6

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb.git for_ingo

Jason Wessel (3):
softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume x86,hw_breakpoint,kgdb: kgdb to use hw_breakpoint API perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger

 arch/x86/kernel/hw_breakpoint.c |    5 +-
 arch/x86/kernel/kgdb.c          |  251 ++++++++++++++++++++++++++++++---------
 include/linux/hw_breakpoint.h   |    2 +
 include/linux/sched.h           |    4 +
 kernel/hw_breakpoint.c          |   52 +++++++--
 kernel/kgdb.c                   |    9 +-
 kernel/softlockup.c             |   15 +++
 7 files changed, 264 insertions(+), 74 deletions(-)
#1 lore raw

PATCH 1/3 softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, sched_clock() gets the time from hardware such as the TSC on x86. In this configuration kgdb will report a softlock warning message on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ... touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...

    scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
    clock" to "hardware time" = 180s) ...  }

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }

note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"

Why is "touch_timestamp" 100 sec, instead of 180 sec?

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, the call trace of get_timestamp() is:

get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)

The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" value is too big for sched_clock_data, it will be ignored.

The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(). This allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.

[[email protected]: Use per cpu instead of an array]
CC Ingo Molnar <[email protected]>
---
 include/linux/sched.h |    4 ++++
 kernel/kgdb.c         |    6 +++---
 kernel/softlockup.c   |   15 +++++++++++++++
 3 files changed, 22 insertions(+), 3 deletions(-)
include/linux/sched.h +4 -0
index 6f7bba9..8923215 100644
@@ -310,6 +310,7 @@ extern void sched_show_task(struct task_struct *p);
 #ifdef CONFIG_DETECT_SOFTLOCKUP
 extern void softlockup_tick(void);
 extern void touch_softlockup_watchdog(void);
+extern void touch_softlockup_watchdog_sync(void);
 extern void touch_all_softlockup_watchdogs(void);
 extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write,
 				    void __user *buffer,
@@ -323,6 +324,9 @@ static inline void softlockup_tick(void)
 static inline void touch_softlockup_watchdog(void)
 {
 }
+static inline void touch_softlockup_watchdog_sync(void)
+{
+}
 static inline void touch_all_softlockup_watchdogs(void)
 {
 }
kernel/kgdb.c +3 -3
index 2eb517e..87f2cc5 100644
@@ -596,7 +596,7 @@ static void kgdb_wait(struct pt_regs *regs)
 
 	/* Signal the primary CPU that we are done: */
 	atomic_set(&cpu_in_kgdb[cpu], 0);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 }
@@ -1450,7 +1450,7 @@ acquirelock:
 	    (kgdb_info[cpu].task &&
 	     kgdb_info[cpu].task->pid != kgdb_sstep_pid) && --sstep_tries) {
 		atomic_set(&kgdb_active, -1);
-		touch_softlockup_watchdog();
+		touch_softlockup_watchdog_sync();
 		clocksource_touch_watchdog();
 		local_irq_restore(flags);
 
@@ -1550,7 +1550,7 @@ kgdb_restore:
 	}
 	/* Free kgdb_active */
 	atomic_set(&kgdb_active, -1);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 
kernel/softlockup.c +15 -0
index d225790..0d4c789 100644
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock);
 static DEFINE_PER_CPU(unsigned long, softlockup_touch_ts); /* touch timestamp */
 static DEFINE_PER_CPU(unsigned long, softlockup_print_ts); /* print timestamp */
 static DEFINE_PER_CPU(struct task_struct *, softlockup_watchdog);
+static DEFINE_PER_CPU(bool, softlock_touch_sync);
 
 static int __read_mostly did_panic;
 int __read_mostly softlockup_thresh = 60;
@@ -79,6 +80,12 @@ void touch_softlockup_watchdog(void)
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+void touch_softlockup_watchdog_sync(void)
+{
+	__raw_get_cpu_var(softlock_touch_sync) = true;
+	__raw_get_cpu_var(softlockup_touch_ts) = 0;
+}
+
 void touch_all_softlockup_watchdogs(void)
 {
 	int cpu;
@@ -118,6 +125,14 @@ void softlockup_tick(void)
 	}
 
 	if (touch_ts == 0) {
+		if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			per_cpu(softlock_touch_sync, this_cpu) = false;
+			sched_clock_tick();
+		}
 		__touch_softlockup_watchdog();
 		return;
 	}
Signature · 1 line
1.6.4.rc1
#2 lore raw

Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 4 lines · Jason Wessel
kernel/softlockup.c +0 -0
@@ -118,6 +125,14 @@ void softlockup_tick(void)
 	}
 
 	if (touch_ts == 0) {

kernel/softlockup.c void softlockup_tick(void) · line 137

Quoted 10 lines
kernel/softlockup.c +8 -0
+		if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			per_cpu(softlock_touch_sync, this_cpu) = false;
+			sched_clock_tick();
+		}
 		__touch_softlockup_watchdog();
 		return;

Shouldnt just all of touch_softlockup_watchdog() gain this new sched_clock_tick() call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?

That would also make the patch much simpler i guess, as we'd only have the chunk above.

Ingo

#3 lore raw

Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Jason Wessel
kernel/softlockup.c +1 -0
@@ -118,6 +125,14 @@ void softlockup_tick(void)
 	}
 
 	if (touch_ts == 0) {
+		if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {

kernel/softlockup.c void softlockup_tick(void) · line 138

Quoted 10 lines
kernel/softlockup.c +7 -0
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			per_cpu(softlock_touch_sync, this_cpu) = false;
+			sched_clock_tick();
+		}
 		__touch_softlockup_watchdog();
 		return;
 

Shouldnt just all of touch_softlockup_watchdog() gain this new sched_clock_tick() call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?

Quoted 3 lines

That would also make the patch much simpler i guess, as we'd only have the chunk above.

We have already been down that road, and it breaks other cases.

http://lkml.org/lkml/2009/7/28/204

Specifically the test case of:

echo 3 > /proc/sys/kernel/softlockup_thresh

And then some kernel code in a thread like:

        local_irq_disable();
        printk("Disable local irq for 11 seconds\n");
        mdelay(11000);
        local_irq_enable();

I could consider calling sched_cpu_clock() before returning the kernel to normal execution, but that didn't look very safe to call from the exception context, which is why it was delayed until the next time the soft lockup code ran.

Resuming from a long sleep is a ugly problem, so I am open to short term and long term suggestions, including a polling time API (obviously we would prefer not to go down that rat hole :-)

Jason.

#4 lore raw

Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 34 lines · Jason Wessel
Quoted 23 lines · Ingo Molnar
Quoted 5 lines · Jason Wessel
kernel/softlockup.c +1 -0
@@ -118,6 +125,14 @@ void softlockup_tick(void)
      }
 
      if (touch_ts == 0) {
+            if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {

kernel/softlockup.c void softlockup_tick(void) · line 138

Quoted 10 lines
kernel/softlockup.c +7 -0
+                    /*
+                     * If the time stamp was touched atomically
+                     * make sure the scheduler tick is up to date.
+                     */
+                    per_cpu(softlock_touch_sync, this_cpu) = false;
+                    sched_clock_tick();
+            }
              __touch_softlockup_watchdog();
              return;
 

Shouldnt just all of touch_softlockup_watchdog() gain this new sched_clock_tick() call, instead of doing this ugly flaggery? Or would that lock up or misbehave in other ways in some cases?

Quoted 3 lines

That would also make the patch much simpler i guess, as we'd only have the chunk above.

We have already been down that road, and it breaks other cases.

http://lkml.org/lkml/2009/7/28/204

Specifically the test case of:

Quoted 7 lines

echo 3 > /proc/sys/kernel/softlockup_thresh

And then some kernel code in a thread like:
       local_irq_disable();
       printk("Disable local irq for 11 seconds\n"); mdelay(11000);
       local_irq_enable();

Hi Jason,

Maybe this problem was fixed by
commit baf48f6577e581a9adb8fe849dc80e24b21d171d - "softlock: fix false panic which can occur if softlockup_thresh is reduced".

Thanks,
Dongdong

Quoted 18 lines

I could consider calling sched_cpu_clock() before returning the kernel to normal execution, but that didn't look very safe to call from the exception context, which is why it was delayed until the next time the soft lockup code ran.

Resuming from a long sleep is a ugly problem, so I am open to short term and long term suggestions, including a polling time API (obviously we would prefer not to go down that rat hole :-)

Jason.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#5 lore raw

Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 12 lines · Dongdong Deng
Quoted 8 lines · Jason Wessel

echo 3 > /proc/sys/kernel/softlockup_thresh

And then some kernel code in a thread like:

       local_irq_disable();
       printk("Disable local irq for 11 seconds\n");
       mdelay(11000);
       local_irq_enable();
Quoted 6 lines

Hi Jason,

Maybe this problem was fixed by
commit baf48f6577e581a9adb8fe849dc80e24b21d171d - "softlock: fix false panic which can occur if softlockup_thresh is reduced".

That is not the same problem. The fix you referenced is a corner case
where you end up with the stack trace at the point in time you reduce the threshold. The only reason I reduce the threshold in the first place is just to shorten the amount of time it takes to observe the problem.

You can just change the numbers for the mdelay and use the default softlockup threshold values and still see the problem I reported.

Jason.

#6 lore raw

Re: [PATCH 1/3] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 21 lines · Jason Wessel
Quoted 12 lines · Dongdong Deng
Quoted 8 lines · Jason Wessel

echo 3 > /proc/sys/kernel/softlockup_thresh

And then some kernel code in a thread like:
       local_irq_disable();
       printk("Disable local irq for 11 seconds\n"); mdelay(11000);
       local_irq_enable();

Quoted 6 lines

Hi Jason,

Maybe this problem was fixed by
commit baf48f6577e581a9adb8fe849dc80e24b21d171d - "softlock: fix false panic which can occur if softlockup_thresh is reduced".

Quoted 4 lines

That is not the same problem.   The fix you referenced is a corner case where you end up with the stack trace at the point in time you reduce the threshold.  The only reason I reduce the threshold in the first place is just to shorten the amount of time it takes to observe the problem.

Thanks for your nice explanation. :)

Dongdong

Quoted 5 lines

You can just change the numbers for the mdelay and use the default softlockup threshold values and still see the problem I reported.

Jason.

#7 lore raw

[tip:core/urgent] softlockup: Add sched_clock_tick() to avoid kernel warning on kgdb resume

In reply to Jason Wessel
Commit-ID:  d6ad3e286d2c075a60b9f11075a2c55aeeeca2ad
Gitweb:     http://git.kernel.org/tip/d6ad3e286d2c075a60b9f11075a2c55aeeeca2ad
Author:     Jason Wessel <[email protected]>
AuthorDate: Wed, 27 Jan 2010 16:25:22 -0600
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 1 Feb 2010 08:22:32 +0100

softlockup: Add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, sched_clock() gets the time from hardware such as the TSC on x86. In this configuration kgdb will report a softlock warning message on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 sec and on

    exit the following is called ...  touch_softlockup_watchdog() -->
    __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...

    scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
    clock" to "hardware time" = 180s) ...  }

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }

note: ***(A) reset "touch_timestamp" to
"get_timestamp(this_cpu)"

Why is "touch_timestamp" 100 sec, instead of 180 sec?

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set, the call trace of get_timestamp() is:

get_timestamp(this_cpu)
-->cpu_clock(this_cpu)
-->sched_clock_cpu(this_cpu)
-->__update_sched_clock(sched_clock_data, now)

The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" value is too big for sched_clock_data, it will be ignored.

The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(). This allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.

LKML-Reference: <[email protected]>

---
 include/linux/sched.h |    4 ++++
 kernel/kgdb.c         |    6 +++---
 kernel/softlockup.c   |   15 +++++++++++++++
 3 files changed, 22 insertions(+), 3 deletions(-)
include/linux/sched.h +4 -0
index 6f7bba9..8923215 100644
@@ -310,6 +310,7 @@ extern void sched_show_task(struct task_struct *p);
 #ifdef CONFIG_DETECT_SOFTLOCKUP
 extern void softlockup_tick(void);
 extern void touch_softlockup_watchdog(void);
+extern void touch_softlockup_watchdog_sync(void);
 extern void touch_all_softlockup_watchdogs(void);
 extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write,
 				    void __user *buffer,
@@ -323,6 +324,9 @@ static inline void softlockup_tick(void)
 static inline void touch_softlockup_watchdog(void)
 {
 }
+static inline void touch_softlockup_watchdog_sync(void)
+{
+}
 static inline void touch_all_softlockup_watchdogs(void)
 {
 }
kernel/kgdb.c +3 -3
index 2eb517e..87f2cc5 100644
@@ -596,7 +596,7 @@ static void kgdb_wait(struct pt_regs *regs)
 
 	/* Signal the primary CPU that we are done: */
 	atomic_set(&cpu_in_kgdb[cpu], 0);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 }
@@ -1450,7 +1450,7 @@ acquirelock:
 	    (kgdb_info[cpu].task &&
 	     kgdb_info[cpu].task->pid != kgdb_sstep_pid) && --sstep_tries) {
 		atomic_set(&kgdb_active, -1);
-		touch_softlockup_watchdog();
+		touch_softlockup_watchdog_sync();
 		clocksource_touch_watchdog();
 		local_irq_restore(flags);
 
@@ -1550,7 +1550,7 @@ kgdb_restore:
 	}
 	/* Free kgdb_active */
 	atomic_set(&kgdb_active, -1);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 
kernel/softlockup.c +15 -0
index d225790..0d4c789 100644
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock);
 static DEFINE_PER_CPU(unsigned long, softlockup_touch_ts); /* touch timestamp */
 static DEFINE_PER_CPU(unsigned long, softlockup_print_ts); /* print timestamp */
 static DEFINE_PER_CPU(struct task_struct *, softlockup_watchdog);
+static DEFINE_PER_CPU(bool, softlock_touch_sync);
 
 static int __read_mostly did_panic;
 int __read_mostly softlockup_thresh = 60;
@@ -79,6 +80,12 @@ void touch_softlockup_watchdog(void)
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+void touch_softlockup_watchdog_sync(void)
+{
+	__raw_get_cpu_var(softlock_touch_sync) = true;
+	__raw_get_cpu_var(softlockup_touch_ts) = 0;
+}
+
 void touch_all_softlockup_watchdogs(void)
 {
 	int cpu;
@@ -118,6 +125,14 @@ void softlockup_tick(void)
 	}
 
 	if (touch_ts == 0) {
+		if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			per_cpu(softlock_touch_sync, this_cpu) = false;
+			sched_clock_tick();
+		}
 		__touch_softlockup_watchdog();
 		return;
 	}
#8 lore raw

PATCH 2/3 x86,hw_breakpoint,kgdb: kgdb to use hw_breakpoint API

In reply to Jason Wessel

In the 2.6.33 kernel, the hw_breakpoint API is now used for the performance event counters. The hw_breakpoint_handler() now consumes the hw breakpoints that were previously set by kgdb arch specific code. In order for kgdb to work in conjunction with this core API change, kgdb must use some of the low level functions of the hw_breakpoint API to install, uninstall, and receive call backs for hw breakpoints.

The kgdb core needs to call kgdb_disable_hw_debug anytime a slave cpu enters kgdb_wait() in order to keep all the hw breakpoints in sync as well as to prevent hitting a hw breakpoint while kgdb is active.

During the architecture specific initialization of kgdb, it will pre-allocate 4 disabled (struct perf event **) structures. Kgdb will use these to manage the capabilities for the 4 hw breakpoint registers. Right now the hw_breakpoint API does not have a way to ask how many breakpoints are available, on each CPU so it is possible that the install of a breakpoint might fail when kgdb restores the system to the run state. The intent of this patch is to first get the basic functionality of hw breakpoints working and leave it to the person debugging the kernel to understand what hw breakpoints are in use and what restrictions have been imposed as a result.

While atomic, the x86 specific kgdb code will call arch_uninstall_hw_breakpoint() and arch_install_hw_breakpoint() to manage the cpu specific hw breakpoints.

The arch specific hw_breakpoint_handler() was changed to restore the cpu specific dr7 instead of the dr7 that was locally saved, because the dr7 can be modified while in a call back to kgdb.

The net result of these changes allow kgdb to use the same pool of hw_breakpoints that are used by the perf event API, but neither knows about future reservations for the available hw breakpoint slots.

CC Frederic Weisbecker <[email protected]>
CC Ingo Molnar <[email protected]>
CC K.Prasad <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Alan Stern <[email protected]>
---
 arch/x86/kernel/hw_breakpoint.c |    5 +-
 arch/x86/kernel/kgdb.c          |  206 ++++++++++++++++++++++++++++-----------
 kernel/kgdb.c                   |    3 +
 3 files changed, 152 insertions(+), 62 deletions(-)
arch/x86/kernel/hw_breakpoint.c +2 -3
index 05d5fec..cbf19e0 100644
@@ -466,7 +466,7 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)
 {
 	int i, cpu, rc = NOTIFY_STOP;
 	struct perf_event *bp;
-	unsigned long dr7, dr6;
+	unsigned long dr6;
 	unsigned long *dr6_p;
 
 	/* The DR6 value is pointed by args->err */
@@ -477,7 +477,6 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)
 	if ((dr6 & DR_TRAP_BITS) == 0)
 		return NOTIFY_DONE;
 
-	get_debugreg(dr7, 7);
 	/* Disable breakpoints during exception handling */
 	set_debugreg(0UL, 7);
 	/*
@@ -525,7 +524,7 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args)
 	if (dr6 & (~DR_TRAP_BITS))
 		rc = NOTIFY_DONE;
 
-	set_debugreg(dr7, 7);
+	set_debugreg(__get_cpu_var(cpu_dr7), 7);
 	put_cpu();
 
 	return rc;
arch/x86/kernel/kgdb.c +147 -59

315 lines not on this page · raw message

kernel/kgdb.c +3 -0
index 87f2cc5..761fdd2 100644
@@ -583,6 +583,9 @@ static void kgdb_wait(struct pt_regs *regs)
 	smp_wmb();
 	atomic_set(&cpu_in_kgdb[cpu], 1);
 
+	/* Disable any cpu specific hw breakpoints */
+	kgdb_disable_hw_debug(regs);
+
 	/* Wait till primary CPU is done with debugging */
 	while (atomic_read(&passive_cpu_wait[cpu]))
 		cpu_relax();
Signature · 1 line
1.6.4.rc1
#9 lore raw

PATCH 3/3 perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger

In reply to Jason Wessel

The kernel debugger cannot use any mutex_lock() calls because it can start the kernel running from an invalid context.

The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.

Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.

CC Frederic Weisbecker <[email protected]>
CC Ingo Molnar <[email protected]>
CC K.Prasad <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Alan Stern <[email protected]>
---
 arch/x86/kernel/kgdb.c        |   49 +++++++++++++++++++++++++++++++++++++-
 include/linux/hw_breakpoint.h |    2 +
 kernel/hw_breakpoint.c        |   52 +++++++++++++++++++++++++++++++++--------
 3 files changed, 92 insertions(+), 11 deletions(-)
arch/x86/kernel/kgdb.c +48 -1
index 9f47cd3..7c3e929 100644
@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)
 	hw_breakpoint_restore();
 }
 
+static int hw_break_reserve_slot(int breakno)
+{
+	int cpu;
+	int cnt = 0;
+	struct perf_event **pevent;
+
+	for_each_online_cpu(cpu) {
+		cnt++;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		if (dbg_reserve_bp_slot(*pevent))
+			goto fail;
+	}
+
+	return 0;
+
+fail:
+	for_each_online_cpu(cpu) {
+		cnt--;
+		if (!cnt)
+			break;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		dbg_release_bp_slot(*pevent);
+	}
+	return -1;
+}
+
+static int hw_break_release_slot(int breakno)
+{
+	struct perf_event **pevent;
+	int ret;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		ret = dbg_release_bp_slot(*pevent);
+	}
+	return ret;
+}
+
 static int
 kgdb_remove_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype)
 {
@@ -250,6 +289,10 @@ kgdb_remove_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype)
 	if (i == 4)
 		return -1;
 
+	if (hw_break_release_slot(i)) {
+		printk(KERN_ERR "Cannot remove hw breakpoint at %lx\n", addr);
+		return -1;
+	}
 	breakinfo[i].enabled = 0;
 
 	return 0;
@@ -313,9 +356,13 @@ kgdb_set_hw_break(unsigned long addr, int len, enum kgdb_bptype bptype)
 		break;
 #endif
 	default:
-	return -1;
+		return -1;
 	}
 	breakinfo[i].addr = addr;
+	if (hw_break_reserve_slot(i)) {
+		breakinfo[i].addr = 0;
+		return -1;
+	}
 	breakinfo[i].enabled = 1;
 
 	return 0;
include/linux/hw_breakpoint.h +2 -0
index 41235c9..070ba06 100644
@@ -75,6 +75,8 @@ extern int __register_perf_hw_breakpoint(struct perf_event *bp);
 extern void unregister_hw_breakpoint(struct perf_event *bp);
 extern void unregister_wide_hw_breakpoint(struct perf_event **cpu_events);
 
+extern int dbg_reserve_bp_slot(struct perf_event *bp);
+extern int dbg_release_bp_slot(struct perf_event *bp);
 extern int reserve_bp_slot(struct perf_event *bp);
 extern void release_bp_slot(struct perf_event *bp);
 
kernel/hw_breakpoint.c +42 -10

80 lines not on this page · raw message

Signature · 1 line
1.6.4.rc1
#10 lore raw

Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kernel debugger

Quoted 57 lines · Jason Wessel

The kernel debugger cannot use any mutex_lock() calls because it can start the kernel running from an invalid context.

The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.

Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.

CC Frederic Weisbecker <[email protected]>
CC Ingo Molnar <[email protected]>
CC K.Prasad <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Alan Stern <[email protected]>
---
 arch/x86/kernel/kgdb.c        |   49 +++++++++++++++++++++++++++++++++++++-
 include/linux/hw_breakpoint.h |    2 +
 kernel/hw_breakpoint.c        |   52 +++++++++++++++++++++++++++++++++--------
 3 files changed, 92 insertions(+), 11 deletions(-)
arch/x86/kernel/kgdb.c +25 -0
index 9f47cd3..7c3e929 100644
@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)
 	hw_breakpoint_restore();
 }
 
+static int hw_break_reserve_slot(int breakno)
+{
+	int cpu;
+	int cnt = 0;
+	struct perf_event **pevent;
+
+	for_each_online_cpu(cpu) {
+		cnt++;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		if (dbg_reserve_bp_slot(*pevent))
+			goto fail;
+	}
+
+	return 0;
+
+fail:
+	for_each_online_cpu(cpu) {
+		cnt--;
+		if (!cnt)
+			break;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		dbg_release_bp_slot(*pevent);
+	}
+	return -1;
+}

arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void) · line 276

Quoted 10 lines
arch/x86/kernel/kgdb.c +10 -0
+
+static int hw_break_release_slot(int breakno)
+{
+	struct perf_event **pevent;
+	int ret;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		ret = dbg_release_bp_slot(*pevent);

So, you are missing some return errors there. Actually, a slot release shouldn't return an error.

Quoted 12 lines
arch/x86/kernel/kgdb.c +12 -0
+/*
+ * Allow the kernel debugger to reserve breakpoint slots without
+ * taking a lock using the dbg_* variant of for the reserve and
+ * release breakpoint slots.
+ */
+int dbg_reserve_bp_slot(struct perf_event *bp)
+{
+	if (mutex_is_locked(&nr_bp_mutex))
+		return -1;
+
+	return __reserve_bp_slot(bp);
+}

arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void)

Quoted 10 lines
arch/x86/kernel/kgdb.c +10 -0
+
+int dbg_release_bp_slot(struct perf_event *bp)
+{
+	if (mutex_is_locked(&nr_bp_mutex))
+		return -1;
+
+	__release_bp_slot(bp);
+
+	return 0;
+}

Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual release_bp_slot, may be toward the workqueues.

Quoted 6 lines

int register_perf_hw_breakpoint(struct perf_event *bp) {

Signature · 1 line
1.6.4.rc1
#11 lore raw

Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kerneldebugger

Quoted 67 lines · Frederic Weisbecker
Quoted 65 lines · Jason Wessel

The kernel debugger cannot use any mutex_lock() calls because it can start the kernel running from an invalid context.

The possibility for a breakpoint reservation to be concurrently processed at the time that kgdb interrupts the system is improbable. As a safety check against this condition the kernel debugger will prohibit updating the hardware breakpoint reservations and an error will be returned to the end user.

Any time the kernel debugger reserves a hardware breakpoint it will be a system wide reservation.

CC Frederic Weisbecker <[email protected]>
CC Ingo Molnar <[email protected]>
CC K.Prasad <[email protected]>
CC Peter Zijlstra <[email protected]>
CC Alan Stern <[email protected]>
---
 arch/x86/kernel/kgdb.c        |   49 +++++++++++++++++++++++++++++++++++++-
 include/linux/hw_breakpoint.h |    2 +
 kernel/hw_breakpoint.c        |   52 +++++++++++++++++++++++++++++++++--------
 3 files changed, 92 insertions(+), 11 deletions(-)
arch/x86/kernel/kgdb.c +33 -0
index 9f47cd3..7c3e929 100644
@@ -239,6 +239,45 @@ static void kgdb_correct_hw_break(void)
 	hw_breakpoint_restore();
 }
 
+static int hw_break_reserve_slot(int breakno)
+{
+	int cpu;
+	int cnt = 0;
+	struct perf_event **pevent;
+
+	for_each_online_cpu(cpu) {
+		cnt++;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		if (dbg_reserve_bp_slot(*pevent))
+			goto fail;
+	}
+
+	return 0;
+
+fail:
+	for_each_online_cpu(cpu) {
+		cnt--;
+		if (!cnt)
+			break;
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		dbg_release_bp_slot(*pevent);
+	}
+	return -1;
+}
+
+static int hw_break_release_slot(int breakno)
+{
+	struct perf_event **pevent;
+	int ret;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
Quoted 11 lines

arch/x86/kernel/kgdb.c static void kgdb_correct_hw_break(void)

Quoted 3 lines
arch/x86/kernel/kgdb.c +2 -0
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		ret = dbg_release_bp_slot(*pevent);
 

So, you are missing some return errors there. Actually, a slot release shouldn't return an error.

This is a trick so to speak. Either all the slot releases will return
0 or -1 depending on if the mutex is available, so it is not really missed.

Certainly I can change this to just quit immediately on error.

Quoted 23 lines
Quoted 23 lines
arch/x86/kernel/kgdb.c +22 -0
+/*
+ * Allow the kernel debugger to reserve breakpoint slots without
+ * taking a lock using the dbg_* variant of for the reserve and
+ * release breakpoint slots.
+ */
+int dbg_reserve_bp_slot(struct perf_event *bp)
+{
+	if (mutex_is_locked(&nr_bp_mutex))
+		return -1;
+
+	return __reserve_bp_slot(bp);
+}
+
+int dbg_release_bp_slot(struct perf_event *bp)
+{
+	if (mutex_is_locked(&nr_bp_mutex))
+		return -1;
+
+	__release_bp_slot(bp);
+
+	return 0;
+}
Quoted 12 lines
Quoted 1 line

Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual release_bp_slot, may be toward the workqueues.

If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).

Quoted 10 lines
Quoted 8 lines

int register_perf_hw_breakpoint(struct perf_event *bp) {

Signature · 1 line
1.6.4.rc1
#12 lore raw

Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken for kerneldebugger

Quoted 20 lines · Jason Wessel
Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines
arch/x86/kernel/kgdb.c +7 -0
+static int hw_break_release_slot(int breakno)
+{
+	struct perf_event **pevent;
+	int ret;
+	int cpu;
+
+	for_each_online_cpu(cpu) {
Quoted 11 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+		pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+		ret = dbg_release_bp_slot(*pevent);

So, you are missing some return errors there. Actually, a slot release shouldn't return an error.

Quoted 3 lines

This is a trick so to speak. Either all the slot releases will return
0 or -1 depending on if the mutex is available, so it is not really missed.

Oh right, I forgot everything was freezed here :)

Quoted 9 lines
Quoted 8 lines

Ok, best effort fits well for reserve, but is certainly not suitable for release. We can't leave a fake occupied slot like this. If it fails, we should do this asynchronously, using the usual release_bp_slot, may be toward the workqueues.

Quoted 3 lines

If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).

Yeah truly it's a corner case, especially if the debugger can handle that later.

May be just add a comment so that future reviewers don't stick to this part.

Thanks!

#13 lore raw

Re: [PATCH 3/3] perf,hw_breakpoint,kgdb: No mutex taken forkerneldebugger

Quoted 11 lines · Frederic Weisbecker
Quoted 3 lines

If it fails the debugger tried to remove it again later. It seems to
me like it is a don't care corner case. You get a printk if it ever
does happen (which it really shouldn't).

Yeah truly it's a corner case, especially if the debugger can handle that later.

May be just add a comment so that future reviewers don't stick to this part.

If you approve, I'll add your ack.

It looks like this now:

+static int hw_break_release_slot(int breakno)
+{
+ struct perf_event **pevent;
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ pevent = per_cpu_ptr(breakinfo[breakno].pev, cpu);
+ if (dbg_release_bp_slot(*pevent))
+ /*
+ * The debugger is responisble for handing the retry on
+ * remove failure.
+ */
+ return -1;
+ }
+ return 0;
+}
+

Thanks,
Jason.