Skip to contentSkip to conversation

watchdog: Fix race on single cpu boot

PATCH In review Jiri Olsa 4 participants 5 messages Jul 31, 2015 – Aug 2, 2015

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

PATCH 1/1 watchdog: Fix race on single cpu boot

We are experiencing following crashes under kdump
kernel execution:

  [    1.930428] BUG: unable to handle kernel NULL pointer dereference at 0000000000000080

SNIP

  [    2.478604] Call Trace:
  [    2.479364]  [<ffffffff81158f06>] perf_event_enable+0x16/0x40
  [    2.481035]  [<ffffffff81119a2c>] watchdog_nmi_enable+0x10c/0x170
  [    2.482906]  [<ffffffff81119c14>] watchdog_enable+0x54/0xc0
  [    2.484583]  [<ffffffff810ad209>] smpboot_thread_fn+0xb9/0x1a0
  [    2.486302]  [<ffffffff81634bb9>] ? schedule+0x29/0x70
  [    2.487811]  [<ffffffff810ad150>] ? lg_double_unlock+0x90/0x90
  [    2.4810a484f>] kthread+0xcf/0xe0
  [    2.991478]  [<ffffffff810a4780>] ? kthread_create_on_node+0x140/0x140
  [    2.993519]  [<ffffffff8163fad8>] ret_from_fork+0x58/0x90
  [    2.995192]  [<ffffffff810a4780>] ? kthread_create_on_node+0x140/0x140
  [    3.005787] RIP  [<ffffffff8115744a>] perf_event_ctx_lock_nested.isra.50+0x5a/0xa0
  [    3.008350]  RSP <ffff88003199fde0>
  [    3.009417] CR2: 0000000000000080
  [    3.010432] ---[ end trace 892dd7be017c5cde ]---
  [    3.011792] Kernel panic - not syncing: Fatal exception
  [    3.013379] Rebooting in 10 seconds..

It's caused by watchdog_nmi_disable_all/watchdog_nmi_enable_all calls executed by fixup_ht_bug (as fallback for HT bug) and watchdog_enable function executed by watchdog's smp_hotplug_thread setup callback.

The watchdog_enable function calls watchdog_nmi_enable which is not serialized with watchdog_nmi_disable_all/watchdog_nmi_enable_all. We see following race during the boot.

(booting with 'maxcpus=1' to emulate kdump mode and execute HT bug fallback).

---
kernel_init

    kernel_init_freeable
      lockup_detector_init
        smpboot_register_percpu_thread

        -> schedules out to the new kthread
  ---
  smpboot_thread_fn
    watchdog_enable
      watchdog_nmi_enable
        perf_event_create_kernel_counter
        per_cpu(watchdog_ev, cpu) = event
        perf_event_enable
          perf_event_ctx_lock_nested
          -> schedules out back to the kernel_init
  ---
  kernel_init
     ...
     do_basic_setup
       do_initcalls
         fixup_ht_bug
           watchdog_nmi_disable_all
           -> release per_cpu(watchdog_ev, cpu) initialized by watchdog_enable

           watchdog_nmi_enable_all
     ...
     -> schedules out to the smpboot_thread_fn
  ---
  smpboot_thread_fn
     ...
     -> crash in perf_event_ctx_lock_nested because the event
        was released

Protecting watchdog_nmi_(enable|disable) calls with new per-cpu mutex.

I can easily reproduce this on SNB server with 'maxcpus=1' command line option, and I guess it could be triggered on any server that deals with the HT bug. I dig out following:

case 42: /* 32nm SandyBridge */
case 45: /* 32nm SandyBridge-E/EN/EP */
case 58: /* 22nm IvyBridge */
case 62: /* 22nm IvyBridge-EP/EX */
case 60: /* 22nm Haswell Core */
case 63: /* 22nm Haswell Server */
case 69: /* 22nm Haswell ULT */
case 70: /* 22nm Haswell + GT3e (Intel Iris Pro graphics) */

---
 kernel/watchdog.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
kernel/watchdog.c +20 -2
index a6ffa43f2993..8fcc75116277 100644
@@ -82,6 +82,7 @@ static DEFINE_PER_CPU(bool, hard_watchdog_warn);
 static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
 static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved);
 static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
+static DEFINE_PER_CPU(struct mutex, watchdog_ev_lock);
 #endif
 static unsigned long soft_lockup_nmi_warn;
 
@@ -526,7 +527,10 @@ static unsigned long cpu0_err;
 static int watchdog_nmi_enable(unsigned int cpu)
 {
 	struct perf_event_attr *wd_attr;
-	struct perf_event *event = per_cpu(watchdog_ev, cpu);
+	struct perf_event *event;
+
+	mutex_lock(&per_cpu(watchdog_ev_lock, cpu));
+	event = per_cpu(watchdog_ev, cpu);
 
 	/* nothing to do if the hard lockup detector is disabled */
 	if (!(watchdog_enabled & NMI_WATCHDOG_ENABLED))
@@ -557,6 +561,8 @@ static int watchdog_nmi_enable(unsigned int cpu)
 		goto out_save;
 	}
 
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
+
 	/*
 	 * Disable the hard lockup detector if _any_ CPU fails to set up
 	 * set up the hardware perf event. The watchdog() function checks
@@ -593,12 +599,16 @@ out_save:
 out_enable:
 	perf_event_enable(per_cpu(watchdog_ev, cpu));
 out:
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
 	return 0;
 }
 
 static void watchdog_nmi_disable(unsigned int cpu)
 {
-	struct perf_event *event = per_cpu(watchdog_ev, cpu);
+	struct perf_event *event;
+
+	mutex_lock(&per_cpu(watchdog_ev_lock, cpu));
+	event = per_cpu(watchdog_ev, cpu);
 
 	if (event) {
 		perf_event_disable(event);
@@ -611,6 +621,7 @@ static void watchdog_nmi_disable(unsigned int cpu)
 		/* watchdog_nmi_enable() expects this to be zero initially. */
 		cpu0_err = 0;
 	}
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
 }
 
 void watchdog_nmi_enable_all(void)
@@ -928,6 +939,8 @@ int proc_watchdog_cpumask(struct ctl_table *table, int write,
 
 void __init lockup_detector_init(void)
 {
+	int cpu __maybe_unused;
+
 	set_sample_period();
 
 #ifdef CONFIG_NO_HZ_FULL
@@ -942,6 +955,11 @@ void __init lockup_detector_init(void)
 	cpumask_copy(&watchdog_cpumask, cpu_possible_mask);
 #endif
 
+#ifdef CONFIG_HARDLOCKUP_DETECTOR
+	for_each_possible_cpu(cpu)
+		mutex_init(&per_cpu(watchdog_ev_lock, cpu));
+#endif
+
 	if (watchdog_enabled)
 		watchdog_enable_all_cpus();
 }
Signature · 1 line
2.4.3
#1 lore raw
Quoted 25 lines · Jiri Olsa

We are experiencing following crashes under kdump
kernel execution:

  [    1.930428] BUG: unable to handle kernel NULL pointer dereference at 0000000000000080

SNIP

  [    2.478604] Call Trace:
  [    2.479364]  [<ffffffff81158f06>] perf_event_enable+0x16/0x40
  [    2.481035]  [<ffffffff81119a2c>] watchdog_nmi_enable+0x10c/0x170
  [    2.482906]  [<ffffffff81119c14>] watchdog_enable+0x54/0xc0
  [    2.484583]  [<ffffffff810ad209>] smpboot_thread_fn+0xb9/0x1a0
  [    2.486302]  [<ffffffff81634bb9>] ? schedule+0x29/0x70
  [    2.487811]  [<ffffffff810ad150>] ? lg_double_unlock+0x90/0x90
  [    2.4810a484f>] kthread+0xcf/0xe0
  [    2.991478]  [<ffffffff810a4780>] ? kthread_create_on_node+0x140/0x140
  [    2.993519]  [<ffffffff8163fad8>] ret_from_fork+0x58/0x90
  [    2.995192]  [<ffffffff810a4780>] ? kthread_create_on_node+0x140/0x140
  [    3.005787] RIP  [<ffffffff8115744a>] perf_event_ctx_lock_nested.isra.50+0x5a/0xa0
  [    3.008350]  RSP <ffff88003199fde0>
  [    3.009417] CR2: 0000000000000080
  [    3.010432] ---[ end trace 892dd7be017c5cde ]---
  [    3.011792] Kernel panic - not syncing: Fatal exception
  [    3.013379] Rebooting in 10 seconds..
Quoted 3 lines

It's caused by watchdog_nmi_disable_all/watchdog_nmi_enable_all calls executed by fixup_ht_bug (as fallback for HT bug) and watchdog_enable function executed by watchdog's smp_hotplug_thread setup callback.

Hi Jiri,

Uli privately has been working on a patchset that cleans up a bunch of these race conditions. We believe it should cover this case. It uses the proc_mutex to synchronize everything.

I think he is reaching out to you. If you could try his patchset to see if it fixes things, it might be a cleaner approach than what you are doing.

<small time goes by>

Ok, I was poked on IRC and you have already done this and it passed your testcase. Let me get Uli to post his patches. :-)

Cheers,
Don

Quoted 149 lines

The watchdog_enable function calls watchdog_nmi_enable which is not serialized with watchdog_nmi_disable_all/watchdog_nmi_enable_all. We see following race during the boot.

(booting with 'maxcpus=1' to emulate kdump mode and execute HT bug fallback).

---
kernel_init

    kernel_init_freeable
      lockup_detector_init
        smpboot_register_percpu_thread

        -> schedules out to the new kthread
  ---
  smpboot_thread_fn
    watchdog_enable
      watchdog_nmi_enable
        perf_event_create_kernel_counter
        per_cpu(watchdog_ev, cpu) = event
        perf_event_enable
          perf_event_ctx_lock_nested
          -> schedules out back to the kernel_init
  ---
  kernel_init
     ...
     do_basic_setup
       do_initcalls
         fixup_ht_bug
           watchdog_nmi_disable_all
           -> release per_cpu(watchdog_ev, cpu) initialized by watchdog_enable

           watchdog_nmi_enable_all
     ...
     -> schedules out to the smpboot_thread_fn
  ---
  smpboot_thread_fn
     ...
     -> crash in perf_event_ctx_lock_nested because the event
        was released

Protecting watchdog_nmi_(enable|disable) calls with new per-cpu mutex.

I can easily reproduce this on SNB server with 'maxcpus=1' command line option, and I guess it could be triggered on any server that deals with the HT bug. I dig out following:

case 42: /* 32nm SandyBridge */
case 45: /* 32nm SandyBridge-E/EN/EP */
case 58: /* 22nm IvyBridge */
case 62: /* 22nm IvyBridge-EP/EX */
case 60: /* 22nm Haswell Core */
case 63: /* 22nm Haswell Server */
case 69: /* 22nm Haswell ULT */
case 70: /* 22nm Haswell + GT3e (Intel Iris Pro graphics) */

---
 kernel/watchdog.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
kernel/watchdog.c +20 -2
index a6ffa43f2993..8fcc75116277 100644
@@ -82,6 +82,7 @@ static DEFINE_PER_CPU(bool, hard_watchdog_warn);
 static DEFINE_PER_CPU(bool, watchdog_nmi_touch);
 static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved);
 static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
+static DEFINE_PER_CPU(struct mutex, watchdog_ev_lock);
 #endif
 static unsigned long soft_lockup_nmi_warn;
 
@@ -526,7 +527,10 @@ static unsigned long cpu0_err;
 static int watchdog_nmi_enable(unsigned int cpu)
 {
 	struct perf_event_attr *wd_attr;
-	struct perf_event *event = per_cpu(watchdog_ev, cpu);
+	struct perf_event *event;
+
+	mutex_lock(&per_cpu(watchdog_ev_lock, cpu));
+	event = per_cpu(watchdog_ev, cpu);
 
 	/* nothing to do if the hard lockup detector is disabled */
 	if (!(watchdog_enabled & NMI_WATCHDOG_ENABLED))
@@ -557,6 +561,8 @@ static int watchdog_nmi_enable(unsigned int cpu)
 		goto out_save;
 	}
 
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
+
 	/*
 	 * Disable the hard lockup detector if _any_ CPU fails to set up
 	 * set up the hardware perf event. The watchdog() function checks
@@ -593,12 +599,16 @@ out_save:
 out_enable:
 	perf_event_enable(per_cpu(watchdog_ev, cpu));
 out:
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
 	return 0;
 }
 
 static void watchdog_nmi_disable(unsigned int cpu)
 {
-	struct perf_event *event = per_cpu(watchdog_ev, cpu);
+	struct perf_event *event;
+
+	mutex_lock(&per_cpu(watchdog_ev_lock, cpu));
+	event = per_cpu(watchdog_ev, cpu);
 
 	if (event) {
 		perf_event_disable(event);
@@ -611,6 +621,7 @@ static void watchdog_nmi_disable(unsigned int cpu)
 		/* watchdog_nmi_enable() expects this to be zero initially. */
 		cpu0_err = 0;
 	}
+	mutex_unlock(&per_cpu(watchdog_ev_lock, cpu));
 }
 
 void watchdog_nmi_enable_all(void)
@@ -928,6 +939,8 @@ int proc_watchdog_cpumask(struct ctl_table *table, int write,
 
 void __init lockup_detector_init(void)
 {
+	int cpu __maybe_unused;
+
 	set_sample_period();
 
 #ifdef CONFIG_NO_HZ_FULL
@@ -942,6 +955,11 @@ void __init lockup_detector_init(void)
 	cpumask_copy(&watchdog_cpumask, cpu_possible_mask);
 #endif
 
+#ifdef CONFIG_HARDLOCKUP_DETECTOR
+	for_each_possible_cpu(cpu)
+		mutex_init(&per_cpu(watchdog_ev_lock, cpu));
+#endif
+
 	if (watchdog_enabled)
 		watchdog_enable_all_cpus();
 }
Signature · 7 lines
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#2 lore raw

Don,

Quoted 7 lines

Uli privately has been working on a patchset that cleans up a bunch of these race conditions. We believe it should cover this case. It uses the proc_mutex to synchronize everything.

I think he is reaching out to you. If you could try his patchset to see if it fixes things, it might be a cleaner approach than what you are doing.

Quoted 7 lines

<small time goes by>

Ok, I was poked on IRC and you have already done this and it passed your testcase. Let me get Uli to post his patches. :-)

Cheers,
Don

I posted the patch set here:

https://lkml.org/lkml/2015/8/1/64
https://lkml.org/lkml/2015/8/1/65
https://lkml.org/lkml/2015/8/1/66
https://lkml.org/lkml/2015/8/1/67
https://lkml.org/lkml/2015/8/1/68

Regards,

Uli

#3 lore raw
Quoted 18 lines · Ulrich Obergfell

Don,

Quoted 7 lines

Uli privately has been working on a patchset that cleans up a bunch of these race conditions. We believe it should cover this case. It uses the proc_mutex to synchronize everything.

I think he is reaching out to you. If you could try his patchset to see if it fixes things, it might be a cleaner approach than what you are doing.

Quoted 7 lines

<small time goes by>

Ok, I was poked on IRC and you have already done this and it passed your testcase. Let me get Uli to post his patches. :-)

Cheers,
Don

Quoted 7 lines

I posted the patch set here:

https://lkml.org/lkml/2015/8/1/64
https://lkml.org/lkml/2015/8/1/65
https://lkml.org/lkml/2015/8/1/66
https://lkml.org/lkml/2015/8/1/67
https://lkml.org/lkml/2015/8/1/68

If only you didn't use lkml.org links, that site is flaky beyond unusable.

Please use lkml.kernel.org/r/$msgid, that way people actually have a chance of getting a working link and/or finding it in their local archive.

#4 lore raw

Peter,

Quoted 8 lines
Quoted 7 lines

I posted the patch set here:

https://lkml.org/lkml/2015/8/1/64
https://lkml.org/lkml/2015/8/1/65
https://lkml.org/lkml/2015/8/1/66
https://lkml.org/lkml/2015/8/1/67
https://lkml.org/lkml/2015/8/1/68

Quoted 6 lines

If only you didn't use lkml.org links, that site is flaky beyond unusable.

Please use lkml.kernel.org/r/$msgid, that way people actually have a chance of getting a working link and/or finding it in their local archive.

I apologize for the inconvenience. The following links redirect to marc.info:

- [PATCH 0/4] watchdog: avoid races in watchdog_nmi_{en|disable} functions lkml.kernel.org/r/[email protected]

- [PATCH 1/4] watchdog: introduce watchdog_park_threads() and watchdog_unpark_threads() lkml.kernel.org/r/[email protected]

- [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume() lkml.kernel.org/r/[email protected]

This part of the series has already received a comment. The suggestion is to introduce different function names to avoid confusion with the watchdog subsystem, so I propose

lockup_detector_suspend() instead of watchdog_suspend() lockup_detector_resume() instead of watchdog_resume()

which would be consistent with the existing name lockup_detector_init().

- [PATCH 3/4] watchdog: use park/unpark functions in update_watchdog_all_cpus() lkml.kernel.org/r/[email protected]

- [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug() lkml.kernel.org/r/[email protected]

Regards,

Uli