watchdog: avoid races in watchdog_nmi_{en|disable} functions
Subsystems (3)
cover watchdog: avoid races in watchdog_nmi_{en|disable} functions
Originally watchdog_nmi_enable(cpu) and watchdog_nmi_disable(cpu) were only called in watchdog thread context. However, the following commits utilize these functions outside of watchdog thread context too.
commit 9809b18fcf6b8d8ec4d3643677345907e6b50eca
Author: Michal Hocko <[email protected]>
Date: Tue Sep 24 15:27:30 2013 -0700
watchdog: update watchdog_thresh properly
commit b3738d29323344da3017a91010530cf3a58590fc
Author: Stephane Eranian <[email protected]>
Date: Mon Nov 17 20:07:03 2014 +0100
watchdog: Add watchdog enable/disable all functions
Hence, it is now possible that these functions execute concurrently with the same 'cpu' argument. This concurrency is problematic because per-cpu 'watchdog_ev' can be accessed/modified without adequate synchronization.
The patch series aims to address the above problem. However, instead of introducing locks to protect per-cpu 'watchdog_ev' a different approach is taken: Invoke these functions by parking and unparking the watchdog threads (to ensure they are always called in watchdog thread context).
static struct smp_hotplug_thread watchdog_threads = {
...
.park = watchdog_disable, // calls watchdog_nmi_disable()
.unpark = watchdog_enable, // calls watchdog_nmi_enable()
};Both previously mentioned commits call these functions in a similar way and thus in principle contain some duplicate code. The patch series also avoids this duplication by providing a commonly usable mechanism.
- Patch 1/4 introduces the watchdog_{park|unpark}_threads functions that park/unpark all watchdog threads specified in 'watchdog_cpumask'. They are intended to be called inside of kernel/watchdog.c only.
- Patch 2/4 introduces the watchdog_{suspend|resume} functions which can be utilized by external callers to deactivate the hard and soft lockup detector temporarily.
- Patch 3/4 utilizes watchdog_{park|unpark}_threads to replace some code that was introduced by commit 9809b18fcf6b8d8ec4d3643677345907e6b50eca.
- Patch 4/4 utilizes watchdog_{suspend|resume} to replace some code that was introduced by commit b3738d29323344da3017a91010530cf3a58590fc.
A few corner cases should be mentioned here for completeness.
- kthread_park() of watchdog/N could hang if cpu N is already locked up. However, if watchdog is enabled the lockup will be detected anyway.
- kthread_unpark() of watchdog/N could hang if cpu N got locked up after kthread_park(). The occurrence of this scenario should be _very_ rare in practice, in particular because it is not expected that temporary deactivation will happen frequently, and if it happens at all it is expected that the duration of deactivation will be short.
Ulrich Obergfell (4):
watchdog: introduce watchdog_park_threads() and
watchdog_unpark_threads()
watchdog: introduce watchdog_suspend() and watchdog_resume()
watchdog: use park/unpark functions in update_watchdog_all_cpus()
watchdog: use suspend/resume interface in fixup_ht_bug()
arch/x86/kernel/cpu/perf_event_intel.c | 9 +- include/linux/nmi.h | 2 + include/linux/watchdog.h | 8 -- kernel/watchdog.c | 157 +++++++++++++++++++-------------- 4 files changed, 100 insertions(+), 76 deletions(-)
Signature · 1 line
1.7.11.7
PATCH 1/4 watchdog: introduce watchdog_park_threads() and watchdog_unpark_threads()
These functions are intended to be used only from inside kernel/watchdog.c to park/unpark all watchdog threads that are specified in watchdog_cpumask.
--- kernel/watchdog.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
kernel/watchdog.c +36 -0
@@ -24,6 +24,7 @@#include <asm/irq_regs.h> #include <linux/kvm_para.h> #include <linux/perf_event.h> +#include <linux/kthread.h> /* * The run state of the lockup detectors is controlled by the content of the
@@ -666,6 +667,41 @@ static struct smp_hotplug_thread watchdog_threads = { .unpark = watchdog_enable, }; +/* +* park all watchdog threads that are specified in 'watchdog_cpumask' +*/ +static int watchdog_park_threads (void ) +{ +int cpu, ret =0 ; + +get_online_cpus (); +for_each_watchdog_cpu (cpu) { + ret =kthread_park (per_cpu (softlockup_watchdog, cpu)); +if (ret) +break ; + } +if (ret) { +for_each_watchdog_cpu (cpu) +kthread_unpark (per_cpu (softlockup_watchdog, cpu)); + } +put_online_cpus (); + +return ret; +} + +/* +* unpark all watchdog threads that are specified in 'watchdog_cpumask' +*/ +static void watchdog_unpark_threads (void ) +{ +int cpu; + +get_online_cpus (); +for_each_watchdog_cpu (cpu) +kthread_unpark (per_cpu (softlockup_watchdog, cpu)); +put_online_cpus (); +} +static void restart_watchdog_hrtimer (void *info) {struct hrtimer *hrtimer =raw_cpu_ptr (&watchdog_hrtimer);
Signature · 1 line
1.7.11.7
Re: [PATCH 1/4] watchdog: introduce watchdog_park_threads() and watchdog_unpark_threads()
Quoted 2 lines · Ulrich Obergfell
These functions are intended to be used only from inside
kernel/watchdog.cto park/unpark all watchdog threads that are specified inwatchdog_cpumask.
I would suggest merging this into Patch2. It is usually better to add new functions along with their users.
Quoted 67 lines
Signed-off-by Ulrich Obergfell <[email protected]>--- kernel/watchdog.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)kernel/watchdog.c +36
-0@@ -24,6 +24,7 @@#include <asm/irq_regs.h> #include <linux/kvm_para.h> #include <linux/perf_event.h> +#include <linux/kthread.h> /* * The run state of the lockup detectors is controlled by the content of the @@ -666,6 +667,41 @@ static struct smp_hotplug_thread watchdog_threads = { .unpark = watchdog_enable, }; +/* +* park all watchdog threads that are specified in 'watchdog_cpumask' +*/ +static int watchdog_park_threads (void ) +{ +int cpu, ret =0 ; + +get_online_cpus (); +for_each_watchdog_cpu (cpu) { + ret =kthread_park (per_cpu (softlockup_watchdog, cpu)); +if (ret) +break ; + } +if (ret) { +for_each_watchdog_cpu (cpu) +kthread_unpark (per_cpu (softlockup_watchdog, cpu)); + } +put_online_cpus (); + +return ret; +} + +/* +* unpark all watchdog threads that are specified in 'watchdog_cpumask' +*/ +static void watchdog_unpark_threads (void ) +{ +int cpu; + +get_online_cpus (); +for_each_watchdog_cpu (cpu) +kthread_unpark (per_cpu (softlockup_watchdog, cpu)); +put_online_cpus (); +} +static void restart_watchdog_hrtimer (void *info) {struct hrtimer *hrtimer =raw_cpu_ptr (&watchdog_hrtimer);Signature · 7 lines
1.7.11.7 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/4] watchdog: introduce watchdog_park_threads() and watchdog_unpark_threads()
Quoted 2 lines
----- Original Message -----
From: "Michal Hocko" <[email protected]>
...
Quoted 6 lines · Ulrich Obergfell
Quoted 2 lines
These functions are intended to be used only from inside
kernel/watchdog.cto park/unpark all watchdog threads that are specified inwatchdog_cpumask.I would suggest merging this into Patch2. It is usually better to add new functions along with their users.
Michal,
watchdog_{park|unpark}_threads are called by watchdog_{suspend|resume} in Patch 2/4 and by update_watchdog_all_cpus() in Patch 3/4, so I would have to merge three patches into one, and I would end up with one large patch and one smaller patch. I think the result would be harder to read, review and understand. Thus I'd prefer to leave the patches split as they currently are.
Regards,
Uli
PATCH 2/4 watchdog: introduce watchdog_suspend() and watchdog_resume()
In reply to Ulrich ObergfellThis interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
--- include/linux/nmi.h | 2 ++ kernel/watchdog.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)
include/linux/nmi.h +2 -0
@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );#endif #ifdef CONFIG_HAVE_ACPI_APEI_NMI
kernel/watchdog.c +65 -0
101 lines not on this page · raw message
Signature · 1 line
1.7.11.7
Guenter Roeck
maintainerRe: [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume()
Quoted 16 lines · Ulrich Obergfell
This interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
Signed-off-by Ulrich Obergfell <[email protected]>--- include/linux/nmi.h | 2 ++ kernel/watchdog.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)include/linux/nmi.h +0
-0
include/linux/nmi.h extern int proc_watchdog_thresh(struct ctl_table *, int , · line 84
Quoted 5 lines
include/linux/nmi.h +2
-0@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );
How about nmi_watchdog_enable() and nmi_watchdog_disable() to avoid confusion with the watchdog subsystem ?
Thanks,
Guenter
Re: [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume()
Quoted 17 lines
----- Original Message -----
From: "Guenter Roeck" <[email protected]>
...
Subject: Re: [PATCH 2/4] watchdog: introducewatchdog_suspend()andwatchdog_resume()Quoted 11 lines · Ulrich Obergfell
This interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
Signed-off-by Ulrich Obergfell <[email protected]>--- include/linux/nmi.h | 2 ++ kernel/watchdog.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)
Quoted 13 lines
include/linux/nmi.h extern int proc_watchdog_thresh(struct ctl_table *, int , · line 84
Quoted 10 lines
include/linux/nmi.h +2
-0@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );How about
nmi_watchdog_enable()andnmi_watchdog_disable()to avoid confusion with the watchdog subsystem ?
Guenter,
Good point. However, I would like to avoid the 'nmi_' prefix in the function names as it could be misleading. watchdog_{suspend|resume} affect both -the hard and soft lockup detector- so I think function names like
lockup_detector_suspend() instead of watchdog_suspend()
lockup_detector_resume() instead of watchdog_resume()
would summarize better what these functions are intended to be used for. The above names would also be consistent with the existing name
lockup_detector_init()
Many Thanks,
Uli
Guenter Roeck
maintainerRe: [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume()
Quoted 38 lines · Ulrich Obergfell
Quoted 17 lines
----- Original Message -----
From: "Guenter Roeck" <[email protected]>
...
Subject: Re: [PATCH 2/4] watchdog: introducewatchdog_suspend()andwatchdog_resume()Quoted 11 lines · Ulrich Obergfell
This interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
Signed-off-by Ulrich Obergfell <[email protected]>--- include/linux/nmi.h | 2 ++ kernel/watchdog.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)Quoted 13 lines
include/linux/nmi.h extern int proc_watchdog_thresh(struct ctl_table *, int , · line 84
Quoted 10 lines
include/linux/nmi.h +2
-0@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );How about
nmi_watchdog_enable()andnmi_watchdog_disable()to avoid confusion with the watchdog subsystem ?Guenter,
Good point. However, I would like to avoid the '
nmi_' prefix in the function names as it could be misleading. watchdog_{suspend|resume} affect both -the hard and soft lockup detector- so I think function names like
Quoted 8 lines
lockup_detector_suspend()instead ofwatchdog_suspend()
lockup_detector_resume()instead ofwatchdog_resume()would summarize better what these functions are intended to be used for. The above names would also be consistent with the existing name
lockup_detector_init()
Makes sense.
Thanks,
Guenter
Re: [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume()
In reply to Ulrich ObergfellQuoted 23 lines · On Sat, 1 Aug 2015 14:49:23 +0200 Ulrich Obergfell <[email protected]> wrote:
This interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
...
include/linux/nmi.h +2
-0@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );#endif #ifdef CONFIG_HAVE_ACPI_APEI_NMI kernel/watchdog.c +0
-0
kernel/watchdog.c unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask); · line 70
Quoted 4 lines
kernel/watchdog.c +1
-0@@ -67,6 +67,7 @@ unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask);#define for_each_watchdog_cpu (cpu) \ for_each_cpu_and((cpu), cpu_online_mask, &watchdog_cpumask) +static int __read_mostly watchdog_suspended =0 ;
With my compiler the "= 0" increases the size of watchdog.o data. For some reason by 16 bytes(!).
kernel/watchdog.c unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask); · line 72
Quoted 2 lines
kernel/watchdog.c +0
-0static int __read_mostly watchdog_running;static u64 __read_mostly sample_period;
The relationship between watchdog_running and watchdog_suspended hurts my brain a bit. It appears that all watchdog_running transitions happen under watchdog_proc_mutex so I don't think it's racy, but I wonder if things would be simpler if these were folded into a single up/down counter.
Re: [PATCH 2/4] watchdog: introduce watchdog_suspend() and watchdog_resume()
Quoted 31 lines
----- Original Message -----
From: "Andrew Morton" <[email protected]>
...Quoted 26 lines · On Sat, 1 Aug 2015 14:49:23 +0200 Ulrich Obergfell <[email protected]> wrote:
This interface can be utilized to deactivate the hard and soft lockup detector temporarily. Callers are expected to minimize the duration of deactivation. Multiple deactivations are allowed to occur in parallel but should be rare in practice.
...
include/linux/nmi.h +2
-0@@ -80,6 +80,8 @@ extern int proc_watchdog_thresh(struct ctl_table *, int , void __user *, size_t *, loff_t *);extern int proc_watchdog_cpumask (struct ctl_table *,int ,void __user *,size_t *,loff_t *); +extern int watchdog_suspend (void ); +extern void watchdog_resume (void );#endif #ifdef CONFIG_HAVE_ACPI_APEI_NMI kernel/watchdog.c +0
-0@@ -67,6 +67,7 @@ unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask);#define for_each_watchdog_cpu (cpu) \ for_each_cpu_and((cpu), cpu_online_mask, &watchdog_cpumask)
Quoted 4 lines
kernel/watchdog.c unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask); · line 70
Quoted 1 line
kernel/watchdog.c +1
-0+static int __read_mostly watchdog_suspended =0 ;With my compiler the "= 0" increases the size of watchdog.o data. For some reason by 16 bytes(!).
I see that you already fixed this. Many Thanks.
Quoted 8 lines
kernel/watchdog.c unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask); · line 72
Quoted 2 lines
kernel/watchdog.c +0
-0static int __read_mostly watchdog_running;static u64 __read_mostly sample_period;The relationship between
watchdog_runningandwatchdog_suspendedhurts my brain a bit. It appears that allwatchdog_runningtransitions happen underwatchdog_proc_mutexso I don't think it's racy, but I wonder if things would be simpler if these were folded into a single up/down counter.
The 'watchdog_running' variable indicates whether watchdog threads exist. [Whether they have been launched via smpboot_register_percpu_thread().]
The 'watchdog_suspended' variable indicates whether existing threads are currently parked, and whether multiple requests to suspend the watchdog have been made by different callers.
I think folding them into one variable would not improve the readability of the code, because instead of two variables with distinct semantics we would then have one variable with multiple semantics (which I think would also make code more complex). If we would want to improve the readability I'd prefer to either just add a comment block to explain the semantics of the variables or to rename them -for example- like:
watchdog_running to watchdog_threads_exist
watchdog_suspended to watchdog_suspend_count
Please let me know if you would want any changes in this regard.
I also received these suggestions:
- rename the watchdog_{suspend|resume} functions as discussed in
http://marc.info/?l=linux-kernel&m=143844050610220&w=2
- use pr_debug() instead of pr_info() as discussed in
http://marc.info/?l=linux-kernel&m=143869949229461&w=2
Please let me know if you want me to post a 'version 2' of the patch set or if you want me to post these changes as separate follow-up patches.
Regards,
Uli
PATCH 3/4 watchdog: use park/unpark functions in update_watchdog_all_cpus()
In reply to Ulrich ObergfellRemove update_watchdog() and restart_watchdog_hrtimer() since these functions are no longer needed. Changes of parameters such as the sample period are honored at the time when the watchdog threads are being unparked.
--- kernel/watchdog.c | 40 ++-------------------------------------- 1 file changed, 2 insertions(+), 38 deletions(-)
kernel/watchdog.c +2 -38
@@ -747,46 +747,10 @@ void watchdog_resume(void)mutex_unlock (&watchdog_proc_mutex); } -static void restart_watchdog_hrtimer (void *info) -{ -struct hrtimer *hrtimer =raw_cpu_ptr (&watchdog_hrtimer); -int ret; - -/* -* No need to cancel and restart hrtimer if it is currently executing -* because it will reprogram itself with the new period now. -* We should never see it unqueued here because we are running per-cpu -* with interrupts disabled. -*/ - ret =hrtimer_try_to_cancel (hrtimer); -if (ret ==1 ) -hrtimer_start (hrtimer,ns_to_ktime (sample_period), -HRTIMER_MODE_REL_PINNED ); -} - -static void update_watchdog (int cpu) -{ -/* -* Make sure that perf event counter will adopt to a new -* sampling period. Updating the sampling period directly would -* be much nicer but we do not have an API for that now so -* let's use a big hammer. -* Hrtimer will adopt the new period on the next tick but this -* might be late already so we have to restart the timer as well. -*/ -watchdog_nmi_disable (cpu); -smp_call_function_single (cpu, restart_watchdog_hrtimer,NULL ,1 ); -watchdog_nmi_enable (cpu); -} -static void update_watchdog_all_cpus (void ) { -int cpu; - -get_online_cpus (); -for_each_watchdog_cpu (cpu) -update_watchdog (cpu); -put_online_cpus (); +watchdog_park_threads (); +watchdog_unpark_threads (); }static int watchdog_enable_all_cpus (void )
Signature · 1 line
1.7.11.7
PATCH 4/4 watchdog: use suspend/resume interface in fixup_ht_bug()
In reply to Ulrich ObergfellRemove watchdog_nmi_disable_all() and watchdog_nmi_enable_all() since these functions are no longer needed. If a subsystem has a need to deactivate the watchdog temporarily, it should utilize the watchdog_suspend() and watchdog_resume() functions.
--- arch/x86/kernel/cpu/perf_event_intel.c | 9 +++++--- include/linux/watchdog.h | 8 ------- kernel/watchdog.c | 38 ---------------------------------- 3 files changed, 6 insertions(+), 49 deletions(-)
arch/x86/kernel/cpu/perf_event_intel.c +6 -3
@@ -12,7 +12,7 @@#include <linux/init.h> #include <linux/slab.h> #include <linux/export.h> -#include <linux/watchdog.h> +#include <linux/nmi.h> #include <asm/cpufeature.h> #include <asm/hardirq.h>
@@ -3368,7 +3368,10 @@ static __init int fixup_ht_bug(void)return 0 ; } -watchdog_nmi_disable_all (); +if (watchdog_suspend () !=0 ) { +pr_info ("failed to disable PMU erratum BJ122, BV98, HSD29 workaround\n" ); +return 0 ; + } x86_pmu.flags &= ~(PMU_FL_EXCL_CNTRS |PMU_FL_EXCL_ENABLED );
@@ -3376,7 +3379,7 @@ static __init int fixup_ht_bug(void) x86_pmu.commit_scheduling =NULL ; x86_pmu.stop_scheduling =NULL ; -watchdog_nmi_enable_all (); +watchdog_resume ();get_online_cpus ();
include/linux/watchdog.h +0 -8
@@ -140,12 +140,4 @@ extern int watchdog_init_timeout(struct watchdog_device *wdd,extern int watchdog_register_device (struct watchdog_device *);extern void watchdog_unregister_device (struct watchdog_device *); -#ifdef CONFIG_HARDLOCKUP_DETECTOR -void watchdog_nmi_disable_all (void ); -void watchdog_nmi_enable_all (void ); -#else -static inline void watchdog_nmi_disable_all (void ) {} -static inline void watchdog_nmi_enable_all (void ) {} -#endif -#endif /* ifndef _LINUX_WATCHDOG_H */
kernel/watchdog.c +0 -38
@@ -614,47 +614,9 @@ static void watchdog_nmi_disable(unsigned int cpu) cpu0_err = 0; } } - -void watchdog_nmi_enable_all (void ) -{ -int cpu; - -mutex_lock (&watchdog_proc_mutex); - -if (!(watchdog_enabled &NMI_WATCHDOG_ENABLED )) - goto unlock; - -get_online_cpus (); -for_each_watchdog_cpu (cpu) -watchdog_nmi_enable (cpu); -put_online_cpus (); - -unlock: -mutex_unlock (&watchdog_proc_mutex); -} - -void watchdog_nmi_disable_all (void ) -{ -int cpu; - -mutex_lock (&watchdog_proc_mutex); - -if (!watchdog_running) - goto unlock; - -get_online_cpus (); -for_each_watchdog_cpu (cpu) -watchdog_nmi_disable (cpu); -put_online_cpus (); - -unlock: -mutex_unlock (&watchdog_proc_mutex); -}#else static int watchdog_nmi_enable (unsigned int cpu) {return 0 ; }static void watchdog_nmi_disable (unsigned int cpu) {return ; } -void watchdog_nmi_enable_all (void ) {} -void watchdog_nmi_disable_all (void ) {}#endif /* CONFIG_HARDLOCKUP_DETECTOR */ static struct smp_hotplug_thread watchdog_threads = {
Signature · 1 line
1.7.11.7
Re: [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug()
On Sat 01-08-15 14:49:25, Ulrich Obergfell wrote:
[...]
arch/x86/kernel/cpu/perf_event_intel.c static __init int fixup_ht_bug(void) · line 3374
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel.c +4
-1@@ -3368,7 +3368,10 @@ static __init int fixup_ht_bug(void)return 0 ; } -watchdog_nmi_disable_all (); +if (watchdog_suspend () !=0 ) { +pr_info ("failed to disable PMU erratum BJ122, BV98, HSD29 workaround\n" ); +return 0 ; + }
Is this really worth reporting to the log? What is an admin supposed to do about it?
<looking into the code>
Ok, so kthread_park fails only when the kernel thread has already exited. Can this ever happen during this call path?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug()
Quoted 6 lines · Michal Hocko
On Sat 01-08-15 14:49:25, Ulrich Obergfell wrote:
[...]Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -3368,7 +3368,10 @@ static __init int fixup_ht_bug(void)return 0 ; }
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c static __init int fixup_ht_bug(void) · line 3374
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel.c +4
-1-watchdog_nmi_disable_all (); +if (watchdog_suspend () !=0 ) { +pr_info ("failed to disable PMU erratum BJ122, BV98, HSD29 workaround\n" ); +return 0 ; + }Is this really worth reporting to the log? What is an admin supposed to do about it?
I think it was more for developers to aid in debugging a strange behaviour of the performance counters.
Quoted 3 lines
<looking into the code>
Ok, sokthread_parkfails only when the kernel thread has already exited. Can this ever happen during this call path?
It might be overkill, but it is just a harmless informational failure message.
Cheers,
Don
Re: [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug()
Quoted 16 lines · Don Zickus
Quoted 6 lines · Michal Hocko
On Sat 01-08-15 14:49:25, Ulrich Obergfell wrote:
[...]Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -3368,7 +3368,10 @@ static __init int fixup_ht_bug(void)return 0 ; }Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c static __init int fixup_ht_bug(void) · line 3374
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel.c +4
-1-watchdog_nmi_disable_all (); +if (watchdog_suspend () !=0 ) { +pr_info ("failed to disable PMU erratum BJ122, BV98, HSD29 workaround\n" ); +return 0 ; + }Is this really worth reporting to the log? What is an admin supposed to do about it?
Quoted 2 lines
I think it was more for developers to aid in debugging a strange behaviour of the performance counters.
pr_debug then?
Quoted 6 lines
Quoted 3 lines
<looking into the code>
Ok, sokthread_parkfails only when the kernel thread has already exited. Can this ever happen during this call path?It might be overkill, but it is just a harmless informational failure message.
Maybe we have way too many of those harmless informational failure admins scratch their heads about...
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug()
In reply to Don ZickusQuoted 2 lines
----- Original Message -----
From: "Don Zickus" <[email protected]>
...
Quoted 19 lines · Michal Hocko
Quoted 6 lines · Ulrich Obergfell
[...]
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel.c +0
-0@@ -3368,7 +3368,10 @@ static __init int fixup_ht_bug(void)return 0 ; }Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel.c static __init int fixup_ht_bug(void) · line 3374
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel.c +4
-1-watchdog_nmi_disable_all (); +if (watchdog_suspend () !=0 ) { +pr_info ("failed to disable PMU erratum BJ122, BV98, HSD29 workaround\n" ); +return 0 ; + }Is this really worth reporting to the log? What is an admin supposed to do about it?
I think it was more for developers to aid in debugging a strange behaviour of the performance counters.
Quoted 6 lines
Quoted 3 lines
<looking into the code>
Ok, sokthread_parkfails only when the kernel thread has already exited. Can this ever happen during this call path?It might be overkill, but it is just a harmless informational failure message.
Don, Michal,
the module prints a message if the workaround is enabled and if the workaround is disabled. Hence, I think we should keep the messages consistent and thus inform the user also if we fail to disable the workaround. Even though at the moment this seems to be an unlikely failure case, I agree with Don that the message could be useful in debugging.
Regards,
Uli
Re: [PATCH 4/4] watchdog: use suspend/resume interface in fixup_ht_bug()
In reply to Ulrich ObergfellQuoted 5 lines · On Sat, 1 Aug 2015 14:49:25 +0200 Ulrich Obergfell <[email protected]> wrote:
Remove
watchdog_nmi_disable_all()andwatchdog_nmi_enable_all()since these functions are no longer needed. If a subsystem has a need to deactivate the watchdog temporarily, it should utilize thewatchdog_suspend()andwatchdog_resume()functions.
With x86_64 allnoconfig I'm getting
arch/x86/kernel/cpu/perf_event_intel.c: In function 'fixup_ht_bug':arch/x86/kernel/cpu/perf_event_intel.c:3371: error: implicit declaration of function 'watchdog_suspend' arch/x86/kernel/cpu/perf_event_intel.c:3382: error: implicit declaration of function 'watchdog_resume'
I had to mangle your patches fairly heavily to make them fit against other pending changes. Specifically
http://ozlabs.org/~akpm/mmots/broken-out/watchdog-move-nmi-function-header-declarations-from-watchdogh-to-nmih.patch
and
http://ozlabs.org/~akpm/mmots/broken-out/watchdog-move-nmi-function-header-declarations-from-watchdogh-to-nmih-v2.patch.
But I don't *think* I caused this... I'm testing this:
include/linux/nmi.h renamed from include/linux/nmi.h~watchdog-use-suspend-resume-interface-in-fixup_ht_bug-fix +9 -0
@@ -80,6 +80,15 @@ extern int proc_watchdog_cpumask(struct void __user *, size_t *, loff_t *);extern int watchdog_suspend (void );extern void watchdog_resume (void ); +#else +static inline int watchdog_suspend (void ) +{ +return 0 ; +} + +static inline void watchdog_resume (void ) +{ +}#endif #ifdef CONFIG_HAVE_ACPI_APEI_NMI
Quoted 67 lines · Ulrich Obergfell
Originally
watchdog_nmi_enable(cpu)andwatchdog_nmi_disable(cpu)were only called in watchdog thread context. However, the following commits utilize these functions outside of watchdog thread context too.commit
9809b18fcf6b8d8ec4d3643677345907e6b50eca
Author: Michal Hocko <[email protected]>
Date: Tue Sep 24 15:27:30 2013 -0700watchdog: update
watchdog_threshproperlycommit
b3738d29323344da3017a91010530cf3a58590fc
Author: Stephane Eranian <[email protected]>
Date: Mon Nov 17 20:07:03 2014 +0100watchdog: Add watchdog enable/disable all functions
Hence, it is now possible that these functions execute concurrently with the same 'cpu' argument. This concurrency is problematic because per-cpu '
watchdog_ev' can be accessed/modified without adequate synchronization.The patch series aims to address the above problem. However, instead of introducing locks to protect per-cpu '
watchdog_ev' a different approach is taken: Invoke these functions by parking and unparking the watchdog threads (to ensure they are always called in watchdog thread context).static struct smp_hotplug_thread watchdog_threads = { ... .park = watchdog_disable, // calls watchdog_nmi_disable() .unpark = watchdog_enable, // calls watchdog_nmi_enable() };Both previously mentioned commits call these functions in a similar way and thus in principle contain some duplicate code. The patch series also avoids this duplication by providing a commonly usable mechanism.
- Patch 1/4 introduces the watchdog_{park|unpark}_threads functions that park/unpark all watchdog threads specified in '
watchdog_cpumask'. They are intended to be called inside ofkernel/watchdog.conly.- Patch 2/4 introduces the watchdog_{suspend|resume} functions which can be utilized by external callers to deactivate the hard and soft lockup detector temporarily.
- Patch 3/4 utilizes watchdog_{park|unpark}_threads to replace some code that was introduced by commit
9809b18fcf6b8d8ec4d3643677345907e6b50eca.- Patch 4/4 utilizes watchdog_{suspend|resume} to replace some code that was introduced by commit
b3738d29323344da3017a91010530cf3a58590fc.A few corner cases should be mentioned here for completeness.
-
kthread_park()of watchdog/N could hang if cpu N is already locked up. However, if watchdog is enabled the lockup will be detected anyway.-
kthread_unpark()of watchdog/N could hang if cpu N got locked up afterkthread_park(). The occurrence of this scenario should be _very_ rare in practice, in particular because it is not expected that temporary deactivation will happen frequently, and if it happens at all it is expected that the duration of deactivation will be short.Ulrich Obergfell (4):
watchdog: introducewatchdog_park_threads()and
watchdog_unpark_threads()
watchdog: introducewatchdog_suspend()andwatchdog_resume()
watchdog: use park/unpark functions inupdate_watchdog_all_cpus()
watchdog: use suspend/resume interface infixup_ht_bug()
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel.c | 9 +- include/linux/nmi.h | 2 + include/linux/watchdog.h | 8 -- kernel/watchdog.c | 157 +++++++++++++++++++-------------- 4 files changed, 100 insertions(+), 76 deletions(-)
The whole patch series looks good to me.
Quoted 60 lines · Ulrich Obergfell
Originally
watchdog_nmi_enable(cpu)andwatchdog_nmi_disable(cpu)were only called in watchdog thread context. However, the following commits utilize these functions outside of watchdog thread context too.commit
9809b18fcf6b8d8ec4d3643677345907e6b50eca
Author: Michal Hocko <[email protected]>
Date: Tue Sep 24 15:27:30 2013 -0700watchdog: update
watchdog_threshproperlycommit
b3738d29323344da3017a91010530cf3a58590fc
Author: Stephane Eranian <[email protected]>
Date: Mon Nov 17 20:07:03 2014 +0100watchdog: Add watchdog enable/disable all functions
Hence, it is now possible that these functions execute concurrently with the same 'cpu' argument. This concurrency is problematic because per-cpu '
watchdog_ev' can be accessed/modified without adequate synchronization.The patch series aims to address the above problem. However, instead of introducing locks to protect per-cpu '
watchdog_ev' a different approach is taken: Invoke these functions by parking and unparking the watchdog threads (to ensure they are always called in watchdog thread context).static struct smp_hotplug_thread watchdog_threads = { ... .park = watchdog_disable, // calls watchdog_nmi_disable() .unpark = watchdog_enable, // calls watchdog_nmi_enable() };Both previously mentioned commits call these functions in a similar way and thus in principle contain some duplicate code. The patch series also avoids this duplication by providing a commonly usable mechanism.
- Patch 1/4 introduces the watchdog_{park|unpark}_threads functions that park/unpark all watchdog threads specified in '
watchdog_cpumask'. They are intended to be called inside ofkernel/watchdog.conly.- Patch 2/4 introduces the watchdog_{suspend|resume} functions which can be utilized by external callers to deactivate the hard and soft lockup detector temporarily.
- Patch 3/4 utilizes watchdog_{park|unpark}_threads to replace some code that was introduced by commit
9809b18fcf6b8d8ec4d3643677345907e6b50eca.- Patch 4/4 utilizes watchdog_{suspend|resume} to replace some code that was introduced by commit
b3738d29323344da3017a91010530cf3a58590fc.A few corner cases should be mentioned here for completeness.
-
kthread_park()of watchdog/N could hang if cpu N is already locked up. However, if watchdog is enabled the lockup will be detected anyway.-
kthread_unpark()of watchdog/N could hang if cpu N got locked up afterkthread_park(). The occurrence of this scenario should be _very_ rare in practice, in particular because it is not expected that temporary deactivation will happen frequently, and if it happens at all it is expected that the duration of deactivation will be short.
Quoted 6 lines
Ulrich Obergfell (4):
watchdog: introducewatchdog_park_threads()and
watchdog_unpark_threads()
watchdog: introducewatchdog_suspend()andwatchdog_resume()
watchdog: use park/unpark functions inupdate_watchdog_all_cpus()
watchdog: use suspend/resume interface infixup_ht_bug()
tested the same way as for I did for my other patch for this issue:
http://marc.info/?l=linux-kernel&m=143834383400803&w=2
works nicely ;-)
thanks,
jirka