Skip to contentSkip to conversation

watchdog: softdog: fire watchdog even if softirqs do not get to run

PATCH Superseded Niklas Cassel 4 participants 11 messages Feb 17, 2017 – Feb 27, 2017 Watchdog device drivers

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 watchdog: softdog: fire watchdog even if softirqs do not get to run

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

---
 drivers/watchdog/softdog.c | 40 ++++++++++++++++++++++++----------------
 1 file changed, 24 insertions(+), 16 deletions(-)
drivers/watchdog/softdog.c +24 -16
index c7bdc986dca1..0f67cd068465 100644
@@ -21,13 +21,12 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/hrtimer.h>
 #include <linux/init.h>
-#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/reboot.h>
-#include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/watchdog.h>
 
@@ -54,7 +53,10 @@ module_param(soft_panic, int, 0);
 MODULE_PARM_DESC(soft_panic,
 	"Softdog action, set to 1 to panic, 0 to reboot (default=0)");
 
-static void softdog_fire(unsigned long data)
+static struct hrtimer softdog_ticktock;
+static struct hrtimer softdog_preticktock;
+
+static enum hrtimer_restart softdog_fire(struct hrtimer *timer)
 {
 	module_put(THIS_MODULE);
 	if (soft_noboot) {
@@ -67,41 +69,42 @@ static void softdog_fire(unsigned long data)
 		emergency_restart();
 		pr_crit("Reboot didn't ?????\n");
 	}
-}
 
-static struct timer_list softdog_ticktock =
-		TIMER_INITIALIZER(softdog_fire, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static struct watchdog_device softdog_dev;
 
-static void softdog_pretimeout(unsigned long data)
+static enum hrtimer_restart softdog_pretimeout(struct hrtimer *timer)
 {
 	watchdog_notify_pretimeout(&softdog_dev);
-}
 
-static struct timer_list softdog_preticktock =
-		TIMER_INITIALIZER(softdog_pretimeout, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static int softdog_ping(struct watchdog_device *w)
 {
-	if (!mod_timer(&softdog_ticktock, jiffies + (w->timeout * HZ)))
+	if (!hrtimer_active(&softdog_ticktock))
 		__module_get(THIS_MODULE);
+	hrtimer_start(&softdog_ticktock, ktime_set(w->timeout, 0),
+		      HRTIMER_MODE_REL);
 
 	if (w->pretimeout)
-		mod_timer(&softdog_preticktock, jiffies +
-			  (w->timeout - w->pretimeout) * HZ);
+		hrtimer_start(&softdog_preticktock,
+			      ktime_set(w->timeout - w->pretimeout, 0),
+			      HRTIMER_MODE_REL);
 	else
-		del_timer(&softdog_preticktock);
+		hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
 
 static int softdog_stop(struct watchdog_device *w)
 {
-	if (del_timer(&softdog_ticktock))
+	if (hrtimer_cancel(&softdog_ticktock))
 		module_put(THIS_MODULE);
 
-	del_timer(&softdog_preticktock);
+	hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
@@ -134,6 +137,11 @@ static int __init softdog_init(void)
 	watchdog_set_nowayout(&softdog_dev, nowayout);
 	watchdog_stop_on_reboot(&softdog_dev);
 
+	hrtimer_init(&softdog_ticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_ticktock.function = softdog_fire;
+	hrtimer_init(&softdog_preticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_preticktock.function = softdog_pretimeout;
+
 	ret = watchdog_register_device(&softdog_dev);
 	if (ret)
 		return ret;
Signature · 1 line
2.1.4
maintainer
#1 lore raw
Cc Wolfram for input.
Quoted 10 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

Quoted 8 lines

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

This makes the driver dependent on HIGH_RES_TIMERS, which is not available on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

Is this driver the only one with this problem, or is anything using timers affected ?

Thanks,
Guenter

Quoted 105 lines
---
 drivers/watchdog/softdog.c | 40 ++++++++++++++++++++++++----------------
 1 file changed, 24 insertions(+), 16 deletions(-)
drivers/watchdog/softdog.c +24 -16
index c7bdc986dca1..0f67cd068465 100644
@@ -21,13 +21,12 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/hrtimer.h>
 #include <linux/init.h>
-#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/reboot.h>
-#include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/watchdog.h>
 
@@ -54,7 +53,10 @@ module_param(soft_panic, int, 0);
 MODULE_PARM_DESC(soft_panic,
 	"Softdog action, set to 1 to panic, 0 to reboot (default=0)");
 
-static void softdog_fire(unsigned long data)
+static struct hrtimer softdog_ticktock;
+static struct hrtimer softdog_preticktock;
+
+static enum hrtimer_restart softdog_fire(struct hrtimer *timer)
 {
 	module_put(THIS_MODULE);
 	if (soft_noboot) {
@@ -67,41 +69,42 @@ static void softdog_fire(unsigned long data)
 		emergency_restart();
 		pr_crit("Reboot didn't ?????\n");
 	}
-}
 
-static struct timer_list softdog_ticktock =
-		TIMER_INITIALIZER(softdog_fire, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static struct watchdog_device softdog_dev;
 
-static void softdog_pretimeout(unsigned long data)
+static enum hrtimer_restart softdog_pretimeout(struct hrtimer *timer)
 {
 	watchdog_notify_pretimeout(&softdog_dev);
-}
 
-static struct timer_list softdog_preticktock =
-		TIMER_INITIALIZER(softdog_pretimeout, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static int softdog_ping(struct watchdog_device *w)
 {
-	if (!mod_timer(&softdog_ticktock, jiffies + (w->timeout * HZ)))
+	if (!hrtimer_active(&softdog_ticktock))
 		__module_get(THIS_MODULE);
+	hrtimer_start(&softdog_ticktock, ktime_set(w->timeout, 0),
+		      HRTIMER_MODE_REL);
 
 	if (w->pretimeout)
-		mod_timer(&softdog_preticktock, jiffies +
-			  (w->timeout - w->pretimeout) * HZ);
+		hrtimer_start(&softdog_preticktock,
+			      ktime_set(w->timeout - w->pretimeout, 0),
+			      HRTIMER_MODE_REL);
 	else
-		del_timer(&softdog_preticktock);
+		hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
 
 static int softdog_stop(struct watchdog_device *w)
 {
-	if (del_timer(&softdog_ticktock))
+	if (hrtimer_cancel(&softdog_ticktock))
 		module_put(THIS_MODULE);
 
-	del_timer(&softdog_preticktock);
+	hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
@@ -134,6 +137,11 @@ static int __init softdog_init(void)
 	watchdog_set_nowayout(&softdog_dev, nowayout);
 	watchdog_stop_on_reboot(&softdog_dev);
 
+	hrtimer_init(&softdog_ticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_ticktock.function = softdog_fire;
+	hrtimer_init(&softdog_preticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_preticktock.function = softdog_pretimeout;
+
 	ret = watchdog_register_device(&softdog_dev);
 	if (ret)
 		return ret;
#2 lore raw
Quoted 18 lines · Guenter Roeck
Cc Wolfram for input.
Quoted 15 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

Quoted 8 lines
Quoted 3 lines

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

This makes the driver dependent on HIGH_RES_TIMERS, which is not available on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

It does no such thing; the hrtimer interface is always available.

maintainer
#3 lore raw
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Guenter Roeck
Cc Wolfram for input.
Quoted 18 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Quoted 7 lines
Quoted 4 lines

This makes the driver dependent on HIGH_RES_TIMERS, which is not available on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

It does no such thing; the hrtimer interface is always available.

That is good to hear. Thanks for the correction.

Guenter

#4 lore raw
In reply to Guenter Roeck
Quoted 22 lines · Guenter Roeck
Cc Wolfram for input.
Quoted 10 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

Quoted 8 lines

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Quoted 6 lines

This makes the driver dependent on HIGH_RES_TIMERS, which is not available on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

Is this driver the only one with this problem, or is anything using timers affected ?

Anything using timers is affected.
The timers will still get incremented, but the code checking for timer expiration is run from a softirq, which in this case never gets to run, so the timers will never expire.

Before 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), softirqs were never deferred, so they always got to run when exiting an irq.

So previously with a user space process using all the CPU, like:
chrt -r 99 sh -c "while :; do :; done"
the softdog would still fire.

If we ask the system to run something all the time, and the system does that, I don't think we can blame the system. It is however important that the watchdog can still detect and fire when this happens. Other drivers, not so much.

I guess another solution would be to modify the if-statements in kernel/softirq.c to sometimes do the softirq directly, even if ksoftirqd is in state TASK_RUNNING, if we also meet some other condition. However, do we want to add that extra complexity?
Perhaps someone with more softirq/scheduler knowledge can give some input on this.

#5 lore raw
Quoted 38 lines · Niklas Cassel
Quoted 16 lines · Guenter Roeck
Cc Wolfram for input.
Quoted 13 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while

ksoftirqd,

Quoted 1 line
Quoted 1 line

which is supposed to handle the deferred softirqs, will never get to

run.

Quoted 6 lines
Quoted 4 lines

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

This makes the driver dependent on HIGH_RES_TIMERS, which is not

available

Quoted 5 lines

on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

Is this driver the only one with this problem, or is anything using timers affected ?

Anything using timers is affected.
The timers will still get incremented, but the code checking for timer expiration is run from a softirq, which in this case never gets to run, so the timers will never expire.

Quoted 8 lines

Before 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), softirqs were never deferred, so they always got to run when exiting an irq.

So previously with a user space process using all the CPU, like:
chrt -r 99 sh -c "while :; do :; done"
the softdog would still fire.

So the question is :

If some RT process does an infinite loop, should we care about system being functional ?

Looks like the OS is now WAI (Working As Intended)

Quoted 12 lines

If we ask the system to run something all the time, and the system does that, I don't think we can blame the system. It is however important that the watchdog can still detect and fire when this happens. Other drivers, not so much.

I guess another solution would be to modify the if-statements in kernel/softirq.c to sometimes do the softirq directly, even if ksoftirqd is in state TASK_RUNNING, if we also meet some other condition. However, do we want to add that extra complexity?
Perhaps someone with more softirq/scheduler knowledge can give some input on this.

maintainer
#6 lore raw
In reply to Niklas Cassel
Quoted 47 lines · Niklas Cassel
Quoted 22 lines · Guenter Roeck
Cc Wolfram for input.
Quoted 10 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

Quoted 8 lines

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Quoted 6 lines

This makes the driver dependent on HIGH_RES_TIMERS, which is not available on all architectures. Before adding that restriction, I would like to see some discussion if this is the only feasible solution.

Is this driver the only one with this problem, or is anything using timers affected ?

Anything using timers is affected.
The timers will still get incremented, but the code checking for timer expiration is run from a softirq, which in this case never gets to run, so the timers will never expire.

Before 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), softirqs were never deferred, so they always got to run when exiting an irq.

So previously with a user space process using all the CPU, like:
chrt -r 99 sh -c "while :; do :; done"
the softdog would still fire.

If we ask the system to run something all the time, and the system does that, I don't think we can blame the system. It is however important that the watchdog can still detect and fire when this happens. Other drivers, not so much.

Quoted 4 lines

I guess another solution would be to modify the if-statements in kernel/softirq.c to sometimes do the softirq directly, even if ksoftirqd is in state TASK_RUNNING, if we also meet some other condition. However, do we want to add that extra complexity?

No, in that respect using hrtimers is ok. I was more concerned about other users of timers. But, as you say, I guess that is considered ok.

Guenter

maintainer Reviewed-by on v1 1/1
#7 lore raw
In reply to Niklas Cassel
Quoted 15 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

Quoted 4 lines

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Quoted 104 lines
---
 drivers/watchdog/softdog.c | 40 ++++++++++++++++++++++++----------------
 1 file changed, 24 insertions(+), 16 deletions(-)
drivers/watchdog/softdog.c +24 -16
index c7bdc986dca1..0f67cd068465 100644
@@ -21,13 +21,12 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/hrtimer.h>
 #include <linux/init.h>
-#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/reboot.h>
-#include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/watchdog.h>
 
@@ -54,7 +53,10 @@ module_param(soft_panic, int, 0);
 MODULE_PARM_DESC(soft_panic,
 	"Softdog action, set to 1 to panic, 0 to reboot (default=0)");
 
-static void softdog_fire(unsigned long data)
+static struct hrtimer softdog_ticktock;
+static struct hrtimer softdog_preticktock;
+
+static enum hrtimer_restart softdog_fire(struct hrtimer *timer)
 {
 	module_put(THIS_MODULE);
 	if (soft_noboot) {
@@ -67,41 +69,42 @@ static void softdog_fire(unsigned long data)
 		emergency_restart();
 		pr_crit("Reboot didn't ?????\n");
 	}
-}
 
-static struct timer_list softdog_ticktock =
-		TIMER_INITIALIZER(softdog_fire, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static struct watchdog_device softdog_dev;
 
-static void softdog_pretimeout(unsigned long data)
+static enum hrtimer_restart softdog_pretimeout(struct hrtimer *timer)
 {
 	watchdog_notify_pretimeout(&softdog_dev);
-}
 
-static struct timer_list softdog_preticktock =
-		TIMER_INITIALIZER(softdog_pretimeout, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static int softdog_ping(struct watchdog_device *w)
 {
-	if (!mod_timer(&softdog_ticktock, jiffies + (w->timeout * HZ)))
+	if (!hrtimer_active(&softdog_ticktock))
 		__module_get(THIS_MODULE);
+	hrtimer_start(&softdog_ticktock, ktime_set(w->timeout, 0),
+		      HRTIMER_MODE_REL);
 
 	if (w->pretimeout)
-		mod_timer(&softdog_preticktock, jiffies +
-			  (w->timeout - w->pretimeout) * HZ);
+		hrtimer_start(&softdog_preticktock,
+			      ktime_set(w->timeout - w->pretimeout, 0),
+			      HRTIMER_MODE_REL);
 	else
-		del_timer(&softdog_preticktock);
+		hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
 
 static int softdog_stop(struct watchdog_device *w)
 {
-	if (del_timer(&softdog_ticktock))
+	if (hrtimer_cancel(&softdog_ticktock))
 		module_put(THIS_MODULE);
 
-	del_timer(&softdog_preticktock);
+	hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
@@ -134,6 +137,11 @@ static int __init softdog_init(void)
 	watchdog_set_nowayout(&softdog_dev, nowayout);
 	watchdog_stop_on_reboot(&softdog_dev);
 
+	hrtimer_init(&softdog_ticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_ticktock.function = softdog_fire;
+	hrtimer_init(&softdog_preticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_preticktock.function = softdog_pretimeout;
+
 	ret = watchdog_register_device(&softdog_dev);
 	if (ret)
 		return ret;
maintainer
#8 lore raw
In reply to Niklas Cassel
Quoted 15 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

Quoted 5 lines

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Niklas,

Please rebase onto current mainline, test, and resubmit.

Thanks,
Guenter

Quoted 103 lines
---
 drivers/watchdog/softdog.c | 40 ++++++++++++++++++++++++----------------
 1 file changed, 24 insertions(+), 16 deletions(-)
drivers/watchdog/softdog.c +24 -16
index c7bdc986dca1..0f67cd068465 100644
@@ -21,13 +21,12 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/hrtimer.h>
 #include <linux/init.h>
-#include <linux/jiffies.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
 #include <linux/reboot.h>
-#include <linux/timer.h>
 #include <linux/types.h>
 #include <linux/watchdog.h>
 
@@ -54,7 +53,10 @@ module_param(soft_panic, int, 0);
 MODULE_PARM_DESC(soft_panic,
 	"Softdog action, set to 1 to panic, 0 to reboot (default=0)");
 
-static void softdog_fire(unsigned long data)
+static struct hrtimer softdog_ticktock;
+static struct hrtimer softdog_preticktock;
+
+static enum hrtimer_restart softdog_fire(struct hrtimer *timer)
 {
 	module_put(THIS_MODULE);
 	if (soft_noboot) {
@@ -67,41 +69,42 @@ static void softdog_fire(unsigned long data)
 		emergency_restart();
 		pr_crit("Reboot didn't ?????\n");
 	}
-}
 
-static struct timer_list softdog_ticktock =
-		TIMER_INITIALIZER(softdog_fire, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static struct watchdog_device softdog_dev;
 
-static void softdog_pretimeout(unsigned long data)
+static enum hrtimer_restart softdog_pretimeout(struct hrtimer *timer)
 {
 	watchdog_notify_pretimeout(&softdog_dev);
-}
 
-static struct timer_list softdog_preticktock =
-		TIMER_INITIALIZER(softdog_pretimeout, 0, 0);
+	return HRTIMER_NORESTART;
+}
 
 static int softdog_ping(struct watchdog_device *w)
 {
-	if (!mod_timer(&softdog_ticktock, jiffies + (w->timeout * HZ)))
+	if (!hrtimer_active(&softdog_ticktock))
 		__module_get(THIS_MODULE);
+	hrtimer_start(&softdog_ticktock, ktime_set(w->timeout, 0),
+		      HRTIMER_MODE_REL);
 
 	if (w->pretimeout)
-		mod_timer(&softdog_preticktock, jiffies +
-			  (w->timeout - w->pretimeout) * HZ);
+		hrtimer_start(&softdog_preticktock,
+			      ktime_set(w->timeout - w->pretimeout, 0),
+			      HRTIMER_MODE_REL);
 	else
-		del_timer(&softdog_preticktock);
+		hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
 
 static int softdog_stop(struct watchdog_device *w)
 {
-	if (del_timer(&softdog_ticktock))
+	if (hrtimer_cancel(&softdog_ticktock))
 		module_put(THIS_MODULE);
 
-	del_timer(&softdog_preticktock);
+	hrtimer_cancel(&softdog_preticktock);
 
 	return 0;
 }
@@ -134,6 +137,11 @@ static int __init softdog_init(void)
 	watchdog_set_nowayout(&softdog_dev, nowayout);
 	watchdog_stop_on_reboot(&softdog_dev);
 
+	hrtimer_init(&softdog_ticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_ticktock.function = softdog_fire;
+	hrtimer_init(&softdog_preticktock, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	softdog_preticktock.function = softdog_pretimeout;
+
 	ret = watchdog_register_device(&softdog_dev);
 	if (ret)
 		return ret;
#9 lore raw
Quoted 16 lines · Guenter Roeck
Quoted 15 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

Quoted 8 lines
Quoted 5 lines

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Niklas,

Please rebase onto current mainline, test, and resubmit.

I've sent out a v2 now :)

Although, I thought that it was a bit weird that the conflicting patch was in Linus tree but is not in next-20170227.

I'm not sure if you are the web master for www.linux-watchdog.org, but their cgit seems to be broken:
http://www.linux-watchdog.org/cgi-bin/gitweb.cgi?p=linux-watchdog.git;a=summary
http://www.linux-watchdog.org/cgi-bin/gitweb.cgi?p=linux-watchdog-next.git;a=summary

Quoted 4 lines

Thanks,
Guenter

maintainer
#10 lore raw
Quoted 22 lines · Niklas Cassel
Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Niklas Cassel

From: Niklas Cassel <[email protected]>

Checking for timer expiration is done from the softirq TIMER_SOFTIRQ.

Since commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job"), pending softirqs are no longer always handled immediately, instead, if there are pending softirqs, and ksoftirqd is in state TASK_RUNNING, the handling of the softirqs are deferred, and are instead supposed to be handled by ksoftirqd, when ksoftirqd gets scheduled.

If a user space process with a real-time policy starts to misbehave by never relinquishing the CPU while ksoftirqd is in state TASK_RUNNING, what will happen is that all softirqs will get deferred, while ksoftirqd, which is supposed to handle the deferred softirqs, will never get to run.

To make sure that the watchdog is able to fire even when we do not get to run softirqs, replace the timers with hrtimers.

Quoted 9 lines
Quoted 3 lines

Niklas,

Please rebase onto current mainline, test, and resubmit.

I've sent out a v2 now :)

Although, I thought that it was a bit weird that the conflicting patch was in Linus tree but is not in next-20170227.

It was in the watchdog-next branch of my repository at kernel.org, which is not in -next. I'll have to sort that out with Wim at some point.

Quoted 5 lines

I'm not sure if you are the web master for www.linux-watchdog.org, but their cgit seems to be broken:
http://www.linux-watchdog.org/cgi-bin/gitweb.cgi?p=linux-watchdog.git;a=summary
http://www.linux-watchdog.org/cgi-bin/gitweb.cgi?p=linux-watchdog-next.git;a=summary

Wim knows about it, and plans to replace the server.

Guenter