Skip to contentSkip to conversation

Differentiate scenarios when watchdog is closed

PATCH In review Charles Haithcock 3 participants 8 messages Aug 24 – Sep 1 Watchdog device drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Differentiate scenarios when watchdog is closed

Presenty, when a watchdog device is closed, we print "watchdog did not stop" in a few different scenarios;

1. When nowayout is set
2. When the watchdog is able to close, has received the magic character to stop, but fails to close in device-specific code paths
3. When userspace delierately closes it without stopping it

For 1, we explicitly print we can not close because of nowayout. Nothing differentiates the other two however.

This change adds a print to indicate the watchdog was closed while still running.

---

 drivers/watchdog/watchdog_dev.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/watchdog/watchdog_dev.c +8 -5
index d7895009a2..a571dea353 100644
@@ -955,14 +955,17 @@ static int watchdog_release(struct inode *inode, struct file *file)
 	if (!watchdog_active(wdd))
 		err = 0;
 	else if (test_and_clear_bit(_WDOG_ALLOW_RELEASE, &wd_data->status) ||
-		 !(wdd->info->options & WDIOF_MAGICCLOSE))
+		 !(wdd->info->options & WDIOF_MAGICCLOSE)) {
 		err = watchdog_stop(wdd);
 
-	/* If the watchdog was not stopped, send a keepalive ping */
-	if (err < 0) {
-		pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
-		watchdog_ping(wdd);
+		/* If the watchdog was not stopped, send a keepalive ping */
+		if (err < 0) {
+			pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
+			watchdog_ping(wdd);
+		}
 	}
+	else
+		pr_info("watchdog%d: closing while running!\n", wdd->id);
 
 	watchdog_update_worker(wdd);
 
Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

New issues:
- [High] The patch inadvertently omits the keepalive ping when a watchdog device is closed without the magic close character, altering expected timeout behavior.

Pre-existing issues:
- [High] If the hardware stop operation fails, the WDOG_HW_RUNNING bit is incorrectly left cleared, leading to a premature module reference release and a subsequent Use-After-Free kernel panic.
- [High] Unbalanced module_put() in the error path of watchdog_open() if the hardware is already running, which can lead to a Use-After-Free.
- [High] watchdog_open() calls watchdog_start() without holding wd_data->lock, creating a data race with the background ping worker.
--

--- Patch [1]: [PATCH] Differentiate scenarios when watchdog is closed --- commit a2b7eb1d1568deccc7f35e939965c301a7ed7c1c
Author: Charles Haithcock <[email protected]>

Differentiate scenarios when watchdog is closed

This commit updates the watchdog closing logic to print a specific message when the watchdog is closed unexpectedly while still running. It separates this scenario from the failure to stop when nowayout is set or when the device fails to stop in the hardware driver.

Quoted 14 lines
drivers/watchdog/watchdog_dev.c +1 -3
index 9a5e544b886bf..72f687aee067a 100644
@@ -955,14 +955,17 @@ static int watchdog_release(struct inode *inode, struct file *file)
 	if (!watchdog_active(wdd))
 		err = 0;
 	else if (test_and_clear_bit(_WDOG_ALLOW_RELEASE, &wd_data->status) ||
-		 !(wdd->info->options & WDIOF_MAGICCLOSE))
+		 !(wdd->info->options & WDIOF_MAGICCLOSE)) {
 		err = watchdog_stop(wdd);
 
-	/* If the watchdog was not stopped, send a keepalive ping */
-	if (err < 0) {

drivers/watchdog/watchdog_dev.c static int watchdog_release(struct inode *inode, struct file *file) · line 968

Quoted 10 lines
drivers/watchdog/watchdog_dev.c +7 -2
-		pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
-		watchdog_ping(wdd);
+		/* If the watchdog was not stopped, send a keepalive ping */
+		if (err < 0) {
+			pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
+			watchdog_ping(wdd);
+		}
 	}
+	else
+		pr_info("watchdog%d: closing while running!\n", wdd->id);

[Severity: High]
By nesting the keepalive ping logic inside the else-if block, does this skip sending the keepalive ping when the watchdog is closed unexpectedly (without the magic character)?

This seems to remove the watchdog's ability to reset its hardware timer on an unexpected close, which could cause the system to reboot based on the last explicit ping time rather than providing a full timeout window for recovery.

drivers/watchdog/watchdog_dev.c static int watchdog_release(struct inode *inode, struct file *file) · line 971

Quoted 3 lines
drivers/watchdog/watchdog_dev.c +0 -0
 	watchdog_update_worker(wdd);
 

[Severity: High]
This is a pre-existing issue, but if the hardware stop operation fails, is the WDOG_HW_RUNNING bit incorrectly left cleared?

Looking at watchdog_stop():

drivers/watchdog/watchdog_dev.c:watchdog_stop() {
	if (wdd->ops->stop) {
		clear_bit(WDOG_HW_RUNNING, &wdd->status);
		err = wdd->ops->stop(wdd);
		trace_watchdog_stop(wdd, err);
	} else {

...
}

If wdd->ops->stop() fails, err is returned but WDOG_HW_RUNNING is never restored.

Later in watchdog_release():

drivers/watchdog/watchdog_dev.c:watchdog_release() {
...

	running = wdd && watchdog_hw_running(wdd);
	mutex_unlock(&wd_data->lock);

	if (!running) {
		module_put(wd_data->cdev.owner);
		put_device(&wd_data->dev);
	}

...
}

Would this cause the module reference to be dropped erroneously, allowing the module to be unloaded while the hardware is still running, and potentially leading to a use-after-free when the background worker fires?

[Severity: High]
This is a pre-existing issue, but is there an unbalanced module_put() in the error path of watchdog_open() if the hardware is already running?

drivers/watchdog/watchdog_dev.c:watchdog_open() {
	hw_running = watchdog_hw_running(wdd);
	if (!hw_running && !try_module_get(wdd->ops->owner)) {
		err = -EBUSY;
		goto out_clear;
	}

	err = watchdog_start(wdd);
	if (err < 0)
		goto out_mod;
...
out_mod:
	module_put(wd_data->wdd->ops->owner);
}

If hw_running is true, try_module_get() is skipped. If watchdog_start() then fails, execution jumps to out_mod and calls module_put(). Could this drop a module reference that was never acquired, leading to a premature unload and a use-after-free?

[Severity: High]
This is a pre-existing issue, but does watchdog_open() call watchdog_start() without holding wd_data->lock?

drivers/watchdog/watchdog_dev.c:watchdog_open() {
...

	err = watchdog_start(wdd);
	if (err < 0)
		goto out_mod;

...
}

If the watchdog hardware is running before being opened, the background ping worker may execute concurrently. Does this lack of synchronization allow data races with the worker that can corrupt internal states?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#2 lore raw
In reply to Charles Haithcock
Quoted 1 line · Charles Haithcock

Presenty, when a watchdog device is closed, we print "watchdog did not

Presently

Also, the subject should start with the subsystem name ("watchdog:")

Quoted 6 lines

stop" in a few different scenarios;

1. When nowayout is set
2. When the watchdog is able to close, has received the magic character to stop, but fails to close in device-specific code paths
3. When userspace delierately closes it without stopping it

deliberately

Quoted 27 lines

For 1, we explicitly print we can not close because of nowayout. Nothing differentiates the other two however.

This change adds a print to indicate the watchdog was closed while still running.

---

  drivers/watchdog/watchdog_dev.c | 13 ++++++++-----
  1 file changed, 8 insertions(+), 5 deletions(-)
drivers/watchdog/watchdog_dev.c +1 -3
index d7895009a2..a571dea353 100644
@@ -955,14 +955,17 @@ static int watchdog_release(struct inode *inode, struct file *file)
  	if (!watchdog_active(wdd))
  		err = 0;
  	else if (test_and_clear_bit(_WDOG_ALLOW_RELEASE, &wd_data->status) ||
-		 !(wdd->info->options & WDIOF_MAGICCLOSE))
+		 !(wdd->info->options & WDIOF_MAGICCLOSE)) {
  		err = watchdog_stop(wdd);
 
-	/* If the watchdog was not stopped, send a keepalive ping */
-	if (err < 0) {

drivers/watchdog/watchdog_dev.c static int watchdog_release(struct inode *inode, struct file *file) · line 968

Quoted 10 lines
drivers/watchdog/watchdog_dev.c +7 -2
-		pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
-		watchdog_ping(wdd);
+		/* If the watchdog was not stopped, send a keepalive ping */
+		if (err < 0) {
+			pr_crit("watchdog%d: watchdog did not stop!\n", wdd->id);
+			watchdog_ping(wdd);
+		}
  	}
+	else
+		pr_info("watchdog%d: closing while running!\n", wdd->id);

As Sashiko points out, this changes behavior if the watchdog is active and was not stopped. Also, I personally find "closing while running" not very informative. Also, while technically userspace may close the watchdog deliberately while it is running, that is not what happens on a regular basis. I find the previous unconditional "watchdog did not stop" message more informative and relevant.

If you want to make a change, I would suggest to add an error message into watchdog_stop() to report an error if the stop callback returns an error. That would distinguish 2/3 without making functional changes.

Thanks,
Guenter

#3 lore raw
Quoted 5 lines · Guenter Roeck

Presently
[...]
Also, the subject should start with the subsystem name ("watchdog:") [...]
deliberately

Thank you for catching these! Please accept my apologies. I can fix those up in the next submission.

Quoted 3 lines

[...] Also, while technically userspace may close the watchdog deliberately while it is running, that is not what happens on a regular basis.

This is actually what initiated a bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1991285 it turns out systemd explicitly does this to help ensure a system shutting down actually eventually goes down even if the shutdown process hits some snags. It does this on every shutdown. Given the prevalence of systemd, this is a regular occurrence. The end result is that, when using iTCO, it shows an error on every shutdown when systemd is in use as init.

Quoted 3 lines

If you want to make a change, I would suggest to add an error message into watchdog_stop() to report an error if the stop callback returns an error. That would distinguish 2/3 without making functional changes.

Thank you! So something like this?

       if (wdd->ops->stop) {
               clear_bit(WDOG_HW_RUNNING, &wdd->status);
               err = wdd->ops->stop(wdd);
+               if (err < 0)
+                       pr_info("watchdog%d: closed while still enabled!\n");
               trace_watchdog_stop(wdd, err);
       } else {
               set_bit(WDOG_HW_RUNNING, &wdd->status);

While responding to this, an additional thought occurred to me; given the primary reason a user would see this is because systemd is shutting down a system, it may be more worth while to have systemd log something about closing the watchdog without disarming it to at least explain a pr_crit kernel log line. Otherwise, it just looks like "something bad happened" with watchdog. I am additionally unsure of what would be best to go in watchdog_stop that helps differentiate intentional closing of the watchdog without disabling vs malicious/accidental closing. The intent would lie within the entity closing the watchdog; "closed while still enabled!" still seems like "something bad happened" with info on if it was intentional or not.

Thank you!
- Charles

maintainer
#4 lore raw
Quoted 21 lines · On 8/27/26 10:09, [email protected] wrote:
Quoted 5 lines · Guenter Roeck

Presently
[...]
Also, the subject should start with the subsystem name ("watchdog:") [...]
deliberately

Thank you for catching these! Please accept my apologies. I can fix those up in the next submission.

Quoted 3 lines

[...] Also, while technically userspace may close the watchdog deliberately while it is running, that is not what happens on a regular basis.

This is actually what initiated a bug report at https://bugzilla.redhat.com/show_bug.cgi?id=1991285 it turns out systemd explicitly does this to help ensure a system shutting down actually eventually goes down even if the shutdown process hits some snags. It does this on every shutdown. Given the prevalence of systemd, this is a regular occurrence. The end result is that, when using iTCO, it shows an error on every shutdown when systemd is in use as init.

Quoted 3 lines

If you want to make a change, I would suggest to add an error message into watchdog_stop() to report an error if the stop callback returns an error. That would distinguish 2/3 without making functional changes.

Quoted 7 lines

Thank you! So something like this?

       if (wdd->ops->stop) {
               clear_bit(WDOG_HW_RUNNING, &wdd->status);
               err = wdd->ops->stop(wdd);
+               if (err < 0)
+                       pr_info("watchdog%d: closed while still enabled!\n");

More like

			pr_err(""watchdog%d: Failed to stop watchdog: %pe\n", wdd->id, ERR_PTR(err));

since this would be a real error.

The "watchdog%d: watchdog did not stop!" message will then follow (unconditionally).

Quoted 7 lines

               trace_watchdog_stop(wdd, err); } else {
               set_bit(WDOG_HW_RUNNING, &wdd->status);

While responding to this, an additional thought occurred to me; given the primary reason a user would see this is because systemd is shutting down a system, it may be more worth while to have systemd log something about closing the watchdog without disarming it to at least explain a pr_crit kernel log line. Otherwise, it just looks like "something bad happened" with watchdog. I am additionally unsure of what would be best to go in watchdog_stop that helps differentiate intentional closing of the watchdog without disabling vs malicious/accidental closing. The intent would lie within the entity closing the watchdog; "closed while still enabled!" still seems like "something bad happened" with info on if it was intentional or not.

Problem is that we don't know if "something bad happened". The same message will be seen if the watchdog daemon was killed or crashed. We can not just assume that closing the watchdog device was intentional.

Guenter

#5 lore raw
Quoted 5 lines · Guenter Roeck

Problem is that we don't know if "something bad happened". The same message
will be seen if the watchdog daemon was killed or crashed. We can not just
assume that closing the watchdog device was intentional.

That's the same sticking point I've come across. Differentiating between positive intentional closure versus malicious or buggy closure would somehow need to be communicated to the kernel. It makes more sense that users of the watchdog which intend to close the watchdog while still running maybe denote such. I opened a PR with systemd at
https://github.com/systemd/systemd/pull/43552 to see if there was some way this could be tackled from that perspective.

maintainer
#6 lore raw
Quoted 7 lines · On 8/31/26 13:05, [email protected] wrote:
Quoted 5 lines · Guenter Roeck

Problem is that we don't know if "something bad happened". The same message
will be seen if the watchdog daemon was killed or crashed. We can not just
assume that closing the watchdog device was intentional.

Quoted 7 lines

That's the same sticking point I've come across. Differentiating between positive intentional closure versus malicious or buggy closure would somehow need to be communicated to the kernel. It makes more sense that users of the watchdog which intend to close the watchdog while still running maybe denote such. I opened a PR with systemd at
https://github.com/systemd/systemd/pull/43552 to see if there was some way this could be tackled from that perspective.

I'd want to see a message even if the watchdog is closed but left running on purpose. The best we can do is to add another message if there was an error when trying to stop the watchdog. That should not happen too often (well, not at all, really, unless there is a bug or hardware problem), so it would not really add noise.

Thanks,
Guenter

#7 lore raw
Quoted 9 lines · Guenter Roeck

I'd want to see a message even if the watchdog is closed but left running
on purpose. The best we can do is to add another message if there was an
error when trying to stop the watchdog. That should not happen too often
(well, not at all, really, unless there is a bug or hardware problem),
so it would not really add noise.

Understood; I just submitted a second patch with your recommendations. Thank you again!