Skip to contentSkip to conversation

reboot: Add timeout for device shutdown

PATCH Superseded Soumya Khasnis 3 participants 5 messages May 29, 2024 – Jun 6, 2024 Driver core

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 reboot: Add timeout for device shutdown

The device shutdown callbacks invoked during shutdown/reboot are prone to errors depending on the device state or mishandling by one or more driver. In order to prevent a device hang in such scenarios, we bail out after a timeout while dumping a meaningful call trace of the shutdown callback which blocks the shutdown or reboot process.

---
 drivers/base/Kconfig | 15 +++++++++++++++
 kernel/reboot.c      | 46 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 1 deletion(-)
drivers/base/Kconfig +15 -0
index 2b8fd6bb7da0..d06e379b6281 100644
@@ -243,3 +243,18 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT
 	  work on.
 
 endmenu
+
+config DEVICE_SHUTDOWN_TIMEOUT
+	bool "device shutdown timeout"
+	default n
+	help
+	   Enable timeout for device shutdown. Helps in case device shutdown
+	   is hung during shoutdonw and reboot.
+
+
+config DEVICE_SHUTDOWN_TIMEOUT_SEC
+	int "device shutdown timeout in seconds"
+	default 5
+	depends on DEVICE_SHUTDOWN_TIMEOUT
+	help
+	  sets time for device shutdown timeout in seconds
kernel/reboot.c +45 -1

77 lines not on this page · raw message

Signature · 1 line
2.40.0
maintainer
#1 lore raw
Quoted 7 lines · Soumya Khasnis

The device shutdown callbacks invoked during shutdown/reboot are prone to errors depending on the device state or mishandling by one or more driver. In order to prevent a device hang in such scenarios, we bail out after a timeout while dumping a meaningful call trace of the shutdown callback which blocks the shutdown or reboot process.

Quoted 7 lines
---
 drivers/base/Kconfig | 15 +++++++++++++++
 kernel/reboot.c      | 46 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 1 deletion(-)

Hi,

This is the friendly patch-bot of Greg Kroah-Hartman. You have sent him a patch that has triggered this response. He used to manually respond to these common problems, but in order to save his sanity (he kept writing the same thing over and over, yet to different people), I was created. Hopefully you will not take offence and will fix the problem in your patch and resubmit it so that it can be accepted into the Linux kernel tree.

You are receiving this message because of the following common error(s) as indicated below:

- This looks like a new version of a previously submitted patch, but you did not list below the --- line any changes from the previous version. Please read the section entitled "The canonical patch format" in the kernel file, Documentation/process/submitting-patches.rst for what needs to be done here to properly describe this.

If you wish to discuss this problem further, or you have questions about how to resolve this issue, please feel free to respond to this email and Greg will reply once he has dug out from the pending patches received from other developers.

thanks,

greg k-h's patch email bot

maintainer
#2 lore raw
In reply to Soumya Khasnis
Quoted 3 lines · Soumya Khasnis

The device shutdown callbacks invoked during shutdown/reboot are prone to errors depending on the device state or mishandling by one or more driver.

Why not fix those drivers? A release callback should not stall, and if it does, that's a bug that should be fixed there.

Or use a watchdog and just reboot if that triggers at shutdown time.

Quoted 4 lines

In order to prevent a device hang in such
scenarios, we bail out after a timeout while dumping a meaningful call trace of the shutdown callback which blocks the shutdown or reboot process.

Dump it where?

Quoted 13 lines
---
 drivers/base/Kconfig | 15 +++++++++++++++
 kernel/reboot.c      | 46 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 1 deletion(-)
drivers/base/Kconfig +0 -0
index 2b8fd6bb7da0..d06e379b6281 100644

drivers/base/Kconfig config FW_DEVLINK_SYNC_STATE_TIMEOUT · line 249

Quoted 7 lines
drivers/base/Kconfig +4 -0
@@ -243,3 +243,18 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT
 	  work on.
 
 endmenu
+
+config DEVICE_SHUTDOWN_TIMEOUT
+	bool "device shutdown timeout"
+	default n

That is the default, so no need for this.

drivers/base/Kconfig config FW_DEVLINK_SYNC_STATE_TIMEOUT · line 260

Quoted 11 lines
drivers/base/Kconfig +11 -0
+	help
+	   Enable timeout for device shutdown. Helps in case device shutdown
+	   is hung during shoutdonw and reboot.
+
+
+config DEVICE_SHUTDOWN_TIMEOUT_SEC
+	int "device shutdown timeout in seconds"
+	default 5
+	depends on DEVICE_SHUTDOWN_TIMEOUT
+	help
+	  sets time for device shutdown timeout in seconds

You need much more help text for all of these.

And why are these in the drivers/base/Kconfig file? It has nothing to do with "devices", or the driver core, it's all core kernel reboot logic.

kernel/reboot.c line 21

Quoted 10 lines
kernel/reboot.c +1 -1
index 22c16e2564cc..8460bd24563b 100644
@@ -18,7 +18,7 @@
 #include <linux/syscalls.h>
 #include <linux/syscore_ops.h>
 #include <linux/uaccess.h>
-
+#include <linux/sched/debug.h>

Why remove the blank line?

Quoted 4 lines
kernel/reboot.c +0 -0
 /*
  * this indicates whether you can reboot with ctrl-alt-del: the default is yes
  */

kernel/reboot.c int reboot_cpu; · line 57

Quoted 10 lines
kernel/reboot.c +7 -0
@@ -48,6 +48,14 @@ int reboot_cpu;
 enum reboot_type reboot_type = BOOT_ACPI;
 int reboot_force;
 
+#ifdef CONFIG_DEVICE_SHUTDOWN_TIMEOUT
+struct device_shutdown_timeout {
+	struct timer_list timer;
+	struct task_struct *task;
+} devs_shutdown;
+#define SHUTDOWN_TIMEOUT CONFIG_DEVICE_SHUTDOWN_TIMEOUT_SEC
+#endif

#ifdefs should not be in .c files, please put this in a .h file where it belongs. Same for the other #ifdefs.

kernel/reboot.c void emergency_restart(void) · line 102

Quoted 12 lines
kernel/reboot.c +5 -0
+
 struct sys_off_handler {
 	struct notifier_block nb;
 	int (*sys_off_cb)(struct sys_off_data *data);
@@ -88,12 +96,46 @@ void emergency_restart(void)
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
 
+#ifdef CONFIG_DEVICE_SHUTDOWN_TIMEOUT
+static void device_shutdown_timeout_handler(struct timer_list *t)
+{
+	pr_emerg("**** device shutdown timeout ****\n");

What does this have to do with "devices"? This is a whole-system issue, or really a "broken driver" issue.

kernel/reboot.c void emergency_restart(void) · line 103

Quoted 1 line
kernel/reboot.c +1 -0
+	show_stack(devs_shutdown.task, NULL, KERN_EMERG);

How do you know this is the 'device shutdown' stack? What is a "device shutdown"?

kernel/reboot.c void emergency_restart(void) · line 112

Quoted 9 lines
kernel/reboot.c +9 -0
+	if (system_state == SYSTEM_RESTART)
+		emergency_restart();
+	else
+		machine_power_off();
+}
+
+static void device_shutdown_timer_set(void)
+{
+	devs_shutdown.task = current;

It's just the normal shutdown stack/process, why call it a device?

Quoted 16 lines
kernel/reboot.c +16 -0
+	timer_setup(&devs_shutdown.timer, device_shutdown_timeout_handler, 0);
+	devs_shutdown.timer.expires = jiffies + SHUTDOWN_TIMEOUT * HZ;
+	add_timer(&devs_shutdown.timer);
+}
+
+static void device_shutdown_timer_clr(void)
+{
+	del_timer(&devs_shutdown.timer);
+}
+#else
+static inline void device_shutdown_timer_set(void)
+{
+}
+static inline void device_shutdown_timer_clr(void)
+{
+}

kernel/reboot.c void emergency_restart(void) · line 138

Quoted 10 lines
kernel/reboot.c +4 -0
+#endif
+
 void kernel_restart_prepare(char *cmd)
 {
 	blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd);
 	system_state = SYSTEM_RESTART;
 	usermodehelper_disable();
+	device_shutdown_timer_set();
 	device_shutdown();
+	device_shutdown_timer_clr();

Why isn't all of this done in device_shutdown() if you think it is a device issue? Why put it in reboot.c?

thanks,

greg k-h

#4 lore raw
In reply to Soumya Khasnis
kernel test robot noticed the following build warnings:

Hi Soumya,

kernel test robot noticed the following build warnings:

[auto build test WARNING on driver-core/driver-core-testing]
[also build test WARNING on driver-core/driver-core-next driver-core/driver-core-linus linus/master v6.10-rc1 next-20240529] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Soumya-Khasnis/reboot-Add-timeout-for-device-shutdown/20240529-190203
base:   driver-core/driver-core-testing
patch link:    https://lore.kernel.org/r/20240529110049.GA73111%40sony.com
patch subject: [PATCH v2] reboot: Add timeout for device shutdown
config: x86_64-randconfig-r132-20240530 (https://download.01.org/0day-ci/archive/20240530/[email protected]/config)
compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240530/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

>> kernel/reboot.c:55:3: sparse: sparse: symbol 'devs_shutdown' was not declared. Should it be static?

kernel/reboot.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/mm.h, ...):
include/linux/page-flags.h:240:46: sparse: sparse: self-comparison always evaluates to false include/linux/page-flags.h:240:46: sparse: sparse: self-comparison always evaluates to false

vim +/devs_shutdown +55 kernel/reboot.c

    50	
    51	#ifdef CONFIG_DEVICE_SHUTDOWN_TIMEOUT
    52	struct device_shutdown_timeout {
    53		struct timer_list timer;
    54		struct task_struct *task;
  > 55	} devs_shutdown;
    56	#define SHUTDOWN_TIMEOUT CONFIG_DEVICE_SHUTDOWN_TIMEOUT_SEC
    57	#endif
    58	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw
In reply to Greg KH
Quoted 7 lines · Greg KH
Quoted 3 lines · Soumya Khasnis

The device shutdown callbacks invoked during shutdown/reboot are prone to errors depending on the device state or mishandling by one or more driver.

Why not fix those drivers? A release callback should not stall, and if it does, that's a bug that should be fixed there.

Yes, the drivers must be fixed. But currently there is no good mechanism which detects such stalls and there by resulting in a system hang. This serves as a fail-safe mechanism to prevent such stalls at reboot/shutdown.

Quoted 2 lines

Or use a watchdog and just reboot if that triggers at shutdown time.

The hard or software watchdog enabled by config_lockup_detector couldn’t detect the cases when stalled on IO wait (wait_for_completion/io)

Quoted 7 lines
Quoted 5 lines

In order to prevent a device hang in such
scenarios, we bail out after a timeout while dumping a meaningful call trace of the shutdown callback which blocks the shutdown or reboot process.

Dump it where?

Dump to kernel log and there by the console. Will update the commit message

Quoted 17 lines
Quoted 17 lines
---
 drivers/base/Kconfig | 15 +++++++++++++++
 kernel/reboot.c      | 46 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 1 deletion(-)
drivers/base/Kconfig +0 -0
index 2b8fd6bb7da0..d06e379b6281 100644
@@ -243,3 +243,18 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT
 	  work on.
 
Quoted 7 lines

drivers/base/Kconfig config FW_DEVLINK_SYNC_STATE_TIMEOUT · line 249

Quoted 5 lines
drivers/base/Kconfig +4 -0
 endmenu
+
+config DEVICE_SHUTDOWN_TIMEOUT
+	bool "device shutdown timeout"
+	default n

That is the default, so no need for this.

Was mindful of the “slow” devices in the system with long IO wait times. Will have to consider increasing DEVICE_SHUTDOWN_TIMEOUT_SEC and make this a default y

Quoted 15 lines

drivers/base/Kconfig config FW_DEVLINK_SYNC_STATE_TIMEOUT · line 260

Quoted 13 lines
drivers/base/Kconfig +11 -0
+	help
+	   Enable timeout for device shutdown. Helps in case device shutdown
+	   is hung during shoutdonw and reboot.
+
+
+config DEVICE_SHUTDOWN_TIMEOUT_SEC
+	int "device shutdown timeout in seconds"
+	default 5
+	depends on DEVICE_SHUTDOWN_TIMEOUT
+	help
+	  sets time for device shutdown timeout in seconds

You need much more help text for all of these.

Will update the commit with more help text

Quoted 4 lines

And why are these in the drivers/base/Kconfig file? It has nothing to do with "devices", or the driver core, it's all core kernel reboot logic.

This is handling only device shutdown, not complete reboot part, so we want to keep it here.

Quoted 14 lines

kernel/reboot.c line 21

Quoted 12 lines
kernel/reboot.c +1 -1
index 22c16e2564cc..8460bd24563b 100644
@@ -18,7 +18,7 @@
 #include <linux/syscalls.h>
 #include <linux/syscore_ops.h>
 #include <linux/uaccess.h>
-
+#include <linux/sched/debug.h>

Why remove the blank line?

Will fix it.

Quoted 8 lines
Quoted 8 lines
kernel/reboot.c +0 -0
 /*
  * this indicates whether you can reboot with ctrl-alt-del: the default is yes
  */
@@ -48,6 +48,14 @@ int reboot_cpu;
 enum reboot_type reboot_type = BOOT_ACPI;
 int reboot_force;
 
Quoted 10 lines

kernel/reboot.c int reboot_cpu; · line 57

Quoted 7 lines
kernel/reboot.c +7 -0
+#ifdef CONFIG_DEVICE_SHUTDOWN_TIMEOUT
+struct device_shutdown_timeout {
+	struct timer_list timer;
+	struct task_struct *task;
+} devs_shutdown;
+#define SHUTDOWN_TIMEOUT CONFIG_DEVICE_SHUTDOWN_TIMEOUT_SEC
+#endif

#ifdefs should not be in .c files, please put this in a .h file where it belongs. Same for the other #ifdefs.

Will fix it

Quoted 11 lines
Quoted 11 lines
kernel/reboot.c +1 -0
+
 struct sys_off_handler {
 	struct notifier_block nb;
 	int (*sys_off_cb)(struct sys_off_data *data);
@@ -88,12 +96,46 @@ void emergency_restart(void)
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
 
Quoted 7 lines

kernel/reboot.c void emergency_restart(void) · line 102

Quoted 4 lines
kernel/reboot.c +4 -0
+#ifdef CONFIG_DEVICE_SHUTDOWN_TIMEOUT
+static void device_shutdown_timeout_handler(struct timer_list *t)
+{
+	pr_emerg("**** device shutdown timeout ****\n");

What does this have to do with "devices"? This is a whole-system issue, or really a "broken driver" issue.

Shutdown/reboot procedure involves quite a few routines. We are specifically dealing with a broken driver or device malfunction.

Quoted 5 lines

kernel/reboot.c void emergency_restart(void) · line 103

Quoted 2 lines
kernel/reboot.c +1 -0
+	show_stack(devs_shutdown.task, NULL, KERN_EMERG);

How do you know this is the 'device shutdown' stack? What is a "device shutdown"?

The timeout handler is invoked in the context of reboot/shutdown process. We are interested in the device shutdown callback which was stuck and preventing the reboot/shutdown. By device shutdown, we refer to device class/bus/driver shutdown calls.

Quoted 12 lines

kernel/reboot.c void emergency_restart(void) · line 112

Quoted 10 lines
kernel/reboot.c +9 -0
+	if (system_state == SYSTEM_RESTART)
+		emergency_restart();
+	else
+		machine_power_off();
+}
+
+static void device_shutdown_timer_set(void)
+{
+	devs_shutdown.task = current;

It's just the normal shutdown stack/process, why call it a device?

We are specifically dealing with a broken driver or device malfunction, So to indicate same, we want to name it as device_shutdown stack.

Quoted 22 lines
Quoted 22 lines
kernel/reboot.c +18 -0
+	timer_setup(&devs_shutdown.timer, device_shutdown_timeout_handler, 0);
+	devs_shutdown.timer.expires = jiffies + SHUTDOWN_TIMEOUT * HZ;
+	add_timer(&devs_shutdown.timer);
+}
+
+static void device_shutdown_timer_clr(void)
+{
+	del_timer(&devs_shutdown.timer);
+}
+#else
+static inline void device_shutdown_timer_set(void)
+{
+}
+static inline void device_shutdown_timer_clr(void)
+{
+}
+#endif
+
 void kernel_restart_prepare(char *cmd)
 {
 	blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd);
Quoted 8 lines

kernel/reboot.c void emergency_restart(void) · line 138

Quoted 5 lines
kernel/reboot.c +2 -0
 	system_state = SYSTEM_RESTART;
 	usermodehelper_disable();
+	device_shutdown_timer_set();
 	device_shutdown();
+	device_shutdown_timer_clr();

Why isn't all of this done in device_shutdown() if you think it is a device issue? Why put it in reboot.c?

Will consider moving to device_shutdown()..

Quoted 4 lines

thanks,

greg k-h

Thanks and Regards,
Soumya.