power: suspend: Add suspend timeout handler
Subsystems (4)
PATCH 1/1 power: suspend: Add suspend timeout handler
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 > /sys/power/pm_hang
adb shell echo mem > /sys/power/state
--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)
include/linux/console.h +1 -0
@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );/* Suspend and resume console messages over PM events */ extern void suspend_console (void );
kernel/power/Kconfig +9 -0
@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool y depends on PM_DEBUG && PM_SLEEP +config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds. + config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERT
kernel/power/main.c +66 -0
78 lines not on this page · raw message
kernel/power/suspend.c +162 -0
216 lines not on this page · raw message
kernel/printk/printk.c +5 -0
@@ -2259,6 +2259,11 @@ module_param_named(console_suspend, console_suspend_enabled,MODULE_PARM_DESC (console_suspend,"suspend console during suspend" " and hibernate operations" ); +int is_console_suspended (void ) +{ +return console_suspended; +} +/** * suspend_console - suspend the console subsystem *
Signature · 1 line
2.29.0.rc1.297.gfa9743e501-goog
Greg Kroah-Hartman
maintainerQuoted 1 line · Joseph Jang
From: josephjang <[email protected]>
Please use your name as spelled out like you did above in the email header.
Quoted 10 lines
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by josephjang <[email protected]>
Need a blank line before the signed-off-by: and again, spell your name the same way.
Quoted 13 lines
--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)include/linux/console.h +0
-0
include/linux/console.h static inline void console_sysfs_notify(void) · line 195
Quoted 4 lines
include/linux/console.h +1
-0@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );
For global functions, how about:
bool console_is_suspended(void);
?
Quoted 9 lines
include/linux/console.h +0
-0/* Suspend and resume console messages over PM events */ extern void suspend_console (void );kernel/power/Kconfig +0
-0@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool y
kernel/power/Kconfig config PM_SLEEP_DEBUG · line 217
Quoted 10 lines
kernel/power/Kconfig +8
-0depends on PM_DEBUG && PM_SLEEP +config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds.
Ouch, are you sure you want to panic?
Quoted 10 lines
kernel/power/Kconfig +1
-0+ config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERTkernel/power/main.c +0
-0@@ -575,6 +575,69 @@ void __pm_pr_dbg(bool defer, const char *fmt, ...)static inline void pm_print_times_init (void ) {}
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 585
Quoted 10 lines
kernel/power/main.c +8
-0#endif /* CONFIG_PM_SLEEP_DEBUG */ +#ifdef CONFIG_PM_SLEEP_MONITOR +/* If set, devices will stuck at suspend for verification */ +static bool pm_hang_enabled; + +static int pm_notify_test (struct notifier_block *nb, +unsigned long mode,void *_unused) +{ +pr_info ("Jump into infinite loop now\n" );
Why do you have debugging code still enabled?
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 590
Quoted 5 lines
kernel/power/main.c +5
-0+ +/* Suspend thread stuck at a loop forever */ +for (;;) + ; +
Don't busy spin, that will burn power.
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 591
Quoted 1 line
kernel/power/main.c +1
-0+pr_info ("Fail to stuck at loop\n" );
And how can this happen?
Quoted 53 lines
kernel/power/main.c +49
-0+ +return 0 ; +} + +static struct notifier_block pm_notify_nb = { + .notifier_call = pm_notify_test, +}; + +static ssize_t pm_hang_show (struct kobject *kobj,struct kobj_attribute *attr, +char *buf) +{ +return snprintf (buf,10 ,"%d\n" , pm_hang_enabled); +} + +static ssize_t pm_hang_store (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t n) +{ +unsigned long val; +int result; + +if (kstrtoul (buf,10 , &val)) +return -EINVAL ; + +if (val >1 ) +return -EINVAL ; + + pm_hang_enabled = !!val; + +if (pm_hang_enabled == true) { + + result =register_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not register suspend notifier, return %d\n" , + result); + + }else { + + result =unregister_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not unregister suspend notifier, return %d\n" , + result); + } + +return n; +} + +power_attr (pm_hang); +#endif +struct kobject *power_kobj;/**
kernel/power/main.c static struct attribute * g[] = { · line 976
Quoted 5 lines
kernel/power/main.c +2
-0@@ -909,6 +972,9 @@ static struct attribute * g[] = { &pm_wakeup_irq_attr.attr, &pm_debug_messages_attr.attr, #endif +#ifdef CONFIG_PM_SLEEP_MONITOR + &pm_hang_attr.attr,
You added a sysfs file, but no Documentation/ABI/ update? That's not ok.
Quoted 9 lines
kernel/power/main.c +1
-0+#endif #endif #ifdef CONFIG_FREEZER &pm_freeze_timeout_attr.attr,kernel/power/suspend.c +0
-0
kernel/power/suspend.c line 38
Quoted 9 lines
kernel/power/suspend.c +6
-0@@ -30,6 +30,12 @@#include <trace/events/power.h> #include <linux/compiler.h> #include <linux/moduleparam.h> +#ifdef CONFIG_PM_SLEEP_MONITOR +#include <linux/sched/debug.h> +#include <linux/kthread.h> +#include <linux/sched.h> +#include <uapi/linux/sched/types.h> +#endif
Don't #ifdef include files.
And why the uapi file?
Quoted 16 lines
kernel/power/suspend.c +9
-0#include "power.h" @@ -61,6 +67,133 @@ static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head);enum s2idle_states __read_mostly s2idle_state;static DEFINE_RAW_SPINLOCK (s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +/* Suspend monitor thread toggle reason */ +enum toggle_reason { +TOGGLE_NONE , +TOGGLE_START , +TOGGLE_STOP , +}; + +#define SUSPEND_TIMER_TIMEOUT_MS 30000
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 88
Quoted 10 lines
kernel/power/suspend.c +10
-0+static struct task_struct *ksuspend_mon_tsk; +static DECLARE_WAIT_QUEUE_HEAD (power_suspend_waitqueue); +static enum toggle_reason suspend_mon_toggle; +static DEFINE_MUTEX (suspend_mon_lock); + +static void start_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_START ; +mutex_unlock (&suspend_mon_lock);
Why do you need a lock for a single integer?
Quoted 17 lines
kernel/power/suspend.c +17
-0+wake_up (&power_suspend_waitqueue); +} + +static void stop_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_STOP ; +mutex_unlock (&suspend_mon_lock); +wake_up (&power_suspend_waitqueue); +} + +static void suspend_timeout (int timeout_count) +{ +char *null_pointer =NULL ; + +pr_info ("Suspend monitor timeout (timer is %d seconds)\n" , + (SUSPEND_TIMER_TIMEOUT_MS /1000 ));
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 115
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +show_state_filter (TASK_UNINTERRUPTIBLE ); + +if (timeout_count <2 ) +return ; + +if (is_console_suspended ()) +resume_console (); + +pr_info ("Trigger a panic\n" );
Again, debugging code?
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 118
Quoted 3 lines
kernel/power/suspend.c +3
-0+ +/* Trigger a NULL pointer dereference */ + *null_pointer ='a' ;
Are you sure this will work on all platforms? We do have a panic function if you really want to do that.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 128
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +/* Should not reach here */ +pr_err ("Trigger panic failed!\n" ); +} + +static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 };
Ick, no, call the scheduler functions properly, don't do this "by hand" ever.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 132
Quoted 4 lines
kernel/power/suspend.c +4
-0+static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" );
Again, debugging code :(
Quoted 68 lines
kernel/power/suspend.c +64
-0+ +sched_setscheduler (current,SCHED_FIFO , ¶m); + + timeout_count =0 ; + timeout =MAX_SCHEDULE_TIMEOUT ; + +do { +/* Wait suspend timer timeout */ + err =wait_event_interruptible_timeout ( + power_suspend_waitqueue, + (suspend_mon_toggle !=TOGGLE_NONE ), + timeout); + +mutex_lock (&suspend_mon_lock); +/* suspend monitor state change */ +if (suspend_mon_toggle !=TOGGLE_NONE ) { +if (suspend_mon_toggle ==TOGGLE_START ) { + timeout =msecs_to_jiffies ( +SUSPEND_TIMER_TIMEOUT_MS ); +pr_info ("Start suspend monitor\n" ); + }else if (suspend_mon_toggle ==TOGGLE_STOP ) { + timeout =MAX_SCHEDULE_TIMEOUT ; + timeout_count =0 ; +pr_info ("Stop suspend monitor\n" ); + } + suspend_mon_toggle =TOGGLE_NONE ; +mutex_unlock (&suspend_mon_lock); +continue ; + } +mutex_unlock (&suspend_mon_lock); + +/* suspend monitor event handler */ +if (err ==0 ) { + timeout_count++; +suspend_timeout (timeout_count); + }else if (err == -ERESTARTSYS ) { +pr_info ("Exit ksuspend_mon!" ); +break ; + } + }while (1 ); + + return 0; +} + +static void init_suspend_monitor_thread (void ) +{ +int ret; + + ksuspend_mon_tsk =kthread_create (suspend_monitor_kthread, +NULL ,"ksuspend_mon" ); +if (IS_ERR (ksuspend_mon_tsk)) { + ret =PTR_ERR (ksuspend_mon_tsk); + ksuspend_mon_tsk =NULL ; +pr_err ("Create suspend_monitor_kthread failed! ret = %d\n" , + ret); +return ; + } + + suspend_mon_toggle =TOGGLE_NONE ; +wake_up_process (ksuspend_mon_tsk); + +} +#endif +/** * pm_suspend_default_s2idle - Check if suspend-to-idle is the default suspend. *
kernel/power/suspend.c static void s2idle_enter(void) · line 227
Quoted 6 lines
kernel/power/suspend.c +3
-0@@ -89,6 +222,10 @@ static void s2idle_enter(void) {trace_suspend_resume (TPS ("machine_suspend" ),PM_SUSPEND_TO_IDLE , true); +#ifdef CONFIG_PM_SLEEP_MONITOR +stop_suspend_mon (); +#endif
Do not put #ifdef in .c files, that's not the proper kernel coding style. Especially for single function calls.
I've stopped here...
greg k-h
A message that is not in the archive (sent off-list or not yet received): CAPaOXERGzo8uF9gh4aAoicEAi_TtHn1M2Yno5LAWQPcWmq_evQ@mail.gmail.com
Greg Kroah-Hartman
maintainerQuoted 1 line · Joseph Jang
Thank you Greg's promptly reply.
<snip>
You just sent html email, which got rejected by all of the mailing lists :(
Please fix your email client to be sand and resend.
thanks,
greg k-h
Thank you Greg's promptly reply.
let me try to explain detail in following. Sorry I forgot to switch to plain text mode in gmail.
Quoted 1 line · Joseph Jang
From: josephjang <[email protected]>
Please use your name as spelled out like you did above in the email header.
Sure, I will update the patch again like following.
From b3afca8f3ee1d587c188b830d94c683fe66cbfb3 Mon Sep 17 00:00:00 2001
From: Joseph Jang <[email protected]>
Date: Fri, 16 Oct 2020 16:31:38 +0800
Subject: [PATCH] power: suspend: Add suspend timeout handler
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Quoted 10 lines
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by josephjang <[email protected]>
Need a blank line before the signed-off-by: and again, spell your name the same way.
Sure, I will update the patch again like following.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 > /sys/power/pm_hang
adb shell echo mem > /sys/power/state
---
Quoted 13 lines
--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)include/linux/console.h +0
-0
include/linux/console.h static inline void console_sysfs_notify(void) · line 195
Quoted 4 lines
include/linux/console.h +1
-0@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );
For global functions, how about:
bool console_is_suspended(void);
?
Agree, I will update like following.
extern bool console_suspend_enabled;
+extern int console_is_suspended(void);
Quoted 9 lines
include/linux/console.h +0
-0/* Suspend and resume console messages over PM events */ extern void suspend_console (void );kernel/power/Kconfig +0
-0@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool y
kernel/power/Kconfig config PM_SLEEP_DEBUG · line 217
Quoted 10 lines
kernel/power/Kconfig +8
-0depends on PM_DEBUG && PM_SLEEP +config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds.
Ouch, are you sure you want to panic?
Yes, we would like to trigger kernel panic to reboot system and prevent system hang there.
Quoted 10 lines
kernel/power/Kconfig +1
-0+ config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERTkernel/power/main.c +0
-0@@ -575,6 +575,69 @@ void __pm_pr_dbg(bool defer, const char *fmt, ...)static inline void pm_print_times_init (void ) {}
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 585
Quoted 10 lines
kernel/power/main.c +8
-0#endif /* CONFIG_PM_SLEEP_DEBUG */ +#ifdef CONFIG_PM_SLEEP_MONITOR +/* If set, devices will stuck at suspend for verification */ +static bool pm_hang_enabled; + +static int pm_notify_test (struct notifier_block *nb, +unsigned long mode,void *_unused) +{ +pr_info ("Jump into infinite loop now\n" );
Why do you have debugging code still enabled?
Because we want to make sure system hang was cause by our test code here.
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 590
Quoted 5 lines
kernel/power/main.c +5
-0+ +/* Suspend thread stuck at a loop forever */ +for (;;) + ; +
Don't busy spin, that will burn power.
Okay, I will add msleep() to prevent system burn power during testing.
+static int pm_notify_test(struct notifier_block *nb,
+ unsigned long mode, void *_unused)
+{
+ pr_info("Jump into infinite loop now\n");
+
+ /* Suspend thread stuck at a loop forever */
+ for (;;)
+ msleep(100);
+
+ pr_info("Fail to stuck at loop\n");
+
+ return 0;
+}
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 591
Quoted 1 line
kernel/power/main.c +1
-0+pr_info ("Fail to stuck at loop\n" );
And how can this happen?
I think we should not see this log unless the for loop has problem.
Sure, I add some description in the document like following.
Documentation/ABI/testing/sysfs-power +15 -0
@@ -420,3 +420,18 @@ Description: disables it. Reads from the file return the current value. The default is "1" if the build-time "SUSPEND_SKIP_SYNC" config flag is unset, or "0" otherwise. + +What: /sys/power/pm_hang +Date: Oct 2020 +Contact: Joseph Jang <[email protected]> +Description: + The /sys/power/pm_hang file controls the variable pm_hang_enabled + which controls whether you need to simulate system hang during + suspend. + + Writing a "1" to this file will set the pm_hang_enabled to "1" and + register a pm_notify function pm_notify_test() to simulate system + hang during suspend. + Writing a "0" to this file will set the pm_hang_enabled to "0" and + unregister a pm_notify function pm_notify_test(). + Reads from this file return the current value of pm_hang_enabled.
Quoted 9 lines
kernel/power/main.c +1
-0+#endif #endif #ifdef CONFIG_FREEZER &pm_freeze_timeout_attr.attr,kernel/power/suspend.c +0
-0
kernel/power/suspend.c line 38
Quoted 9 lines
kernel/power/suspend.c +6
-0@@ -30,6 +30,12 @@#include <trace/events/power.h> #include <linux/compiler.h> #include <linux/moduleparam.h> +#ifdef CONFIG_PM_SLEEP_MONITOR +#include <linux/sched/debug.h> +#include <linux/kthread.h> +#include <linux/sched.h> +#include <uapi/linux/sched/types.h> +#endif
Don't #ifdef include files.
And why the uapi file?
I refer to linux-stable/kernel/rcu/tree.c source code. In order to prevent build error for MAX_RT_PRIO, I need to include this header file
<uapi/linux/sched/types.h>.
Quoted 16 lines
kernel/power/suspend.c +9
-0#include "power.h" @@ -61,6 +67,133 @@ static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head);enum s2idle_states __read_mostly s2idle_state;static DEFINE_RAW_SPINLOCK (s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +/* Suspend monitor thread toggle reason */ +enum toggle_reason { +TOGGLE_NONE , +TOGGLE_START , +TOGGLE_STOP , +}; + +#define SUSPEND_TIMER_TIMEOUT_MS 30000
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 88
Quoted 10 lines
kernel/power/suspend.c +10
-0+static struct task_struct *ksuspend_mon_tsk; +static DECLARE_WAIT_QUEUE_HEAD (power_suspend_waitqueue); +static enum toggle_reason suspend_mon_toggle; +static DEFINE_MUTEX (suspend_mon_lock); + +static void start_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_START ; +mutex_unlock (&suspend_mon_lock);
Why do you need a lock for a single integer?
Since we have two kernel thread may reference variable "suspend_mon_toggle", one is suspend thread, another is suspend_monitor_kthread. We use it to start/stop
suspend monitor.
Yes, in order to notify user this kernel panic was triggered by suspend_timeout(), I add debug code to prevent confused.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 91
Quoted 3 lines
kernel/power/suspend.c +3
-0+ +/* Trigger a NULL pointer dereference */ + *null_pointer ='a' ;
Are you sure this will work on all platforms? We do have a panic function if you really want to do that.
I agree to change to use panic() like following. +static void suspend_timeout(int timeout_count)
+{
+ pr_info("Suspend monitor timeout (timer is %d seconds)\n",
+ (SUSPEND_TIMER_TIMEOUT_MS/1000));
+
+ show_state_filter(TASK_UNINTERRUPTIBLE);
+
+ if (timeout_count < 2)
+ return;
+
+ if (console_is_suspended())
+ resume_console();
+
+ /* Trigger a NULL pointer dereference */
+ panic("suspend timeout and panic");
+}
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 101
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +/* Should not reach here */ +pr_err ("Trigger panic failed!\n" ); +} + +static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 };
Ick, no, call the scheduler functions properly, don't do this "by hand"
ever.
I refer to kernel/sched/core.c and they use MAX_RT_PRIO as thread priority directly.
If you have other suggestion, I am willing to try.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 105
Quoted 4 lines
kernel/power/suspend.c +4
-0+static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" );
Again, debugging code :(
In order to make sure ksuspend_mon thread has been initialize, I add log here. If you think that is not necessary, I can remove it.
Sure, I got it, could you help to review if we can use IS_ENABLED() like following?
+ if (IS_ENABLED(CONFIG_PM_SLEEP_MONITOR))
+ stop_suspend_mon();
+
Thank you,
Joseph.
Joseph Jang <[email protected]> 於 2020年10月16日 週五 下午4:58寫道:
Quoted 850 lines
Thank you Greg's promptly reply.
let me try to explain detail in following.
Quoted 1 line · Joseph Jang
From: josephjang <[email protected]>
Please use your name as spelled out like you did above in the email header.
Sure, I will update the patch again like following.
From
b3afca8f3ee1d587c188b830d94c683fe66cbfb3Mon Sep 17 00:00:00 2001
From: Joseph Jang <[email protected]>
Date: Fri, 16 Oct 2020 16:31:38 +0800
Subject: [PATCH] power: suspend: Add suspend timeout handlerAdd suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Quoted 10 lines
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by josephjang <[email protected]>Need a blank line before the signed-off-by: and again, spell your name the same way.
Sure, I will update the patch again like following.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by Joseph Jang <[email protected]>---
Quoted 13 lines
--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)include/linux/console.h +0
-0include/linux/console.h static inline void console_sysfs_notify(void) · line 195
Quoted 4 lines
include/linux/console.h +1
-0@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );For global functions, how about:
bool console_is_suspended(void);?
Agree, I will update like following.
extern boolconsole_suspend_enabled;
+extern intconsole_is_suspended(void);Quoted 9 lines
include/linux/console.h +0
-0/* Suspend and resume console messages over PM events */ extern void suspend_console (void );kernel/power/Kconfig +0
-0@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool ykernel/power/Kconfig config PM_SLEEP_DEBUG · line 217
Quoted 10 lines
kernel/power/Kconfig +8
-0depends on PM_DEBUG && PM_SLEEP +config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds.Ouch, are you sure you want to panic?
Yes, we would like to trigger kernel panic to reboot system and prevent system hang there.
Quoted 10 lines
kernel/power/Kconfig +1
-0+ config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERTkernel/power/main.c +0
-0@@ -575,6 +575,69 @@ void __pm_pr_dbg(bool defer, const char *fmt, ...)static inline void pm_print_times_init (void ) {}kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 585
Quoted 10 lines
kernel/power/main.c +8
-0#endif /* CONFIG_PM_SLEEP_DEBUG */ +#ifdef CONFIG_PM_SLEEP_MONITOR +/* If set, devices will stuck at suspend for verification */ +static bool pm_hang_enabled; + +static int pm_notify_test (struct notifier_block *nb, +unsigned long mode,void *_unused) +{ +pr_info ("Jump into infinite loop now\n" );Why do you have debugging code still enabled?
Because we want to make sure system hang was cause by our test code here.
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 590
Quoted 5 lines
kernel/power/main.c +5
-0+ +/* Suspend thread stuck at a loop forever */ +for (;;) + ; +Don't busy spin, that will burn power.
Okay, I will add
msleep()to prevent system burn power during testing.kernel/power/suspend.c +13
-0+static int pm_notify_test (struct notifier_block *nb, +unsigned long mode,void *_unused) +{ +pr_info ("Jump into infinite loop now\n" ); + +/* Suspend thread stuck at a loop forever */ +for (;;) +msleep (100 ); + +pr_info ("Fail to stuck at loop\n" ); + +return 0 ; +}kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 591
Quoted 3 lines
kernel/power/main.c +1
-0+pr_info ("Fail to stuck at loop\n" );And how can this happen?
I think we should not see this log unless the for loop has problem.
Quoted 53 lines
kernel/power/main.c +49
-0+ +return 0 ; +} + +static struct notifier_block pm_notify_nb = { + .notifier_call = pm_notify_test, +}; + +static ssize_t pm_hang_show (struct kobject *kobj,struct kobj_attribute *attr, +char *buf) +{ +return snprintf (buf,10 ,"%d\n" , pm_hang_enabled); +} + +static ssize_t pm_hang_store (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t n) +{ +unsigned long val; +int result; + +if (kstrtoul (buf,10 , &val)) +return -EINVAL ; + +if (val >1 ) +return -EINVAL ; + + pm_hang_enabled = !!val; + +if (pm_hang_enabled == true) { + + result =register_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not register suspend notifier, return %d\n" , + result); + + }else { + + result =unregister_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not unregister suspend notifier, return %d\n" , + result); + } + +return n; +} + +power_attr (pm_hang); +#endif +struct kobject *power_kobj;/** kernel/power/main.c static struct attribute * g[] = { · line 976
Quoted 5 lines
kernel/power/main.c +2
-0@@ -909,6 +972,9 @@ static struct attribute * g[] = { &pm_wakeup_irq_attr.attr, &pm_debug_messages_attr.attr, #endif +#ifdef CONFIG_PM_SLEEP_MONITOR + &pm_hang_attr.attr,You added a sysfs file, but no
Documentation/ABI/update? That's not ok.Sure, I add some description in the document like following.
Documentation/ABI/testing/sysfs-power +15
-0@@ -420,3 +420,18 @@ Description: disables it. Reads from the file return the current value. The default is "1" if the build-time "SUSPEND_SKIP_SYNC" config flag is unset, or "0" otherwise. + +What: /sys/power/pm_hang +Date: Oct 2020 +Contact: Joseph Jang <[email protected]> +Description: + The /sys/power/pm_hang file controls the variable pm_hang_enabled + which controls whether you need to simulate system hang during + suspend. + + Writing a "1" to this file will set the pm_hang_enabled to "1" and + register a pm_notify function pm_notify_test() to simulate system + hang during suspend. + Writing a "0" to this file will set the pm_hang_enabled to "0" and + unregister a pm_notify function pm_notify_test(). + Reads from this file return the current value of pm_hang_enabled.Quoted 10 lines
kernel/power/main.c +1
-0+#endif #endif #ifdef CONFIG_FREEZER &pm_freeze_timeout_attr.attr,kernel/power/suspend.c +0
-0kernel/power/suspend.c line 38
Quoted 9 lines
kernel/power/suspend.c +6
-0@@ -30,6 +30,12 @@#include <trace/events/power.h> #include <linux/compiler.h> #include <linux/moduleparam.h> +#ifdef CONFIG_PM_SLEEP_MONITOR +#include <linux/sched/debug.h> +#include <linux/kthread.h> +#include <linux/sched.h> +#include <uapi/linux/sched/types.h> +#endif Don't #ifdef include files.
And why the uapi file?
I refer to
linux-stable/kernel/rcu/tree.csource code.
In order to prevent build error forMAX_RT_PRIO, I need to include this header file <uapi/linux/sched/types.h>.Quoted 16 lines
kernel/power/suspend.c +9
-0#include "power.h" @@ -61,6 +67,133 @@ static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head);enum s2idle_states __read_mostly s2idle_state;static DEFINE_RAW_SPINLOCK (s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +/* Suspend monitor thread toggle reason */ +enum toggle_reason { +TOGGLE_NONE , +TOGGLE_START , +TOGGLE_STOP , +}; + +#define SUSPEND_TIMER_TIMEOUT_MS 30000kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 88
Quoted 10 lines
kernel/power/suspend.c +10
-0+static struct task_struct *ksuspend_mon_tsk; +static DECLARE_WAIT_QUEUE_HEAD (power_suspend_waitqueue); +static enum toggle_reason suspend_mon_toggle; +static DEFINE_MUTEX (suspend_mon_lock); + +static void start_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_START ; +mutex_unlock (&suspend_mon_lock);Why do you need a lock for a single integer?
Since we have two kernel thread may reference variable "
suspend_mon_toggle", one is suspend thread, another issuspend_monitor_kthread. We use it to start/stop suspend monitor.Quoted 17 lines
kernel/power/suspend.c +17
-0+wake_up (&power_suspend_waitqueue); +} + +static void stop_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_STOP ; +mutex_unlock (&suspend_mon_lock); +wake_up (&power_suspend_waitqueue); +} + +static void suspend_timeout (int timeout_count) +{ +char *null_pointer =NULL ; + +pr_info ("Suspend monitor timeout (timer is %d seconds)\n" , + (SUSPEND_TIMER_TIMEOUT_MS /1000 ));kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 115
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +show_state_filter (TASK_UNINTERRUPTIBLE ); + +if (timeout_count <2 ) +return ; + +if (is_console_suspended ()) +resume_console (); + +pr_info ("Trigger a panic\n" );Again, debugging code?
Yes, in order to notify user this kernel panic was triggered by
suspend_timeout(), I add debug code to prevent confused.kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 118
Quoted 3 lines
kernel/power/suspend.c +3
-0+ +/* Trigger a NULL pointer dereference */ + *null_pointer ='a' ;Are you sure this will work on all platforms? We do have a panic function if you really want to do that.
I agree to change to use
panic()like following.Documentation/ABI/testing/sysfs-power +16
-0+static void suspend_timeout(int timeout_count) +{ + pr_info("Suspend monitor timeout (timer is %d seconds)\n", + (SUSPEND_TIMER_TIMEOUT_MS/1000)); + + show_state_filter(TASK_UNINTERRUPTIBLE); + + if (timeout_count < 2) + return; + + if (console_is_suspended()) + resume_console(); + + /* Trigger a NULL pointer dereference */ + panic("suspend timeout and panic"); +}kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 128
Quoted 11 lines
kernel/power/suspend.c +10
-0+ +/* Should not reach here */ +pr_err ("Trigger panic failed!\n" ); +} + +static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 };Ick, no, call the scheduler functions properly, don't do this "by hand"
ever.
I refer tokernel/sched/core.cand they useMAX_RT_PRIOas thread priority directly. If you have other suggestion, I am willing to try.kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 132
Quoted 4 lines
kernel/power/suspend.c +4
-0+static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" );Again, debugging code :(
In order to make sure
ksuspend_monthread has been initialize, I add log here. If you think that is not necessary, I can remove it.Quoted 68 lines
kernel/power/suspend.c +64
-0+ +sched_setscheduler (current,SCHED_FIFO , ¶m); + + timeout_count =0 ; + timeout =MAX_SCHEDULE_TIMEOUT ; + +do { +/* Wait suspend timer timeout */ + err =wait_event_interruptible_timeout ( + power_suspend_waitqueue, + (suspend_mon_toggle !=TOGGLE_NONE ), + timeout); + +mutex_lock (&suspend_mon_lock); +/* suspend monitor state change */ +if (suspend_mon_toggle !=TOGGLE_NONE ) { +if (suspend_mon_toggle ==TOGGLE_START ) { + timeout =msecs_to_jiffies ( +SUSPEND_TIMER_TIMEOUT_MS ); +pr_info ("Start suspend monitor\n" ); + }else if (suspend_mon_toggle ==TOGGLE_STOP ) { + timeout =MAX_SCHEDULE_TIMEOUT ; + timeout_count =0 ; +pr_info ("Stop suspend monitor\n" ); + } + suspend_mon_toggle =TOGGLE_NONE ; +mutex_unlock (&suspend_mon_lock); +continue ; + } +mutex_unlock (&suspend_mon_lock); + +/* suspend monitor event handler */ +if (err ==0 ) { + timeout_count++; +suspend_timeout (timeout_count); + }else if (err == -ERESTARTSYS ) { +pr_info ("Exit ksuspend_mon!" ); +break ; + } + }while (1 ); + + return 0; +} + +static void init_suspend_monitor_thread (void ) +{ +int ret; + + ksuspend_mon_tsk =kthread_create (suspend_monitor_kthread, +NULL ,"ksuspend_mon" ); +if (IS_ERR (ksuspend_mon_tsk)) { + ret =PTR_ERR (ksuspend_mon_tsk); + ksuspend_mon_tsk =NULL ; +pr_err ("Create suspend_monitor_kthread failed! ret = %d\n" , + ret); +return ; + } + + suspend_mon_toggle =TOGGLE_NONE ; +wake_up_process (ksuspend_mon_tsk); + +} +#endif +/** * pm_suspend_default_s2idle - Check if suspend-to-idle is the default suspend. * kernel/power/suspend.c static void s2idle_enter(void) · line 227
Quoted 6 lines
kernel/power/suspend.c +3
-0@@ -89,6 +222,10 @@ static void s2idle_enter(void) {trace_suspend_resume (TPS ("machine_suspend" ),PM_SUSPEND_TO_IDLE , true); +#ifdef CONFIG_PM_SLEEP_MONITOR +stop_suspend_mon (); +#endif Do not put #ifdef in .c files, that's not the proper kernel coding style. Especially for single function calls.
Sure, I got it, could you help to review if we can use
IS_ENABLED()like following?
+ if (IS_ENABLED(CONFIG_PM_SLEEP_MONITOR))
+stop_suspend_mon();
+Thank you,
Joseph.Greg Kroah-Hartman <[email protected]> 於 2020年10月16日 週五 下午1:43寫道:
Quoted 355 lines · Joseph Jang
Quoted 1 line
From: josephjang <[email protected]>
Please use your name as spelled out like you did above in the email header.
Quoted 10 lines
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by josephjang <[email protected]>Need a blank line before the signed-off-by: and again, spell your name the same way.
Quoted 13 lines
--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)include/linux/console.h +0
-0include/linux/console.h static inline void console_sysfs_notify(void) · line 195
Quoted 4 lines
include/linux/console.h +1
-0@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );For global functions, how about:
bool console_is_suspended(void);?
Quoted 9 lines
include/linux/console.h +0
-0/* Suspend and resume console messages over PM events */ extern void suspend_console (void );kernel/power/Kconfig +0
-0@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool ykernel/power/Kconfig config PM_SLEEP_DEBUG · line 217
Quoted 10 lines
kernel/power/Kconfig +8
-0depends on PM_DEBUG && PM_SLEEP +config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds.Ouch, are you sure you want to panic?
Quoted 10 lines
kernel/power/Kconfig +1
-0+ config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERTkernel/power/main.c +0
-0@@ -575,6 +575,69 @@ void __pm_pr_dbg(bool defer, const char *fmt, ...)static inline void pm_print_times_init (void ) {}kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 585
Quoted 10 lines
kernel/power/main.c +8
-0#endif /* CONFIG_PM_SLEEP_DEBUG */ +#ifdef CONFIG_PM_SLEEP_MONITOR +/* If set, devices will stuck at suspend for verification */ +static bool pm_hang_enabled; + +static int pm_notify_test (struct notifier_block *nb, +unsigned long mode,void *_unused) +{ +pr_info ("Jump into infinite loop now\n" );Why do you have debugging code still enabled?
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 590
Quoted 5 lines
kernel/power/main.c +5
-0+ +/* Suspend thread stuck at a loop forever */ +for (;;) + ; +Don't busy spin, that will burn power.
kernel/power/main.c void __pm_pr_dbg(bool defer, const char *fmt, ...) · line 591
Quoted 1 line
kernel/power/main.c +1
-0+pr_info ("Fail to stuck at loop\n" );And how can this happen?
Quoted 53 lines
kernel/power/main.c +49
-0+ +return 0 ; +} + +static struct notifier_block pm_notify_nb = { + .notifier_call = pm_notify_test, +}; + +static ssize_t pm_hang_show (struct kobject *kobj,struct kobj_attribute *attr, +char *buf) +{ +return snprintf (buf,10 ,"%d\n" , pm_hang_enabled); +} + +static ssize_t pm_hang_store (struct kobject *kobj,struct kobj_attribute *attr, +const char *buf,size_t n) +{ +unsigned long val; +int result; + +if (kstrtoul (buf,10 , &val)) +return -EINVAL ; + +if (val >1 ) +return -EINVAL ; + + pm_hang_enabled = !!val; + +if (pm_hang_enabled == true) { + + result =register_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not register suspend notifier, return %d\n" , + result); + + }else { + + result =unregister_pm_notifier (&pm_notify_nb); +if (result) +pr_warn ("Can not unregister suspend notifier, return %d\n" , + result); + } + +return n; +} + +power_attr (pm_hang); +#endif +struct kobject *power_kobj;/** kernel/power/main.c static struct attribute * g[] = { · line 976
Quoted 5 lines
kernel/power/main.c +2
-0@@ -909,6 +972,9 @@ static struct attribute * g[] = { &pm_wakeup_irq_attr.attr, &pm_debug_messages_attr.attr, #endif +#ifdef CONFIG_PM_SLEEP_MONITOR + &pm_hang_attr.attr,You added a sysfs file, but no
Documentation/ABI/update? That's not ok.Quoted 9 lines
kernel/power/main.c +1
-0+#endif #endif #ifdef CONFIG_FREEZER &pm_freeze_timeout_attr.attr,kernel/power/suspend.c +0
-0kernel/power/suspend.c line 38
Quoted 9 lines
kernel/power/suspend.c +6
-0@@ -30,6 +30,12 @@#include <trace/events/power.h> #include <linux/compiler.h> #include <linux/moduleparam.h> +#ifdef CONFIG_PM_SLEEP_MONITOR +#include <linux/sched/debug.h> +#include <linux/kthread.h> +#include <linux/sched.h> +#include <uapi/linux/sched/types.h> +#endif Don't #ifdef include files.
And why the uapi file?
Quoted 16 lines
kernel/power/suspend.c +9
-0#include "power.h" @@ -61,6 +67,133 @@ static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head);enum s2idle_states __read_mostly s2idle_state;static DEFINE_RAW_SPINLOCK (s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +/* Suspend monitor thread toggle reason */ +enum toggle_reason { +TOGGLE_NONE , +TOGGLE_START , +TOGGLE_STOP , +}; + +#define SUSPEND_TIMER_TIMEOUT_MS 30000kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 88
Quoted 10 lines
kernel/power/suspend.c +10
-0+static struct task_struct *ksuspend_mon_tsk; +static DECLARE_WAIT_QUEUE_HEAD (power_suspend_waitqueue); +static enum toggle_reason suspend_mon_toggle; +static DEFINE_MUTEX (suspend_mon_lock); + +static void start_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_START ; +mutex_unlock (&suspend_mon_lock);Why do you need a lock for a single integer?
Quoted 17 lines
kernel/power/suspend.c +17
-0+wake_up (&power_suspend_waitqueue); +} + +static void stop_suspend_mon (void ) +{ +mutex_lock (&suspend_mon_lock); + suspend_mon_toggle =TOGGLE_STOP ; +mutex_unlock (&suspend_mon_lock); +wake_up (&power_suspend_waitqueue); +} + +static void suspend_timeout (int timeout_count) +{ +char *null_pointer =NULL ; + +pr_info ("Suspend monitor timeout (timer is %d seconds)\n" , + (SUSPEND_TIMER_TIMEOUT_MS /1000 ));kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 115
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +show_state_filter (TASK_UNINTERRUPTIBLE ); + +if (timeout_count <2 ) +return ; + +if (is_console_suspended ()) +resume_console (); + +pr_info ("Trigger a panic\n" );Again, debugging code?
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 118
Quoted 3 lines
kernel/power/suspend.c +3
-0+ +/* Trigger a NULL pointer dereference */ + *null_pointer ='a' ;Are you sure this will work on all platforms? We do have a panic function if you really want to do that.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 128
Quoted 10 lines
kernel/power/suspend.c +10
-0+ +/* Should not reach here */ +pr_err ("Trigger panic failed!\n" ); +} + +static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 };Ick, no, call the scheduler functions properly, don't do this "by hand" ever.
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 132
Quoted 4 lines
kernel/power/suspend.c +4
-0+static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" );Again, debugging code :(
Quoted 68 lines
kernel/power/suspend.c +64
-0+ +sched_setscheduler (current,SCHED_FIFO , ¶m); + + timeout_count =0 ; + timeout =MAX_SCHEDULE_TIMEOUT ; + +do { +/* Wait suspend timer timeout */ + err =wait_event_interruptible_timeout ( + power_suspend_waitqueue, + (suspend_mon_toggle !=TOGGLE_NONE ), + timeout); + +mutex_lock (&suspend_mon_lock); +/* suspend monitor state change */ +if (suspend_mon_toggle !=TOGGLE_NONE ) { +if (suspend_mon_toggle ==TOGGLE_START ) { + timeout =msecs_to_jiffies ( +SUSPEND_TIMER_TIMEOUT_MS ); +pr_info ("Start suspend monitor\n" ); + }else if (suspend_mon_toggle ==TOGGLE_STOP ) { + timeout =MAX_SCHEDULE_TIMEOUT ; + timeout_count =0 ; +pr_info ("Stop suspend monitor\n" ); + } + suspend_mon_toggle =TOGGLE_NONE ; +mutex_unlock (&suspend_mon_lock); +continue ; + } +mutex_unlock (&suspend_mon_lock); + +/* suspend monitor event handler */ +if (err ==0 ) { + timeout_count++; +suspend_timeout (timeout_count); + }else if (err == -ERESTARTSYS ) { +pr_info ("Exit ksuspend_mon!" ); +break ; + } + }while (1 ); + + return 0; +} + +static void init_suspend_monitor_thread (void ) +{ +int ret; + + ksuspend_mon_tsk =kthread_create (suspend_monitor_kthread, +NULL ,"ksuspend_mon" ); +if (IS_ERR (ksuspend_mon_tsk)) { + ret =PTR_ERR (ksuspend_mon_tsk); + ksuspend_mon_tsk =NULL ; +pr_err ("Create suspend_monitor_kthread failed! ret = %d\n" , + ret); +return ; + } + + suspend_mon_toggle =TOGGLE_NONE ; +wake_up_process (ksuspend_mon_tsk); + +} +#endif +/** * pm_suspend_default_s2idle - Check if suspend-to-idle is the default suspend. * kernel/power/suspend.c static void s2idle_enter(void) · line 227
Quoted 6 lines
kernel/power/suspend.c +3
-0@@ -89,6 +222,10 @@ static void s2idle_enter(void) {trace_suspend_resume (TPS ("machine_suspend" ),PM_SUSPEND_TO_IDLE , true); +#ifdef CONFIG_PM_SLEEP_MONITOR +stop_suspend_mon (); +#endif Do not put #ifdef in .c files, that's not the proper kernel coding style. Especially for single function calls.
Quoted 3 lines
I've stopped here...
greg k-h
--
Embedded Software engineer
Signature · 1 line
Embedded Software engineer
Petr Mladek
maintainerQuoted 6 lines · Joseph Jang
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
A better solution would be to resume instead of panic().
Quoted 2 lines
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
This looks dangerous. It adds a simple way to panic() the system.
First, it should get enabled separately. e.g.CONFIG_TEST_PM_SLEEP_MONITOR.
Second, I would add it as a module that might get loaded and unloaded.
Quoted 46 lines
kernel/power/suspend.c +42
-0+static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 }; +static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" ); + +sched_setscheduler (current,SCHED_FIFO , ¶m); + + timeout_count =0 ; + timeout =MAX_SCHEDULE_TIMEOUT ; + + do { + /* Wait suspend timer timeout */ + err = wait_event_interruptible_timeout( + power_suspend_waitqueue, + (suspend_mon_toggle != TOGGLE_NONE), + timeout); + + mutex_lock(&suspend_mon_lock); + /* suspend monitor state change */ + if (suspend_mon_toggle != TOGGLE_NONE) { + if (suspend_mon_toggle == TOGGLE_START) { + timeout = msecs_to_jiffies( + SUSPEND_TIMER_TIMEOUT_MS); + pr_info("Start suspend monitor\n"); + } else if (suspend_mon_toggle == TOGGLE_STOP) { + timeout = MAX_SCHEDULE_TIMEOUT; + timeout_count = 0; + pr_info("Stop suspend monitor\n"); + } + suspend_mon_toggle = TOGGLE_NONE; + mutex_unlock(&suspend_mon_lock); + continue; + } + mutex_unlock(&suspend_mon_lock); + + /* suspend monitor event handler */ + if (err == 0) {
kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 175
Quoted 10 lines
kernel/power/suspend.c +10
-0+ timeout_count++; +suspend_timeout (timeout_count); + }else if (err == -ERESTARTSYS ) { +pr_info ("Exit ksuspend_mon!" ); +break ; + } + }while (1 ); + +return 0 ; +}
Using kthread looks like an overkill to me. I wonder how this actually works when the kthreads get freezed. It might be enough to implement just a timer callback. Start the timer in start_suspend_mon() and delete it in stop_suspend_mon(). Or do I miss anything?
Anyway, the kthread implementation looks a but hairy. If you really need to use kthread, I suggest to use kthread_worker API. You would need to run an init work to setup the RT scheduling. Then you could just call kthread_queue_delayed_work(()
and kthread_cancel_delayed_work_sync() to start and stop the monitor.
kernel/power/suspend.c static void s2idle_enter(void) · line 256
Quoted 7 lines
kernel/power/suspend.c +3
-0@@ -114,6 +251,10 @@ static void s2idle_enter(void) s2idle_state =S2IDLE_STATE_NONE ;raw_spin_unlock_irq (&s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +start_suspend_mon (); +#endif
It is better to solve this by defining start_suspend_mon() as empty function when the config option is disabled. For example, see how vgacon_text_force() is defined in console.h.
Best Regards,
Petr
Thanks Petr promptly response.
Quoted 6 lines · Joseph Jang
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
A better solution would be to resume instead of panic().
[Joseph] suspend_timeout() will trigger kernel panic() only when suspend thread stuck (deadlock/hang) for 2*30 seconds. At that moment, I don't know how to resume the suspend thread. So I just could trigger panic to reboot system.
If you have better suggestions, I am willing to study it.
Quoted 2 lines
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
This looks dangerous. It adds a simple way to panic() the system.
First, it should get enabled separately. e.g.CONFIG_TEST_PM_SLEEP_MONITOR.
Second, I would add it as a module that might get loaded and unloaded.
[Joseph] Agree to enable new compile flag for test module.
I think it is better to create separate patch for the new test module right?
kernel/power/suspend.c
Quoted 4 lines
kernel/power/suspend.c +0
-0
Using kthread looks like an overkill to me. I wonder how this actually works when the kthreads get freezed. It might be enough to implement just a timer callback. Start the timer in start_suspend_mon() and delete it in stop_suspend_mon(). Or do I miss anything?
Anyway, the kthread implementation looks a but hairy. If you really need to use kthread, I suggest to use kthread_worker API. You would need to run an init work to setup the RT scheduling. Then you could just call kthread_queue_delayed_work(()
and kthread_cancel_delayed_work_sync() to start and stop the monitor.
[Joseph]
Actually, I had ever think we just need to use add_timer()/del_timer_sync() for start_suspend_mon()/stop_suspend_mon() before.
But I am not sure if add_timer() may cause any performance impact in suspend thread or not.
So I try to create a suspend monitor kthread and just flip the flag in suspend thread.
Thank you,
Joseph.
kernel/power/suspend.c static void s2idle_enter(void) · line 256
Quoted 7 lines
kernel/power/suspend.c +3
-0@@ -114,6 +251,10 @@ static void s2idle_enter(void) s2idle_state =S2IDLE_STATE_NONE ;raw_spin_unlock_irq (&s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +start_suspend_mon (); +#endif
It is better to solve this by defining start_suspend_mon() as empty function when the config option is disabled. For example, see how vgacon_text_force() is defined in console.h.
[Joseph] Thank you for good suggestions.
May I know if I could use IS_ENABLED() ?
if (IS_ENABLED(CONFIG_PM_SLEEP_MONITOR))start_suspend_mon();
Best Regards,
Petr
Thank you,
Joseph.
Petr Mladek <[email protected]> 於 2020年10月16日 週五 下午5:01寫道:
Quoted 106 lines · Joseph Jang
Quoted 6 lines
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
A better solution would be to resume instead of
panic().Quoted 2 lines
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hangThis looks dangerous. It adds a simple way to
panic()the system.First, it should get enabled separately. e.g.
CONFIG_TEST_PM_SLEEP_MONITOR.Second, I would add it as a module that might get loaded and unloaded.
Quoted 46 lines
kernel/power/suspend.c +42
-0+static int suspend_monitor_kthread (void *arg) +{ +long err; +struct sched_param param = {.sched_priority + =MAX_RT_PRIO -1 }; +static int timeout_count; +static long timeout; + +pr_info ("Init ksuspend_mon thread\n" ); + +sched_setscheduler (current,SCHED_FIFO , ¶m); + + timeout_count =0 ; + timeout =MAX_SCHEDULE_TIMEOUT ; + + do { + /* Wait suspend timer timeout */ + err = wait_event_interruptible_timeout( + power_suspend_waitqueue, + (suspend_mon_toggle != TOGGLE_NONE), + timeout); + + mutex_lock(&suspend_mon_lock); + /* suspend monitor state change */ + if (suspend_mon_toggle != TOGGLE_NONE) { + if (suspend_mon_toggle == TOGGLE_START) { + timeout = msecs_to_jiffies( + SUSPEND_TIMER_TIMEOUT_MS); + pr_info("Start suspend monitor\n"); + } else if (suspend_mon_toggle == TOGGLE_STOP) { + timeout = MAX_SCHEDULE_TIMEOUT; + timeout_count = 0; + pr_info("Stop suspend monitor\n"); + } + suspend_mon_toggle = TOGGLE_NONE; + mutex_unlock(&suspend_mon_lock); + continue; + } + mutex_unlock(&suspend_mon_lock); + + /* suspend monitor event handler */ + if (err == 0) {kernel/power/suspend.c static DECLARE_SWAIT_QUEUE_HEAD(s2idle_wait_head); · line 175
Quoted 10 lines
kernel/power/suspend.c +10
-0+ timeout_count++; +suspend_timeout (timeout_count); + }else if (err == -ERESTARTSYS ) { +pr_info ("Exit ksuspend_mon!" ); +break ; + } + }while (1 ); + +return 0 ; +}Using kthread looks like an overkill to me. I wonder how this actually works when the kthreads get freezed. It might be enough to implement just a timer callback. Start the timer in
start_suspend_mon()and delete it instop_suspend_mon(). Or do I miss anything?Anyway, the kthread implementation looks a but hairy. If you really need to use kthread, I suggest to use
kthread_workerAPI. You would need to run an init work to setup the RT scheduling. Then you could just call kthread_queue_delayed_work(()
andkthread_cancel_delayed_work_sync()to start and stop the monitor.kernel/power/suspend.c static void s2idle_enter(void) · line 256
Quoted 7 lines
kernel/power/suspend.c +3
-0@@ -114,6 +251,10 @@ static void s2idle_enter(void) s2idle_state =S2IDLE_STATE_NONE ;raw_spin_unlock_irq (&s2idle_lock); +#ifdef CONFIG_PM_SLEEP_MONITOR +start_suspend_mon (); +#endif It is better to solve this by defining
start_suspend_mon()as empty function when the config option is disabled. For example, see howvgacon_text_force()is defined inconsole.h.Best Regards,
Petr
Signature · 1 line
Embedded Software engineer
Rafael J. Wysocki
maintainerQuoted 10 lines · Petr Mladek
Quoted 6 lines · Joseph Jang
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
A better solution would be to resume instead of
panic().
Well, abort the suspend if it happens during suspend or continue if it happens during resume,
But we have a suspend watchdog already, don't we?
Rafael J. Wysocki
maintainerQuoted 39 lines · Joseph Jang
From: josephjang <[email protected]>
Add suspend timeout handler to prevent device stuck during suspend/ resume process. Suspend timeout handler will dump disk sleep task at first round timeout and trigger kernel panic at second round timeout. The default timer for each round is 30 seconds.
Note: Can use following command to simulate suspend hang for testing.
adb shell echo 1 >/sys/power/pm_hang
adb shell echo mem >/sys/power/stateSigned-off-by josephjang <[email protected]>--- include/linux/console.h | 1 + kernel/power/Kconfig | 9 +++ kernel/power/main.c | 66 ++++++++++++++++ kernel/power/suspend.c | 162 ++++++++++++++++++++++++++++++++++++++++ kernel/printk/printk.c | 5 ++ 5 files changed, 243 insertions(+)include/linux/console.h +1
-0@@ -192,6 +192,7 @@ static inline void console_sysfs_notify(void) { }#endif extern bool console_suspend_enabled; +extern int is_console_suspended (void );/* Suspend and resume console messages over PM events */ extern void suspend_console (void );kernel/power/Kconfig +0
-0@@ -207,6 +207,15 @@ config PM_SLEEP_DEBUG def_bool y depends on PM_DEBUG && PM_SLEEP
kernel/power/Kconfig config PM_SLEEP_DEBUG · line 218
Quoted 10 lines
kernel/power/Kconfig +9
-0+config PM_SLEEP_MONITOR + bool "Linux kernel suspend/resume process monitor" + depends on PM_SLEEP + help + This option will enable suspend/resume monitor to prevent device + stuck during suspend/resume process. Suspend timeout handler will + dump disk sleep task at first round timeout and trigger kernel panic + at second round timeout. The default timer for each round is 30 seconds. +
The facility associated with the Kconfig entry right below is supposed to do exactly the same thing.
What's the reason to add another one? What is missing?
Quoted 3 lines
kernel/power/Kconfig +0
-0config DPM_WATCHDOG bool "Device suspend/resume watchdog" depends on PM_DEBUG && PSTORE && EXPERT