mmc: core: Fix system shutdown hang in mmc_bus_shutdown
Subsystems (2)
PATCH RESEND 1/1 mmc: core: Fix system shutdown hang in mmc_bus_shutdown
During system shutdown, mmc_bus_shutdown() calls __mmc_stop_host() which uses cancel_delayed_work_sync(). This can block indefinitely if the work queue is stuck, causing the system to hang during shutdown.
This patch introduces a new function __mmc_stop_host_no_sync() that skips the synchronous work cancellation, preventing potential shutdown hangs. The function is used in mmc_bus_shutdown() where blocking is not acceptable during system shutdown.
Changes:
- Add __mmc_stop_host_no_sync() function that avoids cancel_delayed_work_sync()
- Update mmc_bus_shutdown() to use the new non-blocking function
- Keep the original __mmc_stop_host() unchanged for normal operation
This ensures graceful system shutdown while maintaining existing functionality for regular MMC host operations.
stack information when an error occurs:
INFO: task init:1 blocked for more than 720 seconds.
Tainted: G OE 5.15.185-android13-8-00043-gd00fb6bce7ed-ab13792018 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:init state:D stack: 0 pid: 1 ppid: 0 flags:0x04000008
Call trace:
__switch_to+0x234/0x470
__schedule+0x694/0xb8c
schedule+0x150/0x254
schedule_timeout+0x48/0x138
wait_for_common+0x144/0x308
__flush_work+0x3d8/0x508
__cancel_work_timer+0x120/0x2e8
mmc_bus_shutdown+0x90/0x158
device_shutdown+0x204/0x434
kernel_restart+0x54/0x220
kernel_restart+0x0/0x220
invoke_syscall+0x60/0x150
el0_svc_common+0xb8/0xf8
do_el0_svc+0x28/0x98
el0_svc+0x24/0x84
el0t_64_sync_handler+0x88/0xec
el0t_64_sync+0x1b8/0x1bc
INFO: task kworker/1:1:73 blocked for more than 721 seconds.
Tainted: G OE 5.15.185-android13-8-00043-gd00fb6bce7ed-ab13792018 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/1:1 state:D stack: 0 pid: 73 ppid: 2 flags:0x00000008
Workqueue: events_freezable mmc_rescan.cfi_jt
Call trace:
__switch_to+0x234/0x470
__schedule+0x694/0xb8c
schedule+0x150/0x254
schedule_preempt_disabled+0x2c/0x4c
__mutex_lock+0x360/0xb00
__mutex_lock_slowpath+0x18/0x28
mutex_lock+0x48/0x12c
device_del+0x48/0x8d0
mmc_remove_card+0x128/0x158
mmc_sdio_remove+0x190/0x1ac
mmc_sdio_detect+0x7c/0x118
mmc_rescan+0xe8/0x42c
process_one_work+0x248/0x55c
worker_thread+0x3b0/0x740
kthread+0x168/0x1dc
ret_from_fork+0x10/0x20--- drivers/mmc/core/bus.c | 2 +- drivers/mmc/core/core.c | 14 ++++++++++++++ drivers/mmc/core/core.h | 1 + 3 files changed, 16 insertions(+), 1 deletion(-)
drivers/mmc/core/bus.c +1 -1
@@ -149,7 +149,7 @@ static void mmc_bus_shutdown(struct device *dev)if (dev->driver && drv->shutdown) drv->shutdown (card); -__mmc_stop_host (host); +__mmc_stop_host_no_sync (host);if (host->bus_ops->shutdown) { ret = host->bus_ops->shutdown (host);
drivers/mmc/core/core.c +14 -0
@@ -2336,6 +2336,20 @@ void __mmc_stop_host(struct mmc_host *host)cancel_delayed_work_sync (&host->detect); } +void __mmc_stop_host_no_sync (struct mmc_host *host) +{ +if (host->rescan_disable) +return ; + +if (host->slot.cd_irq >=0 ) { +mmc_gpio_set_cd_wake (host, false); +disable_irq (host->slot.cd_irq); + } + + host->rescan_disable =1 ; +/* Skip cancel_delayed_work_sync to avoid potential blocking */ +} +void mmc_stop_host (struct mmc_host *host) {__mmc_stop_host (host);
drivers/mmc/core/core.h +1 -0
@@ -71,6 +71,7 @@ static inline void mmc_delay(unsigned int ms)void mmc_rescan (struct work_struct *work);void mmc_start_host (struct mmc_host *host);void __mmc_stop_host (struct mmc_host *host); +void __mmc_stop_host_no_sync (struct mmc_host *host);void mmc_stop_host (struct mmc_host *host); void _mmc_detect_change(struct mmc_host *host, unsigned long delay,
Signature · 1 line
2.29.0
Hi Michael,
thanks for your patch!
Quoted 11 lines · Michael Wu
During system shutdown,
mmc_bus_shutdown()calls__mmc_stop_host()which usescancel_delayed_work_sync(). This can block indefinitely if the work queue is stuck, causing the system to hang during shutdown.This patch introduces a new function
__mmc_stop_host_no_sync()that skips the synchronous work cancellation, preventing potential shutdown hangs. The function is used inmmc_bus_shutdown()where blocking is not acceptable during system shutdown.Changes:
- Add__mmc_stop_host_no_sync()function that avoidscancel_delayed_work_sync()
Why is this function prefixed with __two underscores? The __mmc_stop_host is named like that because there is another function with the same name, but here is no mmc_stop_host_no_sync() so just name it without underscores.
Yours,
Linus Walleij
Quoted 6 lines · Michael Wu
During system shutdown,
mmc_bus_shutdown()calls__mmc_stop_host()which usescancel_delayed_work_sync(). This can block indefinitely if the work queue is stuck, causing the system to hang during shutdown.This patch introduces a new function
__mmc_stop_host_no_sync()that skips the synchronous work cancellation, preventing potential shutdown hangs.
mmc core must ensure there are no ongoing operations racing with shutdown. Leaving the work running does not look like it would achieve that.
Perhaps it can be cancelled earlier? There seems to be a "reboot" notifier associated with shutdown, refer reboot_notifier_list and register_reboot_notifier(). Note, in that case, it is also necessary to ensure nothing can queue the work again.
Quoted 114 lines
The function is used in
mmc_bus_shutdown()where blocking is not acceptable during system shutdown.Changes:
- Add__mmc_stop_host_no_sync()function that avoidscancel_delayed_work_sync()
- Updatemmc_bus_shutdown()to use the new non-blocking function
- Keep the original__mmc_stop_host()unchanged for normal operationThis ensures graceful system shutdown while maintaining existing functionality for regular MMC host operations.
stack information when an error occurs:
INFO: task init:1 blocked for more than 720 seconds. Tainted: G OE 5.15.185-android13-8-00043-gd00fb6bce7ed-ab13792018 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:init state:D stack: 0 pid: 1 ppid: 0 flags:0x04000008 Call trace: __switch_to+0x234/0x470 __schedule+0x694/0xb8c schedule+0x150/0x254 schedule_timeout+0x48/0x138 wait_for_common+0x144/0x308 __flush_work+0x3d8/0x508 __cancel_work_timer+0x120/0x2e8 mmc_bus_shutdown+0x90/0x158 device_shutdown+0x204/0x434 kernel_restart+0x54/0x220 kernel_restart+0x0/0x220 invoke_syscall+0x60/0x150 el0_svc_common+0xb8/0xf8 do_el0_svc+0x28/0x98 el0_svc+0x24/0x84 el0t_64_sync_handler+0x88/0xec el0t_64_sync+0x1b8/0x1bc INFO: task kworker/1:1:73 blocked for more than 721 seconds. Tainted: G OE 5.15.185-android13-8-00043-gd00fb6bce7ed-ab13792018 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/1:1 state:D stack: 0 pid: 73 ppid: 2 flags:0x00000008 Workqueue: events_freezable mmc_rescan.cfi_jt Call trace: __switch_to+0x234/0x470 __schedule+0x694/0xb8c schedule+0x150/0x254 schedule_preempt_disabled+0x2c/0x4c __mutex_lock+0x360/0xb00 __mutex_lock_slowpath+0x18/0x28 mutex_lock+0x48/0x12c device_del+0x48/0x8d0 mmc_remove_card+0x128/0x158 mmc_sdio_remove+0x190/0x1ac mmc_sdio_detect+0x7c/0x118 mmc_rescan+0xe8/0x42c process_one_work+0x248/0x55c worker_thread+0x3b0/0x740 kthread+0x168/0x1dc ret_from_fork+0x10/0x20Signed-off-by Michael Wu <[email protected]>--- drivers/mmc/core/bus.c | 2 +- drivers/mmc/core/core.c | 14 ++++++++++++++ drivers/mmc/core/core.h | 1 + 3 files changed, 16 insertions(+), 1 deletion(-)drivers/mmc/core/bus.c +1
-1@@ -149,7 +149,7 @@ static void mmc_bus_shutdown(struct device *dev)if (dev->driver && drv->shutdown) drv->shutdown (card); -__mmc_stop_host (host); +__mmc_stop_host_no_sync (host);if (host->bus_ops->shutdown) { ret = host->bus_ops->shutdown (host);drivers/mmc/core/core.c +14
-0@@ -2336,6 +2336,20 @@ void __mmc_stop_host(struct mmc_host *host)cancel_delayed_work_sync (&host->detect); } +void __mmc_stop_host_no_sync (struct mmc_host *host) +{ +if (host->rescan_disable) +return ; + +if (host->slot.cd_irq >=0 ) { +mmc_gpio_set_cd_wake (host, false); +disable_irq (host->slot.cd_irq); + } + + host->rescan_disable =1 ; +/* Skip cancel_delayed_work_sync to avoid potential blocking */ +} +void mmc_stop_host (struct mmc_host *host) {__mmc_stop_host (host);drivers/mmc/core/core.h +1
-0@@ -71,6 +71,7 @@ static inline void mmc_delay(unsigned int ms)void mmc_rescan (struct work_struct *work);void mmc_start_host (struct mmc_host *host);void __mmc_stop_host (struct mmc_host *host); +void __mmc_stop_host_no_sync (struct mmc_host *host);void mmc_stop_host (struct mmc_host *host); void _mmc_detect_change(struct mmc_host *host, unsigned long delay,
Hello,
The execution timing of this host->detect work actually depends on when the WiFi driver executes its unload procedure. This is something we cannot control and cannot handle using reboot_notifier_list and register_reboot_notifier() operations. Moreover, different WiFi drivers may have different handling sequences for this timing.
From this perspective, we believe it's not convenient to address the issue in that way.
We noticed that the mmc_bus_shutdown function calling cancel_delayed_work_sync was newly added. Therefore, we suggest removing this part first.
Quoted 2 lines · Adrian Hunter
reboot_notifier_listandregister_reboot_notifier()
Signature · 2 lines
Regards, Michael Wu
Quoted 3 lines · Michael Wu
Hello,
The execution timing of this
host->detectwork actually depends on when the WiFi driver executes its unload procedure.
Devices should not really be removed during shutdown. The requirement for shutdown is to ensure there are no on-going operations and to prepare the hardware for power-off.
So this sounds like the WiFi driver should not be executing an unload procedure at shutdown.
Quoted 10 lines
This is something we cannot control and cannot handle using
reboot_notifier_listandregister_reboot_notifier()operations. Moreover, different WiFi drivers may have different handling sequences for this timing.From this perspective, we believe it's not convenient to address the issue in that way.
We noticed that the
mmc_bus_shutdownfunction callingcancel_delayed_work_syncwas newly added. Therefore, we suggest removing this part first.Quoted 2 lines · Adrian Hunter
reboot_notifier_listandregister_reboot_notifier()