fs: Use a seperate wq for do_sync_work() to avoid a potential deadlock
RFC PATCH 1/1 fs: Use a seperate wq for do_sync_work() to avoid a potential deadlock
Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():
* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.
* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.
This patch introduces a separate workqueue for do_sync_work() to avoid a the described deadlock.
--- fs/sync.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
fs/sync.c +4 -9
@@ -15,6 +15,7 @@#include <linux/pagemap.h> #include <linux/quotaops.h> #include <linux/backing-dev.h> +#include <linux/kthread.h> #include "internal.h" #define VALID_FLAGS (SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE| \
@@ -114,7 +115,7 @@ SYSCALL_DEFINE0(sync)return 0 ; } -static void do_sync_work (struct work_struct *work) +static int do_sync_work (void *dummy) {int nowait =0 ;
@@ -129,18 +130,12 @@ static void do_sync_work(struct work_struct *work)iterate_supers (sync_fs_one_sb, &nowait);iterate_bdevs (fdatawrite_one_bdev,NULL );printk ("Emergency Sync complete\n" ); -kfree (work); +return 0 ; }void emergency_sync (void ) { -struct work_struct *work; - - work =kmalloc (sizeof (*work),GFP_ATOMIC ); -if (work) { -INIT_WORK (work, do_sync_work); -schedule_work (work); - } +kthread_run (do_sync_work,NULL ,"sync_work_thread" ); }/*
Signature · 1 line
1.9.3
Quoted 6 lines · Aaron Tomlin
Since
do_sync_work()is a deferred function it can block indefinitely by design. At presentdo_sync_work()is added to the globalsystem_wq. As such a deadlock is theoretically possible betweensys_unmount()andsync_filesystems():
Quoted 7 lines
* The current work fn on the
system_wq(do_sync_work()) is blocked waiting to aquire a sb'ss_umountfor reading.* The "umount" task is the current owner of the
s_umountin question but is waiting fordo_sync_work()to continue. Thus we hit a deadlock situation.
I can't comment the patches in this area, but I am just curious...
Could you explain this deadlock in more details? I simply can't understand what "waiting for do_sync_work()" actually means.
Quoted 2 lines
This patch introduces a separate workqueue for
do_sync_work()to avoid a the described deadlock.
The subject and the changelog do not match the patch, it doesn't add/use another workqueue.
Oleg.
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin
Since
do_sync_work()is a deferred function it can block indefinitely by design. At presentdo_sync_work()is added to the globalsystem_wq. As such a deadlock is theoretically possible betweensys_unmount()andsync_filesystems():* The current work fn on the
system_wq(do_sync_work()) is blocked waiting to aquire a sb'ss_umountfor reading.
Quoted 8 lines
Quoted 4 lines
* The "umount" task is the current owner of the
s_umountin question but is waiting fordo_sync_work()to continue. Thus we hit a deadlock situation.I can't comment the patches in this area, but I am just curious...
Could you explain this deadlock in more details? I simply can't understand what "waiting for
do_sync_work()" actually means.
Hopefully this helps:
"umount" "events/1"
sys_umount sysrq_handle_sync
deactivate_super(sb) emergency_sync
{ schedule_work(work)
... queue_work(system_wq, work)
down_write(&s->s_umount) do_sync_work(work)
... sync_filesystems(0)
kill_block_super(s) ...
generic_shutdown_super(sb) down_read(&sb->s_umount)
// sop->put_super(sb)
ext4_put_super(sb)
invalidate_bdev(sb->s_bdev)
lru_add_drain_all()
for_each_online_cpu(cpu) {
schedule_work_on(cpu, work)
queue_work_on(cpu, system_wq, work)
...
}}
- Both lru_add_drain and do_sync_work work items are added to the same global system_wq
- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading
- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.
Signature · 1 line
Aaron Tomlin
Quoted 46 lines · Aaron Tomlin
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin
Since
do_sync_work()is a deferred function it can block indefinitely by design. At presentdo_sync_work()is added to the globalsystem_wq. As such a deadlock is theoretically possible betweensys_unmount()andsync_filesystems():* The current work fn on the
system_wq(do_sync_work()) is blocked waiting to aquire a sb'ss_umountfor reading.Quoted 8 lines
Quoted 4 lines
* The "umount" task is the current owner of the
s_umountin question but is waiting fordo_sync_work()to continue. Thus we hit a deadlock situation.I can't comment the patches in this area, but I am just curious...
Could you explain this deadlock in more details? I simply can't understand what "waiting for
do_sync_work()" actually means.Hopefully this helps:
"umount" "events/1"
sys_umountsysrq_handle_sync
deactivate_super(sb)emergency_sync
{schedule_work(work)... queue_work(system_wq, work) down_write(&s->s_umount) do_sync_work(work) ... sync_filesystems(0) kill_block_super(s) ... generic_shutdown_super(sb) down_read(&sb->s_umount) // sop->put_super(sb) ext4_put_super(sb) invalidate_bdev(sb->s_bdev) lru_add_drain_all() for_each_online_cpu(cpu) { schedule_work_on(cpu, work) queue_work_on(cpu, system_wq, work) ... }}
- Both
lru_add_drainanddo_sync_workwork items are added to the same globalsystem_wq
Quoted 6 lines
- The current work fn on the
system_wqisdo_sync_workand is blocked waiting to aquire an sb'ss_umountfor reading- The umount task is the current owner of the
s_umountin question but is waiting fordo_sync_workto continue. Thus we hit a deadlock situation.
What kernel did you see this deadlock on?
I don't see a deadlock here on a mainline kernel. The emergency sync work blocks, the new work gets queued, and the workqueue infrastructure simply pulls another kworker thread from the pool and runs the new work. IOWs, I can't see how this would deadlock unless the system_wq kworker pool has been fully depleted it's defined per-cpu concurrency depth. If the kworker thread pool is depleted then you have bigger problems than emergency sync not deadlocking....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 11 lines · Dave Chinner
Quoted 9 lines
- Both
lru_add_drainanddo_sync_workwork items are added to the same globalsystem_wq- The current work fn on the
system_wqisdo_sync_workand is blocked waiting to aquire an sb'ss_umountfor reading- The umount task is the current owner of the
s_umountin question but is waiting fordo_sync_workto continue. Thus we hit a deadlock situation.What kernel did you see this deadlock on?
Sorry for the noise. This deadlock was produced under a kernel whereby the workqueue implementation is significantly less sophisticated.
Signature · 1 line
Aaron Tomlin
Quoted 42 lines · Aaron Tomlin
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin
Since
do_sync_work()is a deferred function it can block indefinitely by design. At presentdo_sync_work()is added to the globalsystem_wq. As such a deadlock is theoretically possible betweensys_unmount()andsync_filesystems():* The current work fn on the
system_wq(do_sync_work()) is blocked waiting to aquire a sb'ss_umountfor reading.Quoted 8 lines
Quoted 4 lines
* The "umount" task is the current owner of the
s_umountin question but is waiting fordo_sync_work()to continue. Thus we hit a deadlock situation.I can't comment the patches in this area, but I am just curious...
Could you explain this deadlock in more details? I simply can't understand what "waiting for
do_sync_work()" actually means.Hopefully this helps:
"umount" "events/1"
sys_umountsysrq_handle_sync
deactivate_super(sb)emergency_sync
{schedule_work(work)... queue_work(system_wq, work) down_write(&s->s_umount) do_sync_work(work) ... sync_filesystems(0) kill_block_super(s) ... generic_shutdown_super(sb) down_read(&sb->s_umount) // sop->put_super(sb) ext4_put_super(sb) invalidate_bdev(sb->s_bdev) lru_add_drain_all() for_each_online_cpu(cpu) { schedule_work_on(cpu, work) queue_work_on(cpu, system_wq, work) ... }
Quoted 4 lines
}
- Both
lru_add_drainanddo_sync_workwork items are added to the same globalsystem_wq
Aha. Perhaps you hit this bug under the older kernel?
"same workqueue" doesn't mean "same worker thread" today, every CPU can run up to ->max_active works. And for system_wq uses max_active = 256.
Quoted 2 lines
- The current work fn on the
system_wqisdo_sync_workand is blocked waiting to aquire an sb'ss_umountfor reading
OK,
Quoted 3 lines
- The umount task is the current owner of the
s_umountin question but is waiting fordo_sync_workto continue. Thus we hit a deadlock situation.
I don't this this can happen, another worker threaf from worker_pool can handle this work.
Oleg.
Quoted 23 lines · Oleg Nesterov
Quoted 23 lines
Hopefully this helps:
"umount" "events/1"
sys_umountsysrq_handle_sync
deactivate_super(sb)emergency_sync
{schedule_work(work)... queue_work(system_wq, work) down_write(&s->s_umount) do_sync_work(work) ... sync_filesystems(0) kill_block_super(s) ... generic_shutdown_super(sb) down_read(&sb->s_umount) // sop->put_super(sb) ext4_put_super(sb) invalidate_bdev(sb->s_bdev) lru_add_drain_all() for_each_online_cpu(cpu) { schedule_work_on(cpu, work) queue_work_on(cpu, system_wq, work) ... }}
Quoted 4 lines
Quoted 2 lines
- Both
lru_add_drainanddo_sync_workwork items are added to the same globalsystem_wqAha. Perhaps you hit this bug under the older kernel?
I did. Sorry for the noise.
Quoted 6 lines
"same workqueue" doesn't mean "same worker thread" today, every CPU can run up to ->max_active works. And for
system_wqusesmax_active= 256.Quoted 2 lines
- The current work fn on the
system_wqisdo_sync_workand is blocked waiting to aquire an sb'ss_umountfor reading
Quoted 8 lines
OK,
Quoted 3 lines
- The umount task is the current owner of the
s_umountin question but is waiting fordo_sync_workto continue. Thus we hit a deadlock situation.I don't this this can happen, another worker threaf from
worker_poolcan handle this work.
Understood.
Signature · 1 line
Aaron Tomlin