Skip to contentSkip to conversation

fs: Use a seperate wq for do_sync_work() to avoid a potential deadlock

RFC PATCH In review Aaron Tomlin 3 participants 7 messages Sep 17, 2014 – Sep 19, 2014 VFS

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 fs: Use a seperate wq for do_sync_work() to avoid a potential deadlock

Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():

* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.

* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.

This patch introduces a separate workqueue for do_sync_work() to avoid a the described deadlock.

---
 fs/sync.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
fs/sync.c +4 -9
index bdc729d..df455d0 100644
@@ -15,6 +15,7 @@
 #include <linux/pagemap.h>
 #include <linux/quotaops.h>
 #include <linux/backing-dev.h>
+#include <linux/kthread.h>
 #include "internal.h"
 
 #define VALID_FLAGS (SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE| \
@@ -114,7 +115,7 @@ SYSCALL_DEFINE0(sync)
 	return 0;
 }
 
-static void do_sync_work(struct work_struct *work)
+static int do_sync_work(void *dummy)
 {
 	int nowait = 0;
 
@@ -129,18 +130,12 @@ static void do_sync_work(struct work_struct *work)
 	iterate_supers(sync_fs_one_sb, &nowait);
 	iterate_bdevs(fdatawrite_one_bdev, NULL);
 	printk("Emergency Sync complete\n");
-	kfree(work);
+	return 0;
 }
 
 void emergency_sync(void)
 {
-	struct work_struct *work;
-
-	work = kmalloc(sizeof(*work), GFP_ATOMIC);
-	if (work) {
-		INIT_WORK(work, do_sync_work);
-		schedule_work(work);
-	}
+	kthread_run(do_sync_work, NULL, "sync_work_thread");
 }
 
 /*
Signature · 1 line
1.9.3
#1 lore raw
Quoted 6 lines · Aaron Tomlin

Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():

Quoted 7 lines

* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.

* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.

I can't comment the patches in this area, but I am just curious...

Could you explain this deadlock in more details? I simply can't understand what "waiting for do_sync_work()" actually means.

Quoted 2 lines

This patch introduces a separate workqueue for do_sync_work() to avoid a the described deadlock.

The subject and the changelog do not match the patch, it doesn't add/use another workqueue.

Oleg.

#2 lore raw
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin

Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():

* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.

Quoted 8 lines
Quoted 4 lines

* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.

I can't comment the patches in this area, but I am just curious...

Could you explain this deadlock in more details? I simply can't understand what "waiting for do_sync_work()" actually means.

Hopefully this helps:

"umount" "events/1"

sys_umount sysrq_handle_sync
deactivate_super(sb) emergency_sync
{ schedule_work(work)

    ...						    	  queue_work(system_wq, work)
    down_write(&s->s_umount)			    	    do_sync_work(work)
    ...						      	      sync_filesystems(0)
    kill_block_super(s)				    		...
      generic_shutdown_super(sb)		    		down_read(&sb->s_umount)
      // sop->put_super(sb)
      ext4_put_super(sb)
	invalidate_bdev(sb->s_bdev)
	  lru_add_drain_all()
	    for_each_online_cpu(cpu) {
	      schedule_work_on(cpu, work)
		queue_work_on(cpu, system_wq, work)
		...
	    }

}

- Both lru_add_drain and do_sync_work work items are added to the same global system_wq

- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading

- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.

Signature · 1 line
Aaron Tomlin
#3 lore raw
Quoted 46 lines · Aaron Tomlin
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin

Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():

* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.

Quoted 8 lines
Quoted 4 lines

* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.

I can't comment the patches in this area, but I am just curious...

Could you explain this deadlock in more details? I simply can't understand what "waiting for do_sync_work()" actually means.

Hopefully this helps:

"umount" "events/1"

sys_umount sysrq_handle_sync
deactivate_super(sb) emergency_sync
{ schedule_work(work)

    ...						    	  queue_work(system_wq, work)
    down_write(&s->s_umount)			    	    do_sync_work(work)
    ...						      	      sync_filesystems(0)
    kill_block_super(s)				    		...
      generic_shutdown_super(sb)		    		down_read(&sb->s_umount)
      // sop->put_super(sb)
      ext4_put_super(sb)
	invalidate_bdev(sb->s_bdev)
	  lru_add_drain_all()
	    for_each_online_cpu(cpu) {
	      schedule_work_on(cpu, work)
		queue_work_on(cpu, system_wq, work)
		...
	    }

}

- Both lru_add_drain and do_sync_work work items are added to the same global system_wq

Quoted 6 lines

- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading

- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.

What kernel did you see this deadlock on?

I don't see a deadlock here on a mainline kernel. The emergency sync work blocks, the new work gets queued, and the workqueue infrastructure simply pulls another kworker thread from the pool and runs the new work. IOWs, I can't see how this would deadlock unless the system_wq kworker pool has been fully depleted it's defined per-cpu concurrency depth. If the kworker thread pool is depleted then you have bigger problems than emergency sync not deadlocking....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw
Quoted 11 lines · Dave Chinner
Quoted 9 lines

- Both lru_add_drain and do_sync_work work items are added to the same global system_wq

- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading

- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.

What kernel did you see this deadlock on?

Sorry for the noise. This deadlock was produced under a kernel whereby the workqueue implementation is significantly less sophisticated.

Signature · 1 line
Aaron Tomlin
#5 lore raw
In reply to Aaron Tomlin
Quoted 42 lines · Aaron Tomlin
Quoted 10 lines · Oleg Nesterov
Quoted 9 lines · Aaron Tomlin

Since do_sync_work() is a deferred function it can block indefinitely by design. At present do_sync_work() is added to the global system_wq. As such a deadlock is theoretically possible between sys_unmount() and sync_filesystems():

* The current work fn on the system_wq (do_sync_work()) is blocked waiting to aquire a sb's s_umount for reading.

Quoted 8 lines
Quoted 4 lines

* The "umount" task is the current owner of the s_umount in question but is waiting for do_sync_work() to continue. Thus we hit a deadlock situation.

I can't comment the patches in this area, but I am just curious...

Could you explain this deadlock in more details? I simply can't understand what "waiting for do_sync_work()" actually means.

Hopefully this helps:

"umount" "events/1"

sys_umount sysrq_handle_sync
deactivate_super(sb) emergency_sync
{ schedule_work(work)

    ...						    	  queue_work(system_wq, work)
    down_write(&s->s_umount)			    	    do_sync_work(work)
    ...						      	      sync_filesystems(0)
    kill_block_super(s)				    		...
      generic_shutdown_super(sb)		    		down_read(&sb->s_umount)
      // sop->put_super(sb)
      ext4_put_super(sb)
	invalidate_bdev(sb->s_bdev)
	  lru_add_drain_all()
	    for_each_online_cpu(cpu) {
	      schedule_work_on(cpu, work)
		queue_work_on(cpu, system_wq, work)
		...
	    }
Quoted 4 lines

}

- Both lru_add_drain and do_sync_work work items are added to the same global system_wq

Aha. Perhaps you hit this bug under the older kernel?

"same workqueue" doesn't mean "same worker thread" today, every CPU can run up to ->max_active works. And for system_wq uses max_active = 256.

Quoted 2 lines

- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading

OK,

Quoted 3 lines

- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.

I don't this this can happen, another worker threaf from worker_pool can handle this work.

Oleg.

#6 lore raw
Quoted 23 lines · Oleg Nesterov
Quoted 23 lines

Hopefully this helps:

"umount" "events/1"

sys_umount sysrq_handle_sync
deactivate_super(sb) emergency_sync
{ schedule_work(work)

    ...						  queue_work(system_wq, work)
    down_write(&s->s_umount)			    do_sync_work(work)
    ...						      sync_filesystems(0)
    kill_block_super(s)				    	...
      generic_shutdown_super(sb)		    		down_read(&sb->s_umount)
      // sop->put_super(sb)
      ext4_put_super(sb)
	invalidate_bdev(sb->s_bdev)
	  lru_add_drain_all()
	    for_each_online_cpu(cpu) {
	      schedule_work_on(cpu, work)
		queue_work_on(cpu, system_wq, work)
		...
	    }

}

Quoted 4 lines
Quoted 2 lines

- Both lru_add_drain and do_sync_work work items are added to the same global system_wq

Aha. Perhaps you hit this bug under the older kernel?

I did. Sorry for the noise.

Quoted 6 lines

"same workqueue" doesn't mean "same worker thread" today, every CPU can run up to ->max_active works. And for system_wq uses max_active = 256.

Quoted 2 lines

- The current work fn on the system_wq is do_sync_work and is blocked waiting to aquire an sb's s_umount for reading

Quoted 8 lines

OK,

Quoted 3 lines

- The umount task is the current owner of the s_umount in question but is waiting for do_sync_work to continue. Thus we hit a deadlock situation.

I don't this this can happen, another worker threaf from worker_pool can handle this work.

Understood.

Signature · 1 line
Aaron Tomlin