Skip to contentSkip to conversation

loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

PATCH Kay Sievers 2 participants 4 messages Jul 30, 2011 – Jul 31, 2011 Block layer

lore
Share & tools
lore

Kay Sievers

author
#0 lore raw

From: Kay Sievers <[email protected]>
Subject: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

LOOP_CLR_FD takes lo->lo_ctl_mutex and tries to remove the loop sysfs files. Sysfs calls show() and waits for lo->lo_ctl_mutex. LOOP_CLR_FD waits for show() to finish to remove the sysfs file.

cat /sys/class/block/loop0/loop/backing_file

    mutex_lock_nested+0x176/0x350
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    dev_attr_show+0x1b/0x60
    ? sysfs_read_file+0x86/0x1a0
    ? __get_free_pages+0x12/0x50
    sysfs_read_file+0xaf/0x1a0

ioctl(LOOP_CLR_FD):

    wait_for_common+0x12c/0x180
    ? try_to_wake_up+0x2a0/0x2a0
    wait_for_completion+0x18/0x20
    sysfs_deactivate+0x178/0x180
    ? sysfs_addrm_finish+0x43/0x70
    ? sysfs_addrm_start+0x1d/0x20
    sysfs_addrm_finish+0x43/0x70
    sysfs_hash_and_remove+0x85/0xa0
    sysfs_remove_group+0x59/0x100
    loop_clr_fd+0x1dc/0x3f0 [loop]
    lo_ioctl+0x223/0x7a0 [loop]

Instead of taking the lo_ctl_mutex from sysfs code, take the inner lo->lo_lock, to protect the access to the backing_file data.

Thanks to Tejun for help debugging and finding a solution.

Cc Milan Broz <[email protected]>
Cc Tejun Heo <[email protected]>
Signed-off-by Kay Sievers <[email protected]>
---
 drivers/block/loop.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/block/loop.c +4 -2
@@ -743,10 +743,10 @@ static ssize_t loop_attr_backing_file_sh
 	ssize_t ret;
 	char *p = NULL;
 
-	mutex_lock(&lo->lo_ctl_mutex);
+	spin_lock_irq(&lo->lo_lock);
 	if (lo->lo_backing_file)
 		p = d_path(&lo->lo_backing_file->f_path, buf, PAGE_SIZE - 1);
-	mutex_unlock(&lo->lo_ctl_mutex);
+	spin_unlock_irq(&lo->lo_lock);
 
 	if (IS_ERR_OR_NULL(p))
 		ret = PTR_ERR(p);

@@ -1000,7 +1000,9 @@ static int loop_clr_fd(struct loop_devic
 
 	kthread_stop(lo->lo_thread);
 
+	spin_lock_irq(&lo->lo_lock);
 	lo->lo_backing_file = NULL;
+	spin_unlock_irq(&lo->lo_lock);
 
 	loop_release_xfer(lo);
 	lo->transfer = NULL;
#1 lore raw
Quoted 30 lines · Kay Sievers

From: Kay Sievers <[email protected]>
Subject: loop: fix deadlock when sysfs and LOOP_CLR_FD race against each other

LOOP_CLR_FD takes lo->lo_ctl_mutex and tries to remove the loop sysfs files. Sysfs calls show() and waits for lo->lo_ctl_mutex. LOOP_CLR_FD waits for show() to finish to remove the sysfs file.

cat /sys/class/block/loop0/loop/backing_file

    mutex_lock_nested+0x176/0x350
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    ? loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    loop_attr_do_show_backing_file+0x2f/0xd0 [loop]
    dev_attr_show+0x1b/0x60
    ? sysfs_read_file+0x86/0x1a0
    ? __get_free_pages+0x12/0x50
    sysfs_read_file+0xaf/0x1a0

ioctl(LOOP_CLR_FD):

    wait_for_common+0x12c/0x180
    ? try_to_wake_up+0x2a0/0x2a0
    wait_for_completion+0x18/0x20
    sysfs_deactivate+0x178/0x180
    ? sysfs_addrm_finish+0x43/0x70
    ? sysfs_addrm_start+0x1d/0x20
    sysfs_addrm_finish+0x43/0x70
    sysfs_hash_and_remove+0x85/0xa0
    sysfs_remove_group+0x59/0x100
    loop_clr_fd+0x1dc/0x3f0 [loop]
    lo_ioctl+0x223/0x7a0 [loop]
Quoted 4 lines

Instead of taking the lo_ctl_mutex from sysfs code, take the inner lo->lo_lock, to protect the access to the backing_file data.

Thanks to Tejun for help debugging and finding a solution.

Looks good, looks like something that should have a stable tag as well?

Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Kay Sievers

Instead of taking the lo_ctl_mutex from sysfs code, take the inner lo->lo_lock, to protect the access to the backing_file data.

Thanks to Tejun for help debugging and finding a solution.

Looks good, looks like something that should have a stable tag as well?

Right, I think it makes sense to have that in -stable.

It's pretty hard to trigger, I had multiple threads running, crawling /sys and adding/binding/unbinding/removing 1000s of loop devices, and it takes several minutes sometimes until its hit. So I only tested it on top of the 3 loop-control patches, but the issue should exist in the current code as well.

Thanks,
Kay

#3 lore raw
Quoted 9 lines · Kay Sievers
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Kay Sievers

Instead of taking the lo_ctl_mutex from sysfs code, take the inner lo->lo_lock, to protect the access to the backing_file data.

Thanks to Tejun for help debugging and finding a solution.

Looks good, looks like something that should have a stable tag as well?

Quoted 7 lines

Right, I think it makes sense to have that in -stable.

It's pretty hard to trigger, I had multiple threads running, crawling /sys and adding/binding/unbinding/removing 1000s of loop devices, and it takes several minutes sometimes until its hit. So I only tested it on top of the 3 loop-control patches, but the issue should exist in the current code as well.

I applied those for 3.1 as well, but I'm thinking they probably should have been queued up for 3.2 instead.

Signature · 1 line
Jens Axboe