loop: fput() called in loop_clr_fd() may cause bd_mutex recursive locking

PATCH NAK Sergey Senozhatsky 2 participants 9 messages Dec 17, 2011 Block layer

Share & tools

Sergey Senozhatsky

author

2011-12-17 21:57 UTC #0 lore raw

PATCH 1/1 loop: fput() called in loop_clr_fd() may cause bd_mutex recursive locking

Unmonting mounted with `-o loop' block device causes recursive bd_mutex locking. fput() calls blkdev_put() for bdev that issued disk->fops->release() (loop_clr_fd()) call:

[23044.654647] umount/24442 is trying to acquire lock:
[23044.654652]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654670] 
[23044.654672] but task is already holding lock:
[23044.654677]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654690] 
[23044.654692] other info that might help us debug this:
[23044.654697]  Possible unsafe locking scenario:
[23044.654727] 
[23044.654731] 1 lock held by umount/24442:
[23044.654735]  #0:  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654748] 
[23044.654762] Call Trace:
[23044.654773]  [<ffffffff81075611>] __lock_acquire+0x15bf/0x1659
[23044.654784]  [<ffffffff8114b3e3>] ? inotify_free_group_priv+0x4f/0x4f
[23044.654792]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654799]  [<ffffffff81075c6a>] lock_acquire+0x138/0x1b3
[23044.654807]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654814]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654824]  [<ffffffff8147ce67>] mutex_lock_nested+0x5e/0x325
[23044.654831]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654838]  [<ffffffff81149963>] ? fsnotify+0x441/0x459
[23044.654846]  [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654853]  [<ffffffff81144443>] blkdev_close+0x20/0x22
[23044.654863]  [<ffffffff81116b21>] fput+0x117/0x1cf
[23044.654874]  [<ffffffffa016eb71>] loop_clr_fd+0x1f2/0x201 [loop]
[23044.654882]  [<ffffffffa016f861>] lo_release+0x40/0x6f [loop]
[23044.654890]  [<ffffffff81144244>] __blkdev_put+0xd6/0x184
[23044.654898]  [<ffffffff8114441a>] blkdev_put+0x128/0x131
[23044.654906]  [<ffffffff8111704e>] kill_block_super+0x60/0x65
[23044.654914]  [<ffffffff81117366>] deactivate_locked_super+0x32/0x63
[23044.654922]  [<ffffffff81117cc9>] deactivate_super+0x3a/0x3e
[23044.654931]  [<ffffffff8112fc5d>] mntput_no_expire+0xbf/0xc4
[23044.654939]  [<ffffffff811309c7>] sys_umount+0x2c5/0x2f3
[23044.654949]  [<ffffffff81484b12>] system_call_fastpath+0x16/0x1b

Signed-off-by Sergey Senozhatsky <[email protected]>

---

drivers/block/loop.c +9 -0

index 1e888c9..b004779 100644
@@ -1028,6 +1028,15 @@ static int loop_clr_fd(struct loop_device *lo)
 	 * lock dependency possibility warning as fput can take
 	 * bd_mutex which is usually taken before lo_ctl_mutex.
 	 */
+	/*
+	 * Need to put file f_op, otherwise fput() may cause
+	 * recursive locking on bd_mutex, calling blkdev_put()
+	 * for bdev that issued disk->fops->release() call.
+	 */
+	if (bdev && bdev == bdev->bd_contains) {
+		fops_put(filp->f_op);
+		filp->f_op = NULL;
+	}
 	fput(filp);
 	return 0;
 }

Al Viro

NAK

2011-12-17 22:12 UTC #1 lore raw

Quoted 55 lines · Sergey Senozhatsky

Unmonting mounted with `-o loop' block device causes recursive bd_mutex locking. fput() calls blkdev_put() for bdev that issued disk->fops->release() (loop_clr_fd()) call:

[23044.654647] umount/24442 is trying to acquire lock:
[23044.654652]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654670] 
[23044.654672] but task is already holding lock:
[23044.654677]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654690] 
[23044.654692] other info that might help us debug this:
[23044.654697]  Possible unsafe locking scenario:
[23044.654727] 
[23044.654731] 1 lock held by umount/24442:
[23044.654735]  #0:  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654748] 
[23044.654762] Call Trace:
[23044.654773]  [<ffffffff81075611>] __lock_acquire+0x15bf/0x1659
[23044.654784]  [<ffffffff8114b3e3>] ? inotify_free_group_priv+0x4f/0x4f
[23044.654792]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654799]  [<ffffffff81075c6a>] lock_acquire+0x138/0x1b3
[23044.654807]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654814]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654824]  [<ffffffff8147ce67>] mutex_lock_nested+0x5e/0x325
[23044.654831]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654838]  [<ffffffff81149963>] ? fsnotify+0x441/0x459
[23044.654846]  [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654853]  [<ffffffff81144443>] blkdev_close+0x20/0x22
[23044.654863]  [<ffffffff81116b21>] fput+0x117/0x1cf
[23044.654874]  [<ffffffffa016eb71>] loop_clr_fd+0x1f2/0x201 [loop]
[23044.654882]  [<ffffffffa016f861>] lo_release+0x40/0x6f [loop]
[23044.654890]  [<ffffffff81144244>] __blkdev_put+0xd6/0x184
[23044.654898]  [<ffffffff8114441a>] blkdev_put+0x128/0x131
[23044.654906]  [<ffffffff8111704e>] kill_block_super+0x60/0x65
[23044.654914]  [<ffffffff81117366>] deactivate_locked_super+0x32/0x63
[23044.654922]  [<ffffffff81117cc9>] deactivate_super+0x3a/0x3e
[23044.654931]  [<ffffffff8112fc5d>] mntput_no_expire+0xbf/0xc4
[23044.654939]  [<ffffffff811309c7>] sys_umount+0x2c5/0x2f3
[23044.654949]  [<ffffffff81484b12>] system_call_fastpath+0x16/0x1b

Signed-off-by Sergey Senozhatsky <[email protected]>

---

drivers/block/loop.c +2 -0

index 1e888c9..b004779 100644
@@ -1028,6 +1028,15 @@ static int loop_clr_fd(struct loop_device *lo)
 	 * lock dependency possibility warning as fput can take
 	 * bd_mutex which is usually taken before lo_ctl_mutex.
 	 */
+	/*
+	 * Need to put file f_op, otherwise fput() may cause

drivers/block/loop.c static int loop_clr_fd(struct loop_device *lo) · line 1042

Quoted 10 lines

drivers/block/loop.c +7 -0

+	 * recursive locking on bd_mutex, calling blkdev_put()
+	 * for bdev that issued disk->fops->release() call.
+	 */
+	if (bdev && bdev == bdev->bd_contains) {
+		fops_put(filp->f_op);
+		filp->f_op = NULL;
+	}
 	fput(filp);
 	return 0;
 }

NAK - you've "fixed" a false positive from lock checker by failing to close the underlying device.

Sergey Senozhatsky

author

2011-12-17 22:22 UTC #2 lore raw

Quoted 36 lines · Al Viro

Quoted 36 lines

[23044.654647] umount/24442 is trying to acquire lock:
[23044.654652]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654670] 
[23044.654672] but task is already holding lock:
[23044.654677]  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654690] 
[23044.654692] other info that might help us debug this:
[23044.654697]  Possible unsafe locking scenario:
[23044.654727] 
[23044.654731] 1 lock held by umount/24442:
[23044.654735]  #0:  (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811441a1>] __blkdev_put+0x33/0x184
[23044.654748] 
[23044.654762] Call Trace:
[23044.654773]  [<ffffffff81075611>] __lock_acquire+0x15bf/0x1659
[23044.654784]  [<ffffffff8114b3e3>] ? inotify_free_group_priv+0x4f/0x4f
[23044.654792]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654799]  [<ffffffff81075c6a>] lock_acquire+0x138/0x1b3
[23044.654807]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654814]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654824]  [<ffffffff8147ce67>] mutex_lock_nested+0x5e/0x325
[23044.654831]  [<ffffffff81144311>] ? blkdev_put+0x1f/0x131
[23044.654838]  [<ffffffff81149963>] ? fsnotify+0x441/0x459
[23044.654846]  [<ffffffff81144311>] blkdev_put+0x1f/0x131
[23044.654853]  [<ffffffff81144443>] blkdev_close+0x20/0x22
[23044.654863]  [<ffffffff81116b21>] fput+0x117/0x1cf
[23044.654874]  [<ffffffffa016eb71>] loop_clr_fd+0x1f2/0x201 [loop]
[23044.654882]  [<ffffffffa016f861>] lo_release+0x40/0x6f [loop]
[23044.654890]  [<ffffffff81144244>] __blkdev_put+0xd6/0x184
[23044.654898]  [<ffffffff8114441a>] blkdev_put+0x128/0x131
[23044.654906]  [<ffffffff8111704e>] kill_block_super+0x60/0x65
[23044.654914]  [<ffffffff81117366>] deactivate_locked_super+0x32/0x63
[23044.654922]  [<ffffffff81117cc9>] deactivate_super+0x3a/0x3e
[23044.654931]  [<ffffffff8112fc5d>] mntput_no_expire+0xbf/0xc4
[23044.654939]  [<ffffffff811309c7>] sys_umount+0x2c5/0x2f3
[23044.654949]  [<ffffffff81484b12>] system_call_fastpath+0x16/0x1b

[..]

Quoted 4 lines

NAK - you've "fixed" a false positive from lock checker by failing to close the underlying device.

Sorry, why is that a false positive?

blkdev_put() calls lo_release() while holding bd_mutex, lo_release() calls loop_clr_fd() -> fput(). fput() once again attempts to grub already held bd_mutex calling blkdev_put(). Looks like a recursion to me.

Sergey

Al Viro

2011-12-17 22:30 UTC #3 lore raw

Quoted 6 lines · Sergey Senozhatsky

Sorry, why is that a false positive?
blkdev_put() calls lo_release() while holding bd_mutex, lo_release() calls loop_clr_fd() -> fput(). fput() once again attempts to grub already held bd_mutex calling blkdev_put(). Looks like a recursion to me.

Because of this:

        /* Avoid recursion */
        f = file;
        while (is_loop_device(f)) {
                struct loop_device *l;

                if (f->f_mapping->host->i_bdev == bdev)
                        goto out_putf;

                l = f->f_mapping->host->i_bdev->bd_disk->private_data;
                if (l->lo_state == Lo_unbound) {
                        error = -EINVAL;
                        goto out_putf;
                }
                f = l->lo_backing_file;
        }

in loop_set_fd(). Think of it for a minute - if we could run into the same bdev in that recursion, what would have happened on read() from that sucker? So yes, it is a false positive. And your patch would simply leave the underlying device opened, with all the consequences...

Sergey Senozhatsky

author

2011-12-17 22:41 UTC #4 lore raw

Quoted 23 lines · Al Viro

Quoted 6 lines
Sorry, why is that a false positive?
blkdev_put() calls lo_release() while holding bd_mutex, lo_release() calls loop_clr_fd() -> fput(). fput() once again attempts to grub already held bd_mutex calling blkdev_put(). Looks like a recursion to me.
Because of this:
        /* Avoid recursion */
        f = file;
        while (is_loop_device(f)) {
                struct loop_device *l;

                if (f->f_mapping->host->i_bdev == bdev)
                        goto out_putf;

                l = f->f_mapping->host->i_bdev->bd_disk->private_data;
                if (l->lo_state == Lo_unbound) {
                        error = -EINVAL;
                        goto out_putf;
                }
                f = l->lo_backing_file;
        }

Quoted 1 line

in loop_set_fd().

Oh, thanks. I didn't notice that one.

Quoted 3 lines

Think of it for a minute - if we could run into the
same bdev in that recursion, what would have happened on read() from that sucker? So yes, it is a false positive.

I've tried read()/write() some time ago and it worked. Perhaps, I just wasn't "lucky" enough to hit any problems.

Quoted 3 lines

And your patch would simply leave the underlying device opened, with all the consequences...

well, that sucks.

Sergey

Al Viro

2011-12-17 22:58 UTC #5 lore raw

Quoted 6 lines · Sergey Senozhatsky

Quoted 3 lines
Think of it for a minute - if we could run into the
same bdev in that recursion, what would have happened on read() from that sucker? So yes, it is a false positive.
I've tried read()/write() some time ago and it worked. Perhaps, I just wasn't "lucky" enough to hit any problems.

Sure - exactly because of that loop prevention logics. *If* we really had been able to set a loop0 -> loop1 -> loop2 -> loop0 or something of that sort, this warning wouldn't be a false positive. But on any such setup, where would IO attempts end up doing? IOW, we have to prevent such setups anyway and not just because of problems on close() - they would be deadly on read() and write()...

Sergey Senozhatsky

author

2011-12-17 23:24 UTC #6 lore raw

Quoted 5 lines · Al Viro

Quoted 4 lines · Sergey Senozhatsky
Quoted 3 lines
Think of it for a minute - if we could run into the
same bdev in that recursion, what would have happened on read() from that sucker? So yes, it is a false positive.

Quoted 7 lines

Quoted 2 lines
I've tried read()/write() some time ago and it worked. Perhaps, I just wasn't "lucky" enough to hit any problems.
Sure - exactly because of that loop prevention logics. *If* we really had been able to set a loop0 -> loop1 -> loop2 -> loop0 or something of that sort, this warning wouldn't be a false positive. But on any such setup, where would IO attempts end up doing?

Thanks for your explanations.

Quoted 3 lines

IOW, we have to prevent such setups anyway and not just because of problems on close() - they would be deadly on read() and write()...

Preventing recursion in the first place? For example, in lo_open()?

Sergey

Al Viro

2011-12-17 23:39 UTC #7 lore raw

Quoted 5 lines · Sergey Senozhatsky

Quoted 3 lines
IOW, we have to prevent such setups anyway and not just because of problems on close() - they would be deadly on read() and write()...
Preventing recursion in the first place? For example, in lo_open()?

Uh? No, in loop_set_fd(), where we decide what loop devices will be backed by what - the code I quoted to you upthread.

Sergey Senozhatsky

author

2011-12-17 23:50 UTC #8 lore raw

Quoted 11 lines · Al Viro

Quoted 5 lines · Sergey Senozhatsky
Quoted 3 lines
IOW, we have to prevent such setups anyway and not just because of problems on close() - they would be deadly on read() and write()...
Preventing recursion in the first place? For example, in lo_open()?
Uh? No, in loop_set_fd(), where we decide what loop devices will be backed by what - the code I quoted to you upthread.

Oops, sorry. Long day at work . Need to sleep.

Sergey