Skip to contentSkip to conversation

INFO: task hung in lo_ioctl

DISCUSSION Dmitry Vyukov 4 participants 10 messages Dec 12, 2017 – Jan 20, 2019 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 175 lines · syzbot

Hello,

syzkaller hit the following crash on
ad4dac17f9d563b9e34aab78a34293b10993e9b5
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

Unfortunately, I don't have any reproducer for this bug yet.

INFO: task syz-executor3:12157 blocked for more than 120 seconds.
      Not tainted 4.15.0-rc2-next-20171208+ #63
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor3   D25536 12157   3362 0x00000004
Call Trace:
 context_switch kernel/sched/core.c:2800 [inline]
 __schedule+0x8eb/0x2060 kernel/sched/core.c:3376
 schedule+0xf5/0x430 kernel/sched/core.c:3435
 schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3493
 __mutex_lock_common kernel/locking/mutex.c:833 [inline]
 __mutex_lock+0xaad/0x1a80 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
 __blkdev_driver_ioctl block/ioctl.c:303 [inline]
 blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
 block_ioctl+0xea/0x130 fs/block_dev.c:1860
 vfs_ioctl fs/ioctl.c:46 [inline]
 do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686
 SYSC_ioctl fs/ioctl.c:701 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
 entry_SYSCALL_64_fastpath+0x1f/0x96
RIP: 0033:0x452a39
RSP: 002b:00007fd85b57ec58 EFLAGS: 00000212 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000758190 RCX: 0000000000452a39
RDX: 0000000020e67fd8 RSI: 00000000400454ca RDI: 0000000000000016
RBP: 000000000000005b R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000212 R12: 00000000006ee928
R13: 00000000ffffffff R14: 00007fd85b57f6d4 R15: 0000000000000002

Showing all locks held in the system:
2 locks held by khungtaskd/671:
 #0:  (rcu_read_lock){....}, at: [<0000000053bc3983>]
check_hung_uninterruptible_tasks kernel/hung_task.c:175 [inline]
 #0:  (rcu_read_lock){....}, at: [<0000000053bc3983>] watchdog+0x1c5/0xd60
kernel/hung_task.c:249
 #1:  (tasklist_lock){.+.+}, at: [<00000000c2c6381b>]
debug_show_all_locks+0xd3/0x400 kernel/locking/lockdep.c:4554
2 locks held by getty/3115:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
2 locks held by getty/3116:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
2 locks held by getty/3117:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
2 locks held by getty/3118:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
2 locks held by getty/3119:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
2 locks held by getty/3120:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
1 lock held by syz-executor3/12157:
 #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
1 lock held by syz-executor7/12154:
 #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
1 lock held by syz-executor7/12168:
 #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
1 lock held by syz-executor5/12216:
 #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355
2 locks held by getty/12231:
 #0:  (&tty->ldisc_sem){++++}, at: [<00000000e7f92d1f>]
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
 #1:  (&ldata->atomic_read_lock){+.+.}, at: [<00000000f15a7c4f>]
n_tty_read+0x2f2/0x1a10 drivers/tty/n_tty.c:2131
1 lock held by blkid/12232:
 #0:  (&lo->lo_ctl_mutex/1){+.+.}, at: [<00000000d5130113>]
lo_ioctl+0x8b/0x1b90 drivers/block/loop.c:1355

=============================================

NMI backtrace for cpu 0

CPU: 0 PID: 671 Comm: khungtaskd Not tainted 4.15.0-rc2-next-20171208+ #63
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:53
 nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103
 nmi_trigger_cpumask_backtrace+0x122/0x180 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_all_cpu_backtrace include/linux/nmi.h:138 [inline]
 check_hung_task kernel/hung_task.c:132 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:190 [inline]
 watchdog+0x90c/0xd60 kernel/hung_task.c:249
 kthread+0x37a/0x440 kernel/kthread.c:238
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc2-next-20171208+ #63
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
RIP: 0010:unwind_next_frame.part.6+0x5f1/0xb40 include/linux/compiler.h:183
RSP: 0018:ffff8801db306e28 EFLAGS: 00000806
RAX: dffffc0000000000 RBX: ffff8801db306f48 RCX: ffff8801db306ef8
RDX: ffff8801db306f70 RSI: 1ffff1003b660dcf RDI: ffff8801d9f97f48
RBP: ffff8801db306f20 R08: ffff8801db306f80 R09: ffff8801d9f88300
R10: 000000000000000b R11: ffffed003b660df3 R12: 1ffff1003b660dcb
R13: ffff8801db307490 R14: 1ffff1003b660dcf R15: ffff8801db306f98
FS:  0000000000000000(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001e5e000 CR3: 00000001cb1b5000 CR4: 00000000001406e0
DR0: 0000000020000000 DR1: 0000000020000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 <IRQ>
 unwind_next_frame+0x3e/0x50 arch/x86/kernel/unwind_frame.c:287
 __save_stack_trace+0x6e/0xd0 arch/x86/kernel/stacktrace.c:44
 save_stack_trace+0x1a/0x20 arch/x86/kernel/stacktrace.c:60
 add_xhlock kernel/locking/lockdep.c:4880 [inline]
 check_add_xhlock kernel/locking/lockdep.c:4907 [inline]
 lock_acquire_crosslock kernel/locking/lockdep.c:4979 [inline]
 __lock_acquire+0x324e/0x47f0 kernel/locking/lockdep.c:3503
 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:159
 rq_lock_irqsave kernel/sched/sched.h:1750 [inline]
 update_blocked_averages+0x124/0x1b60 kernel/sched/fair.c:7353
 rebalance_domains+0x145/0xcc0 kernel/sched/fair.c:9122
 run_rebalance_domains+0x381/0x780 kernel/sched/fair.c:9383
 __do_softirq+0x29d/0xbb2 kernel/softirq.c:285
 invoke_softirq kernel/softirq.c:365 [inline]
 irq_exit+0x1d3/0x210 kernel/softirq.c:405
 scheduler_ipi+0x32a/0x830 kernel/sched/core.c:1804
 smp_reschedule_interrupt+0xe6/0x670 arch/x86/kernel/smp.c:277
 reschedule_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:944
 </IRQ>
RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:54
RSP: 0018:ffff8801d9f97da8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff02
RAX: dffffc0000000000 RBX: 1ffff1003b3f2fb8 RCX: 0000000000000000
RDX: 1ffffffff0c5975c RSI: 0000000000000001 RDI: ffffffff862cbae0
RBP: ffff8801d9f97da8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8801d9f97e60 R14: ffffffff869efaa0 R15: 0000000000000000
 arch_safe_halt arch/x86/include/asm/paravirt.h:93 [inline]
 default_idle+0xbf/0x430 arch/x86/kernel/process.c:355
 arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:346
 default_idle_call+0x36/0x90 kernel/sched/idle.c:98
 cpuidle_idle_call kernel/sched/idle.c:156 [inline]
 do_idle+0x24a/0x3b0 kernel/sched/idle.c:246
 cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:351
 start_secondary+0x330/0x460 arch/x86/kernel/smpboot.c:277
 secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:237
Code: 8b 6d 00 4c 89 f6 48 c1 ee 03 80 3c 06 00 0f 85 e9 04 00 00 49 c1 ee
Quoted 2 lines

03 48 b8 00 00 00 00 00 fc ff df 4c 89 69 80 41 c6 04 06 f8 <e9> 97 fb ff ff 80 3d f7 60 6e 05 00 0f 85 fc fc ff ff 49 8d be

This seems to be about loop device, but was mis-attributed to scheduler. So +loop maintainers.

FTR, mis-attribution in syzbot was fixed in:
https://github.com/google/syzkaller/commit/27f5dfefff3cd00795d9048ecc7bc70965e03ea7

Quoted 25 lines

---
This bug is generated by a dumb bot. It may contain errors. See https://goo.gl/tpsmEJ for details.
Direct all questions to [email protected].
Please credit me with: Reported-by: syzbot <[email protected]>

syzbot will keep track of this bug report.
Once a fix for this bug is merged into any tree, reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug report.
Note: all commands must start from beginning of the line in the email body.

--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/syzkaller-bugs/94eb2c0810d04f5a46055ffc71aa%40google.com.
For more options, visit https://groups.google.com/d/optout.

#2 lore raw

This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?).

When PID=6540 was (reported as hung) at mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836), it was already holding down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING) (id=566d4c39). But when PID=6541 was (which would have been reported as hung if sysctl_hung_task_panic were not set) at down_read(&sb->s_umount) (id=566d4c39), it was already holding mutex_lock_nested(&lo->lo_ctl_mutex, 1) (id=43ca8836).

----------------------------------------

INFO: task syz-executor0:6540 blocked for more than 120 seconds.
      Not tainted 4.16.0+ #13
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor0   D23560  6540   4521 0x80000004
Call Trace:
 context_switch kernel/sched/core.c:2848 [inline]
 __schedule+0x8fb/0x1ef0 kernel/sched/core.c:3490
 schedule+0xf5/0x430 kernel/sched/core.c:3549
 schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3607
 __mutex_lock_common kernel/locking/mutex.c:833 [inline]
 __mutex_lock+0xb7f/0x1810 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355
 __blkdev_driver_ioctl block/ioctl.c:303 [inline]
 blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
 ioctl_by_bdev+0xa5/0x110 fs/block_dev.c:2060
 isofs_get_last_session fs/isofs/inode.c:567 [inline]
 isofs_fill_super+0x2ba9/0x3bc0 fs/isofs/inode.c:660
 mount_bdev+0x2b7/0x370 fs/super.c:1119
 isofs_mount+0x34/0x40 fs/isofs/inode.c:1560
 mount_fs+0x66/0x2d0 fs/super.c:1222

2 locks held by syz-executor0/6540:
#0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: alloc_super fs/super.c:211 [inline]
#0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: sget_userns+0x3b2/0xe60 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */ #1: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
3 locks held by syz-executor7/6541:
#0: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */ #1: 000000007bf3d3f9 (&bdev->bd_mutex){+.+.}, at: blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
#2: 00000000566d4c39 (&type->s_umount_key#50){.+.+}, at: __get_super.part.10+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
----------------------------------------

sget() is using down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING) with a comment block asserting that there is no risk of deadlock

/*
* sget() can have s_umount recursion.
*
* When it cannot find a suitable sb, it allocates a new
* one (this one), and tries again to find a suitable old
* one.
*
* In case that succeeds, it will acquire the s_umount
* lock of the old one. Since these are clearly distrinct
* locks, and this object isn't exposed yet, there's no
* risk of deadlocks.
*
* Annotate this by putting this lock in a different
* subclass.
*/

but this object (id=566d4c39) is already locked by other thread. What is happening here?

#3 lore raw
Quoted 3 lines · Tetsuo Handa

This seems to be an AB-BA deadlock where the lockdep cannot report (due to use of nested lock?). What is happening here?

This patch does not directly fix the bug syzbot is reporting, but could be relevant. Maybe try replacing mutex_lock_killable_nested() with mutex_lock_killable() and check what the lockdep will say?

Quoted 1 line

From 0b006d536b2e439f347eb857e482fc304e84fd1d Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Fri, 6 Apr 2018 20:52:10 +0900
Subject: [PATCH] block/loop: fix lock imbalance bug at lo_ioctl

Commit 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding lo_ctl_mutex") introduced mutex lock imbalance bug where syzbot would trigger. The caller of loop_get_status64() assumes that mutex_unlock() is already called by loop_get_status() but loop_get_status64() does not always call loop_get_status().

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()") also dropped the mutex for deadlock avoidance reason. But we should recheck whether we have to drop the mutex. Dropping the mutex at the middle of an ioctl() request is not nice, but syzbot is reporting a deadlock inside loop_reread_partitions() which is called by loop_set_fd() and loop_change_fd().

Fixes 2d1d4c1e591fd40b ("loop: don't call into filesystem while holding lo_ctl_mutex")
Cc Omar Sandoval <[email protected]>
Cc Nikanth Karthikesan <[email protected]>
Cc Jens Axboe <[email protected]>
---
 drivers/block/loop.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/block/loop.c +6 -11
index 264abaa..64033e7 100644
@@ -1362,7 +1362,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 
 	err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
 	if (err)
-		goto out_unlocked;
+		return err;
 
 	switch (cmd) {
 	case LOOP_SET_FD:
@@ -1372,10 +1372,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		err = loop_change_fd(lo, bdev, arg);
 		break;
 	case LOOP_CLR_FD:
-		/* loop_clr_fd would have unlocked lo_ctl_mutex on success */
 		err = loop_clr_fd(lo);
-		if (!err)
-			goto out_unlocked;
 		break;
 	case LOOP_SET_STATUS:
 		err = -EPERM;
@@ -1385,8 +1382,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS:
 		err = loop_get_status_old(lo, (struct loop_info __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_STATUS64:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1395,8 +1391,7 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 		break;
 	case LOOP_GET_STATUS64:
 		err = loop_get_status64(lo, (struct loop_info64 __user *) arg);
-		/* loop_get_status() unlocks lo_ctl_mutex */
-		goto out_unlocked;
+		break;
 	case LOOP_SET_CAPACITY:
 		err = -EPERM;
 		if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
@@ -1415,9 +1410,9 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
 	default:
 		err = lo->ioctl ? lo->ioctl(lo, cmd, arg) : -EINVAL;
 	}
-	mutex_unlock(&lo->lo_ctl_mutex);
-
-out_unlocked:
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);
 	return err;
 }
 
Signature · 1 line
1.8.3.1
#4 lore raw

drivers/block/loop.c static int lo_ioctl(struct block_device *bdev, fmode_t mode, · line 1415

Quoted 3 lines · Tetsuo Handa
drivers/block/loop.c +3 -0
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

#5 lore raw
Quoted 7 lines · Peter Zijlstra

drivers/block/loop.c static int lo_ioctl(struct block_device *bdev, fmode_t mode, · line 1415

Quoted 3 lines · Tetsuo Handa
drivers/block/loop.c +3 -0
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking") is using __mutex_owner(). ;-)

Of course, regarding loop module, we will be able to add a flag variable associated with lo->lo_ctl_mutex. But that will be done after we solved the deadlock problem. I think whether we need to drop lo->lo_ctl_mutex is not clear. Maybe

- err = mutex_lock_killable_nested(&lo->lo_ctl_mutex, 1);
+ err = mutex_lock_killable(&lo->lo_ctl_mutex);

on top of this patch and listen to the lockdep?

Commit f028f3b2f987ebc6 ("loop: fix circular locking in loop_clr_fd()") says

A simple way to silence lockdep could be to mark the lo_ctl_mutex in ioctl to be a sub class, but this might mask some other real bugs.

and we are currently hitting a deadlock problem.

#6 lore raw
Quoted 11 lines · Tetsuo Handa
Quoted 7 lines · Peter Zijlstra

drivers/block/loop.c static int lo_ioctl(struct block_device *bdev, fmode_t mode, · line 1415

Quoted 3 lines · Tetsuo Handa
drivers/block/loop.c +3 -0
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking") is using __mutex_owner(). ;-)

That got removed and the warning added.

#7 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Tetsuo Handa
Quoted 7 lines · Peter Zijlstra

Review

Quoted 3 lines · Tetsuo Handa
(quoted diff) +3 -0
+	/* Temporary hack for handling lock imbalance. */
+	if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+		mutex_unlock(&lo->lo_ctl_mutex);

ARGGH.. you didn't read the comment we put on that?

Quoted 4 lines
Quoted 2 lines

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking") is using __mutex_owner(). ;-)

That got removed and the warning added.

Seems easy enough to fix without resorting to __mutex_owner() (untested):

drivers/block/loop.c +6 -5
index 264abaaff662..cee258d12a1e 100644
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
 	struct loop_info64 info64;
-	int err = 0;
+	int err;
 
-	if (!arg)
-		err = -EINVAL;
-	if (!err)
-		err = loop_get_status(lo, &info64);
+	if (!arg) {
+		mutex_unlock(&lo->lo_ctl_mutex);
+		return -EINVAL;
+	}
+	err = loop_get_status(lo, &info64);
 	if (!err && copy_to_user(arg, &info64, sizeof(info64)))
 		err = -EFAULT;
 

I'll test it and send it up when I get into the office.

#8 lore raw
Quoted 45 lines · Omar Sandoval
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Tetsuo Handa
Quoted 1 line · Peter Zijlstra

On Fri, Apr 06, 2018 at 09:04:18PM +0900, Tetsuo Handa wrote:

+ /* Temporary hack for handling lock imbalance. */

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
(quoted diff) +2 -0
+       if (__mutex_owner(&lo->lo_ctl_mutex) == current)
+               mutex_unlock(&lo->lo_ctl_mutex);
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

ARGGH.. you didn't read the comment we put on that?

Commit 5b52330bbfe63b33 ("audit: fix auditd/kernel connection state tracking") is using __mutex_owner(). ;-)

That got removed and the warning added.

Seems easy enough to fix without resorting to __mutex_owner() (untested):

drivers/block/loop.c +6 -5
index 264abaaff662..cee258d12a1e 100644
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
        struct loop_info64 info64;
-       int err = 0;
+       int err;
 
-       if (!arg)
-               err = -EINVAL;
-       if (!err)
-               err = loop_get_status(lo, &info64);
+       if (!arg) {
+               mutex_unlock(&lo->lo_ctl_mutex);
+               return -EINVAL;
+       }
+       err = loop_get_status(lo, &info64);
        if (!err && copy_to_user(arg, &info64, sizeof(info64)))
                err = -EFAULT;
 
Quoted 1 line

I'll test it and send it up when I get into the office.

Was this ever submitted? Or some other fix for this?

The bug is still open, but last happened 289 days ago:
https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce

But it also has 10 duplicates, some of which happened much more recently. If a fix was submitted, but Reported-by tag wasn't added this open bug can now mask lots of other new bugs.

#9 lore raw
Quoted 32 lines · Dmitry Vyukov
Quoted 27 lines

Seems easy enough to fix without resorting to __mutex_owner() (untested):

drivers/block/loop.c +6 -5
index 264abaaff662..cee258d12a1e 100644
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
        struct loop_info64 info64;
-       int err = 0;
+       int err;
 
-       if (!arg)
-               err = -EINVAL;
-       if (!err)
-               err = loop_get_status(lo, &info64);
+       if (!arg) {
+               mutex_unlock(&lo->lo_ctl_mutex);
+               return -EINVAL;
+       }
+       err = loop_get_status(lo, &info64);
        if (!err && copy_to_user(arg, &info64, sizeof(info64)))
                err = -EFAULT;
 
Quoted 1 line

I'll test it and send it up when I get into the office.

Was this ever submitted? Or some other fix for this?

Quoted 7 lines

The bug is still open, but last happened 289 days ago:
https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce

But it also has 10 duplicates, some of which happened much more recently. If a fix was submitted, but Reported-by tag wasn't added this open bug can now mask lots of other new bugs.

The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS lock imbalance"). But the root cause of these hung tasks would be fixed by a series containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks on loop device"). We were not aware of these bugs when you marked these reports as duplicates on 2017/12/12. You can undup them and fix them if you want.

#10 lore raw
Quoted 42 lines · Tetsuo Handa
Quoted 32 lines · Dmitry Vyukov
Quoted 27 lines

Seems easy enough to fix without resorting to __mutex_owner() (untested):

drivers/block/loop.c +6 -5
index 264abaaff662..cee258d12a1e 100644
@@ -1300,12 +1300,13 @@ loop_get_status_old(struct loop_device *lo, struct loop_info __user *arg) {
 static int
 loop_get_status64(struct loop_device *lo, struct loop_info64 __user *arg) {
        struct loop_info64 info64;
-       int err = 0;
+       int err;
 
-       if (!arg)
-               err = -EINVAL;
-       if (!err)
-               err = loop_get_status(lo, &info64);
+       if (!arg) {
+               mutex_unlock(&lo->lo_ctl_mutex);
+               return -EINVAL;
+       }
+       err = loop_get_status(lo, &info64);
        if (!err && copy_to_user(arg, &info64, sizeof(info64)))
                err = -EFAULT;
 
Quoted 1 line

I'll test it and send it up when I get into the office.

Was this ever submitted? Or some other fix for this?

Quoted 7 lines

The bug is still open, but last happened 289 days ago:
https://syzkaller.appspot.com/bug?id=608144371e7fc2cb6285b9ed871fb1eb817a61ce

But it also has 10 duplicates, some of which happened much more recently. If a fix was submitted, but Reported-by tag wasn't added this open bug can now mask lots of other new bugs.

Quoted 6 lines

The commit for this specific patch is bdac616db9bbadb9 ("loop: fix LOOP_GET_STATUS lock imbalance"). But the root cause of these hung tasks would be fixed by a series containing commit 1dded9acf6dc9a34 ("Avoid circular locking dependency between loop_ctl_mutex and bd_mutex") or commit 04906b2f542c2362 ("blockdev: Fix livelocks on loop device"). We were not aware of these bugs when you marked these reports as duplicates on 2017/12/12. You can undup them and fix them if you want.

OK, let's just do then:

#syz fix: blockdev: Fix livelocks on loop device