Skip to contentSkip to conversation

INFO: rcu detected stall in sys_lsetxattr

syzbot In review syzbot 3 participants 6 messages Apr 19, 2022 – Apr 21, 2022 VFS

lore
Share & tools
lore

syzbot

author syzbot
#0 lore raw

Hello,

syzbot found the following issue on:

HEAD commit:    40354149f4d7 Add linux-next specific files for 20220414
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000

The issue was bisected to:

commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67

Author: Al Viro <[email protected]>
Date:   Tue Mar 1 04:04:20 2022 +0000

mount_setattr(): clean the control flow and calling conventions

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10499 ticks this GP) idle=23b/1/0x4000000000000000 softirq=5447/5447 fqs=5210 
	(t=10500 jiffies g=4401 q=63 ncpus=2)
NMI backtrace for cpu 1
CPU: 1 PID: 3614 Comm: syz-executor153 Not tainted 5.18.0-rc2-next-20220414-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1e6/0x230 lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x262/0x3f0 kernel/rcu/tree_stall.h:369
 print_cpu_stall kernel/rcu/tree_stall.h:665 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:749 [inline]
 rcu_pending kernel/rcu/tree.c:4010 [inline]
 rcu_sched_clock_irq.cold+0x144/0x8fc kernel/rcu/tree.c:2675
 update_process_times+0x16d/0x200 kernel/time/timer.c:1811
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:243
 tick_sched_timer+0xee/0x120 kernel/time/tick-sched.c:1473
 __run_hrtimer kernel/time/hrtimer.c:1685 [inline]
 __hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
 __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__mnt_want_write+0xdd/0x2e0 fs/namespace.c:348
Code: 00 02 00 00 89 de e8 22 64 9c ff 85 db 74 42 48 b8 00 00 00 00 00 fc ff df 4d 89 ec 49 c1 ec 03 49 01 c4 e8 e5 61 9c ff f3 90 <41> 0f b6 04 24 84 c0 74 08 3c 03 0f 8e 99 01 00 00 8b 5d 10 31 ff
RSP: 0018:ffffc90003acfdf0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
RDX: ffff8880209957c0 RSI: ffffffff81dda00b RDI: 0000000000000003
RBP: ffff888140174c20 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff81dda030 R11: 1ffffffff1f09899 R12: ffffed102802e986
R13: ffff888140174c30 R14: ffff888140174c50 R15: 0000000000000000
 mnt_want_write+0x13d/0x3e0 fs/namespace.c:394
 path_setxattr+0xb2/0x1c0 fs/xattr.c:627
 __do_sys_lsetxattr fs/xattr.c:652 [inline]
 __se_sys_lsetxattr fs/xattr.c:648 [inline]
 __x64_sys_lsetxattr+0xbd/0x150 fs/xattr.c:648
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f7262608cc9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 11 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f72625992f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000bd
RAX: ffffffffffffffda RBX: 00007f72626904b0 RCX: 00007f7262608cc9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000600
RBP: 00007f726265e074 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0030656c69662f2e
R13: 695f70756f72672c R14: 695f726573752c30 R15: 00007f72626904b8
 </TASK>
----------------
Code disassembly (best guess):
   0:	00 02                	add    %al,(%rdx)
   2:	00 00                	add    %al,(%rax)
   4:	89 de                	mov    %ebx,%esi
   6:	e8 22 64 9c ff       	callq  0xff9c642d
   b:	85 db                	test   %ebx,%ebx
   d:	74 42                	je     0x51
   f:	48 b8 00 00 00 00 00 	movabs $0xdffffc0000000000,%rax
  16:	fc ff df
  19:	4d 89 ec             	mov    %r13,%r12
  1c:	49 c1 ec 03          	shr    $0x3,%r12
  20:	49 01 c4             	add    %rax,%r12
  23:	e8 e5 61 9c ff       	callq  0xff9c620d
  28:	f3 90                	pause
* 2a:	41 0f b6 04 24       	movzbl (%r12),%eax <-- trapping instruction
  2f:	84 c0                	test   %al,%al
  31:	74 08                	je     0x3b
  33:	3c 03                	cmp    $0x3,%al
  35:	0f 8e 99 01 00 00    	jle    0x1d4
  3b:	8b 5d 10             	mov    0x10(%rbp),%ebx
  3e:	31 ff                	xor    %edi,%edi

---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

#1 lore raw
Quoted 21 lines · syzbot

Hello,

syzbot found the following issue on:

HEAD commit:    40354149f4d7 Add linux-next specific files for 20220414
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000

The issue was bisected to:

commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67

Author: Al Viro <[email protected]>
Date:   Tue Mar 1 04:04:20 2022 +0000

mount_setattr(): clean the control flow and calling conventions

Quoted 7 lines
bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:

Reported-by [email protected]
Fixes e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup

author syzbot
#2 lore raw
syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by [email protected]

Tested on:

commit:         bbc1e8c5 fs: unset MNT_WRITE_HOLD on failure
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup
kernel config:  https://syzkaller.appspot.com/x/.config?x=c01066a8395ef6d7
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

#3 lore raw

PATCH 1/1 fs: unset MNT_WRITE_HOLD on failure

In reply to syzbot

After mnt_hold_writers() has been called we will always have set MNT_WRITE_HOLD and consequently we always need to pair mnt_hold_writers() with mnt_unhold_writers(). After the recent cleanup in [1] where Al switched from a do-while to a for loop the cleanup currently fails to unset MNT_WRITE_HOLD for the first mount that was changed. Fix this and make sure that the first mount will be cleaned up and add some comments to make it more obvious.

Reported-by [email protected]
Reported-by [email protected]
Fixes e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions") [1]
Cc Hillf Danton <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Al Viro <[email protected]>

---
This should fix the syzbot issue. This is only relevant for making a mount or mount tree read-only:
1. successul recursive read-only mount tree change:
Cleanup loop isn't executed.
2. failed recursive read-only mount tree change:
m will point to the mount we failed to handle. The cleanup loop will run until p == m and then terminate.
3. successful single read-only mount change:
Cleanup loop won't be executed.
4. failed single read-only mount change:
m will point to mnt and the cleanup loop will terminate if p == m.
I don't think there's any other weird corner cases since we now that MNT_WRITE_HOLD can only have been set by us as it requires lock_mount_hash() which we hold. So unconditionally unsetting it is fine. But please make sure to take a close look at the changed loop.

---
 fs/namespace.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/namespace.c +13 -1
index a0a36bfa3aa0..afe2b64b14f1 100644
@@ -4058,10 +4058,22 @@ static int mount_setattr_prepare(struct mount_kattr *kattr, struct mount *mnt)
 	if (err) {
 		struct mount *p;
 
-		for (p = mnt; p != m; p = next_mnt(p, mnt)) {
+		/*
+		 * If we had to call mnt_hold_writers() MNT_WRITE_HOLD will
+		 * be set in @mnt_flags. The loop unsets MNT_WRITE_HOLD for all
+		 * mounts and needs to take care to include the first mount.
+		 */
+		for (p = mnt; p; p = next_mnt(p, mnt)) {
 			/* If we had to hold writers unblock them. */
 			if (p->mnt.mnt_flags & MNT_WRITE_HOLD)
 				mnt_unhold_writers(p);
+
+			/*
+			 * We're done once the first mount we changed got
+			 * MNT_WRITE_HOLD unset.
+			 */
+			if (p == m)
+				break;
 		}
 	}
 	return err;

base-commit: b2d229d4ddb17db541098b83524d901257e93845

Signature · 1 line
2.32.0
#4 lore raw

Re: [PATCH] fs: unset MNT_WRITE_HOLD on failure

Quoted 25 lines · Christian Brauner

After mnt_hold_writers() has been called we will always have set MNT_WRITE_HOLD and consequently we always need to pair mnt_hold_writers() with mnt_unhold_writers(). After the recent cleanup in [1] where Al switched from a do-while to a for loop the cleanup currently fails to unset MNT_WRITE_HOLD for the first mount that was changed. Fix this and make sure that the first mount will be cleaned up and add some comments to make it more obvious.

Reported-by [email protected]
Reported-by [email protected]
Fixes e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions") [1]
Cc Hillf Danton <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Al Viro <[email protected]>

---
This should fix the syzbot issue. This is only relevant for making a mount or mount tree read-only:
1. successul recursive read-only mount tree change:
Cleanup loop isn't executed.
2. failed recursive read-only mount tree change:
m will point to the mount we failed to handle. The cleanup loop will run until p == m and then terminate.
3. successful single read-only mount change:

Quoted 8 lines

Cleanup loop won't be executed.
4. failed single read-only mount change:
m will point to mnt and the cleanup loop will terminate if p == m.
I don't think there's any other weird corner cases since we now that MNT_WRITE_HOLD can only have been set by us as it requires lock_mount_hash() which we hold. So unconditionally unsetting it is fine. But please make sure to take a close look at the changed loop.
---

Unless I hear objections I'll route this upstream before -rc4 to get this fixed because it's pretty trivial to trigger this.