general protection fault in io_sqpoll_wq_cpu_affinity
syzbot
syzbotHello,
syzbot found the following issue on:
HEAD commit: 626932085009 Add linux-next specific files for 20230825 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=12a97797a80000 kernel config: https://syzkaller.appspot.com/x/.config?x=8a8c992a790e5073 dashboard link: https://syzkaller.appspot.com/bug?extid=c74fea926a78b8a91042 compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/46ec18b3c2fb/disk-62693208.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/b4ea0cb78498/vmlinux-62693208.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5fb3938c7272/bzImage-62693208.xz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
general protection fault, probably for non-canonical address 0xdffffc000000011d: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x00000000000008e8-0x00000000000008ef] CPU: 1 PID: 27342 Comm: syz-executor.5 Not tainted 6.5.0-rc7-next-20230825-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023 RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433 Code: 48 c1 ea 03 80 3c 02 00 75 64 4c 8b a3 a8 00 00 00 48 b8 00 00 00 00 00 fc ff df 49 8d bc 24 e8 08 00 00 48 89 fa 48 c1 ea 03 <80> 3c 02 00 75 37 49 8b bc 24 e8 08 00 00 48 89 ee e8 7e ac 02 00 RSP: 0018:ffffc900051dfe00 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: ffff888056c72400 RCX: 0000000000000000 RDX: 000000000000011d RSI: ffffffff841f12be RDI: 00000000000008e8 RBP: ffffc900051dfec8 R08: 0000000000000000 R09: ffffed100ad8e482 R10: ffffc900051dfde8 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000011 R15: ffffc900051dfec8 FS: 00007f1f3e2bc6c0(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffe2c9b3010 CR3: 0000000029652000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> __io_register_iowq_aff io_uring/io_uring.c:4207 [inline] __io_register_iowq_aff+0xa7/0xe0 io_uring/io_uring.c:4198 io_register_iowq_aff io_uring/io_uring.c:4240 [inline] __io_uring_register io_uring/io_uring.c:4447 [inline] __do_sys_io_uring_register+0xf58/0x2250 io_uring/io_uring.c:4539 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x38/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd RIP: 0033:0x7f1f3d47cae9 Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f1f3e2bc0c8 EFLAGS: 00000246 ORIG_RAX: 00000000000001ab RAX: ffffffffffffffda RBX: 00007f1f3d59bf80 RCX: 00007f1f3d47cae9 RDX: 0000000020000140 RSI: 0000000000000011 RDI: 0000000000000003 RBP: 00007f1f3d4c847a R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 R13: 000000000000000b R14: 00007f1f3d59bf80 R15: 00007fffb91ba9f8 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433 Code: 48 c1 ea 03 80 3c 02 00 75 64 4c 8b a3 a8 00 00 00 48 b8 00 00 00 00 00 fc ff df 49 8d bc 24 e8 08 00 00 48 89 fa 48 c1 ea 03 <80> 3c 02 00 75 37 49 8b bc 24 e8 08 00 00 48 89 ee e8 7e ac 02 00 RSP: 0018:ffffc900051dfe00 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: ffff888056c72400 RCX: 0000000000000000 RDX: 000000000000011d RSI: ffffffff841f12be RDI: 00000000000008e8 RBP: ffffc900051dfec8 R08: 0000000000000000 R09: ffffed100ad8e482 R10: ffffc900051dfde8 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000011 R15: ffffc900051dfec8 FS: 00007f1f3e2bc6c0(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f6671b75198 CR3: 0000000029652000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 ---------------- Code disassembly (best guess): 0: 48 c1 ea 03 shr $0x3,%rdx 4: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) 8: 75 64 jne 0x6e a: 4c 8b a3 a8 00 00 00 mov 0xa8(%rbx),%r12 11: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax 18: fc ff df 1b: 49 8d bc 24 e8 08 00 lea 0x8e8(%r12),%rdi 22: 00 23: 48 89 fa mov %rdi,%rdx 26: 48 c1 ea 03 shr $0x3,%rdx * 2a: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction 2e: 75 37 jne 0x67 30: 49 8b bc 24 e8 08 00 mov 0x8e8(%r12),%rdi 37: 00 38: 48 89 ee mov %rbp,%rsi 3b: e8 7e ac 02 00 call 0x2acbe
---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title
If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)
If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report
If you want to undo deduplication, reply with:
#syz undup
PATCH 1/1 io_uring: Don't set affinity on a dying sqpoll thread
Quoted 18 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: 626932085009 Add linux-next specific files for 20230825 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=12a97797a80000 kernel config: https://syzkaller.appspot.com/x/.config?x=8a8c992a790e5073 dashboard link: https://syzkaller.appspot.com/bug?extid=c74fea926a78b8a91042 compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/46ec18b3c2fb/disk-62693208.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/b4ea0cb78498/vmlinux-62693208.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5fb3938c7272/bzImage-62693208.xz
Quoted 8 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]general protection fault, probably for non-canonical address 0xdffffc000000011d: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x00000000000008e8-0x00000000000008ef] CPU: 1 PID: 27342 Comm: syz-executor.5 Not tainted 6.5.0-rc7-next-20230825-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023 RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433
Jens,
I'm not sure I got the whole story on this one, but it seems fairly trivial to reproduce and I can't see another way it could be triggered. What do you think?
Thanks,
-- >8 --
Subject: [PATCH] io_uring: Don't set affinity on a dying sqpoll thread
Syzbot reported a null-ptr-deref of sqd->thread inside io_sqpoll_wq_cpu_affinity. It turns out the sqd->thread can go away from under us during io_uring_register, in case the process gets a fatal signal during io_uring_register.
It is not particularly hard to hit the race, and while I am not sure this is the exact case hit by syzbot, it solves it. Finally, checking ->thread is enough to close the race because we locked sqd while "parking" the thread, thus preventing it from going away.
I reproduced it fairly consistently with a program that does:
int main(void) {
...
io_uring_queue_init(RING_LEN, &ring1, IORING_SETUP_SQPOLL);
while (1) {
io_uring_register_iowq_aff(ring, 1, &mask); }
}
Executed in a loop with timeout to trigger SIGTERM:
while true; do timeout 1 /a.out ; done
This will hit the following BUG() in very few attempts.
BUG: kernel NULL pointer dereference, address: 00000000000007a8 PGD 800000010e949067 P4D 800000010e949067 PUD 10e46e067 PMD 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 15715 Comm: dead-sqpoll Not tainted 6.5.0-rc7-next-20230825-g193296236fa0-dirty #23 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:io_sqpoll_wq_cpu_affinity+0x27/0x70 Code: 90 90 90 0f 1f 44 00 00 55 53 48 8b 9f 98 03 00 00 48 85 db 74 4f 48 89 df 48 89 f5 e8 e2 f8 ff ff 48 8b 43 38 48 85 c0 74 22 <48> 8b b8 a8 07 00 00 48 89 ee e8 ba b1 00 00 48 89 df 89 c5 e8 70 RSP: 0018:ffffb04040ea7e70 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff93c010749e40 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffffa7653331 RDI: 00000000ffffffff RBP: ffffb04040ea7eb8 R08: 0000000000000000 R09: c0000000ffffdfff R10: ffff93c01141b600 R11: ffffb04040ea7d18 R12: ffff93c00ea74840 R13: 0000000000000011 R14: 0000000000000000 R15: ffff93c00ea74800 FS: 00007fb7c276ab80(0000) GS:ffff93c36f200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000007a8 CR3: 0000000111634003 CR4: 0000000000370ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die_body+0x1a/0x60 ? page_fault_oops+0x154/0x440 ? do_user_addr_fault+0x174/0x7b0 ? exc_page_fault+0x63/0x140 ? asm_exc_page_fault+0x22/0x30 ? io_sqpoll_wq_cpu_affinity+0x27/0x70 __io_register_iowq_aff+0x2b/0x60 __io_uring_register+0x614/0xa70 __x64_sys_io_uring_register+0xaa/0x1a0 do_syscall_64+0x3a/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7fb7c226fec9 Code: 2e 00 b8 ca 00 00 00 0f 05 eb a5 66 0f 1f 44 00 00 48 89 f8 48 89
f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 7f 2d 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe2c0674f8 EFLAGS: 00000246 ORIG_RAX: 00000000000001ab
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb7c226fec9
RDX: 00007ffe2c067530 RSI: 0000000000000011 RDI: 0000000000000003
RBP: 00007ffe2c0675d0 R08: 00007ffe2c067550 R09: 00007ffe2c067550
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffe2c067750 R14: 0000000000000000 R15: 0000000000000000
</TASK> Modules linked in: CR2: 00000000000007a8 ---[ end trace 0000000000000000 ]---
--- io_uring/sqpoll.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
io_uring/sqpoll.c +3 -1
@@ -430,7 +430,9 @@ __cold int io_sqpoll_wq_cpu_affinity(struct io_ring_ctx *ctx,if (sqd) {io_sq_thread_park (sqd); - ret =io_wq_cpu_affinity (sqd->thread->io_uring, mask); +/* Don't set affinity for a dying thread */ +if (sqd->thread) + ret =io_wq_cpu_affinity (sqd->thread->io_uring, mask);io_sq_thread_unpark (sqd); }
Signature · 1 line
2.41.0
Re: [PATCH] io_uring: Don't set affinity on a dying sqpoll thread
...
Quoted 5 lines
Jens,
I'm not sure I got the whole story on this one, but it seems fairly trivial to reproduce and I can't see another way it could be triggered. What do you think?
Right, it can be null and that should be checked. I haven't taken a look at the report but the patch looks good regardless, thanks
Quoted 98 lines
-- >8 --
Subject: [PATCH]io_uring: Don't set affinity on a dying sqpoll threadSyzbot reported a null-ptr-deref of
sqd->threadinsideio_sqpoll_wq_cpu_affinity. It turns out thesqd->threadcan go away from under us duringio_uring_register, in case the process gets a fatal signal duringio_uring_register.It is not particularly hard to hit the race, and while I am not sure this is the exact case hit by syzbot, it solves it. Finally, checking ->thread is enough to close the race because we locked sqd while "parking" the thread, thus preventing it from going away.
I reproduced it fairly consistently with a program that does:
int main(void) { ... io_uring_queue_init(RING_LEN, &ring1, IORING_SETUP_SQPOLL); while (1) { io_uring_register_iowq_aff(ring, 1, &mask); } }Executed in a loop with timeout to trigger SIGTERM:
while true; do timeout 1 /a.out ; doneThis will hit the following
BUG()in very few attempts.BUG: kernel NULL pointer dereference, address: 00000000000007a8 PGD 800000010e949067 P4D 800000010e949067 PUD 10e46e067 PMD 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 15715 Comm: dead-sqpoll Not tainted 6.5.0-rc7-next-20230825-g193296236fa0-dirty #23 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:io_sqpoll_wq_cpu_affinity+0x27/0x70 Code: 90 90 90 0f 1f 44 00 00 55 53 48 8b 9f 98 03 00 00 48 85 db 74 4f 48 89 df 48 89 f5 e8 e2 f8 ff ff 48 8b 43 38 48 85 c0 74 22 <48> 8b b8 a8 07 00 00 48 89 ee e8 ba b1 00 00 48 89 df 89 c5 e8 70 RSP: 0018:ffffb04040ea7e70 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff93c010749e40 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffffffa7653331 RDI: 00000000ffffffff RBP: ffffb04040ea7eb8 R08: 0000000000000000 R09: c0000000ffffdfff R10: ffff93c01141b600 R11: ffffb04040ea7d18 R12: ffff93c00ea74840 R13: 0000000000000011 R14: 0000000000000000 R15: ffff93c00ea74800 FS: 00007fb7c276ab80(0000) GS:ffff93c36f200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000007a8 CR3: 0000000111634003 CR4: 0000000000370ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die_body+0x1a/0x60 ? page_fault_oops+0x154/0x440 ? do_user_addr_fault+0x174/0x7b0 ? exc_page_fault+0x63/0x140 ? asm_exc_page_fault+0x22/0x30 ? io_sqpoll_wq_cpu_affinity+0x27/0x70 __io_register_iowq_aff+0x2b/0x60 __io_uring_register+0x614/0xa70 __x64_sys_io_uring_register+0xaa/0x1a0 do_syscall_64+0x3a/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7fb7c226fec9 Code: 2e 00 b8 ca 00 00 00 0f 05 eb a5 66 0f 1f 44 00 00 48 89 f8 48 89f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 7f 2d 00 f7 d8 64 89 01 48
RSP: 002b:00007ffe2c0674f8 EFLAGS: 00000246ORIG_RAX:00000000000001ab
RAX: ffffffffffffffda RBX: 0000000000000000 RCX:00007fb7c226fec9
RDX:00007ffe2c067530RSI: 0000000000000011 RDI: 0000000000000003
RBP:00007ffe2c0675d0R08:00007ffe2c067550R09:00007ffe2c067550
R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
R13:00007ffe2c067750R14: 0000000000000000 R15: 0000000000000000</TASK> Modules linked in: CR2: 00000000000007a8 ---[ end trace 0000000000000000 ]---Reported-by [email protected]Fixes ebdfefc09c6d ("io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used")Signed-off-by Gabriel Krisman Bertazi <[email protected]>--- io_uring/sqpoll.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)io_uring/sqpoll.c +3
-1@@ -430,7 +430,9 @@ __cold int io_sqpoll_wq_cpu_affinity(struct io_ring_ctx *ctx,if (sqd) {io_sq_thread_park (sqd); - ret =io_wq_cpu_affinity (sqd->thread->io_uring, mask); +/* Don't set affinity for a dying thread */ +if (sqd->thread) + ret =io_wq_cpu_affinity (sqd->thread->io_uring, mask);io_sq_thread_unpark (sqd); }
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerRe: [PATCH] io_uring: Don't set affinity on a dying sqpoll thread
In reply to Gabriel Krisman BertaziQuoted 29 lines · Gabriel Krisman Bertazi
Quoted 18 lines · syzbot
Hello,
syzbot found the following issue on:
HEAD commit: 626932085009 Add linux-next specific files for 20230825 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=12a97797a80000 kernel config: https://syzkaller.appspot.com/x/.config?x=8a8c992a790e5073 dashboard link: https://syzkaller.appspot.com/bug?extid=c74fea926a78b8a91042 compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/46ec18b3c2fb/disk-62693208.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/b4ea0cb78498/vmlinux-62693208.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5fb3938c7272/bzImage-62693208.xzQuoted 8 lines
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]general protection fault, probably for non-canonical address 0xdffffc000000011d: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x00000000000008e8-0x00000000000008ef] CPU: 1 PID: 27342 Comm: syz-executor.5 Not tainted 6.5.0-rc7-next-20230825-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023 RIP: 0010:io_sqpoll_wq_cpu_affinity+0x8c/0xe0 io_uring/sqpoll.c:433
Quoted 5 lines
Jens,
I'm not sure I got the whole story on this one, but it seems fairly trivial to reproduce and I can't see another way it could be triggered. What do you think?
Yep looks like the right fix, we should check the thread after parking. I'll get this added to the queue, thanks.
Signature · 1 line
Jens Axboe
Re: [PATCH] io_uring: Don't set affinity on a dying sqpoll thread
In reply to Gabriel Krisman BertaziQuoted 11 lines · Gabriel Krisman Bertazi
Syzbot reported a null-ptr-deref of
sqd->threadinsideio_sqpoll_wq_cpu_affinity. It turns out thesqd->threadcan go away from under us duringio_uring_register, in case the process gets a fatal signal duringio_uring_register.It is not particularly hard to hit the race, and while I am not sure this is the exact case hit by syzbot, it solves it. Finally, checking ->thread is enough to close the race because we locked sqd while "parking" the thread, thus preventing it from going away.
[...]
Applied, thanks!
[1/1] io_uring: Don't set affinity on a dying sqpoll thread
commit: 37b1499057f61de370dbd8f0c296274ab7186605
Best regards,
Signature · 1 line
Jens Axboe