Crash in cgroup_procs_show
Hi,
Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279
BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0 PGD 0 P4D 0 Oops: 0000 [#1] SMP Modules linked in: fuse vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle bridge iptable_raw stp llc iptable_security ebtable_filter ebtables ip6table_filter ip6_tables sunrpc vfat fat edac_mce_amd eeepc_wmi kvm_amd asus_wmi wmi_bmof sparse_keymap rfkill video kvm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi irqbypass crct10dif_pclmul snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel snd_hda_core snd_hwdep snd_seq joydev snd_seq_device snd_pcm snd_timer sp5100_tco i2c_piix4 ccp snd soundcore tpm_tis shpchp tpm_tis_core tpm wmi acpi_cpufreq binfmt_misc btrfs xor zstd_decompress zstd_compress xxhash raid6_pq hid_logitech_hidpp amdkfd amd_iommu_v2 amdgpu i2c_algo_bit drm_kms_helper ttm drm crc32c_intel r8169 hid_logitech_dj mii nvme nvme_core i2c_dev CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1 Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017 task: ffff88c4baee8000 task.stack: ffff96d5c3158000 RIP: 0010:__task_pid_nr_ns+0xc7/0xf0 RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250 RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00 RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005 R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18 R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000 FS: 00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0 Call Trace: cgroup_procs_show+0x19/0x30 cgroup_seqfile_show+0x4c/0xb0 kernfs_seq_show+0x21/0x30 seq_read+0x2ec/0x3f0 kernfs_fop_read+0x134/0x180 __vfs_read+0x37/0x160 ? security_file_permission+0x9b/0xc0 vfs_read+0x8e/0x130 SyS_read+0x55/0xc0 entry_SYSCALL_64_fastpath+0x1a/0xa5 RIP: 0033:0x7f94455f942d RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60 R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0 R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560 Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50
It looks like a few others reported similar problems
https://marc.info/?l=linux-kernel&m=151210295120013
https://marc.info/?l=linux-kernel&m=151151651301420
I haven't yet asked the reporter to try and bisect. Is there a fix queued up for this yet?
Thanks,
Laura
Tejun Heo
maintainerHello, Laura.
Quoted 7 lines · Laura Abbott
Hi,
Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279
BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0
Any chance you can map this back to the source line with addr2line?
Thanks.
Signature · 1 line
tejun
Quoted 8 lines · Tejun Heo
Hello, Laura.
Quoted 5 lines · Laura Abbott
Hi,
Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279
Quoted 7 lines
Quoted 2 lines
BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0Any chance you can map this back to the source line with addr2line?
Thanks.
annotated with decode_stacktrace.sh
IP: __task_pid_nr_ns (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:506 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:535)
cgroup_procs_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/cgroup/cgroup.c:4240)cgroup_seqfile_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/cgroup/cgroup.c:3413)kernfs_seq_show (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/kernfs/file.c:169)seq_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/seq_file.c:269)kernfs_fop_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/kernfs/file.c:252)__vfs_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:416)
? security_file_permission (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/include/linux/fsnotify.h:56 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/security/security.c:867) vfs_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:448)SyS_read (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:574 /usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/fs/read_write.c:566) entry_SYSCALL_64_fastpath (/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/arch/x86/entry/entry_64.S:206)
All code
========
0: 04 74 add $0x74,%al 2: 0e (bad) 3: 89 f6 mov %esi,%esi 5: 48 8d 04 76 lea (%rsi,%rsi,2),%rax 9: 48 8d 04 c5 f0 05 00 lea 0x5f0(,%rax,8),%rax 10: 00 11: 48 8b bf b8 05 00 00 mov 0x5b8(%rdi),%rdi 18: 48 01 c7 add %rax,%rdi 1b: 31 c0 xor %eax,%eax 1d: 48 8b 0f mov (%rdi),%rcx 20: 48 85 c9 test %rcx,%rcx 23: 74 18 je 0x3d 25: 8b b2 30 08 00 00 mov 0x830(%rdx),%esi 2b:* 3b 71 04 cmp 0x4(%rcx),%esi <-- trapping instruction 2e: 77 0d ja 0x3d 30: 48 c1 e6 05 shl $0x5,%rsi 34: 48 01 f1 add %rsi,%rcx 37: 48 3b 51 38 cmp 0x38(%rcx),%rdx 3b: 74 09 je 0x46 3d: 5d pop %rbp 3e: c3 retq 3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 3b 71 04 cmp 0x4(%rcx),%esi 3: 77 0d ja 0x12 5: 48 c1 e6 05 shl $0x5,%rsi 9: 48 01 f1 add %rsi,%rcx c: 48 3b 51 38 cmp 0x38(%rcx),%rdx 10: 74 09 je 0x1b 12: 5d pop %rbp 13: c3 retq 14: 8b .byte 0x8b
Tejun Heo
maintainerHello, Laura.
Quoted 14 lines · Laura Abbott
Quoted 13 lines · Tejun Heo
Hello, Laura.
Quoted 7 lines · Laura Abbott
Hi,
Fedora got a bug report on 4.14.4 of a crash on reboot https://bugzilla.redhat.com/show_bug.cgi?id=1525279
BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0Any chance you can map this back to the source line with addr2line?
Quoted 7 lines
Quoted 2 lines
Thanks.
annotated with
decode_stacktrace.shIP:
__task_pid_nr_ns(/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:506/usr/src/debug/kernel-4.14.fc26/linux-4.14.4-200.fc26.x86_64/kernel/pid.c:535)
Thank you very much for the trace. I *think* I found the place where the css task iterator points to a task without holding its ref allowing the task to go away before dereferenced. I'll update once I learn more.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 1/1 cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC
Hello,
Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.
Thank you very much.
------ 8< ------
From 74d0833c659a8a54735e5efdd44f4b225af68586 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Wed, 20 Dec 2017 07:09:19 -0800
While teaching css_task_iter to handle skipping over tasks which aren't group leaders, bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS") introduced a silly bug.
CSS_TASK_ITER_PROCS is implemented by repeating css_task_iter_advance() while the advanced cursor is pointing to a non-leader thread. However, the cursor variable, @l, wasn't updated when the iteration has to advance to the next css_set and the following repetition would operate on the terminal @l from the previous iteration which isn't pointing to a valid task leading to oopses like the following or infinite looping.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0 PGD 0 P4D 0 Oops: 0000 [#1] SMP ... CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1 Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017 task: ffff88c4baee8000 task.stack: ffff96d5c3158000 RIP: 0010:__task_pid_nr_ns+0xc7/0xf0 RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250 RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00 RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005 R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18 R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000 FS: 00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0 Call Trace: cgroup_procs_show+0x19/0x30 cgroup_seqfile_show+0x4c/0xb0 kernfs_seq_show+0x21/0x30 seq_read+0x2ec/0x3f0 kernfs_fop_read+0x134/0x180 __vfs_read+0x37/0x160 ? security_file_permission+0x9b/0xc0 vfs_read+0x8e/0x130 SyS_read+0x55/0xc0 entry_SYSCALL_64_fastpath+0x1a/0xa5 RIP: 0033:0x7f94455f942d RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60 R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0 R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560 Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50
Fix it by moving the initialization of the cursor below the repeat label. While at it, rename it to @next for readability.
--- kernel/cgroup/cgroup.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/cgroup/cgroup.c +6 -8
@@ -4125,26 +4125,24 @@ static void css_task_iter_advance_css_set(struct css_task_iter *it)static void css_task_iter_advance (struct css_task_iter *it) { -struct list_head *l = it->task_pos; +struct list_head *next;lockdep_assert_held (&css_set_lock); -WARN_ON_ONCE (!l); - repeat:/* * Advance iterator to find next entry. cset->tasks is consumed * first and then ->mg_tasks. After ->mg_tasks, we move onto the * next cset. */ - l = l->next; + next = it->task_pos->next; -if (l == it->tasks_head) - l = it->mg_tasks_head->next; +if (next == it->tasks_head) + next = it->mg_tasks_head->next; -if (l == it->mg_tasks_head) +if (next == it->mg_tasks_head)css_task_iter_advance_css_set (it);else - it->task_pos = l; + it->task_pos = next;/* if PROCS, skip over tasks which aren't group leaders */ if ((it->flags & CSS_TASK_ITER_PROCS) && it->task_pos &&
Signature · 1 line
2.9.5
Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC
This resolves the bug for me.
Thank you,
George
Quoted 120 lines · Tejun Heo
Hello,
Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.
Thank you very much.
------ 8< ------
From74d0833c659a8a54735e5efdd44f4b225af68586Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Wed, 20 Dec 2017 07:09:19 -0800While teaching
css_task_iterto handle skipping over tasks which aren't group leaders,bc2fb7ed089f("cgroup: add @flags tocss_task_iter_start()and implementCSS_TASK_ITER_PROCS") introduced a silly bug.
CSS_TASK_ITER_PROCSis implemented by repeatingcss_task_iter_advance()while the advanced cursor is pointing to a non-leader thread. However, the cursor variable, @l, wasn't updated when the iteration has to advance to the nextcss_setand the following repetition would operate on the terminal @l from the previous iteration which isn't pointing to a valid task leading to oopses like the following or infinite looping.BUG: unable to handle kernel NULL pointer dereference at 0000000000000254 IP: __task_pid_nr_ns+0xc7/0xf0 PGD 0 P4D 0 Oops: 0000 [#1] SMP ... CPU: 2 PID: 1 Comm: systemd Not tainted 4.14.4-200.fc26.x86_64 #1 Hardware name: System manufacturer System Product Name/PRIME B350M-A, BIOS 3203 11/09/2017 task: ffff88c4baee8000 task.stack: ffff96d5c3158000 RIP: 0010:__task_pid_nr_ns+0xc7/0xf0 RSP: 0018:ffff96d5c315bd50 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff88c4b68c6000 RCX: 0000000000000250 RDX: ffffffffa5e47960 RSI: 0000000000000000 RDI: ffff88c490f6ab00 RBP: ffff96d5c315bd50 R08: 0000000000001000 R09: 0000000000000005 R10: ffff88c4be006b80 R11: ffff88c42f1b8004 R12: ffff96d5c315bf18 R13: ffff88c42d7dd200 R14: ffff88c490f6a510 R15: ffff88c4b68c6000 FS: 00007f9446f8ea00(0000) GS:ffff88c4be680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000254 CR3: 00000007f956f000 CR4: 00000000003406e0 Call Trace: cgroup_procs_show+0x19/0x30 cgroup_seqfile_show+0x4c/0xb0 kernfs_seq_show+0x21/0x30 seq_read+0x2ec/0x3f0 kernfs_fop_read+0x134/0x180 __vfs_read+0x37/0x160 ? security_file_permission+0x9b/0xc0 vfs_read+0x8e/0x130 SyS_read+0x55/0xc0 entry_SYSCALL_64_fastpath+0x1a/0xa5 RIP: 0033:0x7f94455f942d RSP: 002b:00007ffe81ba2d00 EFLAGS: 00000293 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 00005574e2233f00 RCX: 00007f94455f942d RDX: 0000000000001000 RSI: 00005574e2321a90 RDI: 000000000000002b RBP: 0000000000000000 R08: 00005574e2321a90 R09: 00005574e231de60 R10: 00007f94458c8b38 R11: 0000000000000293 R12: 00007f94458c8ae0 R13: 00007ffe81ba3800 R14: 0000000000000000 R15: 00005574e2116560 Code: 04 74 0e 89 f6 48 8d 04 76 48 8d 04 c5 f0 05 00 00 48 8b bf b8 05 00 00 48 01 c7 31 c0 48 8b 0f 48 85 c9 74 18 8b b2 30 08 00 00 <3b> 71 04 77 0d 48 c1 e6 05 48 01 f1 48 3b 51 38 74 09 5d c3 8b RIP: __task_pid_nr_ns+0xc7/0xf0 RSP: ffff96d5c315bd50Fix it by moving the initialization of the cursor below the repeat label. While at it, rename it to @next for readability.
Signed-off-by Tejun Heo <[email protected]>Fixes bc2fb7ed089f ("cgroup: add @flags to css_task_iter_start() and implement CSS_TASK_ITER_PROCS")Cc [email protected] # v4.14+Reported-by Laura Abbott <[email protected]>Reported-by Bronek Kozicki <[email protected]>Reported-by George Amanakis <[email protected]>Signed-off-by Tejun Heo <[email protected]>--- kernel/cgroup/cgroup.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)kernel/cgroup/cgroup.c +6
-9@@ -4125,26 +4125,24 @@ static void css_task_iter_advance_css_set(struct css_task_iter *it)static void css_task_iter_advance (struct css_task_iter *it) { -struct list_head *l = it->task_pos; +struct list_head *next;lockdep_assert_held (&css_set_lock); -WARN_ON_ONCE (!l); - repeat:/* * Advance iterator to find next entry. cset->tasks is consumed * first and then ->mg_tasks. After ->mg_tasks, we move onto the * next cset. */ - l = l->next; + next = it->task_pos->next; -if (l == it->tasks_head) - l = it->mg_tasks_head->next; +if (next == it->tasks_head) + next = it->mg_tasks_head->next; -if (l == it->mg_tasks_head) +if (next == it->mg_tasks_head)css_task_iter_advance_css_set (it);else - it->task_pos = l; + it->task_pos = next;/* if PROCS, skip over tasks which aren't group leaders */ if ((it->flags & CSS_TASK_ITER_PROCS) && it->task_pos && --2.9.5
Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC
In reply to Tejun HeoQuoted 6 lines · Tejun Heo
Hello,
Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. I could reproduce the problem reliably and am pretty sure this is the right fix but I'd greatly appreciate if you guys can confirm the fix too.
Fixed here as well, 4.14.8 stable with your fix on top.
Thanks,
Rouven
Re: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC
In reply to Tejun HeoHello,
Quoted 2 lines · Tejun Heo
Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. [...]
I can't find your fixes in master yet, did you perhaps forget to push out?
Thanks and a happy new year,
Rouven
Tejun Heo
maintainerRe: [PATCH cgroup/for-4.15-fixes] cgroup: fix css_task_iter crash on CSS_TASK_ITER_PROC
Quoted 9 lines · Rouven Czerwinski
Hello,
Quoted 2 lines · Tejun Heo
Applied the following to cgroup/for-4.15-fixes. Will push out to linus later this week. [...]
I can't find your fixes in master yet, did you perhaps forget to push out?
Thanks for reminding me. Sent out the pull request.
Happy new year!
Signature · 1 line
tejun