Soft lockups during reading /proc/PID/smaps
Subsystems (2)
I'm getting weird soft lockups while reading smaps on loaded systems with some background cgroups usage. This issue can be reproduced with the most recent kernel.
Here's the stack trace:
[ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857] [ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt psmouse serio_raw ppdev parport_pc i2c_piix4 parport xen_fbfront fb_sys_fops syscopyarea sysfillrect sysimgblt mac_hid isofs raid10 raid456 async_memcpy async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd floppy
[ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted 3.15.5-031505-generic #201407091543 [ 1748.312052] Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 [ 1748.312052] task: ffff8800eab41930 ti: ffff8803b9a94000 task.ti: ffff8803b9a94000 [ 1748.312052] RIP: 0010:[<ffffffff81013111>] [<ffffffff81013111>] KSTK_ESP+0x11/0x40 [ 1748.312052] RSP: 0018:ffff8803b9a97c68 EFLAGS: 00000287 [ 1748.312052] RAX: 0000000000000000 RBX: ffff8803b60de3aa RCX: 00007f49ec000000 [ 1748.312052] RDX: 0000000000000001 RSI: ffff8800eba1c730 RDI: ffff880399434b90 [ 1748.312052] RBP: ffff8803b9a97c68 R08: 000000000000000a R09: 000000000000fffe [ 1748.312052] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000001 [ 1748.312052] R13: ffff8803b7a74108 R14: 00007f49ec021000 R15: ffff8803b9a9ffff [ 1748.312052] FS: 00007fcc9562b740(0000) GS:ffff8803cfcc0000(0000) knlGS:0000000000000000 [ 1748.312052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1748.312052] CR2: 00007fcc955ed180 CR3: 00000003b97a5000 CR4: 00000000001406e0 [ 1748.312052] Stack: [ 1748.312052] ffff8803b9a97ca8 ffffffff8117bcc9 ffff8803b9a97df8 ffff880399435030 [ 1748.312052] ffff88003684d000 ffff8800eba1c730 0000000000000000 ffff8803b5dfc980 [ 1748.312052] ffff8803b9a97d38 ffffffff81236612 ffff88030000002d 00007f4900000070 [ 1748.312052] Call Trace: [ 1748.312052] [<ffffffff8117bcc9>] vm_is_stack+0x59/0xe0 [ 1748.312052] [<ffffffff81236612>] show_map_vma+0x212/0x280 [ 1748.312052] [<ffffffff81236805>] show_smap+0x85/0x250 [ 1748.312052] [<ffffffff81237bc0>] ? smaps_pte_entry.isra.21+0x220/0x220 [ 1748.312052] [<ffffffff81236a03>] show_pid_smap+0x13/0x20 [ 1748.312052] [<ffffffff811f6016>] seq_read+0x256/0x3e0 [ 1748.312052] [<ffffffff811d30e1>] vfs_read+0xb1/0x180 [ 1748.312052] [<ffffffff811d335f>] SyS_read+0x4f/0xb0 [ 1748.312052] [<ffffffff8178527f>] tracesys+0xe1/0xe6 [ 1748.312052] Code: 89 f8 48 89 f2 89 c6 48 89 e5 65 48 8b 3c 25 c0 c7 00
00 e8 b2 f9 ff ff 5d c3 0f 1f 44 00 00 48 8b 47 08 55 48 89 e5 48 8b 40 10 <a9> 00 00 02 00 75 10 48 8b 87 10 06 00 00 5d c3 0f 1f 80 00 00
Quoted 16 lines · Aleksei Besogonov
I'm getting weird soft lockups while reading smaps on loaded systems with some background cgroups usage. This issue can be reproduced with the most recent kernel.
Here's the stack trace:
[ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857] [ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
iptable_filteript_MASQUERADEiptable_natnf_conntrack_ipv4nf_defrag_ipv4nf_nat_ipv4nf_natnf_conntrackip_tablesx_tablesbridge stp llcdm_thin_pooldm_persistent_datadm_bio_prisondm_bufiolibcrc32c nfsdauth_rpcgssnfs_aclnfs lockd sunrpc fscachedm_cryptpsmouseserio_rawppdevparport_pci2c_piix4parportxen_fbfrontfb_sys_fopssyscopyarea sysfillrect sysimgbltmac_hidisofs raid10 raid456async_memcpyasync_raid6_recovasync_pqasync_xorasync_txxorraid6_pqraid1 raid0 multipath linearcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelaesni_intelaes_x86_64lrw gf128mulglue_helperablk_helpercryptd floppy
Quoted 2 lines
[ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted3.15.5-031505-generic #201407091543
This isn't the most recent kernel, we're at 3.16-rc7 now, but I don't think there are any changes that would prevent this.
Quoted 24 lines
[ 1748.312052] Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 [ 1748.312052] task: ffff8800eab41930 ti: ffff8803b9a94000 task.ti: ffff8803b9a94000 [ 1748.312052] RIP: 0010:[<ffffffff81013111>] [<ffffffff81013111>] KSTK_ESP+0x11/0x40 [ 1748.312052] RSP: 0018:ffff8803b9a97c68 EFLAGS: 00000287 [ 1748.312052] RAX: 0000000000000000 RBX: ffff8803b60de3aa RCX: 00007f49ec000000 [ 1748.312052] RDX: 0000000000000001 RSI: ffff8800eba1c730 RDI: ffff880399434b90 [ 1748.312052] RBP: ffff8803b9a97c68 R08: 000000000000000a R09: 000000000000fffe [ 1748.312052] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000001 [ 1748.312052] R13: ffff8803b7a74108 R14: 00007f49ec021000 R15: ffff8803b9a9ffff [ 1748.312052] FS: 00007fcc9562b740(0000) GS:ffff8803cfcc0000(0000) knlGS:0000000000000000 [ 1748.312052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1748.312052] CR2: 00007fcc955ed180 CR3: 00000003b97a5000 CR4: 00000000001406e0 [ 1748.312052] Stack: [ 1748.312052] ffff8803b9a97ca8 ffffffff8117bcc9 ffff8803b9a97df8 ffff880399435030 [ 1748.312052] ffff88003684d000 ffff8800eba1c730 0000000000000000 ffff8803b5dfc980 [ 1748.312052] ffff8803b9a97d38 ffffffff81236612 ffff88030000002d 00007f4900000070 [ 1748.312052] Call Trace: [ 1748.312052] [<ffffffff8117bcc9>] vm_is_stack+0x59/0xe0
Quoted 8 lines
[ 1748.312052] [<ffffffff81236612>] show_map_vma+0x212/0x280 [ 1748.312052] [<ffffffff81236805>] show_smap+0x85/0x250 [ 1748.312052] [<ffffffff81237bc0>] ? smaps_pte_entry.isra.21+0x220/0x220 [ 1748.312052] [<ffffffff81236a03>] show_pid_smap+0x13/0x20 [ 1748.312052] [<ffffffff811f6016>] seq_read+0x256/0x3e0 [ 1748.312052] [<ffffffff811d30e1>] vfs_read+0xb1/0x180 [ 1748.312052] [<ffffffff811d335f>] SyS_read+0x4f/0xb0 [ 1748.312052] [<ffffffff8178527f>] tracesys+0xe1/0xe6
The while_each_thread() in vm_is_stack() looks suspicious since the task isn't current and rcu won't protect the iteration, and we also don't hold sighand lock or a readlock on tasklist_lock.
I think Oleg will know how to proceed, cc'd.
Quoted 19 lines · David Rientjes
Quoted 16 lines · Aleksei Besogonov
I'm getting weird soft lockups while reading smaps on loaded systems with some background cgroups usage. This issue can be reproduced with the most recent kernel.
Here's the stack trace:
[ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857] [ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
iptable_filteript_MASQUERADEiptable_natnf_conntrack_ipv4nf_defrag_ipv4nf_nat_ipv4nf_natnf_conntrackip_tablesx_tablesbridge stp llcdm_thin_pooldm_persistent_datadm_bio_prisondm_bufiolibcrc32c nfsdauth_rpcgssnfs_aclnfs lockd sunrpc fscachedm_cryptpsmouseserio_rawppdevparport_pci2c_piix4parportxen_fbfrontfb_sys_fopssyscopyarea sysfillrect sysimgbltmac_hidisofs raid10 raid456async_memcpyasync_raid6_recovasync_pqasync_xorasync_txxorraid6_pqraid1 raid0 multipath linearcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelaesni_intelaes_x86_64lrw gf128mulglue_helperablk_helpercryptd floppyQuoted 2 lines
[ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted3.15.5-031505-generic #201407091543
Quoted 2 lines
This isn't the most recent kernel, we're at 3.16-rc7 now, but I don't think there are any changes that would prevent this.
Yes, I tested it with the rc7, the error report is from a previous run with an Ubuntu 14.04 kernel.
Quoted 4 lines
The
while_each_thread()invm_is_stack()looks suspicious since the task isn't current and rcu won't protect the iteration, and we also don't hold sighand lock or a readlock ontasklist_lock.
I think Oleg will know how to proceed, cc'd.
I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
#!/usr/bin/env python2.7
from os import mkdir
from threading import Thread
from time import sleep
import os
__author__ = 'cyberax'
count = 0
def threadproc():
global count
count += 1
sleep(0.01)
count -= 1def do_threads():
sleep(2)
while True:
while count > 200:
sleep(0.01)
th = Thread(target=threadproc)
th.start()def do_reader(pid):
while True:
with open("/sys/fs/cgroup/memory/ck/1001/tasks", "r") as fl:
fl.readlines()
with open("/sys/fs/cgroup/memory/ck/1001/delegate/tasks", "r") as fl:
lines = fl.readlines()
for l in lines:
try:
with open("/proc/%s/smaps" % l.strip(), "r") as fl:
fl.readlines()
except:
passpid = os.fork()
if pid == 0:
do_threads()
exit(0)try:
mkdir('/sys/fs/cgroup/memory/ck')
mkdir('/sys/fs/cgroup/memory/ck/1001')
mkdir('/sys/fs/cgroup/memory/ck/1001/delegate')except:
pass
with open('/sys/fs/cgroup/memory/ck/1001/delegate/tasks', 'w') as fl:
fl.write('%d\n' % pid)do_reader(pid)
Quoted 8 lines · Aleksei Besogonov
Quoted 4 lines · David Rientjes
The
while_each_thread()invm_is_stack()looks suspicious since the task isn't current and rcu won't protect the iteration, and we also don't hold sighand lock or a readlock ontasklist_lock.
I think Oleg will know how to proceed, cc'd.I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
Thanks. I think David is right and we need the simple patch below. This reminds me I should kill while_each_thread :/
Any chance you can test it? If not, I will do this later and send the patch if it helps.
Oleg.
mm/util.c +3 -6
@@ -277,17 +277,14 @@ pid_t vm_is_stack(struct task_struct *taif (in_group) {struct task_struct *t; -rcu_read_lock (); -if (!pid_alive (task)) - goto done; - t = task; - do { +rcu_read_lock (); +for_each_thread (task, t) {if (vm_is_stack_for_task (t, vma)) { ret = t->pid; goto done; } - } while_each_thread(task, t); + } done:rcu_read_unlock (); }
Quoted 8 lines · Oleg Nesterov
Quoted 7 lines · Aleksei Besogonov
Quoted 4 lines · David Rientjes
The
while_each_thread()invm_is_stack()looks suspicious since the task isn't current and rcu won't protect the iteration, and we also don't hold sighand lock or a readlock ontasklist_lock.
I think Oleg will know how to proceed, cc'd.
Quoted 5 lines
Quoted 1 line
I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
Thanks. I think David is right and we need the simple patch below. This reminds me I should kill
while_each_thread:/
Any chance you can test it? If not, I will do this later and send the patch if it helps.
Thanks, it works on the rc7 kernel.
I can make a backported version for earlier kernels if nobody else is interested.
Quoted 27 lines
Oleg.
mm/util.c +3
-6@@ -277,17 +277,14 @@ pid_t vm_is_stack(struct task_struct *taif (in_group) {struct task_struct *t; -rcu_read_lock (); -if (!pid_alive (task)) - goto done; - t = task; - do { +rcu_read_lock (); +for_each_thread (task, t) {if (vm_is_stack_for_task (t, vma)) { ret = t->pid; goto done; } - } while_each_thread(task, t); + }done:
rcu_read_unlock(); }
Quoted 2 lines · Aleksei Besogonov
Thanks, it works on the rc7 kernel.
Great!
I didn't have time today, will write the changelog and send the patch tomorrow.
Thanks a lot Aleksei.
Oleg.