sched: softlockups in multi_cpu_stop
Hi all,
I'm seeing the following lockup pretty often while fuzzing with trinity:
[ 880.960250] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 447s! [migration/1:14] [ 880.960700] Modules linked in: [ 880.960700] irq event stamp: 380954 [ 880.960700] hardirqs last enabled at (380953): restore_args (arch/x86/kernel/entry_64.S:780) [ 880.960700] hardirqs last disabled at (380954): apic_timer_interrupt (arch/x86/kernel/entry_64.S:920) [ 880.960700] softirqs last enabled at (380952): __do_softirq (./arch/x86/include/asm/preempt.h:22 kernel/softirq.c:300) [ 880.960700] softirqs last disabled at (380947): irq_exit (kernel/softirq.c:350 kernel/softirq.c:391) [ 880.960700] CPU: 1 PID: 14 Comm: migration/1 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880052cc8000 ti: ffff880052cd0000 task.ti: ffff880052cd0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880052cd7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: 0000000042506841 RCX: 1ffff1000a5992d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880052cd7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9315e211 [ 880.960700] R13: ffff880052cd7ae8 R14: ffffffff93220213 R15: ffff880052cd7ad8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880053200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002d88fd8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8805d4f67728 0000000000000282 [ 880.960700] ffff880000000000 00ff880052cc8000 ffff880052cd7bd8 dffffc0000000000 [ 880.960700] ffff880053218440 ffff8805d4f67778 ffff8805d4f67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] sending NMI to other CPUs: [ 881.290044] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.336 msecs [ 881.292380] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.083 msecs [ 881.297603] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 7.423 msecs [ 880.960700] NMI backtrace for cpu 0 [ 880.960700] CPU: 0 PID: 11 Comm: migration/0 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88000d6e3000 ti: ffff88000d6f0000 task.ti: ffff88000d6f0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88000d6f7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0007c21ee5 RCX: 1ffff10001adc8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88000d6f7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88003e10f738 R14: 0000000000000001 R15: ffff88003e10f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880029000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002d31fb0 CR3: 000000002082c000 CR4: 00000000000007b0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88003e10f728 0000000000000282 [ 880.960700] ffff880000000000 00ff88000d6e3000 ffff88000d6f7bd8 dffffc0000000000 [ 880.960700] ffff880029018440 ffff88003e10f778 ffff88003e10f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 2 [ 880.960700] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802ccca0000 ti: ffff88000dee8000 task.ti: ffff88000dee8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000deefd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000deefd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000deeffd8 R14: 0000000000000000 R15: ffff88000deeffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f6baacd10f8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000deefd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000deeffd8 0000000000000000 ffff88000deefd78 ffffffff931629ef [ 880.960700] ffff88000deefe68 ffffffff934018e0 ffff88000deefde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 3 [ 880.960700] CPU: 3 PID: 32 Comm: migration/3 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8800a6cc0000 ti: ffff8800a6cc8000 task.ti: ffff8800a6cc8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8800a6ccfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0067e09ee5 RCX: 1ffff10014d982d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8800a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88033f04f738 R14: 0000000000000001 R15: ffff88033f04f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800a7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001490fe8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88033f04f728 0000000000000282 [ 880.960700] ffff880000000000 00ff8800a6cc0000 ffff8800a6ccfbd8 dffffc0000000000 [ 880.960700] ffff8800a7218440 ffff88033f04f778 ffff88033f04f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 4 [ 880.960700] CPU: 4 PID: 41 Comm: migration/4 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8800cacb3000 ti: ffff8800cacc0000 task.ti: ffff8800cacc0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8800cacc7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed006e615ee5 RCX: 1ffff100195968d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8800cacc7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8803730af738 R14: 0000000000000001 R15: ffff8803730af718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800cf200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000189efd8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803730af728 0000000000000282 [ 880.960700] ffff880000000000 00ff8800cacb3000 ffff8800cacc7bd8 dffffc0000000000 [ 880.960700] ffff8800cf218440 ffff8803730af778 ffff8803730af718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 5 [ 880.960700] CPU: 5 PID: 50 Comm: migration/5 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880128c6b000 ti: ffff880128c88000 task.ti: ffff880128c88000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880128c8fb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008c0bbee5 RCX: 1ffff1002518d8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880128c8fbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8804605df738 R14: 0000000000000001 R15: ffff8804605df718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880129000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002bf8ff0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8804605df728 0000000000000282 [ 880.960700] ffff880100000000 00ff880128c6b000 ffff880128c8fbd8 dffffc0000000000 [ 880.960700] ffff880129018440 ffff8804605df778 ffff8804605df718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 6 [ 880.960700] CPU: 6 PID: 59 Comm: migration/6 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880152cb3000 ti: ffff880152cc0000 task.ti: ffff880152cc0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880152cc7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0063d8cee5 RCX: 1ffff1002a5968d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880152cc7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88031ec67738 R14: 0000000000000001 R15: ffff88031ec67718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880153200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002b54fa0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88031ec67728 0000000000000282 [ 880.960700] ffff880100000000 00ff880152cb3000 ffff880152cc7bd8 dffffc0000000000 [ 880.960700] ffff880153218440 ffff88031ec67778 ffff88031ec67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 7 [ 880.960700] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8808dd0b0000 ti: ffff88000e2d0000 task.ti: ffff88000e2d0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0000:ffff88000e2d7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e2d7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e2d7fd8 R14: 0000000000000000 R15: ffff88000e2d7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88017d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000000000 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e2d7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e2d7fd8 0000000000000000 ffff88000e2d7d78 ffffffff931629ef [ 880.960700] ffff88000e2d7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 8 [ 880.960700] CPU: 8 PID: 77 Comm: migration/8 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801a6cc0000 ti: ffff8801a6cc8000 task.ti: ffff8801a6cc8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8801a6ccfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed003abaaee5 RCX: 1ffff10034d982d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8801a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8801d5d57738 R14: 0000000000000001 R15: ffff8801d5d57718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801a7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000e69fc8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8801d5d57728 0000000000000282 [ 880.960700] ffff880100000000 00ff8801a6cc0000 ffff8801a6ccfbd8 dffffc0000000000 [ 880.960700] ffff8801a7218440 ffff8801d5d57778 ffff8801d5d57718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 9 [ 880.960700] CPU: 9 PID: 86 Comm: migration/9 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801d0c88000 ti: ffff8801d0c90000 task.ti: ffff8801d0c90000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8801d0c97b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed007245bee5 RCX: 1ffff1003a1912d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8801d0c97bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8803922df738 R14: 0000000000000001 R15: ffff8803922df718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801d1000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029a8fb0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803922df728 0000000000000282 [ 880.960700] ffff880100000000 00ff8801d0c88000 ffff8801d0c97bd8 dffffc0000000000 [ 880.960700] ffff8801d1018440 ffff8803922df778 ffff8803922df718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 10 [ 880.960700] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880128c68000 ti: ffff88000e6c0000 task.ti: ffff88000e6c0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6c7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6c7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6c7fd8 R14: 0000000000000000 R15: ffff88000e6c7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801fb200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000636668 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6c7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6c7fd8 0000000000000000 ffff88000e6c7d78 ffffffff931629ef [ 880.960700] ffff88000e6c7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 11 [ 880.960700] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801a6ca8000 ti: ffff88000e6c8000 task.ti: ffff88000e6c8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6cfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6cfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6cffd8 R14: 0000000000000000 R15: ffff88000e6cffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880225200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000030db000 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6cfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6cffd8 0000000000000000 ffff88000e6cfd78 ffffffff931629ef [ 880.960700] ffff88000e6cfe68 ffffffff934018e0 ffff88000e6cfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 12 [ 880.960700] CPU: 12 PID: 113 Comm: migration/12 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88024ecd0000 ti: ffff88024ecd8000 task.ti: ffff88024ecd8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88024ecdfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0073a32ee5 RCX: 1ffff10049d9a2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88024ecdfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88039d197738 R14: 0000000000000001 R15: ffff88039d197718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88024f200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002648fe0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039d197728 0000000000000282 [ 880.960700] ffff880200000000 00ff88024ecd0000 ffff88024ecdfbd8 dffffc0000000000 [ 880.960700] ffff88024f218440 ffff88039d197778 ffff88039d197718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 13 [ 880.960700] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802f6cc0000 ti: ffff88000e6d8000 task.ti: ffff88000e6d8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6dfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6dfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6dffd8 R14: 0000000000000000 R15: ffff88000e6dffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880279000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000063a69c CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6dffd8 0000000000000000 ffff88000e6dfd78 ffffffff931629ef [ 880.960700] ffff88000e6dfe68 ffffffff934018e0 ffff88000e6dfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 14 [ 880.960700] CPU: 14 PID: 131 Comm: migration/14 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802a2cdb000 ti: ffff8802a2ce8000 task.ti: ffff8802a2ce8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802a2cefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0059614ee5 RCX: 1ffff1005459b8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802a2cefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8802cb0a7738 R14: 0000000000000001 R15: ffff8802cb0a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802a3200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002cc8ff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802cb0a7728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802a2cdb000 ffff8802a2cefbd8 dffffc0000000000 [ 880.960700] ffff8802a3218440 ffff8802cb0a7778 ffff8802cb0a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 15 [ 880.960700] CPU: 15 PID: 140 Comm: migration/15 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802ccca3000 ti: ffff8802cccd0000 task.ti: ffff8802cccd0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802cccd7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0023834ee5 RCX: 1ffff100599948d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802cccd7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88011c1a7738 R14: 0000000000000001 R15: ffff88011c1a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802cd200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001ad0a08 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88011c1a7728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802ccca3000 ffff8802cccd7bd8 dffffc0000000000 [ 880.960700] ffff8802cd218440 ffff88011c1a7778 ffff88011c1a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 16 [ 880.960700] CPU: 16 PID: 149 Comm: migration/16 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802f6ce8000 ti: ffff8802f6cf0000 task.ti: ffff8802f6cf0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802f6cf7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149d5ee5 RCX: 1ffff1005ed9d2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802f6cf7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8800a4eaf738 R14: 0000000000000001 R15: ffff8800a4eaf718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802f7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001f6afe8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4eaf728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802f6ce8000 ffff8802f6cf7bd8 dffffc0000000000 [ 880.960700] ffff8802f7218440 ffff8800a4eaf778 ffff8800a4eaf718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 17 [ 880.960700] CPU: 17 PID: 158 Comm: migration/17 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880320c7b000 ti: ffff880320c88000 task.ti: ffff880320c88000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880320c8fb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed000b451ee5 RCX: 1ffff1006418f8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880320c8fbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88005a28f738 R14: 0000000000000001 R15: ffff88005a28f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880321000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000161bff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88005a28f728 0000000000000282 [ 880.960700] ffff880300000000 00ff880320c7b000 ffff880320c8fbd8 dffffc0000000000 [ 880.960700] ffff880321018440 ffff88005a28f778 ffff88005a28f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 18 [ 880.960700] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8806ad0b0000 ti: ffff88000eac0000 task.ti: ffff88000eac0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eac7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eac7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eac7fd8 R14: 0000000000000000 R15: ffff88000eac7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88034b200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001692ac0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eac7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eac7fd8 0000000000000000 ffff88000eac7d78 ffffffff931629ef [ 880.960700] ffff88000eac7e68 ffffffff934018e0 ffff88000eac7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 19 [ 880.960700] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8807ed0c8000 ti: ffff88000eac8000 task.ti: ffff88000eac8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eacfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eacfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eacffd8 R14: 0000000000000000 R15: ffff88000eacffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880375200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007ffe1ae26d00 CR3: 000000000660a000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eacfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eacffd8 0000000000000000 ffff88000eacfd78 ffffffff931629ef [ 880.960700] ffff88000eacfe68 ffffffff934018e0 ffff88000eacfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 20 [ 880.960700] CPU: 20 PID: 185 Comm: migration/20 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88039ece0000 ti: ffff88039ece8000 task.ti: ffff88039ece8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88039ecefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008ddccee5 RCX: 1ffff10073d9c2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88039ecefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88046ee67738 R14: 0000000000000001 R15: ffff88046ee67718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88039f200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029aeff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ee67728 0000000000000282 [ 880.960700] ffff880300000000 00ff88039ece0000 ffff88039ecefbd8 dffffc0000000000 [ 880.960700] ffff88039f218440 ffff88046ee67778 ffff88046ee67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 21 [ 880.960700] CPU: 21 PID: 194 Comm: migration/21 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803c8ca8000 ti: ffff8803c8cb0000 task.ti: ffff8803c8cb0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8803c8cb7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149caee5 RCX: 1ffff100791952d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8803c8cb7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8800a4e57738 R14: 0000000000000001 R15: ffff8800a4e57718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803c9000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002914fb0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4e57728 0000000000000282 [ 880.960700] ffff880300000000 00ff8803c8ca8000 ffff8803c8cb7bd8 dffffc0000000000 [ 880.960700] ffff8803c9018440 ffff8800a4e57778 ffff8800a4e57718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 22 [ 880.960700] CPU: 22 PID: 203 Comm: migration/22 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803f2cdb000 ti: ffff8803f2ce8000 task.ti: ffff8803f2ce8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8803f2cefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008dde3ee5 RCX: 1ffff1007e59b8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8803f2cefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88046ef1f738 R14: 0000000000000001 R15: ffff88046ef1f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803f3200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002e38b80 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ef1f728 0000000000000282 [ 880.960700] ffff880300000000 00ff8803f2cdb000 ffff8803f2cefbd8 dffffc0000000000 [ 880.960700] ffff8803f3218440 ffff88046ef1f778 ffff88046ef1f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 23 [ 880.960700] CPU: 23 PID: 212 Comm: migration/23 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88041cc7b000 ti: ffff88041cce0000 task.ti: ffff88041cce0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88041cce7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed006926eee5 RCX: 1ffff1008398f8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88041cce7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff880349377738 R14: 0000000000000001 R15: ffff880349377718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88041d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f9c055599d0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff880349377728 0000000000000282 [ 880.960700] ffff880400000000 00ff88041cc7b000 ffff88041cce7bd8 dffffc0000000000 [ 880.960700] ffff88041d218440 ffff880349377778 ffff880349377718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 24 [ 880.960700] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801d0c68000 ti: ffff88000eed0000 task.ti: ffff88000eed0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eed7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eed7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eed7fd8 R14: 0000000000000000 R15: ffff88000eed7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880447200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029e4220 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eed7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eed7fd8 0000000000000000 ffff88000eed7d78 ffffffff931629ef [ 880.960700] ffff88000eed7e68 ffffffff934018e0 ffff88000eed7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 25 [ 880.960700] CPU: 25 PID: 12438 Comm: trinity-c68 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802cb318000 ti: ffff8802cb3a8000 task.ti: ffff8802cb3a8000 [ 880.960700] RIP: rwsem_down_write_failed (./arch/x86/include/asm/processor.h:658 ./arch/x86/include/asm/processor.h:663 kernel/locking/rwsem-xadd.c:413 kernel/locking/rwsem-xadd.c:442) [ 880.960700] RSP: 0018:ffff8802cb3afc08 EFLAGS: 00000286 [ 880.960700] RAX: ffffffff00000001 RBX: ffffffffa18e12a8 RCX: ffff8802cb318000 [ 880.960700] RDX: fffffffe00000001 RSI: fffffffeffffffff RDI: 0000000100000000 [ 880.960700] RBP: ffff8802cb3afd98 R08: 0000000000000000 R09: ffffffff00000001 [ 880.960700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803f10c3000 [ 880.960700] R13: ffffffffa18e12a0 R14: ffff8802cb3afd68 R15: ffffffffa18e1250 [ 880.960700] FS: 00007f9c05559700(0000) GS:ffff880471000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 880.960700] CR2: 00007f9c0549b614 CR3: 00000002cb385000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffff9dbf3ced ffff8802cb318d00 0000000000000000 ffff8802cb318000 [ 880.960700] 00000000000004ff ffffffffa3ebf6b0 1ffff10059675f89 1ffff10059675f8d [ 880.960700] 0000000041b58ab3 ffff8802cb3a8010 1ffff10059675002 ffff8802cb3a8000 [ 880.960700] Call Trace: [ 880.960700] ? rwsem_down_write_failed (include/linux/rcupdate.h:912 kernel/locking/rwsem-xadd.c:306 kernel/locking/rwsem-xadd.c:381 kernel/locking/rwsem-xadd.c:442) [ 880.960700] ? rwsem_down_read_failed (kernel/locking/rwsem-xadd.c:433) [ 880.960700] ? trace_hardirqs_off (kernel/locking/lockdep.c:2647) [ 880.960700] ? __acct_update_integrals (kernel/tsacct.c:125) [ 880.960700] ? taskstats_exit (kernel/tsacct.c:125) [ 880.960700] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:104) [ 880.960700] ? down_write (kernel/locking/rwsem.h:4 kernel/locking/rwsem.c:52) [ 880.960700] ? ipcget (ipc/util.c:349 ipc/util.c:646) [ 880.960700] ipcget (ipc/util.c:349 ipc/util.c:646) [ 880.960700] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609) [ 880.960700] ? syscall_trace_enter_phase1 (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1486) [ 880.960700] SyS_semget (ipc/sem.c:591 ipc/sem.c:572) [ 880.960700] ? sem_exit_ns (ipc/sem.c:572) [ 880.960700] ? trace_hardirqs_on_thunk (arch/x86/lib/thunk_64.S:42) [ 880.960700] tracesys_phase2 (arch/x86/kernel/entry_64.S:347) [ 880.960700] Code: 8d 0c 0a 48 89 d0 f0 49 0f b1 0f 48 39 c2 0f 84 2b 04 00 00 48 89 c2 48 8d 04 3a 48 85 f0 74 df 4d 85 e4 0f 84 50 04 00 00 f3 90 <e9> 57 ff ff ff e8 02 62 87 f5 84 c0 0f 85 f5 fd ff ff 48 c7 c2 All code ======== 0: 8d 0c 0a lea (%rdx,%rcx,1),%ecx 3: 48 89 d0 mov %rdx,%rax 6: f0 49 0f b1 0f lock cmpxchg %rcx,(%r15) b: 48 39 c2 cmp %rax,%rdx e: 0f 84 2b 04 00 00 je 0x43f 14: 48 89 c2 mov %rax,%rdx 17: 48 8d 04 3a lea (%rdx,%rdi,1),%rax 1b: 48 85 f0 test %rsi,%rax 1e: 74 df je 0xffffffffffffffff 20: 4d 85 e4 test %r12,%r12 23: 0f 84 50 04 00 00 je 0x479 29: f3 90 pause 2b:* e9 57 ff ff ff jmpq 0xffffffffffffff87 <-- trapping instruction 30: e8 02 62 87 f5 callq 0xfffffffff5876237 35: 84 c0 test %al,%al 37: 0f 85 f5 fd ff ff jne 0xfffffffffffffe32 3d: 48 rex.W 3e: c7 .byte 0xc7 3f: c2 .byte 0xc2 ...
Code starting with the faulting instruction
===========================================
0: e9 57 ff ff ff jmpq 0xffffffffffffff5c 5: e8 02 62 87 f5 callq 0xfffffffff587620c a: 84 c0 test %al,%al c: 0f 85 f5 fd ff ff jne 0xfffffffffffffe07 12: 48 rex.W 13: c7 .byte 0xc7 14: c2 .byte 0xc2 ... [ 880.960700] NMI backtrace for cpu 26 [ 880.960700] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803c8c78000 ti: ffff88000eee0000 task.ti: ffff88000eee0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eee7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eee7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eee7fd8 R14: 0000000000000000 R15: ffff88000eee7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88049b200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f0f2bc1d5a8 CR3: 000000000678f000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eee7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eee7fd8 0000000000000000 ffff88000eee7d78 ffffffff931629ef [ 880.960700] ffff88000eee7e68 ffffffff934018e0 ffff88000eee7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 27 [ 880.960700] CPU: 27 PID: 248 Comm: migration/27 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8804c4cf8000 ti: ffff8804c3b00000 task.ti: ffff8804c3b00000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8804c3b07b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0072423ee5 RCX: 1ffff1009899f2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8804c3b07bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88039211f738 R14: 0000000000000001 R15: ffff88039211f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804c5200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029c4ff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039211f728 0000000000000282 [ 880.960700] ffff880400000000 00ff8804c4cf8000 ffff8804c3b07bd8 dffffc0000000000 [ 880.960700] ffff8804c5218440 ffff88039211f778 ffff88039211f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 28 [ 880.960700] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88065d0c0000 ti: ffff88000eef0000 task.ti: ffff88000eef0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eef7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eef7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eef7fd8 R14: 0000000000000000 R15: ffff88000eef7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804ef200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000b34fb8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eef7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eef7fd8 0000000000000000 ffff88000eef7d78 ffffffff931629ef [ 880.960700] ffff88000eef7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 29 [ 880.960700] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88079d0b0000 ti: ffff88000eef8000 task.ti: ffff88000eef8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eeffd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eeffd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eefffd8 R14: 0000000000000000 R15: ffff88000eefffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880519000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f0f2b8e5210 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eeffd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eefffd8 0000000000000000 ffff88000eeffd78 ffffffff931629ef [ 880.960700] ffff88000eeffe68 ffffffff934018e0 ffff88000eeffde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 30 [ 880.960700] CPU: 30 PID: 269 Comm: migration/30 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880542cf8000 ti: ffff880541b00000 task.ti: ffff880541b00000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880541b07b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed005ea54ee5 RCX: 1ffff100a859f2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880541b07bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8802f52a7738 R14: 0000000000000001 R15: ffff8802f52a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880543200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f42552112c0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802f52a7728 0000000000000282 [ 880.960700] ffff880500000000 00ff880542cf8000 ffff880541b07bd8 dffffc0000000000 [ 880.960700] ffff880543218440 ffff8802f52a7778 ffff8802f52a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...
Code starting with the faulting instruction
===========================================
0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 31 [ 880.960700] CPU: 31 PID: 0 Comm: swapper/31 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88000f2d0000 ti: ffff88000f2d8000 task.ti: ffff88000f2d8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0000:ffff88000f2dfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000f2dfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000f2dffd8 R14: 0000000000000000 R15: ffff88000f2dffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88056d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007fff144e2980 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000f2dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000f2dffd8 0000000000000000 ffff88000f2dfd78 ffffffff931629ef [ 880.960700] ffff88000f2dfe68 ffffffff934018e0 ffff88000f2dfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)
Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00
a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax)
Thanks,
Sasha
A message that is not in the archive (sent off-list or not yet received): CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com
Some more info:
multi_cpu_stop seems to be spinning inside do { ... } while (curstate != MULTI_STOP_EXIT);
So, multi_cpu_stop is an offload ([migration]) for: migrate_swap -> stop_two_cpus -> wait_for_completion() sequence... for cross-migrating 2 tasks.
Based on task structs from callers stacks:
PID 14990 CPU 05 -> PID 14996 CPU 00
PID 14991 CPU 30 -> PID 14998 CPU 01 (30 -> 1, different curr, same ) PID 14992 CPU 30 -> PID 14998 CPU 01 (30 -> 1, different curr) PID 14996 CPU 00 -> PID 14992 CPU 30
PID 14998 CPU 01 -> PID 14990 CPU 05
RUNNING migration threads (cpu_stopper_thread -> multi_cpu_stop):
PID 118 = LAST CPU 09, CPU 09
PID 102 = LAST CPU 06, CPU 06
PID 143 = LAST CPU 14, CPU 14
PID 148 = LAST CPU 15, CPU 15
PID 153 = LAST CPU 16, CPU 16
### backtraces and task structs from stack
PID: 14990 TASK: ffff883e59e717f0 CPU: 5 COMMAND: "beam.smp"
#5 [ffff883e607edbc8] migrate_swap at ffffffff810987fa
ffff883e607edbd0: ffff883e59e717f0 ffff883e59e82fe0
ffff883e607edbe0: 000000100000000f ffff883e607edcc0
ffff883e607edbf0: ffffffff810a0827ffff883e59e717f0 = task_struct -> pid 14990 (last cpu = 5) ffff883e59e82fe0 = task_struct -> pid 14996 (last cpu = 0)
PID: 14991 TASK: ffff883e59e72fe0 CPU: 30 COMMAND: "beam.smp"
#5 [ffff883e59e0bbc8] migrate_swap at ffffffff810987fa
ffff883e59e0bbd0: ffff883e59e72fe0 ffff883e59e85fc0
ffff883e59e0bbe0: 000000060000000e ffff883e59e0bcc0
ffff883e59e0bbf0: ffffffff810a0827ffff883e59e72fe0 = task_struct -> pid 14991 (last cpu = 30) ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1)
PID: 14992 TASK: ffff883e59e747d0 CPU: 30 COMMAND: "beam.smp"
#5 [ffff883e59cadbc8] migrate_swap at ffffffff810987fa
ffff883e59cadbd0: ffff883e59e747d0 ffff883e59e85fc0
ffff883e59cadbe0: 0000000600000009 ffff883e59cadcc0
ffff883e59cadbf0: ffffffff810a0827ffff883e59e747d0 = task_struct -> pid 14992 (last cpu = 30) ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1)
PID: 14996 TASK: ffff883e59e82fe0 CPU: 0 COMMAND: "beam.smp"
#5 [ffff883f55d01bc8] migrate_swap at ffffffff810987fa
ffff883f55d01bd0: ffff883e59e82fe0 ffff883e59e747d0
ffff883f55d01be0: 0000000900000010 ffff883f55d01cc0
ffff883f55d01bf0: ffffffff810a0827ffff883e59e82fe0 = task_struct -> pid 14996 (last cpu = 0) ffff883e59e747d0 = task_struct -> pid 14992 (last cpu = 30)
PID: 14998 TASK: ffff883e59e85fc0 CPU: 1 COMMAND: "beam.smp"
#5 [ffff883e59e05bc8] migrate_swap at ffffffff810987fa
ffff883e59e05bd0: ffff883e59e85fc0 ffff883e59e717f0
ffff883e59e05be0: 0000000f00000006 ffff883e59e05cc0
ffff883e59e05bf0: ffffffff810a0827ffff883e59e85fc0 = task_struct -> pid 14998 (last cpu = 1) ffff883e59e717f0 = task_struct -> pid 14990 (last cpu = 5)
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
Thanks,
Sasha
Quoted 2,456 lines · Sasha Levin
Hi all,
I'm seeing the following lockup pretty often while fuzzing with trinity:
[ 880.960250] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 447s! [migration/1:14] [ 880.960700] Modules linked in: [ 880.960700] irq event stamp: 380954 [ 880.960700] hardirqs last enabled at (380953): restore_args (arch/x86/kernel/entry_64.S:780) [ 880.960700] hardirqs last disabled at (380954): apic_timer_interrupt (arch/x86/kernel/entry_64.S:920) [ 880.960700] softirqs last enabled at (380952): __do_softirq (./arch/x86/include/asm/preempt.h:22 kernel/softirq.c:300) [ 880.960700] softirqs last disabled at (380947): irq_exit (kernel/softirq.c:350 kernel/softirq.c:391) [ 880.960700] CPU: 1 PID: 14 Comm: migration/1 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880052cc8000 ti: ffff880052cd0000 task.ti: ffff880052cd0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880052cd7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: 0000000042506841 RCX: 1ffff1000a5992d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880052cd7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9315e211 [ 880.960700] R13: ffff880052cd7ae8 R14: ffffffff93220213 R15: ffff880052cd7ad8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880053200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002d88fd8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8805d4f67728 0000000000000282 [ 880.960700] ffff880000000000 00ff880052cc8000 ffff880052cd7bd8 dffffc0000000000 [ 880.960700] ffff880053218440 ffff8805d4f67778 ffff8805d4f67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] sending NMI to other CPUs: [ 881.290044] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.336 msecs [ 881.292380] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.083 msecs [ 881.297603] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 7.423 msecs [ 880.960700] NMI backtrace for cpu 0 [ 880.960700] CPU: 0 PID: 11 Comm: migration/0 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88000d6e3000 ti: ffff88000d6f0000 task.ti: ffff88000d6f0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88000d6f7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0007c21ee5 RCX: 1ffff10001adc8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88000d6f7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88003e10f738 R14: 0000000000000001 R15: ffff88003e10f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880029000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002d31fb0 CR3: 000000002082c000 CR4: 00000000000007b0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88003e10f728 0000000000000282 [ 880.960700] ffff880000000000 00ff88000d6e3000 ffff88000d6f7bd8 dffffc0000000000 [ 880.960700] ffff880029018440 ffff88003e10f778 ffff88003e10f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 2 [ 880.960700] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802ccca0000 ti: ffff88000dee8000 task.ti: ffff88000dee8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000deefd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000deefd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000deeffd8 R14: 0000000000000000 R15: ffff88000deeffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f6baacd10f8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000deefd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000deeffd8 0000000000000000 ffff88000deefd78 ffffffff931629ef [ 880.960700] ffff88000deefe68 ffffffff934018e0 ffff88000deefde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 3 [ 880.960700] CPU: 3 PID: 32 Comm: migration/3 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8800a6cc0000 ti: ffff8800a6cc8000 task.ti: ffff8800a6cc8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8800a6ccfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0067e09ee5 RCX: 1ffff10014d982d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8800a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88033f04f738 R14: 0000000000000001 R15: ffff88033f04f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800a7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001490fe8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88033f04f728 0000000000000282 [ 880.960700] ffff880000000000 00ff8800a6cc0000 ffff8800a6ccfbd8 dffffc0000000000 [ 880.960700] ffff8800a7218440 ffff88033f04f778 ffff88033f04f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 4 [ 880.960700] CPU: 4 PID: 41 Comm: migration/4 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8800cacb3000 ti: ffff8800cacc0000 task.ti: ffff8800cacc0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8800cacc7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed006e615ee5 RCX: 1ffff100195968d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8800cacc7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8803730af738 R14: 0000000000000001 R15: ffff8803730af718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800cf200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000189efd8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803730af728 0000000000000282 [ 880.960700] ffff880000000000 00ff8800cacb3000 ffff8800cacc7bd8 dffffc0000000000 [ 880.960700] ffff8800cf218440 ffff8803730af778 ffff8803730af718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 5 [ 880.960700] CPU: 5 PID: 50 Comm: migration/5 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880128c6b000 ti: ffff880128c88000 task.ti: ffff880128c88000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880128c8fb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008c0bbee5 RCX: 1ffff1002518d8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880128c8fbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8804605df738 R14: 0000000000000001 R15: ffff8804605df718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880129000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002bf8ff0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8804605df728 0000000000000282 [ 880.960700] ffff880100000000 00ff880128c6b000 ffff880128c8fbd8 dffffc0000000000 [ 880.960700] ffff880129018440 ffff8804605df778 ffff8804605df718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 6 [ 880.960700] CPU: 6 PID: 59 Comm: migration/6 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880152cb3000 ti: ffff880152cc0000 task.ti: ffff880152cc0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880152cc7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0063d8cee5 RCX: 1ffff1002a5968d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880152cc7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88031ec67738 R14: 0000000000000001 R15: ffff88031ec67718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880153200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002b54fa0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88031ec67728 0000000000000282 [ 880.960700] ffff880100000000 00ff880152cb3000 ffff880152cc7bd8 dffffc0000000000 [ 880.960700] ffff880153218440 ffff88031ec67778 ffff88031ec67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 7 [ 880.960700] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8808dd0b0000 ti: ffff88000e2d0000 task.ti: ffff88000e2d0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0000:ffff88000e2d7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e2d7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e2d7fd8 R14: 0000000000000000 R15: ffff88000e2d7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88017d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000000000 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e2d7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e2d7fd8 0000000000000000 ffff88000e2d7d78 ffffffff931629ef [ 880.960700] ffff88000e2d7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 8 [ 880.960700] CPU: 8 PID: 77 Comm: migration/8 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801a6cc0000 ti: ffff8801a6cc8000 task.ti: ffff8801a6cc8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8801a6ccfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed003abaaee5 RCX: 1ffff10034d982d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8801a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8801d5d57738 R14: 0000000000000001 R15: ffff8801d5d57718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801a7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000e69fc8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8801d5d57728 0000000000000282 [ 880.960700] ffff880100000000 00ff8801a6cc0000 ffff8801a6ccfbd8 dffffc0000000000 [ 880.960700] ffff8801a7218440 ffff8801d5d57778 ffff8801d5d57718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 9 [ 880.960700] CPU: 9 PID: 86 Comm: migration/9 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801d0c88000 ti: ffff8801d0c90000 task.ti: ffff8801d0c90000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8801d0c97b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed007245bee5 RCX: 1ffff1003a1912d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8801d0c97bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8803922df738 R14: 0000000000000001 R15: ffff8803922df718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801d1000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029a8fb0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803922df728 0000000000000282 [ 880.960700] ffff880100000000 00ff8801d0c88000 ffff8801d0c97bd8 dffffc0000000000 [ 880.960700] ffff8801d1018440 ffff8803922df778 ffff8803922df718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 10 [ 880.960700] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880128c68000 ti: ffff88000e6c0000 task.ti: ffff88000e6c0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6c7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6c7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6c7fd8 R14: 0000000000000000 R15: ffff88000e6c7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801fb200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000636668 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6c7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6c7fd8 0000000000000000 ffff88000e6c7d78 ffffffff931629ef [ 880.960700] ffff88000e6c7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 11 [ 880.960700] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801a6ca8000 ti: ffff88000e6c8000 task.ti: ffff88000e6c8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6cfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6cfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6cffd8 R14: 0000000000000000 R15: ffff88000e6cffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880225200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000030db000 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6cfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6cffd8 0000000000000000 ffff88000e6cfd78 ffffffff931629ef [ 880.960700] ffff88000e6cfe68 ffffffff934018e0 ffff88000e6cfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 12 [ 880.960700] CPU: 12 PID: 113 Comm: migration/12 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88024ecd0000 ti: ffff88024ecd8000 task.ti: ffff88024ecd8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88024ecdfb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0073a32ee5 RCX: 1ffff10049d9a2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88024ecdfbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88039d197738 R14: 0000000000000001 R15: ffff88039d197718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88024f200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002648fe0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039d197728 0000000000000282 [ 880.960700] ffff880200000000 00ff88024ecd0000 ffff88024ecdfbd8 dffffc0000000000 [ 880.960700] ffff88024f218440 ffff88039d197778 ffff88039d197718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 13 [ 880.960700] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802f6cc0000 ti: ffff88000e6d8000 task.ti: ffff88000e6d8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000e6dfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000e6dfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000e6dffd8 R14: 0000000000000000 R15: ffff88000e6dffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880279000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000063a69c CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000e6dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000e6dffd8 0000000000000000 ffff88000e6dfd78 ffffffff931629ef [ 880.960700] ffff88000e6dfe68 ffffffff934018e0 ffff88000e6dfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 14 [ 880.960700] CPU: 14 PID: 131 Comm: migration/14 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802a2cdb000 ti: ffff8802a2ce8000 task.ti: ffff8802a2ce8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802a2cefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0059614ee5 RCX: 1ffff1005459b8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802a2cefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8802cb0a7738 R14: 0000000000000001 R15: ffff8802cb0a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802a3200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002cc8ff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802cb0a7728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802a2cdb000 ffff8802a2cefbd8 dffffc0000000000 [ 880.960700] ffff8802a3218440 ffff8802cb0a7778 ffff8802cb0a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 15 [ 880.960700] CPU: 15 PID: 140 Comm: migration/15 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802ccca3000 ti: ffff8802cccd0000 task.ti: ffff8802cccd0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802cccd7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0023834ee5 RCX: 1ffff100599948d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802cccd7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88011c1a7738 R14: 0000000000000001 R15: ffff88011c1a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802cd200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001ad0a08 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88011c1a7728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802ccca3000 ffff8802cccd7bd8 dffffc0000000000 [ 880.960700] ffff8802cd218440 ffff88011c1a7778 ffff88011c1a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 16 [ 880.960700] CPU: 16 PID: 149 Comm: migration/16 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802f6ce8000 ti: ffff8802f6cf0000 task.ti: ffff8802f6cf0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8802f6cf7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149d5ee5 RCX: 1ffff1005ed9d2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8802f6cf7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8800a4eaf738 R14: 0000000000000001 R15: ffff8800a4eaf718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802f7200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001f6afe8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4eaf728 0000000000000282 [ 880.960700] ffff880200000000 00ff8802f6ce8000 ffff8802f6cf7bd8 dffffc0000000000 [ 880.960700] ffff8802f7218440 ffff8800a4eaf778 ffff8800a4eaf718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 17 [ 880.960700] CPU: 17 PID: 158 Comm: migration/17 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880320c7b000 ti: ffff880320c88000 task.ti: ffff880320c88000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880320c8fb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed000b451ee5 RCX: 1ffff1006418f8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880320c8fbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88005a28f738 R14: 0000000000000001 R15: ffff88005a28f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880321000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 000000000161bff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88005a28f728 0000000000000282 [ 880.960700] ffff880300000000 00ff880320c7b000 ffff880320c8fbd8 dffffc0000000000 [ 880.960700] ffff880321018440 ffff88005a28f778 ffff88005a28f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 18 [ 880.960700] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8806ad0b0000 ti: ffff88000eac0000 task.ti: ffff88000eac0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eac7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eac7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eac7fd8 R14: 0000000000000000 R15: ffff88000eac7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88034b200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000001692ac0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eac7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eac7fd8 0000000000000000 ffff88000eac7d78 ffffffff931629ef [ 880.960700] ffff88000eac7e68 ffffffff934018e0 ffff88000eac7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 19 [ 880.960700] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8807ed0c8000 ti: ffff88000eac8000 task.ti: ffff88000eac8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eacfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eacfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eacffd8 R14: 0000000000000000 R15: ffff88000eacffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880375200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007ffe1ae26d00 CR3: 000000000660a000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eacfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eacffd8 0000000000000000 ffff88000eacfd78 ffffffff931629ef [ 880.960700] ffff88000eacfe68 ffffffff934018e0 ffff88000eacfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 20 [ 880.960700] CPU: 20 PID: 185 Comm: migration/20 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88039ece0000 ti: ffff88039ece8000 task.ti: ffff88039ece8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88039ecefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008ddccee5 RCX: 1ffff10073d9c2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88039ecefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88046ee67738 R14: 0000000000000001 R15: ffff88046ee67718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88039f200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029aeff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ee67728 0000000000000282 [ 880.960700] ffff880300000000 00ff88039ece0000 ffff88039ecefbd8 dffffc0000000000 [ 880.960700] ffff88039f218440 ffff88046ee67778 ffff88046ee67718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 21 [ 880.960700] CPU: 21 PID: 194 Comm: migration/21 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803c8ca8000 ti: ffff8803c8cb0000 task.ti: ffff8803c8cb0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8803c8cb7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149caee5 RCX: 1ffff100791952d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8803c8cb7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8800a4e57738 R14: 0000000000000001 R15: ffff8800a4e57718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803c9000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002914fb0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4e57728 0000000000000282 [ 880.960700] ffff880300000000 00ff8803c8ca8000 ffff8803c8cb7bd8 dffffc0000000000 [ 880.960700] ffff8803c9018440 ffff8800a4e57778 ffff8800a4e57718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 22 [ 880.960700] CPU: 22 PID: 203 Comm: migration/22 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803f2cdb000 ti: ffff8803f2ce8000 task.ti: ffff8803f2ce8000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8803f2cefb78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed008dde3ee5 RCX: 1ffff1007e59b8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8803f2cefbd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88046ef1f738 R14: 0000000000000001 R15: ffff88046ef1f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803f3200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000002e38b80 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ef1f728 0000000000000282 [ 880.960700] ffff880300000000 00ff8803f2cdb000 ffff8803f2cefbd8 dffffc0000000000 [ 880.960700] ffff8803f3218440 ffff88046ef1f778 ffff88046ef1f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 23 [ 880.960700] CPU: 23 PID: 212 Comm: migration/23 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88041cc7b000 ti: ffff88041cce0000 task.ti: ffff88041cce0000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff88041cce7b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed006926eee5 RCX: 1ffff1008398f8d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff88041cce7bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff880349377738 R14: 0000000000000001 R15: ffff880349377718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88041d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f9c055599d0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff880349377728 0000000000000282 [ 880.960700] ffff880400000000 00ff88041cc7b000 ffff88041cce7bd8 dffffc0000000000 [ 880.960700] ffff88041d218440 ffff880349377778 ffff880349377718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 24 [ 880.960700] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8801d0c68000 ti: ffff88000eed0000 task.ti: ffff88000eed0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eed7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eed7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eed7fd8 R14: 0000000000000000 R15: ffff88000eed7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880447200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029e4220 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eed7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eed7fd8 0000000000000000 ffff88000eed7d78 ffffffff931629ef [ 880.960700] ffff88000eed7e68 ffffffff934018e0 ffff88000eed7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 25 [ 880.960700] CPU: 25 PID: 12438 Comm: trinity-c68 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8802cb318000 ti: ffff8802cb3a8000 task.ti: ffff8802cb3a8000 [ 880.960700] RIP: rwsem_down_write_failed (./arch/x86/include/asm/processor.h:658 ./arch/x86/include/asm/processor.h:663 kernel/locking/rwsem-xadd.c:413 kernel/locking/rwsem-xadd.c:442) [ 880.960700] RSP: 0018:ffff8802cb3afc08 EFLAGS: 00000286 [ 880.960700] RAX: ffffffff00000001 RBX: ffffffffa18e12a8 RCX: ffff8802cb318000 [ 880.960700] RDX: fffffffe00000001 RSI: fffffffeffffffff RDI: 0000000100000000 [ 880.960700] RBP: ffff8802cb3afd98 R08: 0000000000000000 R09: ffffffff00000001 [ 880.960700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803f10c3000 [ 880.960700] R13: ffffffffa18e12a0 R14: ffff8802cb3afd68 R15: ffffffffa18e1250 [ 880.960700] FS: 00007f9c05559700(0000) GS:ffff880471000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 880.960700] CR2: 00007f9c0549b614 CR3: 00000002cb385000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffff9dbf3ced ffff8802cb318d00 0000000000000000 ffff8802cb318000 [ 880.960700] 00000000000004ff ffffffffa3ebf6b0 1ffff10059675f89 1ffff10059675f8d [ 880.960700] 0000000041b58ab3 ffff8802cb3a8010 1ffff10059675002 ffff8802cb3a8000 [ 880.960700] Call Trace: [ 880.960700] ? rwsem_down_write_failed (include/linux/rcupdate.h:912 kernel/locking/rwsem-xadd.c:306 kernel/locking/rwsem-xadd.c:381 kernel/locking/rwsem-xadd.c:442) [ 880.960700] ? rwsem_down_read_failed (kernel/locking/rwsem-xadd.c:433) [ 880.960700] ? trace_hardirqs_off (kernel/locking/lockdep.c:2647) [ 880.960700] ? __acct_update_integrals (kernel/tsacct.c:125) [ 880.960700] ? taskstats_exit (kernel/tsacct.c:125) [ 880.960700] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:104) [ 880.960700] ? down_write (kernel/locking/rwsem.h:4 kernel/locking/rwsem.c:52) [ 880.960700] ? ipcget (ipc/util.c:349 ipc/util.c:646) [ 880.960700] ipcget (ipc/util.c:349 ipc/util.c:646) [ 880.960700] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609) [ 880.960700] ? syscall_trace_enter_phase1 (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1486) [ 880.960700] SyS_semget (ipc/sem.c:591 ipc/sem.c:572) [ 880.960700] ? sem_exit_ns (ipc/sem.c:572) [ 880.960700] ? trace_hardirqs_on_thunk (arch/x86/lib/thunk_64.S:42) [ 880.960700] tracesys_phase2 (arch/x86/kernel/entry_64.S:347) [ 880.960700] Code: 8d 0c 0a 48 89 d0 f0 49 0f b1 0f 48 39 c2 0f 84 2b 04 00 00 48 89 c2 48 8d 04 3a 48 85 f0 74 df 4d 85 e4 0f 84 50 04 00 00 f3 90 <e9> 57 ff ff ff e8 02 62 87 f5 84 c0 0f 85 f5 fd ff ff 48 c7 c2 All code ======== 0: 8d 0c 0a lea (%rdx,%rcx,1),%ecx 3: 48 89 d0 mov %rdx,%rax 6: f0 49 0f b1 0f lock cmpxchg %rcx,(%r15) b: 48 39 c2 cmp %rax,%rdx e: 0f 84 2b 04 00 00 je 0x43f 14: 48 89 c2 mov %rax,%rdx 17: 48 8d 04 3a lea (%rdx,%rdi,1),%rax 1b: 48 85 f0 test %rsi,%rax 1e: 74 df je 0xffffffffffffffff 20: 4d 85 e4 test %r12,%r12 23: 0f 84 50 04 00 00 je 0x479 29: f3 90 pause 2b:* e9 57 ff ff ff jmpq 0xffffffffffffff87 <-- trapping instruction 30: e8 02 62 87 f5 callq 0xfffffffff5876237 35: 84 c0 test %al,%al 37: 0f 85 f5 fd ff ff jne 0xfffffffffffffe32 3d: 48 rex.W 3e: c7 .byte 0xc7 3f: c2 .byte 0xc2 ...Code starting with the faulting instruction
===========================================0: e9 57 ff ff ff jmpq 0xffffffffffffff5c 5: e8 02 62 87 f5 callq 0xfffffffff587620c a: 84 c0 test %al,%al c: 0f 85 f5 fd ff ff jne 0xfffffffffffffe07 12: 48 rex.W 13: c7 .byte 0xc7 14: c2 .byte 0xc2 ... [ 880.960700] NMI backtrace for cpu 26 [ 880.960700] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8803c8c78000 ti: ffff88000eee0000 task.ti: ffff88000eee0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eee7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eee7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eee7fd8 R14: 0000000000000000 R15: ffff88000eee7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88049b200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f0f2bc1d5a8 CR3: 000000000678f000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eee7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eee7fd8 0000000000000000 ffff88000eee7d78 ffffffff931629ef [ 880.960700] ffff88000eee7e68 ffffffff934018e0 ffff88000eee7de8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 27 [ 880.960700] CPU: 27 PID: 248 Comm: migration/27 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff8804c4cf8000 ti: ffff8804c3b00000 task.ti: ffff8804c3b00000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff8804c3b07b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed0072423ee5 RCX: 1ffff1009899f2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff8804c3b07bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff88039211f738 R14: 0000000000000001 R15: ffff88039211f718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804c5200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00000000029c4ff8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039211f728 0000000000000282 [ 880.960700] ffff880400000000 00ff8804c4cf8000 ffff8804c3b07bd8 dffffc0000000000 [ 880.960700] ffff8804c5218440 ffff88039211f778 ffff88039211f718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 28 [ 880.960700] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88065d0c0000 ti: ffff88000eef0000 task.ti: ffff88000eef0000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eef7d38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eef7d38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eef7fd8 R14: 0000000000000000 R15: ffff88000eef7fd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804ef200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 0000000000b34fb8 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eef7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eef7fd8 0000000000000000 ffff88000eef7d78 ffffffff931629ef [ 880.960700] ffff88000eef7e68 ffffffff934018e0 0000000000000000 0000000000000000 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 29 [ 880.960700] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88079d0b0000 ti: ffff88000eef8000 task.ti: ffff88000eef8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0018:ffff88000eeffd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000eeffd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000eefffd8 R14: 0000000000000000 R15: ffff88000eefffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880519000000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f0f2b8e5210 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000eeffd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000eefffd8 0000000000000000 ffff88000eeffd78 ffffffff931629ef [ 880.960700] ffff88000eeffe68 ffffffff934018e0 ffff88000eeffde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax) [ 880.960700] NMI backtrace for cpu 30 [ 880.960700] CPU: 30 PID: 269 Comm: migration/30 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff880542cf8000 ti: ffff880541b00000 task.ti: ffff880541b00000 [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192) [ 880.960700] RSP: 0000:ffff880541b07b78 EFLAGS: 00000246 [ 880.960700] RAX: 0000000000000000 RBX: ffffed005ea54ee5 RCX: 1ffff100a859f2d5 [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4 [ 880.960700] RBP: ffff880541b07bd8 R08: 0000000000000001 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 [ 880.960700] R13: ffff8802f52a7738 R14: 0000000000000001 R15: ffff8802f52a7718 [ 880.960700] FS: 0000000000000000(0000) GS:ffff880543200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007f42552112c0 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802f52a7728 0000000000000282 [ 880.960700] ffff880500000000 00ff880542cf8000 ffff880541b07bd8 dffffc0000000000 [ 880.960700] ffff880543218440 ffff8802f52a7778 ffff8802f52a7718 ffffffff935335b0 [ 880.960700] Call Trace: [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170) [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437) [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449) [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1)) [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] ? sort_range (kernel/smpboot.c:105) [ 880.960700] kthread (kernel/kthread.c:207) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565) [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176) [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5 All code ======== 0: eb 27 jmp 0x29 2: 0f 1f 00 nopl (%rax) 5: 41 83 fe 03 cmp $0x3,%r14d 9: 75 0a jne 0x15 b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp) f: 0f 85 50 01 00 00 jne 0x165 15: f0 41 ff 4f 24 lock decl 0x24(%r15) 1a: 74 71 je 0x8d 1c: 41 83 fe 04 cmp $0x4,%r14d 20: 0f 84 d7 00 00 00 je 0xfd 26: 45 89 f0 mov %r14d,%r8d 29: f3 90 pause 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction 2e: 48 c1 e8 03 shr $0x3,%rax 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax 37: 84 c0 test %al,%al 39: 74 08 je 0x43 3b: 3c 03 cmp $0x3,%al 3d: 0f .byte 0xf 3e: 8e f5 mov %ebp,%? ...Code starting with the faulting instruction
===========================================0: 4c 89 e8 mov %r13,%rax 3: 48 c1 e8 03 shr $0x3,%rax 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax c: 84 c0 test %al,%al e: 74 08 je 0x18 10: 3c 03 cmp $0x3,%al 12: 0f .byte 0xf 13: 8e f5 mov %ebp,%? ... [ 880.960700] NMI backtrace for cpu 31 [ 880.960700] CPU: 31 PID: 0 Comm: swapper/31 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971 [ 880.960700] task: ffff88000f2d0000 ti: ffff88000f2d8000 task.ti: ffff88000f2d8000 [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50) [ 880.960700] RSP: 0000:ffff88000f2dfd38 EFLAGS: 00000282 [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000 [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420 [ 880.960700] RBP: ffff88000f2dfd38 R08: 0000000000000000 R09: 0000000000000000 [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4 [ 880.960700] R13: ffff88000f2dffd8 R14: 0000000000000000 R15: ffff88000f2dffd8 [ 880.960700] FS: 0000000000000000(0000) GS:ffff88056d200000(0000) knlGS:0000000000000000 [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 880.960700] CR2: 00007fff144e2980 CR3: 000000002082c000 CR4: 00000000000007a0 [ 880.960700] Stack: [ 880.960700] ffff88000f2dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4 [ 880.960700] ffff88000f2dffd8 0000000000000000 ffff88000f2dfd78 ffffffff931629ef [ 880.960700] ffff88000f2dfe68 ffffffff934018e0 ffff88000f2dfde8 ffffffff9dbf7ec5 [ 880.960700] Call Trace: [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314) [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306) [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191) [ 880.960700] ? arch_cpu_idle_prepare (??:?) [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191) [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481) [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219) [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219) [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 00 55 48 add %dl,0x48(%rbp) 7: 89 e5 mov %esp,%ebp 9: fa cli a: 5d pop %rbp b: c3 retq c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 13: 00 00 15: 55 push %rbp 16: 48 89 e5 mov %rsp,%rbp 19: fb sti 1a: 5d pop %rbp 1b: c3 retq 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 23: 00 00 25: 55 push %rbp 26: 48 89 e5 mov %rsp,%rbp 29: fb sti 2a: f4 hlt 2b:* 5d pop %rbp <-- trapping instruction 2c: c3 retq 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 34: 00 35: 55 push %rbp 36: 48 89 e5 mov %rsp,%rbp 39: f4 hlt 3a: 5d pop %rbp 3b: c3 retq 3c: 66 data16 3d: 0f .byte 0xf 3e: 1f (bad) 3f: 84 00 test %al,(%rax)Code starting with the faulting instruction
===========================================
0: 5d pop %rbp
1: c3 retq
2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
9: 00a: 55 push %rbp b: 48 89 e5 mov %rsp,%rbp e: f4 hlt f: 5d pop %rbp 10: c3 retq 11: 66 data16 12: 0f .byte 0xf 13: 1f (bad) 14: 84 00 test %al,(%rax)Thanks,
Sasha
Ingo Molnar
Acked-by Jason LowQuoted 1 line · Sasha Levin
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
That would be:
1a99367023f6 ("locking/rwsem: Check for active lock before bailing on spinning")
attached below.
Thanks,
Ingo
===========================>
Quoted 1 line
From
1a99367023f6ac664365a37fa508b059e31d0e88Mon Sep 17 00:00:00 2001
From: Davidlohr Bueso <[email protected]>
Date: Fri, 30 Jan 2015 01:14:27 -0800
Subject: [PATCH] locking/rwsem: Check for active lock before bailing on spinning
37e9562453b ("locking/rwsem: Allow conservative optimistic spinning when readers have lock") forced the default for optimistic spinning to be disabled if the lock owner was nil, which makes much sense for readers. However, while it is not our priority, we can make some optimizations for write-mostly workloads. We can bail the spinning step and still be conservative if there are any active tasks, otherwise there's really no reason not to spin, as the semaphore is most likely unlocked.
This patch recovers most of a Unixbench 'execl' benchmark throughput by sleeping less and making better average system usage:
before:
CPU %user %nice %system %iowait %steal %idle
all 0.60 0.00 8.02 0.00 0.00 91.38
after:
CPU %user %nice %system %iowait %steal %idle
all 1.22 0.00 70.18 0.00 0.00 28.60
--- kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
kernel/locking/rwsem-xadd.c +17 -10
@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem) {struct task_struct *owner; -bool on_cpu = false; +bool ret = true;if (need_resched ())return false;rcu_read_lock (); owner =ACCESS_ONCE (sem->owner); -if (owner) - on_cpu = owner->on_cpu; -rcu_read_unlock (); +if (!owner) { +long count =ACCESS_ONCE (sem->count); +/* +* If sem->owner is not set, yet we have just recently entered the +* slowpath with the lock being active, then there is a possibility +* reader(s) may have the lock. To be safe, bail spinning in these +* situations. +*/ +if (count &RWSEM_ACTIVE_MASK ) + ret = false; + goto done; + } -/* -* If sem->owner is not set, yet we have just recently entered the -* slowpath, then there is a possibility reader(s) may have the lock. -* To be safe, avoid spinning in these situations. -*/ -return on_cpu; + ret = owner->on_cpu; +done: +rcu_read_unlock (); +return ret; } static inline bool owner_running(struct rw_semaphore *sem,
Are you sure about this ? I have a core dump locked on the same place (state machine for powering cpu down for the task swap) from a 3.13 (+ upstream patches) and this commit wasn't backported yet.
-> multi_cpu_stop -> do { } while (curstate != MULTI_STOP_EXIT);
In my case, curstate is WAY different from enum containing MULTI_STOP_EXIT (4).
Register totally messed up (probably after cpu_relax(), right where you were trapped -> after the pause instruction).
my case:
PID: 118 TASK: ffff883fd28ec7d0 CPU: 9 COMMAND: "migration/9"
...
[exception RIP: multi_cpu_stop+0x64]
RIP: ffffffff810f5944 RSP: ffff883fd2907d98 RFLAGS: 00000246
RAX: 0000000000000010 RBX: 0000000000000010 RCX: 0000000000000246
RDX: ffff883fd2907d98 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffffff810f5944 R8: ffffffff810f5944 R9: 0000000000000000
R10: ffff883fd2907d98 R11: 0000000000000246 R12: ffffffffffffffff
R13: ffff883f55d01b48 R14: 0000000000000000 R15: 0000000000000001
ORIG_RAX: 0000000000000001 CS: 0010 SS: 0000--- <NMI exception stack> ---
#4 [ffff883fd2907d98] multi_cpu_stop+0x64 at ffffffff810f5944
208 } while (curstate != MULTI_STOP_EXIT);
---> RIP
RIP 0xffffffff810f5944 <+100>: cmp $0x4,%edx
---> CHECKING FOR MULTI_STOP_EXITRDX: ffff883fd2907d98 -> does not make any sense
###
If i'm reading this right,
"""
CPU 05 - PID 14990
do_numa_pagetask_numa_faultnuma_migrate_preferredtask_numa_migratemigrate_swap (curr: 14990, task: 14996)stop_two_cpus (cpu1=05(14996), cpu2=00(14990))wait_for_completion
14990 - CPU05
14996 - CPU00
stop_two_cpus:
multi_stop_data (msdata->state = MULTI_STOP_PREPARE)
smp_call_function_single (min=cpu2=00, irq_cpu_stop_queue_work, wait=1)
smp_call_function_single (ran on lowest CPU, 00 for this case)
irq_cpu_stop_queue_work
cpu_stop_queue_work(cpu1=05(14996)) # add work
(multi_cpu_stop) to cpu 05 cpu_stopper queue
cpu_stop_queue_work(cpu2=00(14990)) # add work
(multi_cpu_stop) to cpu 00 cpu_stopper queue
wait_for_completion() --> HERE"""
in my case, checking task structs for tasks scheduled when "waiting_for_completion()":
PID 14990 CPU 05 -> PID 14996 CPU 00
PID 14991 CPU 30 -> PID 14998 CPU 01
PID 14992 CPU 30 -> PID 14998 CPU 01
PID 14996 CPU 00 -> PID 14992 CPU 30
PID 14998 CPU 01 -> PID 14990 CPU 05
AND
Quoted 5 lines
102 2 6 ffff881fd2ea97f0 RU 0.0 0 0 [migration/6] 118 2 9 ffff883fd28ec7d0 RU 0.0 0 0 [migration/9] 143 2 14 ffff883fd29d47d0 RU 0.0 0 0 [migration/14] 148 2 15 ffff883fd29fc7d0 RU 0.0 0 0 [migration/15] 153 2 16 ffff881fd2f517f0 RU 0.0 0 0 [migration/16]
THEN
I am still waiting for 5 cpu_stopper_thread -> multi_cpu_stop just scheduled (probably in the per cpu's queue of cpus 0,1,5,30), not running yet.
AND
I don't have any "wait_for_completion" for those "OLDER" migration threads (6, 9, 14, 15 and 16)
Probably wait_for_completion signaled done.completion before racing.
Looks like something messed up with curstate in the "multi_cpu_stop" state machine.
/* Simple state machine */
do {
/* Chill out and ensure we re-read multi_stop_state. */ cpu_relax();
cpu_relax maybe ?
--
Rafael Tinoco
Quoted 107 lines · Ingo Molnar
Quoted 1 line · Sasha Levin
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
That would be:
1a99367023f6("locking/rwsem: Check for active lock before bailing on spinning")attached below.
Thanks,
Ingo
===========================>
From1a99367023f6ac664365a37fa508b059e31d0e88Mon Sep 17 00:00:00 2001
From: Davidlohr Bueso <[email protected]>
Date: Fri, 30 Jan 2015 01:14:27 -0800
Subject: [PATCH] locking/rwsem: Check for active lock before bailing on spinning
37e9562453b("locking/rwsem: Allow conservative optimistic spinning when readers have lock") forced the default for optimistic spinning to be disabled if the lock owner was nil, which makes much sense for readers. However, while it is not our priority, we can make some optimizations for write-mostly workloads. We can bail the spinning step and still be conservative if there are any active tasks, otherwise there's really no reason not to spin, as the semaphore is most likely unlocked.This patch recovers most of a Unixbench 'execl' benchmark throughput by sleeping less and making better average system usage:
before:
CPU %user %nice %system %iowait %steal %idle
all 0.60 0.00 8.02 0.00 0.00 91.38after:
CPU %user %nice %system %iowait %steal %idle
all 1.22 0.00 70.18 0.00 0.00 28.60Signed-off-by Davidlohr Bueso <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Acked-by Jason Low <[email protected]>Cc Linus Torvalds <[email protected]>Cc Michel Lespinasse <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Tim Chen <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)kernel/locking/rwsem-xadd.c +17
-11@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem) {struct task_struct *owner; -bool on_cpu = false; +bool ret = true;if (need_resched ())return false;rcu_read_lock (); owner =ACCESS_ONCE (sem->owner); -if (owner) - on_cpu = owner->on_cpu; -rcu_read_unlock (); +if (!owner) { +long count =ACCESS_ONCE (sem->count); +/* +* If sem->owner is not set, yet we have just recently entered the +* slowpath with the lock being active, then there is a possibility +* reader(s) may have the lock. To be safe, bail spinning in these +* situations. +*/ +if (count &RWSEM_ACTIVE_MASK ) + ret = false; + goto done; + } -/* -* If sem->owner is not set, yet we have just recently entered the -* slowpath, then there is a possibility reader(s) may have the lock. -* To be safe, avoid spinning in these situations. -*/ -return on_cpu; + ret = owner->on_cpu; +done: +rcu_read_unlock (); +return ret; } static inline bool owner_running(struct rw_semaphore *sem, --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Rafael David Tinoco
Are you sure about this ? I have a core dump locked on the same place (state machine for powering cpu down for the task swap) from a 3.13 (+ upstream patches) and this commit wasn't backported yet.
bisect took me to that same commit twice, and I'm running a -next with that commit reverted and not seeing those hangs anymore.
I also didn't see the hang up until a month or so ago (which matches with that commit date), and definitely not back in 3.13.
Thanks,
Sasha
Quoted 10 lines · Sasha Levin
Quoted 3 lines · Rafael David Tinoco
Are you sure about this ? I have a core dump locked on the same place (state machine for powering cpu down for the task swap) from a 3.13 (+ upstream patches) and this commit wasn't backported yet.
bisect took me to that same commit twice, and I'm running a -next with that commit reverted and not seeing those hangs anymore.
I also didn't see the hang up until a month or so ago (which matches with that commit date), and definitely not back in 3.13.
I just remembered that I was seeing something similar back then, which was fixed:
http://lists.openwall.net/linux-kernel/2014/05/12/725
Thanks,
Sasha
Quoted 9 lines · Ingo Molnar
Quoted 1 line · Sasha Levin
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
That would be:
1a99367023f6("locking/rwsem: Check for active lock before bailing on spinning")attached below.
[...]
Quoted 20 lines
kernel/locking/rwsem-xadd.c +2
-4@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem) {struct task_struct *owner; -bool on_cpu = false; +bool ret = true;if (need_resched ())return false;rcu_read_lock (); owner =ACCESS_ONCE (sem->owner); -if (owner) - on_cpu = owner->on_cpu; -rcu_read_unlock (); +if (!owner) {
kernel/locking/rwsem-xadd.c static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) · line 318
Quoted 10 lines
kernel/locking/rwsem-xadd.c +10
-0+long count =ACCESS_ONCE (sem->count); +/* +* If sem->owner is not set, yet we have just recently entered the +* slowpath with the lock being active, then there is a possibility +* reader(s) may have the lock. To be safe, bail spinning in these +* situations. +*/ +if (count &RWSEM_ACTIVE_MASK ) + ret = false; + goto done;
Hmmm so the lockup would be due to this (when owner is non-nil the patch has no effect), telling users to spin instead of sleep -- _except_ for this condition. And when spinning we're always checking for need_resched to be safe. So even if this function was completely bogus, we'd end up needlessly spinning but I'm surprised about the lockup. Maybe coffee will make things clearer.
Quoted 28 lines · Davidlohr Bueso
Quoted 28 lines
kernel/locking/rwsem-xadd.c +0
-0Quoted 27 lines
index
1c0d11e8ce34..e4ad019e23f5100644kernel/locking/rwsem-xadd.c +10
-4@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem) {struct task_struct *owner; -bool on_cpu = false; +bool ret = true;if (need_resched ())return false;rcu_read_lock (); owner =ACCESS_ONCE (sem->owner); -if (owner) - on_cpu = owner->on_cpu; -rcu_read_unlock (); +if (!owner) { +long count =ACCESS_ONCE (sem->count); +/* +* If sem->owner is not set, yet we have just recently entered the +* slowpath with the lock being active, then there is a possibility +* reader(s) may have the lock. To be safe, bail spinning in these +* situations. +*/ +if (count &RWSEM_ACTIVE_MASK )
Quoted 8 lines
Quoted 2 lines
kernel/locking/rwsem-xadd.c static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) · line 318
Quoted 2 lines
kernel/locking/rwsem-xadd.c +2
-0+ ret = false; + goto done;Hmmm so the lockup would be due to this (when owner is non-nil the patch has no effect), telling users to spin instead of sleep -- _except_ for this condition. And when spinning we're always checking for
need_reschedto be safe. So even if this function was completely bogus, we'd end up needlessly spinning but I'm surprised about the lockup. Maybe coffee will make things clearer.
There's always the possibility that bisect went wrong. I did it twice, but since I don't have a sure way of reproducing it I was basing my good/bad decisions on whether I saw it within a reasonable amount of time.
I can go redo that again if you suspect that that commit is not the cause.
Thanks,
Sasha
Quoted 1 line · Sasha Levin
I can go redo that again if you suspect that that commit is not the cause.
I took a closer look at the logs, and I'm seeing hangs that begin this way as well:
[ 2298.020237] NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [trinity-c19:839] [ 2298.020237] Modules linked in: [ 2298.020237] CPU: 19 PID: 839 Comm: trinity-c19 Not tainted 4.0.0-rc2-next-20150306-sasha-00056-g61886e8 #2005 [ 2298.020237] task: ffff880278d62000 ti: ffff880254fe8000 task.ti: ffff880254fe8000 [ 2298.020237] RIP: 0010:[<ffffffffa442702f>] [<ffffffffa442702f>] __rcu_read_unlock+0x9f/0x130 [ 2298.020237] RSP: 0000:ffff880254fefbd8 EFLAGS: 00000207 [ 2298.020237] RAX: dffffc0000000000 RBX: ffff880254fe8000 RCX: 1ffff1004a9fd002 [ 2298.020237] RDX: 1ffff1004f1ac4e2 RSI: ffff8802c3ff6000 RDI: ffff880278d62714 [ 2298.020237] RBP: ffff880254fefbe8 R08: ffff880362b2e080 R09: ffffffff00000001 [ 2298.020237] R10: ffff880362b2e140 R11: ffffea000e253800 R12: 0000000000000a3e [ 2298.020237] R13: ffff880278d62cb0 R14: ffffed014e1e4899 R15: 0034c1c55efd9eff [ 2298.020237] FS: 00007f183b9c3700(0000) GS:ffff880375200000(0000) knlGS:0000000000000000 [ 2298.020237] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2298.020237] CR2: 0000000000bc8fe8 CR3: 0000000259a0a000 CR4: 00000000000007a0 [ 2298.020237] Stack: [ 2298.020237] ffff8802c3ff6000 ffff880126cfdc28 ffff880254fefc38 ffffffffa43fcb75 [ 2298.020237] ffff8802c3ff6000 ffff8802c3ff6000 ffff880278d62714 ffff880126cfdc48 [ 2298.020237] ffff8802c3ff6000 ffff880126cfdc44 ffff880126cfdc28 ffff880254fefd78 [ 2298.020237] Call Trace: [ 2298.020237] [<ffffffffa43fcb75>] rwsem_spin_on_owner+0x165/0x250 [ 2298.020237] [<ffffffffae92a67f>] rwsem_down_write_failed+0x22f/0x750 [ 2298.020237] [<ffffffffae92a450>] ? rwsem_down_read_failed+0x260/0x260 [ 2298.020237] [<ffffffffa438fc31>] ? get_parent_ip+0x11/0x50 [ 2298.020237] [<ffffffffa438fd76>] ? preempt_count_add+0x106/0x160 [ 2298.020237] [<ffffffffa5f77c77>] ? debug_smp_processor_id+0x17/0x20 [ 2298.020237] [<ffffffffa47128c0>] ? cmpxchg_double_slab.isra.25+0x210/0x240 [ 2298.020237] [<ffffffffa47119df>] ? free_debug_processing+0x19f/0x320 [ 2298.020237] [<ffffffffa5f4da33>] call_rwsem_down_write_failed+0x13/0x20 [ 2298.020237] [<ffffffffae9296a9>] ? down_write+0x29/0x70 [ 2298.020237] [<ffffffffa46acd32>] validate_mm+0xa2/0x910 [ 2298.020237] [<ffffffffa46b5fd1>] do_munmap+0x421/0xf50 [ 2298.020237] [<ffffffffa41627d0>] ? send_sigtrap+0x1e0/0x1e0 [ 2298.020237] [<ffffffffa46b6b5f>] vm_munmap+0x5f/0x80 [ 2298.020237] [<ffffffffa46b9562>] SyS_munmap+0x22/0x30 [ 2298.020237] [<ffffffffae92e60d>] system_call_fastpath+0x16/0x1b [ 2298.020237] Code: 02 84 c0 74 04 3c 03 7e 7c c7 83 10 07 00 00 00 00 00 80 48 8d bb 14 07 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <0f> b6 14 02 48 89 f8 83 e0 07 83 c0 01 38 d0 7c 04 84 d2 75 52
So it seems that we end up spinning for quite a while?
Thanks,
Sasha
Quoted 38 lines · Davidlohr Bueso
Quoted 37 lines · Ingo Molnar
Quoted 1 line · Sasha Levin
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
That would be:
1a99367023f6("locking/rwsem: Check for active lock before bailing on spinning")kernel/locking/rwsem-xadd.c +11
-4@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)static inline bool rwsem_can_spin_on_owner (struct rw_semaphore *sem) {struct task_struct *owner; -bool on_cpu = false; +bool ret = true;if (need_resched ())return false;rcu_read_lock (); owner =ACCESS_ONCE (sem->owner); -if (owner) - on_cpu = owner->on_cpu; -rcu_read_unlock (); +if (!owner) { +long count =ACCESS_ONCE (sem->count); +/* +* If sem->owner is not set, yet we have just recently entered the +* slowpath with the lock being active, then there is a possibility +* reader(s) may have the lock. To be safe, bail spinning in these +* situations. +*/ +if (count &RWSEM_ACTIVE_MASK ) + ret = false;
Quoted 8 lines
kernel/locking/rwsem-xadd.c static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) · line 318
Quoted 1 line
kernel/locking/rwsem-xadd.c +1
-0+ goto done;Hmmm so the lockup would be due to this (when owner is non-nil the patch has no effect), telling users to spin instead of sleep -- _except_ for this condition. And when spinning we're always checking for
need_reschedto be safe. So even if this function was completely bogus, we'd end up needlessly spinning but I'm surprised about the lockup. Maybe coffee will make things clearer.
Right, the can_spin_on_owner() was originally added to the mutex spinning code for optimization purposes, particularly so that we can avoid adding the spinner to the OSQ only to find that it doesn't need to spin. This function needing to return a correct value should really only affect performance, so yes, lockups due to this seems surprising.
Quoted 6 lines · Jason Low
Right, the
can_spin_on_owner()was originally added to the mutex spinning code for optimization purposes, particularly so that we can avoid adding the spinner to the OSQ only to find that it doesn't need to spin. This function needing to return a correct value should really only affect performance, so yes, lockups due to this seems surprising.
Well, softlockups aren't about "correct behavior". They are about certain things not happening in a timely manner.
Clearly the mutex code now tries to hold on to the CPU too aggressively.
At some point people need to admit that busy-looping isn't always a good idea. Especially if
(a) we could idle the core instead
(b) the tuning has been done based on som especial-purpose benchmark
that is likely not realistic
(c) we get reports from people that it causes problems.
In other words: Let's just undo that excessive busy-looping. The performance numbers were dubious to begin with. Real scalability comes from fixing the locking, not from trying to play games with the locks themselves. Particularly games that then cause problems.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Jason Low
Right, the
can_spin_on_owner()was originally added to the mutex spinning code for optimization purposes, particularly so that we can avoid adding the spinner to the OSQ only to find that it doesn't need to spin. This function needing to return a correct value should really only affect performance, so yes, lockups due to this seems surprising.
Quoted 4 lines
Well, softlockups aren't about "correct behavior". They are about certain things not happening in a timely manner.
Clearly the mutex code now tries to hold on to the CPU too aggressively.
This patch was a performance "fix" for rwsems, where it works well mutexes.
Quoted 9 lines
At some point people need to admit that busy-looping isn't always a good idea. Especially if
(a) we could idle the core instead
(b) the tuning has been done based on som especial-purpose benchmark
that is likely not realistic
Quoted 6 lines
(c) we get reports from people that it causes problems.
In other words: Let's just undo that excessive busy-looping. The performance numbers were dubious to begin with. Real scalability comes from fixing the locking, not from trying to play games with the locks themselves. Particularly games that then cause problems.
I obviously agree with all those points, however fyi most of the testing on rwsems I do includes scaling address space ops stressing the mmap_sem, which is a real world concern. So while it does include microbenchmarks, it is not guided by them.
Thanks,
Davidlohr
Quoted 5 lines · Davidlohr Bueso
I obviously agree with all those points, however fyi most of the testing on rwsems I do includes scaling address space ops stressing the
mmap_sem, which is a real world concern. So while it does include microbenchmarks, it is not guided by them.
So I agree that mmap_sem is problematic.
We probably still end up holding it over many actual IO operations, for example. The whole "FAULT_RETRY" thing should have helped a lot, in that hopefully at least a fair amount of the time we now end up waiting for the IO without holding the semaphore, but I bet many other cases remain.
And I also suspect that we could try to be even more aggressive, and allow some entirely unlocked cases. For example, long long ago we used to have a completely SMP-unsafe model where we would do things optimistically - doing IO without holding any locks, and then before we "committed" to it, we'd re-try. And I wonder if we might want to re-introduce that for the cases where we hit in caches and could use RCU.
IOW, I wonder if we could special-case the common non-IO fault-handling path something along the lines of:
- look up the vma in the vma lookup cache
- look up the page in the page cache
- get the page table spinlock
- re-check the vma now (it ends up being stable if it can't be torn
down due to the page table spinlock)
because I suspect that page faults are the biggest users of that mmap_sem, and we could probably handle a fairly large common case (making it simpler by special-casing it and punting in any even _slightly_ complicated situations) without even getting the semaphore at all, since we have to serialize on the actual page table *anyway*.
Basically, to me, the whole "if a lock is so contended that we need to play locking games, then we should look at why we *use* the lock, rather than at the lock itself" is a religion.
Linus
Quoted 3 lines · Linus Torvalds
Basically, to me, the whole "if a lock is so contended that we need to play locking games, then we should look at why we *use* the lock, rather than at the lock itself" is a religion.
Oh absolutely, I'm only mentioning the locking primitive side of mmap_sem, for which minor optimizations will _never_ solve the actual scalability issues. But yes, point taken.
Thanks,
Davidlohr
Quoted 4 lines · Linus Torvalds
IOW, I wonder if we could special-case the common non-IO fault-handling path something along the lines of:
- look up the vma in the vma lookup cache
But you'd still need mmap_sem there to at least get the VMA's first value.
Quoted 21 lines
- look up the page in the page cache
- get the page table spinlock
- re-check the vma now (it ends up being stable if it can't be torn
down due to the page table spinlock)because I suspect that page faults are the biggest users of that
mmap_sem, and we could probably handle a fairly large common case (making it simpler by special-casing it and punting in any even _slightly_ complicated situations) without even getting the semaphore at all, since we have to serialize on the actual page table *anyway*.Basically, to me, the whole "if a lock is so contended that we need to play locking games, then we should look at why we *use* the lock, rather than at the lock itself" is a religion.
Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Davidlohr Bueso
Quoted 4 lines · Linus Torvalds
IOW, I wonder if we could special-case the common non-IO fault-handling path something along the lines of:
- look up the vma in the vma lookup cache
But you'd still need
mmap_semthere to at least get the VMA's first value.
Incomplete msg... what I'm getting at is that one way or another mmap_sem contention is easy to hit -- yes, avoiding it in IO paths is obviously good for hold times. But I also think that it protects way too much, ie making pthread intense workloads really suck as well.
Quoted 5 lines · Davidlohr Bueso
Quoted 2 lines
- look up the vma in the vma lookup cache
But you'd still need
mmap_semthere to at least get the VMA's first value.
So my theory was that the vma cache is such a trivial data structure that we could trivially make it be rcu-protected.
The vma allocations are already SLAB_DESTROY_BY_RCU, because we play games with the anon-vma stuff. Or something. i forget the exact details.
So I think that vmacache_find() would *already* basically work under just the RCU read lock, and we can look at the resulting vma without having to worry about it getting free'd.
Yes, the actual field values may change (ie start/end offsets etc due to vma merging etc), but again, that's not necessarily deadly if we are careful and make use of the vmacache sequence number. We can optimistically do things like page cache lookups (which is already RCU safe), and then before we actually *use* the result, we do another vmacache sequence number validation.
So I *think* we could do at least that limited "we hit in the vma cache, and it's a nice normal simple vma with regular vma ops" with just a RCU read lock, and skip the mmap_sem entirely. Of course, we'd have to fall back on the mmap_sem if anything fails (not in the vma cache, or the sequence number changes before we can actually insert the result in the page tables etc).
The page cache is already RCU-safe, and the actual page table operations are protected by another lock anyway (which should scale better because it's a spinlock and held for shorter times, _and_ is spread out by pte address).
Is it some trivial one-liner? No. But I suspect we could make a trial "lockless page lookup for the simple cases that hit in the caches" without a *lot* of effort.
Linus
Quoted 21 lines · Linus Torvalds
Quoted 6 lines · Jason Low
Right, the
can_spin_on_owner()was originally added to the mutex spinning code for optimization purposes, particularly so that we can avoid adding the spinner to the OSQ only to find that it doesn't need to spin. This function needing to return a correct value should really only affect performance, so yes, lockups due to this seems surprising.Well, softlockups aren't about "correct behavior". They are about certain things not happening in a timely manner.
Clearly the mutex code now tries to hold on to the CPU too aggressively.
At some point people need to admit that busy-looping isn't always a good idea. Especially if
(a) we could idle the core instead
(b) the tuning has been done based on som especial-purpose benchmark
that is likely not realistic
Quoted 6 lines
(c) we get reports from people that it causes problems.
In other words: Let's just undo that excessive busy-looping. The performance numbers were dubious to begin with. Real scalability comes from fixing the locking, not from trying to play games with the locks themselves. Particularly games that then cause problems.
Hi Linus,
Agreed, this is an issue we need to address, though we're just trying to figure out if the change to rwsem_can_spin_on_owner() in "commit:37e9562453b" is really the one that's causing the issue.
For example, it looks like Ming recently found another change in the same patchset: commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving lock spinners) to be causing lockups.
Re: softlockups in multi_cpu_stop
Quoted 11 lines · Jason Low
Hi Linus,
Agreed, this is an issue we need to address, though we're just trying to figure out if the change to
rwsem_can_spin_on_owner()in "commit:37e9562453b" is really the one that's causing the issue.For example, it looks like Ming recently found another change in the same patchset: commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving lock spinners) to be causing lockups.
So I think I may have spotted a problem in the tip commit:
Commit b3fd4f03ca0b995 (locking/rwsem: Avoid deceiving lock spinners).
In owner_running() there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).
So we we really want this right (not yet tested):
---
Subject: [PATCH] locking/rwsem: Avoid spinning when owner is not running
--- kernel/locking/rwsem-xadd.c | 28 ++++++++-------------------- 1 files changed, 8 insertions(+), 20 deletions(-)
kernel/locking/rwsem-xadd.c +8 -20
@@ -324,32 +324,20 @@ done:return ret; } -static inline bool owner_running (struct rw_semaphore *sem, -struct task_struct *owner) -{ -if (sem->owner != owner) -return false; - -/* -* Ensure we emit the owner->on_cpu, dereference _after_ checking -* sem->owner still matches owner, if that fails, owner might -* point to free()d memory, if it still matches, the rcu_read_lock() -* ensures the memory stays valid. -*/ -barrier (); - -return owner->on_cpu; -} -static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) {long count;rcu_read_lock (); -while (owner_running (sem, owner)) { -/* abort spinning when need_resched */ -if (need_resched ()) { +while (true) { +if (sem->owner != owner) +break ; + +barrier (); + +/* abort spinning when need_resched or owner is not running*/ +if (!owner->on_cpu ||need_resched ()) {rcu_read_unlock ();return false; }
Signature · 1 line
1.7.2.5
Re: softlockups in multi_cpu_stop
kernel/locking/rwsem-xadd.c done: · line 335
Quoted 4 lines · Jason Low
kernel/locking/rwsem-xadd.c +3
-0+while (true) { +if (sem->owner != owner) +break ;
That looks *really* odd.
Why is this not
while (sem->owner == owner) {Also, this "barrier()" now lost the comment:
Quoted 1 line
+
barrier();
so it looks very odd indeed.
Linus
Re: softlockups in multi_cpu_stop
Quoted 11 lines · Linus Torvalds
kernel/locking/rwsem-xadd.c done: · line 335
Quoted 4 lines · Jason Low
kernel/locking/rwsem-xadd.c +3
-0+while (true) { +if (sem->owner != owner) +break ;That looks *really* odd.
Why is this not
while (sem->owner == owner) {
Yes, this looks more readable.
That while (true) thing was something we recently did for mutexes which was why I originally had that.
Quoted 5 lines
Also, this "
barrier()" now lost the comment:Quoted 1 line
+
barrier();so it looks very odd indeed.
Right, we should keep the comment for the barrier().
Re: softlockups in multi_cpu_stop
In reply to Jason LowQuoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).
So you're rationale is that we're missing this need_resched:
while (owner_running(sem, owner)) {
/* abort spinning when need_resched */
if (need_resched()) {
rcu_read_unlock();
return false;
}
}Because the owner_running() would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.
Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
Thanks,
Davidlohr
Re: softlockups in multi_cpu_stop
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }
Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
kernel/locking/rwsem-xadd.c +3 -2
@@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) }rcu_read_unlock (); -if (READ_ONCE (sem->owner)) -return true;/* new owner, continue spinning */ + owner =READ_ONCE (sem->owner); +if (owner && owner->on_cpu) +return true;/* * When the owner is not set, the lock could be free or
Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 20 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }
Quoted 8 lines
Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
Quoted 21 lines
kernel/locking/rwsem-xadd.c +3
-2@@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) }rcu_read_unlock (); -if (READ_ONCE (sem->owner)) -return true;/* new owner, continue spinning */ + owner =READ_ONCE (sem->owner); +if (owner && owner->on_cpu) +return true;/* * When the owner is not set, the lock could be free or Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 25 lines · Davidlohr Bueso
Quoted 24 lines · Ming Lei
Quoted 21 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
But the check on owner->on_cpu should be moved outside the loop because new owner can be scheduled out too, right?
Quoted 23 lines
Quoted 21 lines
kernel/locking/rwsem-xadd.c +3
-2@@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) }rcu_read_unlock (); -if (READ_ONCE (sem->owner)) -return true;/* new owner, continue spinning */ + owner =READ_ONCE (sem->owner); +if (owner && owner->on_cpu) +return true;/* * When the owner is not set, the lock could be free or Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 31 lines · Ming Lei
Quoted 30 lines · Davidlohr Bueso
Quoted 28 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
Quoted 6 lines
Quoted 3 lines
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?
That's not what this is about. New lock _owners_ need to worry about burning cycles trying to acquire the lock ;)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
kernel/locking/rwsem-xadd.c +0
-0@@ -358,8 +358,9 @@ bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) }rcu_read_unlock ();
Quoted 5 lines
Quoted 5 lines
kernel/locking/rwsem-xadd.c bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) · line 363
Quoted 5 lines
kernel/locking/rwsem-xadd.c +3
-2-if (READ_ONCE (sem->owner)) -return true;/* new owner, continue spinning */ + owner =READ_ONCE (sem->owner); +if (owner && owner->on_cpu) +return true;
So if I'm understanding this right, your patch works because you add another on_cpu check and at this point we could very well have sem->owner == owner -- such that owner_running return false for the same reason in the first place! So Jason's patch takes on the issue directly by never allowing ups to reach this point.
Thanks,
Davidlohr
Re: softlockups in multi_cpu_stop
Quoted 1 line · Davidlohr Bueso
That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
Re: softlockups in multi_cpu_stop
Quoted 3 lines · Davidlohr Bueso
Quoted 1 line · Davidlohr Bueso
That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
Sorry, could you explain a bit why new owner can't be scheduled out(on_cpu becomes zero)? If that is possible, it still can cause soft lockup like current problem.
Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 8 lines · Ming Lei
Quoted 3 lines · Davidlohr Bueso
Quoted 1 line · Davidlohr Bueso
That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
Sorry, could you explain a bit why new owner can't be scheduled
out(on_cpu becomes zero)? If that is possible, it still can cause soft lockup like current problem.
Oh its not that it can't be scheduled out. The point is we don't care what happens with the lock owner itself (new or not). We care about, and the point of this discussion, how _other_ threads handle themselves when trying to take that lock (a lock having an owner implies the lock is not free, of course). So if a lock owner gets scheduled out... so what? That's already taken into account by spinners.
Re: softlockups in multi_cpu_stop
Quoted 10 lines · Davidlohr Bueso
Quoted 8 lines · Ming Lei
Quoted 3 lines · Davidlohr Bueso
Quoted 1 line · Davidlohr Bueso
That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
Sorry, could you explain a bit why new owner can't be scheduled
out(on_cpu becomes zero)? If that is possible, it still can cause soft lockup like current problem.
Quoted 6 lines
Oh its not that it can't be scheduled out. The point is we don't care what happens with the lock owner itself (new or not). We care about, and the point of this discussion, how _other_ threads handle themselves when trying to take that lock (a lock having an owner implies the lock is not free, of course). So if a lock owner gets scheduled out... so what? That's already taken into account by spinners.
Not exactly, current problem is just in spinner because it ignores scheduled out owner and continues to spin, then cause lockup, isn't it?
Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 18 lines · Ming Lei
Quoted 10 lines · Davidlohr Bueso
Quoted 8 lines · Ming Lei
Quoted 3 lines · Davidlohr Bueso
Quoted 1 line · Davidlohr Bueso
That's not what this is about. New lock _owners_ need to worry about
^^^ make that "need not"
Sorry, could you explain a bit why new owner can't be scheduled
out(on_cpu becomes zero)? If that is possible, it still can cause soft lockup like current problem.Quoted 6 lines
Oh its not that it can't be scheduled out. The point is we don't care what happens with the lock owner itself (new or not). We care about, and the point of this discussion, how _other_ threads handle themselves when trying to take that lock (a lock having an owner implies the lock is not free, of course). So if a lock owner gets scheduled out... so what? That's already taken into account by spinners.
Quoted 3 lines
Not exactly, current problem is just in spinner because it ignores scheduled out owner and continues to spin, then cause lockup, isn't it?
Exactly my point, Ming. It's the _spinner_ that has the problem, hence the fix in the part of the code that must decide just that. By the time we're doing this:
if (READ_ONCE(sem->owner)) return true; /* new owner, continue spinning */
We need to have already taken into account the owner->on_cpu situation. We fix spinners, not lock owners.
I'm really running out of ways to explain this, and you are going in circles, which is getting annoying given that you haven't even tried the other patch.
Re: softlockups in multi_cpu_stop
In reply to Ming LeiQuoted 31 lines · Ming Lei
Quoted 30 lines · Davidlohr Bueso
Quoted 28 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
Quoted 6 lines
Quoted 3 lines
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?
We should keep the owner->on_cpu check inside the loop, otherwise we could continue spinning if the owner is not running.
Re: softlockups in multi_cpu_stop
Quoted 36 lines · Jason Low
Quoted 35 lines · Ming Lei
Quoted 33 lines · Davidlohr Bueso
Quoted 20 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Quoted 8 lines
Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
Quoted 5 lines
Quoted 2 lines
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?We should keep the
owner->on_cpucheck inside the loop, otherwise we could continue spinning if the owner is not running.
So how about checking in this way outside the loop for avoiding the spin?
if (owner)
return owner->on_cpu;Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 40 lines · Ming Lei
Quoted 39 lines · Jason Low
Quoted 37 lines · Ming Lei
Quoted 25 lines · Davidlohr Bueso
Quoted 24 lines · Ming Lei
Quoted 21 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?
Quoted 7 lines
Quoted 2 lines
We should keep the
owner->on_cpucheck inside the loop, otherwise we could continue spinning if the owner is not running.So how about checking in this way outside the loop for avoiding the spin?
if (owner) return owner->on_cpu;
Ming are you reading the thread?? Have you at least tried jason's patch?? *sigh*
Re: softlockups in multi_cpu_stop
In reply to Ming LeiQuoted 40 lines · Ming Lei
Quoted 39 lines · Jason Low
Quoted 37 lines · Ming Lei
Quoted 25 lines · Davidlohr Bueso
Quoted 24 lines · Ming Lei
Quoted 21 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?
Quoted 7 lines
Quoted 2 lines
We should keep the
owner->on_cpucheck inside the loop, otherwise we could continue spinning if the owner is not running.So how about checking in this way outside the loop for avoiding the spin?
if (owner) return owner->on_cpu;
So these owner->on_cpu checks outside of the loop "fixes" the issue as well, but I don't see the benefit of needing to guess why we break out of the spin loop (which may make things less readable) and checking owner->on_cpu duplicate times when one check is enough.
Re: softlockups in multi_cpu_stop
Quoted 46 lines · Jason Low
Quoted 45 lines · Ming Lei
Quoted 36 lines · Jason Low
Quoted 35 lines · Ming Lei
Quoted 33 lines · Davidlohr Bueso
Quoted 20 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Quoted 8 lines
Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
Quoted 5 lines
Quoted 2 lines
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?We should keep the
owner->on_cpucheck inside the loop, otherwise we could continue spinning if the owner is not running.So how about checking in this way outside the loop for avoiding the spin?
Quoted 7 lines
Quoted 2 lines
if (owner) return owner->on_cpu;So these
owner->on_cpuchecks outside of the loop "fixes" the issue as well, but I don't see the benefit of needing to guess why we break out of the spin loop (which may make things less readable) and checkingowner->on_cpuduplicate times when one check is enough.
I mean moving the check on owner->on_cpu outside loop, so there is only one check for both new and old owner. If it is inside loop, the check is only on old owner.
That is correct to keep it inside loop if you guys are sure new owner can't be scheduled out, but better to add comment why it can't, looks no one explained yet.
Thanks,
Re: softlockups in multi_cpu_stop
Quoted 55 lines · Ming Lei
Quoted 46 lines · Jason Low
Quoted 45 lines · Ming Lei
Quoted 36 lines · Jason Low
Quoted 35 lines · Ming Lei
Quoted 33 lines · Davidlohr Bueso
Quoted 20 lines · Ming Lei
Quoted 17 lines · On Fri, 06 Mar 2015 14:15:37 -0800 Davidlohr Bueso <[email protected]> wrote:
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }Quoted 8 lines
Quoted 5 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.Sasha, Ming (Cc'ed), does this address the issues you guys are seeing?
For the xfstest lockup, what matters is that the owner isn't running, since the following simple change does fix the issue:
I much prefer Jason's approach, which should also take care of the issue, as it includes the !owner->on_cpu stop condition to stop spinning.
Quoted 5 lines
Quoted 2 lines
But the check on
owner->on_cpushould be moved outside the loop because new owner can be scheduled out too, right?We should keep the
owner->on_cpucheck inside the loop, otherwise we could continue spinning if the owner is not running.So how about checking in this way outside the loop for avoiding the spin?
Quoted 7 lines
Quoted 2 lines
if (owner) return owner->on_cpu;So these
owner->on_cpuchecks outside of the loop "fixes" the issue as well, but I don't see the benefit of needing to guess why we break out of the spin loop (which may make things less readable) and checkingowner->on_cpuduplicate times when one check is enough.
Quoted 7 lines
I mean moving the check on
owner->on_cpuoutside loop, so there is only one check for both new and old owner. If it is inside loop, the check is only on old owner.That is correct to keep it inside loop if you guys are sure new owner can't be scheduled out, but better to add comment why it can't, looks no one explained yet.
The new owner can get rescheduled.
And if there's a new owner, then the spinner goes to rwsem_spin_on_owner() again and checks the new owner's on_cpu.
Re: softlockups in multi_cpu_stop
In reply to Davidlohr BuesoQuoted 17 lines · Davidlohr Bueso
Quoted 5 lines · Jason Low
In
owner_running()there are 2 conditions that would make it return
false: if the owner changed or if the owner is not running. However, that patch continues spinning if there is a "new owner" but it does not take into account that we may want to stop spinning if the owner is not running (due to getting rescheduled).So you're rationale is that we're missing this
need_resched:while (owner_running(sem, owner)) { /* abort spinning when need_resched */ if (need_resched()) { rcu_read_unlock(); return false; } }
Quoted 3 lines
Because the
owner_running()would return false, right? Yeah that makes sense, as missing a resched is a bug, as opposed to our heuristics being so painfully off.
Actually, the rationale is that when the lock owner reschedules while holding the lock, we'd want the spinners to stop spinning. The original owner_running() check takes care of this since it returns false if ->on_cpu gets set to false and the sem->owner != NULL would be false causing us to stop spinning . However, with the patch, when owner_running returns false, we check sem->owner, which causes the ->on_cpu check to essentially get ignored.
Re: softlockups in multi_cpu_stop
In reply to Jason LowOn Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Just in case, here's the updated patch which addresses Linus's comments and with a changelog.
Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving lock spinners), though I still haven't seen full confirmation that it addresses all of the lockup reports.
------
Subject: [PATCH] rwsem: Avoid spinning when owner is not running
Fixes tip commmit b3fd4f03ca0b (locking/rwsem: Avoid deceiving lock spinners).
When doing optimistic spinning in rwsem, threads should stop spinning when the lock owner is not running. While a thread is spinning on owner, if the owner reschedules, owner->on_cpu returns false and we stop spinning.
However, commit b3fd4f03ca0b essentially caused the check to get ignored because when we break out of the spin loop due to !on_cpu, we continue spinning if sem->owner != NULL.
This patch fixes this by making sure we stop spinning if the owner is not running. Furthermore, just like with mutexes, refactor the code such that we don't have separate checks for owner_running(). This makes it more straightforward in terms of why we exit the spin on owner loop and we would also avoid needing to "guess" why we broke out of the loop to make this more readable.
--- kernel/locking/rwsem-xadd.c | 31 +++++++++++-------------------- 1 files changed, 11 insertions(+), 20 deletions(-)
kernel/locking/rwsem-xadd.c +11 -20
@@ -324,32 +324,23 @@ done:return ret; } -static inline bool owner_running (struct rw_semaphore *sem, -struct task_struct *owner) -{ -if (sem->owner != owner) -return false; - -/* -* Ensure we emit the owner->on_cpu, dereference _after_ checking -* sem->owner still matches owner, if that fails, owner might -* point to free()d memory, if it still matches, the rcu_read_lock() -* ensures the memory stays valid. -*/ -barrier (); - -return owner->on_cpu; -} -static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) {long count;rcu_read_lock (); -while (owner_running (sem, owner)) { -/* abort spinning when need_resched */ -if (need_resched ()) { +while (sem->owner == owner) { +/* +* Ensure we emit the owner->on_cpu, dereference _after_ +* checking sem->owner still matches owner, if that fails, +* owner might point to free()d memory, if it still matches, +* the rcu_read_lock() ensures the memory stays valid. +*/ +barrier (); + +/* abort spinning when need_resched or owner is not running */ +if (!owner->on_cpu ||need_resched ()) {rcu_read_unlock ();return false; }
Signature · 1 line
1.7.2.5
Davidlohr Bueso
Acked-byRe: softlockups in multi_cpu_stop
Quoted 14 lines · Jason Low
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Just in case, here's the updated patch which addresses Linus's comments and with a changelog.
Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving lock spinners), though I still haven't seen full confirmation that it addresses all of the lockup reports.
------
Subject: [PATCH] rwsem: Avoid spinning when owner is not runningFixes tip commmit
b3fd4f03ca0b(locking/rwsem: Avoid deceiving lock spinners).
Quoted 7 lines
When doing optimistic spinning in rwsem, threads should stop spinning when the lock owner is not running. While a thread is spinning on owner, if the owner reschedules,
owner->on_cpureturns false and we stop spinning.However, commit
b3fd4f03ca0bessentially caused the check to get ignored because when we break out of the spin loop due to !on_cpu, we continue spinning ifsem->owner!= NULL.
I would mention the actual effects of the bug, either just a "lockup" and/or a fragment of the trace. But ultimately this comes down to missing a need_resched() condition.
Quoted 10 lines
This patch fixes this by making sure we stop spinning if the owner is not running. Furthermore, just like with mutexes, refactor the code such that we don't have separate checks for
owner_running(). This makes it more straightforward in terms of why we exit the spin on owner loop and we would also avoid needing to "guess" why we broke out of the loop to make this more readable.Cc Ming Lei <[email protected]>Cc Davidlohr Bueso <[email protected]>
Re: softlockups in multi_cpu_stop
Quoted 19 lines · Davidlohr Bueso
Quoted 18 lines · Jason Low
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Just in case, here's the updated patch which addresses Linus's comments and with a changelog.
Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving lock spinners), though I still haven't seen full confirmation that it addresses all of the lockup reports.
------
Subject: [PATCH] rwsem: Avoid spinning when owner is not runningFixes tip commmit
b3fd4f03ca0b(locking/rwsem: Avoid deceiving lock spinners).When doing optimistic spinning in rwsem, threads should stop spinning when the lock owner is not running. While a thread is spinning on owner, if the owner reschedules,
owner->on_cpureturns false and we stop spinning.
Quoted 6 lines
Quoted 3 lines
However, commit
b3fd4f03ca0bessentially caused the check to get ignored because when we break out of the spin loop due to !on_cpu, we continue spinning ifsem->owner!= NULL.I would mention the actual effects of the bug, either just a "lockup" and/or a fragment of the trace.
Right, we should mention about the lockup in the changelog.
Quoted 4 lines
Quoted 2 lines
Cc Ming Lei <[email protected]>Cc Davidlohr Bueso <[email protected]>Acked-by Davidlohr Bueso <[email protected]>
Thanks!
Re: softlockups in multi_cpu_stop
In reply to Jason LowQuoted 29 lines · Jason Low
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Just in case, here's the updated patch which addresses Linus's comments and with a changelog.
Note: The changelog says that it fixes (locking/rwsem: Avoid deceiving lock spinners), though I still haven't seen full confirmation that it addresses all of the lockup reports.
------
Subject: [PATCH] rwsem: Avoid spinning when owner is not runningFixes tip commmit
b3fd4f03ca0b(locking/rwsem: Avoid deceiving lock spinners).When doing optimistic spinning in rwsem, threads should stop spinning when the lock owner is not running. While a thread is spinning on owner, if the owner reschedules,
owner->on_cpureturns false and we stop spinning.However, commit
b3fd4f03ca0bessentially caused the check to get ignored because when we break out of the spin loop due to !on_cpu, we continue spinning ifsem->owner!= NULL.This patch fixes this by making sure we stop spinning if the owner is not running. Furthermore, just like with mutexes, refactor the code such that we don't have separate checks for
owner_running(). This makes it more straightforward in terms of why we exit the spin on owner loop and we would also avoid needing to "guess" why we broke out of the loop to make this more readable.
Quoted 3 lines
Cc Ming Lei <[email protected]>Cc Davidlohr Bueso <[email protected]>Signed-off-by Jason Low <[email protected]>
Quoted 39 lines
--- kernel/locking/rwsem-xadd.c | 31 +++++++++++-------------------- 1 files changed, 11 insertions(+), 20 deletions(-)kernel/locking/rwsem-xadd.c +1
-20@@ -324,32 +324,23 @@ done:return ret; } -static inline bool owner_running (struct rw_semaphore *sem, -struct task_struct *owner) -{ -if (sem->owner != owner) -return false; - -/* -* Ensure we emit the owner->on_cpu, dereference _after_ checking -* sem->owner still matches owner, if that fails, owner might -* point to free()d memory, if it still matches, the rcu_read_lock() -* ensures the memory stays valid. -*/ -barrier (); - -return owner->on_cpu; -} -static noinline bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner) {long count;rcu_read_lock (); -while (owner_running (sem, owner)) { -/* abort spinning when need_resched */ -if (need_resched ()) { +while (sem->owner == owner) {
kernel/locking/rwsem-xadd.c done: · line 343
Quoted 10 lines
kernel/locking/rwsem-xadd.c +10
-0+/* +* Ensure we emit the owner->on_cpu, dereference _after_ +* checking sem->owner still matches owner, if that fails, +* owner might point to free()d memory, if it still matches, +* the rcu_read_lock() ensures the memory stays valid. +*/ +barrier (); + +/* abort spinning when need_resched or owner is not running */ +if (!owner->on_cpu ||need_resched ()) {
BTW, could the need_resched() be handled in loop of rwsem_optimistic_spin() directly? Then code may get simplified a bit.
Thanks,
Ming Lei
Re: softlockups in multi_cpu_stop
Quoted 7 lines · Ming Lei
Quoted 4 lines · Jason Low
On Fri, 2015-03-06 at 13:12 -0800, Jason Low wrote:
Cc Ming Lei <[email protected]>Cc Davidlohr Bueso <[email protected]>Signed-off-by Jason Low <[email protected]>Reported-and-tested-by Ming Lei <[email protected]>
Thanks!
Quoted 21 lines
Quoted 10 lines
static noinline
boolrwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner){ long count; rcu_read_lock();kernel/locking/rwsem-xadd.c +1
-3-while (owner_running (sem, owner)) { -/* abort spinning when need_resched */ -if (need_resched ()) { +while (sem->owner == owner) {kernel/locking/rwsem-xadd.c done: · line 343
Quoted 10 lines
kernel/locking/rwsem-xadd.c +10
-0+/* +* Ensure we emit the owner->on_cpu, dereference _after_ +* checking sem->owner still matches owner, if that fails, +* owner might point to free()d memory, if it still matches, +* the rcu_read_lock() ensures the memory stays valid. +*/ +barrier (); + +/* abort spinning when need_resched or owner is not running */ +if (!owner->on_cpu ||need_resched ()) {
Quoted 3 lines
BTW, could the
need_resched()be handled in loop ofrwsem_optimistic_spin()directly? Then code may get simplified a bit.
We still need the need_resched() check here, since if the thread needs to reschedule, it should immediately stop spinning for the lock. Otherwise, it could potentially spin for a long time before it checks for it needs to reschedule.