rcu,sched: spinlock recursion on 3.5-rc2

Hey all,

I've got the following splat while fuzzing with trinity in a KVM tools guest on 3.5-rc2:

[ 8110.274070] BUG: spinlock recursion on CPU#0, rcu_torture_rea/2658
[ 8110.275014]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8110.275014] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8110.275014] Call Trace:
[ 8110.275014]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8110.275014]  [<ffffffff8197deab>] spin_bug+0x2b/0x40
[ 8110.275014]  [<ffffffff8197dfee>] do_raw_spin_lock+0x4e/0x140
[ 8110.275014]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.275014]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8110.275014]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8110.275014]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8110.275014]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8110.275014]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8110.275014]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8110.275014]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8110.275014]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8110.275014]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.275014]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8110.275014]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8110.275014]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8110.275014]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8110.275014]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8110.275014]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8110.275014]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8110.275014]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.275014]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.275014]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.275014]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.275014]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.275014]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] BUG: spinlock lockup on CPU#1, rcu_torture_rea/2653
[ 8110.306059]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ>  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8110.306059]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI>  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] sending NMI to all CPUs:
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8110.306059] RSP: 0018:ffff88001b803bd0  EFLAGS: 00000086
[ 8110.306059] RAX: 0000000000000400 RBX: 0000000000000001 RCX: 0000000000000830
[ 8110.306059] RDX: 0000000000000001 RSI: 0000000000000400 RDI: 0000000000000830
[ 8110.306059] RBP: ffff88001b803bd0 R08: ffffffff851259c0 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 8110.306059] R13: ffffffff851259c0 R14: 0000000000000001 R15: 000000000000ef40
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c40 ffffffff81093d2d 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c50 0000000000000086 0000000000000002 0000000200080000
[ 8110.306059]  ffff88001b803c20 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8110.306059]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8110.306059]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8110.306059]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff81116887>]  [<ffffffff81116887>] sub_preempt_count+0x87/0xb0
[ 8111.347769] RSP: 0018:ffff88000cf959e8  EFLAGS: 00000002
[ 8111.347769] RAX: ffff88000cf95fd8 RBX: 000000004eacf13e RCX: 0000000000000000
[ 8111.347769] RDX: 00000000000012db RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf959e8 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf94010
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000000 R15: 000000004eacf17b
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a28 ffffffff819674aa ffff88000cf95a18 ffff88000d9d6140
[ 8111.347769]  000000008abefb96 00000000948a9c80 0000000000000001 0000000000000086
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 01 f0 04 00 75 3e be 4b 0c 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff <eb> 17 0f 1f 80 00 00 00 00 81 ff fe 00 00 00 77 e0 eb aa 66 0f 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff819674a0>]  [<ffffffff819674a0>] delay_tsc+0x100/0x140
[ 8111.351463] RSP: 0018:ffff88000d575c90  EFLAGS: 00000006
[ 8111.351463] RAX: 0000000000000029 RBX: 000000006bcb86d7 RCX: 000000006bcb8700
[ 8111.351463] RDX: 00000000000012db RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575cc0 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000d574010
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: 000000006bcb8700
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575d10 ffff88000d9d6140 0000000092cd334a 00000000948a9c80
[ 8111.351463]  0000000000000001 ffff88000d9d6140 ffff88000d575cd0 ffffffff8196736a
[ 8111.351463]  ffff88000d575d10 ffffffff8197e07d ffff88000d575d10 0000000000000086
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: cf 0d 03 00 75 04 0f 0b eb fe 44 29 fb 41 89 c6 41 01 dd e8 f3 91 70 ff 66 90 89 c3 e9 6a ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 <bf> 01 00 00 00 e8 56 f3 7a ff 65 48 8b 04 25 88 d9 00 00 48 8b 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff819673c9>]  [<ffffffff819673c9>] delay_tsc+0x29/0x140
[ 8111.011063] RSP: 0018:ffff880041803d78  EFLAGS: 00000002
[ 8111.011063] RAX: 0000000000000004 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.011063] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: 000000008c9addbe
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803df8 ffff88000d9d6140 000000008c9addbe 00000000948a9c80
[ 8111.011063]  0000000000000001 0000000000000001 ffff880041803db8 ffffffff8196736a
[ 8111.011063]  ffff880041803df8 ffffffff8197e07d ffff880041803df8 0000000000000086
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 <48> 83 3d 37 d0 0d 03 00 75 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff819673c9>]  [<ffffffff819673c9>] delay_tsc+0x29/0x140
[ 8112.229101] RSP: 0018:ffff8800354d7ca0  EFLAGS: 00000006
[ 8112.229101] RAX: 0000000000000003 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8112.229101] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000090092bc0
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7d20 ffff88000d9d6140 0000000090092bc0 00000000948a9c80
[ 8112.229101]  0000000000000001 ffff88000d9d6140 ffff8800354d7ce0 ffffffff8196736a
[ 8112.229101]  ffff8800354d7d20 ffffffff8197e07d ffff8800354d7d20 0000000000000086
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 <48> 83 3d 37 d0 0d 03 00 75 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 
[ 8111.351463] BUG: spinlock lockup on CPU#2, rcub/5/11
[ 8111.351463]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.351463]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] sending NMI to all CPUs:
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.351463] RSP: 0018:ffff88000d575c30  EFLAGS: 00000086
[ 8111.351463] RAX: 0000000000000400 RBX: 0000000000000002 RCX: 0000000000000830
[ 8111.351463] RDX: 0000000000000002 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.351463] RBP: ffff88000d575c30 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[ 8111.351463] R13: ffffffff851259c0 R14: 0000000000000002 R15: 000000000000ef40
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575ca0 ffffffff81093d2d 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cb0 0000000000000086 0000000000000002 0000000200080000
[ 8111.351463]  ffff88000d575c80 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.351463]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.351463]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.351463]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff81116890>]  [<ffffffff81116890>] sub_preempt_count+0x90/0xb0
[ 8110.306059] RSP: 0018:ffff88001b803c20  EFLAGS: 00000006
[ 8110.306059] RAX: ffff88000cf73fd8 RBX: 00000000c3c0e658 RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c20 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000c3c0e67c
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c60 ffffffff819674aa 0000000000000000 ffff88000d9d6140
[ 8110.306059]  0000000001978cb7 00000000948a9c80 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c70 ffffffff8196736a ffff88001b803cb0 ffffffff8197e07d
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff eb 17 0f 1f 80 00 00 00 00 <81> ff fe 00 00 00 77 e0 eb aa 66 0f 1f 44 00 00 c9 0f 1f 80 00 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff819673d1>]  [<ffffffff819673d1>] delay_tsc+0x31/0x140
[ 8111.347769] RSP: 0018:ffff88000cf959f8  EFLAGS: 00000082
[ 8111.347769] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf95a28 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 000000008bfc4879
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a18 ffff88000d9d6140 000000008bfc4879 00000000948a9c80
[ 8111.347769]  0000000000000001 0000000000000086 ffff88000cf95a38 ffffffff8196736a
[ 8111.347769]  ffff88000cf95a78 ffffffff8197e07d ffff88000cf95a78 0000000000000086
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 48 83 3d 37 d0 0d 03 00 <75> 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 9e 92 70 ff 66 90 65 4c 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8112.229101] RSP: 0018:ffff8800354d7c90  EFLAGS: 00000082
[ 8112.229101] RAX: 00000000cfcf63b2 RBX: 00000000cfcf6383 RCX: 00000000cfcf6383
[ 8112.229101] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7c90 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800354d6010
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cd0 ffffffff81967425 ffff8800354d7d20 ffff88000d9d6140
[ 8112.229101]  00000000913d633e 00000000948a9c80 0000000000000001 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7ce0 ffffffff8196736a ffff8800354d7d20 ffffffff8197e07d
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff81967425>] delay_tsc+0x85/0x140
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8111.011063] RSP: 0018:ffff880041803d78  EFLAGS: 00000082
[ 8111.011063] RAX: 000012dccfa8fa68 RBX: 00000000cfa8fa68 RCX: 00000000cfa8fa68
[ 8111.011063] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000cf74010
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000004 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803df8 ffff88000d9d6140 000000008daabef7 00000000948a9c80
[ 8111.011063]  0000000000000001 0000000000000001 ffff880041803db8 ffffffff8196736a
[ 8111.011063]  ffff880041803df8 ffffffff8197e07d ffff880041803df8 0000000000000086
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 
[ 8112.229101] BUG: spinlock lockup on CPU#3, watchdog/3/27
[ 8112.229101]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8112.229101]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] sending NMI to all CPUs:
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff819673a1>]  [<ffffffff819673a1>] delay_tsc+0x1/0x140
[ 8111.347769] RSP: 0018:ffff88000cf95a28  EFLAGS: 00000002
[ 8111.347769] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf95a38 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 000000008f1a16ab
[ 8111.347769] R13: 00000000948a9c80 R14: 0000000000000001 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769]  ffff88000cf95a78 0000000000000086 ffff88000d9d6140 ffff88000d9d6158
[ 8111.347769]  00000000001d6140 000000000000008b ffff88000cf95aa8 ffffffff837c0b2b
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8196736a>] ? __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 00 00 00 48 89 e5 65 48 8b 14 25 58 60 1d 00 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a 01 ff 15 08 0c 47 03 c9 c3 66 0f 1f 44 00 00 55 <48> 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8112.229101] RSP: 0018:ffff8800354d7c40  EFLAGS: 00000086
[ 8112.229101] RAX: 0000000000000400 RBX: 0000000000000003 RCX: 0000000000000830
[ 8112.229101] RDX: 0000000000000003 RSI: 0000000000000400 RDI: 0000000000000830
[ 8112.229101] RBP: ffff8800354d7c40 R08: ffffffff851259c0 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 8112.229101] R13: ffffffff851259c0 R14: 0000000000000003 R15: 000000000000ef40
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cb0 ffffffff81093d2d 0000000000000000 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7cc0 0000000000000086 0000000000000002 0000000200080000
[ 8112.229101]  ffff8800354d7c90 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8112.229101]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8112.229101]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8112.229101]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.351463] RSP: 0018:ffff88000d575c80  EFLAGS: 00000082
[ 8111.351463] RAX: 00000000e96c109c RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.351463] RDX: 00000000000012df RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575c80 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000033fccab
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575cc0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8111.351463]  00000000033fccab 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cd0 ffffffff8196736a ffff88000d575d10 ffffffff8197e07d
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8196736a>]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059] RSP: 0018:ffff88001b803c70  EFLAGS: 00000082
[ 8110.306059] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012df RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c70 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000058864be
[ 8110.306059] R13: 00000000948a9c80 R14: 0000000000000000 R15: ffff88001b803d70
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803cb0 ffffffff8197e07d ffff88001b803cb0 0000000000000086
[ 8110.306059]  ffff88000d9d6140 ffff88000d9d6158 0000000000000287 0000000000000000
[ 8110.306059]  ffff88001b803ce0 ffffffff837c0b2b ffffffff81126620 000000000000fff8
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 55 48 c7 05 54 0c 47 03 a0 73 96 81 48 89 e5 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 ff 15 36 0c 47 03 <c9> c3 0f 1f 40 00 55 48 8d 04 bd 00 00 00 00 48 89 e5 65 48 8b 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff81070688>]  [<ffffffff81070688>] native_read_tsc+0x8/0x20
[ 8111.011063] RSP: 0018:ffff880041803d68  EFLAGS: 00000082
[ 8111.011063] RAX: 0000000007eddbe7 RBX: ffff88000d9d6140 RCX: 0000000007eddbe7
[ 8111.011063] RDX: 00000000000012e0 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803d68 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000091460d14
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000004 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803da8 ffffffff819673e2 ffff880041803df8 ffff88000d9d6140
[ 8111.011063]  0000000091460d14 00000000948a9c80 0000000000000001 0000000000000001
[ 8111.011063]  ffff880041803db8 ffffffff8196736a ffff880041803df8 ffffffff8197e07d
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 <c9> 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 00 00 
[ 8111.011063] BUG: spinlock lockup on CPU#4, rcu_torture_rea/2654
[ 8111.011063]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ>  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.011063]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI>  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] sending NMI to all CPUs:
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.011063] RSP: 0018:ffff880041803d18  EFLAGS: 00000086
[ 8111.011063] RAX: 0000000000000400 RBX: 0000000000000004 RCX: 0000000000000830
[ 8111.011063] RDX: 0000000000000004 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.011063] RBP: ffff880041803d18 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004
[ 8111.011063] R13: ffffffff851259c0 R14: 0000000000000004 R15: 000000000000ef40
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803d88 ffffffff81093d2d 0000000000000000 0000000000000001
[ 8111.011063]  ffff880041803d98 0000000000000086 0000000000000002 0000000200080000
[ 8111.011063]  ffff880041803d68 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.011063]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.011063]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.011063]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.347769] RSP: 0018:ffff88000cf959e8  EFLAGS: 00000082
[ 8111.347769] RAX: 0000000069137f7d RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf959e8 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000932d7753
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a28 ffffffff819673e2 ffff88000cf95a18 ffff88000d9d6140
[ 8111.347769]  00000000932d7753 00000000948a9c80 0000000000000001 0000000000000086
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8110.306059] RSP: 0018:ffff88001b803c30  EFLAGS: 00000082
[ 8110.306059] RAX: 000012e369144e60 RBX: 0000000069144e60 RCX: 0000000069144e60
[ 8110.306059] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c60 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88001b803d70
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  0000000000000000 ffff88000d9d6140 000000000a12546a 00000000948a9c80
[ 8110.306059]  0000000000000000 ffff88001b803d70 ffff88001b803c70 ffffffff8196736a
[ 8110.306059]  ffff88001b803cb0 ffffffff8197e07d ffff88001b803cb0 0000000000000086
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff81070688>]  [<ffffffff81070688>] native_read_tsc+0x8/0x20
[ 8112.229101] RSP: 0018:ffff8800354d7c90  EFLAGS: 00000082
[ 8112.229101] RAX: 0000000069147143 RBX: ffff88000d9d6140 RCX: 0000000069147143
[ 8112.229101] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7c90 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000003e7b190
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cd0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8112.229101]  0000000003e7b190 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7ce0 ffffffff8196736a ffff8800354d7d20 ffffffff8197e07d
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 <c9> 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 00 00 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.351463] RSP: 0018:ffff88000d575c80  EFLAGS: 00000082
[ 8111.351463] RAX: 00000000686f9b15 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.351463] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575c80 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000006a0401f
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575cc0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8111.351463]  0000000006a0401f 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cd0 ffffffff8196736a ffff88000d575d10 ffffffff8197e07d
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8111.347769] BUG: spinlock lockup on CPU#0, rcu_torture_rea/2658
[ 8111.347769]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.347769]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] sending NMI to all CPUs:
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.347769] RSP: 0018:ffff88000cf95998  EFLAGS: 00000086
[ 8111.347769] RAX: 0000000000000400 RBX: 0000000000000000 RCX: 0000000000000830
[ 8111.347769] RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.347769] RBP: ffff88000cf95998 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 8111.347769] R13: ffffffff851259c0 R14: 0000000000000000 R15: 000000000000ef40
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a08 ffffffff81093d2d 0000000000000000 0000000000000086
[ 8111.347769]  ffff88000cf95a18 0000000000000086 0000000000000002 0000000200080000
[ 8111.347769]  ffff88000cf959e8 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.347769]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.347769]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.347769]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff81116890>]  [<ffffffff81116890>] sub_preempt_count+0x90/0xb0
[ 8110.306059] RSP: 0018:ffff88001b803c20  EFLAGS: 00000006
[ 8110.306059] RAX: ffff88000cf73fd8 RBX: 0000000046560e3e RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c20 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000046560e6d
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c60 ffffffff819674aa 0000000000000000 ffff88000d9d6140
[ 8110.306059]  000000000c21d807 00000000948a9c80 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c70 ffffffff8196736a ffff88001b803cb0 ffffffff8197e07d
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff eb 17 0f 1f 80 00 00 00 00 <81> ff fe 00 00 00 77 e0 eb aa 66 0f 1f 44 00 00 c9 0f 1f 80 00 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff819674d1>]  [<ffffffff819674d1>] delay_tsc+0x131/0x140
[ 8111.351463] RSP: 0018:ffff88000d575cc8  EFLAGS: 00000082
[ 8111.351463] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 0000000000000000
[ 8111.351463] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000008a00398
[ 8111.351463] R13: 00000000948a9c80 R14: 0000000000000000 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffffffff8196736a ffff88000d575d10 ffffffff8197e07d ffff88000d575d10
[ 8111.351463]  0000000000000086 ffff88000d9d6140 ffff88000d9d6158 0000000000000002
[ 8111.351463]  0000000000000000 ffff88000d575d40 ffffffff837c0b2b ffffffff8112bfec
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8196736a>] ? __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 56 f3 7a ff 65 48 8b 04 25 88 d9 00 00 48 8b 80 38 e0 ff ff a8 08 74 05 e8 bd 85 e5 01 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f c9 <c3> 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8d 04 bf 48 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff819673a4>]  [<ffffffff819673a4>] delay_tsc+0x4/0x140
[ 8111.011063] RSP: 0018:ffff880041803da8  EFLAGS: 00000002
[ 8111.011063] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.011063] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000191edad
[ 8111.011063] R13: 00000000948a9c80 R14: 0000000000000000 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803db8 ffffffff8196736a ffff880041803df8 ffffffff8197e07d
[ 8111.011063]  ffff880041803df8 0000000000000086 ffff88000d9d6140 ffff88000d9d6158
[ 8111.011063]  0000000000000001 0000000000000000 ffff880041803e28 ffffffff837c0b2b
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 48 89 e5 65 48 8b 14 25 58 60 1d 00 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a 01 ff 15 08 0c 47 03 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 <41> 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8112.229101] RSP: 0018:ffff8800354d7ca0  EFLAGS: 00000082
[ 8112.229101] RAX: 000012e54536e761 RBX: 000000004536e761 RCX: 000000004536e761
[ 8112.229101] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800354d6010
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  0000000000000000 ffff88000d9d6140 0000000005bcbf34 00000000948a9c80
[ 8112.229101]  0000000000000000 ffff88000d9d6140 ffff8800354d7ce0 ffffffff8196736a
[ 8112.229101]  ffff8800354d7d20 ffffffff8197e07d ffff8800354d7d20 0000000000000086
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 

Ok, that's nasty.

The torture thread got preempted. rcu_preempt_note_context_switch() tries to unlock the boosting rt mutex.

Though rcu_preempt_note_context_switch() is called with rq lock held. So it's not a surprise that the code will dead lock.

My brain hurts already from looking, so Paul to the rescue!

Thanks,

tglx

My brain hurts from beating my head on my desk. It seems that attempts to enhance PREEMPT_RCU's read-side performance require even more paranoia than I normally bring to bear. :-/

Please see below for what I expect is the relevant revert.

Thanx, Paul

------------------------------------------------------------------------

Revert "rcu: Move PREEMPT_RCU preemption to switch_to() invocation"

This reverts commit 616c310e83b872024271c915c1b9ab505b9efad9 (Move PREEMPT_RCU preemption to switch_to() invocation) which can result in runqueue deadlock.

index 88e466b..43b39d6 100644
@@ -705,7 +705,6 @@ static void stack_proc(void *arg)
 	struct task_struct *from = current, *to = arg;
 
 	to->thread.saved_task = from;
-	rcu_switch_from(from);
 	switch_to(from, to, from);
 }
 

index 26d1a47..9cac722 100644
@@ -184,7 +184,6 @@ static inline int rcu_preempt_depth(void)
 /* Internal to kernel */
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
-extern void rcu_preempt_note_context_switch(void);
 extern void rcu_check_callbacks(int cpu, int user);
 struct notifier_block;
 extern void rcu_idle_enter(void);

index adb5e5a..148c6db 100644
@@ -87,6 +87,10 @@ static inline void kfree_call_rcu(struct rcu_head *head,
 
 #ifdef CONFIG_TINY_RCU
 
+static inline void rcu_preempt_note_context_switch(void)
+{
+}
+
 static inline int rcu_needs_cpu(int cpu)
 {
 	return 0;

@@ -94,6 +98,7 @@ static inline int rcu_needs_cpu(int cpu)
 
 #else /* #ifdef CONFIG_TINY_RCU */
 
+void rcu_preempt_note_context_switch(void);
 int rcu_preempt_needs_cpu(void);
 
 static inline int rcu_needs_cpu(int cpu)

@@ -106,6 +111,7 @@ static inline int rcu_needs_cpu(int cpu)
 static inline void rcu_note_context_switch(int cpu)
 {
 	rcu_sched_qs(cpu);
+	rcu_preempt_note_context_switch();
 }
 
 /*

index 4059c0f..06a4c5f 100644
@@ -1871,22 +1871,12 @@ static inline void rcu_copy_process(struct task_struct *p)
 	INIT_LIST_HEAD(&p->rcu_node_entry);
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-	if (prev->rcu_read_lock_nesting != 0)
-		rcu_preempt_note_context_switch();
-}
-
 #else
 
 static inline void rcu_copy_process(struct task_struct *p)
 {
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-}
-
 #endif
 
 #ifdef CONFIG_SMP

index 0da7b88..d994530 100644
@@ -201,6 +201,7 @@ void rcu_note_context_switch(int cpu)
 {
 	trace_rcu_utilization("Start context switch");
 	rcu_sched_qs(cpu);
+	rcu_preempt_note_context_switch(cpu);
 	trace_rcu_utilization("End context switch");
 }
 EXPORT_SYMBOL_GPL(rcu_note_context_switch);

index 7f5d138..bd220db 100644
@@ -430,6 +430,7 @@ DECLARE_PER_CPU(char, rcu_cpu_has_work);
 /* Forward declarations for rcutree_plugin.h */
 static void rcu_bootup_announce(void);
 long rcu_batches_completed(void);
+static void rcu_preempt_note_context_switch(int cpu);
 static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp);
 #ifdef CONFIG_HOTPLUG_CPU
 static void rcu_report_unblock_qs_rnp(struct rcu_node *rnp,

index 2411000..fb6e07e 100644
@@ -153,7 +153,7 @@ static void rcu_preempt_qs(int cpu)
  *
  * Caller must disable preemption.
  */
-void rcu_preempt_note_context_switch(void)
+static void rcu_preempt_note_context_switch(int cpu)
 {
 	struct task_struct *t = current;
 	unsigned long flags;

@@ -164,7 +164,7 @@ void rcu_preempt_note_context_switch(void)
 	    (t->rcu_read_unlock_special & RCU_READ_UNLOCK_BLOCKED) == 0) {
 
 		/* Possibly blocking in an RCU read-side critical section. */
-		rdp = __this_cpu_ptr(rcu_preempt_state.rda);
+		rdp = per_cpu_ptr(rcu_preempt_state.rda, cpu);
 		rnp = rdp->mynode;
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		t->rcu_read_unlock_special |= RCU_READ_UNLOCK_BLOCKED;

@@ -228,7 +228,7 @@ void rcu_preempt_note_context_switch(void)
 	 * means that we continue to block the current grace period.
 	 */
 	local_irq_save(flags);
-	rcu_preempt_qs(smp_processor_id());
+	rcu_preempt_qs(cpu);
 	local_irq_restore(flags);
 }
 

@@ -1002,6 +1002,14 @@ void rcu_force_quiescent_state(void)
 EXPORT_SYMBOL_GPL(rcu_force_quiescent_state);
 
 /*
+ * Because preemptible RCU does not exist, we never have to check for
+ * CPUs being in quiescent states.
+ */
+static void rcu_preempt_note_context_switch(int cpu)
+{
+}
+
+/*
  * Because preemptible RCU does not exist, there are never any preempted
  * RCU readers.
  */

index d5594a4..eaead2d 100644
@@ -2081,7 +2081,6 @@ context_switch(struct rq *rq, struct task_struct *prev,
 #endif
 
 	/* Here we just switch the register state and the stack. */
-	rcu_switch_from(prev);
 	switch_to(prev, next, prev);
 
 	barrier();

Hmm, not sure. The deadlock was not triggered in switch_to. It was just at the beginning of __schedule()

need_resched:

	preempt_disable();
        cpu = smp_processor_id();
	rq = cpu_rq(cpu);
        rcu_note_context_switch(cpu);
	
	Which ends up in rcu_read_unlock_special() which tries to
	unlock the rtmutex.

So that code is still there ....

Thanks,

tglx

How can you tell? switch_to() is a macro not a function, it won't ever show up on a stack-trace.. but I think you meant context_switch() but that typically gets inlined, similar problem.

Bah, I misread the trace and the rcu code. :(

So, yes the revert should solve the problem. Looks like I need more coffee, not Paul :)

Thanks,

tglx

Well, given that the commit that I reverted was my own, I definitely needed more of something or another... :-)

Thanx, Paul

addr2line -i confirmed it was pointing to switch_to(), if it matters.

btw, wouldn't it be nice if there were a short script that would take an oops and convert all those kernel addresses into file+line numbers based on a given vmlinux? It would make our lives so much easier.

???

rcu_note_context_switch() does not call rcu_preempt_note_context_switch() unless the patch is reverted. No rcu_preempt_note_context_switch(), no rcu_read_unlock_special().

Besides which, unless I am even more confused than normal, __schedule() doesn't hold any runqueue locks at this point. However, those locks really are held across the call to switch_to() in context_switch().

I bet that gcc inlined context_switch().

Thanx, Paul