frequent lockups in 3.18rc4

yeah just a wild guess, just that that trace looked so.. clean.

It sure failed quickly, but not in the "machine is totally locked up" sense, just "shit is all corrupted" sense. So it might be a completely different thing, or it could be a different manifestation of a corruptor.

I guess we'll see how things go now that I marked it 'bad'.

I'll give it a quick run with just lsetxattr again just to see what happens, but before I leave this one run over the weekend, I'll switch it back to "do everything", and pick it up again on Monday.

Dave

Looking at the callchain: up to the point where dump_stack() is called everything is preemtible context. So dump_stack() would need to loop for a few seconds to trigger the NMI watchdog.

I doubt that dump_trace() itself is broken, but the call site might have handed in something which causes memory corruption. And looking at set_track() and the completely undocumented way how it retrieves the storage for the trace entries via get_track() makes my brain melt.

Thanks,

tglx

So, every time there is a slab allocation the entire stack trace gets saved as human readable text. And for each line in the trace, is_module_text_address() can be called, which has huge overhead walking the entire list of loaded modules. No wonder there are timeouts...

Hi,

I've been following this thread with some interrest because my mythtv based media centre suffered from sudden freezes. I thought I had tried everything to solve the problem (including exchanging hardware) until I caught the slashdot article discussing this bug. I was running on 3.17.3 and and experienced daily freezes while watching DVB-C recorded H.264 (HD) video, both live and pre-recorded streams. Both using VAAPI GPU accelaration and CPU decoding. Ever since downgrading to 3.16.7 my system has been solid as a rock again.

The load on this machine is marginal. Most playback is done using GPU and other than that, the main load is hammering at most 2 HD (1G/10mins) streams to SSD. When idle, the system crawls DVB channels for EPG info. Crashes happened mostly while watching TV and ~5 minutes after a background recording started or after a couple of hours watching live TV, but were very hard to trigger. When freezing the system was completely inaccessible, without any panic logging on (ssh remote connected) console. I have never been able to find any evidence in old logs after reboot. The freezes happened both with and without NMI watchdog enabled.

Hardware is J1900 BayTrail (ValleyView) based ASRock miniITX board. The DVB-C card requires out-of-tree built ddbridge drivers (~endriss/media_build_expermintal) so the kernels (both stable and unstable) are tainted I guess?

I'm a moderately experienced Linux user. I do build my own kernels, but am not very knowledgable about the inner workings you guys are discussing here. The system is a (family) production device so there is not a lot of room for testing, let alone bisecting. I do now have a spare J1900 lying around, so there is opportunity to build a dedicated test system if needed.

Hope this may help in finding the right direction for this bug?

Regards,
Martin

If 'but' was any useful, it would be a logic operator

If you can reproduce it with your spare J1900 system and could perhaps bisect it there, that would be a huge help.

Linus

Side note: your load sounds superficially more like the one Dâniel Fraga had, who saw lockups with 3.17 but 3.18-rc7 actually works for him.

But if you're running 3.17.3, I think you should already have all the relevant fixes throuigh the -stable tree.

Linus

I'll give it a shot and see if I can get to freeze it on 3.17.3 as a start by playing content from the prd backend, but don't expect fast respons times... busy man...

M.

I've finally received the memory I needed to prepare the spare J1900 and have it now configured as mythfrontend to the DVB-C backend that was freezing on 3.17.3. It's been playing liveTV for hours now and is still going strong. I'd say the freezes can't be reproduced this way. The only difference being the disk I/O that is missing on the front-end.

I will give 3.18 a try on production J1900. Knowing I can go back to safety in 3.16.7 won't hurt too much of my reputation I hope.

Best regards,
Martin

If 'but' was any useful, it would be a logic operator

3.18 froze twice (just to be sure) as well. Will commence the slow and painful bisect between 3.16.7 and 3.17rc1 after I've established that the latter freezes reliably.
I assume my problems are not linked to the RCU problems described in this thread since I have no logging in kernel and different symptoms? Should I start a new thread, or maybe better file a bug and report progress on the bisect there? I won't be too noisy on the list with my findings untill I'm done though.

Best regards,
Martin
--
If 'but' was any useful, it would be a logic operator

Wrong. It gets saved as address. No conversion to text at all.

You would have to have a gazillion of modules to make that overhead big enough to trigger a multi seconds watchdog.

Thanks,

tglx

I went back through all of the traces Dave has posted in this thread. This one looks like vm debugging is on:

http://marc.info/?l=linux-kernel&m=141632237304726&w=2

Another had a function call from CONFIG_DEBUG_PAGEALLOC:

http://marc.info/?l=linux-kernel&m=141701248210949&w=2

So one idea is that our allocation/freeing of pages is dramatically more expensive and we're hitting a strange edge condition. Maybe we're even faulting on a readonly page from a horrible place?

[83246.925234] end_request: I/O error, dev sda, sector 0

Ext3/4 shouldn't be doing IO to sector zero. Something is stomping on ram?

-chris

Well, various allocators have definitely shown up a lot. DEBUG_PAGEALLOC does horrible things to performance, though, and the kernel will just spend a *lot* of time in memory allocators when it is on. So it might just be "yeah, the traces show allocations a lot, but that might just be because allocation is slow". The last one showed slub debugging - getting a call trace for the allocation.

I'd buy memory corruption through wild pointers as the reason, but quite frankly, that tends to have completely different failure modes. Not NMI watchdogs.

So it must be some very particular corruption. I still vote for "let's see if Dave can narrow it down with bisection".

Linus

I vote for "plane ticket to drinking island".

So I've been continuing the bisect, leaving the 'good' cases running for as long as a day and a half, and it's taking me down a path of staging commits again (which is crap, as I don't even have that enabled).

git bisect start
git bisect bad 7d1311b93e58ed55f3a31cc8f94c4b8fe988a2b9 git bisect good 19583ca584d6f574384e17fe7613dfaeadcdc4a6 git bisect bad ae045e2455429c418a418a3376301a9e5753a0a8 git bisect bad 53ee983378ff23e8f3ff95ecf99dea7c6c221900 git bisect good 2042088cd67d0064d18c52c13c69af2499907bb1 git bisect good 98959948a7ba33cf8c708626e0d2a1456397e1c6 git bisect bad 6f929b4e5a022c3ca806c1675ccb833c42086853

So either one of those 'good's actually wasn't, or I'm just cursed.

I'm going to lose access to this machine next week, so unless I see this on different hardware in the new year, I'm running out of options.

(on the bright side, having people stare at various suspect code
throughout this thread does some to have yielded some good from all of this).

Dave

Even if there was a good that wasn't, that last "bad" (6f929b4e5a02) is already sufficient just on its own to say that likely v3.16 already had the problem.

Just do

gitk v3.16..6f929b4e5a02

and cry.

(or "git diff --stat -M v3.16...6f929b4e5a02" to see what that commit brought in from the common ancestor).

So I'd call that bisect a failure, and your "v3.16 is fine" is actually suspect after all. Which *might* mean that it's some hardware issue after all. Or there are multiple different problems, and while v3.16 was fine, the problem was introduced earlier (in the common ancestor of that staging tree), then fixed for 3.16, and then re-introduced later again.

Anyway, you might as well stop bisecting. Regardless of where it lands in the remaining pile, it's not going to give us any useful information, methinks.

I'm stumped.

Maybe it's worth it to concentrate on just testing current kernels, and instead try to limit the triggering some other way. In particular, you had a trinity run that was *only* testing lsetxattr(). Is that really *all* that was going on? Obviously trinity will be using timers, fork, and other things? Can you recreate that lsetxattr thing, and just try to get as many problem reports as possible from one particular kernel (say, 3.18, since that should be a reasonable modern base with hopefully not a lot of other random issues)?

Together with perhaps config checks. You've done some those already. Did it reproduce without preemption, for example?

Does anybody have any smart ideas?

Linus

Is it possible that Dave and myself were seeing the same problem after all?

I'll go bisect it even further back...

Thanks,
Sasha

For both of you, I'm curious how long 3.18 lasts if you turn off the serial console (and netconsole) completely.

-chris

I didn't try turning it off, but I tried switching debug level to critical which meant that nothing was going out. I still saw the same hang...

Thanks,
Sasha

Could be. You do have commonalities, even if the actual symptoms then differ. And while it looked different when you could trigger it with 3.16 but DaveJ couldn't, that's up in the air now that I doubt that 3.16 really is ok for DaveJ after all..

And you might have a better luck bisecting it, since you seem to be able to trigger your RCU lockup much more quickly (and apparently reliably? Correct?)

Linus

Right, and it reproduces in 3.10 as well, so it's not really a new thing.

What's odd is that I don't remember seeing this bug so long in the past, I'll try bisecting trinity rather than the kernel - it's the only other thing that changed.

Thanks,
Sasha

So I think DaveJ mentioned it that Trinity recently changed its test task count and is now more aggressively loading the system. Such a change might have made a dormant, resource limits related bug or load dependent race more likely.

I think at this point it would also be useful to debug the hang itself directly: using triggered printks and kgdb and drilling into all the data structures to figure out why the system isn't progressing.

If the bug triggers in a VM (which your testing uses) the failed kernel state ought to be a lot more accessible than bare metal.

That it triggers in a VM, and if it's the same bug as DaveJ's, that also makes the hardware bug theory a lot less likely.

Thanks,

Ingo

So I checked out trinity from half a year ago, and could not reproduce the stall any more. Not on v3.16 nor on the current -next.

I ran bisection on trinity, rather than the kernel, and got the following result:

commit f2be2d5ffe4bf896eb5418972013822a2bef0cee

Author: Dave Jones <[email protected]>
Date:   Mon Aug 4 19:55:17 2014 -0400

begin some infrastructure to use a bunch of test files for fsx like ops.

I've been running trinity f2be2d5ff^ on -next for two hours now, and there's no sign of a lockup. Previously it took ~10 minutes trigger.

Thanks,
Sasha

Heh. That commit is pretty small, but I guess the effect of having a number of regular files open and being used on the trinity loads can be almost arbitrarily large.

Where do those files get opened? What filesystem?

That might actually explain your and DaveJ's big differences: you are running in virtualization, and I remember some 9p traces with virtio etc from your reports.

While DaveJ obviously runs on bare hardware, possibly on /tmp and tmpfs?

DaveJ? Is there anything limiting the size of those files?

Linus

Right, it's virtio-9p. However, virtio-9p acts merely as a proxy to an underlying tmpfs - so while it's slow, I don't think it's way slower than the average disk backed ext4.

Thanks,
Sasha

I was thinking more in the sense of "how much of the trouble is about something like tmpfs eating tons of memory when trinity starts doing random system calls on those files".

I was also thinking that some of it might be filesystem-specific. We already *did* see one trace where it was in the loop getting virtio channel data. Maybe it's actually possible to overwhelm the 9p filesystem exactly because the backing store is tmpfs, and basically have a CPU 100% busy handling ring events from the virtual filesystem..

But I'm just flailing..

Linus

I started seeing this behavior somewhere around 3.16 with CONFIG_PREEMPT set. Setting CONFIG_PREEMPT off seems to help. And, yes, it happens on high load (compiling mozilla, xul) and using qemu chroot to compile mesa.

I'm seeing a few persons bisecting already. If you want, I could start bisecting too, but 3.16 was unstable for me as I'm still on reiserfs.

Jeff.

in the cwd where trinity is run from. In my case, ext4

Nope. We could for eg, do a random truncate() with a huge size, and it would try and create something enormous. write()'s are limited to page size, there might be some other syscalls I haven't added safety guards to.

Dave

Don't know if smart but we can check whether it is a hardware failure if we try to reproduce the exact same failure on a second, identical machine. If Dave gets one, of course.

Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

yeah, likewise. I don't recall any bug that's given me this much headache. I don't think it's helped that the symptoms are vague enough that a number of people have thought they've seen the same thing, which have turned out to be unrelated incidents. At least some of those have gotten closure though it seems.

I'll let it run overnight, but so far after 4hrs, on .18 it's not done anything.

Next kernel build I try, I'll turn that off. I don't remember if we've already tried that. I *think* we just tried the non-preempt rcu stuff, but not "no preemption at all". I wish I'd kept better notes about everything tried so far too, but I hadn't anticipated this dragging out so long. Live and learn..

Dave

Two hours later, it had spewed this, but survived. (Trinity had quit after that point because /proc/sys/kernel/tainted changed).

[18755.303442] WARNING: CPU: 1 PID: 25572 at kernel/watchdog.c:317 watchdog_overflow_callback+0xdd/0x130()
[18755.303472] Watchdog detected hard LOCKUP on cpu 1
[18755.303487] CPU: 1 PID: 25572 Comm: trinity-c25 Not tainted 3.18.0+ #101 
[18755.303527]  ffffffff81a66315 00000000c1ad8e75 ffff880244205b88 ffffffff817d317e
[18755.303556]  0000000000110001 ffff880244205be0 ffff880244205bc8 ffffffff81078a01
[18755.303586]  0000000000000000 0000000000000001 0000000000000000 ffff880244205d30
[18755.303616] Call Trace:
[18755.303627]  <NMI>  [<ffffffff817d317e>] dump_stack+0x4f/0x7c
[18755.303654]  [<ffffffff81078a01>] warn_slowpath_common+0x81/0xa0
[18755.303675]  [<ffffffff81078a75>] warn_slowpath_fmt+0x55/0x70
[18755.303696]  [<ffffffff8112fea0>] ? restart_watchdog_hrtimer+0x60/0x60
[18755.303718]  [<ffffffff8112ff7d>] watchdog_overflow_callback+0xdd/0x130
[18755.303742]  [<ffffffff81173a7c>] __perf_event_overflow+0xac/0x2a0
[18755.303765]  [<ffffffff81019952>] ? x86_perf_event_set_period+0xe2/0x150
[18755.303787]  [<ffffffff81174644>] perf_event_overflow+0x14/0x20
[18755.303809]  [<ffffffff8101f479>] intel_pmu_handle_irq+0x209/0x410
[18755.303831]  [<ffffffff8101875b>] perf_event_nmi_handler+0x2b/0x50
[18755.303853]  [<ffffffff81007634>] nmi_handle+0xa4/0x1e0
[18755.303872]  [<ffffffff81007595>] ? nmi_handle+0x5/0x1e0
[18755.303892]  [<ffffffff810079aa>] default_do_nmi+0x7a/0x1d0
[18755.303911]  [<ffffffff81007bb8>] do_nmi+0xb8/0xf0
[18755.303929]  [<ffffffff817e0c2a>] end_repeat_nmi+0x1e/0x2e
[18755.303948]  <<EOE>>  <UNK> 
[18755.303959] ---[ end trace 6362f5b39b85eb2c ]---
[18755.303983] perf interrupt took too long (7018 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[18758.481443] ------------[ cut here ]------------
[18758.481475] WARNING: CPU: 3 PID: 25965 at kernel/watchdog.c:317 watchdog_overflow_callback+0xdd/0x130()
[18758.481512] Watchdog detected hard LOCKUP on cpu 3
[18758.481531] CPU: 3 PID: 25965 Comm: trinity-c418 Tainted: G        W      3.18.0+ #101 
[18758.482554]  ffffffff81a66315 00000000c93c9d92 ffff880244605b88 ffffffff817d317e
[18758.483585]  0000000000110004 ffff880244605be0 ffff880244605bc8 ffffffff81078a01
[18758.484614]  0000000000000000 0000000000000003 0000000000000000 ffff880244605d30
[18758.485651] Call Trace:
[18758.486657]  <NMI>  [<ffffffff817d317e>] dump_stack+0x4f/0x7c
[18758.487670]  [<ffffffff81078a01>] warn_slowpath_common+0x81/0xa0
[18758.488676]  [<ffffffff81078a75>] warn_slowpath_fmt+0x55/0x70
[18758.489675]  [<ffffffff8112fea0>] ? restart_watchdog_hrtimer+0x60/0x60
[18758.490681]  [<ffffffff8112ff7d>] watchdog_overflow_callback+0xdd/0x130
[18758.491687]  [<ffffffff81173a7c>] __perf_event_overflow+0xac/0x2a0
[18758.492677]  [<ffffffff81019952>] ? x86_perf_event_set_period+0xe2/0x150
[18758.493668]  [<ffffffff81174644>] perf_event_overflow+0x14/0x20
[18758.494662]  [<ffffffff8101f479>] intel_pmu_handle_irq+0x209/0x410
[18758.495653]  [<ffffffff8101875b>] perf_event_nmi_handler+0x2b/0x50
[18758.496652]  [<ffffffff81007634>] nmi_handle+0xa4/0x1e0
[18758.497646]  [<ffffffff81007595>] ? nmi_handle+0x5/0x1e0
[18758.498644]  [<ffffffff811080cf>] ? is_module_text_address+0x3f/0x50
[18758.499644]  [<ffffffff810079aa>] default_do_nmi+0x7a/0x1d0
[18758.500643]  [<ffffffff81007bb8>] do_nmi+0xb8/0xf0
[18758.501633]  [<ffffffff817e0c2a>] end_repeat_nmi+0x1e/0x2e
[18758.502619]  [<ffffffff811080cf>] ? is_module_text_address+0x3f/0x50
[18758.503606]  [<ffffffff811080cf>] ? is_module_text_address+0x3f/0x50
[18758.504583]  [<ffffffff811080cf>] ? is_module_text_address+0x3f/0x50
[18758.505548]  <<EOE>>  [<ffffffff810986b8>] __kernel_text_address+0x58/0x80
[18758.506526]  [<ffffffff81182a24>] ? free_one_page+0x1c4/0x520
[18758.507503]  [<ffffffff81006d8f>] print_context_stack+0x8f/0x100
[18758.508483]  [<ffffffff81005710>] dump_trace+0x140/0x370
[18758.509464]  [<ffffffff811b0961>] ? remove_vma+0x71/0x80
[18758.510445]  [<ffffffff81013ecf>] save_stack_trace+0x2f/0x50
[18758.511428]  [<ffffffff811d4f20>] set_track+0x70/0x140
[18758.512412]  [<ffffffff817d0f85>] free_debug_processing+0x157/0x22a
[18758.513387]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[18758.514344]  [<ffffffff817d10ad>] __slab_free+0x55/0x320
[18758.515298]  [<ffffffff8138e016>] ? debug_check_no_obj_freed+0x156/0x250
[18758.516252]  [<ffffffff811d7fd2>] kmem_cache_free+0x262/0x280
[18758.517201]  [<ffffffff811b0961>] ? remove_vma+0x71/0x80
[18758.518133]  [<ffffffff811b0961>] remove_vma+0x71/0x80
[18758.519042]  [<ffffffff811b3f1c>] exit_mmap+0x13c/0x1a0
[18758.519938]  [<ffffffff81075a2b>] mmput+0x6b/0x100
[18758.520826]  [<ffffffff8107a02e>] do_exit+0x29e/0xba0
[18758.521707]  [<ffffffff81088d41>] ? get_signal+0x2c1/0x710
[18758.522585]  [<ffffffff8138cb27>] ? debug_smp_processor_id+0x17/0x20
[18758.523463]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[18758.524336]  [<ffffffff810c5466>] ? lock_release_holdtime.part.24+0xe6/0x160
[18758.525220]  [<ffffffff8107b9dc>] do_group_exit+0x4c/0xc0
[18758.526099]  [<ffffffff81088d8c>] get_signal+0x30c/0x710
[18758.526977]  [<ffffffff8138cb27>] ? debug_smp_processor_id+0x17/0x20
[18758.527858]  [<ffffffff81002477>] do_signal+0x37/0x770
[18758.528735]  [<ffffffff8138d131>] ? free_object+0x81/0xb0
[18758.529608]  [<ffffffff8138db87>] ? debug_object_free+0xf7/0x150
[18758.530480]  [<ffffffff810ec285>] ? hrtimer_nanosleep+0x155/0x1c0
[18758.531357]  [<ffffffff810eac60>] ? hrtimer_get_res+0x50/0x50
[18758.532225]  [<ffffffff81002c15>] do_notify_resume+0x65/0x80
[18758.533102]  [<ffffffff8137f9ce>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[18758.533980]  [<ffffffff817deeff>] int_signal+0x12/0x17
[18758.534858] ---[ end trace 6362f5b39b85eb2d ]---
[18758.535840] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 54.297 msecs
[18758.537332] perf interrupt took too long (426767 > 10000), lowering kernel.perf_event_max_sample_rate to 12500

Few seconds later rcu craps itself..

[18801.941908] INFO: rcu_preempt detected stalls on CPUs/tasks:
[18801.942920] 	3: (3 GPs behind) idle=bf4/0/0 softirq=1597256/1597257 
[18801.943890] 	(detected by 0, t=6002 jiffies, g=763359, c=763358, q=0)
[18801.944843] Task dump for CPU 3:
[18801.945770] swapper/3       R  running task    14576     0      1 0x00200000
[18801.946706]  0000000342b6fe28 def23185c07e1b3d ffffe8ffff403518 0000000000000001
[18801.947629]  ffffffff81cb2000 0000000000000003 ffff880242b6fe78 ffffffff8166cb95
[18801.948557]  0000111242adb59f ffffffff81cb2070 ffff880242b6c000 ffffffff81d21ab0
[18801.949478] Call Trace:
[18801.950384]  [<ffffffff8166cb95>] ? cpuidle_enter_state+0x55/0x1c0
[18801.951303]  [<ffffffff8166cdb7>] ? cpuidle_enter+0x17/0x20
[18801.952211]  [<ffffffff810bf303>] ? cpu_startup_entry+0x423/0x4d0
[18801.953125]  [<ffffffff810314c3>] ? start_secondary+0x1a3/0x220

More of the same a minute later..

[18861.937095] INFO: rcu_preempt detected stalls on CPUs/tasks:
[18861.938050] 	1: (3 GPs behind) idle=89a/0/0 softirq=1498125/1498197 
[18861.938992] 	3: (4 GPs behind) idle=bf6/0/0 softirq=1597256/1597257 
[18861.939897] 	(detected by 0, t=6002 jiffies, g=763360, c=763359, q=0)
[18861.940812] Task dump for CPU 1:
[18861.941719] swapper/1       R  running task    14576     0      1 0x00200000
[18861.942649]  0000000142b5be28 8d64c020bc383a15 ffffe8ffff003518 0000000000000005
[18861.943584]  ffffffff81cb2000 0000000000000001 ffff880242b5be78 ffffffff8166cb95
[18861.944531]  0000112a29d67eee ffffffff81cb21d0 ffff880242b58000 ffffffff81d21ab0
[18861.945482] Call Trace:
[18861.946417]  [<ffffffff8166cb95>] ? cpuidle_enter_state+0x55/0x1c0
[18861.947368]  [<ffffffff8166cdb7>] ? cpuidle_enter+0x17/0x20
[18861.948315]  [<ffffffff810bf303>] ? cpu_startup_entry+0x423/0x4d0
[18861.949262]  [<ffffffff810314c3>] ? start_secondary+0x1a3/0x220
[18861.950214] Task dump for CPU 3:
[18861.951168] swapper/3       R  running task    14576     0      1 0x00200000
[18861.952143]  0000000342b6fe28 def23185c07e1b3d ffffe8ffff403518 0000000000000001
[18861.953117]  ffffffff81cb2000 0000000000000003 ffff880242b6fe78 ffffffff8166cb95
[18861.954099]  0000111c49e2522b ffffffff81cb2070 ffff880242b6c000 ffffffff81d21ab0
[18861.955082] Call Trace:
[18861.956054]  [<ffffffff8166cb95>] ? cpuidle_enter_state+0x55/0x1c0
[18861.957045]  [<ffffffff8166cdb7>] ? cpuidle_enter+0x17/0x20
[18861.958034]  [<ffffffff810bf303>] ? cpu_startup_entry+0x423/0x4d0
[18861.959019]  [<ffffffff810314c3>] ? start_secondary+0x1a3/0x220

CPU2 also gets 'stuck'.

[18889.800920] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 133s! [kworker/2:2:17558]
[18889.801789] CPU: 2 PID: 17558 Comm: kworker/2:2 Tainted: G        W      3.18.0+ #101 
[18889.802646] Workqueue: events free_obj_work
[18889.803490] task: ffff880231380000 ti: ffff8801b8c34000 task.ti: ffff8801b8c34000
[18889.804347] RIP: 0010:[<ffffffff81379d90>]  [<ffffffff81379d90>] memchr_inv+0x30/0x150
[18889.805219] RSP: 0018:ffff8801b8c37b08  EFLAGS: 00000287
[18889.806090] RAX: ffff8801cc9208a0 RBX: ffffffff81801b30 RCX: 000000000000005a
[18889.806973] RDX: 0000000000000008 RSI: 000000000000005a RDI: ffff8801cc92089c
[18889.807862] RBP: ffff8801b8c37b08 R08: ffff8801cc920898 R09: 000000000000005a
[18889.808756] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801b8c37af8
[18889.809650] R13: 0000000000000000 R14: ffffffffffffc000 R15: ffff8801b8c37b68
[18889.810545] FS:  0000000000000000(0000) GS:ffff880244400000(0000) knlGS:0000000000000000
[18889.811489] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18889.812384] CR2: 00007f1da06eaf30 CR3: 0000000001c11000 CR4: 00000000001407e0
[18889.813290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18889.814194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[18889.815096] Stack:
[18889.815990]  ffff8801b8c37b58 ffffffff811d5aeb ffffea0007324800 ffff8801cc920898
[18889.816912]  00000000000000ce ffff880243c0fa80 ffff8801cc920730 ffffea0007324800
[18889.817839]  00000000000000bb ffff880243c0fa80 ffff8801b8c37b98 ffffffff811d6e78
[18889.818764] Call Trace:
[18889.819681]  [<ffffffff811d5aeb>] check_bytes_and_report+0x3b/0x110
[18889.820608]  [<ffffffff811d6e78>] check_object+0xa8/0x250
[18889.821525]  [<ffffffff811d7328>] __free_slab+0x158/0x1b0
[18889.822444]  [<ffffffff811d73b9>] discard_slab+0x39/0x50
[18889.823355]  [<ffffffff817d133a>] __slab_free+0x2e2/0x320
[18889.824250]  [<ffffffff8138ceed>] ? free_obj_work+0x5d/0xa0
[18889.825130]  [<ffffffff8138cb27>] ? debug_smp_processor_id+0x17/0x20
[18889.826007]  [<ffffffff811d7fd2>] kmem_cache_free+0x262/0x280
[18889.826882]  [<ffffffff8138cefc>] ? free_obj_work+0x6c/0xa0
[18889.827749]  [<ffffffff8138cefc>] free_obj_work+0x6c/0xa0
[18889.828612]  [<ffffffff810942ad>] process_one_work+0x1fd/0x590
[18889.829468]  [<ffffffff81094227>] ? process_one_work+0x177/0x590
[18889.830321]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[18889.831215]  [<ffffffff8109475b>] worker_thread+0x11b/0x490
[18889.832054]  [<ffffffff81094640>] ? process_one_work+0x590/0x590
[18889.832896]  [<ffffffff81099f79>] kthread+0xf9/0x110
[18889.833738]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[18889.834580]  [<ffffffff81099e80>] ? kthread_create_on_node+0x250/0x250
[18889.835425]  [<ffffffff817deb6c>] ret_from_fork+0x7c/0xb0
[18889.836268]  [<ffffffff81099e80>] ? kthread_create_on_node+0x250/0x250
[18889.837107] Code: 48 89 e5 77 3e 40 0f b6 f6 85 d2 89 d0 89 f1 74 2b 40 3a 37 0f 85 f1 00 00 00 83 e8 01 48 8d 44 07 01 eb 0f 0f 1f 80 00 00 00 00 <3a> 0f 0f 85 d8 00 00 00 48 83 c7 01 48 39 c7 75 ef 31 c0 5d c3 
[18889.838955] sending NMI to other CPUs:
[18889.839845] NMI backtrace for cpu 0
[18889.840760] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W      3.18.0+ #101 
[18889.841697] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[18889.842634] RIP: 0010:[<ffffffff813e1b15>]  [<ffffffff813e1b15>] intel_idle+0xd5/0x180
[18889.843583] RSP: 0018:ffffffff81c03e28  EFLAGS: 00000046
[18889.844523] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[18889.845467] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[18889.846410] RBP: ffffffff81c03e58 R08: 000000008baf90f8 R09: 0000000000000000
[18889.847355] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[18889.848300] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[18889.849240] FS:  0000000000000000(0000) GS:ffff880244000000(0000) knlGS:0000000000000000
[18889.850183] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18889.851118] CR2: 00007f7c0148c443 CR3: 0000000001c11000 CR4: 00000000001407f0
[18889.852058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18889.852998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[18889.853923] Stack:
[18889.854837]  0000000081c03e58 1972379d2eec42e9 ffffe8fffee03518 0000000000000005
[18889.855780]  ffffffff81cb2000 0000000000000000 ffffffff81c03ea8 ffffffff8166cb95
[18889.856727]  00001130b01225d0 ffffffff81cb21d0 ffffffff81c00000 ffffffff81d21ab0
[18889.857674] Call Trace:
[18889.858606]  [<ffffffff8166cb95>] cpuidle_enter_state+0x55/0x1c0
[18889.859532]  [<ffffffff8166cdb7>] cpuidle_enter+0x17/0x20
[18889.860435]  [<ffffffff810bf303>] cpu_startup_entry+0x423/0x4d0
[18889.861332]  [<ffffffff817ca403>] rest_init+0xc3/0xd0
[18889.862219]  [<ffffffff817ca345>] ? rest_init+0x5/0xd0
[18889.863099]  [<ffffffff81f21ee0>] ? ftrace_init+0xa8/0x13b
[18889.863975]  [<ffffffff81f0304c>] start_kernel+0x49d/0x4be
[18889.864840]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[18889.865703]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[18889.866564]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[18889.867421] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[18889.869301] NMI backtrace for cpu 1
[18889.869312] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 29.456 msecs
[18889.871119] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W      3.18.0+ #101 
[18889.872056] task: ffff8802428bade0 ti: ffff880242b58000 task.ti: ffff880242b58000
[18889.872994] RIP: 0010:[<ffffffff813e1b15>]  [<ffffffff813e1b15>] intel_idle+0xd5/0x180
[18889.873934] RSP: 0018:ffff880242b5bdf8  EFLAGS: 00000046
[18889.874867] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[18889.875814] RDX: 0000000000000000 RSI: ffff880242b5bfd8 RDI: 0000000000000001
[18889.876751] RBP: ffff880242b5be28 R08: 000000008baf90f8 R09: 0000000000000000
[18889.877685] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[18889.878619] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880242b58000
[18889.879556] FS:  0000000000000000(0000) GS:ffff880244200000(0000) knlGS:0000000000000000
[18889.880479] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18889.881425] CR2: 00007f0f10335000 CR3: 0000000001c11000 CR4: 00000000001407e0
[18889.882361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18889.883292] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[18889.884219] Stack:
[18889.885136]  0000000142b5be28 8d64c020bc383a15 ffffe8ffff003518 0000000000000005
[18889.886087]  ffffffff81cb2000 0000000000000001 ffff880242b5be78 ffffffff8166cb95
[18889.887044]  00001130b011a3f3 ffffffff81cb21d0 ffff880242b58000 ffffffff81d21ab0
[18889.888005] Call Trace:
[18889.888970]  [<ffffffff8166cb95>] cpuidle_enter_state+0x55/0x1c0
[18889.889949]  [<ffffffff8166cdb7>] cpuidle_enter+0x17/0x20
[18889.890918]  [<ffffffff810bf303>] cpu_startup_entry+0x423/0x4d0
[18889.891884]  [<ffffffff810314c3>] start_secondary+0x1a3/0x220
[18889.892843] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[18889.894938] NMI backtrace for cpu 3
[18889.894946] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 55.092 msecs
[18889.896934] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W      3.18.0+ #101 
[18889.897911] task: ffff880242b65bc0 ti: ffff880242b6c000 task.ti: ffff880242b6c000
[18889.898890] RIP: 0010:[<ffffffff813e1b15>]  [<ffffffff813e1b15>] intel_idle+0xd5/0x180
[18889.899884] RSP: 0018:ffff880242b6fdf8  EFLAGS: 00000046
[18889.900889] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[18889.901859] RDX: 0000000000000000 RSI: ffff880242b6ffd8 RDI: 0000000000000003
[18889.902800] RBP: ffff880242b6fe28 R08: 000000008baf90f8 R09: 0000000000000000
[18889.903734] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[18889.904666] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880242b6c000
[18889.905599] FS:  0000000000000000(0000) GS:ffff880244600000(0000) knlGS:0000000000000000
[18889.906546] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18889.907502] CR2: 00007f0f10335000 CR3: 0000000001c11000 CR4: 00000000001407e0
[18889.908452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18889.909396] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[18889.910329] Stack:
[18889.911297]  0000000342b6fe28 def23185c07e1b3d ffffe8ffff403518 0000000000000005
[18889.912265]  ffffffff81cb2000 0000000000000003 ffff880242b6fe78 ffffffff8166cb95
[18889.913254]  00001130b01223e8 ffffffff81cb21d0 ffff880242b6c000 ffffffff81d21ab0
[18889.914291] Call Trace:
[18889.915361]  [<ffffffff8166cb95>] cpuidle_enter_state+0x55/0x1c0
[18889.916402]  [<ffffffff8166cdb7>] cpuidle_enter+0x17/0x20
[18889.917448]  [<ffffffff810bf303>] cpu_startup_entry+0x423/0x4d0
[18889.918434]  [<ffffffff810314c3>] start_secondary+0x1a3/0x220
[18889.919385] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[18889.921511] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 81.657 msecs
[18889.922586] perf interrupt took too long (423445 > 19841), lowering kernel.perf_event_max_sample_rate to 6300

Then things are idle. Curiously, periodically I continued to see these..

[19729.163946] perf interrupt took too long (420164 > 39062), lowering kernel.perf_event_max_sample_rate to 3200
[20944.488815] perf interrupt took too long (416920 > 78125), lowering kernel.perf_event_max_sample_rate to 1600
[22102.596107] perf interrupt took too long (413690 > 156250), lowering kernel.perf_event_max_sample_rate to 800

I've seen those messages a fair bit on other machines do, and they drive me nuts because there's no 'perf' being run. I think it means "NMI watchdog", but is worded badly. Still, it's curious that they appeared during what should have been idle time.

I'll reboot the box and give it another shot, and see what falls out in the morning.

Oh, worth noting: on this run, I gave Chris's idea of disabling usb serial console a try. I don't know if that's why I didn't see a total lockup this time or not..

Also this was from a run with just lsetxattr, 512 children..

Dave

Same deal again. It happened pretty quick after I'd gone to bed (typical).

[ 2754.509747] Clocksource tsc unstable (delta = -243594587656 ns)
[ 2754.519197] Switched to clocksource hpet
[ 2754.782940] INFO: rcu_preempt self-detected stall on CPU
[ 2754.782972] 	0: (1 GPs behind) idle=6ef/140000000000001/0 softirq=247160/247161 
[ 2754.782999] 	 (t=24343 jiffies g=104086 c=104085 q=0)
[ 2754.783022] Task dump for CPU 0:
[ 2754.783037] trinity-c387    R  running task    14016 13658  12780 0x00000008
[ 2754.783070]  ffff880222365bc0 000000005ed04994 ffff880244003d68 ffffffff810a8d46
[ 2754.783104]  ffffffff810a8cb2 0000000000000000 0000000000000001 0000000000000000
[ 2754.783138]  ffffffff81c51e40 0000000000000092 ffff880244003d88 ffffffff810acf4d
[ 2754.783171] Call Trace:
[ 2754.783184]  <IRQ>  [<ffffffff810a8d46>] sched_show_task+0x116/0x180
[ 2754.783215]  [<ffffffff810a8cb2>] ? sched_show_task+0x82/0x180
[ 2754.783239]  [<ffffffff810acf4d>] dump_cpu_task+0x3d/0x50
[ 2754.783261]  [<ffffffff810dc0c0>] rcu_dump_cpu_stacks+0x90/0xd0
[ 2754.783286]  [<ffffffff810e3db3>] rcu_check_callbacks+0x573/0x850
[ 2754.783311]  [<ffffffff8138cb43>] ? __this_cpu_preempt_check+0x13/0x20
[ 2754.783337]  [<ffffffff810ec043>] ? hrtimer_run_queues+0x43/0x130
[ 2754.783361]  [<ffffffff810ea5eb>] update_process_times+0x4b/0x80
[ 2754.783386]  [<ffffffff810fb2cc>] tick_sched_timer+0x4c/0x1b0
[ 2754.783409]  [<ffffffff810eb5db>] ? __run_hrtimer+0xbb/0x2e0
[ 2754.783432]  [<ffffffff810eb5db>] __run_hrtimer+0xbb/0x2e0
[ 2754.783454]  [<ffffffff810eb984>] ? hrtimer_interrupt+0x94/0x260
[ 2754.783478]  [<ffffffff810fb280>] ? tick_init_highres+0x20/0x20
[ 2754.783501]  [<ffffffff810eb9f7>] hrtimer_interrupt+0x107/0x260
[ 2754.783526]  [<ffffffff81033258>] local_apic_timer_interrupt+0x38/0x70
[ 2754.783552]  [<ffffffff817e16f5>] smp_apic_timer_interrupt+0x45/0x60
[ 2754.783578]  [<ffffffff817dfadf>] apic_timer_interrupt+0x6f/0x80
[ 2754.783600]  <EOI>  [<ffffffff810c541d>] ? lock_release_holdtime.part.24+0x9d/0x160
[ 2754.783634]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[ 2754.783659]  [<ffffffff8118524c>] ? __alloc_pages_nodemask+0x1ac/0xb60
[ 2754.783684]  [<ffffffff811cf4be>] ? alloc_pages_vma+0xee/0x1b0
[ 2754.783708]  [<ffffffff810ad575>] ? local_clock+0x25/0x30
[ 2754.783731]  [<ffffffff810c6e2c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[ 2754.783756]  [<ffffffff810a73e1>] ? get_parent_ip+0x11/0x50
[ 2754.783779]  [<ffffffff810c541d>] ? lock_release_holdtime.part.24+0x9d/0x160
[ 2754.784613]  [<ffffffff811cf4be>] alloc_pages_vma+0xee/0x1b0
[ 2754.785452]  [<ffffffff811aa21a>] ? do_wp_page+0xca/0x7d0
[ 2754.786305]  [<ffffffff811aa21a>] do_wp_page+0xca/0x7d0
[ 2754.787140]  [<ffffffff811acb6b>] handle_mm_fault+0x6cb/0xe90
[ 2754.787948]  [<ffffffff81042b20>] ? __do_page_fault+0x140/0x600
[ 2754.788748]  [<ffffffff81042b84>] __do_page_fault+0x1a4/0x600
[ 2754.789562]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[ 2754.790340]  [<ffffffff810c541d>] ? lock_release_holdtime.part.24+0x9d/0x160
[ 2754.791113]  [<ffffffff810a73e1>] ? get_parent_ip+0x11/0x50
[ 2754.791879]  [<ffffffff810a755b>] ? preempt_count_sub+0x7b/0x100
[ 2754.792646]  [<ffffffff8137fa0d>] ? trace_hardirqs_off_thunk+0x3a/0x3f
[ 2754.793413]  [<ffffffff81042fec>] do_page_fault+0xc/0x10
[ 2754.794176]  [<ffffffff817e0862>] page_fault+0x22/0x30
[ 2754.794938] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 2754.795736] 	Tasks blocked on level-0 rcu_node (CPUs 0-7):
[ 2754.796528] 	0: (1 GPs behind) idle=6ef/140000000000001/0 softirq=247160/247161 
[ 2754.797334] 	Tasks blocked on level-0 rcu_node (CPUs 0-7):
[ 2754.798153] 	(detected by 3, t=24343 jiffies, g=104086, c=104085, q=0)
[ 2754.798981] Task dump for CPU 0:
[ 2754.799797] trinity-c387    R  running task    14016 13658  12780 0x00000008
[ 2754.800630]  ffff880222365bc0 0000000000000246 0000000127e77e08 8000000000000865
[ 2754.801461]  ffff8802000000a9 800000008d044865 0000000000000000 ffff8802256d3c70
[ 2754.802288]  ffff880227e77e28 00000000000000a9 0000000000d8eff8 ffff880227e77f58
[ 2754.803109] Call Trace:
[ 2754.803929]  [<ffffffff81042b84>] ? __do_page_fault+0x1a4/0x600
[ 2754.804765]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[ 2754.805596]  [<ffffffff810c541d>] ? lock_release_holdtime.part.24+0x9d/0x160
[ 2754.806444]  [<ffffffff810a73e1>] ? get_parent_ip+0x11/0x50
[ 2754.807267]  [<ffffffff810a755b>] ? preempt_count_sub+0x7b/0x100
[ 2754.808088]  [<ffffffff8137fa0d>] ? trace_hardirqs_off_thunk+0x3a/0x3f
[ 2754.808920]  [<ffffffff81042fec>] ? do_page_fault+0xc/0x10
[ 2754.809742]  [<ffffffff817e0862>] ? page_fault+0x22/0x30
[ 2771.561356] ------------[ cut here ]------------
[ 2771.562079] WARNING: CPU: 0 PID: 13696 at kernel/watchdog.c:317 watchdog_overflow_callback+0xdd/0x130()
[ 2771.562879] Watchdog detected hard LOCKUP on cpu 0
[ 2771.562895] CPU: 0 PID: 13696 Comm: trinity-c425 Not tainted 3.18.0+ #101 
[ 2771.564490]  ffffffff81a66315 00000000fce35109 ffff880244005b88 ffffffff817d317e
[ 2771.565315]  0000000000110004 ffff880244005be0 ffff880244005bc8 ffffffff81078a01
[ 2771.566136]  0000000000000000 0000000000000000 0000000000000000 ffff880244005d30
[ 2771.566954] Call Trace:
[ 2771.567759]  <NMI>  [<ffffffff817d317e>] dump_stack+0x4f/0x7c
[ 2771.568584]  [<ffffffff81078a01>] warn_slowpath_common+0x81/0xa0
[ 2771.569405]  [<ffffffff81078a75>] warn_slowpath_fmt+0x55/0x70
[ 2771.570253]  [<ffffffff8112fea0>] ? restart_watchdog_hrtimer+0x60/0x60
[ 2771.571074]  [<ffffffff8112ff7d>] watchdog_overflow_callback+0xdd/0x130
[ 2771.571894]  [<ffffffff81173a7c>] __perf_event_overflow+0xac/0x2a0
[ 2771.572721]  [<ffffffff81019952>] ? x86_perf_event_set_period+0xe2/0x150
[ 2771.573551]  [<ffffffff81174644>] perf_event_overflow+0x14/0x20
[ 2771.574378]  [<ffffffff8101f479>] intel_pmu_handle_irq+0x209/0x410
[ 2771.575210]  [<ffffffff8101875b>] perf_event_nmi_handler+0x2b/0x50
[ 2771.576040]  [<ffffffff81007634>] nmi_handle+0xa4/0x1e0
[ 2771.576868]  [<ffffffff81007595>] ? nmi_handle+0x5/0x1e0
[ 2771.577698]  [<ffffffff81006de1>] ? print_context_stack+0xe1/0x100
[ 2771.578526]  [<ffffffff810079aa>] default_do_nmi+0x7a/0x1d0
[ 2771.579354]  [<ffffffff81007bb8>] do_nmi+0xb8/0xf0
[ 2771.580206]  [<ffffffff817e0c2a>] end_repeat_nmi+0x1e/0x2e
[ 2771.581023]  [<ffffffff817d0f85>] ? free_debug_processing+0x157/0x22a
[ 2771.581836]  [<ffffffff817d0f85>] ? free_debug_processing+0x157/0x22a
[ 2771.582644]  [<ffffffff81006de1>] ? print_context_stack+0xe1/0x100
[ 2771.583452]  [<ffffffff81006de1>] ? print_context_stack+0xe1/0x100
[ 2771.584253]  [<ffffffff81006de1>] ? print_context_stack+0xe1/0x100
[ 2771.585042]  <<EOE>>  [<ffffffff81005710>] dump_trace+0x140/0x370
[ 2771.585841]  [<ffffffff812005c6>] ? final_putname+0x26/0x50
[ 2771.586636]  [<ffffffff81013ecf>] save_stack_trace+0x2f/0x50
[ 2771.587430]  [<ffffffff811d4f20>] set_track+0x70/0x140
[ 2771.588217]  [<ffffffff817d0f85>] free_debug_processing+0x157/0x22a
[ 2771.589015]  [<ffffffff810c50fe>] ? put_lock_stats.isra.23+0xe/0x30
[ 2771.589815]  [<ffffffff817d10ad>] __slab_free+0x55/0x320
[ 2771.590636]  [<ffffffff8138e016>] ? debug_check_no_obj_freed+0x156/0x250
[ 2771.591442]  [<ffffffff81212294>] ? mntput+0x24/0x40
[ 2771.592242]  [<ffffffff811d7fd2>] kmem_cache_free+0x262/0x280
[ 2771.593036]  [<ffffffff812005c6>] ? final_putname+0x26/0x50
[ 2771.593831]  [<ffffffff812005c6>] final_putname+0x26/0x50
[ 2771.594622]  [<ffffffff81200869>] putname+0x29/0x40
[ 2771.595411]  [<ffffffff8120166e>] user_path_at_empty+0x6e/0xc0
[ 2771.596199]  [<ffffffff81212197>] ? mntput_no_expire+0x67/0x140
[ 2771.596986]  [<ffffffff81212135>] ? mntput_no_expire+0x5/0x140
[ 2771.597766]  [<ffffffff81207df6>] ? dput+0x56/0x190
[ 2771.598542]  [<ffffffff812016d1>] user_path_at+0x11/0x20
[ 2771.599311]  [<ffffffff812187ec>] path_setxattr+0x4c/0xe0
[ 2771.600097]  [<ffffffff81218a51>] SyS_lsetxattr+0x11/0x20
[ 2771.600848]  [<ffffffff817dec12>] system_call_fastpath+0x12/0x17
[ 2771.601598] ---[ end trace 7b78126c55dcb717 ]---
[ 2771.602404] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 40.989 msecs
[ 2771.603175] perf interrupt took too long (322423 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[ 3471.463812] perf interrupt took too long (319933 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[ 4563.539619] perf interrupt took too long (317460 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
[ 5676.723413] perf interrupt took too long (315015 > 19841), lowering kernel.perf_event_max_sample_rate to 6300
[ 6800.751151] perf interrupt took too long (312583 > 39062), lowering kernel.perf_event_max_sample_rate to 3200
[ 8056.882309] perf interrupt took too long (310176 > 78125), lowering kernel.perf_event_max_sample_rate to 1600
[ 9233.809073] perf interrupt took too long (307790 > 156250), lowering kernel.perf_event_max_sample_rate to 800

again, the box survived. Next run I'll try undoing Chris' idea of no serial,
and see if it wedges after the spew. After that, I'll do a no preempt run.

Dave

In this one, the CPU is at least non-idle. ;-)

Looks like standard scheduling-clock interrupt above this point.

If this was an acquisition rather than a release, I would suspect high lock contention. Could just be luck of the draw,I suppose.

Or am I missing something subtle here?

Thanx, Paul

[ . . . ]

Very strange. Both cpuidle_enter() and cpuidle_enter_state() should be within the idle loop, so that RCU should be ignoring this CPU. And the "idle=bf4/0/0" means that it really has marked itself as being idle from an RCU perspective. So I am guessing that the RCU grace-period kthread has not gotten a chance to run.

If you are willing to live a bit dangerously, could you please see if the (not for mainline) patch below clears this up?

Thanx, Paul

------------------------------------------------------------------------

rcu: Run grace-period kthreads at real-time priority

This is a experimental commit that attempts to better handle high-load situations.

index cecce1b13825..6db1f304157c 100644
@@ -677,7 +677,6 @@ config RCU_BOOST
 config RCU_KTHREAD_PRIO
 	int "Real-time priority to use for RCU worker threads"
 	range 1 99
-	depends on RCU_BOOST
 	default 1
 	help
 	  This option specifies the SCHED_FIFO priority value that will be

index 93bca38925a9..57fd8f5bd1ad 100644
@@ -156,6 +156,10 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
 static void invoke_rcu_core(void);
 static void invoke_rcu_callbacks(struct rcu_state *rsp, struct rcu_data *rdp);
 
+/* rcuc/rcub kthread realtime priority */
+static int kthread_prio = CONFIG_RCU_KTHREAD_PRIO;
+module_param(kthread_prio, int, 0644);
+
 /*
  * Track the rcutorture test sequence number and the update version
  * number within a given test.  The rcutorture_testseq is incremented

@@ -3631,15 +3635,19 @@ static int __init rcu_spawn_gp_kthread(void)
 	unsigned long flags;
 	struct rcu_node *rnp;
 	struct rcu_state *rsp;
+	struct sched_param sp;
 	struct task_struct *t;
 
 	rcu_scheduler_fully_active = 1;
 	for_each_rcu_flavor(rsp) {
-		t = kthread_run(rcu_gp_kthread, rsp, "%s", rsp->name);
+		t = kthread_create(rcu_gp_kthread, rsp, "%s", rsp->name);
 		BUG_ON(IS_ERR(t));
 		rnp = rcu_get_root(rsp);
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		rsp->gp_kthread = t;
+		sp.sched_priority = kthread_prio;
+		sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
+		wake_up_process(t);
 		raw_spin_unlock_irqrestore(&rnp->lock, flags);
 	}
 	rcu_spawn_nocb_kthreads();

index cf3b4d532379..564944964f14 100644
@@ -34,10 +34,6 @@
 
 #include "../locking/rtmutex_common.h"
 
-/* rcuc/rcub kthread realtime priority */
-static int kthread_prio = CONFIG_RCU_KTHREAD_PRIO;
-module_param(kthread_prio, int, 0644);
-
 /*
  * Control variables for per-CPU and per-rcu_node kthreads.  These
  * handle all flavors of RCU.

I'll try anything at this point, regardless of danger level :)

Dave

Something that's still making me wonder if it's some kind of hardware problem is the non-deterministic nature of this bug.
Take the example above, by limiting trinity to doing nothing but lsetxattr's. Why would the bug sometimes take 3-4 hours to shake out, and another run take just 45 minutes.

"different entropy" really shouldn't matter a huge amount here. Even if we end up picking different pathnames to pass in, it's the same source
(proc,sys,/dev). The other arguments are a crapshoot, but it seems
unlikely that it would matter hugely whatever values they are.

If it *is* a kernel bug, it's not going to be in lsetxattr, but rather some kind of scheduling or mm related thing that happens in some corner case when we're under extreme load. That I can drive up the loadavg with lsetxattr is I suspect just a symptom rather than the cause.

If enough callers pass in huge 'len' arguments, and an mmap that's big enough to cover that size, I could see that giving the kernel a lot of work to do.

Another thing I keep thinking is "well, how is this different from a forkbomb?". The user account I'm running under has no ulimit set on the maximum memory size for eg, but if that were the problem, surely I'd be seeing the oom-killer rather than lockups.

Dave

I'd expect it to be a race condition, though. Which can easily cause these kinds of issues, and the timing will be pretty random even if the load is very regular.

And we know that the scheduler has an integer overflow under Sasha's loads, although I didn't hear anything from Ingo and friends about it. Ingo/Peter, you were cc'd on that report, where at least one of the multiplcations in wake_affine() ended up overflowing..

Some scheduler thing that overflows only under heavy load, and screws up scheduling could easily account for the RCU thread thing. I see it *less* easily accounting for DaveJ's case, though, because the watchdog is running at RT priority, and the scheduler would have to screw up much more to then not schedule an RT task, but..

I'm also not sure if the bug ever happens with preemption disabled. Sasha, was that you who reported that you cannot reproduce it without preemption? It strikes me that there's a race condition in __cond_resched() wrt preemption, for example: we do

        __preempt_count_add(PREEMPT_ACTIVE);
        __schedule();
        __preempt_count_sub(PREEMPT_ACTIVE);

and in between the __schedule() and __preempt_count_sub(), if an interrupt comes in and wakes up some important process, it won't reschedule (because preemption is active), but then we enable preemption again and don't check whether we should reschedule (again), and we just go on our merry ways.

Now, I don't see how that could really matter for a long time - returning to user space will check need_resched, and sleeping will obviously force a reschedule anyway, so these kinds of races should at most delay things by just a tiny amount, but maybe there is some case where we screw up in a bigger way. So I do *not* believe that the one in __cond_resched() matters, but I'm giving it as an example of the kind of things that could go wrong.

Linus

After tomorrow, I'm not going to be in front of this machine until Wednesday, so I'll leave a no-preempt build running for the duration. Hopefully that will give us some clues. I might be able to log into it while travelling, so I'll provide updates where possible as long as it doesn't wedge solid.

Dave

If the machine has NOHZ and has a cpu bound userspace task, it could take quite a while before userspace would trigger a reschedule (at least if I've understood the comments on this thread properly)

David Lang

The thing is, we'd have to return to user space for that to happen. And when we do that, we check the "should we schedule" flag again. So races like this really shouldn't matter, but there could be something kind-of-similar that just ends up causing a wakeup to be delayed.

But it would need to be delayed for seconds (for the RCU threads) or for tens of seconds (for the watchdog) to matter.

Which just seems unlikely. Even the "very high load" thing shouldn't really matter, since while that could delay one particular thread being scheduled, it shouldn't delay the next "should we schedule" test. In fact, high load would normally be extected to make the next "should we schedule" come faster.

But this is where some load calculation overflow might screw things up, of course.

Linus

Furthermore there ought to be a scheduler tick active in that case - which won't be as fast as an immediate reschedule, but fast enough to beat the softlockup watchdog's threshold of 20 seconds or so.

That is why I think it would be interesting to examine how the locked up state looks like: is the system truly locked up, impossible to log in to, locks held but not released, etc., or is the lockup transient?

Also, the percpu watchdog threads are SCHED_FIFO:99, woken up through percpu hrtimers, which are not easy to delay through high SCHED_OTHER load.

Thanks,

Ingo

Dave, Sasha, if you guys are running CONFIG_NO_HZ_FULL=y and CONFIG_NO_HZ_FULL_ALL=y, please let me know. I am currently assuming that none of your CPUs are in NO_HZ_FULL mode. If this assumption is incorrect, there are some other pieces of RCU that I should be taking a hard look at.

Thanx, Paul

This is my no_hz related config:

$ grep NO_HZ .config
CONFIG_NO_HZ_COMMON=y
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ_FULL_ALL=y
CONFIG_NO_HZ_FULL_SYSIDLE=y
CONFIG_NO_HZ_FULL_SYSIDLE_SMALL=8
CONFIG_NO_HZ=y
CONFIG_RCU_FAST_NO_HZ=y

And from dmesg:

[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  RCU debugfs-based tracing is enabled.
[    0.000000]  Hierarchical RCU autobalancing is disabled.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  Additional per-CPU info printed with stalls.
[    0.000000]  RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=28.
[    0.000000]  RCU kthread priority: 1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=28
[    0.000000] NR_IRQS:524544 nr_irqs:648 16
[    0.000000] NO_HZ: Clearing 0 from nohz_full range for timekeeping
[    0.000000] NO_HZ: Full dynticks CPUs: 1-27.
[    0.000000]  Offload RCU callbacks from CPUs: 1-27.

Thanks,
Sasha

Just curious, if you disable NO_HZ_FULL_ALL, does the bug change?

Thanks,

Ingo

I've tried both nohz_full=4-27 and disabling CONFIG_NO_HZ_FULL_ALL altogether, but I'm still seeing the stall:

[  725.670017] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  725.670017]  0: (11 ticks this GP) idle=bbd/140000000000002/0 softirq=11529/11529 fqs=0 last_accelerate: 9d0e/a648, nonlazy_posted: 721357, ..
[  725.670017]  (detected by 16, t=2102 jiffies, g=9857, c=9856, q=2581)
[  725.670017] Task dump for CPU 0:
[  725.670017] kworker/0:1     S ffff8800633abde8 13016   520      2 0x10080008
[  725.670017]  ffffffffb03027a8 ffff880a70f24017 ffffffffb043ef40 ffff88005ffea310
[  725.670017]  0000000000000000 dfffe90000000000 0000000000000000 1ffffffff63bcdeb
[  725.670017]  ffff88006be15030 ffffffffb1de6f58 ffffffffffffff10 ffffffffb0301237
[  725.670017] Call Trace:
[  725.670017]  [<ffffffffb03027a8>] ? retint_restore_args+0x13/0x13
[  725.670017]  [<ffffffffb0301237>] ? _raw_spin_unlock_irq+0x57/0x200
[  725.670017]  [<ffffffffb0301203>] ? _raw_spin_unlock_irq+0x23/0x200
[  725.670017]  [<ffffffffa04630cb>] ? worker_thread+0x15b/0x1680
[  725.670017]  [<ffffffffb02effef>] ? __schedule+0xf6f/0x2fc0
[  725.670017]  [<ffffffffa0462f70>] ? process_one_work+0x1650/0x1650
[  725.670017]  [<ffffffffa047ae12>] ? kthread+0x1f2/0x2b0
[  725.670017]  [<ffffffffa047ac20>] ? kthread_worker_fn+0x6a0/0x6a0
[  725.670017]  [<ffffffffb03018bc>] ? ret_from_fork+0x7c/0xb0
[  725.670017]  [<ffffffffa047ac20>] ? kthread_worker_fn+0x6a0/0x6a0

Thanks,
Sasha

And again looping in workqueues, despite the cond_resched_rcu_qs() there. And the reason for that is that cond_resched_rcu_qs() currently only provides quiescent states for tasks RCU. I will put together something that makes it work for other RCU flavors.

Not that this is likely to do much about Dave Jones's lockup, but one thing at a time...

Thanx, Paul

And here is a patch for this purpose that passes moderate rcutorture testing. Please note that this patch is designed to help the case you were seeing, which is workqueues running indefinitely on the housekeeping CPU, namely CPU 0. If this also ends up happening on the other CPUs, I will need to get you another CPU-kicking mechanism.

Thanx, Paul

------------------------------------------------------------------------

rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors

Although cond_resched_rcu_qs() only applies to TASKS_RCU, it is used in places where it would be useful for it to apply to the normal RCU flavors, rcu_preempt, rcu_sched, and rcu_bh. This is especially the case for workloads that aggressively overload the system, particularly those that generate large numbers of RCU updates on systems running NO_HZ_FULL CPUs. This commit therefore communicates quiescent states from cond_resched_rcu_qs() to the normal RCU flavors.

Note that it is unfortunately necessary to leave the old ->passed_quiesce mechanism in place to allow quiescent states that apply to only one flavor to be recorded. (Yes, we could decrement ->rcu_qs_ctr_snap in that case, but that is not so good for debugging of RCU internals.)

index b63b9bb3bc0c..08651da15448 100644
@@ -56,14 +56,14 @@ rcuboost:
 
 The output of "cat rcu/rcu_preempt/rcudata" looks as follows:
 
-  0!c=30455 g=30456 pq=1 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
-  1!c=30719 g=30720 pq=1 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
-  2!c=30150 g=30151 pq=1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
-  3 c=31249 g=31250 pq=1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
-  4!c=29502 g=29503 pq=1 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
-  5 c=31201 g=31202 pq=1 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
-  6!c=30253 g=30254 pq=1 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
-  7 c=31178 g=31178 pq=1 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
+  0!c=30455 g=30456 pq=1/0 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
+  1!c=30719 g=30720 pq=1/0 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
+  2!c=30150 g=30151 pq=1/1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
+  3 c=31249 g=31250 pq=1/1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
+  4!c=29502 g=29503 pq=1/0 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
+  5 c=31201 g=31202 pq=1/0 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
+  6!c=30253 g=30254 pq=1/0 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
+  7 c=31178 g=31178 pq=1/0 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
 
 This file has one line per CPU, or eight for this 8-CPU system.
 The fields are as follows:

@@ -188,14 +188,14 @@ o	"ca" is the number of RCU callbacks that have been adopted by this
 Kernels compiled with CONFIG_RCU_BOOST=y display the following from
 /debug/rcu/rcu_preempt/rcudata:
 
-  0!c=12865 g=12866 pq=1 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
-  1 c=14407 g=14408 pq=1 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
-  2 c=14407 g=14408 pq=1 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
-  3 c=14407 g=14408 pq=1 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
-  4 c=14405 g=14406 pq=1 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
-  5!c=14168 g=14169 pq=1 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
-  6 c=14404 g=14405 pq=1 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
-  7 c=14407 g=14408 pq=1 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
+  0!c=12865 g=12866 pq=1/0 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
+  1 c=14407 g=14408 pq=1/0 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
+  2 c=14407 g=14408 pq=1/0 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
+  3 c=14407 g=14408 pq=1/0 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
+  4 c=14405 g=14406 pq=1/0 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
+  5!c=14168 g=14169 pq=1/0 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
+  6 c=14404 g=14405 pq=1/0 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
+  7 c=14407 g=14408 pq=1/0 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
 
 This is similar to the output discussed above, but contains the following
 additional fields:

index 386ba288084a..7f08f5079757 100644
@@ -331,6 +331,7 @@ static inline void rcu_init_nohz(void)
 extern struct srcu_struct tasks_rcu_exit_srcu;
 #define rcu_note_voluntary_context_switch(t) \
 	do { \
+		rcu_all_qs(); \
 		if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \
 			ACCESS_ONCE((t)->rcu_tasks_holdout) = false; \
 	} while (0)

index 984192160e9b..937edaeb150d 100644
@@ -186,7 +186,10 @@ static inline bool rcu_is_watching(void)
 	return true;
 }
 
-
 #endif /* #else defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
 
+static inline void rcu_all_qs(void)
+{
+}
+
 #endif /* __LINUX_RCUTINY_H */

index c0dd124e69ec..fbfd61d40e73 100644
@@ -100,4 +100,10 @@ extern int rcu_scheduler_active __read_mostly;
 
 bool rcu_is_watching(void);
 
+DECLARE_PER_CPU(unsigned long, rcu_qs_ctr);
+static inline void rcu_all_qs(void)
+{
+	this_cpu_inc(rcu_qs_ctr);
+}
+
 #endif /* __LINUX_RCUTREE_H */

index 57fd8f5bd1ad..58ea6bf55fd2 100644
@@ -219,6 +219,9 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
 #endif /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
 };
 
+DEFINE_PER_CPU_SHARED_ALIGNED(unsigned long, rcu_qs_ctr);
+EXPORT_PER_CPU_SYMBOL_GPL(rcu_qs_ctr);
+
 /*
  * Let the RCU core know that this CPU has gone through the scheduler,
  * which is a quiescent state.  This is called when the need for a

@@ -1630,6 +1633,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,
 		rdp->gpnum = rnp->gpnum;
 		trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpustart"));
 		rdp->passed_quiesce = 0;
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
 		zero_cpu_stall_ticks(rdp);
 		ACCESS_ONCE(rdp->gpwrap) = false;

@@ -2096,8 +2100,10 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)
 	rnp = rdp->mynode;
 	raw_spin_lock_irqsave(&rnp->lock, flags);
 	smp_mb__after_unlock_lock();
-	if (rdp->passed_quiesce == 0 || rdp->gpnum != rnp->gpnum ||
-	    rnp->completed == rnp->gpnum || rdp->gpwrap) {
+	if ((rdp->passed_quiesce == 0 &&
+	     rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr)) ||
+	    rdp->gpnum != rnp->gpnum || rnp->completed == rnp->gpnum ||
+	    rdp->gpwrap) {
 
 		/*
 		 * The grace period in which this quiescent state was

@@ -2106,6 +2112,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)
 		 * within the current grace period.
 		 */
 		rdp->passed_quiesce = 0;	/* need qs for new gp. */
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		raw_spin_unlock_irqrestore(&rnp->lock, flags);
 		return;
 	}

@@ -2150,7 +2157,8 @@ rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
 	 * Was there a quiescent state since the beginning of the grace
 	 * period? If no, then exit and wait for the next call.
 	 */
-	if (!rdp->passed_quiesce)
+	if (!rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr))
 		return;
 
 	/*

@@ -3206,9 +3214,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
 	if (rcu_scheduler_fully_active &&
-	    rdp->qs_pending && !rdp->passed_quiesce) {
+	    rdp->qs_pending && !rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr)) {
 		rdp->n_rp_qs_pending++;
-	} else if (rdp->qs_pending && rdp->passed_quiesce) {
+	} else if (rdp->qs_pending &&
+		   (rdp->passed_quiesce ||
+		    rdp->rcu_qs_ctr_snap != __this_cpu_read(rcu_qs_ctr))) {
 		rdp->n_rp_report_qs++;
 		return 1;
 	}

@@ -3542,6 +3553,7 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 			rdp->gpnum = rnp->completed;
 			rdp->completed = rnp->completed;
 			rdp->passed_quiesce = 0;
+			rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 			rdp->qs_pending = 0;
 			trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpuonl"));
 		}

index 4c156b43ff1a..f1dfc0bbb498 100644
@@ -252,6 +252,8 @@ struct rcu_data {
 					/*  in order to detect GP end. */
 	unsigned long	gpnum;		/* Highest gp number that this CPU */
 					/*  is aware of having started. */
+	unsigned long	rcu_qs_ctr_snap;/* Snapshot of rcu_qs_ctr to check */
+					/*  for rcu_all_qs() invocations. */
 	bool		passed_quiesce;	/* User-mode/idle loop etc. */
 	bool		qs_pending;	/* Core waits for quiesc state. */
 	bool		beenonline;	/* CPU online at least once. */

index 5cdc62e1beeb..4ec028a9987a 100644
@@ -115,11 +115,13 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)
 
 	if (!rdp->beenonline)
 		return;
-	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d qp=%d",
+	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d/%d qp=%d",
 		   rdp->cpu,
 		   cpu_is_offline(rdp->cpu) ? '!' : ' ',
 		   ulong2long(rdp->completed), ulong2long(rdp->gpnum),
-		   rdp->passed_quiesce, rdp->qs_pending);
+		   rdp->passed_quiesce,
+		   rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr),
+		   rdp->qs_pending);
 	seq_printf(m, " dt=%d/%llx/%d df=%lu",
 		   atomic_read(&rdp->dynticks->dynticks),
 		   rdp->dynticks->dynticks_nesting,

Does it depend on anything not currently in -next? My build fails with

kernel/rcu/tree.c: In function ‘rcu_report_qs_rdp’:
kernel/rcu/tree.c:2099:6: error: ‘struct rcu_data’ has no member named ‘gpwrap’

   rdp->gpwrap) {
      ^

On an unrelated subject, I've tried disabling preemption, and am seeing different stalls even when I have the testfiles fuzzing in trinity disabled (which means I'm not seeing hangs in the preempt case):

[  332.920142] INFO: rcu_sched self-detected stall on CPU
[  332.920142] 	19: (2099 ticks this GP) idle=f7d/140000000000001/0 softirq=21726/21726 fqs=1751
[  332.920142] 	 (t=2100 jiffies g=10656 c=10655 q=212427)
[  332.920142] Task dump for CPU 19:
[  332.920142] trinity-c522    R  running task    13544  9447   8279 0x1008000a
[  332.920142]  00000000000034e8 00000000000034e8 ffff8808a678a000 ffff8808bc203c18
[  332.920142]  ffffffff814b66f6 dfffe900000054de 0000000000000013 ffff8808bc215800
[  332.920142]  0000000000000013 ffffffff9cb5d018 dfffe90000000000 ffff8808bc203c48
[  332.920142] Call Trace:
[  332.920142] <IRQ> sched_show_task (kernel/sched/core.c:4541)
[  332.920142] dump_cpu_task (kernel/sched/core.c:8383)
[  332.940081] INFO: rcu_sched detected stalls on CPUs/tasks:
[  332.920142] rcu_dump_cpu_stacks (kernel/rcu/tree.c:1093)
[  332.920142] rcu_check_callbacks (kernel/rcu/tree.c:1199 kernel/rcu/tree.c:1261 kernel/rcu/tree.c:3194 kernel/rcu/tree.c:3254 kernel/rcu/tree.c:2507)
[  332.920142] update_process_times (./arch/x86/include/asm/preempt.h:22 kernel/time/timer.c:1386)
[  332.920142] tick_sched_timer (kernel/time/tick-sched.c:152 kernel/time/tick-sched.c:1128)
[  332.920142] __run_hrtimer (kernel/time/hrtimer.c:1216 (discriminator 3))
[  332.920142] ? tick_init_highres (kernel/time/tick-sched.c:1115)
[  332.920142] hrtimer_interrupt (include/linux/timerqueue.h:37 kernel/time/hrtimer.c:1275)
[  332.920142] ? acct_account_cputime (kernel/tsacct.c:168)
[  332.920142] local_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:921)
[  332.920142] smp_apic_timer_interrupt (./arch/x86/include/asm/apic.h:660 arch/x86/kernel/apic/apic.c:945)
[  332.920142] apic_timer_interrupt (arch/x86/kernel/entry_64.S:983)
[  332.920142] <EOI> ? retint_restore_args (arch/x86/kernel/entry_64.S:844)
[  332.920142] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:160 kernel/locking/spinlock.c:191)
[  332.920142] __debug_check_no_obj_freed (lib/debugobjects.c:713)
[  332.920142] debug_check_no_obj_freed (lib/debugobjects.c:727)
[  332.920142] free_pages_prepare (mm/page_alloc.c:829)
[  332.920142] free_hot_cold_page (mm/page_alloc.c:1496)
[  332.920142] __free_pages (mm/page_alloc.c:2982)
[  332.920142] ? __vunmap (mm/vmalloc.c:1459 (discriminator 2))
[  332.920142] __vunmap (mm/vmalloc.c:1455 (discriminator 2))
[  332.920142] vfree (mm/vmalloc.c:1500)
[  332.920142] SyS_init_module (kernel/module.c:2483 kernel/module.c:3359 kernel/module.c:3346)
[  332.920142] ia32_do_call (arch/x86/ia32/ia32entry.S:446)

Thanks,
Sasha

Indeed it does. Please see below for a port to current mainline.

More than 200K RCU callbacks queued. Impressive! ;-)

If this still appears after applying the below patch, I would be tempted to place a few cond_resched_rcu_qs() calls in the above three functions.

Thanx, Paul

------------------------------------------------------------------------

rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors

Although cond_resched_rcu_qs() only applies to TASKS_RCU, it is used in places where it would be useful for it to apply to the normal RCU flavors, rcu_preempt, rcu_sched, and rcu_bh. This is especially the case for workloads that aggressively overload the system, particularly those that generate large numbers of RCU updates on systems running NO_HZ_FULL CPUs. This commit therefore communicates quiescent states from cond_resched_rcu_qs() to the normal RCU flavors.

Note that it is unfortunately necessary to leave the old ->passed_quiesce mechanism in place to allow quiescent states that apply to only one flavor to be recorded. (Yes, we could decrement ->rcu_qs_ctr_snap in that case, but that is not so good for debugging of RCU internals.)

index b63b9bb3bc0c..08651da15448 100644
@@ -56,14 +56,14 @@ rcuboost:
 
 The output of "cat rcu/rcu_preempt/rcudata" looks as follows:
 
-  0!c=30455 g=30456 pq=1 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
-  1!c=30719 g=30720 pq=1 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
-  2!c=30150 g=30151 pq=1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
-  3 c=31249 g=31250 pq=1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
-  4!c=29502 g=29503 pq=1 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
-  5 c=31201 g=31202 pq=1 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
-  6!c=30253 g=30254 pq=1 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
-  7 c=31178 g=31178 pq=1 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
+  0!c=30455 g=30456 pq=1/0 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
+  1!c=30719 g=30720 pq=1/0 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
+  2!c=30150 g=30151 pq=1/1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
+  3 c=31249 g=31250 pq=1/1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
+  4!c=29502 g=29503 pq=1/0 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
+  5 c=31201 g=31202 pq=1/0 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
+  6!c=30253 g=30254 pq=1/0 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
+  7 c=31178 g=31178 pq=1/0 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
 
 This file has one line per CPU, or eight for this 8-CPU system.
 The fields are as follows:

@@ -188,14 +188,14 @@ o	"ca" is the number of RCU callbacks that have been adopted by this
 Kernels compiled with CONFIG_RCU_BOOST=y display the following from
 /debug/rcu/rcu_preempt/rcudata:
 
-  0!c=12865 g=12866 pq=1 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
-  1 c=14407 g=14408 pq=1 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
-  2 c=14407 g=14408 pq=1 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
-  3 c=14407 g=14408 pq=1 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
-  4 c=14405 g=14406 pq=1 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
-  5!c=14168 g=14169 pq=1 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
-  6 c=14404 g=14405 pq=1 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
-  7 c=14407 g=14408 pq=1 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
+  0!c=12865 g=12866 pq=1/0 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
+  1 c=14407 g=14408 pq=1/0 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
+  2 c=14407 g=14408 pq=1/0 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
+  3 c=14407 g=14408 pq=1/0 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
+  4 c=14405 g=14406 pq=1/0 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
+  5!c=14168 g=14169 pq=1/0 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
+  6 c=14404 g=14405 pq=1/0 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
+  7 c=14407 g=14408 pq=1/0 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
 
 This is similar to the output discussed above, but contains the following
 additional fields:

index ed4f5939a452..04eb366ab7fa 100644
@@ -331,6 +331,7 @@ static inline void rcu_init_nohz(void)
 extern struct srcu_struct tasks_rcu_exit_srcu;
 #define rcu_note_voluntary_context_switch(t) \
 	do { \
+		rcu_all_qs(); \
 		if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \
 			ACCESS_ONCE((t)->rcu_tasks_holdout) = false; \
 	} while (0)

index 0e5366200154..fabd3fad8516 100644
@@ -154,7 +154,10 @@ static inline bool rcu_is_watching(void)
 	return true;
 }
 
-
 #endif /* #else defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
 
+static inline void rcu_all_qs(void)
+{
+}
+
 #endif /* __LINUX_RCUTINY_H */

index 52953790dcca..3344783af1f2 100644
@@ -97,4 +97,10 @@ extern int rcu_scheduler_active __read_mostly;
 
 bool rcu_is_watching(void);
 
+DECLARE_PER_CPU(unsigned long, rcu_qs_ctr);
+static inline void rcu_all_qs(void)
+{
+	this_cpu_inc(rcu_qs_ctr);
+}
+
 #endif /* __LINUX_RCUTREE_H */

index 7680fc275036..38f0009b999b 100644
@@ -215,6 +215,9 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
 #endif /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
 };
 
+DEFINE_PER_CPU_SHARED_ALIGNED(unsigned long, rcu_qs_ctr);
+EXPORT_PER_CPU_SYMBOL_GPL(rcu_qs_ctr);
+
 /*
  * Let the RCU core know that this CPU has gone through the scheduler,
  * which is a quiescent state.  This is called when the need for a

@@ -1554,6 +1557,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,
 		rdp->gpnum = rnp->gpnum;
 		trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpustart"));
 		rdp->passed_quiesce = 0;
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
 		zero_cpu_stall_ticks(rdp);
 	}

@@ -2020,6 +2024,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)
 		 * within the current grace period.
 		 */
 		rdp->passed_quiesce = 0;	/* need qs for new gp. */
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		raw_spin_unlock_irqrestore(&rnp->lock, flags);
 		return;
 	}

@@ -2064,7 +2069,8 @@ rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
 	 * Was there a quiescent state since the beginning of the grace
 	 * period? If no, then exit and wait for the next call.
 	 */
-	if (!rdp->passed_quiesce)
+	if (!rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr))
 		return;
 
 	/*

@@ -3109,9 +3115,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
 	if (rcu_scheduler_fully_active &&
-	    rdp->qs_pending && !rdp->passed_quiesce) {
+	    rdp->qs_pending && !rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr)) {
 		rdp->n_rp_qs_pending++;
-	} else if (rdp->qs_pending && rdp->passed_quiesce) {
+	} else if (rdp->qs_pending &&
+		   (rdp->passed_quiesce ||
+		    rdp->rcu_qs_ctr_snap != __this_cpu_read(rcu_qs_ctr))) {
 		rdp->n_rp_report_qs++;
 		return 1;
 	}

@@ -3444,6 +3453,7 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 			rdp->gpnum = rnp->completed;
 			rdp->completed = rnp->completed;
 			rdp->passed_quiesce = 0;
+			rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 			rdp->qs_pending = 0;
 			trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpuonl"));
 		}

index 8e7b1843896e..c259a0bc0d97 100644
@@ -257,6 +257,8 @@ struct rcu_data {
 					/*  in order to detect GP end. */
 	unsigned long	gpnum;		/* Highest gp number that this CPU */
 					/*  is aware of having started. */
+	unsigned long	rcu_qs_ctr_snap;/* Snapshot of rcu_qs_ctr to check */
+					/*  for rcu_all_qs() invocations. */
 	bool		passed_quiesce;	/* User-mode/idle loop etc. */
 	bool		qs_pending;	/* Core waits for quiesc state. */
 	bool		beenonline;	/* CPU online at least once. */

index 5cdc62e1beeb..4ec028a9987a 100644
@@ -115,11 +115,13 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)
 
 	if (!rdp->beenonline)
 		return;
-	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d qp=%d",
+	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d/%d qp=%d",
 		   rdp->cpu,
 		   cpu_is_offline(rdp->cpu) ? '!' : ' ',
 		   ulong2long(rdp->completed), ulong2long(rdp->gpnum),
-		   rdp->passed_quiesce, rdp->qs_pending);
+		   rdp->passed_quiesce,
+		   rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr),
+		   rdp->qs_pending);
 	seq_printf(m, " dt=%d/%llx/%d df=%lu",
 		   atomic_read(&rdp->dynticks->dynticks),
 		   rdp->dynticks->dynticks_nesting,

With the patch:

[  620.340045] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  620.341087]  (detected by 22, t=8407 jiffies, g=10452, c=10451, q=3622)
[  620.342154] All QSes seen, last rcu_preempt kthread activity 4294990929/4294999330, jiffies_till_next_fqs=1
[  643.710049] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  643.710073] INFO: rcu_sched detected stalls on CPUs/tasks:
[  643.710093]  0: (6 ticks this GP) idle=bd5/140000000000002/0 softirq=12421/12421 fqs=0 last_accelerate: 5283/8643, nonlazy_posted: 643841, ..
[  643.710110]  (detected by 1, t=2102 jiffies, g=-292, c=-293, q=0)
[  643.710112] Task dump for CPU 0:
[  643.710129] kworker/0:1     R  running task    13016   628      2 0x10080008
[  643.710148] Workqueue: events vmstat_update
[  643.710156]  ffffffffb0301dc4 ffff88006be15000 ffff880060ba1c70 0000000000000000
[  643.710161]  ffff88006be10680 ffff8800633efde8 ffffffffa0461f1b ffff88006a776000
[  643.710166]  ffff880060ba1cb8 ffff880060ba1c78 ffff880060ba1c80 ffff880060ba1c90
[  643.710168] Call Trace:
[  643.710181]  [<ffffffffb0301dc4>] ? _raw_spin_unlock_irq+0x64/0x200
[  643.710191]  [<ffffffffa0461f1b>] ? process_one_work+0x5fb/0x1660
[  643.710197]  [<ffffffffa0463545>] ? worker_thread+0x5c5/0x1680
[  643.710205]  [<ffffffffb02f0b6f>] ? __schedule+0xf6f/0x2fc0
[  643.710211]  [<ffffffffa0462f80>] ? process_one_work+0x1660/0x1660
[  643.710216]  [<ffffffffa047ae22>] ? kthread+0x1f2/0x2b0
[  643.710221]  [<ffffffffa047ac30>] ? kthread_worker_fn+0x6a0/0x6a0
[  643.710226]  [<ffffffffb030243c>] ? ret_from_fork+0x7c/0xb0
[  643.710233]  [<ffffffffa047ac30>] ? kthread_worker_fn+0x6a0/0x6a0
[  643.711486]
[  643.711486]  (detected by 22, t=2104 jiffies, g=10453, c=10452, q=1570)
[  643.711486] All QSes seen, last rcu_preempt kthread activity 4294999565/4295001669, jiffies_till_next_fqs=1

Thanks,
Sasha

OK, 8401 jiffies since the grace-period kthread ran. This is without the "Run grace-period kthreads at real-time priority" patch?

But this one is real.

Which in theory should have been addressed by the "Make cond_resched_rcu_qs() apply to normal RCU flavors" patch, given that this is CPU 0, which should be taking scheduling clock interrupts. Well, I guess that theory and practice are only the same in theory. :-/

Will dig into it more.

Thanx, Paul

And maybe it would help if I did the CONFIG_TASKS_RCU=n case as well as the CONFIG_TASKS_RCU=y case. Please see below for an updated patch.

Thanx, Paul

------------------------------------------------------------------------

rcu: Make cond_resched_rcu_qs() apply to normal RCU flavors

Although cond_resched_rcu_qs() only applies to TASKS_RCU, it is used in places where it would be useful for it to apply to the normal RCU flavors, rcu_preempt, rcu_sched, and rcu_bh. This is especially the case for workloads that aggressively overload the system, particularly those that generate large numbers of RCU updates on systems running NO_HZ_FULL CPUs. This commit therefore communicates quiescent states from cond_resched_rcu_qs() to the normal RCU flavors.

Note that it is unfortunately necessary to leave the old ->passed_quiesce mechanism in place to allow quiescent states that apply to only one flavor to be recorded. (Yes, we could decrement ->rcu_qs_ctr_snap in that case, but that is not so good for debugging of RCU internals.)

index b63b9bb3bc0c..08651da15448 100644
@@ -56,14 +56,14 @@ rcuboost:
 
 The output of "cat rcu/rcu_preempt/rcudata" looks as follows:
 
-  0!c=30455 g=30456 pq=1 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
-  1!c=30719 g=30720 pq=1 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
-  2!c=30150 g=30151 pq=1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
-  3 c=31249 g=31250 pq=1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
-  4!c=29502 g=29503 pq=1 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
-  5 c=31201 g=31202 pq=1 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
-  6!c=30253 g=30254 pq=1 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
-  7 c=31178 g=31178 pq=1 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
+  0!c=30455 g=30456 pq=1/0 qp=1 dt=126535/140000000000000/0 df=2002 of=4 ql=0/0 qs=N... b=10 ci=74572 nci=0 co=1131 ca=716
+  1!c=30719 g=30720 pq=1/0 qp=0 dt=132007/140000000000000/0 df=1874 of=10 ql=0/0 qs=N... b=10 ci=123209 nci=0 co=685 ca=982
+  2!c=30150 g=30151 pq=1/1 qp=1 dt=138537/140000000000000/0 df=1707 of=8 ql=0/0 qs=N... b=10 ci=80132 nci=0 co=1328 ca=1458
+  3 c=31249 g=31250 pq=1/1 qp=0 dt=107255/140000000000000/0 df=1749 of=6 ql=0/450 qs=NRW. b=10 ci=151700 nci=0 co=509 ca=622
+  4!c=29502 g=29503 pq=1/0 qp=1 dt=83647/140000000000000/0 df=965 of=5 ql=0/0 qs=N... b=10 ci=65643 nci=0 co=1373 ca=1521
+  5 c=31201 g=31202 pq=1/0 qp=1 dt=70422/0/0 df=535 of=7 ql=0/0 qs=.... b=10 ci=58500 nci=0 co=764 ca=698
+  6!c=30253 g=30254 pq=1/0 qp=1 dt=95363/140000000000000/0 df=780 of=5 ql=0/0 qs=N... b=10 ci=100607 nci=0 co=1414 ca=1353
+  7 c=31178 g=31178 pq=1/0 qp=0 dt=91536/0/0 df=547 of=4 ql=0/0 qs=.... b=10 ci=109819 nci=0 co=1115 ca=969
 
 This file has one line per CPU, or eight for this 8-CPU system.
 The fields are as follows:

@@ -188,14 +188,14 @@ o	"ca" is the number of RCU callbacks that have been adopted by this
 Kernels compiled with CONFIG_RCU_BOOST=y display the following from
 /debug/rcu/rcu_preempt/rcudata:
 
-  0!c=12865 g=12866 pq=1 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
-  1 c=14407 g=14408 pq=1 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
-  2 c=14407 g=14408 pq=1 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
-  3 c=14407 g=14408 pq=1 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
-  4 c=14405 g=14406 pq=1 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
-  5!c=14168 g=14169 pq=1 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
-  6 c=14404 g=14405 pq=1 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
-  7 c=14407 g=14408 pq=1 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
+  0!c=12865 g=12866 pq=1/0 qp=1 dt=83113/140000000000000/0 df=288 of=11 ql=0/0 qs=N... kt=0/O ktl=944 b=10 ci=60709 nci=0 co=748 ca=871
+  1 c=14407 g=14408 pq=1/0 qp=0 dt=100679/140000000000000/0 df=378 of=7 ql=0/119 qs=NRW. kt=0/W ktl=9b6 b=10 ci=109740 nci=0 co=589 ca=485
+  2 c=14407 g=14408 pq=1/0 qp=0 dt=105486/0/0 df=90 of=9 ql=0/89 qs=NRW. kt=0/W ktl=c0c b=10 ci=83113 nci=0 co=533 ca=490
+  3 c=14407 g=14408 pq=1/0 qp=0 dt=107138/0/0 df=142 of=8 ql=0/188 qs=NRW. kt=0/W ktl=b96 b=10 ci=121114 nci=0 co=426 ca=290
+  4 c=14405 g=14406 pq=1/0 qp=1 dt=50238/0/0 df=706 of=7 ql=0/0 qs=.... kt=0/W ktl=812 b=10 ci=34929 nci=0 co=643 ca=114
+  5!c=14168 g=14169 pq=1/0 qp=0 dt=45465/140000000000000/0 df=161 of=11 ql=0/0 qs=N... kt=0/O ktl=b4d b=10 ci=47712 nci=0 co=677 ca=722
+  6 c=14404 g=14405 pq=1/0 qp=0 dt=59454/0/0 df=94 of=6 ql=0/0 qs=.... kt=0/W ktl=e57 b=10 ci=55597 nci=0 co=701 ca=811
+  7 c=14407 g=14408 pq=1/0 qp=1 dt=68850/0/0 df=31 of=8 ql=0/0 qs=.... kt=0/W ktl=14bd b=10 ci=77475 nci=0 co=508 ca=1042
 
 This is similar to the output discussed above, but contains the following
 additional fields:

index ed4f5939a452..aa894d4fe375 100644
@@ -331,12 +331,13 @@ static inline void rcu_init_nohz(void)
 extern struct srcu_struct tasks_rcu_exit_srcu;
 #define rcu_note_voluntary_context_switch(t) \
 	do { \
+		rcu_all_qs(); \
 		if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \
 			ACCESS_ONCE((t)->rcu_tasks_holdout) = false; \
 	} while (0)
 #else /* #ifdef CONFIG_TASKS_RCU */
 #define TASKS_RCU(x) do { } while (0)
-#define rcu_note_voluntary_context_switch(t)	do { } while (0)
+#define rcu_note_voluntary_context_switch(t)	do { rcu_do_qs(); } while (0)
 #endif /* #else #ifdef CONFIG_TASKS_RCU */
 
 /**

index 0e5366200154..fabd3fad8516 100644
@@ -154,7 +154,10 @@ static inline bool rcu_is_watching(void)
 	return true;
 }
 
-
 #endif /* #else defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
 
+static inline void rcu_all_qs(void)
+{
+}
+
 #endif /* __LINUX_RCUTINY_H */

index 52953790dcca..3344783af1f2 100644
@@ -97,4 +97,10 @@ extern int rcu_scheduler_active __read_mostly;
 
 bool rcu_is_watching(void);
 
+DECLARE_PER_CPU(unsigned long, rcu_qs_ctr);
+static inline void rcu_all_qs(void)
+{
+	this_cpu_inc(rcu_qs_ctr);
+}
+
 #endif /* __LINUX_RCUTREE_H */

index 7680fc275036..38f0009b999b 100644
@@ -215,6 +215,9 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
 #endif /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
 };
 
+DEFINE_PER_CPU_SHARED_ALIGNED(unsigned long, rcu_qs_ctr);
+EXPORT_PER_CPU_SYMBOL_GPL(rcu_qs_ctr);
+
 /*
  * Let the RCU core know that this CPU has gone through the scheduler,
  * which is a quiescent state.  This is called when the need for a

@@ -1554,6 +1557,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,
 		rdp->gpnum = rnp->gpnum;
 		trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpustart"));
 		rdp->passed_quiesce = 0;
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);
 		zero_cpu_stall_ticks(rdp);
 	}

@@ -2020,6 +2024,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)
 		 * within the current grace period.
 		 */
 		rdp->passed_quiesce = 0;	/* need qs for new gp. */
+		rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 		raw_spin_unlock_irqrestore(&rnp->lock, flags);
 		return;
 	}

@@ -2064,7 +2069,8 @@ rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp)
 	 * Was there a quiescent state since the beginning of the grace
 	 * period? If no, then exit and wait for the next call.
 	 */
-	if (!rdp->passed_quiesce)
+	if (!rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr))
 		return;
 
 	/*

@@ -3109,9 +3115,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
 	if (rcu_scheduler_fully_active &&
-	    rdp->qs_pending && !rdp->passed_quiesce) {
+	    rdp->qs_pending && !rdp->passed_quiesce &&
+	    rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr)) {
 		rdp->n_rp_qs_pending++;
-	} else if (rdp->qs_pending && rdp->passed_quiesce) {
+	} else if (rdp->qs_pending &&
+		   (rdp->passed_quiesce ||
+		    rdp->rcu_qs_ctr_snap != __this_cpu_read(rcu_qs_ctr))) {
 		rdp->n_rp_report_qs++;
 		return 1;
 	}

@@ -3444,6 +3453,7 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)
 			rdp->gpnum = rnp->completed;
 			rdp->completed = rnp->completed;
 			rdp->passed_quiesce = 0;
+			rdp->rcu_qs_ctr_snap = __this_cpu_read(rcu_qs_ctr);
 			rdp->qs_pending = 0;
 			trace_rcu_grace_period(rsp->name, rdp->gpnum, TPS("cpuonl"));
 		}

index 8e7b1843896e..c259a0bc0d97 100644
@@ -257,6 +257,8 @@ struct rcu_data {
 					/*  in order to detect GP end. */
 	unsigned long	gpnum;		/* Highest gp number that this CPU */
 					/*  is aware of having started. */
+	unsigned long	rcu_qs_ctr_snap;/* Snapshot of rcu_qs_ctr to check */
+					/*  for rcu_all_qs() invocations. */
 	bool		passed_quiesce;	/* User-mode/idle loop etc. */
 	bool		qs_pending;	/* Core waits for quiesc state. */
 	bool		beenonline;	/* CPU online at least once. */

index 5cdc62e1beeb..4ec028a9987a 100644
@@ -115,11 +115,13 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)
 
 	if (!rdp->beenonline)
 		return;
-	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d qp=%d",
+	seq_printf(m, "%3d%cc=%ld g=%ld pq=%d/%d qp=%d",
 		   rdp->cpu,
 		   cpu_is_offline(rdp->cpu) ? '!' : ' ',
 		   ulong2long(rdp->completed), ulong2long(rdp->gpnum),
-		   rdp->passed_quiesce, rdp->qs_pending);
+		   rdp->passed_quiesce,
+		   rdp->rcu_qs_ctr_snap == __this_cpu_read(rcu_qs_ctr),
+		   rdp->qs_pending);
 	seq_printf(m, " dt=%d/%llx/%d df=%lu",
 		   atomic_read(&rdp->dynticks->dynticks),
 		   rdp->dynticks->dynticks_nesting,

I do have CONFIG_TASKS_RCU=y

Thanks,
Sasha

OK, back to the drawing board...

Thanx, Paul

Thank you, Sasha. Looks like I have a few more places to take a hard look at, then!

Thanx, Paul

And one effect of CONFIG_NO_HZ_FULL=y and CONFIG_NO_HZ_FULL_ALL=y is that all the grace-period kthread are pinned to CPU 0. In addition, all of CPUs 1-27 are offloaded, and all of the resulting rcuo kthreads (which invoke RCU callbacks) are also pinned to CPU 0. If you are then running a heavy in-kernel workload that generates lots of callbacks, it is easy to imagine that CPU 0 might be getting overloaded. After all, this combination of Kconfig parameters was designed for HPC and real-time workloads that spend most of their time in userspace.

If you are allowing your workload to run on CPU 0, it would be very interesting to see what happens if you restrict your workload to run on CPUs 1-27.

Alternatively, your could boot with nohz_full=2-27 (or maybe even nohz_full=4-27). This will override CONFIG_NO_HZ_FULL_ALL=y and will provide two (or four with 4-27) housekeeping CPUs that are available to run things like RCU grace-period kthreads and RCU callback processing. This might allow RCU to get the CPU bandwidth it needs despite competition from your workload.

Thanx, Paul

Indeed, that's a really good find regardless of whether it's the source of these lockups - the (untested) patch below ought to cure that.

(as you later note) NOHZ is somewhat special in this regard, because there we try really hard not to run anything periodically, so a lost reschedule will matter more.

But ... I'd be surprised if this patch made a difference: it should normally not be possible to go idle with tasks on the runqueue (even with this bug present), and with at least one busy task on the CPU we get the regular scheduler tick which ought to hide such latencies.

It's nevertheless a good thing to fix, I'm just not sure it's the root cause of the observed lockup here.

Thanks,

Ingo

--

index bb398c0c5f08..532809aa0544 100644
@@ -4207,6 +4207,8 @@ static void __cond_resched(void)
 	__preempt_count_add(PREEMPT_ACTIVE);
 	__schedule();
 	__preempt_count_sub(PREEMPT_ACTIVE);
+	if (need_resched())
+		__schedule();
 }
 
 int __sched _cond_resched(void)

So I like your patch, but quite frankly, can we go one step further?

Look at the callers of __schedule().

EVERY SINGLE ONE now has that loop around it that goes along the lines of

   do {
      .. disable preemption somehow ..
      __schedule();
      ...enable preemption without scheduling ..
   } while (need_resced());

except for one - the regular "schedule()" function.

Furthermore, look inside __schedule() itself: it has the same loop, except with a count of one.

So I would suggest going the extra mile, and
- remove the loop from __schedule() itself
- add the same loop as everywhere else to "schedule()"

IOW, just make this "you have to loop and disable preemption" thing be a rule that __schedule() can depend on.

Linus

That sounds like a good idea. Unless the loop inside __schedule() is very frequent and sensitive enough to show visible overhead if we force it to pass through the preemp_count_add/sub() and local_irq_*() operations in the preempt_schedule_*() functions.

I suspect it's not, so I'm cooking that patch.

Right. I'm doing that too.

Ok. It would be nice if we could have a common function that does the loop and PREEMPT_ACTIVE increments. But the variable code is inside that loop so that's only factorizable with a function pointer (no-go in that fast-path) or a macro that would make things even worse and ugly.

So I think I'll just keep all those loops explicit.

Thanks.

Just to make sure, is there any other wake_affine report other than the one in this thread? (I tried a wake_affine full text search on my inbox and didn't find anything that appeared relevant.)

Yeah, the RT scheduler is harder (but not impossible) to confuse due to its simplicity, but scheduler counts overflowing could definitely cause all sorts of trouble and make debugging harder, so we want to fix it regardless of its likelihood of causing lockups.

Thanks,

Ingo

Found the report from Sasha:

sched: odd values for effective load calculations

right?

Thanks,

Ingo

Yup, that's the one.

Thanks,
Sasha

Bah, so I see some watchdog traces with preemption off, and that then taints the kernel, and the fuzzing stops. I'll hack something up so it ignores the taint and keeps going. All I really care about here is the "machine hangs completely" case, which the trace below didn't hit..

(back to fuzzing almost everything, not just lsetxattr btw)

[34917.468470] WARNING: CPU: 1 PID: 9226 at kernel/watchdog.c:317 watchdog_overflow_callback+0x9c/0xd0()
[34917.468500] Watchdog detected hard LOCKUP on cpu 1
[34917.468516] CPU: 1 PID: 9226 Comm: trinity-c107 Not tainted 3.18.0+ #102 
[34917.468542] [loadavg: 155.62 139.10 140.12 10/405 11756]
[34917.468559]  ffffffff81a65d99 000000005606cf60 ffff880244205b98 ffffffff817c4f75
[34917.468591]  ffffffff810cd5a1 ffff880244205bf0 ffff880244205bd8 ffffffff81077cb1
[34917.468623]  ffff880244205bd8 ffff880243c55388 0000000000000000 ffff880244205d30
[34917.468655] Call Trace:
[34917.468667]  <NMI>  [<ffffffff817c4f75>] dump_stack+0x4e/0x68
[34917.468696]  [<ffffffff810cd5a1>] ? console_unlock+0x1f1/0x4e0
[34917.468718]  [<ffffffff81077cb1>] warn_slowpath_common+0x81/0xa0
[34917.468740]  [<ffffffff81077d25>] warn_slowpath_fmt+0x55/0x70
[34917.468761]  [<ffffffff817c3710>] ? __slab_alloc+0x3c4/0x58f
[34917.468783]  [<ffffffff8112bce0>] ? restart_watchdog_hrtimer+0x60/0x60
[34917.468806]  [<ffffffff8112bd7c>] watchdog_overflow_callback+0x9c/0xd0
[34917.468830]  [<ffffffff8116ebed>] __perf_event_overflow+0x9d/0x2a0
[34917.468856]  [<ffffffff8116d7c3>] ? perf_event_update_userpage+0x103/0x180
[34917.469785]  [<ffffffff8116d6c0>] ? perf_event_task_disable+0x90/0x90
[34917.470705]  [<ffffffff8116f7c4>] perf_event_overflow+0x14/0x20
[34917.471632]  [<ffffffff8101e749>] intel_pmu_handle_irq+0x1f9/0x3f0
[34917.472553]  [<ffffffff81017cbb>] perf_event_nmi_handler+0x2b/0x50
[34917.473459]  [<ffffffff81007330>] nmi_handle+0xc0/0x1b0
[34917.474355]  [<ffffffff81007275>] ? nmi_handle+0x5/0x1b0
[34917.475245]  [<ffffffff8100761a>] default_do_nmi+0x4a/0x140
[34917.476128]  [<ffffffff810077d0>] do_nmi+0xc0/0x100
[34917.477012]  [<ffffffff817d237a>] end_repeat_nmi+0x1e/0x2e
[34917.477902]  [<ffffffff81383a37>] ? debug_check_no_obj_freed+0xe7/0x250
[34917.478788]  [<ffffffff81383a37>] ? debug_check_no_obj_freed+0xe7/0x250
[34917.479660]  [<ffffffff81383a37>] ? debug_check_no_obj_freed+0xe7/0x250
[34917.480523]  <<EOE>>  [<ffffffff8117b87f>] free_pages_prepare+0x1af/0x240
[34917.481396]  [<ffffffff8117dd51>] __free_pages_ok+0x21/0x100
[34917.482270]  [<ffffffff8117de4b>] free_compound_page+0x1b/0x20
[34917.483144]  [<ffffffff81184d23>] __put_compound_page+0x23/0x30
[34917.484022]  [<ffffffff81184da8>] put_compound_page+0x48/0x2e0
[34917.484895]  [<ffffffff811854d9>] release_pages+0x239/0x270
[34917.485768]  [<ffffffff811b9d1d>] free_pages_and_swap_cache+0x8d/0xa0
[34917.486648]  [<ffffffff811a25d4>] tlb_flush_mmu_free+0x34/0x60
[34917.487530]  [<ffffffff811a4021>] unmap_single_vma+0x6d1/0x900
[34917.488405]  [<ffffffff811a4d51>] unmap_vmas+0x51/0xa0
[34917.489277]  [<ffffffff811ae175>] exit_mmap+0xe5/0x1a0
[34917.490143]  [<ffffffff81074d6b>] mmput+0x6b/0x100
[34917.490995]  [<ffffffff810792ae>] do_exit+0x29e/0xb60
[34917.491823]  [<ffffffff8107abfc>] do_group_exit+0x4c/0xc0
[34917.492645]  [<ffffffff8107ac84>] SyS_exit_group+0x14/0x20
[34917.493462]  [<ffffffff817d0589>] tracesys_phase2+0xd4/0xd9
[34917.494268] ---[ end trace c48441b18b6523a2 ]---
[34917.495171] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 26.690 msecs
[34917.496031] perf interrupt took too long (211387 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[34967.056860] INFO: rcu_sched detected stalls on CPUs/tasks:
[34967.057898] 	1: (0 ticks this GP) idle=b4c/0/0 softirq=2168971/2168971 
[34967.058900] 	(detected by 2, t=6002 jiffies, g=1058044, c=1058043, q=0)
[34967.059867] Task dump for CPU 1:
[34967.060827] swapper/1       R  running task    14576     0      1 0x00200000
[34967.061802]  0000000142b4be38 4a979bec19cdc3d2 ffffe8ffff003200 0000000000000003
[34967.062786]  ffffffff81cb1b80 0000000000000001 ffff880242b4be88 ffffffff8165ff65
[34967.063759]  00001fcecca713d2 ffffffff81cb1ca0 ffffffff81cb1b80 ffffffff81d215b0
[34967.064721] Call Trace:
[34967.065649]  [<ffffffff8165ff65>] ? cpuidle_enter_state+0x55/0x190
[34967.066570]  [<ffffffff81660157>] ? cpuidle_enter+0x17/0x20
[34967.067498]  [<ffffffff810bd6c5>] ? cpu_startup_entry+0x355/0x410
[34967.068425]  [<ffffffff8103016a>] ? start_secondary+0x1aa/0x230
[35027.731690] INFO: rcu_sched detected stalls on CPUs/tasks:
[35027.732701] 	1: (0 ticks this GP) idle=b82/0/0 softirq=2168971/2168971 
[35027.733652] 	(detected by 2, t=6002 jiffies, g=1058047, c=1058046, q=0)
[35027.734593] Task dump for CPU 1:
[35027.735514] swapper/1       R  running task    14576     0      1 0x00200000
[35027.736445]  0000000142b4be38 4a979bec19cdc3d2 ffffe8ffff003200 0000000000000004
[35027.737369]  ffffffff81cb1b80 0000000000000001 ffff880242b4be88 ffffffff8165ff65
[35027.738285]  00001fde808fc8c8 ffffffff81cb1cf8 ffffffff81cb1b80 ffffffff81d215b0
[35027.739206] Call Trace:
[35027.740114]  [<ffffffff8165ff65>] ? cpuidle_enter_state+0x55/0x190
[35027.741032]  [<ffffffff81660157>] ? cpuidle_enter+0x17/0x20
[35027.741949]  [<ffffffff810bd6c5>] ? cpu_startup_entry+0x355/0x410
[35027.742858]  [<ffffffff8103016a>] ? start_secondary+0x1aa/0x230
[35982.698415] perf interrupt took too long (209762 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
[37306.241794] perf interrupt took too long (208160 > 19841), lowering kernel.perf_event_max_sample_rate to 6300
[38626.487390] perf interrupt took too long (206565 > 39062), lowering kernel.perf_event_max_sample_rate to 3200
[39781.429034] perf interrupt took too long (204990 > 78125), lowering kernel.perf_event_max_sample_rate to 1600
[41041.380281] perf interrupt took too long (203427 > 156250), lowering kernel.perf_event_max_sample_rate to 800

Hmmm... This one looks like the RCU grace-period kthread is getting
starved: "idle=b4c/0/0". Is this running with the "dangerous" patch that sets these kthreads to RT priority?

Thanx, Paul

sorry, no. Ran out of time yesterday. I'll try and get to applying that later this evening if I get chance.

Dave

Whew!!! You had me worried there for a bit! ;-)

Thanx, Paul

Ok, I think we can rule out preemption. I just checked on it, and found it wedged. Here's what I got over usb-serial. (tainting was from previous post).

[76132.505590] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [trinity-c8:19387]
[76132.506438] CPU: 3 PID: 19387 Comm: trinity-c8 Tainted: G        W      3.18.0+ #102 [loadavg: 148.33 137.64 140.62 48/406 19489]
[76132.507293] task: ffff880226a9ada0 ti: ffff8801aee08000 task.ti: ffff8801aee08000
[76132.508149] RIP: 0010:[<ffffffff81045d3c>]  [<ffffffff81045d3c>] kernel_map_pages+0xbc/0x120
[76132.509022] RSP: 0000:ffff8801aee0ba08  EFLAGS: 00000202
[76132.509889] RAX: 00000000001407e0 RBX: 0000000000000000 RCX: 0000000000140760
[76132.510760] RDX: 0000000000000202 RSI: ffff880000000188 RDI: 0000000000000001
[76132.511636] RBP: ffff8801aee0ba68 R08: 8000000000000063 R09: ffff880000000000
[76132.512512] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[76132.513394] R13: 0000000000000000 R14: 0000000001b5f000 R15: 0000000000000000
[76132.514269] FS:  00007fb1263cc740(0000) GS:ffff880244600000(0000) knlGS:0000000000000000
[76132.515152] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76132.516055] CR2: 000000000277dff8 CR3: 0000000233290000 CR4: 00000000001407e0
[76132.516957] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76132.517858] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76132.518757] Stack:
[76132.519650]  ffff880097a32000 ffff8801aee0ba08 0000000000000000 0000000000000003
[76132.520590]  0000000000000000 0000000100000001 0000000000097a31 0000000000000000
[76132.521530]  0000000000000000 0000000078052420 ffff8802447d7348 0000000000000001
[76132.522447] Call Trace:
[76132.523359]  [<ffffffff8117f9d4>] get_page_from_freelist+0x4a4/0xaa0
[76132.524281]  [<ffffffff811801fe>] __alloc_pages_nodemask+0x22e/0xb40
[76132.525205]  [<ffffffff810abb55>] ? local_clock+0x25/0x30
[76132.526135]  [<ffffffff810c518c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[76132.527065]  [<ffffffff8118545d>] ? release_pages+0x1bd/0x270
[76132.528004]  [<ffffffff810c367f>] ? lock_release_holdtime.part.24+0xf/0x190
[76132.528944]  [<ffffffff811c941e>] alloc_pages_vma+0xee/0x1b0
[76132.529883]  [<ffffffff811a465a>] ? do_wp_page+0xca/0x770
[76132.530823]  [<ffffffff8109faff>] ? __might_sleep+0x1f/0x140
[76132.531770]  [<ffffffff811a465a>] do_wp_page+0xca/0x770
[76132.532705]  [<ffffffff811a6eab>] handle_mm_fault+0x6cb/0xe90
[76132.533633]  [<ffffffff810423f8>] ? __do_page_fault+0x198/0x5c0
[76132.534561]  [<ffffffff8104245c>] __do_page_fault+0x1fc/0x5c0
[76132.535465]  [<ffffffff817cf790>] ? _raw_spin_unlock_irq+0x30/0x40
[76132.536372]  [<ffffffff8109f2bd>] ? finish_task_switch+0x7d/0x120
[76132.537269]  [<ffffffff8109f27f>] ? finish_task_switch+0x3f/0x120
[76132.538154]  [<ffffffff817c9822>] ? __schedule+0x352/0x8c0
[76132.539043]  [<ffffffff8137576d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[76132.539921]  [<ffffffff8104282c>] do_page_fault+0xc/0x10
[76132.540791]  [<ffffffff817d1fb2>] page_fault+0x22/0x30
[76132.541654] Code: 65 48 33 04 25 28 00 00 00 75 75 48 83 c4 50 5b 41 5c 5d c3 0f 1f 00 9c 5a fa 0f 20 e0 48 89 c1 80 e1 7f 0f 22 e1 0f 22 e0 52 9d <eb> cf 66 90 49 bc 00 00 00 00 00 88 ff ff 48 63 f6 49 01 fc 48 
[76132.543541] sending NMI to other CPUs:
[76132.544438] NMI backtrace for cpu 1
[76132.545300] CPU: 1 PID: 17326 Comm: trinity-c93 Tainted: G        W      3.18.0+ #102 [loadavg: 148.33 137.64 140.62 48/406 19489]
[76132.546193] task: ffff8800098b2da0 ti: ffff8801aec38000 task.ti: ffff8801aec38000
[76132.547085] RIP: 0010:[<ffffffff810c50a2>]  [<ffffffff810c50a2>] __lock_acquire.isra.31+0x142/0x9f0
[76132.547987] RSP: 0018:ffff8801aec3bd58  EFLAGS: 00000082
[76132.548883] RAX: 0000000000000000 RBX: ffff8800098b2da0 RCX: 0000000000000002
[76132.549786] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81c0a098
[76132.550683] RBP: ffff8801aec3bdc8 R08: 0000000000000000 R09: 0000000000000000
[76132.551576] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff825a4ab0
[76132.552462] R13: 0000000000000000 R14: ffffffff81c0a098 R15: 0000000000000000
[76132.553347] FS:  00007fb1263cc740(0000) GS:ffff880244200000(0000) knlGS:0000000000000000
[76132.554237] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76132.555118] CR2: 00007fc56b68e000 CR3: 0000000176d68000 CR4: 00000000001407e0
[76132.556014] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76132.556908] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76132.557805] Stack:
[76132.558679]  ffff8801aec3bdd8 0000000000000046 0000000000000000 0000000000000000
[76132.559582]  0000000000000000 0000000000000000 ffff8801aec3bdf8 0000000000000046
[76132.560486]  0000000000000000 0000000000000246 0000000000000000 0000000000000000
[76132.561394] Call Trace:
[76132.562285]  [<ffffffff810c605f>] lock_acquire+0x9f/0x120
[76132.563172]  [<ffffffff8107a939>] ? do_wait+0xd9/0x280
[76132.564038]  [<ffffffff817cf851>] _raw_read_lock+0x41/0x80
[76132.564891]  [<ffffffff8107a939>] ? do_wait+0xd9/0x280
[76132.565734]  [<ffffffff8107a939>] do_wait+0xd9/0x280
[76132.566566]  [<ffffffff8107af00>] SyS_wait4+0x80/0x110
[76132.567389]  [<ffffffff810789e0>] ? task_stopped_code+0x60/0x60
[76132.568220]  [<ffffffff817d0392>] system_call_fastpath+0x12/0x17
[76132.569044] Code: 00 00 45 31 ed e9 f3 01 00 00 0f 1f 80 00 00 00 00 44 89 e8 4d 8b 64 c6 08 4d 85 e4 0f 84 21 ff ff ff f0 41 ff 84 24 98 01 00 00 <8b> 3d 70 31 ab 01 44 8b ab 68 07 00 00 85 ff 75 0a 41 83 fd 2f 
[76132.570883] NMI backtrace for cpu 2
[76132.570886] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 26.444 msecs
[76132.572634] CPU: 2 PID: 18775 Comm: trinity-c59 Tainted: G        W      3.18.0+ #102 [loadavg: 148.33 137.64 140.62 49/406 19489]
[76132.573536] task: ffff880096135b40 ti: ffff8802408f0000 task.ti: ffff8802408f0000
[76132.574440] RIP: 0010:[<ffffffff810fbd4e>]  [<ffffffff810fbd4e>] generic_exec_single+0xee/0x1b0
[76132.575364] RSP: 0018:ffff8802408f3d28  EFLAGS: 00000202
[76132.576275] RAX: ffff880223a67d00 RBX: ffff8802408f3d40 RCX: ffff880223a67d40
[76132.577188] RDX: ffff8802447d3ac0 RSI: ffff8802408f3d40 RDI: ffff8802408f3d40
[76132.578096] RBP: ffff8802408f3d88 R08: 0000000000000001 R09: 0000000000000001
[76132.579011] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[76132.579918] R13: 0000000000000001 R14: ffff880096159290 R15: ffff8802408f3e80
[76132.580822] FS:  00007fb1263cc740(0000) GS:ffff880244400000(0000) knlGS:0000000000000000
[76132.581737] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76132.582653] CR2: 0000000081000000 CR3: 00000001ccf39000 CR4: 00000000001407e0
[76132.583573] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76132.584495] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76132.585411] Stack:
[76132.586319]  0000000000000001 ffff880096135b40 ffff8802408f3d48 ffff880223a67d40
[76132.587256]  ffffffff81166730 ffff8802258737b0 0000000000000003 00000000803d2842
[76132.588198]  ffff8802408f3da8 00000000ffffffff 0000000000000003 ffffffff81166730
[76132.589142] Call Trace:
[76132.590074]  [<ffffffff81166730>] ? perf_swevent_add+0x110/0x110
[76132.591018]  [<ffffffff81166730>] ? perf_swevent_add+0x110/0x110
[76132.591957]  [<ffffffff810fbeb0>] smp_call[76148.896166] sched: RT throttling activated
[76172.492491] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u16:1:24452]
[76172.493490] CPU: 3 PID: 24452 Comm: kworker/u16:1 Tainted: G        W    L 3.18.0+ #102 [loadavg: 180.20 147.58 143.86 23/399 24452]
[76172.494533] task: ffff88007309c470 ti: ffff880223a18000 task.ti: ffff880223a18000
[76172.495572] RIP: 0010:[<ffffffff817cf799>]  [<ffffffff817cf799>] _raw_spin_unlock_irq+0x39/0x40
[76172.496598] RSP: 0018:ffff880223a1bec8  EFLAGS: 00000286
[76172.497628] RAX: 0000000000000003 RBX: 0000000000000046 RCX: 0000000000000380
[76172.498691] RDX: ffff88024460dc40 RSI: 0000000000000000 RDI: ffff8802447d2f00
[76172.499749] RBP: ffff880223a1bed8 R08: 0000000000000000 R09: 0000000000000000
[76172.500804] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[76172.501920] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[76172.502958] FS:  0000000000000000(0000) GS:ffff880244600000(0000) knlGS:0000000000000000
[76172.504054] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76172.505140] CR2: 00007fe411e27000 CR3: 0000000042ae7000 CR4: 00000000001407e0
[76172.506171] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76172.507176] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76172.508171] Stack:
[76172.509170]  0000000000000001 ffff8802447d2f00 ffff880223a1bf28 ffffffff8109f2bd
[76172.510185]  ffffffff8109f27f 0000000000000000 0000000000000000 ffff8802447d2f00
[76172.511234]  ffff8802447d2f00 ffff880094cc5dc0 0000000000000000 ffff8801aed38208
[76172.512313] Call Trace:
[76172.513371]  [<ffffffff8109f2bd>] finish_task_switch+0x7d/0x120
[76172.514399]  [<ffffffff8109f27f>] ? finish_task_switch+0x3f/0x120
[76172.515429]  [<ffffffff810a5c87>] schedule_tail+0x27/0xb0
[76172.516455]  [<ffffffff817d027f>] ret_from_fork+0xf/0xb0
[76172.517479]  [<ffffffff8108e390>] ? call_helper+0x20/0x20
[76172.518515] Code: 53 48 89 fb 48 8d 7f 18 48 83 ec 08 48 8b 55 08 e8 8d 6c 8f ff 48 89 df e8 b5 9f 8f ff e8 d0 aa 97 ff fb 65 ff 0c 25 e0 a9 00 00 <48> 83 c4 08 5b 5d c3 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 
[76172.520732] sending NMI to other CPUs:
[76172.521757] NMI backtrace for cpu 1
[76172.522745] CPU: 1 PID: 24454 Comm: modprobe Tainted: G        W    L 3.18.0+ #102 [loadavg: 180.20 147.58 143.86 19/402 24456]
[76172.523759] task: ffff880223174470 ti: ffff880225ba4000 task.ti: ffff880225ba4000
[76172.524750] RIP: 0033:[<000000336f609ffd>]  [<000000336f609ffd>] 0x336f609ffd
[76172.525747] RSP: 002b:00007fffcd99c458  EFLAGS: 00000202
[76172.526721] RAX: 0000000000000001 RBX: 0000000000000003 RCX: 0000000000000000
[76172.527697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000336ec20908
[76172.528662] RBP: 000000336ee80fc0 R08: 000000336ec20908 R09: 0000000000000000
[76172.529632] R10: 000000336ee879b0 R11: 0000000000000010 R12: 00007fffcd99c4d0
[76172.530599] R13: 0000000000418c2f R14: 00007fffcd99d918 R15: 0000000000000000
[76172.531567] FS:  00007f8ca6d7e740(0000) GS:ffff880244200000(0000) knlGS:0000000000000000
[76172.532543] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76172.533512] CR2: 000000336eed9c00 CR3: 00000000963b7000 CR4: 00000000001407e0
[76172.534486] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76172.535450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76172.536413] 
[76172.537368] NMI backtrace for cpu 2
[76172.538331] CPU: 2 PID: 20261 Comm: trinity-c241 Tainted: G        W    L 3.18.0+ #102 [loadavg: 180.20 147.58 143.86 18/402 24456]
[76172.539329] task: ffff88017968ada0 ti: ffff8801c0508000 task.ti: ffff8801c0508000
[76172.540316] RIP: 0033:[<000000336ee39e40>]  [<000000336ee39e40>] 0x336ee39e40
[76172.541314] RSP: 002b:00007fffe71236e8  EFLAGS: 00000206
[76172.542300] RAX: 0000000000000003 RBX: 0000000001b66720 RCX: 000000336f1b70cc
[76172.543289] RDX: 000000336f1b70c4 RSI: 00007fffe71236fc RDI: 000000336f1b76e0
[76172.544282] RBP: 000000000000000f R08: 000000336f1b713c R09: 000000336f1b7140
[76172.545273] R10: ffffffffffff9f00 R11: 0000000000000202 R12: 00007fe41545d000
[76172.546266] R13: 00007fe41545d068 R14: 0000000000000000 R15: 0000000000000000
[76172.547263] FS:  00007fe416231740(0000) GS:ffff880244400000(0000) knlGS:0000000000000000
[76172.548267] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76172.549270] CR2: 0000000000000008 CR3: 000000017304b000 CR4: 00000000001407e0
[76172.550284] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76172.551300] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76172.552316] 
[76172.553304] NMI backtrace for cpu 0
[76172.554289] CPU: 0 PID: 22642 Comm: trinity-c51 Tainted: G        W    L 3.18.0+ #102 [loadavg: 180.20 147.58 143.86 19/400 24457]
[76172.555284] task: ffff88009f375b40 ti: ffff880229be8000 task.ti: ffff880229be8000
[76172.556277] RIP: 0010:[<ffffffff811a5de0>]  [<ffffffff811a5de0>] copy_page_range+0x550/0xa20
[76172.557304] RSP: 0018:ffff880229bebc50  EFLAGS: 00000286
[76172.558340] RAX: 001ffe000008007c RBX: 00007fe413bbf000 RCX: 00000000022c3700
[76172.559328] RDX: ffffea00022c3700 RSI: 00007fe413bbf000 RDI: ffff880201b78c00
[76172.560375] RBP: ffff880229bebd80 R08: 0000000000000000 R09: 0000000000000001
[76172.561423] R10: 0000000000000000 R11: 0000000000000000 R12: 800000008b0dc007
[76172.562465] R13: 00007fe413c00000 R14: ffff88022475ddf8 R15: 0000000000000018
[76172.563432] FS:  00007fe416231740(0000) GS:ffff880244000000(0000) knlGS:0000000000000000
[76172.564401] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76172.565375] CR2: 0000000001d04b68 CR3: 00000001aee5d000 CR4: 00000000001407f0
[76172.566408] DR0: 00007f07ef05a000 DR1: 00007fb7761bb000 DR2: 0000000000000000
[76172.567423] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[76172.568394] Stack:
[76172.569353]  ffffffff00000000 00007fe413ea6fff 0000000000000000 ffff88021242b988
[76172.570337]  00007fe413ea6fff 00007fe4134a7000 ffff88006681f7f8 ffff880175a72d08
[76172.571323]  ffff8800905d9c80 ffff8801aee5d7f8 00007fe413ea7000 ffff880223ff3c80
[76172.572320] Call Trace:
[76172.573300]  [<ffffffff810764ef>] copy_process.part.26+0x146f/0x1a40

Dave

Ok, one more. Mind checking what happens without CONFIG_DEBUG_PAGEALLOC?

Linus

Side note: I think I've found a real potential lockup bug in fs/namespace.c, but afaik it could only trigger with the RT patches.

I'm looking at what lxsetattr() does, since you had that lxsetattr-only lockup. I doubt it's really related to lxsetattr(), but whatever. The generic code does that mnt_want_write/mnt_drop_write dance adound the call to setxattr, and that in turn does

        while (ACCESS_ONCE(mnt->mnt.mnt_flags) & MNT_WRITE_HOLD)
                cpu_relax();

with preemption explicitly disabled. It's waitingo for mnt_make_readonly() to go away if it is racing with it.

But mnt_make_readonly() doesn't actually explicitly disable preemption while it sets that MNT_WRITE_HOLD bit. Instead, it depends on lock_mount_hash() to disable preemption for it. Which it does, because it is a seq-writelock, which uses a spinlock, which will disable preemption.

Except it won't with the RT patches, I guess. So it looks like you could have:\

- mnt_make_readonly() sets that bit
- gets preempted with the RT patches
- we run mnt_want_write() on all CPU's, which disables preemption and
waits for the bit to be cleared
- nothing happens.

This is clearly not what happens in your lockup, but it does seem to be a potential issue for the RT kernel.

Added Al and Thomas to the cc, for fs/namespace.c and RT kernel respectively. Maybe the RT patches already fix this, I didn't actually check.

Linus

Btw, I see no reason why mnt_want_write/mnt_drop_write disables preemption. They don't care, they just care about the ordering of the write counts and the MNT_WRITE_HOLD bit. It's the code that sets the bit that should care, afaik. But maybe I'm missing something.

Linus

Er... There's much more direct reason - suppose we get a timer interrupt right in the middle of mnt_drop_write(). And lost the timeslice. On UP we have mnt->mnt_writers--, with no locks held. On SMP we have this_cpu_dec() instead, also without any locks. You really don't want to lose the timeslice in the middle of either...

So?

You didn't have preemption disabled in *between* the mnt_want_write() and mnt_drop_write(), there's absolutely no reason to have it inside of them.

Nobody cares if you get preempted and go away for a while. It's exactly equivalent to sleeping while doing the write that the pair was protecting.

Seriously, the preemption disable looks like just voodoo code. It doesn't protect anything, it doesn't fix anything, it doesn't change anything. All it does is disable preemption over a random sequence of code.

Linus

Huh? Sure, we can enable it after mnt_inc_writers() and disable just prior to mnt_dec_writers(), but we absolutely *do* need it disabled during either. Is that what you are talking about? If so, yes, we can do that.

But that applies only to __mnt_want_write() - __mnt_drop_write() is pure mnt_dec_writers() and we can't call that one with preemption enabled. Seriously, look at the mnt_dec_writers():
static inline void mnt_dec_writers(struct mount *mnt)
{

#ifdef CONFIG_SMP
        this_cpu_dec(mnt->mnt_pcp->mnt_writers);  
#else
        mnt->mnt_writers--;

#endif
}
It's load/modify/store, without any kind of atomicity; get preempted in the middle of that sequence by another caller of mnt_dec_writers() and obvious bad things will happen...

Al, really confused by now...

Ugh, yes ok, the UP case needs it for the actual counter itself. Ugh. What an ugly mess. I'd rather have the preemption disable where it is actually *needed*, in that function itself for the UP case (or just make it "atomic_t", which would likely be better still.

Linus

So does SMP - this_cpu_dec() relies on preemption being disabled. On x86 we might get away with that, what with having it compiled into decl %gs:const, but on generic it turns into

	*raw_cpu_ptr(&pcp) -= 1;
and compiler has every right to turn it into
	p = raw_cpu_ptr(&pcp);
	(*p)--;

again, with no locking. Lose the timeslice in the middle of that and you are risking to get a different CPU when you are scheduled again, with another process doing this_cpu_dec() on your old CPU. Have fun - two non-atomic decrements of the same variable by different CPUs in parallel...

We really need preemtion disabled there, UP or no UP.

No. really. It very much does not. Not on x86, not elsewhere. It's part of the whole point of "this_cpu_p()". They are preemption and interrupt safe.

It's the "__this_cpu_op()" ones that need external protection.

Linus

Right you are - I really need to get some coffee... Sorry...

FWIW, do we need to disable interrupts there? After all, mnt_want_write() and mnt_drop_write() shouldn't be done from interrupt context - they can happen via schedule_delayed_work(), but that's it...

OK, having looked through the tree - we really don't need to bother with disabling interrupts (fortunately - or UP case would be broken). So how about turning those into __this_cpu_{inc,dec} and yes, moving preempt disabling into mnt_{inc,dec}_writers()? Like this:

index 5b66b2b..48cb162 100644
@@ -274,20 +274,32 @@ EXPORT_SYMBOL_GPL(__mnt_is_readonly);
 
 static inline void mnt_inc_writers(struct mount *mnt)
 {
+	preempt_disable();
 #ifdef CONFIG_SMP
-	this_cpu_inc(mnt->mnt_pcp->mnt_writers);
+	__this_cpu_inc(mnt->mnt_pcp->mnt_writers);
 #else
 	mnt->mnt_writers++;
 #endif
+	preempt_enable();
 }
 
-static inline void mnt_dec_writers(struct mount *mnt)
+/**
+ * __mnt_drop_write - give up write access to a mount
+ * @mnt: the mount on which to give up write access
+ *
+ * Tells the low-level filesystem that we are done
+ * performing writes to it.  Must be matched with
+ * __mnt_want_write() call above.
+ */
+void __mnt_drop_write(struct vfsmount *m)
 {
+	preempt_disable();
 #ifdef CONFIG_SMP
-	this_cpu_dec(mnt->mnt_pcp->mnt_writers);
+	__this_cpu_dec(real_mount(m)->mnt_pcp->mnt_writers);
 #else
-	mnt->mnt_writers--;
+	real_mount(m)->mnt_writers--;
 #endif
+	preempt_enable();
 }
 
 static unsigned int mnt_get_writers(struct mount *mnt)

@@ -336,7 +348,6 @@ int __mnt_want_write(struct vfsmount *m)
 	struct mount *mnt = real_mount(m);
 	int ret = 0;
 
-	preempt_disable();
 	mnt_inc_writers(mnt);
 	/*
 	 * The store to mnt_inc_writers must be visible before we pass

@@ -353,10 +364,9 @@ int __mnt_want_write(struct vfsmount *m)
 	 */
 	smp_rmb();
 	if (mnt_is_readonly(m)) {
-		mnt_dec_writers(mnt);
+		__mnt_drop_write(m);
 		ret = -EROFS;
 	}
-	preempt_enable();
 
 	return ret;
 }

@@ -399,9 +409,7 @@ int mnt_clone_write(struct vfsmount *mnt)
 	/* superblock may be r/o */
 	if (__mnt_is_readonly(mnt))
 		return -EROFS;
-	preempt_disable();
 	mnt_inc_writers(real_mount(mnt));
-	preempt_enable();
 	return 0;
 }
 EXPORT_SYMBOL_GPL(mnt_clone_write);

@@ -441,21 +449,6 @@ int mnt_want_write_file(struct file *file)
 EXPORT_SYMBOL_GPL(mnt_want_write_file);
 
 /**
- * __mnt_drop_write - give up write access to a mount
- * @mnt: the mount on which to give up write access
- *
- * Tells the low-level filesystem that we are done
- * performing writes to it.  Must be matched with
- * __mnt_want_write() call above.
- */
-void __mnt_drop_write(struct vfsmount *mnt)
-{
-	preempt_disable();
-	mnt_dec_writers(real_mount(mnt));
-	preempt_enable();
-}
-
-/**
  * mnt_drop_write - give up write access to a mount
  * @mnt: the mount on which to give up write access
  *

I agree that it's a thing to keep in mind on the RT side of things, but IMO it belongs in RT patches...

Crap. Looks like it wedged. It's stuck that way until I get back to it on Wednesday.

[ 6188.985536] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c175:14205]
[ 6188.985612] CPU: 1 PID: 14205 Comm: trinity-c175 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 179/407 17316]
[ 6188.985652] task: ffff880056ac96d0 ti: ffff8800975d8000 task.ti: ffff8800975d8000
[ 6188.985680] RIP: 0010:[<ffffffff810c6430>]  [<ffffffff810c6430>] lock_release+0xc0/0x240
[ 6188.985714] RSP: 0018:ffff8800975dbaa8  EFLAGS: 00000292
[ 6188.985734] RAX: ffff880056ac96d0 RBX: ffff8800975dbaf0 RCX: 00000000000003a0
[ 6188.985759] RDX: ffff88024500dd20 RSI: 0000000000000000 RDI: ffff880056ac9e40
[ 6188.985785] RBP: ffff8800975dbad8 R08: 0000000000000000 R09: 0000000000000000
[ 6188.985810] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000292
[ 6188.985835] R13: ffff8800975dba28 R14: 0000000000000292 R15: 0000000000000292
[ 6188.985861] FS:  00007f107fc69740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[ 6188.985890] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6188.985912] CR2: 00007fff7457af40 CR3: 00000000145ed000 CR4: 00000000001407e0
[ 6188.985937] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6188.985963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 6188.985988] Stack:
[ 6188.986000]  ffff8800975dbb08 0000000000000000 0000000000000000 00000000001ce380
[ 6188.986034]  ffff8800975dbd08 ffff8802451ce380 ffff8800975dbbd8 ffffffff8116f928
[ 6188.986067]  ffffffff8116f842 ffff8800975dbaf0 ffff8800975dbaf0 0000000000000001
[ 6188.986101] Call Trace:
[ 6188.986116]  [<ffffffff8116f928>] __perf_sw_event+0x168/0x240
[ 6188.987079]  [<ffffffff8116f842>] ? __perf_sw_event+0x82/0x240
[ 6188.988045]  [<ffffffff81178ab2>] ? __lock_page_or_retry+0xb2/0xc0
[ 6188.989008]  [<ffffffff811a68f8>] ? handle_mm_fault+0x458/0xe90
[ 6188.989986]  [<ffffffff8104250e>] __do_page_fault+0x28e/0x5c0
[ 6188.990940]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6188.991884]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6188.992826]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6188.993773]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6188.994715]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6188.995658]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6188.996590]  [<ffffffff81375266>] ? __clear_user+0x36/0x60
[ 6188.997518]  [<ffffffff81375247>] ? __clear_user+0x17/0x60
[ 6188.998440]  [<ffffffff8100f3f1>] save_xstate_sig+0x81/0x220
[ 6188.999362]  [<ffffffff817cf1cf>] ? _raw_spin_unlock_irqrestore+0x4f/0x60
[ 6189.000291]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6189.001220]  [<ffffffff81209acf>] ? mnt_drop_write+0x2f/0x40
[ 6189.002164]  [<ffffffff811e527e>] ? chmod_common+0xfe/0x150
[ 6189.003096]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6189.004038]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6189.004972]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6189.005899] Code: ff 0f 85 7c 00 00 00 4c 89 ea 4c 89 e6 48 89 df e8 26 fc ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 41 56 9d <48> 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d f3 c3 65 ff 04 25 e0 
[ 6189.007935] sending NMI to other CPUs:
[ 6189.008904] NMI backtrace for cpu 2
[ 6189.009755] CPU: 2 PID: 14224 Comm: trinity-c194 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 179/407 17316]
[ 6189.010618] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000
[ 6189.011555] RIP: 0010:[<ffffffff81176890>]  [<ffffffff81176890>] pagecache_get_page+0x0/0x220
[ 6189.012501] RSP: 0018:ffff880225aefb50  EFLAGS: 00000282
[ 6189.013442] RAX: ffff88023f4b9d00 RBX: 00007fff7457b07f RCX: 0000000000000000
[ 6189.014396] RDX: 0000000000000000 RSI: 000000000001dda7 RDI: ffffffff81c6aa80
[ 6189.015357] RBP: ffff880225aefb58 R08: 0000000000000000 R09: 0000000007769c80
[ 6189.016317] R10: 0000000000000000 R11: 0000000000000029 R12: ffff88022a63be00
[ 6189.017283] R13: ffff8801c64afd10 R14: ffff880000000bd8 R15: ffff880187efea40
[ 6189.018253] FS:  00007f107fc69740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[ 6189.019247] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6189.020245] CR2: 00007fff7457b07f CR3: 000000022789e000 CR4: 00000000001407e0
[ 6189.021230] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6189.022193] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 6189.023155] Stack:
[ 6189.024104]  ffffffff811b99da ffff880225aefbf8 ffffffff811a68a1 ffff880225aefbd8
[ 6189.025084]  0000000000000246 ffffffff81042418 ffffffff00000000 0000000000000000
[ 6189.026063]  0000000000000246 0000000100000000 ffff880187efeb58 0000000000000080
[ 6189.027025] Call Trace:
[ 6189.027962]  [<ffffffff811b99da>] ? lookup_swap_cache+0x2a/0x70
[ 6189.028897]  [<ffffffff811a68a1>] handle_mm_fault+0x401/0xe90
[ 6189.029819]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6189.030731]  [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0
[ 6189.031635]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6189.032537]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6189.033432]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6189.034334]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6189.035238]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6189.036146]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6189.037043]  [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220
[ 6189.037934]  [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220
[ 6189.038819]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6189.039699]  [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40
[ 6189.040583]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6189.041464]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6189.042340]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6189.043210] Code: f0 80 a6 81 48 89 df e8 7f a5 02 00 0f 0b 48 89 df e8 45 fd ff ff 48 89 df e8 8d e4 00 00 eb 83 66 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 41 57 45 89 c7 41 56 49 89 f6 41 55 
[ 6189.045130] NMI backtrace for cpu 3
[ 6189.045244] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 36.225 msecs
[ 6189.046980] CPU: 3 PID: 14076 Comm: trinity-c46 Not tainted 3.18.0+ #103 [loadavg: 200.63 151.07 150.40 181/407 17316]
[ 6189.047934] task: ffff88008a6c4470 ti: ffff8801cdb58000 task.ti: ffff8801cdb58000
[ 6189.048893] RIP: 0010:[<ffffffff810c63a4>]  [<ffffffff810c63a4>] lock_release+0x34/0x240
[ 6189.049867] RSP: 0000:ffff8801cdb5bad0  EFLAGS: 00000296
[ 6189.050834] RAX: ffff88008a6c4470 RBX: ffff88013f39e4a8 RCX: 00000000000003a0
[ 6189.051815] RDX: ffffffff81178a7f RSI: 0000000000000001 RDI: ffff88013f39e518
[ 6189.052799] RBP: ffff8801cdb5bb08 R08: 0000000000000000 R09: 00000000073e8480
[ 6189.053781] R10: ffffea0007927a80 R11: 0000000000000029 R12: ffff88013f39e518
[ 6189.054764] R13: ffffffff81178a7f R14: ffff880000000bd8 R15: 0000000000000001
[ 6189.055748] FS:  00007f107fc69740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[ 6189.056746] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6189.057751] CR2: 00007fff7457b07f CR3: 0000000226f46000 CR4: 00000000001407e0
[ 6189.058770] DR0: 00007f322081b000 DR1: 0000000000[ 6216.969357] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c175:14205]
[ 6216.970331] CPU: 1 PID: 14205 Comm: trinity-c175 Tainted: G             L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316]
[ 6216.971359] task: ffff880056ac96d0 ti: ffff8800975d8000 task.ti: ffff8800975d8000
[ 6216.972366] RIP: 0010:[<ffffffff817cf1b8>]  [<ffffffff817cf1b8>] _raw_spin_unlock_irqrestore+0x38/0x60
[ 6216.973391] RSP: 0018:ffff8800975dba18  EFLAGS: 00000292
[ 6216.974423] RAX: 0000000000000001 RBX: ffff880056ac96d0 RCX: 0000000000005040
[ 6216.975459] RDX: ffff88024502f580 RSI: 0000000000000000 RDI: ffff88024e581e28
[ 6216.976507] RBP: ffff8800975dba28 R08: 0000000000000000 R09: ffff8800975dbaf0
[ 6216.977551] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000046
[ 6216.978594] R13: ffff8800975db9e8 R14: 0000000000000000 R15: 0000000000000000
[ 6216.979635] FS:  00007f107fc69740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[ 6216.980686] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6216.981735] CR2: 00007fff7457af40 CR3: 00000000145ed000 CR4: 00000000001407e0
[ 6216.982774] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6216.983792] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 6216.984799] Stack:
[ 6216.985785]  ffff8800975dbad8 ffff8800975dbaf0 ffff8800975dba58 ffffffff810bcd56
[ 6216.986789]  ffff8800975dbac0 ffff8800975dbad8 ffff88024e581e28 0000000000000082
[ 6216.987794]  ffff8800975dbaa8 ffffffff817c9f4e ffff8800975dba78 000000010073e800
[ 6216.988770] Call Trace:
[ 6216.989729]  [<ffffffff810bcd56>] finish_wait+0x56/0x70
[ 6216.990693]  [<ffffffff817c9f4e>] __wait_on_bit+0x7e/0x90
[ 6216.991661]  [<ffffffff811789d7>] wait_on_page_bit_killable+0xc7/0xf0
[ 6216.992632]  [<ffffffff810bd050>] ? autoremove_wake_function+0x40/0x40
[ 6216.993609]  [<ffffffff81178ab2>] __lock_page_or_retry+0xb2/0xc0
[ 6216.994586]  [<ffffffff811a6e5c>] handle_mm_fault+0x9bc/0xe90
[ 6216.995555]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6216.996516]  [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0
[ 6216.997468]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6216.998428]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6216.999386]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6217.000330]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6217.001269]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6217.002205]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6217.003136]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6217.004055]  [<ffffffff81375266>] ? __clear_user+0x36/0x60
[ 6217.004972]  [<ffffffff81375247>] ? __clear_user+0x17/0x60
[ 6217.005882]  [<ffffffff8100f3f1>] save_xstate_sig+0x81/0x220
[ 6217.006800]  [<ffffffff817cf1cf>] ? _raw_spin_unlock_irqrestore+0x4f/0x60
[ 6217.007716]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6217.008635]  [<ffffffff81209acf>] ? mnt_drop_write+0x2f/0x40
[ 6217.009555]  [<ffffffff811e527e>] ? chmod_common+0xfe/0x150
[ 6217.010470]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6217.011382]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6217.012297]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6217.013210] Code: fc 48 8b 55 08 53 48 8d 7f 18 48 89 f3 be 01 00 00 00 e8 cc 71 8f ff 4c 89 e7 e8 f4 a4 8f ff f6 c7 02 74 17 e8 0a b0 97 ff 53 9d <5b> 65 ff 0c 25 e0 a9 00 00 41 5c 5d c3 0f 1f 00 53 9d e8 f1 ae 
[ 6217.015229] sending NMI to other CPUs:
[ 6217.016191] NMI backtrace for cpu 3
[ 6217.017110] CPU: 3 PID: 14076 Comm: trinity-c46 Tainted: G             L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316]
[ 6217.018066] task: ffff88008a6c4470 ti: ffff8801cdb58000 task.ti: ffff8801cdb58000
[ 6217.019021] RIP: 0010:[<ffffffff810c5071>]  [<ffffffff810c5071>] __lock_acquire.isra.31+0x1b1/0x9f0
[ 6217.019997] RSP: 0000:ffff8801cdb5b9d8  EFLAGS: 00000083
[ 6217.020972] RAX: 000000000000001e RBX: ffff88008a6c4470 RCX: 0000000000000002
[ 6217.021966] RDX: 0000000000000157 RSI: 0000000000000008 RDI: 0000000000000000
[ 6217.022961] RBP: ffff8801cdb5ba48 R08: 0000000000000000 R09: 0000000000000000
[ 6217.023953] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001d
[ 6217.024945] R13: 0000000000000001 R14: ffffffff81c50e60 R15: ffff88008a6c4c18
[ 6217.025939] FS:  00007f107fc69740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[ 6217.026943] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6217.027943] CR2: 00007fff7457b07f CR3: 0000000226f46000 CR4: 00000000001407e0
[ 6217.028933] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6217.029896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 6217.030846] Stack:
[ 6217.031772]  ffff88008a6c4470 ffff88024e52e040 ffff8801cdb5ba28 0000000000000092
[ 6217.032708]  ffff8801cdb5ba08 ffffffff810abab5 ffff8801cdb5ba88 ffffffff810c50ec
[ 6217.033625]  0000000000000296 0000000000000246 0000000000000000 0000000000000000
[ 6217.034537] Call Trace:
[ 6217.035425]  [<ffffffff810abab5>] ? local_clock+0x25/0x30
[ 6217.036316]  [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0
[ 6217.037210]  [<ffffffff810c5fbf>] lock_acquire+0x9f/0x120
[ 6217.038103]  [<ffffffff811766d5>] ? find_get_entry+0x5/0x120
[ 6217.038995]  [<ffffffff81176717>] find_get_entry+0x47/0x120
[ 6217.039891]  [<ffffffff811766d5>] ? find_get_entry+0x5/0x120
[ 6217.040776]  [<ffffffff811768bf>] pagecache_get_page+0x2f/0x220
[ 6217.041653]  [<ffffffff8116f842>] ? __perf_sw_event+0x82/0x240
[ 6217.042527]  [<ffffffff811b99da>] lookup_swap_cache+0x2a/0x70
[ 6217.043399]  [<ffffffff811a68a1>] handle_mm_fault+0x401/0xe90
[ 6217.044273]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6217.045140]  [<ffffffff8104247c>] __do_page_fault+0x1fc/0x5c0
[ 6217.045999]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6217.046857]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6217.047713]  [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0
[ 6217.048562]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6217.049414]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6217.050263]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6217.051108]  [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220
[ 6217.051953]  [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220
[ 6217.052787]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6217.053620]  [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40
[ 6217.054457]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6217.055294]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6217.056133]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6217.056977] Code: ea 48 8d 34 d5 00 00 00 00 48 c1 e2 06 48 29 f2 4c 8d bc 13 70 07 00 00 41 0f b7 57 f8 81 e2 ff 1f 00 00 39 d0 0f 84 3f 01 00 00 <41> 0f b7 57 30 66 25 ff 1f 4c 89 4d c8 41 c1 e2 07 83 e1 03 4d 
[ 6217.058855] NMI backtrace for cpu 2
[ 6217.059739] CPU: 2 PID: 14224 Comm: trinity-c194 Tainted: G             L 3.18.0+ #103 [loadavg: 221.85 160.04 153.39 183/407 17316]
[ 6217.060662] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000
[ 6217.061589] RIP: 0010:[<ffffffff810c6396>]  [<ffffffff810c6396>] lock_release+0x26/0x240
[ 6217.062533] RSP: 0018:ffff880225aefa10  EFLAGS: 00000046
[ 6217.063477] RAX: ffff880224af5b40 RBX: 0000000000000296 RCX: 0000000000000002
[ 6217.064435] RDX: ffffffff810bcd56 RSI: 0000000000000001 RDI: ffff88024e54da40
[ 6217.065392] RBP: ffff880225aefa28 R08: 0000000000000000 R09: ffff880225aefb10
[ 6217.066342] R[ 6217.158570] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 142.293 msecs
[ 6225.814243] INFO: rcu_sched self-detected stall on CPU
[ 6225.815127] 	3: (5990 ticks this GP) idle=f83/140000000000001/0 softirq=390686/390686 
[ 6225.816000] 	 (t=6000 jiffies g=166553 c=166552 q=0)
[ 6225.816870] Task dump for CPU 3:
[ 6225.817736] trinity-c46     R  running task    13568 14076  13551 0x1000000c
[ 6225.818626]  ffff88008a6c4470 00000000ef08e0c8 ffff880245403d68 ffffffff810a73fc
[ 6225.819531]  ffffffff810a7362 0000000000000003 0000000000000008 0000000000000003
[ 6225.820432]  ffffffff81c523c0 0000000000000092 ffff880245403d88 ffffffff810ab4ad
[ 6225.821340] Call Trace:
[ 6225.822218]  <IRQ>  [<ffffffff810a73fc>] sched_show_task+0x11c/0x190
[ 6225.823122]  [<ffffffff810a7362>] ? sched_show_task+0x82/0x190
[ 6225.824021]  [<ffffffff810ab4ad>] dump_cpu_task+0x3d/0x50
[ 6225.824916]  [<ffffffff810d9bf0>] rcu_dump_cpu_stacks+0x90/0xd0
[ 6225.825813]  [<ffffffff810e0783>] rcu_check_callbacks+0x503/0x770
[ 6225.826697]  [<ffffffff811304dc>] ? acct_account_cputime+0x1c/0x20
[ 6225.827581]  [<ffffffff810abde7>] ? account_system_time+0x97/0x180
[ 6225.828464]  [<ffffffff810e645b>] update_process_times+0x4b/0x80
[ 6225.829350]  [<ffffffff810f6e13>] ? tick_sched_timer+0x23/0x1b0
[ 6225.830233]  [<ffffffff810f6e3f>] tick_sched_timer+0x4f/0x1b0
[ 6225.831108]  [<ffffffff810e72ff>] __run_hrtimer+0xaf/0x240
[ 6225.831977]  [<ffffffff810e76eb>] ? hrtimer_interrupt+0x16b/0x260
[ 6225.832844]  [<ffffffff810f6df0>] ? tick_init_highres+0x20/0x20
[ 6225.833709]  [<ffffffff810e7687>] hrtimer_interrupt+0x107/0x260
[ 6225.834565]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[ 6225.835384]  [<ffffffff817d28c5>] smp_apic_timer_interrupt+0x45/0x60
[ 6225.836203]  [<ffffffff817d0caf>] apic_timer_interrupt+0x6f/0x80
[ 6225.837023]  <EOI>  [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0
[ 6225.837858]  [<ffffffff810c5fd4>] ? lock_acquire+0xb4/0x120
[ 6225.838688]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6225.839517]  [<ffffffff810c225a>] down_read_trylock+0x5a/0x60
[ 6225.840345]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6225.841175]  [<ffffffff81042418>] __do_page_fault+0x198/0x5c0
[ 6225.842004]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6225.842836]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6225.843672]  [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0
[ 6225.844506]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6225.845340]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6225.846174]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6225.847002]  [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220
[ 6225.847827]  [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220
[ 6225.848648]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6225.849468]  [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40
[ 6225.850287]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6225.851104]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6225.851927]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6225.852746] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 6225.853609] 	3: (5991 ticks this GP) idle=f83/140000000000000/0 softirq=390686/390686 
[ 6225.854481] 	(detected by 1, t=6004 jiffies, g=166553, c=166552, q=0)
[ 6225.855354] Task dump for CPU 3:
[ 6225.856225] trinity-c46     R  running task    13568 14076  13551 0x1000000c
[ 6225.857127]  ffffffff810bcd38 ffff88008a6c4470 ffff8801cdb5b9c8 ffffffff810abab5
[ 6225.858045]  ffff88024e52e040 0000000000000046 ffff88008a6c4470 ffff88024e52e040
[ 6225.858951]  ffff8801cdb5ba28 0000000000000092 ffffffff810bcb37 ffffffff810abab5
[ 6225.859860] Call Trace:
[ 6225.860761]  [<ffffffff810c35df>] ? lock_release_holdtime.part.24+0xf/0x190
[ 6225.861690]  [<ffffffff810abab5>] ? local_clock+0x25/0x30
[ 6225.862612]  [<ffffffff810c35df>] lock_release_holdtime.part.24+0xf/0x190
[ 6225.863543]  [<ffffffff810abab5>] ? local_clock+0x25/0x30
[ 6225.864473]  [<ffffffff810c50ec>] ? __lock_acquire.isra.31+0x22c/0x9f0
[ 6225.865402]  [<ffffffff810bcd56>] ? finish_wait+0x56/0x70
[ 6225.866329]  [<ffffffff817c9f4e>] ? __wait_on_bit+0x7e/0x90
[ 6225.867236]  [<ffffffff811789d7>] ? wait_on_page_bit_killable+0xc7/0xf0
[ 6225.868122]  [<ffffffff810bd050>] ? autoremove_wake_function+0x40/0x40
[ 6225.868996]  [<ffffffff811b99da>] ? lookup_swap_cache+0x2a/0x70
[ 6225.869855]  [<ffffffff811a68f8>] ? handle_mm_fault+0x458/0xe90
[ 6225.870706]  [<ffffffff810c225a>] ? down_read_trylock+0x5a/0x60
[ 6225.871545]  [<ffffffff8104247c>] ? __do_page_fault+0x1fc/0x5c0
[ 6225.872385]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6225.873219]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6225.874046]  [<ffffffff81042358>] ? __do_page_fault+0xd8/0x5c0
[ 6225.874873]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6225.875692]  [<ffffffff8104284c>] ? do_page_fault+0xc/0x10
[ 6225.876503]  [<ffffffff817d1a32>] ? page_fault+0x22/0x30
[ 6225.877307]  [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220
[ 6225.878107]  [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220
[ 6225.878901]  [<ffffffff810029e7>] ? do_signal+0x5c7/0x740
[ 6225.879695]  [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40
[ 6225.880496]  [<ffffffff81002bc5>] ? do_notify_resume+0x65/0x80
[ 6225.881292]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6225.882097]  [<ffffffff817d00ff>] ? int_signal+0x12/0x17
[ 6244.953181] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [trinity-c194:14224]
[ 6244.953995] CPU: 2 PID: 14224 Comm: trinity-c194 Tainted: G             L 3.18.0+ #103 [loadavg: 238.82 170.06 156.93 185/407 17316]
[ 6244.954854] task: ffff880224af5b40 ti: ffff880225aec000 task.ti: ffff880225aec000
[ 6244.955699] RIP: 0010:[<ffffffff810c5f60>]  [<ffffffff810c5f60>] lock_acquire+0x40/0x120
[ 6244.956560] RSP: 0018:ffff880225aefb78  EFLAGS: 00000246
[ 6244.957418] RAX: ffff880224af5b40 RBX: ffff8802453ce380 RCX: 0000000000000001
[ 6244.958281] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000
[ 6244.959146] RBP: ffff880225aefbd8 R08: 0000000000000001 R09: 0000000000000000
[ 6244.960008] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000001ce380
[ 6244.960867] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880225aefb28
[ 6244.961716] FS:  00007f107fc69740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[ 6244.962571] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6244.963429] CR2: 00007fff7457b07f CR3: 000000022789e000 CR4: 00000000001407e0
[ 6244.964296] DR0: 00007f322081b000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6244.965160] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 6244.966025] Stack:
[ 6244.966881]  0000000005080021 ffffffff00000000 0000000000000000 0000000000000246
[ 6244.967776]  0000000100000000 ffff880187efeb58 0000000000000000 0000000000000029
[ 6244.968678]  00007fff7457b07f ffff880225aefd28 0000000000000002 ffff880187efea40
[ 6244.969564] Call Trace:
[ 6244.970445]  [<ffffffff810c225a>] down_read_trylock+0x5a/0x60
[ 6244.971336]  [<ffffffff81042418>] ? __do_page_fault+0x198/0x5c0
[ 6244.972226]  [<ffffffff81042418>] __do_page_fault+0x198/0x5c0
[ 6244.973125]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6244.974021]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6244.974913]  [<ffffffff8107c25d>] ? __do_softirq+0x1ed/0x310
[ 6244.975800]  [<ffffffff817d09e0>] ? retint_restore_args+0xe/0xe
[ 6244.976685]  [<ffffffff8137511d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 6244.977573]  [<ffffffff8104284c>] do_page_fault+0xc/0x10
[ 6244.978455]  [<ffffffff817d1a32>] page_fault+0x22/0x30
[ 6244.979327]  [<ffffffff8100f408>] ? save_xstate_sig+0x98/0x220
[ 6244.980190]  [<ffffffff8100f3f1>] ? save_xstate_sig+0x81/0x220
[ 6244.981045]  [<ffffffff810029e7>] do_signal+0x5c7/0x740
[ 6244.981892]  [<ffffffff817cf210>] ? _raw_spin_unlock_irq+0x30/0x40
[ 6244.982743]  [<ffffffff81002bc5>] do_notify_resume+0x65/0x80
[ 6244.983582]  [<ffffffff813750de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 6244.984425]  [<ffffffff817d00ff>] int_signal+0x12/0x17
[ 6244.985266] Code: 65 48 8b 04 25 00 aa 00 00 8b b8 6c 07 00 00 44 89 45 c4 85 ff 0f 85 84 00 00 00 41 89 f4 41 89 d5 41 89 ce 4d 89 cf 9c 8f 45 b8 <fa> c7 80 6c 07 00 00 01 00 00 00 0f 1f 44 00 00 65 ff 04 25 e0 
[ 6244.987120] sending NMI to other CPUs:

Hey, this is not "crap" at all. Quite the reverse. I think you may have hit on the real bug now, and it's possible that the DEBUG_PAGEALLOC code was hiding it because it was trying to handle the page fault.

Or something.

Anyway, this time your backtrace is *interesting*. It's showing something that looks real. Namely "save_xstate_sig" apparently taking a page fault. And unlike your earlier traces, now all the different CPU traces show very similar things, which again is something that makes a lot more sense than your previous lockups have.

That said, maybe I'm just being optimistic, because while the NMI watchdog messages now look ostensibly much saner, I'm not actually seeing what's really going on. But at least this time I *could* imagine that it's something like infinitely taking a page fault in save_xstate_sig. This is some pretty special code, with the whole FPU save state handling being one mess of random really subtle issues with FPU exceptions, page faults, delayed allocation, yadda yadda.

And I could fairly easily imagine endless page faults due to the exception table, or even endless signal handling loops due to getting a signal while trying to handle a signal. Both things that would actually reasonably result in a watchdog.

So I'm adding some x86 FPU save people to the cc.

Can anybody make sense of that backtrace, keeping in mind that we're looking for some kind of endless loop where we don't make progress?

There's more in the original email (see on lkml if you haven't seen the thread earlier already), but they look similar with that whole do_signal -> save_xstate_sig -> do_page_fault thing just on other CPU's.

DaveJ, do you have the kernel image for this? I'd love to see what the code is around that "save_xstate_sig+0x81" or around those __clear_user+0x17/0x36 points...

Linus

It was more a "Crap it's wedged and I can't do anything about it for days". I didn't look closely at the actual traces this time.

Not until I get back on Wednesday and reboot it.

Dave

So looking at all the backtraces, which is kind of messy because there's some missing data (presumably buffers overflowed from all the CPU's printing at the same time), it looks like:

- CPU 0 is missing. No idea why.
- CPU's 1-3 all have the same trace for

    int_signal ->
    do_notify_resume ->
    do_signal ->
      ....
    page_fault ->
    do_page_fault

and "save_xstate_sig+0x81" shows up on all stacks, although only on CPU1 does it show up as a "guaranteed" part of the stack chain (ie it matches frame pointer data too). CPU1 also has that __clear_user show up (which is called from save_xstate_sig), but not other CPU's. CPU2 and CPU3 have "save_xstate_sig+0x98" in addition to that +0x81 thing.

My guess is that "save_xstate_sig+0x81" is the instruction after the __clear_user call, and that CPU1 took the fault in __clear_user(), while CPU2 and CPU3 took the fault at "save_xstate_sig+0x98" instead, which I'd guess is the

        xsave64 (%rdi)

and in fact, with CONFIG_FTRACE on, my own kernel build gives exactly those two offsets for those things in save_xstate_sig().

So I'm pretty certain that on all three CPU's, we had page faults for save_xstate_sig() accessing user space, with the only difference being that on CPU1 it happened from __clear_user, while on CPU's 2/3 it happened on the xsaveq instruction itself.

That sounds like much more than coincidence. I have no idea where CPU0 is hiding, and all CPU's were at different stages of actually handling the fault, but that's to be expected if the page fault just keeps repeating.

In fact, CPU2 shows up three different times, and the call trace changes in between, so it's "making progress", just never getting out of that loop. The traces are

    pagecache_get_page+0x0/0x220
    ? lookup_swap_cache+0x2a/0x70
    handle_mm_fault+0x401/0xe90
    ? __do_page_fault+0x198/0x5c0
    __do_page_fault+0x1fc/0x5c0
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    ? __do_softirq+0x1ed/0x310
    ? retint_restore_args+0xe/0xe
    ? trace_hardirqs_off_thunk+0x3a/0x3c
    do_page_fault+0xc/0x10
    page_fault+0x22/0x30
    ? save_xstate_sig+0x98/0x220
    ? save_xstate_sig+0x81/0x220
    do_signal+0x5c7/0x740
    ? _raw_spin_unlock_irq+0x30/0x40
    do_notify_resume+0x65/0x80
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    int_signal+0x12/0x17

and

    ? __lock_acquire.isra.31+0x22c/0x9f0
    ? lock_acquire+0xb4/0x120
    ? __do_page_fault+0x198/0x5c0
    down_read_trylock+0x5a/0x60
    ? __do_page_fault+0x198/0x5c0
    __do_page_fault+0x198/0x5c0
    ? __do_softirq+0x1ed/0x310
    ? retint_restore_args+0xe/0xe
    ? __do_page_fault+0xd8/0x5c0
    ? trace_hardirqs_off_thunk+0x3a/0x3c
    do_page_fault+0xc/0x10
    page_fault+0x22/0x30
    ? save_xstate_sig+0x98/0x220
    ? save_xstate_sig+0x81/0x220
    do_signal+0x5c7/0x740
    ? _raw_spin_unlock_irq+0x30/0x40
    do_notify_resume+0x65/0x80
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    int_signal+0x12/0x17

and

    lock_acquire+0x40/0x120
    down_read_trylock+0x5a/0x60
    ? __do_page_fault+0x198/0x5c0
    __do_page_fault+0x198/0x5c0
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    ? __do_softirq+0x1ed/0x310
    ? retint_restore_args+0xe/0xe
    ? trace_hardirqs_off_thunk+0x3a/0x3c
    do_page_fault+0xc/0x10
    page_fault+0x22/0x30
    ? save_xstate_sig+0x98/0x220
    ? save_xstate_sig+0x81/0x220
    do_signal+0x5c7/0x740
    ? _raw_spin_unlock_irq+0x30/0x40
    do_notify_resume+0x65/0x80
    ? trace_hardirqs_on_thunk+0x3a/0x3f
    int_signal+0x12/0x17

so it's always in __do_page_fault, but at sometimes it has gotten into handle_mm_fault too. So it really really looks like it is taking an endless stream of page faults on that "xsaveq" instruction. Presumably the page faulting never actually makes any progress, even though it *thinks* the page tables are fine.

DaveJ - you've seen that "endless page faults" behavior before. You had a few traces that showed it. That was in that whole "pipe/page fault oddness." email thread, where you would get endless faults in copy_page_to_iter() with an error_code=0x2.

That was the one where I chased it down to "page table entry must be marked with _PAGE_PROTNONE", but VM_WRITE in the vma, because your machine was alive enough that you got traces out of the endless loop.

Very odd.

Linus

We had a flashback to that old bug last month too.
See this mail & your followup. : https://lkml.org/lkml/2014/11/25/1171
That was during a bisect though, so may have been something entirely different, but it is a spooky coincidence.

Dave

Yeah, there's something funny going on there.

Anyway, I've looked at the page fault patch, and I mentioned this last time it came up: there's a nasty possible kernel loop in the "retry" case if there's also a fatal signal pending, and we're returning to kernel mode rather than returning to user mode.

If we return to user mode, the return will handle signals, and we'll kill the process due to the fatal pending signal and everything is fine.

But if we're returning to kernel mode, we'll just take the page fault again. And again. And again. Until the condition that caused the retry is finally cleared.

Now, normally finishing IO on the page or whatever should get things done, but whatever. Us busy-looping on it in kernel space might end up delaying that too forever.

So let's just fix it. Here's a completely untested patch. It looks bigger than it really is: it moves the "up_read()" up a bit in __do_page_fault(), so that all the logic is saner. This is "tested" in the sense that I am running a kernel with this patch, but I could easily have screwed up some fault handling case.

Anyway, at least CPU1 in your traces was actually going through that __lock_page_or_retry() code that could trigger this, so...

Linus

 arch/x86/mm/fault.c | 65 +++++++++++++++++++++++++----------------------------
 1 file changed, 30 insertions(+), 35 deletions(-)

index d973e61e450d..b38adc1cd39f 100644
@@ -844,11 +844,8 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address,
 	  unsigned int fault)
 {
 	struct task_struct *tsk = current;
-	struct mm_struct *mm = tsk->mm;
 	int code = BUS_ADRERR;
 
-	up_read(&mm->mmap_sem);
-
 	/* Kernel mode? Handle exceptions or die: */
 	if (!(error_code & PF_USER)) {
 		no_context(regs, error_code, address, SIGBUS, BUS_ADRERR);

@@ -879,7 +876,6 @@ mm_fault_error(struct pt_regs *regs, unsigned long error_code,
 	       unsigned long address, unsigned int fault)
 {
 	if (fatal_signal_pending(current) && !(error_code & PF_USER)) {
-		up_read(&current->mm->mmap_sem);
 		no_context(regs, error_code, address, 0, 0);
 		return;
 	}

@@ -887,14 +883,11 @@ mm_fault_error(struct pt_regs *regs, unsigned long error_code,
 	if (fault & VM_FAULT_OOM) {
 		/* Kernel mode? Handle exceptions or die: */
 		if (!(error_code & PF_USER)) {
-			up_read(&current->mm->mmap_sem);
 			no_context(regs, error_code, address,
 				   SIGSEGV, SEGV_MAPERR);
 			return;
 		}
 
-		up_read(&current->mm->mmap_sem);
-
 		/*
 		 * We ran out of memory, call the OOM killer, and return the
 		 * userspace (which will retry the fault, or kill us if we got

@@ -1062,7 +1055,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,
 	struct vm_area_struct *vma;
 	struct task_struct *tsk;
 	struct mm_struct *mm;
-	int fault;
+	int fault, major = 0;
 	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 
 	tsk = current;

@@ -1237,14 +1230,31 @@ good_area:
 	 * we get VM_FAULT_RETRY back, the mmap_sem has been unlocked.
 	 */
 	fault = handle_mm_fault(mm, vma, address, flags);
+	major |= fault & VM_FAULT_MAJOR;
 
 	/*
-	 * If we need to retry but a fatal signal is pending, handle the
-	 * signal first. We do not need to release the mmap_sem because it
-	 * would already be released in __lock_page_or_retry in mm/filemap.c.
+	 * If we need to retry the mmap_sem has already been released,
+	 * and if there is a fatal signal pending there is no guarantee
+	 * that we made any progress. Handle this case first.
 	 */
-	if (unlikely((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)))
+	if (unlikely(fault & VM_FAULT_RETRY)) {
+		if ((flags & FAULT_FLAG_ALLOW_RETRY) && !fatal_signal_pending(current)) {
+			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
+			 * of starvation. */
+			flags &= ~FAULT_FLAG_ALLOW_RETRY;
+			flags |= FAULT_FLAG_TRIED;
+			goto retry;
+		}
+
+		/* Not returning to user mode? Handle exceptions or die: */
+		if (!(fault & FAULT_FLAG_USER))
+			no_context(regs, error_code, address, SIGBUS, BUS_ADRERR);
+
+		/* User mode? Just return to handle the fatal exception */
 		return;
+	}
+
+	up_read(&mm->mmap_sem);
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {
 		mm_fault_error(regs, error_code, address, fault);

@@ -1252,32 +1262,17 @@ good_area:
 	}
 
 	/*
-	 * Major/minor page fault accounting is only done on the
-	 * initial attempt. If we go through a retry, it is extremely
-	 * likely that the page will be found in page cache at that point.
+	 * Major/minor page fault accounting. If any of the events
+	 * returned VM_FAULT_MAJOR, we account it as a major fault.
 	 */
-	if (flags & FAULT_FLAG_ALLOW_RETRY) {
-		if (fault & VM_FAULT_MAJOR) {
-			tsk->maj_flt++;
-			perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MAJ, 1,
-				      regs, address);
-		} else {
-			tsk->min_flt++;
-			perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MIN, 1,
-				      regs, address);
-		}
-		if (fault & VM_FAULT_RETRY) {
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation. */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
-			flags |= FAULT_FLAG_TRIED;
-			goto retry;
-		}
+	if (major) {
+		tsk->maj_flt++;
+		perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MAJ, 1, regs, address);
+	} else {
+		tsk->min_flt++;
+		perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MIN, 1, regs, address);
 	}
-
 	check_v8086_mode(regs, address, tsk);
-
-	up_read(&mm->mmap_sem);
 }
 NOKPROBE_SYMBOL(__do_page_fault);
 

So after looking at this more, I'm actually really convinced that this was a pretty nasty bug.

I'm *not* convinced that it's necessarily *your* bug, but I still think it could be.

I cleaned up the patch a bit, split it up into two to clarify it, and have committed it to my tree. I'm not marking the patches for stable, because while I'm convinced it's a bug, I'm also not sure why even if it triggers it doesn't eventually recover when the IO completes. So I'd mark them for stable only if they are actually confirmed to fix anything in the wild, and after they've gotten some testing in general. The patches *look* straightforward, they remove more lines than they add, and I think the code is more understandable too, but maybe I just screwed up. Whatever. Some care is warranted, but this is the first time I feel like I actually fixed something that matched at least one of your lockup symptoms.

Anyway, it's there as

26178ec11ef3 ("x86: mm: consolidate VM_FAULT_RETRY handling") 7fb08eca4527 ("x86: mm: move mmap_sem unlock from mm_fault_error() to caller")

and I'll continue to look at the page fault patch. I still have a slight worry that it's something along the lines of corrupted page tables or some core VM issue, but I apart from my general nervousness about the auto-numa code (which will be cleaned up eventually though the pte_protnone patches), I can't actually see how you'd get into endless page faults any other way. So I'm really hoping that the buggy VM_FAULT_RETRY handling explains it.

But me not seeing any other bug clearly doesn't mean it doesn't exist.

Linus

I guess you did "just screwed up"...

I've started seeing this:

[  240.190061] BUG: unable to handle kernel paging request at 00007f341768b000
[  240.190061] IP: [<00007f341baf61fb>] 0x7f341baf61fb
[  240.190061] PGD 12b3e4067 PUD 12b3e5067 PMD 29a700067 PTE 0
[  240.190061] Oops: 0004 [#10] PREEMPT SMP
[  240.190061] Dumping ftrace buffer:
[  240.190061]    (ftrace buffer empty)
[  240.190061] Modules linked in:
[  240.190061] CPU: 6 PID: 9691 Comm: trinity-c619 Tainted: G      D        3.18.0-sasha-08443-g2b40f4a #1618
[  240.190061] task: ffff88012b346000 ti: ffff88012b3d4000 task.ti: ffff88012b3d4000
[  240.190061] RIP: 0033:[<00007f341baf61fb>]  [<00007f341baf61fb>] 0x7f341baf61fb
[  240.190061] RSP: 002b:00007fff39f045f8  EFLAGS: 00010206
[  240.190061] RAX: 00007fff39f04600 RBX: 0000000000000363 RCX: 0000000000000200
[  240.190061] RDX: 0000000000001000 RSI: 00007f341768b000 RDI: 00007fff39f04600
[  240.190061] RBP: 00007fff39f05640 R08: 00007f341bdf20a8 R09: 00007f341bdf2100
[  240.190061] R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000001000
[  240.190061] R13: 0000000000001000 R14: 0000000000362000 R15: 00007fff39f04600
[  240.190061] FS:  00007f341bffb700(0000) GS:ffff8802da400000(0000) knlGS:0000000000000000
[  240.190061] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  240.190061] CR2: 00007f341894801c CR3: 000000012b364000 CR4: 00000000000006a0
[  240.190061] DR0: ffffffff81000000 DR1: 0000000000000000 DR2: 0000000000000000
[  240.190061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000b0602
[  240.190061]
[  240.190061] RIP  [<00007f341baf61fb>] 0x7f341baf61fb
[  240.190061]  RSP <00007fff39f045f8>
[  240.190061] CR2: 00007f341768b000

Which was bisected down to:

	26178ec11ef3 ("x86: mm: consolidate VM_FAULT_RETRY handling")

Thanks,
Sasha

See the email to Dave, pick the fix from there, or from commit cf3c0a1579ef ("x86: mm: fix VM_FAULT_RETRY handling")

Linus

Bah, I was getting all optimistic.
I came home this evening to a locked up machine.
Serial console had a *lot* more traces than usual though.
Full log below. The 12xxx.xxxxxx traces we seemed to recover from, followed by silence for a while, before the real fun begins at 157xx.xxxxxx

[12525.310743] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [trinity-c60:28574]
[12525.311429] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12525.315877] CPU: 2 PID: 28574 Comm: trinity-c60 Not tainted 3.18.0+ #106
[12525.318136] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12525.318931] RIP: 0010:[<ffffffff81373805>]  [<ffffffff81373805>] copy_user_enhanced_fast_string+0x5/0x10
[12525.319758] RSP: 0018:ffff8801c251fbe0  EFLAGS: 00010206
[12525.320555] RAX: 00007f9c44b07f1d RBX: 0000000000000003 RCX: 00000000000009c0
[12525.321403] RDX: 0000000000001000 RSI: 00007f9c44b0855d RDI: ffff88007cb1f640
[12525.322227] RBP: ffff8801c251fc28 R08: 0000000000000000 R09: 0000000000000001
[12525.323046] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817cf22b
[12525.323854] R13: ffff8801c251fb68 R14: ffff880061dccd40 R15: ffff880061dcce60
[12525.324676] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12525.325468] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.326262] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12525.327065] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.327867] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.328666] Stack:
[12525.329459]  ffffffff8119fe56 ffff8801c251fbf8 140000010718f000 0000000000001000
[12525.330283]  140000010718f000 0000000000001000 ffff8801c251fd60 0000000000000000
[12525.331139]  ffff880061dcd070 ffff8801c251fcc8 ffffffff81175fe7 ffff8801c251fc88
[12525.331969] Call Trace:
[12525.332801]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12525.333642]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12525.334475]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12525.335315]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12525.336149]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12525.336981]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.337809]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12525.338630]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12525.339435]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.340238]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12525.341068]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12525.341865]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12525.342664]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12525.343486]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12525.344292]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12525.345099] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[12525.346820] sending NMI to other CPUs:
[12525.347621] NMI backtrace for cpu 3
[12525.348409] CPU: 3 PID: 25560 Comm: trinity-c201 Not tainted 3.18.0+ #106
[12525.350733] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12525.351503] RIP: 0010:[<ffffffff810fbc4e>]  [<ffffffff810fbc4e>] generic_exec_single+0xee/0x1b0
[12525.352270] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12525.353017] RAX: 0000000000000008 RBX: ffff8801c048fa10 RCX: 0000000000000038
[12525.353757] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12525.354479] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12525.355187] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[12525.355878] R13: 0000000000000001 R14: ffff880061e34fc0 R15: 0000000000000003
[12525.356558] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12525.357236] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.357905] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12525.358577] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.359246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.359911] Stack:
[12525.360568]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12525.361250]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12525.361928]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12525.362601] Call Trace:
[12525.363259]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.363926]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.364589]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12525.365250]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12525.365904]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12525.366556]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12525.367204]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12525.367843]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12525.368476]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12525.369102]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12525.369726]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12525.370348]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12525.370971]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12525.371591]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12525.372210]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12525.372825]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12525.373439]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12525.374061] Code: 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[12525.375458] NMI backtrace for cpu 0
[12525.375565] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 27.836 msecs
[12525.376788] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.0+ #106
[12525.378827] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[12525.379514] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12525.380215] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[12525.380905] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12525.381588] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[12525.382254] RBP: ffffffff81c03e68 R08: 000000008baf8a0b R09: 0000000000000000
[12525.382911] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12525.383558] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[12525.384194] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:00000000000000[12525.398359] NMI backtrace for cpu 1
[12525.398368] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 50.720 msecs
[12525.399764] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.18.0+ #106
[12525.401932] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12525.402687] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12525.403457] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12525.404241] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12525.405038] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12525.405832] RBP: ffff88024374be38 R08: 000000008baf8a0b R09: 0000000000000000
[12525.406611] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12525.407377] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12525.408131] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12525.408881] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12525.409601] CR2: 00000000000000d2 CR3: 0000000001c11000 CR4: 00000000001407e0
[12525.410310] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12525.411011] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12525.411694] Stack:
[12525.412363]  000000014374be38 b532cae23f43ed20 ffffe8ffff003200 0000000000000005
[12525.413051]  ffffffff81cb1bc0 0000000000000001 ffff88024374be88 ffffffff8165f7b5
[12525.413744]  00000b65f893e397 ffffffff81cb1d90 ffffffff81cb1bc0 ffffffff81d215f0
[12525.414439] Call Trace:
[12525.415125]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12525.415824]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12525.416523]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12525.417216]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12525.417904] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12525.419435] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 71.790 msecs
[12549.296847] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [trinity-c60:28574]
[12549.297585] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12549.302426] CPU: 2 PID: 28574 Comm: trinity-c60 Tainted: G             L 3.18.0+ #106
[12549.304874] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12549.305730] RIP: 0010:[<ffffffff81373805>]  [<ffffffff81373805>] copy_user_enhanced_fast_string+0x5/0x10
[12549.306609] RSP: 0018:ffff8801c251fbe0  EFLAGS: 00010206
[12549.307502] RAX: 00007f9c44b07f1d RBX: 0000000000000003 RCX: 00000000000009c0
[12549.308400] RDX: 0000000000001000 RSI: 00007f9c44b0855d RDI: ffff88007cb1f640
[12549.309284] RBP: ffff8801c251fc28 R08: 0000000000000000 R09: 0000000000000001
[12549.310159] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817cf22b
[12549.311018] R13: ffff8801c251fb68 R14: ffff880061dccd40 R15: ffff880061dcce60
[12549.311862] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12549.312716] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.313581] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12549.314418] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12549.315239] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12549.316051] Stack:
[12549.316861]  ffffffff8119fe56 ffff8801c251fbf8 140000010718f000 0000000000001000
[12549.317679]  140000010718f000 0000000000001000 ffff8801c251fd60 0000000000000000
[12549.318514]  ffff880061dcd070 ffff8801c251fcc8 ffffffff81175fe7 ffff8801c251fc88
[12549.319338] Call Trace:
[12549.320153]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12549.320988]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12549.321824]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12549.322658]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12549.323484]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12549.324306]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.325133]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12549.325961]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12549.326783]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.327646]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12549.328470]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12549.329279]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12549.330078]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12549.330874]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12549.331672]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12549.332471] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[12549.334222] sending NMI to other CPUs:
[12549.335023] NMI backtrace for cpu 3
[12549.335811] CPU: 3 PID: 25560 Comm: trinity-c201 Tainted: G             L 3.18.0+ #106
[12549.338137] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12549.338904] RIP: 0010:[<ffffffff810fbc4e>]  [<ffffffff810fbc4e>] generic_exec_single+0xee/0x1b0
[12549.339670] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12549.340416] RAX: 0000000000000008 RBX: ffff8801c048fa10 RCX: 0000000000000038
[12549.341157] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12549.341879] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12549.342587] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[12549.343280] R13: 0000000000000001 R14: ffff880061e34fc0 R15: 0000000000000003
[12549.343961] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12549.344639] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.345309] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12549.345981] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12549.346649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12549.347313] Stack:
[12549.347969]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12549.348648]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12549.349325]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12549.349997] Call Trace:
[12549.350653]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.351319]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.351983]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12549.352644]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12549.353298]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12549.353954]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12549.354604]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12549.355245]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12549.355881]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12549.356508]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12549.357131]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12549.357756]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12549.358379]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12549.359001]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12549.359620]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12549.360239]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12549.360855]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12549.361476] Code: 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[12549.362876] NMI backtrace for cpu 1
[12549.363530] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #106
[12549.365548] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12549.366222] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12549.366902] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12549.367578] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12549.368258] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12549.368920] RBP: ffff88024374be38 R08: 000000008baf89f8 R09: 0000000000000000
[12549.369565] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12549.370201] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12549.370828] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12549.371451] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12549.372069][12563.088868] INFO: rcu_sched self-detected stall on CPU
[12563.089596] 	2: (1 GPs behind) idle=d4b/140000000000001/0 softirq=781780/781781 
[12563.090304] 	 (t=6000 jiffies g=382738 c=382737 q=0)
[12563.090996] Task dump for CPU 2:
[12563.091676] trinity-c60     R  running task    12768 28574  19163 0x0000000c
[12563.092378]  ffff880153a20000 000000005c7b7872 ffff880245203d68 ffffffff810a743c
[12563.093085]  ffffffff810a73a2 0000000000000002 0000000000000004 0000000000000002
[12563.093810]  ffffffff81c52380 0000000000000092 ffff880245203d88 ffffffff810ab4ed
[12563.094507] Call Trace:
[12563.095197]  <IRQ>  [<ffffffff810a743c>] sched_show_task+0x11c/0x190
[12563.095910]  [<ffffffff810a73a2>] ? sched_show_task+0x82/0x190
[12563.096636]  [<ffffffff810ab4ed>] dump_cpu_task+0x3d/0x50
[12563.097336]  [<ffffffff810d9b90>] rcu_dump_cpu_stacks+0x90/0xd0
[12563.098019]  [<ffffffff810e0723>] rcu_check_callbacks+0x503/0x770
[12563.098712]  [<ffffffff8113058c>] ? acct_account_cputime+0x1c/0x20
[12563.099408]  [<ffffffff810abe27>] ? account_system_time+0x97/0x180
[12563.100022]  [<ffffffff810e63fb>] update_process_times+0x4b/0x80
[12563.100623]  [<ffffffff810f6db3>] ? tick_sched_timer+0x23/0x1b0
[12563.101214]  [<ffffffff810f6ddf>] tick_sched_timer+0x4f/0x1b0
[12563.101795]  [<ffffffff810e729f>] __run_hrtimer+0xaf/0x240
[12563.102370]  [<ffffffff810e75ab>] ? hrtimer_interrupt+0x8b/0x260
[12563.102941]  [<ffffffff810f6d90>] ? tick_init_highres+0x20/0x20
[12563.103514]  [<ffffffff810e7627>] hrtimer_interrupt+0x107/0x260
[12563.104086]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[12563.104660]  [<ffffffff817d29c5>] smp_apic_timer_interrupt+0x45/0x60
[12563.105234]  [<ffffffff817d0daf>] apic_timer_interrupt+0x6f/0x80
[12563.105805]  <EOI>  [<ffffffff81373805>] ? copy_user_enhanced_fast_string+0x5/0x10
[12563.106387]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12563.106965]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12563.107545]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12563.108128]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12563.108705]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12563.109299]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.109869]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12563.110440]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12563.111003]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.111566]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12563.112125]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12563.112686]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12563.113248]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12563.113810]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12563.114373]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.290614] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [trinity-c201:25560]
[12577.291259] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12577.295639] CPU: 3 PID: 25560 Comm: trinity-c201 Tainted: G             L 3.18.0+ #106
[12577.297656] task: ffff880095342da0 ti: ffff8801c048c000 task.ti: ffff8801c048c000
[12577.298367] RIP: 0010:[<ffffffff810fbc4a>]  [<ffffffff810fbc4a>] generic_exec_single+0xea/0x1b0
[12577.299091] RSP: 0018:ffff8801c048f9f8  EFLAGS: 00000202
[12577.299825] RAX: 0000000000000008 RBX: ffffffff817d0ae0 RCX: 0000000000000038
[12577.300553] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[12577.301296] RBP: ffff8801c048fa58 R08: ffff8802437418f0 R09: 0000000000000000
[12577.302007] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c048f968
[12577.302726] R13: 0000000000001a4a R14: ffff8801c048c000 R15: ffff880095342da0
[12577.303444] FS:  00007f9c4714c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12577.304169] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.304897] CR2: 0000000001ac9498 CR3: 00000002247cb000 CR4: 00000000001407e0
[12577.305636] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.306386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.307114] Stack:
[12577.307830]  0000000000000000 0000000000000000 ffff8801c048fa78 0000000000000000
[12577.308585]  ffffffff81048cd0 ffff8801c048fb08 0000000000000003 00000000639bd7ce
[12577.309355]  ffff880095342da0 00000000ffffffff 0000000000000002 ffffffff81048cd0
[12577.310106] Call Trace:
[12577.310898]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.311664]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.312440]  [<ffffffff810fbdb0>] smp_call_function_single+0x70/0xd0
[12577.313196]  [<ffffffff81048cd0>] ? do_flush_tlb_all+0x60/0x60
[12577.313935]  [<ffffffff810fc4a9>] smp_call_function_many+0x2b9/0x320
[12577.314684]  [<ffffffff811a396d>] ? unmap_single_vma+0x50d/0x900
[12577.315429]  [<ffffffff81049020>] flush_tlb_mm_range+0x90/0x1d0
[12577.316182]  [<ffffffff811a25b2>] tlb_flush_mmu_tlbonly+0x42/0x50
[12577.316913]  [<ffffffff811a2bac>] tlb_flush_mmu+0x1c/0x30
[12577.317664]  [<ffffffff811a2bd4>] tlb_finish_mmu+0x14/0x40
[12577.318399]  [<ffffffff811a3e78>] zap_page_range_single+0x118/0x160
[12577.319123]  [<ffffffff811a4044>] unmap_mapping_range+0x134/0x190
[12577.319871]  [<ffffffff81192a5d>] shmem_fallocate+0x4fd/0x520
[12577.320624]  [<ffffffff810bcb77>] ? prepare_to_wait+0x27/0x90
[12577.321411]  [<ffffffff811e6062>] do_fallocate+0x132/0x1d0
[12577.322149]  [<ffffffff811b8558>] SyS_madvise+0x398/0x870
[12577.322868]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.323617] Code: c0 3a 1d 00 48 89 de 48 03 14 c5 a0 f2 d1 81 48 89 df e8 fa e0 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[12577.325246] sending NMI to other CPUs:
[12577.325989] NMI backtrace for cpu 2
[12577.326717] CPU: 2 PID: 28574 Comm: trinity-c60 Tainted: G             L 3.18.0+ #106
[12577.328845] task: ffff880153a20000 ti: ffff8801c251c000 task.ti: ffff8801c251c000
[12577.329548] RIP: 0010:[<ffffffff810c523d>]  [<ffffffff810c523d>] __lock_acquire.isra.31+0x33d/0x9f0
[12577.330250] RSP: 0018:ffff880245203d88  EFLAGS: 00000097
[12577.330933] RAX: 0000000000000000 RBX: ffff880153a20000 RCX: ffff8802453cff98
[12577.331610] RDX: 0000000000000980 RSI: 0000000000000010 RDI: 0000000000000000
[12577.332272] RBP: ffff880245203df8 R08: 0000000000000001 R09: 0000000000000000
[12577.332924] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000097
[12577.333564] R13: 0000000000000002 R14: ffff8802453cc898 R15: ffff880153a207e0
[12577.334195] FS:  00007f9c4714c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12577.334831] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.335469] CR2: 00007f9c45b905b8 CR3: 000000022a30b000 CR4: 00000000001407e0
[12577.336116] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.336759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.337399] Stack:
[12577.338024]  ffff880153a20000 0000000000000046 ffff880245203dd8 0000000000000092
[12577.338671]  ffff8802453d2f18 ffff8802453d2f00 0000000000000002 0000000000000000
[12577.339313]  ffff880245203dc8 0000000000000046 0000000000000000 0000000000000000
[12577.339955] Call Trace:
[12577.340585]  <IRQ> 
[12577.340593]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[12577.341830]  [<ffffffff810e72c6>] ? __run_hrtimer+0xd6/0x240
[12577.342450]  [<ffffffff810f6db3>] ? tick_sched_timer+0x23/0x1b0
[12577.343065]  [<ffffffff817cee9e>] _raw_spin_lock+0x3e/0x80
[12577.343673]  [<ffffffff810e72c6>] ? __run_hrtimer+0xd6/0x240
[12577.344277]  [<ffffffff810e72c6>] __run_hrtimer+0xd6/0x240
[12577.344886]  [<ffffffff810e75ab>] ? hrtimer_interrupt+0x8b/0x260
[12577.345493]  [<ffffffff810f6d90>] ? tick_init_highres+0x20/0x20
[12577.346099]  [<ffffffff810e7627>] hrtimer_interrupt+0x107/0x260
[12577.346703]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[12577.347309]  [<ffffffff817d29c5>] smp_apic_timer_interrupt+0x45/0x60
[12577.347916]  [<ffffffff817d0daf>] apic_timer_interrupt+0x6f/0x80
[12577.348522]  <EOI> 
[12577.348529]  [<ffffffff81373805>] ? copy_user_enhanced_fast_string+0x5/0x10
[12577.349750]  [<ffffffff8119fe56>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[12577.350375]  [<ffffffff81175fe7>] generic_perform_write+0xf7/0x1f0
[12577.351001]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12577.351626]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12577.352253]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12577.352883]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.353515]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12577.354147]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12577.354763]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.355366]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12577.355956]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12577.356550]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12577.357141]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12577.357713]  [<ffffffff811e98d2>] SyS_pwritev+0xc2/0xf0
[12577.358267]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12577.358815] Code: 00 00 48 83 c4 48 44 89 e8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 41 81 fc fe 1f 00 00 0f 87 34 04 00 00 <45> 85 ed 48 8b 83 60 07 00 00 75 09 48 85 c0 0f 85 df 03 00 00 
[12577.360038] NMI backtrace for cpu 0
[12577.360598] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #106
[12577.381697] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #106
[12577.383862] task: ffff8802434b96d0 ti: ffff880243748000 task.ti: ffff880243748000
[12577.384585] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12577.385332] RSP: 0018:ffff88024374be08  EFLAGS: 00000046
[12577.386066] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[12577.386807] RDX: 0000000000000000 RSI: ffff88024374bfd8 RDI: 0000000000000001
[12577.387545] RBP: ffff88024374be38 R08: 000000008baf89e1 R09: 0000000000000000
[12577.388285] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[12577.389028] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243748000
[12577.389752] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12577.390463] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12577.391170] CR2: 00007f5271c07000 CR3: 0000000001c11000 CR4: 00000000001407e0
[12577.391888] DR0: 00007f6c9030b000 DR1: 0000000000000000 DR2: 0000000000000000
[12577.392584] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12577.393258] Stack:
[12577.393915]  000000014374be38 b532cae23f43ed20 ffffe8ffff003200 0000000000000005
[12577.394595]  ffffffff81cb1bc0 0000000000000001 ffff88024374be88 ffffffff8165f7b5
[12577.395278]  00000b72149ea4e1 ffffffff81cb1d90 ffffffff81cb1bc0 ffffffff81d215f0
[12577.395972] Call Trace:
[12577.396643]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12577.397327]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12577.398010]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12577.398691]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12577.399364] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12577.400864] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 74.851 msecs
[12591.532403] sched: RT throttling activated
[12613.269774] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [trinity-c12:30629]
[12613.270528] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12613.275557] CPU: 3 PID: 30629 Comm: trinity-c12 Tainted: G             L 3.18.0+ #106
[12613.278212] task: ffff8802433b5b40 ti: ffff880180f34000 task.ti: ffff880180f34000
[12613.279152] RIP: 0010:[<ffffffff8117e2df>]  [<ffffffff8117e2df>] free_hot_cold_page+0x11f/0x1a0
[12613.280117] RSP: 0018:ffff880180f37b38  EFLAGS: 00000202
[12613.281030] RAX: 0000000000000003 RBX: 0000000000000002 RCX: 0000000000000240
[12613.281939] RDX: ffff88024540d380 RSI: 0000000000000000 RDI: ffff88024e5d3f40
[12613.282817] RBP: ffff880180f37b78 R08: 0000000000000000 R09: 000000000024e600
[12613.283701] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88024e5d3f40
[12613.284572] R13: 0000000180f37b08 R14: ffffffff8117df5b R15: ffff880180f37b28
[12613.285454] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12613.286408] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.287338] CR2: 00007f688baa9000 CR3: 00000002276f9000 CR4: 00000000001407e0
[12613.288228] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.289102] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.289957] Stack:
[12613.290804]  00000000001ddf58 ffff88024e5d3e80 ffff8802433b5b40 ffffea00076a8540
[12613.291675]  ffffea000777d600 0000000000000000 ffffea00062ec580 ffff880180f37c00
[12613.292574]  ffff880180f37bb8 ffffffff8117e3c5 000000000000000d ffffea00076a8540
[12613.293471] Call Trace:
[12613.294325]  [<ffffffff8117e3c5>] free_hot_cold_page_list+0x65/0xd0
[12613.295179]  [<ffffffff81184f2d>] release_pages+0x1bd/0x270
[12613.296027]  [<ffffffff81185f63>] __pagevec_release+0x43/0x60
[12613.296869]  [<ffffffff81191f80>] shmem_undo_range+0x460/0x710
[12613.297704]  [<ffffffff81192248>] shmem_truncate_range+0x18/0x40
[12613.298541]  [<ffffffff811924d6>] shmem_setattr+0x116/0x1a0
[12613.299383]  [<ffffffff812075d1>] notify_change+0x241/0x390
[12613.300250]  [<ffffffff811e5ad5>] do_truncate+0x75/0xc0
[12613.301103]  [<ffffffff811e5e4a>] ? do_sys_ftruncate.constprop.14+0xda/0x160
[12613.301962]  [<ffffffff811e5e7f>] do_sys_ftruncate.constprop.14+0x10f/0x160
[12613.302815]  [<ffffffff81374f6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[12613.303656]  [<ffffffff811e5f0e>] SyS_ftruncate+0xe/0x10
[12613.304497]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12613.305362] Code: 8d 7b 20 e8 64 3b 20 00 41 8b 04 24 83 c0 01 41 3b 44 24 04 41 89 04 24 7d 52 41 f7 c6 00 02 00 00 74 31 e8 94 bf fc ff 41 56 9d <48> 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 90 49 8d 34 04 
[12613.307230] sending NMI to other CPUs:
[12613.308088] NMI backtrace for cpu 2
[12613.308871] CPU: 2 PID: 30778 Comm: trinity-c161 Tainted: G             L 3.18.0+ #106
[12613.311213] task: ffff8800478eada0 ti: ffff8802251dc000 task.ti: ffff8802251dc000
[12613.311995] RIP: 0010:[<ffffffff810c96bd>]  [<ffffffff810c96bd>] do_raw_spin_trylock+0x2d/0x50
[12613.312776] RSP: 0018:ffff8802251dfd88  EFLAGS: 00000246
[12613.313536] RAX: 0000000000008f8f RBX: ffff880235610238 RCX: 0000000000000000
[12613.314287] RDX: 0000000000008f8f RSI: 000000000000908f RDI: ffff880235610238
[12613.315022] RBP: ffff8802251dfd88 R08: 0000000000000000 R09: 0000000000000000
[12613.315743] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880235610250
[12613.316446] R13: ffff88009f3cc640 R14: 0000000000000000 R15: ffff8802356103c0
[12613.317133] FS:  00007f688fc33740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12613.317813] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.318483] CR2: 000000288076098d CR3: 000000022a0d2000 CR4: 00000000001407e0
[12613.319155] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.319819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.320475] Stack:
[12613.321122]  ffff8802251dfdb8 ffffffff817ceea6 ffffffff81214c6a ffffffff817cf22b
[12613.321796]  ffff8802356101b0 ffff880235610238 ffff8802251dff08 ffffffff81214c6a
[12613.322474]  00000000251dfde8 7fffffffffffffff ffff88009f3cc520 ffff88017b9a3c28
[12613.323153] Call Trace:
[12613.323815]  [<ffffffff817ceea6>] _raw_spin_lock+0x46/0x80
[12613.324479]  [<ffffffff81214c6a>] ? sync_inodes_sb+0x1ca/0x2b0
[12613.325138]  [<ffffffff817cf22b>] ? _raw_spin_unlock+0x2b/0x40
[12613.325798]  [<ffffffff81214c6a>] sync_inodes_sb+0x1ca/0x2b0
[12613.326458]  [<ffffffff817cb01f>] ? wait_for_completion+0xff/0x130
[12613.327114]  [<ffffffff8121c560>] ? vfs_fsync+0x40/0x40
[12613.327761]  [<ffffffff8121c579>] sync_inodes_one_sb+0x19/0x20
[12613.328404]  [<ffffffff811ec002>] iterate_supers+0xb2/0x110
[12613.329041]  [<ffffffff8121c824>] sys_sync+0x44/0xb0
[12613.329669]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12613.330292] Code: 44 00 00 0f b7 17 55 31 c9 48 89 e5 38 d6 74 0e 89 c8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 8d b2 00 01 00 00 89 d0 f0 66 0f b1 37 <66> 39 d0 75 e0 b1 01 5d 65 8b 04 25 2c a0 00 00 89 47 08 65 48 
[12613.331704] NMI backtrace for cpu 0
[12613.332380] CPU: 0 PID: 1460 Comm: trinity-c155 Tainted: G             L 3.18.0+ #106
[12613.334512] task: ffff880232a72da0 ti: ffff880232820000 task.ti: ffff880232820000
[12613.335239] RIP: 0033:[<000000336ee93d7d>]  [<000000336ee93d7d>] 0x336ee93d7d
[12613.335984] RSP: 002b:00007fff9d870a18  EFLAGS: 00000206
[12613.336708] RAX: ffffff68f0ef8610 RBX: 0000000000000200 RCX: 00007fff9d870f00
[12613.337440] RDX: 00007fff9d871a00 RSI: 00007f688e769020 RDI: 00007fff9d870a20
[12613.338173] RBP: 00007fff9d871a60 R08: ffffff68f0ef8600 R09: ffffff68f0ef85f0
[12613.338905] R10: ffffff68f0ef85e0 R11: 0000000000000246 R12: 0000000000000129
[12613.339627] R13: 0000000000001000 R14: 0000000000001000 R15: 00007f688e6a9000
[12613.340332] FS:  00007f688fc33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[12613.341040] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.341753] CR2: 00007f688e702000 CR3: 00000001c9675000 CR4: 00000000001407f0
[12613.342468] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.343161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.343836] 
[12613.344498] NMI backtrace for cpu 1
[12613.345187] CPU: 1 PID: 30783 Comm: trinity-c166 Tainted: G             L 3.18.0+ #106
[12613.347247] task: ffff8801c059ada0 ti: ffff880227058000 task.ti: ffff880227058000
[12613.347978] RIP: 0033:[<000000336ee39bc0>]  [<000000336ee39bc0>] 0x336ee39bc0
[12613.348715] RSP: 002b:00007fff9d8719d8  EFLAGS: 00000202
[12613.349440] RAX: e8ffff3573f747e9 RBX: 00007f688f22e000 RCX: 0000000000000000
[12613.350175] RDX: 8000000000000000 RSI: 00007fff9d8719cc RDI: 000000336f1b76e0
[12613.350903] RBP: 00007f688f22e068 R08: 000000336f1b7120 R09: 000000336f1b7140
[12613.351639] R10: ffffffffffff9f00 R11: 0000000000000202 R12: 00007f688fbb7528
[12613.352362] R13: 00007f688f22e068 R14: 0000000000000000 R15: 0000000000000000
[12613.353089] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12613.353814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12613.354537] CR2: 0000000000000008 CR3: 0000000153ac1000 CR4: 00000000001407e0
[12613.355265] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12613.355989] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12613.356701] 
[12641.253566] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [trinity-main:30108]
[12641.254492] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12641.260655] CPU: 3 PID: 30108 Comm: trinity-main Tainted: G             L 3.18.0+ #106
[12641.263755] task: ffff8800a0d15b40 ti: ffff880234ec4000 task.ti: ffff880234ec4000
[12641.264753] RIP: 0010:[<ffffffff811729d7>]  [<ffffffff811729d7>] task_bp_pinned.isra.6.constprop.12+0x37/0x90
[12641.265768] RSP: 0018:ffff880234ec7a40  EFLAGS: 00000287
[12641.266763] RAX: ffff88023f199d18 RBX: 000000000000005c RCX: 00000000001ce490
[12641.267751] RDX: ffff8801c87c2fa8 RSI: ffff8801ad860000 RDI: 0000000000000007
[12641.268728] RBP: ffff880234ec7a70 R08: ffffffff81d215f8 R09: ffff880245dce490
[12641.269700] R10: 0000000000000000 R11: 0000000000000008 R12: ffff880234ec7a30
[12641.270657] R13: ffffffff810abaf5 R14: ffff880234ec79b0 R15: ffffea000252c000
[12641.271618] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12641.272585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.273547] CR2: 000000336f1b8490 CR3: 0000000229035000 CR4: 00000000001407e0
[12641.274502] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.275443] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 000000000017060a
[12641.276381] Stack:
[12641.277317]  0000000000000002 ffff8800a0d15b40 ffff880234ec7aa0 0000000000000001
[12641.278267]  ffff880094b06618 0000000000000007 ffff880234ec7ad0 ffffffff81172af6
[12641.279214]  ffffffff81c64868 0000000100000246 00000000001ce490 ffffffff81d215f8
[12641.280161] Call Trace:
[12641.281101]  [<ffffffff81172af6>] toggle_bp_slot.constprop.7+0xc6/0x1b0
[12641.282055]  [<ffffffff81172dfb>] __reserve_bp_slot+0x1eb/0x230
[12641.283017]  [<ffffffff81172e77>] reserve_bp_slot+0x27/0x40
[12641.283972]  [<ffffffff81172f68>] register_perf_hw_breakpoint+0x18/0x60
[12641.284884]  [<ffffffff81172fdf>] hw_breakpoint_event_init+0x2f/0x50
[12641.285766]  [<ffffffff8116fcfb>] perf_init_event+0x17b/0x1e0
[12641.286650]  [<ffffffff8116fb80>] ? perf_bp_event+0xd0/0xd0
[12641.287512]  [<ffffffff811700f8>] perf_event_alloc+0x398/0x440
[12641.288373]  [<ffffffff810e3c1c>] ? do_init_timer+0x5c/0x60
[12641.289239]  [<ffffffff81170e3e>] inherit_event.isra.90+0x8e/0x260
[12641.290095]  [<ffffffff8117104e>] inherit_task_group.isra.92.part.93+0x3e/0xe0
[12641.290959]  [<ffffffff811716e3>] perf_event_init_task+0x163/0x2e0
[12641.291812]  [<ffffffff81075756>] copy_process.part.26+0x726/0x1a40
[12641.292664]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12641.293514]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12641.294347]  [<ffffffff810424b2>] ? __do_page_fault+0x232/0x610
[12641.295158]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[12641.295969]  [<ffffffff811ea0b1>] ? __fput+0x191/0x200
[12641.296791]  [<ffffffff81374fe4>] ? lockdep_sys_exit_thunk+0x35/0x67
[12641.297597]  [<ffffffff81077066>] SyS_clone+0x16/0x20
[12641.298395]  [<ffffffff817d0299>] stub_clone+0x69/0x90
[12641.299173]  [<ffffffff817cff12>] ? system_call_fastpath+0x12/0x17
[12641.299935] Code: 54 53 48 83 ec 18 48 8b 05 57 1f af 00 48 8d 98 c0 fe ff ff 48 3d 10 49 c6 81 74 4c 41 89 fd 45 31 e4 eb 17 48 8b 93 40 01 00 00 <48> 8d 9a c0 fe ff ff 48 81 fa 10 49 c6 81 74 39 48 3b b3 28 01 
[12641.301567] sending NMI to other CPUs:
[12641.302311] NMI backtrace for cpu 1
[12641.303001] CPU: 1 PID: 3545 Comm: trinity-c7 Tainted: G             L 3.18.0+ #106
[12641.305035] task: ffff8800963f96d0 ti: ffff880215b28000 task.ti: ffff880215b28000
[12641.305715] RIP: 0010:[<ffffffff810fc466>]  [<ffffffff810fc466>] smp_call_function_many+0x276/0x320
[12641.306405] RSP: 0000:ffff880215b2baa8  EFLAGS: 00000202
[12641.307074] RAX: 0000000000000003 RBX: ffff8802451d3b00 RCX: ffff8802455d9488
[12641.307748] RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
[12641.308423] RBP: ffff880215b2baf8 R08: ffff880243741e30 R09: 0000000100180011
[12641.309097] R10: ffff880244804240 R11: 0000000000000000 R12: 0000000000000003
[12641.309770] R13: 0000000000000000 R14: 0000000000000008 R15: 0000000000000008
[12641.310436] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12641.311099] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.311757] CR2: 0000000002258ff8 CR3: 0000000180fec000 CR4: 00000000001407e0
[12641.312420] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.313082] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12641.313744] Stack:
[12641.314390]  ffff88024e5d4d00 0000000000000001 0000000100000003 00000000001d3ac0
[12641.315059]  0000004000000000 ffffffff82bea298 0000000000000001 ffffffff8117e150
[12641.315726]  0000000000000000 0000000000000000 ffff880215b2bb28 ffffffff810fc652
[12641.316393] Call Trace:
[12641.317045]  [<ffffffff8117e150>] ? drain_pages+0xc0/0xc0
[12641.317698]  [<ffffffff810fc652>] on_each_cpu_mask+0x32/0xa0
[12641.318351]  [<ffffffff8117c371>] drain_all_pages+0x101/0x120
[12641.319003]  [<ffffffff8118024f>] __alloc_pages_nodemask+0x7af/0xb40
[12641.319657]  [<ffffffff811c8ebe>] alloc_pages_vma+0xee/0x1b0
[12641.320310]  [<ffffffff811a416a>] ? do_wp_page+0xca/0x770
[12641.320959]  [<ffffffff811a416a>] do_wp_page+0xca/0x770
[12641.321608]  [<ffffffff811a69bb>] handle_mm_fault+0x6cb/0xe90
[12641.322258]  [<ffffffff8104241d>] ? __do_page_fault+0x19d/0x610
[12641.322910]  [<ffffffff8104248c>] __do_page_fault+0x20c/0x610
[12641.323557]  [<ffffffff811aa2dc>] ? validate_mm+0x15c/0x2c0
[12641.324204]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12641.324847]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12641.325494]  [<ffffffff81374fad>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[12641.326142]  [<ffffffff8104289c>] do_page_fault+0xc/0x10
[12641.326785]  [<ffffffff817d1b32>] page_fault+0x22/0x30
[12641.327423] Code: 00 41 89 c4 39 f0 0f 8d 27 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 a0 f2 d1 81 f6 41 18 01 74 14 0f 1f 44 00 00 f3 90 f6 41 18 01 <75> f8 48 63 35 05 63 c2 00 83 f8 ff 48 8b 7b 08 74 b0 39 c6 77 
[12641.328871] NMI backtrace for cpu 0
[12641.329555] CPU: 0 PID: 30823 Comm: trinity-c206 Tainted: G             L 3.18.0+ #106
[12641.331677] task: ffff8801ad862da0 ti: ffff88017d7d0000 task.ti: ffff88017d7d0000
[12641.332396] RIP: 0010:[<ffffffff810fc466>]  [<ffffffff810fc466>] smp_call_function_many+0x276/0x320
[12641.333117] RSP: 0000:ffff88017d7d3a18  EFLAGS: 00000202
[12641.333827] RAX: 0000000000000003 RBX: ffff880244fd3b00 RCX: ffff8802455d7228
[12641.334546] RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
[12641.335264] RBP: ffff88017d7d3a68 R08: ffff88024483b650 R09: 0000000100180011
[12641.335968] R10: ffff880244804240 R11: 0000000000000000 R12: 0000000000000003
[12641.336653] R13: 0000000000000000 R14: 0000000000000008 R15: 0000000000000008
[12641.337327] FS:  00007f688fc33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[12641.338000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.338665] CR2: 0000000002258ff8 CR3: 0000000232b93000 CR4: 00000000001407f0
[126[12641.356633] NMI backtrace for cpu 2
[12641.357349] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #106
[12641.359555] task: ffff8802434bc470 ti: ffff880243758000 task.ti: ffff880243758000
[12641.360314] RIP: 0010:[<ffffffff813d6cab>]  [<ffffffff813d6cab>] intel_idle+0xdb/0x180
[12641.361070] RSP: 0000:ffff88024375be08  EFLAGS: 00000046
[12641.361812] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[12641.362561] RDX: 0000000000000000 RSI: ffff88024375bfd8 RDI: 0000000000000002
[12641.363289] RBP: ffff88024375be38 R08: 000000008baf89af R09: 0000000000000000
[12641.363998] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[12641.364680] R13: 0000000000000000 R14: 0000000000000001 R15: ffff880243758000
[12641.365350] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12641.366017] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12641.366678] CR2: 0000000002258ff8 CR3: 0000000097dd1000 CR4: 00000000001407e0
[12641.367347] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12641.368016] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[12641.368680] Stack:
[12641.369340]  000000024375be38 5dccb806cd27b931 ffffe8ffff203200 0000000000000001
[12641.370021]  ffffffff81cb1bc0 0000000000000002 ffff88024375be88 ffffffff8165f7b5
[12641.370710]  00000b80fc4e5dd1 ffffffff81cb1c30 ffffffff81cb1bc0 ffffffff81d215f0
[12641.371403] Call Trace:
[12641.372073]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[12641.372758]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[12641.373441]  [<ffffffff810bd665>] cpu_startup_entry+0x355/0x410
[12641.374108]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[12641.374769] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[12669.237353] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [trinity-c59:7816]
[12669.238086] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[12669.242957] CPU: 3 PID: 7816 Comm: trinity-c59 Tainted: G             L 3.18.0+ #106
[12669.245468] task: ffff8802258796d0 ti: ffff88007df64000 task.ti: ffff88007df64000
[12669.246378] RIP: 0010:[<ffffffff810c6014>]  [<ffffffff810c6014>] lock_acquire+0xb4/0x120
[12669.247281] RSP: 0018:ffff88007df67da8  EFLAGS: 00000246
[12669.248181] RAX: ffff8802258796d0 RBX: ffffffff810c512c RCX: ffff8802455cff98
[12669.249074] RDX: 00000000000001e0 RSI: 0000000000000010 RDI: 0000000000000000
[12669.249977] RBP: ffff88007df67e08 R08: 0000000000000000 R09: 0000000000000000
[12669.250881] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810abaf5
[12669.251790] R13: ffff88007df67d18 R14: ffffffff810abaf5 R15: ffff88007df67d08
[12669.252693] FS:  00007f688fc33740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[12669.253585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.254486] CR2: 000000336f1b7740 CR3: 00000000a0ef4000 CR4: 00000000001407e0
[12669.255431] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.256368] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.257263] Stack:
[12669.258097]  ffffffff81086b05 0000000000000000 0000000000000000 0000000000000246
[12669.258933]  0000000000000002 ffffffff81c50e20 ffff88007df67e48 00000000000024ca
[12669.259768]  0000000000000000 ffff88007df67eb0 ffff8800953896d0 0000000000000000
[12669.260638] Call Trace:
[12669.261471]  [<ffffffff81086b05>] ? group_send_sig_info+0x5/0xc0
[12669.262318]  [<ffffffff81086b48>] group_send_sig_info+0x48/0xc0
[12669.263170]  [<ffffffff81086b05>] ? group_send_sig_info+0x5/0xc0
[12669.264028]  [<ffffffff81086d05>] kill_pid_info+0x65/0xb0
[12669.264945]  [<ffffffff81086ca5>] ? kill_pid_info+0x5/0xb0
[12669.265851]  [<ffffffff81086e3c>] SYSC_kill+0xcc/0x240
[12669.266674]  [<ffffffff81086df8>] ? SYSC_kill+0x88/0x240
[12669.267508]  [<ffffffff8107aebb>] ? SyS_wait4+0x8b/0x110
[12669.268328]  [<ffffffff81088cde>] SyS_kill+0xe/0x10
[12669.269145]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12669.269958] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[12669.271728] sending NMI to other CPUs:
[12669.272548] NMI backtrace for cpu 1
[12669.273314] CPU: 1 PID: 7878 Comm: trinity-c219 Tainted: G             L 3.18.0+ #106
[12669.275686] task: ffff880096860000 ti: ffff880097f2c000 task.ti: ffff880097f2c000
[12669.276489] RIP: 0010:[<ffffffff811037c2>]  [<ffffffff811037c2>] is_module_text_address+0x22/0x30
[12669.277296] RSP: 0018:ffff880097f2f9e8  EFLAGS: 00000046
[12669.278089] RAX: 0000000000000000 RBX: ffff880097f2fd70 RCX: 0000000000000004
[12669.278876] RDX: ffff880227e01b80 RSI: 0000000000000000 RDI: ffff880097f2fd70
[12669.279650] RBP: ffff880097f2fa00 R08: ffff880097f2fb40 R09: 0000000000000000
[12669.280408] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[12669.281147] R13: ffff880097f2fd80 R14: ffff880097f2fd50 R15: ffffffff81801b30
[12669.281874] FS:  00007f688fc33740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[12669.282595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.283299] CR2: 0000000001cb9a28 CR3: 00000000948d4000 CR4: 00000000001407e0
[12669.283997] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.284677] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.285348] Stack:
[12669.286003]  ffffffff810973c8 0000000000000000 ffff880097f2fd70 ffff880097f2fa70
[12669.286678]  ffffffff81006a7f ffff880097f2fa20 ffff880097f2fff8 ffff880097f2c000
[12669.287346]  ffff880097f2fd80 ffff880097f2fb40 ffffffffffffc000 0000000000000000
[12669.288011] Call Trace:
[12669.288672]  [<ffffffff810973c8>] ? __kernel_text_address+0x58/0x80
[12669.289343]  [<ffffffff81006a7f>] print_context_stack+0x8f/0x100
[12669.290009]  [<ffffffff8100559f>] dump_trace+0x16f/0x350
[12669.290668]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.291321]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.291963]  [<ffffffff810135bf>] save_stack_trace+0x2f/0x50
[12669.292603]  [<ffffffff811ce860>] set_track+0x70/0x140
[12669.293240]  [<ffffffff817c26de>] alloc_debug_processing+0x92/0x118
[12669.293872]  [<ffffffff817c33b2>] __slab_alloc+0x4da/0x58f
[12669.294498]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.295119]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[12669.295745]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.296362]  [<ffffffff811b22a9>] ? anon_vma_clone+0x49/0x140
[12669.296969]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[12669.297577]  [<ffffffff811b22a9>] anon_vma_clone+0x49/0x140
[12669.298185]  [<ffffffff811b23cd>] anon_vma_fork+0x2d/0x100
[12669.298790]  [<ffffffff81076392>] copy_process.part.26+0x1362/0x1a40
[12669.299398]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.300009]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[12669.300619]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[12669.301232]  [<ffffffff81374fe4>] ? lockdep_sys_exit_thunk+0x35/0x67
[12669.301850]  [<ffffffff81077066>] SyS_clone+0x16/0x20
[12669.302466]  [<ffffffff817d0299>] stub_clone+0x69/0x90
[12669.303062]  [<ffffffff817cff12>] ? system_call_fastpath+0x12/0x17
[12669.303646] Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 65 ff 04 25 e0 a9 00 00 e8 4a b7 ff ff 65 ff 0c 25 e0 a9 00 00 48 85 c0 5d <0f> 95 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 
[12669.304961] NMI backtrace for cpu 0
[12669.305579] CPU: 0 PID: 5863 Comm: trinity-c230 Tainted: G             L 3.18.0+ #106
[12669.307401] task: ffff88023f830000 ti: ffff880096b74000 task.ti: ffff880096b74000
[12669.308023] RIP: 0010:[<ffffffff81191f24>]  [<ffffffff81191f24>] shmem_undo_range+0x404/0x710
[12669.308649] RSP: 0000:ffff880096b77c78  EFLAGS: 00000246
[12669.309266] RAX: 002ffe000008003d RBX: 000000000000000c RCX: 0000000000000835
[12669.309888] RDX: ffffea0005e915c0 RSI: ffffea0005e91580 RDI:[12669.325471] NMI backtrace for cpu 2
[12669.326185] CPU: 2 PID: 7188 Comm: trinity-c210 Tainted: G             L 3.18.0+ #106
[12669.328331] task: ffff8800a0d14470 ti: ffff8801c96d4000 task.ti: ffff8801c96d4000
[12669.329085] RIP: 0010:[<ffffffff810c4f21>]  [<ffffffff810c4f21>] __lock_acquire.isra.31+0x21/0x9f0
[12669.329858] RSP: 0018:ffff8801c96d79b8  EFLAGS: 00000096
[12669.330627] RAX: ffffffff81185068 RBX: ffff8800a0d14470 RCX: 0000000000000000
[12669.331406] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88024e5d3c58
[12669.332191] RBP: ffff8801c96d7a28 R08: 0000000000000001 R09: 0000000000000000
[12669.332984] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[12669.333780] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[12669.334577] FS:  00007f688fc33740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[12669.335370] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12669.336152] CR2: 00007f688d8a9000 CR3: 00000002271f7000 CR4: 00000000001407e0
[12669.336934] DR0: 00007f688e6a9000 DR1: 0000000000000000 DR2: 0000000000000000
[12669.337707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[12669.338474] Stack:
[12669.339224]  ffff8801c96d79f8 0000000093c05f9b 00000000000000d0 ffffffff82c00cc0
[12669.340023]  ffff8800a0d14470 00000000000200da ffff8800a0d15780 0000000000000000
[12669.340803]  ffff8801c96d7a08 0000000000000046 0000000000000000 0000000000000000
[12669.341541] Call Trace:
[12669.342270]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[12669.342998]  [<ffffffff81185068>] ? pagevec_lru_move_fn+0x88/0x100
[12669.343728]  [<ffffffff817cf0a9>] _raw_spin_lock_irqsave+0x49/0x90
[12669.344455]  [<ffffffff81185068>] ? pagevec_lru_move_fn+0x88/0x100
[12669.345190]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12669.345916]  [<ffffffff81185068>] pagevec_lru_move_fn+0x88/0x100
[12669.346640]  [<ffffffff81185160>] ? __pagevec_lru_add+0x20/0x20
[12669.347374]  [<ffffffff81185811>] __lru_cache_add+0x71/0x90
[12669.348097]  [<ffffffff81185bf9>] lru_cache_add_anon+0x19/0x20
[12669.348811]  [<ffffffff81190b08>] shmem_getpage_gfp+0x528/0x7a0
[12669.349559]  [<ffffffff81190dc2>] shmem_write_begin+0x42/0x70
[12669.350260]  [<ffffffff81175fc4>] generic_perform_write+0xd4/0x1f0
[12669.350957]  [<ffffffff81178722>] __generic_file_write_iter+0x162/0x350
[12669.351655]  [<ffffffff811e7940>] ? new_sync_read+0xd0/0xd0
[12669.352348]  [<ffffffff8117894f>] generic_file_write_iter+0x3f/0xb0
[12669.353037]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.353738]  [<ffffffff811e7a88>] do_iter_readv_writev+0x78/0xc0
[12669.354434]  [<ffffffff811e92b8>] do_readv_writev+0xd8/0x2a0
[12669.355126]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.355828]  [<ffffffff81178910>] ? __generic_file_write_iter+0x350/0x350
[12669.356531]  [<ffffffff810c361f>] ? lock_release_holdtime.part.24+0xf/0x190
[12669.357228]  [<ffffffff817cf310>] ? _raw_spin_unlock_irq+0x30/0x40
[12669.357921]  [<ffffffff811e950c>] vfs_writev+0x3c/0x50
[12669.358596]  [<ffffffff811e967c>] SyS_writev+0x5c/0x100
[12669.359258]  [<ffffffff817cff12>] system_call_fastpath+0x12/0x17
[12669.359905] Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 48 44 8b 25 48 80 be 00 65 48 8b 1c 25 00 aa 00 00 <45> 85 e4 0f 84 ef 00 00 00 44 8b 1d e7 32 ab 01 49 89 fe 41 89 
[12669.361354] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 88.798 msecs
[15739.449422] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15739.450087] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15739.454475] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15739.456686] Workqueue: khelper __call_usermodehelper
[15739.457473] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15739.458231] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15739.459015] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15739.459794] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15739.460570] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15739.461343] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15739.462113] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15739.462885] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15739.463648] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15739.464431] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.465203] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15739.465985] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.466766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.467542] Stack:
[15739.468308]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15739.469106]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15739.469912]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15739.470717] Call Trace:
[15739.471511]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15739.472325]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15739.473138]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15739.473947]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15739.474757]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15739.475571]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15739.476374]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15739.477174]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15739.477983]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15739.478781]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15739.479590]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15739.480387]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15739.481184]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15739.481983]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15739.482797]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15739.483598]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15739.484382]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15739.485160]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15739.485922]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15739.486669]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15739.487407]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15739.488126]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15739.488834]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15739.489529]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15739.490197]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15739.490845]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15739.491474]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15739.492092]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15739.492689]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15739.493288] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15739.494617] sending NMI to other CPUs:
[15739.495221] NMI backtrace for cpu 3
[15739.495787] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15739.497539] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15739.498153] RIP: 0010:[<ffffffff810c6014>]  [<ffffffff810c6014>] lock_acquire+0xb4/0x120
[15739.498772] RSP: 0018:ffff880180e07dd8  EFLAGS: 00000246
[15739.499382] RAX: ffff8801adac4470 RBX: 0000000000000246 RCX: ffff8802455cff98
[15739.499994] RDX: 00000000000006a0 RSI: 0000000000000000 RDI: 0000000000000000
[15739.500609] RBP: ffff880180e07e38 R08: 0000000000000000 R09: 0000000000000000
[15739.501224] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15739.501829] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
[15739.502422] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15739.503021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.503621] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15739.504217] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.504812] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.505403] Stack:
[15739.505988]  ffffffff8107a8e9 0000000000000000 ffff8801adac4470 0000000000000246
[15739.506603]  0000000127cc4da0 ffffffff81c0a098 ffff8801adac4470 ffffffff81c0a080
[15739.507220]  ffffffff81c0a098 ffff8801adac4470 ffff8801adac4470 ffff8801adac4470
[15739.507839] Call Trace:
[15739.508448]  [<ffffffff8107a8e9>] ? do_wait+0xd9/0x280
[15739.509064]  [<ffffffff817cf3d1>] _raw_read_lock+0x41/0x80
[15739.509678]  [<ffffffff8107a8e9>] ? do_wait+0xd9/0x280
[15739.510293]  [<ffffffff8107a8e9>] do_wait+0xd9/0x280
[15739.510907]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15739.511518]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15739.512130]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15739.512739] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15739.514121] NMI backtrace for cpu 1
[15739.514769] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15739.516771] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15739.517467] RIP: 0010:[<ffffffff810961f9>]  [<ffffffff810961f9>] find_pid_ns+0x39/0x90
[15739.518177] RSP: 0018:ffff880153a4fe78  EFLAGS: 00000207
[15739.518884] RAX: ffff88024e517120 RBX: 0000000000000d7f RCX: 0000000000000034
[15739.519599] RDX: ffff880094be6500 RSI: ffffffff81c486c0 RDI: 0000000000000d7f
[15739.520316] RBP: ffff880153a4fe78 R08: 0000000000000000 R09: 0000000000000000
[15739.521036] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15739.521752] R13: ffff880096b4c470 R14: 0000000000000000 R15: 0000000000000000
[15739.522464] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15739.523181] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15739.523882] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15739.524580] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15739.525269] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15739.525957] Stack:
[15739.526632]  ffff880153a4fe88 ffffffff8109627f ffff880153a4ff68 ffffffff81086e2a
[15739.527319]  ffffffff81086df8 0000000000000000 ffff880096b4c840 0000000000000000
[15739.527992]  ffff880100000000 000003e800000b21 ffff880094be64c0 00007fff123fb870
[15739.528656] Call Trace:
[15739.529301]  [<ffffffff8109627f>] find_vpid+0x2f/0x50
[15739.529947]  [<ffffffff81086e2a>] SYSC_kill+0xba/0x240
[15739.530587]  [<ffffffff81086df8>] ? SYSC_kill+0x88/0x240
[15739[15763.435537] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15763.436252] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15763.441025] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15763.443320] Workqueue: khelper __call_usermodehelper
[15763.444090] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15763.444861] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15763.445654] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15763.446422] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15763.447201] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15763.447985] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15763.448764] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15763.449545] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15763.450319] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15763.451100] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.451876] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15763.452662] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.453451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.454230] Stack:
[15763.455003]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15763.455818]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15763.456608]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15763.457393] Call Trace:
[15763.458166]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15763.458960]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15763.459753]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15763.460544]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15763.461338]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15763.462127]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15763.462921]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15763.463707]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15763.464503]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15763.465294]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15763.466086]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15763.466875]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15763.467664]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15763.468458]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15763.469251]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15763.470041]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15763.470828]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15763.471603]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15763.472371]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15763.473120]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15763.473854]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15763.474578]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15763.475287]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15763.475975]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15763.476640]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15763.477285]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15763.477915]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15763.478532]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15763.479131]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15763.479741] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15763.481087] sending NMI to other CPUs:
[15763.481693] NMI backtrace for cpu 3
[15763.482255] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15763.484005] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15763.484618] RIP: 0010:[<ffffffff810c47f5>]  [<ffffffff810c47f5>] lock_acquired+0x45/0x370
[15763.485234] RSP: 0018:ffff880180e07db8  EFLAGS: 00000046
[15763.485841] RAX: 0000000000000001 RBX: ffff880227cc4da0 RCX: 0000000000000001
[15763.486453] RDX: 000000000000dbdb RSI: ffffffff810bcc8d RDI: ffff880227cc4db8
[15763.487068] RBP: ffff880180e07df8 R08: 0000000000000000 R09: 0000000000000001
[15763.487687] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801adac4470
[15763.488303] R13: ffff880227cc4db8 R14: 0000000000000046 R15: ffff8801adac4460
[15763.488909] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15763.489522] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.490133] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15763.490747] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.491356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.491964] Stack:
[15763.492564]  0000000100000002 ffffffff810bcc8d ffff880180e07dd8 ffff880227cc4da0
[15763.493194]  ffff880227cc4db8 0000000000000292 ffff8801adac4470 ffff8801adac4460
[15763.493828]  ffff880180e07e38 ffffffff817cf0d5 ffffffff810bcc8d 0000000000000296
[15763.494464] Call Trace:
[15763.495088]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15763.495726]  [<ffffffff817cf0d5>] _raw_spin_lock_irqsave+0x75/0x90
[15763.496364]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15763.497005]  [<ffffffff810bcc8d>] remove_wait_queue+0x1d/0x40
[15763.497645]  [<ffffffff8107a95b>] do_wait+0x14b/0x280
[15763.498283]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15763.498917]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15763.499551]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15763.500183] Code: b8 00 45 85 c9 0f 84 d8 00 00 00 65 4c 8b 24 25 00 aa 00 00 45 8b 84 24 6c 07 00 00 45 85 c0 0f 85 be 00 00 00 49 89 fd 9c 41 5e <fa> 8b 35 1c 3a ab 01 41 c7 84 24 6c 07 00 00 01 00 00 00 41 8b 
[15763.501611] NMI backtrace for cpu 1
[15763.502274] CPU: 1 PID: 3298 Comm: trinity-c183 Tainted: G             L 3.18.0+ #106
[15763.504345] task: ffff880227de16d0 ti: ffff880071060000 task.ti: ffff880071060000
[15763.505065] RIP: 0033:[<000000336eebc2fc>]  [<000000336eebc2fc>] 0x336eebc2fc
[15763.505796] RSP: 002b:00007fff123fb868  EFLAGS: 00000246
[15763.506522] RAX: 0000000000000000 RBX: 0000000000000d7b RCX: ffffffffffffffff
[15763.507257] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7b
[15763.507994] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15763.508729] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7996cf6000
[15763.509460] R13: 00007f7996cf6068 R14: 0000000000000000 R15: 0000000000000000
[15763.510188] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15763.510908] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15763.511612] CR2: 00007f7996f24220 CR3: 000000009a646000 CR4: 00000000001407e0
[15763.512319] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15763.513027] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15763.513727] 
[15763.514403] NMI backtrace for cpu 0
[15763.515067] CPU: 0 PID: 1876 Comm: trinity-c189 Tainted: G             L 3.18.0+ #106
[15763.517087] task: ffff880096b4ada0 ti: ffff8802253d0000 task.ti: ffff8802253d0000
[15763.517776] RIP: 0010:[<[15779.306349] INFO: rcu_sched detected stalls on CPUs/tasks:
[15779.307024] 	(detected by 0, t=6002 jiffies, g=481360, c=481359, q=0)
[15779.307662] INFO: Stall ended before state dump start
[15787.421647] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u16:3:14112]
[15787.422333] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15787.426946] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15787.429181] Workqueue: khelper __call_usermodehelper
[15787.429945] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15787.430719] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15787.431507] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15787.432297] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15787.433094] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15787.433886] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15787.434671] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15787.435452] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15787.436234] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15787.437028] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.437816] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15787.438605] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.439400] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.440183] Stack:
[15787.440959]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15787.441792]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15787.442615]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15787.443422] Call Trace:
[15787.444217]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15787.445027]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15787.445833]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15787.446647]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15787.447449]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15787.448258]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15787.449067]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15787.449879]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15787.450688]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15787.451497]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15787.452307]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15787.453122]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15787.453928]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15787.454733]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15787.455524]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15787.456299]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15787.457062]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15787.457826]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15787.458587]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15787.459336]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15787.460078]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15787.460794]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15787.461495]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15787.462182]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15787.462850]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15787.463496]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15787.464124]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15787.464742]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15787.465341]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15787.465947] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15787.467291] sending NMI to other CPUs:
[15787.467904] NMI backtrace for cpu 3
[15787.468463] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15787.470211] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15787.470824] RIP: 0033:[<000000336eebc2fc>]  [<000000336eebc2fc>] 0x336eebc2fc
[15787.471440] RSP: 002b:00007fff123fb868  EFLAGS: 00000246
[15787.472043] RAX: 0000000000000000 RBX: 0000000000000d7e RCX: ffffffffffffffff
[15787.472652] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7e
[15787.473259] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15787.473868] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7997265000
[15787.474468] R13: 00007f7997265068 R14: 0000000000000000 R15: 0000000000000000
[15787.475063] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15787.475660] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.476254] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15787.476851] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.477442] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.478032] 
[15787.478615] NMI backtrace for cpu 1
[15787.479205] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15787.481052] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15787.481696] RIP: 0010:[<ffffffff817cfea0>]  [<ffffffff817cfea0>] system_call+0x0/0x3
[15787.482351] RSP: 0018:00007fff123fb868  EFLAGS: 00000046
[15787.483007] RAX: 000000000000003d RBX: 0000000000000d7f RCX: 000000336eebc2fc
[15787.483674] RDX: 000000000000000b RSI: 00007fff123fb870 RDI: 0000000000000d7f
[15787.484338] RBP: 0000000000000000 R08: 00007f79977d8740 R09: 0000000000000000
[15787.485000] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f799716e000
[15787.485658] R13: 00007f799716e068 R14: 0000000000000000 R15: 0000000000000000
[15787.486314] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15787.486979] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15787.487641] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15787.488311] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15787.488982] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15787.489650] Stack:
[15787.490312]  000000000041401b 00007f79976944e8 00007f799716e000 0000000000000001
[15787.490999]  00007f799716e07c 00007f799716e000 0000000000000000 00007f799716e07c
[15787.491690]  0000000000416c03 00000000000066b2 0000000000000155 00000000cccccccd
[15787.492383] Call Trace:
[15787.493063]  <UNK> 
[15787.493070] Code: 8b 3c 24 4c 8b 74 24 08 4c 8b 6c 24 10 4c 8b 64 24 18 48 8b 6c 24 20 48 8b 5c 24 28 48 83 c4 30 e9 74 02 00 00 66 0f 1f 44 00 00 <0f> 01 f8 65 48 89 24 25 80 a0 00 00 65 48 8b 24 25 08 aa 00 00 
[15787.495270] NMI backtrace for cpu 0
[15787.495987] CPU: 0 PID: 1876 Comm: trinity-c189 Tainted: G             L 3.18.0+ #106
[15787.498136] task: ffff880096b4ada0 ti: ffff8802253d0000 task.ti: ffff8802253d0000
[15787.498870] RIP: 0010:[<ffffffff810c9698>]  [<ffffffff810c9698>] do_raw_spin_trylock+0x8/0x50
[15787.499618] RSP: 0018:ffff880244e03d00  EFLAGS: 00000092
[15787.500344] RAX: ffff880096b4ada0 RBX: ffff880240c51578 RCX: ffff880244fcff98
[15787.501064] RDX: 0000000000004a4a RSI: 0000000000000018 RDI: ffff880240c51578
[15787.501780] RBP: ffff880244e03d38 R08: 0000000000000001 R09: 0000000000000000
[15787.502487] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880240c51590
[15787.503190] R13: 0000000000000092 R14: ff[15811.407761] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15811.408334] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15811.412296] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15811.414324] Workqueue: khelper __call_usermodehelper
[15811.415025] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15811.415733] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15811.416455] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15811.417171] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15811.417913] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15811.418645] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15811.419374] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15811.420102] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15811.420831] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15811.421572] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.422317] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15811.423070] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.423828] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.424583] Stack:
[15811.425327]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15811.426099]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15811.426872]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15811.427657] Call Trace:
[15811.428437]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15811.429231]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15811.430025]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15811.430814]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15811.431600]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15811.432394]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15811.433180]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15811.433972]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15811.434768]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15811.435559]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15811.436349]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15811.437141]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15811.437937]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.438725]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.439519]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15811.440307]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15811.441094]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.441871]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15811.442637]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15811.443385]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15811.444121]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15811.444838]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15811.445538]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15811.446219]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15811.446883]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15811.447527]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.448159]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15811.448776]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15811.449371]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15811.449968] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15811.451292] sending NMI to other CPUs:
[15811.451901] NMI backtrace for cpu 3
[15811.452459] CPU: 3 PID: 1650 Comm: trinity-c76 Tainted: G             L 3.18.0+ #106
[15811.454206] task: ffff8801adac4470 ti: ffff880180e04000 task.ti: ffff880180e04000
[15811.454817] RIP: 0010:[<ffffffff810c511a>]  [<ffffffff810c511a>] __lock_acquire.isra.31+0x21a/0x9f0
[15811.455437] RSP: 0018:ffff880180e07d18  EFLAGS: 00000002
[15811.456045] RAX: 0000000000000008 RBX: ffff8801adac4470 RCX: 0000000000000000
[15811.456660] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[15811.457274] RBP: ffff880180e07d88 R08: 0000000000000001 R09: 0000000000000000
[15811.457889] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000014e
[15811.458493] R13: 0000000000000000 R14: ffff880227cc4db8 R15: ffff8801adac4be0
[15811.459092] FS:  00007f79977d8740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15811.459699] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.460303] CR2: 0000000000000001 CR3: 00000001c9593000 CR4: 00000000001407e0
[15811.460907] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.461507] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.462108] Stack:
[15811.462702]  ffff880180e07d98 ffffffff810c512c 0000000000000102 0000000000000000
[15811.463325]  ffff880180e07d48 ffffffff810abaf5 ffff880180e07dc8 0000000000000000
[15811.463945]  ffff880180e07dd8 0000000000000046 0000000000000000 0000000000000000
[15811.464564] Call Trace:
[15811.465173]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15811.465797]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15811.466421]  [<ffffffff810c5fff>] lock_acquire+0x9f/0x120
[15811.467047]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15811.467672]  [<ffffffff817cf0a9>] _raw_spin_lock_irqsave+0x49/0x90
[15811.468300]  [<ffffffff810bcc8d>] ? remove_wait_queue+0x1d/0x40
[15811.468924]  [<ffffffff810bcc8d>] remove_wait_queue+0x1d/0x40
[15811.469546]  [<ffffffff8107a95b>] do_wait+0x14b/0x280
[15811.470166]  [<ffffffff8107aeb0>] SyS_wait4+0x80/0x110
[15811.470782]  [<ffffffff81078990>] ? task_stopped_code+0x60/0x60
[15811.471398]  [<ffffffff817d0109>] tracesys_phase2+0xd4/0xd9
[15811.472011] Code: e0 7f 44 09 d0 41 88 47 31 41 0f b6 47 32 83 e0 f0 45 85 c0 0f 95 c2 09 c8 c1 e2 03 09 d0 41 88 47 32 0f b7 55 18 41 0f b7 47 32 <c1> e2 04 83 e0 0f 09 d0 66 41 89 47 32 e8 a4 69 fe ff 4c 8b 4d 
[15811.473409] NMI backtrace for cpu 1
[15811.474061] CPU: 1 PID: 2849 Comm: trinity-c95 Tainted: G             L 3.18.0+ #106
[15811.476098] task: ffff880096b4c470 ti: ffff880153a4c000 task.ti: ffff880153a4c000
[15811.476808] RIP: 0010:[<ffffffff810c63cf>]  [<ffffffff810c63cf>] lock_release+0x1f/0x240
[15811.477535] RSP: 0018:ffff880153a4fe40  EFLAGS: 00000246
[15811.478253] RAX: ffff880096b4c470 RBX: 0000000000000000 RCX: 00000000000003a0
[15811.478982] RDX: ffffffff81086d0c RSI: 0000000000000001 RDI: ffffffff81c50e20
[15811.479707] RBP: ffff880153a4fe48 R08: 0000000000000000 R09: 0000000000000000
[15811.480419] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880094be64c0
[15811.481115] R13: ffff880153a4feb0 R14: 0000000000000000 R15: 0000000000000000
[15811.481803] FS:  00007f79977d8740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15811.482504] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15811.483200] CR2: 00007f7996f24220 CR3: 00000002251b4000 CR4: 00000000001407e0
[15811.483890] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15811.484562] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15811.485221] Stack:
[15811.485862]  0000000000000000 ffff880153a4fe88 ffffffff81086d24 ffffffff81086ca5
[15811.486521]  0000000000000d7f 0000000000000d7f 0000000000000000 ffff880096[15835.393872] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u16:3:14112]
[15835.394432] Modules linked in: bridge 8021q garp stp snd_seq_dummy dlci tun fuse rfcomm hidp bnep af_key llc2 nfnetlink can_bcm scsi_transport_iscsi can_raw sctp libcrc32c nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 usb_debug cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e ptp pps_core snd_timer snd soundcore shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15835.398378] CPU: 2 PID: 14112 Comm: kworker/u16:3 Tainted: G             L 3.18.0+ #106
[15835.400379] Workqueue: khelper __call_usermodehelper
[15835.401080] task: ffff8801c95f0000 ti: ffff880227eac000 task.ti: ffff880227eac000
[15835.401795] RIP: 0010:[<ffffffff817c3407>]  [<ffffffff817c3407>] __slab_alloc+0x52f/0x58f
[15835.402534] RSP: 0018:ffff880227eaf8f8  EFLAGS: 00000246
[15835.403252] RAX: 0000000000000002 RBX: ffff8802304cf5c8 RCX: 00000000000002e0
[15835.403995] RDX: ffff88024520d7e0 RSI: 0000000000000000 RDI: ffff880244802000
[15835.404727] RBP: ffff880227eaf9e8 R08: 0000000000000000 R09: 0000000000000000
[15835.405456] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810135bf
[15835.406185] R13: ffff880227eaf878 R14: 0000000100160015 R15: ffffffff8138278d
[15835.406916] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15835.407667] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.408410] CR2: 0000000000000008 CR3: 0000000225ab9000 CR4: 00000000001407e0
[15835.409165] DR0: 00007fbe591ef000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.409924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.410679] Stack:
[15835.411433]  000000000000005c ffff880240f8d790 ffff880240f8d790 ffff880240f8dd00
[15835.412226]  0000000180230020 000000010000000f ffffffff8112ee12 0000000000000000
[15835.413002]  ffff8802453d7260 000000020023001f ffff880227eaf968 ffffffff8138278d
[15835.413784] Call Trace:
[15835.414563]  [<ffffffff8112ee12>] ? __delayacct_tsk_init+0x22/0x50
[15835.415358]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15835.416162]  [<ffffffff811ce860>] ? set_track+0x70/0x140
[15835.416952]  [<ffffffff811cf35d>] ? init_object+0x3d/0x70
[15835.417745]  [<ffffffff8138278d>] ? __debug_object_init+0x43d/0x450
[15835.418533]  [<ffffffff811d295b>] kmem_cache_alloc+0x1cb/0x1f0
[15835.419328]  [<ffffffff8138278d>] __debug_object_init+0x43d/0x450
[15835.420114]  [<ffffffff813827bb>] debug_object_init+0x1b/0x20
[15835.420904]  [<ffffffff810e66d5>] hrtimer_init+0x25/0xb0
[15835.421701]  [<ffffffff8109f069>] __sched_fork+0x99/0x230
[15835.422491]  [<ffffffff810a59c9>] sched_fork+0x29/0x200
[15835.423282]  [<ffffffff8107568c>] copy_process.part.26+0x65c/0x1a40
[15835.424077]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15835.424865]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15835.425658]  [<ffffffff8108e340>] ? call_helper+0x20/0x20
[15835.426447]  [<ffffffff81076c37>] do_fork+0xe7/0x490
[15835.427239]  [<ffffffff810c512c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15835.428018]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15835.428776]  [<ffffffff81077006>] kernel_thread+0x26/0x30
[15835.429525]  [<ffffffff8108e1b4>] __call_usermodehelper+0x64/0x80
[15835.430259]  [<ffffffff8109301a>] process_one_work+0x1fa/0x550
[15835.430986]  [<ffffffff81092f98>] ? process_one_work+0x178/0x550
[15835.431694]  [<ffffffff8109348b>] worker_thread+0x11b/0x490
[15835.432374]  [<ffffffff81093370>] ? process_one_work+0x550/0x550
[15835.433049]  [<ffffffff81098c89>] kthread+0xf9/0x110
[15835.433694]  [<ffffffff810abaf5>] ? local_clock+0x25/0x30
[15835.434328]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15835.434947]  [<ffffffff817cfe6c>] ret_from_fork+0x7c/0xb0
[15835.435543]  [<ffffffff81098b90>] ? kthread_create_on_node+0x250/0x250
[15835.436148] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 7a 6d 98 ff 4c 89 e0 eb 0f e8 70 6e 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 22 46 
[15835.437494] sending NMI to other CPUs:

CPU 2 seems to be the one making the least progress. I think he's calling fork and then trying to allocate a debug object for his hrtimer, eventually wandering into fill_pool from __debug_object_init():

static void fill_pool(void)

{
       gfp_t gfp = GFP_ATOMIC | __GFP_NORETRY | __GFP_NOWARN;
       struct debug_obj *new;
       unsigned long flags;

       if (likely(obj_pool_free >= ODEBUG_POOL_MIN_LEVEL))
               return;

       if (unlikely(!obj_cache))
               return;

       while (obj_pool_free < ODEBUG_POOL_MIN_LEVEL) {

               new = kmem_cache_zalloc(obj_cache, gfp);
               if (!new)
                       return;

               raw_spin_lock_irqsave(&pool_lock, flags);
               hlist_add_head(&new->node, &obj_pool);
               obj_pool_free++;
               raw_spin_unlock_irqrestore(&pool_lock, flags);
       }
}

It doesn't seem to be making progress out of __slab_alloc+0x52f/0x58f, but maybe the slab code is just a victim of being called in a while loop with GFP_ATOMIC set from a starvation prone loop. Can you please line up where 0x52f is in __slab_alloc?

It might be fun to run with CONFIG_DEBUG_OBJECTS off...Linus' patch clearly helped, I think we're off in a different bug now.

http://codemonkey.org.uk/junk/slub.txt is the whole disassembly. is that at 10ac, the end of an inlined get_freepointer ? If so, the looping thing sounds plausible.

That was going to be my next move in absense of any better ideas.

It certainly not showing the xsave traces any more, which is a good sign. I just this happen..

[36195.185301] WARNING: CPU: 2 PID: 23893 at kernel/watchdog.c:290 watchdog_overflow_callback+0x9c/0xd0()
[36195.185333] Watchdog detected hard LOCKUP on cpu 2
[36195.185347] Modules linked in:
[36195.185363]  8021q garp dlci bridge stp snd_seq_dummy fuse tun rfcomm bnep llc2 af_key hidp can_raw nfnetlink sctp libcrc32c can_bcm nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds scsi_transport_iscsi rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel usb_debug snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm shpchp snd_timer snd e1000e ptp soundcore pps_core nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[36195.186809] CPU: 2 PID: 23893 Comm: modprobe Not tainted 3.18.0+ #106
[36195.189025]  0000000000000000 00000000ac0a0aab ffff880245205b98 ffffffff817c4aef
[36195.190140]  0000000000000000 ffff880245205bf0 ffff880245205bd8 ffffffff81077c61
[36195.191266]  0000000000000000 ffff880244852520 0000000000000000 ffff880245205d30
[36195.192380] Call Trace:
[36195.193482]  <NMI>  [<ffffffff817c4aef>] dump_stack+0x4e/0x68
[36195.194603]  [<ffffffff81077c61>] warn_slowpath_common+0x81/0xa0
[36195.195722]  [<ffffffff81077cd5>] warn_slowpath_fmt+0x55/0x70
[36195.196810]  [<ffffffff8112bcf0>] ? restart_watchdog_hrtimer+0x60/0x60
[36195.197898]  [<ffffffff8112bd8c>] watchdog_overflow_callback+0x9c/0xd0
[36195.198982]  [<ffffffff8116ebfd>] __perf_event_overflow+0x9d/0x2a0
[36195.200058]  [<ffffffff8116d7d3>] ? perf_event_update_userpage+0x103/0x180
[36195.201137]  [<ffffffff8116d6d0>] ? perf_event_task_disable+0x90/0x90
[36195.202291]  [<ffffffff8116f7d4>] perf_event_overflow+0x14/0x20
[36195.203501]  [<ffffffff8101e739>] intel_pmu_handle_irq+0x1f9/0x3f0
[36195.204696]  [<ffffffff81017cab>] perf_event_nmi_handler+0x2b/0x50
[36195.205881]  [<ffffffff81007320>] nmi_handle+0xc0/0x1b0
[36195.207018]  [<ffffffff81007265>] ? nmi_handle+0x5/0x1b0
[36195.208119]  [<ffffffff8100760a>] default_do_nmi+0x4a/0x140
[36195.209191]  [<ffffffff810077c0>] do_nmi+0xc0/0x100
[36195.210259]  [<ffffffff817d1efa>] end_repeat_nmi+0x1e/0x2e
[36195.211323]  <<EOE>>  <UNK> 
[36195.211335] ---[ end trace b7e2af452c79e16a ]---
[36195.213538] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 28.230 msecs
[36195.214595] perf interrupt took too long (223362 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[36225.945721] perf interrupt took too long (221634 > 10000), lowering kernel.perf_event_max_sample_rate to 12500
[36243.519426] INFO: rcu_sched detected stalls on CPUs/tasks:
[36243.520799] 	2: (0 ticks this GP) idle=3b5/140000000000000/0 softirq=2159968/2159968 
[36243.522120] 	(detected by 0, t=6002 jiffies, g=972859, c=972858, q=0)
[36243.523406] Task dump for CPU 2:
[36243.524699] swapper/2       R  running task    14576     0      1 0x00200000
[36243.526016]  000000024375be38 ffffffffffffff10 ffffffff8165f7d9 0000000000000001
[36243.527326]  ffffffff81cb1bc0 0000000000000002 ffff88024375be88 ffffffff8165f7b5
[36243.528664]  000020fb799d5682 ffffffff81cb1c30 00000000001cc300 ffffffff81d215f0
[36243.530015] Call Trace:
[36243.531328]  [<ffffffff810ed5f4>] ? ktime_get+0x94/0x120
[36243.532647]  [<ffffffff8165f7b5>] cpuidle_enter_state+0x55/0x190
[36243.533974]  [<ffffffff8165f9a7>] cpuidle_enter+0x17/0x20
[36243.535290]  [<ffffffff810bd4a4>] cpu_startup_entry+0x194/0x410
[36243.536590]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230

Which didn't lock up the box, but did taint the kernel so it stopped the fuzzer. That <UNK> trace is pretty hopeless.

Dave

So this is interesting. So far today with that change, I've not been able to reproduce the "NMI watchdog kicks in, and then locks up the box", but I keep hitting lockups which cause the kernel to taint, and then the fuzzer would usually quit. I hacked something up so it would ignore that, and keep going, and it spews a lot of traces, but it does actually seem to be staying up.

Example of the spew-o-rama below.

Dave

[15351.809622] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15351.809723] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15351.811011] CPU: 1 PID: 20128 Comm: trinity-c195 Not tainted 3.18.0+ #107
[15351.813048] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15351.814094] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15351.815142] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15351.816174] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15351.817211] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15351.818242] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15351.819265] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15351.820314] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15351.821343] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15351.822373] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15351.823401] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15351.824436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15351.825473] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15351.826507] Stack:
[15351.827536]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15351.828596]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15351.829683]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15351.830752] Call Trace:
[15351.831807]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15351.832873]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15351.833942]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15351.834999]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15351.836044]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15351.837077]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15351.838105]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15351.839141]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15351.840210]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15351.841244]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15351.842280]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15351.843311]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15351.844337]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15351.845358]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15351.846384] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15351.848623] sending NMI to other CPUs:
[15351.849722] NMI backtrace for cpu 2
[15351.850796] CPU: 2 PID: 22823 Comm: trinity-c154 Not tainted 3.18.0+ #107
[15351.852950] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15351.854042] RIP: 0010:[<ffffffff810fb22a>]  [<ffffffff810fb22a>] generic_exec_single+0xea/0x1b0
[15351.855149] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15351.856246] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15351.857348] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15351.858449] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15351.859558] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15351.860668] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15351.861787] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15351.862915] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15351.864023] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15351.865121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15351.866204] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15351.867269] Stack:
[15351.868316]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15351.869381]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15351.870445]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15351.871506] Call Trace:
[15351.872550]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15351.873579]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15351.874582]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15351.875564]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15351.876525]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15351.877469]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15351.878397]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15351.879303]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15351.880192]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15351.881072]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15351.881949]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15351.882818]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15351.883684]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15351.884552]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15351.885416]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15351.886279] Code: c0 3a 1d 00 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[15351.888165] NMI backtrace for cpu 0
[15351.888278] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 38.443 msecs
[15351.889964] CPU: 0 PID: 24440 Comm: trinity-c229 Not tainted 3.18.0+ #107
[15351.891825] task: ffff880240c14470 ti: ffff880095bb4000 task.ti: ffff880095bb4000
[15351.892771] RIP: 0010:[<ffffffff810c6155>]  [<ffffffff810c6155>] lock_release+0xc5/0x240
[15351.893724] RSP: 0018:ffff880095bb7e98  EFLAGS: 00000296
[15351.894666] RAX: ffff880240c14470 RBX: ffff8802236cc380 RCX: 0000000000000360
[15351.895616] RDX: ffff880244e0db60 RSI: 0000000000000000 RDI: ffff880240c14be0
[15351.896562] RBP: ffff880095bb7eb8 R08: 0000000000000000 R09: 0000000000000000
[15351.897507] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81c50da0
[15351.898452] R13: ffffffff810961fd R14: 0000000000000296 R15: 0000000000000001
[15351.899401] FS:  00007f9dcd485740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15351.900361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15351.901318] CR2: 0000000000000008 CR3: 000000009afbc000 CR4: 00000000001407f0
[15351.902288] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15351.903254] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15351.904220] Stack:
[15351.905181]  00007f9dcc74d000 00007fffe01fcb90 0000000000000000 0000000000000000
[15351.906167]  ffff880095bb7ee8 ffffffff81096215 ffffffff810961b5 00007fffe01fcb90
[15351.907153]  0000000000000000 000000000000000b ffff880095bb7f78 ffffffff8107ae75
[15351.908147] Call Trace:
[15351.909131]  [<ffffffff81096215>] find_get_pid+0x65/0x80
[15351.910101]  [<ffffffff810961b5>] ? find_get_pid+0x5/0x80
[15351.911051]  [<ffffffff8107ae75>] SyS_wait4+0x55/0x110
[15351.911995]  [<ffffffff8137432e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[15351.912944]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15351.913886] Code: 00 00 4c 89 ea 4c 89 e6 48 89 df e8 26 fc ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 41 56 9d 48 83 c4 08 5b <41> 5c 41 5d 41 5e 41 5f 5d f3 c3 65 ff 04 25 e0 a9 00 00 48 8b 
[15351.915903] NMI backtrace for cpu 3
[15351.915906] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 66.180 msecs
[15351.917854] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.18.0+ #107
[15351.919856] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15351.920877] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15351.921916] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15351.922951] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15351.923981] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15351.925008] RBP: ffff88024372be38 R08: 000000008baf5f56 R09: 0000000000000000
[15351.926027] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15351.927053] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15351.928080] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15351.929111] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15351.930133] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15351.931161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15351.932184] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15351.933197] Stack:
[15351.934198]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15351.935238]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15351.936280]  00000df86c4d4975 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15351.937324] Call Trace:
[15351.938358]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15351.939406]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15351.940448]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15351.941495]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15351.942533] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15351.944799] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 95.053 msecs

[15375.795837] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15375.796930] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15375.802851] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15375.805152] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15375.806342] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15375.807487] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15375.808618] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15375.809753] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15375.810881] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15375.812007] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15375.813131] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15375.814262] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15375.815399] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15375.816583] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15375.817716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15375.818842] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15375.819963] Stack:
[15375.821085]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15375.822227]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15375.823368]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15375.824506] Call Trace:
[15375.825630]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15375.826798]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15375.827930]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15375.829057]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15375.830190]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15375.831320]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15375.832445]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15375.833558]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15375.834647]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15375.835727]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15375.836821]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15375.837883]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15375.838926]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15375.839948]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15375.840965] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15375.843169] sending NMI to other CPUs:
[15375.844211] NMI backtrace for cpu 2
[15375.845266] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15375.847416] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15375.848501] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15375.849601] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15375.850691] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15375.851789] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15375.852887] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15375.853982] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15375.855070] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15375.856156] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15375.857242] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15375.858325] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15375.859414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15375.860503] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15375.861585] Stack:
[15375.862664]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15375.863747]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15375.864811]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15375.865862] Call Trace:
[15375.866891]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15375.867913]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15375.868904]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15375.869873]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15375.870823]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15375.871751]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15375.872665]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15375.873561]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15375.874438]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15375.875309]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15375.876176]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15375.877032]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15375.877890]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15375.878743]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15375.879599]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15375.880452] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15375.882317] NMI backtrace for cpu 3
[15375.883194] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15375.885009] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15375.885929] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15375.886858] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15375.887779] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15375.888702] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15375.889621] RBP: ffff88024372be38 R08: 000000008baf5f56 R09: 0000000000000000
[15375.890534] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15375.891446] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15375.892362] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15375.893286] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15375.894208] CR2: 00000000a0d5536f CR3: 0000000001c11000 CR4: 00000000001407e0
[15375.895138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15375.896066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15375.896990] Stack:
[15375.897914]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15375.898865]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15375.899819]  00000dfe02d03a98 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15375.900772] Call Trace:
[15375.901721]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15375.902675]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15375.903607]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15375.904520]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15375.905430] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15375.907432] NMI backtrace for cpu 0
[15375.908366] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15375.910239] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15375.911180] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15375.912114] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15375.913039] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15375.913969] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15375.914895] RBP: ffffffff81c03e68 R08: 000000008baf5f56 R09: 0000000000000000
[15375.915851] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15375.916776] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15375.917694] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15375.918612] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15375.919528] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15375.920459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15375.921382] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15375.922294] Stack:
[15375.923199]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15375.924126]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15375.925050]  00000dfe03416b9a ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15375.925994] Call Trace:
[15375.926912]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15375.927843]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15375.928768]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15375.929692]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15375.930613]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15375.931531]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15375.932450]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15375.933364]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15375.934282]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15375.935202]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15375.936136] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15390.907159] INFO: rcu_sched self-detected stall on CPU
[15390.908187] 	1: (1 GPs behind) idle=52b/140000000000001/0 softirq=984789/984790 
[15390.909165] 	 (t=6000 jiffies g=472111 c=472110 q=0)
[15390.910126] Task dump for CPU 1:
[15390.911080] trinity-c195    R  running task    13248 20128  19429 0x0000000c
[15390.912048]  ffff880069c02da0 0000000017646c73 ffff880245003d68 ffffffff810a711c
[15390.913012]  ffffffff810a7082 0000000000000001 0000000000000002 0000000000000001
[15390.913958]  ffffffff81c52300 0000000000000092 ffff880245003d88 ffffffff810ab1cd
[15390.914898] Call Trace:
[15390.915822]  <IRQ>  [<ffffffff810a711c>] sched_show_task+0x11c/0x190
[15390.916767]  [<ffffffff810a7082>] ? sched_show_task+0x82/0x190
[15390.917748]  [<ffffffff810ab1cd>] dump_cpu_task+0x3d/0x50
[15390.918709]  [<ffffffff810d9760>] rcu_dump_cpu_stacks+0x90/0xd0
[15390.919675]  [<ffffffff810e0213>] rcu_check_callbacks+0x503/0x770
[15390.920637]  [<ffffffff8112fadc>] ? acct_account_cputime+0x1c/0x20
[15390.921595]  [<ffffffff810abb07>] ? account_system_time+0x97/0x180
[15390.922549]  [<ffffffff810e5c0b>] update_process_times+0x4b/0x80
[15390.923497]  [<ffffffff810f63a3>] ? tick_sched_timer+0x23/0x1b0
[15390.924449]  [<ffffffff810f63cf>] tick_sched_timer+0x4f/0x1b0
[15390.925402]  [<ffffffff810e6900>] __run_hrtimer+0xa0/0x230
[15390.926350]  [<ffffffff810e6beb>] ? hrtimer_interrupt+0x8b/0x260
[15390.927310]  [<ffffffff810f6380>] ? tick_init_highres+0x20/0x20
[15390.928214]  [<ffffffff810e6c67>] hrtimer_interrupt+0x107/0x260
[15390.929111]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[15390.930006]  [<ffffffff817d0505>] smp_apic_timer_interrupt+0x45/0x60
[15390.930898]  [<ffffffff817ce8ef>] apic_timer_interrupt+0x6f/0x80
[15390.931796]  <EOI>  [<ffffffff810c5cf4>] ? lock_acquire+0xb4/0x120
[15390.932697]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15390.933599]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15390.934483]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15390.935343]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15390.936201]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15390.937050]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15390.937883]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15390.938701]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15390.939505]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15390.940303]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15390.941094]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15390.941893]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15390.942690]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15390.943477]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15390.944276] INFO: rcu_sched detected stalls on CPUs/tasks:
[15390.945074] 	1: (1 GPs behind) idle=52b/140000000000001/0 softirq=984789/984790 
[15390.945854] 	(detected by 0, t=6002 jiffies, g=472111, c=472110, q=0)
[15390.946637] Task dump for CPU 1:
[15390.947442] trinity-c195    R  running task    13248 20128  19429 0x0000000c
[15390.948244]  ffff8801c06dff60 ffffffff817c6eb2 000000000203f56b 0000000017646c73
[15390.949047]  ffffffff813743a4 0000000000000000 00007fffe01fcb40 0000000000000000
[15390.949842]  0000000000000000 0000000000000000 ffff8801c06dff40 ffffffff81077056
[15390.950641] Call Trace:
[15390.951431]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15390.952218]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15390.953006]  [<ffffffff81077056>] ? SyS_clone+0x16/0x20
[15390.953793]  [<ffffffff817cddd9>] ? stub_clone+0x69/0x90
[15390.954582]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15403.789654] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [trinity-c154:22823]
[15403.790519] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15403.795345] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15403.797194] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15403.798137] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15403.799068] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15403.800047] RAX: 0000000000000008 RBX: ffffffff817ce620 RCX: 0000000000000038
[15403.800987] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15403.801923] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15403.802866] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88009455b8a8
[15403.803780] R13: 0000000000406040 R14: ffff880094558000 R15: ffff880235dedb40
[15403.804720] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15403.805633] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15403.806572] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15403.807495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15403.808423] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15403.809361] Stack:
[15403.810339]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15403.811301]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15403.812253]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15403.813215] Call Trace:
[15403.814145]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15403.815102]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15403.816051]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15403.817000]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15403.817966]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15403.818923]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15403.819903]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15403.820853]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15403.821801]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15403.822753]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15403.823711]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15403.824645]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15403.825601]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15403.826536]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15403.827482]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15403.828423] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15403.830537] sending NMI to other CPUs:
[15403.831515] NMI backtrace for cpu 1
[15403.832519] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15403.834564] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15403.835608] RIP: 0010:[<ffffffff810a5d95>]  [<ffffffff810a5d95>] scheduler_tick+0x35/0xe0
[15403.836664] RSP: 0018:ffff880245003e08  EFLAGS: 00000086
[15403.837713] RAX: 0000000000000000 RBX: ffff8802451d2f00 RCX: 0000000000000009
[15403.838768] RDX: 000000000007342f RSI: ffff8802451cc5c0 RDI: ffff8802451ce358
[15403.839816] RBP: ffff880245003e28 R08: 0000000000000000 R09: 0000000000000000
[15403.840866] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15403.841908] R13: 00000000001d2f00 R14: ffff880069c02da0 R15: 00000e2fcb303732
[15403.842944] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15403.843986] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15403.845025] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15403.846067] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15403.847105] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15403.848139] Stack:
[15403.849168]  ffff880069c02da0 0000000000000000 0000000000000001 ffff8802451cc880
[15403.850226]  ffff880245003e58 ffffffff810e5c24 ffffffff810f63a3 ffff8802451cce60
[15403.851285]  ffff8801c06dfb08 00000e048872d5bd ffff880245003e98 ffffffff810f63cf
[15403.852339] Call Trace:
[15403.853358]  <IRQ> 

[15403.854360]  [<ffffffff810e5c24>] update_process_times+0x64/0x80
[15403.855339]  [<ffffffff810f63a3>] ? tick_sched_timer+0x23/0x1b0
[15403.856296]  [<ffffffff810f63cf>] tick_sched_timer+0x4f/0x1b0
[15403.857230]  [<ffffffff810e6900>] __run_hrtimer+0xa0/0x230
[15403.858151]  [<ffffffff810e6beb>] ? hrtimer_interrupt+0x8b/0x260
[15403.859055]  [<ffffffff810f6380>] ? tick_init_highres+0x20/0x20
[15403.859937]  [<ffffffff810e6c67>] hrtimer_interrupt+0x107/0x260
[15403.860809]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[15403.861677]  [<ffffffff817d0505>] smp_apic_timer_interrupt+0x45/0x60
[15403.862539]  [<ffffffff817ce8ef>] apic_timer_interrupt+0x6f/0x80
[15403.863404]  <EOI> 

[15403.864270]  [<ffffffff810c5cf4>] ? lock_acquire+0xb4/0x120
[15403.865133]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15403.865999]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15403.866855]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15403.867709]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15403.868551]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15403.869397]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15403.870244]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15403.871090]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15403.871928]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15403.872757]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15403.873581]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15403.874400]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15403.875215]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15403.876034]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15403.876854] Code: 56 41 55 49 c7 c5 00 2f 1d 00 41 54 65 44 8b 24 25 2c a0 00 00 4d 63 e4 53 4c 89 eb 4a 03 1c e5 a0 f0 d1 81 4c 8b b3 50 09 00 00 <e8> 46 5a 00 00 48 89 df e8 fe 6b 72 00 8b b3 88 00 00 00 85 f6 
[15403.878669] NMI backtrace for cpu 0
[15403.879531] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15403.881306] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15403.882213] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15403.883130] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15403.884046] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[15403.884974] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15403.885895] RBP: ffffffff81c03e68 R08: 000000008baf94ae R09: 0000000000000000
[15403.886819] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004
[15403.887743] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c00000
[15403.888643] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15403.889530] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15403.890410] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15403.891303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15403.892187] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15403.893049] Stack:
[15403.893886]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000004
[15403.894739]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15403.895587]  00000e048864f135 ffffffff81cb1b38 ffffffff81cb19c0 ffffffff81d213f0
[15403.896437] Call Trace:
[15403.897273]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15403.898118]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15403.898959]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15403.899803]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15403.900634]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15403.901461]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15403.902279]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15403.903098]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15403.903919]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15403.904738]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15403.905542] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15403.907331] NMI backtrace for cpu 3
[15403.908216] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15403.910040] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15403.910967] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15403.911886] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15403.912807] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15403.913741] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15403.914671] RBP: ffff88024372be38 R08: 000000008baf94ae R09: 0000000000000000
[15403.915600] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15403.916530] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15403.917459] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15403.918390] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15403.919323] CR2: 00007f3d704ad510 CR3: 0000000001c11000 CR4: 00000000001407e0
[15403.920290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15403.921227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15403.922170] Stack:
[15403.923088]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15403.924041]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15403.924999]  00000e048856e4d7 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15403.925963] Call Trace:
[15403.926915]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15403.927880]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15403.928851]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15403.929856]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15403.930814] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15403.932954] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 101.416 msecs
[15431.763416] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15431.764423] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15431.769803] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15431.771905] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15431.772970] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15431.774087] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15431.775164] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15431.776247] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15431.777335] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15431.778416] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15431.779490] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15431.780563] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15431.781650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15431.782741] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15431.783871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15431.784955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15431.786032] Stack:
[15431.787101]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15431.788192]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15431.789284]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15431.790380] Call Trace:
[15431.791470]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15431.792574]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15431.793719]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15431.794824]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15431.795932]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15431.797039]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15431.798144]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15431.799257]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15431.800350]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15431.801415]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15431.802477]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15431.803569]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15431.804612]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15431.805638]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15431.806658] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15431.808871] sending NMI to other CPUs:
[15431.809913] NMI backtrace for cpu 2
[15431.810970] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15431.813123] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15431.814216] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15431.815316] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15431.816408] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15431.817508] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15431.818611] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15431.819708] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15431.820802] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15431.821891] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15431.822984] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15431.824070] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15431.825163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15431.826259] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15431.827346] Stack:
[15431.828426]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15431.829512]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15431.830583]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15431.831638] Call Trace:
[15431.832672]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15431.833696]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15431.834691]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15431.835665]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15431.836615]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15431.837546]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15431.838465]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15431.839368]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15431.840247]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15431.841119]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15431.841984]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15431.842844]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15431.843703]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15431.844564]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15431.845423]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15431.846275] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15431.848144] NMI backtrace for cpu 3
[15431.849023] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15431.850835] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15431.851759] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15431.852688] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15431.853609] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15431.854534] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15431.855455] RBP: ffff88024372be38 R08: 000000008baf9453 R09: 0000000000000000
[15431.856371] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15431.857285] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15431.858198] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15431.859119] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15431.860043] CR2: 00007fb66fe8d000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15431.860974] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15431.861900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15431.862823] Stack:
[15431.863743]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15431.864692]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15431.865643]  00000e0b0cac5a1b ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15431.866597] Call Trace:
[15431.867545]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15431.868502]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15431.869430]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15431.870340]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15431.871248] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15431.873246] NMI backtrace for cpu 0
[15431.874180] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15431.875997] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15431.876912] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15431.877839] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15431.878761] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15431.879690] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15431.880614] RBP: ffffffff81c03e68 R08: 000000008baf9453 R09: 0000000000000000
[15431.881543] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15431.882464] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15431.883404] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15431.884320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15431.885237] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15431.886166] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15431.887087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15431.887999] Stack:
[15431.888902]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15431.889827]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15431.890750]  00000e0b0d00864a ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15431.891667] Call Trace:
[15431.892585]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15431.893539]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15431.894459]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15431.895379]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15431.896298]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15431.897213]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15431.898129]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15431.899043]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15431.899956]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15431.900870]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15431.901781] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15455.749491] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15455.750478] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15455.755714] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15455.757759] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15455.758788] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15455.759874] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15455.760911] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15455.761959] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15455.763008] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15455.764058] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15455.765101] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15455.766140] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15455.767184] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15455.768231] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15455.769293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15455.770388] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15455.771434] Stack:
[15455.772464]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15455.773521]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15455.774595]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15455.775650] Call Trace:
[15455.776695]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15455.777753]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15455.778815]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15455.779897]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15455.780917]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15455.781937]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15455.782953]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15455.783954]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15455.784939]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15455.785912]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15455.786881]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15455.787847]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15455.788813]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15455.789820]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15455.790787] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15455.792887] sending NMI to other CPUs:
[15455.793879] NMI backtrace for cpu 2
[15455.794911] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15455.796992] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15455.798051] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15455.799120] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15455.800181] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15455.801251] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15455.802312] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15455.803370] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15455.804434] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15455.805497] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15455.806568] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15455.807644] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15455.808728] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15455.809809] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15455.810889] Stack:
[15455.811962]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15455.813062]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15455.814144]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15455.815201] Call Trace:
[15455.816245]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15455.817277]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15455.818277]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15455.819260]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15455.820220]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15455.821159]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15455.822081]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15455.822985]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15455.823872]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15455.824751]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15455.825624]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15455.826490]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15455.827357]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15455.828223]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15455.829087]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15455.829946] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15455.831821] NMI backtrace for cpu 0
[15455.832705] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15455.834530] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15455.835459] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15455.836393] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15455.837317] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15455.838245] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15455.839167] RBP: ffffffff81c03e68 R08: 000000008baf9406 R09: 0000000000000000
[15455.840088] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15455.841005] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15455.841922] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15455.842846] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15455.843771] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15455.844702] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15455.845633] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15455.846557] Stack:
[15455.847479]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15455.848429]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15455.849383]  00000e10a3641194 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15455.850336] Call Trace:
[15455.851281]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15455.852238]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15455.853168]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15455.854074]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15455.854979]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15455.855879]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15455.856777]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15455.857652]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15455.858509]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15455.859360]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15455.860198] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15455.862051] NMI backtrace for cpu 3
[15455.862965] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15455.864827] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15455.865772] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15455.866725] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15455.867666] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15455.868617] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15455.869587] RBP: ffff88024372be38 R08: 000000008baf9406 R09: 0000000000000000
[15455.870547] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15455.871512] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15455.872459] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15455.873402] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15455.874337] CR2: 00007fcb01b8ddc8 CR3: 0000000001c11000 CR4: 00000000001407e0
[15455.875280] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15455.876223] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15455.877167] Stack:
[15455.878099]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15455.879058]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15455.880070]  00000e10a32f62cd ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15455.881036] Call Trace:
[15455.881988]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15455.882958]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15455.883922]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15455.884889]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15455.885855] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15479.735569] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15479.736560] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15479.741807] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15479.743855] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15479.744885] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15479.745974] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15479.747013] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15479.748060] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15479.749108] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15479.750160] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15479.751208] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15479.752252] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15479.753302] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15479.754354] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15479.755417] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15479.756512] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15479.757559] Stack:
[15479.758599]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15479.759664]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15479.760720]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15479.761780] Call Trace:
[15479.762833]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15479.763899]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15479.764966]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15479.766079]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15479.767149]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15479.768222]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15479.769292]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15479.770362]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15479.771437]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15479.772489]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15479.773518]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15479.774548]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15479.775590]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15479.776595]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15479.777587] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15479.779714] sending NMI to other CPUs:
[15479.780720] NMI backtrace for cpu 2
[15479.781764] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15479.783889] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15479.784972] RIP: 0010:[<ffffffff810fb22a>]  [<ffffffff810fb22a>] generic_exec_single+0xea/0x1b0
[15479.786068] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15479.787156] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15479.788247] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15479.789336] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15479.790430] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15479.791519] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15479.792602] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15479.793683] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15479.794764] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15479.795847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15479.796924] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15479.797999] Stack:
[15479.799068]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15479.800165]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15479.801246]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15479.802301] Call Trace:
[15479.803339]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15479.804368]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15479.805369]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15479.806347]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15479.807302]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15479.808240]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15479.809161]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15479.810064]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15479.810945]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15479.811823]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15479.812694]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15479.813558]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15479.814420]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15479.815282]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15479.816142]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15479.816999] Code: c0 3a 1d 00 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[15479.818876] NMI backtrace for cpu 0
[15479.819760] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15479.821583] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15479.822511] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15479.823444] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15479.824367] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15479.825295] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15479.826216] RBP: ffffffff81c03e68 R08: 000000008baf93ba R09: 0000000000000000
[15479.827135] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15479.828051] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15479.828966] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15479.829890] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15479.830816] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15479.831748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15479.832678] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15479.833606] Stack:
[15479.834532]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15479.835483]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15479.836437]  00000e1639f2763b ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15479.837392] Call Trace:
[15479.838340]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15479.839299]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15479.840232]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15479.841143]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15479.842054]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15479.842961]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15479.843859]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15479.844740]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15479.845601]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15479.846457]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15479.847298] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15479.849161] NMI backtrace for cpu 3
[15479.850081] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15479.851956] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15479.852907] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15479.853869] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15479.854814] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15479.855797] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15479.856752] RBP: ffff88024372be38 R08: 000000008baf93ba R09: 0000000000000000
[15479.857706] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15479.858655] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15479.859594] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15479.860536] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15479.861477] CR2: 00007fb66fe8d000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15479.862424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15479.863374] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15479.864320] Stack:
[15479.865259]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15479.866246]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15479.867210]  00000e1639b23221 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15479.868181] Call Trace:
[15479.869141]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15479.870115]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15479.871084]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15479.872056]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15479.873024] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15503.721645] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15503.722660] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15503.728065] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15503.730161] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15503.731220] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15503.732323] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15503.733390] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15503.734464] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15503.735538] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15503.736613] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15503.737686] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15503.738753] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15503.739825] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15503.740900] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15503.742025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15503.743107] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15503.744174] Stack:
[15503.745237]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15503.746321]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15503.747402]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15503.748483] Call Trace:
[15503.749558]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15503.750647]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15503.751777]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15503.752869]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15503.753960]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15503.755052]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15503.756141]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15503.757236]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15503.758334]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15503.759408]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15503.760459]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15503.761510]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15503.762592]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15503.763620]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15503.764630] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15503.766839] sending NMI to other CPUs:
[15503.767869] NMI backtrace for cpu 2
[15503.768911] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15503.771029] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15503.772108] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15503.773197] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15503.774278] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15503.775363] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15503.776449] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15503.777541] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15503.778624] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15503.779703] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15503.780783] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15503.781859] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15503.782940] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15503.784017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15503.785088] Stack:
[15503.786155]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15503.787248]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15503.788321]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15503.789368] Call Trace:
[15503.790402]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15503.791424]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15503.792418]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15503.793393]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15503.794348]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15503.795279]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15503.796195]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15503.797090]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15503.797967]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15503.798842]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15503.799708]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15503.800568]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15503.801426]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15503.802284]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15503.803144]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15503.803995] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15503.805861] NMI backtrace for cpu 0
[15503.806741] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15503.808556] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15503.809478] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15503.810409] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15503.811329] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15503.812252] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15503.813170] RBP: ffffffff81c03e68 R08: 000000008baf936d R09: 0000000000000000
[15503.814082] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15503.814996] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15503.815908] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15503.816831] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15503.817751] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15503.818679] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15503.819606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15503.820528] Stack:
[15503.821448]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15503.822395]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15503.823347]  00000e1bd0854ef6 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15503.824298] Call Trace:
[15503.825238]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15503.826192]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15503.827119]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15503.828030]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15503.828933]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15503.829838]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15503.830737]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15503.831615]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15503.832476]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15503.833324]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15503.834162] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15503.836024] NMI backtrace for cpu 3
[15503.836937] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15503.838808] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15503.839754] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15503.840709] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15503.841668] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15503.842628] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15503.843584] RBP: ffff88024372be38 R08: 000000008baf936d R09: 0000000000000000
[15503.844532] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15503.845472] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15503.846410] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15503.847351] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15503.848287] CR2: 00007f3d70900000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15503.849227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15503.850175] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15503.851116] Stack:
[15503.852075]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15503.853033]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15503.853996]  00000e1bd035579e ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15503.854964] Call Trace:
[15503.855919]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15503.856889]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15503.857856]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15503.858823]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15503.859792] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15527.707724] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15527.708738] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15527.714144] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15527.716239] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15527.717296] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15527.718404] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15527.719467] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15527.720542] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15527.721617] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15527.722698] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15527.723769] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15527.724838] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15527.725913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15527.726988] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15527.728114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15527.729195] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15527.730264] Stack:
[15527.731329]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15527.732413]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15527.733496]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15527.734579] Call Trace:
[15527.735657]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15527.736745]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15527.737876]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15527.738967]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15527.740061]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15527.741153]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15527.742247]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15527.743341]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15527.744436]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15527.745510]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15527.746561]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15527.747613]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15527.748697]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15527.749728]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15527.750744] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15527.752936] sending NMI to other CPUs:
[15527.753966] NMI backtrace for cpu 2
[15527.755011] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15527.757137] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15527.758219] RIP: 0010:[<ffffffff810fb22a>]  [<ffffffff810fb22a>] generic_exec_single+0xea/0x1b0
[15527.759312] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15527.760398] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15527.761486] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15527.762575] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15527.763669] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15527.764759] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15527.765841] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15527.766922] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15527.768002] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15527.769085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15527.770164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15527.771239] Stack:
[15527.772311]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15527.773405]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15527.774481]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15527.775537] Call Trace:
[15527.776573]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15527.777602]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15527.778601]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15527.779578]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15527.780533]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15527.781466]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15527.782385]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15527.783286]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15527.784169]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15527.785043]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15527.785910]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15527.786771]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15527.787633]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15527.788494]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15527.789353]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15527.790209] Code: c0 3a 1d 00 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[15527.792084] NMI backtrace for cpu 0
[15527.792967] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15527.794790] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15527.795714] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15527.796646] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15527.797567] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15527.798491] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15527.799410] RBP: ffffffff81c03e68 R08: 000000008baf9322 R09: 0000000000000000
[15527.800326] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15527.801238] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15527.802152] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15527.803078] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15527.804002] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15527.804932] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15527.805862] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15527.806783] Stack:
[15527.807706]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15527.808654]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15527.809606]  00000e216708a957 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15527.810560] Call Trace:
[15527.811502]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15527.812458]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15527.813387]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15527.814298]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15527.815206]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15527.816111]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15527.817012]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15527.817888]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15527.818748]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15527.819602]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15527.820441] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15527.822303] NMI backtrace for cpu 3
[15527.823217] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15527.825093] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15527.826045] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15527.827004] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15527.827975] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15527.828929] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15527.829884] RBP: ffff88024372be38 R08: 000000008baf9322 R09: 0000000000000000
[15527.830835] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15527.831782] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15527.832725] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15527.833671] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15527.834610] CR2: 00007fb66fe8d000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15527.835553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15527.836504] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15527.837452] Stack:
[15527.838415]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15527.839379]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15527.840346]  00000e2166b83852 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15527.841318] Call Trace:
[15527.842279]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15527.843254]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15527.844223]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15527.845193]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15527.846163] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15551.693803] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]
[15551.694820] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15551.700151] CPU: 1 PID: 20128 Comm: trinity-c195 Tainted: G             L 3.18.0+ #107
[15551.702255] task: ffff880069c02da0 ti: ffff8801c06dc000 task.ti: ffff8801c06dc000
[15551.703316] RIP: 0010:[<ffffffff810c5cf4>]  [<ffffffff810c5cf4>] lock_acquire+0xb4/0x120
[15551.704425] RSP: 0018:ffff8801c06dfbb0  EFLAGS: 00000246
[15551.705491] RAX: ffff880069c02da0 RBX: 0000000180140014 RCX: ffff8802451cff98
[15551.706569] RDX: 00000000000015a0 RSI: 0000000000000018 RDI: 0000000000000000
[15551.707645] RBP: ffff8801c06dfc10 R08: 0000000000000000 R09: 0000000000000000
[15551.708722] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c06dfba0
[15551.709794] R13: ffffffff810ab7d5 R14: ffff8801c06dfb20 R15: ffff88024483fc70
[15551.710862] FS:  00007f9dcd485740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15551.711937] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15551.713015] CR2: 0000000001e259f8 CR3: 00000002289f7000 CR4: 00000000001407e0
[15551.714153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15551.715236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15551.716311] Stack:
[15551.717374]  ffffffff811a4cc3 ffff880000000000 ffff8801c06dfc10 0000000000000246
[15551.718462]  000000019449eff0 ffff88005e137c38 00000000000000d0 ffff88005e137c20
[15551.719545]  ffff88005e137c38 0000000002422000 ffff8802332eb110 0000000000000000
[15551.720630] Call Trace:
[15551.721706]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15551.722796]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15551.723935]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15551.725027]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15551.726119]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15551.727214]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15551.728307]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15551.729402]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15551.730507]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15551.731572]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15551.732624]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15551.733679]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15551.734760]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15551.735791]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15551.736805] Code: d8 49 c1 e8 09 48 89 04 24 49 83 f0 01 41 83 e0 01 e8 01 ef ff ff 65 48 8b 04 25 00 aa 00 00 c7 80 6c 07 00 00 00 00 00 00 53 9d <48> 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 00 65 
[15551.738995] sending NMI to other CPUs:
[15551.740026] NMI backtrace for cpu 2
[15551.741070] CPU: 2 PID: 22823 Comm: trinity-c154 Tainted: G             L 3.18.0+ #107
[15551.743195] task: ffff880235dedb40 ti: ffff880094558000 task.ti: ffff880094558000
[15551.744276] RIP: 0010:[<ffffffff810fb22e>]  [<ffffffff810fb22e>] generic_exec_single+0xee/0x1b0
[15551.745368] RSP: 0000:ffff88009455b938  EFLAGS: 00000202
[15551.746450] RAX: 0000000000000008 RBX: ffff88009455b950 RCX: 0000000000000038
[15551.747536] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[15551.748624] RBP: ffff88009455b998 R08: ffff88024370c3f0 R09: 0000000000000000
[15551.749717] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15551.750805] R13: 0000000000000001 R14: ffff880225e61780 R15: 0000000000000002
[15551.751885] FS:  00007f9dcd485740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15551.752967] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15551.754047] CR2: 0000000001f9cfd8 CR3: 000000009ae2d000 CR4: 00000000001407e0
[15551.755130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15551.756204] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15551.757280] Stack:
[15551.758349]  ffff88009455b948 0000000000000001 ffff88024370c3f0 0000000000000000
[15551.759443]  ffffffff81048cc0 ffff88009455ba48 0000000000000003 0000000077fab51f
[15551.760518]  0000000000000001 00000000ffffffff 0000000000000001 ffffffff81048cc0
[15551.761574] Call Trace:
[15551.762611]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15551.763634]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15551.764628]  [<ffffffff810fb390>] smp_call_function_single+0x70/0xd0
[15551.765605]  [<ffffffff81048cc0>] ? do_flush_tlb_all+0x60/0x60
[15551.766558]  [<ffffffff810fba89>] smp_call_function_many+0x2b9/0x320
[15551.767490]  [<ffffffff81049010>] flush_tlb_mm_range+0x90/0x1d0
[15551.768408]  [<ffffffff811a1a42>] tlb_flush_mmu_tlbonly+0x42/0x50
[15551.769309]  [<ffffffff811a2fa8>] unmap_single_vma+0x6b8/0x900
[15551.770190]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15551.771068]  [<ffffffff811a34d4>] unmap_mapping_range+0x134/0x190
[15551.771935]  [<ffffffff81191f0d>] shmem_fallocate+0x4fd/0x520
[15551.772795]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15551.773656]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15551.774518]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15551.775378]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15551.776235] Code: 48 89 de 48 03 14 c5 a0 f0 d1 81 48 89 df e8 da de 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[15551.778111] NMI backtrace for cpu 0
[15551.778992] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15551.780810] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15551.781738] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15551.782669] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15551.783592] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15551.784519] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15551.785439] RBP: ffffffff81c03e68 R08: 000000008baf92d6 R09: 0000000000000000
[15551.786357] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15551.787273] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15551.788189] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15551.789114] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15551.790036] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407f0
[15551.790968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15551.791900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15551.792824] Stack:
[15551.793747]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15551.794696]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15551.795650]  00000e26fd8b172b ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15551.796601] Call Trace:
[15551.797546]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15551.798502]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15551.799432]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15551.800343]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15551.801247]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15551.802151]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15551.803050]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15551.803928]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15551.804786]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15551.805637]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15551.806475] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15551.808334] NMI backtrace for cpu 3
[15551.809246] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15551.811116] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15551.812066] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15551.813027] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15551.813992] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15551.814942] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15551.815894] RBP: ffff88024372be38 R08: 000000008baf92d6 R09: 0000000000000000
[15551.816843] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15551.817784] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15551.818718] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15551.819658] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15551.820593] CR2: 00007f3d70914008 CR3: 0000000001c11000 CR4: 00000000001407e0
[15551.821533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15551.822478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15551.823418] Stack:
[15551.824374]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15551.825334]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15551.826295]  00000e26fd3b2c1a ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15551.827258] Call Trace:
[15551.828213]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15551.829182]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15551.830143]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15551.831107]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15551.832072] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15570.852710] INFO: rcu_sched self-detected stall on CPU
[15570.853735] 	1: (1 GPs behind) idle=52b/140000000000001/0 softirq=984789/984790 
[15570.854714] 	 (t=24005 jiffies g=472111 c=472110 q=0)
[15570.855675] Task dump for CPU 1:
[15570.856626] trinity-c195    R  running task    13248 20128  19429 0x0000000c
[15570.857593]  ffff880069c02da0 0000000017646c73 ffff880245003d68 ffffffff810a711c
[15570.858554]  ffffffff810a7082 0000000000000001 0000000000000002 0000000000000001
[15570.859513]  ffffffff81c52300 0000000000000092 ffff880245003d88 ffffffff810ab1cd
[15570.860447] Call Trace:
[15570.861368]  <IRQ>  [<ffffffff810a711c>] sched_show_task+0x11c/0x190
[15570.862309]  [<ffffffff810a7082>] ? sched_show_task+0x82/0x190
[15570.863287]  [<ffffffff810ab1cd>] dump_cpu_task+0x3d/0x50
[15570.864250]  [<ffffffff810d9760>] rcu_dump_cpu_stacks+0x90/0xd0
[15570.865213]  [<ffffffff810e0213>] rcu_check_callbacks+0x503/0x770
[15570.866178]  [<ffffffff8112fadc>] ? acct_account_cputime+0x1c/0x20
[15570.867139]  [<ffffffff810abb07>] ? account_system_time+0x97/0x180
[15570.868092]  [<ffffffff810e5c0b>] update_process_times+0x4b/0x80
[15570.869053]  [<ffffffff810f63a3>] ? tick_sched_timer+0x23/0x1b0
[15570.869997]  [<ffffffff810f63cf>] tick_sched_timer+0x4f/0x1b0
[15570.870951]  [<ffffffff810e6900>] __run_hrtimer+0xa0/0x230
[15570.871901]  [<ffffffff810e6beb>] ? hrtimer_interrupt+0x8b/0x260
[15570.872862]  [<ffffffff810f6380>] ? tick_init_highres+0x20/0x20
[15570.873767]  [<ffffffff810e6c67>] hrtimer_interrupt+0x107/0x260
[15570.874670]  [<ffffffff81031e9b>] local_apic_timer_interrupt+0x3b/0x70
[15570.875569]  [<ffffffff817d0505>] smp_apic_timer_interrupt+0x45/0x60
[15570.876467]  [<ffffffff817ce8ef>] apic_timer_interrupt+0x6f/0x80
[15570.877366]  <EOI>  [<ffffffff810c5cf4>] ? lock_acquire+0xb4/0x120
[15570.878275]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15570.879182]  [<ffffffff817ccadc>] _raw_spin_lock_nested+0x3c/0x80
[15570.880088]  [<ffffffff811a4cc3>] ? copy_page_range+0x493/0xa20
[15570.881000]  [<ffffffff811a4cc3>] copy_page_range+0x493/0xa20
[15570.881911]  [<ffffffff8107648f>] copy_process.part.26+0x146f/0x1a40
[15570.882824]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15570.883732]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15570.884647]  [<ffffffff8109ef5d>] ? finish_task_switch+0x7d/0x120
[15570.885544]  [<ffffffff8109ef1f>] ? finish_task_switch+0x3f/0x120
[15570.886410]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15570.887274]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15570.888136]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15570.888977]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15570.889798]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15570.890616] INFO: rcu_sched detected stalls on CPUs/tasks:
[15570.891487] 	1: (1 GPs behind) idle=52b/140000000000001/0 softirq=984789/984790 
[15570.892485] 	(detected by 2, t=24008 jiffies, g=472111, c=472110, q=0)
[15570.893412] Task dump for CPU 1:
[15570.894307] trinity-c195    R  running task    13248 20128  19429 0x0000000c
[15570.895226]  ffff8801c06dff60 ffffffff817c6eb2 000000000203f56b 0000000017646c73
[15570.896250]  ffffffff813743a4 0000000000000000 00007fffe01fcb40 0000000000000000
[15570.897134]  0000000000000000 0000000000000000 ffff8801c06dff40 ffffffff81077056
[15570.898021] Call Trace:
[15570.898891]  [<ffffffff817c6eb2>] ? __schedule+0x352/0x8c0
[15570.899789]  [<ffffffff813743a4>] ? lockdep_sys_exit_thunk+0x35/0x67
[15570.900688]  [<ffffffff81077056>] ? SyS_clone+0x16/0x20
[15570.901556]  [<ffffffff817cddd9>] ? stub_clone+0x69/0x90
[15570.902430]  [<ffffffff817cda52>] ? system_call_fastpath+0x12/0x17
[15589.522001] sched: RT throttling activated
[15611.669013] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [trinity-c170:30232]
[15611.669927] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15611.674793] CPU: 1 PID: 30232 Comm: trinity-c170 Tainted: G             L 3.18.0+ #107
[15611.676697] task: ffff880226e70000 ti: ffff8802276e4000 task.ti: ffff8802276e4000
[15611.677659] RIP: 0010:[<ffffffff8118f845>]  [<ffffffff8118f845>] shmem_write_end+0x65/0xf0
[15611.678654] RSP: 0018:ffff8802276e7c08  EFLAGS: 00000202
[15611.679726] RAX: 001ffe0000080005 RBX: ffff8802276e7c80 RCX: 1000ef5abfc6b000
[15611.680754] RDX: 1000ef5abfc6a000 RSI: ffff88009ac54a00 RDI: ffff88020ccd9500
[15611.681797] RBP: ffff8802276e7c28 R08: 0000000000001000 R09: ffffea00011734c0
[15611.682859] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8118ffe4
[15611.683899] R13: ffff8802276e7c18 R14: 0000000000000003 R15: 0000000000000003
[15611.684945] FS:  00007fdf412f4740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15611.686005] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15611.687067] CR2: 00007fdf412fb055 CR3: 0000000227d30000 CR4: 00000000001407e0
[15611.688105] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15611.689152] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15611.690228] Stack:
[15611.691283]  1000ef5abfc6a000 0000000000001000 ffff8802276e7d60 0000000000000000
[15611.692367]  ffff8802276e7cc8 ffffffff811754da ffff8802276e7d18 0000000000001000
[15611.693438]  ffff880226e70000 0000000081204df1 0000000003341000 0000000000000000
[15611.694522] Call Trace:
[15611.695606]  [<ffffffff811754da>] generic_perform_write+0x11a/0x1f0
[15611.696694]  [<ffffffff81177bf2>] __generic_file_write_iter+0x162/0x350
[15611.697783]  [<ffffffff811e6d90>] ? new_sync_read+0xd0/0xd0
[15611.698872]  [<ffffffff81177e1f>] generic_file_write_iter+0x3f/0xb0
[15611.699974]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.701062]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15611.702161]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15611.703259]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.704386]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.705488]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15611.706610]  [<ffffffff811e895c>] vfs_writev+0x3c/0x50
[15611.707726]  [<ffffffff811e8acc>] SyS_writev+0x5c/0x100
[15611.708838]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15611.709950] Code: df e8 20 63 fe ff 48 89 df e8 a8 47 ff ff 5b 44 89 e0 41 5c 41 5d 41 5e 5d c3 0f 1f 40 00 41 81 fc ff 0f 00 00 76 0f f0 80 0b 08 <eb> c9 66 0f 1f 84 00 00 00 00 00 81 e2 ff 0f 00 00 46 8d 2c 22 
[15611.712349] sending NMI to other CPUs:
[15611.713501] NMI backtrace for cpu 3
[15611.714511] CPU: 3 PID: 29352 Comm: trinity-c56 Tainted: G             L 3.18.0+ #107
[15611.716572] task: ffff8801c067c470 ti: ffff88017c840000 task.ti: ffff88017c840000
[15611.717621] RIP: 0010:[<ffffffff81372bc5>]  [<ffffffff81372bc5>] copy_user_enhanced_fast_string+0x5/0x10
[15611.718691] RSP: 0018:ffff88017c843c60  EFLAGS: 00010286
[15611.719758] RAX: ffff8801711e0000 RBX: ffff88017c843d60 RCX: 0000000000000fc0
[15611.720830] RDX: 0000000000001000 RSI: ffff8801711e0040 RDI: 00007fdf3fdd67c2
[15611.721900] RBP: ffff88017c843cb8 R08: ffff8801711e0000 R09: ffff88022d82a6d8
[15611.722963] R10: ffff88017c843b88 R11: 0000000000000000 R12: 0000000000001000
[15611.724022] R13: 000000000006c782 R14: ffff880095736ad0 R15: 0000000000001000
[15611.725076] FS:  00007fdf412f4740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15611.726136] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15611.727191] CR2: 00007fdf403eaa16 CR3: 0000000223d4e000 CR4: 00000000001407e0
[15611.728253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15611.729308] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15611.730357] Stack:
[15611.731402]  ffffffff8119f74d ffff8801711e0000 00007fdf3fdd6782 ffff8801711e0000
[15611.732469]  ffffea0005c47800 ffffea0005c47800 000000000008d423 0000000000000000
[15611.733543]  ffff88009ac56820 0000000000001000 0000000000000000 ffff88017c843d48
[15611.734620] Call Trace:
[15611.735681]  [<ffffffff8119f74d>] ? copy_page_to_iter+0x19d/0x340
[15611.736742]  [<ffffffff81190b7b>] shmem_file_read_iter+0xcb/0x300
[15611.737783]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15611.738800]  [<ffffffff811e6cc0>] ? do_sync_readv_writev+0xa0/0xa0
[15611.739797]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15611.740770]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15611.741721]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15611.742657]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15611.743576]  [<ffffffff811e8909>] vfs_readv+0x39/0x50
[15611.744474]  [<ffffffff811e89cc>] SyS_readv+0x5c/0x100
[15611.745359]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15611.746238] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[15611.748165] NMI backtrace for cpu 2
[15611.749082] CPU: 2 PID: 28939 Comm: trinity-c179 Tainted: G             L 3.18.0+ #107
[15611.750987] task: ffff880222f696d0 ti: ffff88019cc6c000 task.ti: ffff88019cc6c000
[15611.751942] RIP: 0010:[<ffffffff81372bc5>]  [<ffffffff81372bc5>] copy_user_enhanced_fast_string+0x5/0x10
[15611.752910] RSP: 0000:ffff88019cc6fbe0  EFLAGS: 00010202
[15611.753867] RAX: 00007fdf3d6bdc01 RBX: 0000000000000000 RCX: 0000000000000880
[15611.754833] RDX: 0000000000001000 RSI: 00007fdf3d6be381 RDI: ffff88011f624780
[15611.755806] RBP: ffff88019cc6fc28 R08: 0000000000000000 R09: 0000000000000001
[15611.756775] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880069dae0a8
[15611.757738] R13: 0000000000001000 R14: 0000000000001000 R15: ffff88011f624000
[15611.758699] FS:  00007fdf412f4740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15611.759671] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15611.760632] CR2: 00007fdf400c0727 CR3: 00000001c07a0000 CR4: 00000000001407e0
[15611.761606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15611.762573] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15611.763535] Stack:
[15611.764491]  ffffffff8119f2e6 ffff88019cc6fbf8 000000000f8c4000 0000000000001000
[15611.765477]  000000000f8c4000 0000000000001000 ffff88019cc6fd60 0000000000000000
[15611.766460]  ffff88009ac57710 ffff88019cc6fcc8 ffffffff811754b7 ffff88019cc6fc88
[15611.767449] Call Trace:
[15611.768424]  [<ffffffff8119f2e6>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[15611.769424]  [<ffffffff811754b7>] generic_perform_write+0xf7/0x1f0
[15611.770420]  [<ffffffff81177bf2>] __generic_file_write_iter+0x162/0x350
[15611.771416]  [<ffffffff811e6d90>] ? new_sync_read+0xd0/0xd0
[15611.772417]  [<ffffffff81177e1f>] generic_file_write_iter+0x3f/0xb0
[15611.773419]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.774405]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15611.775369]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15611.776326]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.777292]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.778248]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15611.779188]  [<ffffffff811e895c>] vfs_writev+0x3c/0x50
[15611.780107]  [<ffffffff811e8acc>] SyS_writev+0x5c/0x100
[15611.781012]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15611.781902] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[15611.783859] NMI backtrace for cpu 0
[15611.784812] CPU: 0 PID: 29177 Comm: trinity-c146 Tainted: G             L 3.18.0+ #107
[15611.786780] task: ffff880095be0000 ti: ffff880223dd8000 task.ti: ffff880223dd8000
[15611.787780] RIP: 0010:[<ffffffff8136b93c>]  [<ffffffff8136b93c>] __radix_tree_create+0x6c/0x220
[15611.788788] RSP: 0000:ffff880223ddba88  EFLAGS: 00000046
[15611.789786] RAX: ffff8800984c7100 RBX: 0000000000000000 RCX: 0000000000000000
[15611.790790] RDX: 0000000000000000 RSI: 000000000008d689 RDI: ffff88009ac56a38
[15611.791801] RBP: ffff880223ddbad8 R08: 0000000000000000 R09: 0000000000000001
[15611.792806] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fffffffa
[15611.793794] R13: 000000000008d689 R14: ffff88009ac56a38 R15: 0000000000000009
[15611.794770] FS:  00007fdf412f4740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15611.795754] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15611.796746] CR2: 00007fdf40250096 CR3: 000000017a9d0000 CR4: 00000000001407f0
[15611.797760] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15611.798765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15611.799775] Stack:
[15611.800774]  ffff880223ddbaf8 ffff880223ddbaf0 0000000000000002 ffff880223ddbaf8
[15611.801798]  ffff880223ddbaf8 ffffea0008891980 0000000000000000 000000000008d689
[15611.802817]  ffff88009ac56a38 0000000000000000 ffff880223ddbb28 ffffffff8136bb31
[15611.803841] Call Trace:
[15611.804858]  [<ffffffff8136bb31>] radix_tree_insert+0x41/0xf0
[15611.805881]  [<ffffffff8118f798>] shmem_add_to_page_cache+0xf8/0x140
[15611.806908]  [<ffffffff8118ff9c>] shmem_getpage_gfp+0x50c/0x7a0
[15611.807935]  [<ffffffff81190272>] shmem_write_begin+0x42/0x70
[15611.808950]  [<ffffffff81175494>] generic_perform_write+0xd4/0x1f0
[15611.809917]  [<ffffffff81177bf2>] __generic_file_write_iter+0x162/0x350
[15611.810880]  [<ffffffff811e6d90>] ? new_sync_read+0xd0/0xd0
[15611.811838]  [<ffffffff81177e1f>] generic_file_write_iter+0x3f/0xb0
[15611.812779]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.813705]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15611.814616]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15611.815515]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.816418]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15611.817310]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15611.818198]  [<ffffffff811e895c>] vfs_writev+0x3c/0x50
[15611.819090]  [<ffffffff811e8d22>] SyS_pwritev+0xc2/0xf0
[15611.819979]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15611.820859] Code: 12 fa 74 75 45 31 ff 31 c9 eb 28 0f 1f 40 00 44 89 e1 4d 89 ef 41 83 ec 06 49 d3 ef 41 83 e7 3f 83 eb 01 44 89 fa 48 8b 54 d0 28 <74> 52 48 89 c1 48 89 d0 48 85 c0 75 d7 4c 89 f7 48 89 4d c0 e8 
[15611.822791] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 109.285 msecs
[15639.652762] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [rcuos/3:32]
[15639.653833] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15639.659569] CPU: 1 PID: 32 Comm: rcuos/3 Tainted: G             L 3.18.0+ #107
[15639.661870] task: ffff8802437e4470 ti: ffff88024300c000 task.ti: ffff88024300c000
[15639.663045] RIP: 0010:[<ffffffff817ccdf8>]  [<ffffffff817ccdf8>] _raw_spin_unlock_irqrestore+0x38/0x60
[15639.664199] RSP: 0018:ffff88024300fc48  EFLAGS: 00000292
[15639.665344] RAX: 0000000000000001 RBX: ffffffff811cdcd0 RCX: 00000000000002a0
[15639.666499] RDX: ffff88024500d620 RSI: 0000000000000000 RDI: ffff88024483e200
[15639.667656] RBP: ffff88024300fc58 R08: 0000000000000000 R09: ffff8800984a0448
[15639.668811] R10: ffff880243b13840 R11: 0000000000000000 R12: ffff8800984a04f0
[15639.669955] R13: 00000000000000cc R14: ffffffff810135bf R15: ffff88024300fbb8
[15639.671114] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15639.672277] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15639.673431] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15639.674570] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15639.675708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15639.676821] Stack:
[15639.677905]  ffffea0002612800 ffffea0002612800 ffff88024300fd18 ffffffff817c0513
[15639.679014]  ffff88024300fc78 ffffffff810ab7d5 ffff88024300fcf8 ffff8800984a0000
[15639.680119]  0000000000000000 0000000000000292 0000000000000000 0000000100190016
[15639.681210] Call Trace:
[15639.682277]  [<ffffffff817c0513>] __slab_free+0x75/0x317
[15639.683348]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15639.684402]  [<ffffffff81255bcc>] ? proc_i_callback+0x1c/0x20
[15639.685449]  [<ffffffff811d0bee>] kmem_cache_free+0x1ae/0x240
[15639.686495]  [<ffffffff81255bb0>] ? proc_destroy_inode+0x20/0x20
[15639.687538]  [<ffffffff81255bcc>] proc_i_callback+0x1c/0x20
[15639.688590]  [<ffffffff810dd8df>] rcu_nocb_kthread+0x58f/0xd60
[15639.689645]  [<ffffffff810dd62a>] ? rcu_nocb_kthread+0x2da/0xd60
[15639.690670]  [<ffffffff810bcd30>] ? prepare_to_wait_event+0xf0/0xf0
[15639.691706]  [<ffffffff810dd350>] ? rcu_process_callbacks+0x9d0/0x9d0
[15639.692735]  [<ffffffff81098979>] kthread+0xf9/0x110
[15639.693763]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15639.694793]  [<ffffffff81098880>] ? kthread_create_on_node+0x250/0x250
[15639.695817]  [<ffffffff817cd9ac>] ret_from_fork+0x7c/0xb0
[15639.696852]  [<ffffffff81098880>] ? kthread_create_on_node+0x250/0x250
[15639.697866] Code: fc 48 8b 55 08 53 48 8d 7f 18 48 89 f3 be 01 00 00 00 e8 ac 92 8f ff 4c 89 e7 e8 d4 c5 8f ff f6 c7 02 74 17 e8 aa c9 97 ff 53 9d <5b> 65 ff 0c 25 e0 a9 00 00 41 5c 5d c3 0f 1f 00 53 9d e8 91 c8 
[15639.700102] sending NMI to other CPUs:
[15639.701151] NMI backtrace for cpu 3
[15639.702181] CPU: 3 PID: 5935 Comm: trinity-main Tainted: G             L 3.18.0+ #107
[15639.704275] task: ffff880229bbc470 ti: ffff8801b65f4000 task.ti: ffff8801b65f4000
[15639.705341] RIP: 0010:[<ffffffff811cdcf4>]  [<ffffffff811cdcf4>] set_track+0x94/0x140
[15639.706420] RSP: 0018:ffff8801b65f7bd8  EFLAGS: 00000012
[15639.707490] RAX: 000000000000000a RBX: ffff8802288fa228 RCX: ffff8802288fa230
[15639.708570] RDX: 0000000000000010 RSI: 000000000000000f RDI: 00007fffda3ab4c8
[15639.709650] RBP: ffff8801b65f7c08 R08: ffff8801b65f7bd8 R09: 0000000000000000
[15639.710730] R10: ffff88024480e1c0 R11: 0000000000000000 R12: ffffffff811f8726
[15639.711816] R13: ffff8802288f9180 R14: ffff88024483ea00 R15: ffff8801b65f7ca0
[15639.712902] FS:  00007fdd90bc4740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15639.714004] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15639.715104] CR2: 0000000000bb4000 CR3: 000000019ce21000 CR4: 00000000001407e0
[15639.716211] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15639.717315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[15639.718420] Stack:
[15639.719492]  000000100000000a ffff8802288fa230 ffffea0000000000 00000000ff68a7ed
[15639.720572]  ffffea0008a23e00 ffff88024480e1c0 ffff8801b65f7c68 ffffffff817c03cb
[15639.721639]  ffff88009a3a4fd8 ffff88009a3a4fc0 ffff88024480e1c0 ffff88024483ea00
[15639.722694] Call Trace:
[15639.723719]  [<ffffffff817c03cb>] free_debug_processing+0x157/0x22a
[15639.724736]  [<ffffffff817c04f3>] __slab_free+0x55/0x317
[15639.725727]  [<ffffffff811f6fda>] ? path_lookupat+0x7a/0x770
[15639.726696]  [<ffffffff811f8726>] ? final_putname+0x26/0x50
[15639.727641]  [<ffffffff811d0bee>] kmem_cache_free+0x1ae/0x240
[15639.728570]  [<ffffffff811f8726>] final_putname+0x26/0x50
[15639.729478]  [<ffffffff811f89c9>] putname+0x29/0x40
[15639.730366]  [<ffffffff811f97ce>] user_path_at_empty+0x6e/0xc0
[15639.731243]  [<ffffffff811a1ad7>] ? might_fault+0x47/0x50
[15639.732111]  [<ffffffff811ece47>] ? cp_new_stat+0x157/0x190
[15639.732969]  [<ffffffff811f9831>] user_path_at+0x11/0x20
[15639.733827]  [<ffffffff811ec983>] vfs_fstatat+0x63/0xc0
[15639.734682]  [<ffffffff811ecf64>] SYSC_newfstatat+0x24/0x60
[15639.735543]  [<ffffffff81012217>] ? syscall_trace_enter_phase2+0xa7/0x200
[15639.736404]  [<ffffffff811ed1ce>] SyS_newfstatat+0xe/0x10
[15639.737263]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15639.738114] Code: 00 00 e8 c0 58 e4 ff 8b 55 d0 85 d2 0f 85 95 00 00 00 31 d2 0f 1f 00 48 63 f2 83 c2 01 48 c7 44 f3 08 00 00 00 00 83 fa 0f 7e ec <4c> 89 23 65 8b 04 25 2c a0 00 00 89 83 88 00 00 00 65 48 8b 04 
[15639.739992] NMI backtrace for cpu 2
[15639.740889] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #107
[15639.742712] task: ffff88024348c470 ti: ffff88024371c000 task.ti: ffff88024371c000
[15639.743639] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15639.744575] RSP: 0018:ffff88024371fe08  EFLAGS: 00000046
[15639.745502] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[15639.746434] RDX: 0000000000000000 RSI: ffff88024371ffd8 RDI: 0000000000000002
[15639.747365] RBP: ffff88024371fe38 R08: 000000008baf91c6 R09: 0000000000000000
[15639.748300] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[15639.749235] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88024371c000
[15639.750168] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15639.751109] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15639.752051] CR2: 0000000000000000 CR3: 0000000240f57000 CR4: 00000000001407e0
[15639.753000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15639.753946] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15639.754884] Stack:
[15639.755819]  000000024371fe38 87ce37a9ab100d43 ffffe8ffff202118 0000000000000002
[15639.756778]  ffffffff81cb19c0 0000000000000002 ffff88024371fe88 ffffffff8165d385
[15639.757741]  00000e3b7b884bf4 ffffffff81cb1a88 ffffffff81cb19c0 ffffffff81d213f0
[15639.758707] Call Trace:
[15639.759662]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15639.760606]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15639.761524]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15639.762440]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15639.763358] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15639.765344] NMI backtrace for cpu 0
[15639.766269] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15639.768133] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15639.769074] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15639.770030] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15639.770979] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15639.771933] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15639.772884] RBP: ffffffff81c03e68 R08: 000000008baf91c6 R09: 0000000000000000
[15639.773827] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15639.774767] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[15639.775699] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15639.776639] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15639.777580] CR2: 0000003370219050 CR3: 0000000001c11000 CR4: 00000000001407f0
[15639.778527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15639.779463] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15639.780391] Stack:
[15639.781314]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000005
[15639.782258]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15639.783208]  00000e3b7b79aa5b ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15639.784151] Call Trace:
[15639.785088]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15639.786034]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15639.786978]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15639.787921]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15639.788869]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15639.789815]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15639.790758]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15639.791704]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15639.792649]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15639.793603]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15639.794549] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15667.636533] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c206:14314]
[15667.637609] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15667.643390] CPU: 1 PID: 14314 Comm: trinity-c206 Tainted: G             L 3.18.0+ #107
[15667.645698] task: ffff880227440000 ti: ffff880227c7c000 task.ti: ffff880227c7c000
[15667.646864] RIP: 0010:[<ffffffff81372bc5>]  [<ffffffff81372bc5>] copy_user_enhanced_fast_string+0x5/0x10
[15667.648013] RSP: 0018:ffff880227c7fc60  EFLAGS: 00010286
[15667.649167] RAX: ffff880152231000 RBX: ffffffff81175db2 RCX: 0000000000000fc0
[15667.650341] RDX: 0000000000001000 RSI: ffff880152231040 RDI: 00007fdd8cc56b7f
[15667.651499] RBP: ffff880227c7fcb8 R08: ffff880152231000 R09: ffff8800982fe9f0
[15667.652654] R10: ffff880227c7fb88 R11: 0000000000000000 R12: ffffea0005488c40
[15667.653825] R13: 000100000009cf84 R14: 0000000000000246 R15: ffff880227c7fbf8
[15667.654982] FS:  00007fdd90bc4740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15667.656129] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15667.657274] CR2: 00007fdd8f9f58e0 CR3: 0000000224a01000 CR4: 00000000001407e0
[15667.658421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15667.659556] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[15667.660673] Stack:
[15667.661780]  ffffffff8119f74d ffff880152231000 00007fdd8cc56b3f ffff880152231000
[15667.662915]  ffffea0005488c40 ffffea0005488c40 000100000009cf84 0000000000000000
[15667.664058]  ffff8802252be820 0000000000001000 0000000000000000 ffff880227c7fd48
[15667.665205] Call Trace:
[15667.666336]  [<ffffffff8119f74d>] ? copy_page_to_iter+0x19d/0x340
[15667.667512]  [<ffffffff81190b7b>] shmem_file_read_iter+0xcb/0x300
[15667.668671]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15667.669837]  [<ffffffff811e6cc0>] ? do_sync_readv_writev+0xa0/0xa0
[15667.671003]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15667.672149]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15667.673252]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15667.674329]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15667.675387]  [<ffffffff811e8909>] vfs_readv+0x39/0x50
[15667.676432]  [<ffffffff811e89cc>] SyS_readv+0x5c/0x100
[15667.677537]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15667.678574] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[15667.680846] sending NMI to other CPUs:
[15667.681910] NMI backtrace for cpu 2
[15667.682941] CPU: 2 PID: 15662 Comm: trinity-c188 Tainted: G             L 3.18.0+ #107
[15667.685036] task: ffff8801db9e0000 ti: ffff880226ca8000 task.ti: ffff880226ca8000
[15667.686094] RIP: 0010:[<ffffffff810c44d6>]  [<ffffffff810c44d6>] lock_acquired+0x46/0x370
[15667.687169] RSP: 0018:ffff880226cabdf8  EFLAGS: 00000046
[15667.688244] RAX: 0000000000000001 RBX: ffffffff81c0a080 RCX: ffff8802453cff98
[15667.689324] RDX: 0000000000000680 RSI: ffffffff8107a8d9 RDI: ffffffff81c0a098
[15667.690392] RBP: ffff880226cabe38 R08: 0000000000000000 R09: 0000000000000001
[15667.691464] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801db9e0000
[15667.692532] R13: ffffffff81c0a098 R14: 0000000000000246 R15: ffff8801db9e0000
[15667.693595] FS:  00007fdd90bc4740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15667.694665] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15667.695738] CR2: 000000336f1b7740 CR3: 0000000223f31000 CR4: 00000000001407e0
[15667.696820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15667.697901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[15667.698977] Stack:
[15667.700049]  0000000125edaea0 ffffffff8107a8d9 ffff8801db9e0000 ffffffff81c0a080
[15667.701151]  ffffffff81c0a098 ffff8801db9e0000 ffff8801db9e0000 ffff8801db9e0000
[15667.702257]  ffff880226cabe68 ffffffff817ccf3d ffffffff8107a8d9 ffff880226cabef8
[15667.703361] Call Trace:
[15667.704456]  [<ffffffff8107a8d9>] ? do_wait+0xd9/0x280
[15667.705539]  [<ffffffff817ccf3d>] _raw_read_lock+0x6d/0x80
[15667.706599]  [<ffffffff8107a8d9>] ? do_wait+0xd9/0x280
[15667.707646]  [<ffffffff8107a8d9>] do_wait+0xd9/0x280
[15667.708669]  [<ffffffff8107aea0>] SyS_wait4+0x80/0x110
[15667.709667]  [<ffffffff81078980>] ? task_stopped_code+0x60/0x60
[15667.710642]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15667.711594] Code: 00 45 85 c9 0f 84 d8 00 00 00 65 4c 8b 24 25 00 aa 00 00 45 8b 84 24 6c 07 00 00 45 85 c0 0f 85 be 00 00 00 49 89 fd 9c 41 5e fa <8b> 35 3c ed aa 01 41 c7 84 24 6c 07 00 00 01 00 00 00 41 8b 9c 
[15667.713623] NMI backtrace for cpu 0
[15667.714570] CPU: 0 PID: 12614 Comm: trinity-c134 Tainted: G             L 3.18.0+ #107
[15667.716458] task: ffff8802254716d0 ti: ffff88009463c000 task.ti: ffff88009463c000
[15667.717406] RIP: 0010:[<ffffffff8136c297>]  [<ffffffff8136c297>] rb_first+0x17/0x30
[15667.718363] RSP: 0018:ffff88009463fd10  EFLAGS: 00000282
[15667.719311] RAX: ffff8801be9e1220 RBX: ffff8800955da020 RCX: ffff8800955da620
[15667.720271] RDX: ffff8801be9e2220 RSI: 0000000000000000 RDI: ffff88009ee6abc8
[15667.721229] RBP: ffff88009463fd10 R08: 0000000000000000 R09: ffff8801beaa8398
[15667.722185] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800955da020
[15667.723129] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8800955da620
[15667.724066] FS:  00007fdd90bc4740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15667.725004] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15667.725946] CR2: 0000000000e308f0 CR3: 000000007260b000 CR4: 00000000001407f0
[15667.726902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15667.727844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[15667.728781] Stack:
[15667.729706]  ffff88009463fd40 ffffffff811a98ea ffff8801beaa83b0 ffff8800955da020
[15667.730658]  ffff8800955da020 ffff88009ee6abc8 ffff88009463fd80 ffffffff811aa413
[15667.731604]  ffff88009463fd70 ffff8800955da000 ffff8801beaa83b0 0000000001200011
[15667.732545] Call Trace:
[15667.733480]  [<ffffffff811a98ea>] validate_mm_rb+0x1a/0x70
[15667.734421]  [<ffffffff811aa413>] __vma_link_rb+0xc3/0x100
[15667.735361]  [<ffffffff81076465>] copy_process.part.26+0x1445/0x1a40
[15667.736305]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15667.737250]  [<ffffffff81076c27>] do_fork+0xe7/0x490
[15667.738191]  [<ffffffff81012217>] ? syscall_trace_enter_phase2+0xa7/0x200
[15667.739138]  [<ffffffff81077056>] SyS_clone+0x16/0x20
[15667.740081]  [<ffffffff817cddd9>] stub_clone+0x69/0x90
[15667.741024]  [<ffffffff817cdc49>] ? tracesys_phase2+0xd4/0xd9
[15667.741969] Code: 31 c0 e8 18 1e 45 00 eb d2 90 90 90 90 90 90 90 90 90 90 90 48 8b 07 55 48 89 e5 48 85 c0 75 07 eb 10 66 90 48 89 d0 48 8b 50 10 <48> 85 d2 75 f4 5d c3 31 c0 5d c3 66 66 66 66 66 2e 0f 1f 84 00 
[15667.743980] NMI backtrace for cpu 3
[15667.744977] CPU: 3 PID: 15728 Comm: trinity-c168 Tainted: G             L 3.18.0+ #107
[15667.747026] task: ffff88009af74470 ti: ffff88022752c000 task.ti: ffff88022752c000
[15667.748038] RIP: 0010:[<ffffffff8117d366>]  [<ffffffff8117d366>] free_pcppages_bulk+0x3c6/0x530
[15667.749008] RSP: 0018:ffff88022752fa98  EFLAGS: 00000087
[15667.749964] RAX: ffffea0006cb6000 RBX: 0000000000000584 RCX: 0000000000000002
[15667.750928] RDX: 00000000ffffff80 RSI: 000000000000000a RDI: 00000000ffffff80
[15667.751884] RBP: ffff88022752fb28 R08: ffffea0006cb6100 R09: ffff88024e5d3ed8
[15667.752845] R10: 0000000000000002 R11: 0000000000000001 R12: 0000000000000580
[15667.753803] R13: 0000000000000002 R14: ffffea0006cb6180 R15: 0000000000000002
[15667.754760] FS:  00007fdd90bc4740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15667.755727] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15667.756685] CR2: 0000000000000000 CR3: 0000000223c1b000 CR4: 00000000001407e0
[15667.757649] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15667.758606] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000070602
[15667.759561] Stack:
[15667.760508]  ffff88022752fb18 ffff88024e5d3f40 ffff8802455d7328 0000000900000002
[15667.761476]  0000000000000002 0000000000000586 ffffea0006cb6180 ffff88024e5d3ed8
[15667.762445]  ffffea0006cb6100 000000090000000a ffff8802455d7358 ffff88024e5d3e80
[15667.763410] Call Trace:
[15667.764368]  [<ffffffff8117d7f3>] free_hot_cold_page+0x173/0x1a0
[15667.765331]  [<ffffffff8117d885>] free_hot_cold_page_list+0x65/0xd0
[15667.766290]  [<ffffffff811843ed>] release_pages+0x1bd/0x270
[15667.767252]  [<ffffffff81185423>] __pagevec_release+0x43/0x60
[15667.768213]  [<ffffffff81191430>] shmem_undo_range+0x460/0x710
[15667.769178]  [<ffffffff811916f8>] shmem_truncate_range+0x18/0x40
[15667.770143]  [<ffffffff81191986>] shmem_setattr+0x116/0x1a0
[15667.771107]  [<ffffffff81206a21>] notify_change+0x241/0x390
[15667.772067]  [<ffffffff811e4f25>] do_truncate+0x75/0xc0
[15667.773009]  [<ffffffff811e529a>] ? do_sys_ftruncate.constprop.14+0xda/0x160
[15667.773932]  [<ffffffff811e52cf>] do_sys_ftruncate.constprop.14+0x10f/0x160
[15667.774858]  [<ffffffff811e535e>] SyS_ftruncate+0xe/0x10
[15667.775773]  [<ffffffff817cdc49>] tracesys_phase2+0xd4/0xd9
[15667.776667] Code: 48 c1 e0 06 49 01 c0 41 39 f7 73 24 41 bc 01 00 00 00 41 d3 e4 4d 63 e4 49 31 dc 4c 89 e0 48 29 d8 48 c1 e0 06 4c 01 c0 8b 50 18 <83> fa 80 74 0d 44 89 f9 e9 4d fd ff ff 0f 1f 44 00 00 48 8d 51 
[15695.620295] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c243:22756]
[15695.621317] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15695.626986] CPU: 1 PID: 22756 Comm: trinity-c243 Tainted: G             L 3.18.0+ #107
[15695.629248] task: ffff8801dbac0000 ti: ffff880222ec8000 task.ti: ffff880222ec8000
[15695.630424] RIP: 0010:[<ffffffff81372bc5>]  [<ffffffff81372bc5>] copy_user_enhanced_fast_string+0x5/0x10
[15695.631649] RSP: 0000:ffff880222ecbbe0  EFLAGS: 00010202
[15695.632813] RAX: 00007f5352671e15 RBX: 0000000000000003 RCX: 00000000000003c0
[15695.633981] RDX: 0000000000001000 RSI: 00007f5352672a55 RDI: ffff880022849c40
[15695.635160] RBP: ffff880222ecbc28 R08: 0000000000000000 R09: 0000000000000001
[15695.636360] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817ccd6b
[15695.637495] R13: ffff880222ecbb68 R14: ffff88007758c6d0 R15: ffff88007758c7f0
[15695.638629] FS:  00007f53535df740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15695.639773] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15695.640919] CR2: 00007f5352440748 CR3: 0000000227cf6000 CR4: 00000000001407e0
[15695.642073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15695.643247] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15695.644422] Stack:
[15695.645589]  ffffffff8119f2e6 ffff880222ecbbf8 00000000119c1000 0000000000001000
[15695.646776]  00000000119c1000 0000000000001000 ffff880222ecbd60 0000000000000000
[15695.647982]  ffff88007758ca00 ffff880222ecbcc8 ffffffff811754b7 ffff880222ecbc88
[15695.649174] Call Trace:
[15695.650348]  [<ffffffff8119f2e6>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[15695.651487]  [<ffffffff811754b7>] generic_perform_write+0xf7/0x1f0
[15695.652622]  [<ffffffff81177bf2>] __generic_file_write_iter+0x162/0x350
[15695.653748]  [<ffffffff811e6d90>] ? new_sync_read+0xd0/0xd0
[15695.654855]  [<ffffffff81177e1f>] generic_file_write_iter+0x3f/0xb0
[15695.655943]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15695.657028]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15695.658103]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15695.659170]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15695.660255]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15695.661334]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15695.662409]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15695.663490]  [<ffffffff811e895c>] vfs_writev+0x3c/0x50
[15695.664559]  [<ffffffff811e8acc>] SyS_writev+0x5c/0x100
[15695.665610]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15695.666663] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[15695.668947] sending NMI to other CPUs:
[15695.670050] NMI backtrace for cpu 0
[15695.671085] CPU: 0 PID: 24460 Comm: trinity-c223 Tainted: G             L 3.18.0+ #107
[15695.673189] task: ffff88022551db40 ti: ffff88009afa8000 task.ti: ffff88009afa8000
[15695.674252] RIP: 0010:[<ffffffff815f25f2>]  [<ffffffff815f25f2>] xhci_irq+0x42/0x1c70
[15695.675331] RSP: 0018:ffff880244e03dd8  EFLAGS: 00000086
[15695.676404] RAX: 0000000000000008 RBX: ffff8802400bd480 RCX: 0000000000000000
[15695.677487] RDX: ffff880244e0c3c0 RSI: ffff880240298060 RDI: ffff88022551e2b0
[15695.678572] RBP: ffff880244e03e78 R08: 0000000000000000 R09: 0000000000000001
[15695.679657] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000001c
[15695.680744] R13: ffff8800a171dab8 R14: ffff880240298048 R15: ffff880240298000
[15695.681831] FS:  00007f53535df740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15695.682930] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15695.684032] CR2: 00007f53535ada54 CR3: 00000000a1a8c000 CR4: 00000000001407f0
[15695.685141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15695.686246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15695.687345] Stack:
[15695.688438]  ffff880244e03de8 ffffffff810ab7d5 ffff880244e03e68 ffffffff810c4e0c
[15695.689541]  ffff880240d05450 0000000000000000 00000000fffe8920 0000000000000046
[15695.690619]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[15695.691688] Call Trace:
[15695.692723]  <IRQ> 

[15695.693742]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15695.694734]  [<ffffffff810c4e0c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15695.695708]  [<ffffffff815f4231>] xhci_msi_irq+0x11/0x20
[15695.696664]  [<ffffffff810cf8f5>] handle_irq_event_percpu+0x55/0x1f0
[15695.697605]  [<ffffffff810cfad1>] handle_irq_event+0x41/0x70
[15695.698524]  [<ffffffff810d2b5e>] ? handle_edge_irq+0x1e/0x140
[15695.699425]  [<ffffffff810d2bbf>] handle_edge_irq+0x7f/0x140
[15695.700311]  [<ffffffff810053a1>] handle_irq+0xb1/0x140
[15695.701189]  [<ffffffff8107a9f1>] ? do_wait+0x1f1/0x280
[15695.702059]  [<ffffffff817d0413>] do_IRQ+0x53/0x100
[15695.702916]  [<ffffffff8107a9f1>] ? do_wait+0x1f1/0x280
[15695.703776]  [<ffffffff817ce56f>] common_interrupt+0x6f/0x6f
[15695.704634]  <EOI> 

[15695.705488]  [<ffffffff810c6150>] ? lock_release+0xc0/0x240
[15695.706335]  [<ffffffff817cd103>] _raw_read_unlock+0x23/0x40
[15695.707179]  [<ffffffff8107a9f1>] do_wait+0x1f1/0x280
[15695.708016]  [<ffffffff8107aea0>] SyS_wait4+0x80/0x110
[15695.708850]  [<ffffffff81078980>] ? task_stopped_code+0x60/0x60
[15695.709689]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15695.710525] Code: ec 78 65 48 8b 04 25 28 00 00 00 48 89 45 c8 31 c0 4c 8b bf d0 03 00 00 4d 8d 77 48 4c 89 f7 e8 b5 a3 1d 00 49 8b 47 18 8b 40 04 <83> f8 ff 0f 84 09 01 00 00 a8 08 0f 84 35 02 00 00 a8 04 0f 85 
[15695.712357] NMI backtrace for cpu 3
[15695.713263] CPU: 3 PID: 21578 Comm: trinity-c97 Tainted: G             L 3.18.0+ #107
[15695.715097] task: ffff88009acc2da0 ti: ffff88019cdf8000 task.ti: ffff88019cdf8000
[15695.716028] RIP: 0010:[<ffffffff81185b7b>]  [<ffffffff81185b7b>] cancel_dirty_page+0xb/0xc0
[15695.716970] RSP: 0018:ffff88019cdfbc40  EFLAGS: 00000246
[15695.717914] RAX: 00000000fffffffb RBX: ffffea0007bfe600 RCX: 0000000000000415
[15695.718861] RDX: ffffea0007bfe600 RSI: 0000000000001000 RDI: ffffea0007bfe600
[15695.719804] RBP: ffff88019cdfbc48 R08: 0000000000000000 R09: ffff88010f861c40
[15695.720746] R10: ffff88019cdfbbd8 R11: 0000000000000000 R12: ffff880227e63040
[15695.721687] R13: ffff88019cdfbd40 R14: 0000000000000000 R15: 0000000000047bc6
[15695.722623] FS:  00007f53535df740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15695.723585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15695.724539] CR2: 000000336f1b7740 CR3: 00000002240c0000 CR4: 00000000001407e0
[15695.725502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15695.726465] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15695.727420] Stack:
[15695.728374]  ffff88019cdfbd40 ffff88019cdfbc68 ffffffff8118609e 0000000000000008
[15695.729334]  ffff88019cdfbcd0 ffff88019cdfbdf8 ffffffff811913f3 ffffea0006fad800
[15695.730268]  ffff880227e62e30 0000000000000000 0000000000000000 0000000000000000
[15695.731203] Call Trace:
[15695.732131]  [<ffffffff8118609e>] truncate_inode_page+0x4e/0x90
[15695.733069]  [<ffffffff811913f3>] shmem_undo_range+0x423/0x710
[15695.733989]  [<ffffffff811916f8>] shmem_truncate_range+0x18/0x40
[15695.734891]  [<ffffffff81191986>] shmem_setattr+0x116/0x1a0
[15695.735787]  [<ffffffff81206a21>] notify_change+0x241/0x390
[15695.736664]  [<ffffffff811e4f25>] do_truncate+0x75/0xc0
[15695.737535]  [<ffffffff811e529a>] ? do_sys_ftruncate.constprop.14+0xda/0x160
[15695.738413]  [<ffffffff811e52cf>] do_sys_ftruncate.constprop.14+0x10f/0x160
[15695.739291]  [<ffffffff8137432e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[15695.740176]  [<ffffffff811e535e>] SyS_ftruncate+0xe/0x10
[15695.741050]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15695.741916] Code: eb d5 be f7 02 00 00 48 c7 c7 a2 cc a6 81 48 89 55 d8 e8 e9 21 ef ff 48 8b 55 d8 e9 4f ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 55 <41> 54 41 89 f4 53 48 83 ec 08 f0 0f ba 37 04 72 14 48 83 c4 08 
[15695.743812] NMI backtrace for cpu 2
[15695.744736] CPU: 2 PID: 24408 Comm: trinity-c238 Tainted: G             L 3.18.0+ #107
[15695.746590] task: ffff880240cd8000 ti: ffff880226d38000 task.ti: ffff880226d38000
[15695.747528] RIP: 0010:[<ffffffff81372bc5>]  [<ffffffff81372bc5>] copy_user_enhanced_fast_string+0x5/0x10
[15695.748479] RSP: 0018:ffff880226d3bc60  EFLAGS: 00010286
[15695.749430] RAX: ffff8800020a1000 RBX: ffff880226d3bd60 RCX: 00000000000009c0
[15695.750385] RDX: 0000000000001000 RSI: ffff8800020a1640 RDI: 00007f5351be001e
[15695.751344] RBP: ffff880226d3bcb8 R08: ffff8800020a1000 R09: ffff8800981a8388
[15695.752303] R10: ffff880226d3bb88 R11: 0000000000000000 R12: 0000000000001000
[15695.753261] R13: 000000000018a9de R14: ffff880094522380 R15: 0000000000001000
[15695.754220] FS:  00007f53535df740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15695.755190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15695.756162] CR2: 00007f53524a684a CR3: 0000000226d63000 CR4: 00000000001407e0
[15695.757142] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15695.758127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15695.759108] Stack:
[15695.760127]  ffffffff8119f74d ffff8800020a1000 00007f5351bdf9de ffff8800020a1000
[15695.761224]  ffffea0000082840 ffffea00051adc80 0000000000001965 0000000000000000
[15695.762316]  ffff880227e64e60 0000000000001000 0000000000000000 ffff880226d3bd48
[15695.763408] Call Trace:
[15695.764482]  [<ffffffff8119f74d>] ? copy_page_to_iter+0x19d/0x340
[15695.765503]  [<ffffffff81190b7b>] shmem_file_read_iter+0xcb/0x300
[15695.766524]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15695.767542]  [<ffffffff811e6cc0>] ? do_sync_readv_writev+0xa0/0xa0
[15695.768565]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15695.769583]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15695.770606]  [<ffffffff81190ab0>] ? shmem_fault+0x1c0/0x1c0
[15695.771609]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15695.772593]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15695.773598]  [<ffffffff811e8909>] vfs_readv+0x39/0x50
[15695.774616]  [<ffffffff811e8c32>] SyS_preadv+0xc2/0xf0
[15695.775607]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15695.776553] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[15723.604074] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-main:773]
[15723.605115] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15723.610645] CPU: 1 PID: 773 Comm: trinity-main Tainted: G             L 3.18.0+ #107
[15723.612819] task: ffff880240c116d0 ti: ffff8800582d4000 task.ti: ffff8800582d4000
[15723.613925] RIP: 0010:[<ffffffff817c0f17>]  [<ffffffff817c0f17>] __slab_alloc+0x52f/0x58f
[15723.615063] RSP: 0018:ffff8800582d7a58  EFLAGS: 00000246
[15723.616174] RAX: 0000000000000001 RBX: ffff88014bc57900 RCX: 00000000000002a0
[15723.617287] RDX: ffff88024500d620 RSI: 0000000000000000 RDI: ffff88024483d600
[15723.618407] RBP: ffff8800582d7b48 R08: 0000000000000000 R09: 0000000000000000
[15723.619522] R10: 0000000000000092 R11: ffff88014bc54f30 R12: ffffffff810135bf
[15723.620632] R13: ffff8800582d79d8 R14: 0000000100190010 R15: ffffffff81202809
[15723.621744] FS:  00007f2495d4c740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15723.622868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15723.623990] CR2: 0000000000e39000 CR3: 0000000094e20000 CR4: 00000000001407e0
[15723.625133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15723.626265] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15723.627394] Stack:
[15723.628515]  ffff8800582d7a68 ffffffff810ab7d5 ffff8800582d7ae8 ffffffff810c4e0c
[15723.629664]  ffff8800582d7a98 ffffffff810c32ff ffff880240c116d0 0000000000000000
[15723.630811]  ffff8802451d80a0 0000000200000092 ffff8800582d7ad8 ffffffff81202809
[15723.631984] Call Trace:
[15723.633117]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15723.634276]  [<ffffffff810c4e0c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15723.635346]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15723.636425]  [<ffffffff81202809>] ? __d_alloc+0x29/0x1d0
[15723.637510]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15723.638551]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15723.639590]  [<ffffffff81202809>] ? __d_alloc+0x29/0x1d0
[15723.640631]  [<ffffffff811d172b>] kmem_cache_alloc+0x1cb/0x1f0
[15723.641649]  [<ffffffff81202fcf>] ? __d_lookup+0xdf/0x1c0
[15723.642661]  [<ffffffff81202ef5>] ? __d_lookup+0x5/0x1c0
[15723.643670]  [<ffffffff81202809>] __d_alloc+0x29/0x1d0
[15723.644712]  [<ffffffff812029d1>] d_alloc+0x21/0x80
[15723.645731]  [<ffffffff811f217b>] lookup_dcache+0x8b/0xb0
[15723.646734]  [<ffffffff817c128a>] ? lookup_slow+0x38/0xad
[15723.647731]  [<ffffffff811f21cd>] __lookup_hash+0x2d/0x60
[15723.648709]  [<ffffffff817c1299>] lookup_slow+0x47/0xad
[15723.649687]  [<ffffffff811f76a8>] path_lookupat+0x748/0x770
[15723.650669]  [<ffffffff811d1666>] ? kmem_cache_alloc+0x106/0x1f0
[15723.651666]  [<ffffffff811f879f>] ? getname_flags+0x4f/0x1a0
[15723.652634]  [<ffffffff811f76fb>] filename_lookup+0x2b/0xc0
[15723.653603]  [<ffffffff811f97c3>] user_path_at_empty+0x63/0xc0
[15723.654619]  [<ffffffff811a1ad7>] ? might_fault+0x47/0x50
[15723.655582]  [<ffffffff811ece47>] ? cp_new_stat+0x157/0x190
[15723.656540]  [<ffffffff811f9831>] user_path_at+0x11/0x20
[15723.657495]  [<ffffffff811ec983>] vfs_fstatat+0x63/0xc0
[15723.658458]  [<ffffffff811ecf64>] SYSC_newfstatat+0x24/0x60
[15723.659412]  [<ffffffff8137432e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[15723.660354]  [<ffffffff811ed1ce>] SyS_newfstatat+0xe/0x10
[15723.661310]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15723.662268] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 9a 87 98 ff 4c 89 e0 eb 0f e8 90 88 98 ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 55 c8 65 48 33 14 25 28 00 00 00 74 3c e8 02 6b 
[15723.664427] sending NMI to other CPUs:
[15723.665416] NMI backtrace for cpu 2
[15723.666409] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #107
[15723.668424] task: ffff88024348c470 ti: ffff88024371c000 task.ti: ffff88024371c000
[15723.669432] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15723.670429] RSP: 0018:ffff88024371fe08  EFLAGS: 00000046
[15723.671412] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[15723.672382] RDX: 0000000000000000 RSI: ffff88024371ffd8 RDI: 0000000000000002
[15723.673330] RBP: ffff88024371fe38 R08: 000000008baf90c8 R09: 0000000000000000
[15723.674260] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[15723.675167] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88024371c000
[15723.676047] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15723.676921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15723.677771] CR2: 0000000000da6000 CR3: 000000007252c000 CR4: 00000000001407e0
[15723.678613] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15723.679446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15723.680268] Stack:
[15723.681077]  000000024371fe38 87ce37a9ab100d43 ffffe8ffff202118 0000000000000002
[15723.681913]  ffffffff81cb19c0 0000000000000002 ffff88024371fe88 ffffffff8165d385
[15723.682749]  00000e4f0b188364 ffffffff81cb1a88 ffffffff81cb19c0 ffffffff81d213f0
[15723.683590] Call Trace:
[15723.684416]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15723.685247]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15723.686064]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15723.686880]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15723.687697] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15723.689514] NMI backtrace for cpu 0
[15723.690374] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15723.692130] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15723.693019] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15723.693920] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15723.694813] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[15723.695721] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15723.696616] RBP: ffffffff81c03e68 R08: 000000008baf90c8 R09: 0000000000000000
[15723.697519] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004
[15723.698420] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c00000
[15723.699315] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15723.700217] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15723.701120] CR2: 0000003370219050 CR3: 0000000001c11000 CR4: 00000000001407f0
[15723.702038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15723.702952] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15723.703863] Stack:
[15723.704771]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000004
[15723.705706]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15723.706649]  00000e4f0b0b0da0 ffffffff81cb1b38 ffffffff81cb19c0 ffffffff81d213f0
[15723.707593] Call Trace:
[15723.708528]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15723.709476]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15723.710415]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15723.711355]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15723.712298]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15723.713241]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15723.714184]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15723.715128]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15723.716070]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15723.717018]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15723.717959] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15723.720020] NMI backtrace for cpu 3
[15723.721025] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15723.723020] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15723.724043] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15723.725069] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15723.726079] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15723.727071] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15723.728051] RBP: ffff88024372be38 R08: 000000008baf90c8 R09: 0000000000000000
[15723.729029] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15723.730028] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880243728000
[15723.731028] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15723.732019] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15723.732997] CR2: 0000003370219050 CR3: 0000000001c11000 CR4: 00000000001407e0
[15723.734000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15723.734971] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15723.735943] Stack:
[15723.736901]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000005
[15723.737889]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15723.738904]  00000e4f0b04a497 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15723.739903] Call Trace:
[15723.740880]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15723.741867]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15723.742849]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15723.743828]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15723.744820] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15751.587822] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c127:9822]
[15751.588924] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15751.594608] CPU: 1 PID: 9822 Comm: trinity-c127 Tainted: G             L 3.18.0+ #107
[15751.596824] task: ffff880229bb96d0 ti: ffff8801699f4000 task.ti: ffff8801699f4000
[15751.597973] RIP: 0010:[<ffffffff81117f09>]  [<ffffffff81117f09>] map_id_up+0x9/0x80
[15751.599126] RSP: 0018:ffff8801699f7e78  EFLAGS: 00000246
[15751.600280] RAX: ffff88002394c000 RBX: 0000000000000292 RCX: ffff88022357edb0
[15751.601445] RDX: ffffffff81c48680 RSI: 00000000000003e8 RDI: ffffffff81c41700
[15751.602611] RBP: ffff8801699f7e88 R08: 00007f2495d4c740 R09: 0000000000000001
[15751.603772] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000096
[15751.604933] R13: ffff8801699f7e18 R14: ffff88022fdd3e38 R15: ffff880229bb96d0
[15751.606071] FS:  00007f2495d4c740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15751.607202] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15751.608347] CR2: 0000000000639210 CR3: 00000001dbbc4000 CR4: 00000000001407e0
[15751.609502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15751.610645] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15751.611774] Stack:
[15751.612884]  ffff8801699f7e88 ffffffff81117fae ffff8801699f7f68 ffffffff81086dda
[15751.614045]  00007f2495542000 0000000000000000 ffff880229bb9aa0 0000000000000000
[15751.615175]  ffff880100000000 000000000000265e ffff880061283800 00007fffa68babd0
[15751.616317] Call Trace:
[15751.617454]  [<ffffffff81117fae>] ? from_kuid_munged+0xe/0x20
[15751.618617]  [<ffffffff81086dda>] SYSC_kill+0x7a/0x240
[15751.619749]  [<ffffffff8107aeab>] ? SyS_wait4+0x8b/0x110
[15751.620889]  [<ffffffff81088cce>] SyS_kill+0xe/0x10
[15751.622034]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15751.623177] Code: 04 52 5d 03 74 87 08 89 f0 29 c8 c3 8b 47 0c 01 c8 83 e8 01 39 c6 77 a7 31 d2 eb e2 66 0f 1f 44 00 00 0f 1f 44 00 00 55 44 8b 0f <48> 89 e5 45 85 c9 b8 ff ff ff ff 74 3c 8b 4f 08 39 ce 73 4d 48 
[15751.625658] sending NMI to other CPUs:
[15751.626847] NMI backtrace for cpu 0
[15751.627968] CPU: 0 PID: 9692 Comm: trinity-c41 Tainted: G             L 3.18.0+ #107
[15751.630204] task: ffff88020ce1c470 ti: ffff88009acb0000 task.ti: ffff88009acb0000
[15751.631339] RIP: 0010:[<ffffffff810c93c0>]  [<ffffffff810c93c0>] do_raw_spin_unlock+0x0/0xa0
[15751.632467] RSP: 0018:ffff88009acb3b10  EFLAGS: 00000092
[15751.633576] RAX: ffff88020ce1c470 RBX: ffff88009d38f728 RCX: 0000000000002640
[15751.634685] RDX: ffff880244e1cf80 RSI: 0000000000000000 RDI: ffff88009d38f728
[15751.635789] RBP: ffff88009acb3b28 R08: 0000000000000000 R09: 0000000000000001
[15751.636888] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88009d38f710
[15751.637990] R13: 0000000000000000 R14: ffff88009d38f728 R15: 0000000000000000
[15751.639090] FS:  00007f2495d4c740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15751.640201] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15751.641312] CR2: 0000000000000004 CR3: 0000000097156000 CR4: 00000000001407f0
[15751.642424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15751.643531] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15751.644627] Stack:
[15751.645719]  ffffffff817cce4b ffff88009acb3b28 ffffea000820ddc0 ffff88009acb3b68
[15751.646844]  ffffffff8118f783 ffff88009acb3b58 0000000000000003 0000000000000000
[15751.647958]  0000000000000000 ffff88009d38f500 0000000000000000 ffff88009acb3c18
[15751.649070] Call Trace:
[15751.650165]  [<ffffffff817cce4b>] ? _raw_spin_unlock_irq+0x2b/0x40
[15751.651269]  [<ffffffff8118f783>] shmem_add_to_page_cache+0xe3/0x140
[15751.652368]  [<ffffffff8118ff9c>] shmem_getpage_gfp+0x50c/0x7a0
[15751.653456]  [<ffffffff81190272>] shmem_write_begin+0x42/0x70
[15751.654552]  [<ffffffff81175494>] generic_perform_write+0xd4/0x1f0
[15751.655646]  [<ffffffff81177bf2>] __generic_file_write_iter+0x162/0x350
[15751.656742]  [<ffffffff811e6d90>] ? new_sync_read+0xd0/0xd0
[15751.657819]  [<ffffffff81177e1f>] generic_file_write_iter+0x3f/0xb0
[15751.658872]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15751.659919]  [<ffffffff811e6ed8>] do_iter_readv_writev+0x78/0xc0
[15751.660948]  [<ffffffff811e8708>] do_readv_writev+0xd8/0x2a0
[15751.661950]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15751.662938]  [<ffffffff81177de0>] ? __generic_file_write_iter+0x350/0x350
[15751.663898]  [<ffffffff810c32ff>] ? lock_release_holdtime.part.24+0xf/0x190
[15751.664836]  [<ffffffff817cce50>] ? _raw_spin_unlock_irq+0x30/0x40
[15751.665755]  [<ffffffff811e895c>] vfs_writev+0x3c/0x50
[15751.666657]  [<ffffffff811e8acc>] SyS_writev+0x5c/0x100
[15751.667535]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15751.668406] Code: 00 89 d0 f0 66 0f b1 37 66 39 d0 75 e0 b1 01 5d 65 8b 04 25 2c a0 00 00 89 47 08 65 48 8b 04 25 00 aa 00 00 48 89 47 10 89 c8 c3 <0f> 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 81 7f 04 ad 
[15751.670313] NMI backtrace for cpu 3
[15751.671258] CPU: 3 PID: 8780 Comm: trinity-c136 Tainted: G             L 3.18.0+ #107
[15751.673201] task: ffff88009ed316d0 ti: ffff8802333fc000 task.ti: ffff8802333fc000
[15751.674212] RIP: 0010:[<ffffffff81185b90>]  [<ffffffff81185b90>] cancel_dirty_page+0x20/0xc0
[15751.675256] RSP: 0018:ffff8802333ffc30  EFLAGS: 00000296
[15751.676330] RAX: ffffffff81d24160 RBX: ffff88016985bd20 RCX: 0000000000005ed5
[15751.677366] RDX: ffffea0005d7c700 RSI: 0000000000001000 RDI: ffffea0005d7c700
[15751.678392] RBP: ffff8802333ffc48 R08: 0000000000000000 R09: 0000000000000046
[15751.679383] R10: ffff8802333ffbd8 R11: 0000000000000000 R12: 0000000000001000
[15751.680368] R13: ffff8802333ffd40 R14: 0000000000000000 R15: 0000800000129380
[15751.681341] FS:  00007f2495d4c740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15751.682321] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15751.683301] CR2: 00000000ffffff48 CR3: 0000000226fe9000 CR4: 00000000001407e0
[15751.684300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15751.685278] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15751.686254] Stack:
[15751.687221]  ffffea0005d7c700 ffff88016985bd20 ffff8802333ffd40 ffff8802333ffc68
[15751.688225]  ffffffff8118609e 0000000000000009 ffff8802333ffcd0 ffff8802333ffdf8
[15751.689222]  ffffffff811913f3 ffff880243b12f40 ffff88016985bb10 000000eb00000000
[15751.690199] Call Trace:
[15751.691159]  [<ffffffff8118609e>] truncate_inode_page+0x4e/0x90
[15751.692117]  [<ffffffff811913f3>] shmem_undo_range+0x423/0x710
[15751.693078]  [<ffffffff811916f8>] shmem_truncate_range+0x18/0x40
[15751.694033]  [<ffffffff81191986>] shmem_setattr+0x116/0x1a0
[15751.694964]  [<ffffffff81206a21>] notify_change+0x241/0x390
[15751.695880]  [<ffffffff811e4f25>] do_truncate+0x75/0xc0
[15751.696783]  [<ffffffff811e529a>] ? do_sys_ftruncate.constprop.14+0xda/0x160
[15751.697697]  [<ffffffff811e52cf>] do_sys_ftruncate.constprop.14+0x10f/0x160
[15751.698595]  [<ffffffff8137432e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[15751.699490]  [<ffffffff811e5370>] compat_SyS_ftruncate+0x10/0x20
[15751.700403]  [<ffffffff817d0249>] ia32_do_call+0x13/0x13
[15751.701301] Code: ef ff 48 8b 55 d8 e9 4f ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 41 89 f4 53 48 83 ec 08 f0 0f ba 37 04 72 14 48 83 c4 08 <5b> 41 5c 41 5d 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 5f 08 48 
[15751.703287] NMI backtrace for cpu 2
[15751.704239] CPU: 2 PID: 9809 Comm: trinity-c81 Tainted: G             L 3.18.0+ #107
[15751.706139] task: ffff8802277e0000 ti: ffff880222cb0000 task.ti: ffff880222cb0000
[15751.707119] RIP: 0010:[<ffffffff810c4d22>]  [<ffffffff810c4d22>] __lock_acquire.isra.31+0x142/0x9f0
[15751.708111] RSP: 0018:ffff880222cb3d68  EFLAGS: 00000002
[15751.709102] RAX: 0000000000000000 RBX: ffff8802277e0000 RCX: 0000000000000002
[15751.710083] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81c50da0
[15751.711071] RBP: ffff880222cb3dd8 R08: 0000000000000000 R09: 0000000000000000
[15751.712049] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82596770
[15751.713027] R13: 0000000000000000 R14: ffffffff81c50da0 R15: 0000000000000000
[15751.714017] FS:  00007f2495d4c740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15751.715002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15751.715993] CR2: 0000000000000000 CR3: 000000017cbc0000 CR4: 00000000001407e0
[15751.716989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15751.717995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15751.718991] Stack:
[15751.720012]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[15751.721025]  ffff880222cb3d98 ffffffff810ab7d5 ffff880222cb3e18 ffffffff810c4e0c
[15751.722043]  ffff880222cb3db8 0000000000000246 0000000000000000 0000000000000000
[15751.723057] Call Trace:
[15751.724058]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15751.725094]  [<ffffffff810c4e0c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15751.726114]  [<ffffffff810c5cdf>] lock_acquire+0x9f/0x120
[15751.727134]  [<ffffffff81086c95>] ? kill_pid_info+0x5/0xb0
[15751.728178]  [<ffffffff81086cd8>] kill_pid_info+0x48/0xb0
[15751.729197]  [<ffffffff81086c95>] ? kill_pid_info+0x5/0xb0
[15751.730225]  [<ffffffff81086e2c>] SYSC_kill+0xcc/0x240
[15751.731216]  [<ffffffff81086de8>] ? SYSC_kill+0x88/0x240
[15751.732193]  [<ffffffff8107aeab>] ? SyS_wait4+0x8b/0x110
[15751.733174]  [<ffffffff81088cce>] SyS_kill+0xe/0x10
[15751.734133]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15751.735076] Code: 00 00 45 31 ed e9 f3 01 00 00 0f 1f 80 00 00 00 00 44 89 e8 4d 8b 64 c6 08 4d 85 e4 0f 84 21 ff ff ff f0 41 ff 84 24 98 01 00 00 <8b> 3d f0 e4 aa 01 44 8b ab 68 07 00 00 85 ff 75 0a 41 83 fd 2f 
[15751.737141] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 110.288 msecs
[15779.571634] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0]
[15779.572815] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15779.579104] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #107
[15779.581677] task: ffff8802434896d0 ti: ffff880243714000 task.ti: ffff880243714000
[15779.582931] RIP: 0010:[<ffffffff8165d3a9>]  [<ffffffff8165d3a9>] cpuidle_enter_state+0x79/0x190
[15779.584212] RSP: 0000:ffff880243717e48  EFLAGS: 00000246
[15779.585481] RAX: 00000e5c114d0f5f RBX: ffffffff82bb8048 RCX: 0000000000000019
[15779.586751] RDX: 20c49ba5e353f7cf RSI: 000000000002c480 RDI: 00410b3fa729c634
[15779.588025] RBP: ffff880243717e88 R08: 000000008baf9021 R09: 0000000000000000
[15779.589299] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810c32ff
[15779.590568] R13: ffff880243717dc8 R14: ffffffff810ab7d5 R15: ffff880243717da8
[15779.591843] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15779.593080] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15779.594318] CR2: 0000000000d80c80 CR3: 0000000001c11000 CR4: 00000000001407e0
[15779.595572] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15779.596822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15779.598066] Stack:
[15779.599304]  00000e5c1132f748 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15779.600571]  ffffe8ffff002118 ffff880243714000 ffffffff81cb19c0 ffff880243714000
[15779.601840]  ffff880243717e98 ffffffff8165d577 ffff880243717f08 ffffffff810bd345
[15779.603062] Call Trace:
[15779.604277]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15779.605482]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15779.606660]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15779.607831] Code: c8 48 89 df ff 50 48 41 89 c5 e8 c3 f7 a8 ff 44 8b 63 04 49 89 c7 0f 1f 44 00 00 e8 02 c4 ae ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 2b 7d c0 4c 89 f8 49 c1 ff 3f 48 f7 ea b8 ff ff ff 7f 48 c1 
[15779.610336] sending NMI to other CPUs:
[15779.611504] NMI backtrace for cpu 0
[15779.612633] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15779.614924] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15779.616088] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15779.617265] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15779.618434] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[15779.619614] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15779.620782] RBP: ffffffff81c03e68 R08: 000000008baf9021 R09: 0000000000000000
[15779.621945] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15779.623100] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff81c00000
[15779.624258] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15779.625426] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15779.626584] CR2: 00007f249422c269 CR3: 0000000001c11000 CR4: 00000000001407f0
[15779.627747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15779.628901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15779.630043] Stack:
[15779.631174]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000001
[15779.632331]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15779.633491]  00000e5c139d54ef ffffffff81cb1a30 ffffffff81cb19c0 ffffffff81d213f0
[15779.634649] Call Trace:
[15779.635799]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15779.636965]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15779.638128]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15779.639291]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15779.640454]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15779.641617]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15779.642783]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15779.643943]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15779.645078]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15779.646196]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15779.647291] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15779.649635] NMI backtrace for cpu 3
[15779.650755] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15779.652899] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15779.653930] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15779.654957] RSP: 0018:ffff88024372be08  EFLAGS: 00000046
[15779.655960] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[15779.656955] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15779.657940] RBP: ffff88024372be38 R08: 000000008baf9021 R09: 0000000000000000
[15779.658918] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[15779.659893] R13: 0000000000000010 R14: 0000000000000002 R15: ffff880243728000
[15779.660862] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15779.661833] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15779.662808] CR2: 00007f24941d6269 CR3: 0000000001c11000 CR4: 00000000001407e0
[15779.663797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15779.664768] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15779.665737] Stack:
[15779.666688]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000003
[15779.667667]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15779.668656]  00000e5c1316d5c0 ffffffff81cb1ae0 ffffffff81cb19c0 ffffffff81d213f0
[15779.669642] Call Trace:
[15779.670611]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15779.671604]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15779.672584]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15779.673558]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15779.674526] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15779.676645] NMI backtrace for cpu 2
[15779.677640] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #107
[15779.679625] task: ffff88024348c470 ti: ffff88024371c000 task.ti: ffff88024371c000
[15779.680640] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15779.681674] RSP: 0018:ffff88024371fe08  EFLAGS: 00000046
[15779.682678] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[15779.683669] RDX: 0000000000000000 RSI: ffff88024371ffd8 RDI: 0000000000000002
[15779.684654] RBP: ffff88024371fe38 R08: 000000008baf9021 R09: 0000000000000000
[15779.685629] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[15779.686605] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024371c000
[15779.687581] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15779.688566] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15779.689551] CR2: 00000000006ee1e0 CR3: 0000000001c11000 CR4: 00000000001407e0
[15779.690546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15779.691544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15779.692532] Stack:
[15779.693514]  000000024371fe38 87ce37a9ab100d43 ffffe8ffff202118 0000000000000005
[15779.694530]  ffffffff81cb19c0 0000000000000002 ffff88024371fe88 ffffffff8165d385
[15779.695552]  00000e5c13171780 ffffffff81cb1b90 ffffffff81cb19c0 ffffffff81d213f0
[15779.696576] Call Trace:
[15779.697590]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15779.698615]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15779.699626]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15779.700624]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15779.701619] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15807.555361] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c35:18287]
[15807.556447] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15807.562334] CPU: 1 PID: 18287 Comm: trinity-c35 Tainted: G             L 3.18.0+ #107
[15807.564718] task: ffff8802252edb40 ti: ffff8802253f0000 task.ti: ffff8802253f0000
[15807.565907] RIP: 0010:[<ffffffff811a30c9>]  [<ffffffff811a30c9>] unmap_single_vma+0x7d9/0x900
[15807.567106] RSP: 0018:ffff8802253f38e8  EFLAGS: 00000286
[15807.568294] RAX: 0000000000000001 RBX: 0000000000000246 RCX: 00000000007d6dc0
[15807.569470] RDX: ffff8800972c1010 RSI: ffff8802253f3ac0 RDI: ffff880223c22000
[15807.570629] RBP: ffff8802253f39d8 R08: 0000000000000000 R09: 0000000000000001
[15807.571790] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8802252edb40
[15807.572940] R13: 0000000000000003 R14: 0000000000000246 R15: ffff8802253f38a8
[15807.574084] FS:  00007f1edb7cb740(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15807.575239] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15807.576412] CR2: 0000000000639058 CR3: 00000001dbb98000 CR4: 00000000001407e0
[15807.577594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15807.578759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15807.579916] Stack:
[15807.581064]  00007f1ed9440fff 00007f1ed9440fff ffff88020cfc83d8 ffff8801b64d47f0
[15807.582230]  00007f1ed9441000 00007f1ed9441000 00007f1ed9440fff 00007f1ed9441000
[15807.583396]  ffff8802253f3968 ffff8802253f39a0 ffff8802252edb40 00003ffffffff000
[15807.584565] Call Trace:
[15807.585732]  [<ffffffff811a32ec>] zap_page_range_single+0xfc/0x160
[15807.586897]  [<ffffffff811a3415>] ? unmap_mapping_range+0x75/0x190
[15807.588054]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15807.589203]  [<ffffffff811a3517>] unmap_mapping_range+0x177/0x190
[15807.590353]  [<ffffffff8118607d>] truncate_inode_page+0x2d/0x90
[15807.591490]  [<ffffffff811913f3>] shmem_undo_range+0x423/0x710
[15807.592633]  [<ffffffff811916f8>] shmem_truncate_range+0x18/0x40
[15807.593779]  [<ffffffff81191c47>] shmem_fallocate+0x237/0x520
[15807.594916]  [<ffffffff810bc857>] ? prepare_to_wait+0x27/0x90
[15807.596039]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15807.597135]  [<ffffffff811b79d8>] SyS_madvise+0x398/0x870
[15807.598225]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15807.599315] Code: f7 45 98 00 08 00 00 48 0f 45 c2 49 89 04 24 e9 8b fc ff ff 66 0f 1f 44 00 00 4d 85 f6 0f 84 4e fc ff ff 48 8b 75 a8 48 8b 56 08 <48> 85 d2 74 0a 49 3b 56 08 0f 85 e6 fb ff ff 48 8b 75 a8 48 83 
[15807.601631] sending NMI to other CPUs:
[15807.602717] NMI backtrace for cpu 0
[15807.603771] CPU: 0 PID: 16807 Comm: trinity-c87 Tainted: G             L 3.18.0+ #107
[15807.605890] task: ffff88005e3e16d0 ti: ffff880225c54000 task.ti: ffff880225c54000
[15807.606966] RIP: 0010:[<ffffffff811496a0>]  [<ffffffff811496a0>] trace_hardirqs_off+0x0/0x100
[15807.608063] RSP: 0018:ffff880244e03cc0  EFLAGS: 00000092
[15807.609150] RAX: 0000000000000000 RBX: 0000000000000092 RCX: 0000000000007560
[15807.610243] RDX: ffff880244e3f960 RSI: 0000000000000000 RDI: ffff88023fa7da98
[15807.611329] RBP: ffff880244e03cd8 R08: 0000000000000000 R09: 0000000000000001
[15807.612412] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023fa7da98
[15807.613498] R13: 0000000000000092 R14: ffff88023fa7da98 R15: ffff880240298000
[15807.614589] FS:  00007f1edb7cb740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15807.615682] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15807.616767] CR2: 00007f1ed9c41000 CR3: 0000000225685000 CR4: 00000000001407f0
[15807.617855] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15807.618938] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15807.620007] Stack:
[15807.621068]  ffffffff817cce0f ffff88023fa7d668 ffff88023fa7da98 ffff880244e03d38
[15807.622157]  ffffffff815fbecd ffff88023fa7da98 ffff88023fa7dcb8 000000203fa7da98
[15807.623246]  ffff88023fa7dbb8 ffff88023fa7da98 ffff88023fa7d668 ffff880240d06aa8
[15807.624334] Call Trace:
[15807.625388]  <IRQ> 

[15807.626421]  [<ffffffff817cce0f>] ? _raw_spin_unlock_irqrestore+0x4f/0x60
[15807.627453]  [<ffffffff815fbecd>] usb_serial_generic_write_start+0x11d/0x200
[15807.628469]  [<ffffffff815fc193>] usb_serial_generic_write_bulk_callback+0x93/0x1b0
[15807.629462]  [<ffffffff815b0542>] __usb_hcd_giveback_urb+0x72/0x120
[15807.630430]  [<ffffffff815b0733>] usb_hcd_giveback_urb+0x43/0x120
[15807.631379]  [<ffffffff815f3203>] xhci_irq+0xc53/0x1c70
[15807.632303]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15807.633212]  [<ffffffff810c4e0c>] ? __lock_acquire.isra.31+0x22c/0x9f0
[15807.634107]  [<ffffffff815f4231>] xhci_msi_irq+0x11/0x20
[15807.634981]  [<ffffffff810cf8f5>] handle_irq_event_percpu+0x55/0x1f0
[15807.635848]  [<ffffffff810cfad1>] handle_irq_event+0x41/0x70
[15807.636708]  [<ffffffff810d2b5e>] ? handle_edge_irq+0x1e/0x140
[15807.637557]  [<ffffffff810d2bbf>] handle_edge_irq+0x7f/0x140
[15807.638404]  [<ffffffff810053a1>] handle_irq+0xb1/0x140
[15807.639247]  [<ffffffff817d0413>] do_IRQ+0x53/0x100
[15807.640085]  [<ffffffff817ce56f>] common_interrupt+0x6f/0x6f
[15807.640927]  <EOI> 

[15807.641761]  [<ffffffff81079bc4>] ? wait_consider_task+0x84/0xcc0
[15807.642588]  [<ffffffff8107a8d9>] ? do_wait+0xd9/0x280
[15807.643412]  [<ffffffff8107a920>] do_wait+0x120/0x280
[15807.644234]  [<ffffffff8107aea0>] SyS_wait4+0x80/0x110
[15807.645064]  [<ffffffff81078980>] ? task_stopped_code+0x60/0x60
[15807.645885]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15807.646697] Code: e8 66 f6 ff ff 4c 8b 4d c8 4a 8b 04 cd a0 f0 d1 81 41 c7 04 06 01 00 00 00 f0 41 ff 0f e9 46 ff ff ff 66 0f 1f 84 00 00 00 00 00 <f6> 05 51 a0 bd 00 02 75 07 c3 66 0f 1f 44 00 00 9c 58 f6 c4 02 
[15807.648500] NMI backtrace for cpu 2
[15807.649383] CPU: 2 PID: 15128 Comm: trinity-c180 Tainted: G             L 3.18.0+ #107
[15807.651182] task: ffff880225dd2da0 ti: ffff88009aed8000 task.ti: ffff88009aed8000
[15807.652102] RIP: 0010:[<ffffffff810c610b>]  [<ffffffff810c610b>] lock_release+0x7b/0x240
[15807.653030] RSP: 0018:ffff88009aedbe10  EFLAGS: 00000046
[15807.653960] RAX: 0000000000000000 RBX: ffff880225dd2da0 RCX: ffff8802266f74d8
[15807.654896] RDX: ffffffff8107a9f1 RSI: 0000000000000000 RDI: 0000000000000001
[15807.655839] RBP: ffff88009aedbe48 R08: 0000000000000000 R09: 0000000000000001
[15807.656782] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81c0a098
[15807.657724] R13: ffffffff8107a9f1 R14: 0000000000000296 R15: 0000000000000001
[15807.658669] FS:  00007f1edb7cb740(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15807.659626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15807.660583] CR2: 0000000000000001 CR3: 00000000613c7000 CR4: 00000000001407e0
[15807.661526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15807.662446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15807.663355] Stack:
[15807.664262]  0000000000000046 ffffffff81c0a098 ffffffff81c0a080 ffff880225dd3108
[15807.665197]  ffff880225dd2da0 ffff880225dd2da0 ffff880225dd2d90 ffff88009aedbe68
[15807.666125]  ffffffff817cd103 ffff880225dd2da0 ffff88009aedbef8 ffff88009aedbee8
[15807.667012] Call Trace:
[15807.667883]  [<ffffffff817cd103>] _raw_read_unlock+0x23/0x40
[15807.668761]  [<ffffffff8107a9f1>] do_wait+0x1f1/0x280
[15807.669630]  [<ffffffff8107aea0>] SyS_wait4+0x80/0x110
[15807.670494]  [<ffffffff81078980>] ? task_stopped_code+0x60/0x60
[15807.671365]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15807.672238] Code: 05 0b cc c3 00 8b 3d 75 6d be 00 65 ff 0c 25 e0 a9 00 00 65 48 8b 1c 25 00 aa 00 00 85 ff 74 3a 8b 35 12 d1 aa 01 85 f6 75 0b 9c <58> f6 c4 02 0f 85 6e 01 00 00 8b 93 68 07 00 00 85 d2 0f 8e 45 
[15807.674141] NMI backtrace for cpu 3
[15807.675053] CPU: 3 PID: 18981 Comm: trinity-c225 Tainted: G             L 3.18.0+ #107
[15807.676970] task: ffff88011c4e0000 ti: ffff880225e94000 task.ti: ffff880225e94000
[15807.677939] RIP: 0010:[<ffffffff810c4dfa>]  [<ffffffff810c4dfa>] __lock_acquire.isra.31+0x21a/0x9f0
[15807.678917] RSP: 0018:ffff880225e97a48  EFLAGS: 00000002
[15807.679875] RAX: 0000000000000008 RBX: ffff88011c4e0000 RCX: 0000000000000000
[15807.680839] RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
[15807.681802] RBP: ffff880225e97ab8 R08: 0000000000000001 R09: 0000000000000000
[15807.682757] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000135
[15807.683717] R13: 0000000000000002 R14: ffff88024e5d4398 R15: ffff88011c4e07e0
[15807.684675] FS:  00007f1edb7cb740(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15807.685642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15807.686606] CR2: 00007f1edb72f07c CR3: 0000000229ac5000 CR4: 00000000001407e0
[15807.687579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15807.688548] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15807.689514] Stack:
[15807.690473]  ffff880225e97ac8 ffffea00087f7440 000000021fdd0000 0000000000000000
[15807.691463]  0000000000000000 000ffe0000000000 ffff880225e97ad8 0000000000000000
[15807.692449]  ffffea0006f59d80 0000000000000046 0000000000000000 0000000000000000
[15807.693433] Call Trace:
[15807.694415]  [<ffffffff810c5cdf>] lock_acquire+0x9f/0x120
[15807.695418]  [<ffffffff81184331>] ? release_pages+0x101/0x270
[15807.696412]  [<ffffffff817ccbe9>] _raw_spin_lock_irqsave+0x49/0x90
[15807.697408]  [<ffffffff81184331>] ? release_pages+0x101/0x270
[15807.698408]  [<ffffffff810ab7d5>] ? local_clock+0x25/0x30
[15807.699404]  [<ffffffff81184331>] release_pages+0x101/0x270
[15807.700406]  [<ffffffff81185423>] __pagevec_release+0x43/0x60
[15807.701396]  [<ffffffff81191430>] shmem_undo_range+0x460/0x710
[15807.702391]  [<ffffffff81191d96>] shmem_fallocate+0x386/0x520
[15807.703386]  [<ffffffff811ea115>] ? __sb_start_write+0x105/0x1c0
[15807.704382]  [<ffffffff811e549c>] ? do_fallocate+0x11c/0x1d0
[15807.705389]  [<ffffffff811e549c>] ? do_fallocate+0x11c/0x1d0
[15807.706377]  [<ffffffff811e54b2>] do_fallocate+0x132/0x1d0
[15807.707346]  [<ffffffff811e5598>] SyS_fallocate+0x48/0x80
[15807.708287]  [<ffffffff817cda52>] system_call_fastpath+0x12/0x17
[15807.709228] Code: e0 7f 44 09 d0 41 88 47 31 41 0f b6 47 32 83 e0 f0 45 85 c0 0f 95 c2 09 c8 c1 e2 03 09 d0 41 88 47 32 0f b7 55 18 41 0f b7 47 32 <c1> e2 04 83 e0 0f 09 d0 66 41 89 47 32 e8 a4 69 fe ff 4c 8b 4d 
[15835.539166] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0]
[15835.540319] Modules linked in: 8021q garp bridge stp snd_seq_dummy hidp fuse tun rfcomm bnep af_key llc2 can_raw nfnetlink sctp libcrc32c can_bcm scsi_transport_iscsi nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic microcode serio_raw pcspkr usb_debug snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep shpchp snd_seq e1000e snd_seq_device ptp pps_core snd_pcm snd_timer snd soundcore nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[15835.546410] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #107
[15835.548896] task: ffff8802434896d0 ti: ffff880243714000 task.ti: ffff880243714000
[15835.550154] RIP: 0010:[<ffffffff8165d3a9>]  [<ffffffff8165d3a9>] cpuidle_enter_state+0x79/0x190
[15835.551425] RSP: 0018:ffff880243717e48  EFLAGS: 00000246
[15835.552686] RAX: 00000e691b28f962 RBX: ffffffff82bb8048 RCX: 0000000000000019
[15835.553975] RDX: 20c49ba5e353f7cf RSI: 00000000000276af RDI: 0041352987dbd37c
[15835.555275] RBP: ffff880243717e88 R08: 000000008baf8f7d R09: 0000000000000000
[15835.556557] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff810c32ff
[15835.557841] R13: ffff880243717dc8 R14: ffffffff810ab7d5 R15: ffff880243717da8
[15835.559116] FS:  0000000000000000(0000) GS:ffff880245000000(0000) knlGS:0000000000000000
[15835.560378] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.561620] CR2: 00007f8b7d063000 CR3: 0000000001c11000 CR4: 00000000001407e0
[15835.562868] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.564124] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.565374] Stack:
[15835.566619]  00000e691a90eefa ffffffff81cb1ae0 ffffffff81cb19c0 ffffffff81d213f0
[15835.567895]  ffffe8ffff002118 ffff880243714000 ffffffff81cb19c0 ffff880243714000
[15835.569155]  ffff880243717e98 ffffffff8165d577 ffff880243717f08 ffffffff810bd345
[15835.570338] Call Trace:
[15835.571508]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15835.572690]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15835.573850]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15835.574978] Code: c8 48 89 df ff 50 48 41 89 c5 e8 c3 f7 a8 ff 44 8b 63 04 49 89 c7 0f 1f 44 00 00 e8 02 c4 ae ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 2b 7d c0 4c 89 f8 49 c1 ff 3f 48 f7 ea b8 ff ff ff 7f 48 c1 
[15835.577402] sending NMI to other CPUs:
[15835.578547] NMI backtrace for cpu 0
[15835.579666] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #107
[15835.581963] task: ffffffff81c164c0 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[15835.583133] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15835.584310] RSP: 0018:ffffffff81c03e38  EFLAGS: 00000046
[15835.585475] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[15835.586643] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[15835.587805] RBP: ffffffff81c03e68 R08: 000000008baf8f7d R09: 0000000000000000
[15835.588972] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[15835.590132] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff81c00000
[15835.591285] FS:  0000000000000000(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
[15835.592443] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.593595] CR2: 00000000f0000137 CR3: 0000000001c11000 CR4: 00000000001407f0
[15835.594753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.595903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.597050] Stack:
[15835.598189]  0000000081c03e68 76e04e88cc48b66f ffffe8fffee02118 0000000000000001
[15835.599357]  ffffffff81cb19c0 0000000000000000 ffffffff81c03eb8 ffffffff8165d385
[15835.600530]  00000e691d725867 ffffffff81cb1a30 ffffffff81cb19c0 ffffffff81d213f0
[15835.601707] Call Trace:
[15835.602870]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15835.604053]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15835.605230]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15835.606407]  [<ffffffff817b9639>] rest_init+0xc9/0xd0
[15835.607581]  [<ffffffff817b9575>] ? rest_init+0x5/0xd0
[15835.608761]  [<ffffffff81f21de6>] ? ftrace_init+0xa8/0x13b
[15835.609942]  [<ffffffff81f03041>] start_kernel+0x492/0x4b3
[15835.611123]  [<ffffffff81f0299f>] ? set_init_arg+0x55/0x55
[15835.612296]  [<ffffffff81f02581>] x86_64_start_reservations+0x2a/0x2c
[15835.613451]  [<ffffffff81f02675>] x86_64_start_kernel+0xf2/0xf6
[15835.614573] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15835.616991] NMI backtrace for cpu 3
[15835.618146] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #107
[15835.620322] task: ffff88024348ada0 ti: ffff880243728000 task.ti: ffff880243728000
[15835.621352] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15835.622392] RSP: 0000:ffff88024372be08  EFLAGS: 00000046
[15835.623397] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[15835.624391] RDX: 0000000000000000 RSI: ffff88024372bfd8 RDI: 0000000000000003
[15835.625376] RBP: ffff88024372be38 R08: 000000008baf8f7d R09: 0000000000000000
[15835.626352] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[15835.627324] R13: 0000000000000001 R14: 0000000000000001 R15: ffff880243728000
[15835.628290] FS:  0000000000000000(0000) GS:ffff880245400000(0000) knlGS:0000000000000000
[15835.629259] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.630229] CR2: 0000000001186ff8 CR3: 0000000094ed2000 CR4: 00000000001407e0
[15835.631206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.632173] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.633133] Stack:
[15835.634080]  000000034372be38 492161474c69335b ffffe8ffff402118 0000000000000002
[15835.635053]  ffffffff81cb19c0 0000000000000003 ffff88024372be88 ffffffff8165d385
[15835.636035]  00000e691d2afb93 ffffffff81cb1a88 ffffffff81cb19c0 ffffffff81d213f0
[15835.637018] Call Trace:
[15835.637986]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15835.638969]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15835.639942]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15835.640908]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15835.641872] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[15835.643982] NMI backtrace for cpu 2
[15835.644955] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #107
[15835.646877] task: ffff88024348c470 ti: ffff88024371c000 task.ti: ffff88024371c000
[15835.647860] RIP: 0010:[<ffffffff813d4cab>]  [<ffffffff813d4cab>] intel_idle+0xdb/0x180
[15835.648868] RSP: 0018:ffff88024371fe08  EFLAGS: 00000046
[15835.649868] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[15835.650828] RDX: 0000000000000000 RSI: ffff88024371ffd8 RDI: ffffffff81c11000
[15835.651782] RBP: ffff88024371fe38 R08: 000000008baf8f7d R09: 0000000000000000
[15835.652734] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[15835.653682] R13: 0000000000000010 R14: 0000000000000002 R15: ffff88024371c000
[15835.654633] FS:  0000000000000000(0000) GS:ffff880245200000(0000) knlGS:0000000000000000
[15835.655588] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15835.656543] CR2: 000000336f1b7740 CR3: 0000000001c11000 CR4: 00000000001407e0
[15835.657514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15835.658480] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[15835.659460] Stack:
[15835.660403]  000000024371fe38 87ce37a9ab100d43 ffffe8ffff202118 0000000000000003
[15835.661376]  ffffffff81cb19c0 0000000000000002 ffff88024371fe88 ffffffff8165d385
[15835.662354]  00000e691d059748 ffffffff81cb1ae0 ffffffff81cb19c0 ffffffff81d213f0
[15835.663333] Call Trace:
[15835.664296]  [<ffffffff8165d385>] cpuidle_enter_state+0x55/0x190
[15835.665268]  [<ffffffff8165d577>] cpuidle_enter+0x17/0x20
[15835.666239]  [<ffffffff810bd345>] cpu_startup_entry+0x355/0x410
[15835.667202]  [<ffffffff8103015a>] start_secondary+0x1aa/0x230
[15835.668163] Code: 31 d2 65 48 8b 34 25 08 aa 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 aa 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 

Hmm. Not only does it apparently stay up for you now, the traces seem to be improving in quality.

There's a decided pattern of "copy_page_range()" and "zap_page_range()" here.

Now, what's *also* intriguing is how many "_raw_spin_lock_nested" things there are in there. Which makes me suspect that you are actually hitting some really nasty spinlock contention, and that your 22s lockups could be due to lock hold times going exponential.

So I don't think that it's the copy_page_range() itself that is necessarily all that horribly expensive (although it's certainly not a lightweight function), but the fact that you get contention on some lock inside that loop, and when you have every single CPU hammering on it things just go to hell in a handbasket.

And when spinlocks start getting contention, *nested* spinlocks really really hurt. And you've got all the spinlock debugging on etc, don't you? Which just makes the locks really expensive, and much much easier to start becoming contended (and there's *another* level of nesting, because I think the lockdep stuff has its own locking inside). So you have three levels of spinlock nesting, and the outer one will be completely hammered.

So I think the locks you have are from copy_pte_range:

        dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl);
        if (!dst_pte)
                return -ENOMEM;
        src_pte = pte_offset_map(src_pmd, addr);
        src_ptl = pte_lockptr(src_mm, src_pmd);
        spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);

and we do have some mitigation in place for horrible horrible contention (try to release every few entries, but with all CPU's hammering on these locks, and things being slow due to all the debugging, I think we may finally be hitting the right place here.

Also, you do have this:

sched: RT throttling activated

so there's something going on with RT scheduling too. I'd consider all the softlockups after that point suspect - the softlockup thread has presumably used so much CPU spewing out the debug messages that things aren't really working any more RT-wise.

Lookie here (the "soft lockup" grep is to skip all the cross-CPU traces from other CPU's that weren't necessarily the problem case):

[torvalds@i7 linux]$ grep -5 "soft lockup" ~/0.txt | grep RIP

   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   generic_exec_single+0xee/0x1b0
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   lock_acquire+0xb4/0x120
   RIP: 0010:   shmem_write_end+0x65/0xf0
   RIP: 0010:   _raw_spin_unlock_irqrestore+0x38/0x60
   RIP: 0010:   copy_user_enhanced_fast_string+0x5/0x10
   RIP: 0010:   copy_user_enhanced_fast_string+0x5/0x10
   RIP: 0010:   __slab_alloc+0x52f/0x58f
   RIP: 0010:   map_id_up+0x9/0x80
   RIP: 0010:   cpuidle_enter_state+0x79/0x190
   RIP: 0010:   unmap_single_vma+0x7d9/0x900
   RIP: 0010:   cpuidle_enter_state+0x79/0x190

and notice the pattern? All those early RIP cases are the page table locks for copy_page_range and that one TLB flush for zap_page_range..

So your printouts are finally starting to make sense. But I'm also starting to suspect strongly that the problem is that with all your lock debugging and other overheads (does this still have DEBUG_PAGEALLOC?) you really are getting into a "real" softlockup because things are scaling so horribly badly.

If you now disable spinlock debugging and lockdep, hopefully that page table lock now doesn't always get hung up on the lockdep locking, so it starts scaling much better, and maybe you'd not see this...

Linus

Yeah, though remember this seems to have for some reason gotten worse in more recent builds. I've been running kitchen-sink debug kernels for my trinity runs for the last three years, and it's only this last few months that this has got to be enough of a problem that I'm not seeing the more interesting bugs. (Or perhaps we're just getting better at fixing them in -next now, so my runs are lasting longer..)

I see that fairly often. I've never dug into exactly what causes it, but it seems to be triggerable just by some long running CPU hogs.

I can give it a shot. Hopefully there's some further mitigation that could be done to allow a workload like this to survive under a debug build though, as we've caught *so many* bugs with this stuff in the past.

Dave

Turns out also that this build didn't have PROVE_LOCKING enabled. CONFIG_LOCKDEP was, but that just bloats the structures a little, and afaik doesn't incur the same runtime overhead.

I also forgot to answer the question above, PAGEALLOC is also off.

So the only thing that was on that could cause spinlock overhead was DEBUG_SPINLOCK (and LOCK_STAT, though iirc that's not huge either)

Dave

So DEBUG_SPINLOCK does have one big downside if I recall correctly - the debugging spinlocks are very much not fair. So they don't work like the real ticket spinlocks. That might have serious effects on the contention case, with some thread not making any progress due to just the implementation of the debug spinlocks.

Peter, Ingo, maybe I'm full of crap on the debug spinlock thing., but a quick look tells me thay are all built on top of the "trylock" primitive that does indeed not queue anything, and is thus not fair.

I'm not sure why the debug spinlocks couldn't just be ticket locks instead. But there you are - once more, the debug infrastructure is actually much weaker and inferior to the "real" code.

Linus

Yeah, the DEBUG_SPINLOCK stuff is horrible. The trylock loops were designed to 'detect' actual lockups, but this was all done before lockdep.

I think one can make an argument to remove the trylock loops and fully rely on lockdep to detect such issues. Only keeping the integrity checks; similar to the mutex debugging stuff.

There's a related issue with the trylock loops in that it relies on delay(1) and DVFS heavy (or virt) platforms often have 'dubious' quality delay loops.

Wish DEBUG_SPINLOCK disabled, I see the same behaviour. Lots of traces spewed, but it seems to run and run (at least so far).

Dave

[24053.092097] trinity-c206 (26510) used greatest stack depth: 8888 bytes left
[24998.017355] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c205:636]
[24998.017457] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[24998.019523] CPU: 1 PID: 636 Comm: trinity-c205 Not tainted 3.18.0+ #108
[24998.021412] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[24998.022367] RIP: 0010:[<ffffffff810ee9ca>]  [<ffffffff810ee9ca>] generic_exec_single+0xea/0x1b0
[24998.023332] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[24998.024287] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[24998.025245] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[24998.026194] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[24998.027143] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[24998.028144] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[24998.029095] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[24998.030050] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24998.031005] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[24998.031967] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[24998.032930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[24998.033898] Stack:
[24998.034855]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[24998.035843]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[24998.036832]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[24998.037855] Call Trace:
[24998.038837]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[24998.039826]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[24998.040804]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[24998.041779]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[24998.042748]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[24998.043705]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[24998.044647]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[24998.045585]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[24998.046528]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[24998.047510]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[24998.048456]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[24998.049409]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[24998.050361]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[24998.051309]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[24998.052260]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[24998.053209]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[24998.054156]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[24998.055100]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[24998.056044]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[24998.056992]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[24998.057963] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[24998.059993] sending NMI to other CPUs:
[24998.060963] NMI backtrace for cpu 0
[24998.061989] CPU: 0 PID: 2940 Comm: trinity-c150 Not tainted 3.18.0+ #108
[24998.064073] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[24998.065137] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[24998.066226] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[24998.067310] RAX: 0000000061fece8a RBX: 0000000000510792 RCX: 0000000000000000
[24998.068407] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[24998.069491] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[24998.070569] R10: 0000000000000526 R11: 000000000000000f R12: 000016bf99600917
[24998.071642] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[24998.072713] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[24998.073771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24998.074813] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[24998.075855] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[24998.076891] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[24998.077903] Stack:
[24998.078886]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[24998.079872]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[24998.080832]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[24998.081769] Call Trace:
[24998.082680]  <IRQ> 

[24998.083577]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[24998.084450]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[24998.085315]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[24998.086173]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[24998.087025]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[24998.087877]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[24998.088732]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[24998.089583]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[24998.090435]  <EOI> 

[24998.091279]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[24998.092118]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[24998.092951]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[24998.093779]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[24998.094605]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[24998.095420] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[24998.097220] NMI backtrace for cpu 3
[24998.098074] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.18.0+ #108
[24998.099812] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[24998.100708] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[24998.101617] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[24998.102529] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[24998.103448] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[24998.104364] RBP: ffff88024471be38 R08: 0000000000000010 R09: 000000000000265a
[24998.105283] R10: 0000000000007d5b R11: 00000000000003ff R12: 0000000000000005
[24998.106201] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[24998.107115] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[24998.108039] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24998.108962] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[24998.109894] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[24998.110820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[24998.111747] Stack:
[24998.112668]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[24998.113610]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[24998.114553]  000016bfb0c185fc ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[24998.115501] Call Trace:
[24998.116440]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[24998.117389]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[24998.118309]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[24998.119208]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[24998.120106] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[24998.122074] NMI backtrace for cpu 2
[24998.122981] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.18.0+ #108
[24998.124771] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[24998.125679] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[24998.126593] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[24998.127528] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[24998.128444] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[24998.129357] RBP: ffff88024470fe38 R08: ffff88024e28ecf0 R09: 00000000ffffffff
[24998.130267] R10: 000000000000265c R11: 00000000000003ff R12: 0000000000000005
[24998.131172] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[24998.132071] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[24998.132974] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24998.133876] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[24998.134789] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[24998.135694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[24998.136589] Stack:
[24998.137505]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[24998.138424]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[24998.139340]  000016bfb0c18159 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[24998.140262] Call Trace:
[24998.141176]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[24998.142103]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[24998.143026]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[24998.143948]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[24998.144868] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25026.001132] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c205:636]
[25026.002121] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25026.007365] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25026.009524] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25026.010591] RIP: 0010:[<ffffffff810ee9ce>]  [<ffffffff810ee9ce>] generic_exec_single+0xee/0x1b0
[25026.011710] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25026.012787] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25026.013854] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25026.014902] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25026.015941] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25026.016982] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25026.018019] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25026.019065] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25026.020113] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25026.021205] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25026.022263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25026.023310] Stack:
[25026.024349]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25026.025411]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25026.026472]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25026.027535] Call Trace:
[25026.028585]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25026.029640]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25026.030683]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25026.031747]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25026.032771]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25026.033800]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25026.034826]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25026.035854]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25026.036883]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25026.037896]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25026.038886]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25026.039875]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25026.040865]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25026.041867]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25026.042818]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25026.043763]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25026.044707]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25026.045647]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25026.046587]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25026.047529]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25026.048469] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[25026.050509] sending NMI to other CPUs:
[25026.051516] NMI backtrace for cpu 0
[25026.052541] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25026.054635] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25026.055696] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25026.056761] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[25026.057822] RAX: 0000000079e588fc RBX: 0000000000511d6e RCX: 0000000000000000
[25026.058886] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[25026.059947] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[25026.061009] R10: 0000000000000526 R11: 000000000000000f R12: 000016c61e4e2117
[25026.062073] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[25026.063139] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25026.064215] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25026.065271] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25026.066314] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25026.067349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25026.068359] Stack:
[25026.069339]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[25026.070326]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[25026.071289]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[25026.072225] Call Trace:
[25026.073133]  <IRQ> 

[25026.074028]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25026.074902]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[25026.075767]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25026.076622]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25026.077476]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25026.078328]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25026.079183]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25026.080035]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25026.080885]  <EOI> 

[25026.081728]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25026.082565]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25026.083397]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25026.084226]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25026.085050]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25026.085867] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25026.087667] NMI backtrace for cpu 3
[25026.088525] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25026.090268] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25026.091168] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25026.092081] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25026.092994] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25026.093915] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25026.094834] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff
[25026.095754] R10: 000000000000265a R11: 00000000000003ff R12: 0000000000000005
[25026.096677] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25026.097598] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25026.098526] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25026.099454] CR2: 00007fffcba6fea0 CR3: 0000000001c11000 CR4: 00000000001407e0
[25026.100392] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25026.101324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25026.102257] Stack:
[25026.103183]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25026.104133]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25026.105083]  000016c63648391d ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25026.106036] Call Trace:
[25026.106984]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25026.107936]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25026.108865]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25026.109771]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25026.110674] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25026.112654] NMI backtrace for cpu 2
[25026.113565] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25026.115375] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25026.116284] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25026.117207] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25026.118123] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25026.119041] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25026.119958] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25026.120875] R10: 0000000000002659 R11: 00000000000003ff R12: 0000000000000005
[25026.121814] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25026.122719] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25026.123625] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25026.124533] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25026.125450] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25026.126361] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25026.127262] Stack:
[25026.128157]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25026.129079]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25026.129996]  000016c636483734 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25026.130920] Call Trace:
[25026.131864]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25026.132794]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25026.133725]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25026.134652]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25026.135579] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25037.184648] INFO: rcu_sched detected stalls on CPUs/tasks:
[25037.185608] 	(detected by 0, t=6002 jiffies, g=777861, c=777860, q=0)
[25037.186536] INFO: Stall ended before state dump start
[25053.984908] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636]
[25053.985884] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25053.991028] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25053.993097] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25053.994131] RIP: 0010:[<ffffffff810ee9ca>]  [<ffffffff810ee9ca>] generic_exec_single+0xea/0x1b0
[25053.995202] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25053.996223] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25053.997247] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25053.998270] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25053.999294] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25054.000323] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25054.001349] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25054.002384] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25054.003420] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25054.004455] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25054.005522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25054.006552] Stack:
[25054.007575]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25054.008619]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25054.009658]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25054.010692] Call Trace:
[25054.011716]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25054.012743]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25054.013763]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25054.014779]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25054.015830]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25054.016846]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25054.017846]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25054.018823]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25054.019798]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25054.020774]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25054.021732]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25054.022671]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25054.023604]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25054.024534]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25054.025495]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25054.026421]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25054.027353]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25054.028281]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25054.029206]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25054.030136]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25054.031054] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[25054.033040] sending NMI to other CPUs:
[25054.033992] NMI backtrace for cpu 0
[25054.035027] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25054.037117] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25054.038172] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25054.039238] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[25054.040297] RAX: 0000000091ca7f65 RBX: 0000000000513346 RCX: 0000000000000000
[25054.041365] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[25054.042436] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[25054.043514] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016cca33c3917
[25054.044572] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[25054.045606] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25054.046645] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25054.047685] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25054.048728] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25054.049763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25054.050774] Stack:
[25054.051757]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[25054.052744]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[25054.053709]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[25054.054649] Call Trace:
[25054.055559]  <IRQ> 

[25054.056455]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25054.057326]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[25054.058194]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25054.059050]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25054.059904]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25054.060759]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25054.061614]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25054.062466]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25054.063317]  <EOI> 

[25054.064159]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25054.064994]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25054.065826]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25054.066656]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25054.067480]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25054.068297] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25054.070095] NMI backtrace for cpu 2
[25054.070949] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25054.072692] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25054.073589] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25054.074498] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25054.075411] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25054.076329] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25054.077248] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25054.078167] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005
[25054.079087] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25054.080007] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25054.080934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25054.081856] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25054.082790] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25054.083721] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25054.084651] Stack:
[25054.085578]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25054.086525]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25054.087477]  000016ccba9dadfd ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25054.088429] Call Trace:
[25054.089371]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25054.090324]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25054.091251]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25054.092158]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25054.093059] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25054.095041] NMI backtrace for cpu 3
[25054.095952] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25054.097763] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25054.098676] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25054.099598] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25054.100517] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25054.101441] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25054.102360] RBP: ffff88024471be38 R08: 0000000000000018 R09: 0000000000002655
[25054.103278] R10: 000000000000be78 R11: 00000000000003ff R12: 0000000000000005
[25054.104189] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25054.105118] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25054.106026] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25054.106933] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25054.107847] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25054.108756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25054.109656] Stack:
[25054.110546]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25054.111466]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25054.112383]  000016ccba9dc98b ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25054.113305] Call Trace:
[25054.114222]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25054.115173]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25054.116098]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25054.117022]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25054.117945] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25081.968684] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636]
[25081.969644] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25081.974748] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25081.976852] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25081.977892] RIP: 0010:[<ffffffff810ee9ca>]  [<ffffffff810ee9ca>] generic_exec_single+0xea/0x1b0
[25081.978981] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25081.980030] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25081.981068] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25081.982090] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25081.983107] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25081.984121] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25081.985132] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25081.986155] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25081.987179] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25081.988208] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25081.989271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25081.990294] Stack:
[25081.991306]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25081.992338]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25081.993369]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25081.994406] Call Trace:
[25081.995430]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25081.996456]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25081.997470]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25081.998481]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25081.999512]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25082.000509]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25082.001510]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25082.002510]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25082.003514]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25082.004498]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25082.005463]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25082.006425]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25082.007386]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25082.008335]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25082.009300]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25082.010222]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25082.011144]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25082.012057]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25082.012971]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25082.013889]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25082.014807] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[25082.016796] sending NMI to other CPUs:
[25082.017742] NMI backtrace for cpu 0
[25082.018768] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25082.020865] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25082.021929] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25082.022996] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[25082.024057] RAX: 00000000a9afbd0d RBX: 000000000051491e RCX: 0000000000000000
[25082.025125] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[25082.026185] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[25082.027248] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016d3282a5117
[25082.028317] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[25082.029383] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25082.030459] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25082.031518] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25082.032563] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25082.033598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25082.034612] Stack:
[25082.035595]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[25082.036582]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[25082.037546]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[25082.038485] Call Trace:
[25082.039397]  <IRQ> 

[25082.040290]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25082.041164]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[25082.042032]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25082.042890]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25082.043748]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25082.044603]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25082.045460]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25082.046317]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25082.047170]  <EOI> 

[25082.048018]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25082.048856]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25082.049691]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25082.050520]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25082.051347]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25082.052166] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25082.053966] NMI backtrace for cpu 3
[25082.054824] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25082.056568] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25082.057464] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25082.058376] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25082.059292] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25082.060214] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25082.061132] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff
[25082.062054] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005
[25082.062977] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25082.063899] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25082.064825] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25082.065751] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25082.066685] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25082.067618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25082.068552] Stack:
[25082.069477]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25082.070426]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25082.071378]  000016d33f8be858 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25082.072334] Call Trace:
[25082.073279]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25082.074233]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25082.075161]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25082.076066]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25082.076968] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25082.078949] NMI backtrace for cpu 2
[25082.079865] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25082.081672] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25082.082586] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25082.083507] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25082.084424] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25082.085345] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25082.086261] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25082.087178] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005
[25082.088088] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25082.089016] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25082.089919] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25082.090826] CR2: 00007f274c4061c0 CR3: 0000000001c11000 CR4: 00000000001407e0
[25082.091740] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25082.092652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25082.093556] Stack:
[25082.094451]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25082.095371]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25082.096293]  000016d33f8bcc84 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25082.097220] Call Trace:
[25082.098141]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25082.099095]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25082.100026]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25082.100956]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25082.101882] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25097.149884] INFO: rcu_sched self-detected stall on CPU
[25097.149885] INFO: rcu_sched self-detected stall on CPU

[25097.149886] 	1: (5973 ticks this GP) idle=6b3/140000000000001/0 softirq=1672732/1672732 
[25097.149897] 	
[25097.149898]  (t=6000 jiffies g=777862 c=777861 q=0)
[25097.149899] Task dump for CPU 0:
[25097.149899] trinity-c150    R
[25097.149900]   running task    
[25097.149908] 13312  2940  29562 0x10000008
[25097.149909]  ffff880197e0c000
[25097.149909]  000000002e7fd874
[25097.149909]  ffff880197e0fb88
[25097.149910]  ffffffff810f1c86

[25097.149910]  ffff8801bf3536b0
[25097.149910]  00007fffa4bfd708
[25097.149911]  ffff880197e0fb98
[25097.149911]  000000002e7fd874

[25097.149911]  ffff880197e0fbb8
[25097.149911]  ffffffff810f1c86
[25097.149912]  ffff880197e0fc08
[25097.149912]  00007fffa4bfd708

[25097.149912] Call Trace:

[25097.149915]  [<ffffffff810f1c86>] ? __module_text_address+0x16/0x80

[25097.149917]  [<ffffffff810f1c86>] ? __module_text_address+0x16/0x80

[25097.149919]  [<ffffffff810f63f6>] ? is_module_text_address+0x16/0x30

[25097.149921]  [<ffffffff810942f8>] ? __kernel_text_address+0x58/0x80

[25097.149923]  [<ffffffff81006a2f>] ? print_context_stack+0x8f/0x100

[25097.149925]  [<ffffffff8100556f>] ? dump_trace+0x16f/0x350

[25097.149926]  [<ffffffff811e2966>] ? link_path_walk+0x266/0x850

[25097.149927]  [<ffffffff811e5dff>] ? getname_flags+0x4f/0x1a0

[25097.149929]  [<ffffffff811e5d86>] ? final_putname+0x26/0x50

[25097.149930]  [<ffffffff810133af>] ? save_stack_trace+0x2f/0x50

[25097.149941]  [<ffffffff811bc6e0>] ? set_track+0x70/0x140

[25097.149943]  [<ffffffff8178e291>] ? free_debug_processing+0x1de/0x232

[25097.149944]  [<ffffffff8178e407>] ? __slab_free+0x122/0x2e9

[25097.149946]  [<ffffffff81352ef4>] ? timerqueue_del+0x24/0x70

[25097.149948]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0

[25097.149950]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0

[25097.149951]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0

[25097.149953]  [<ffffffff812c8d6d>] ? SyS_msgget+0x4d/0x70

[25097.149955]  [<ffffffff817994c4>] ? tracesys_phase2+0xd4/0xd9
[25097.149955] Task dump for CPU 1:
[25097.149956] trinity-c205    R
[25097.149956]   running task    
[25097.149958] 13312   636  29562 0x10000008
[25097.149959]  ffff8800962fed60
[25097.149959]  0000000077930768
[25097.149959]  ffff88024e243d78
[25097.149959]  ffffffff810a3932

[25097.149960]  0000000000000001
[25097.149960]  ffffffff81c4bf80
[25097.149960]  ffff88024e243d98
[25097.149960]  ffffffff810a76dd

[25097.149961]  ffff88024e24c440
[25097.149961]  0000000000000002
[25097.149961]  ffff88024e243dc8
[25097.149961]  ffffffff810ce920

[25097.149962] Call Trace:
[25097.149962]  <IRQ> 

[25097.149964]  [<ffffffff810a3932>] sched_show_task+0xd2/0x140

[25097.149965]  [<ffffffff810a76dd>] dump_cpu_task+0x3d/0x50

[25097.149969]  [<ffffffff810ce920>] rcu_dump_cpu_stacks+0x90/0xd0

[25097.149970]  [<ffffffff810d4c33>] rcu_check_callbacks+0x4c3/0x730

[25097.149973]  [<ffffffff81120b4c>] ? acct_account_cputime+0x1c/0x20

[25097.149974]  [<ffffffff810a800e>] ? account_system_time+0x8e/0x190

[25097.149976]  [<ffffffff810da34b>] update_process_times+0x4b/0x80

[25097.149977]  [<ffffffff810e9cff>] tick_sched_timer+0x4f/0x160

[25097.149978]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0

[25097.149979]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20

[25097.149980]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260

[25097.149982]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70

[25097.149983]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60

[25097.149985]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25097.149985]  <EOI> 

[25097.149987]  [<ffffffff81799e0d>] ? retint_restore_args+0xe/0xe

[25097.149989]  [<ffffffff810ee9ce>] ? generic_exec_single+0xee/0x1b0

[25097.149990]  [<ffffffff810eea09>] ? generic_exec_single+0x129/0x1b0

[25097.149991]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0

[25097.149992]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0

[25097.149993]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60

[25097.149994]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60

[25097.149995]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0

[25097.149998]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280

[25097.149999]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60

[25097.150000]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320

[25097.150001]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0

[25097.150003]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50

[25097.150004]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900

[25097.150006]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50

[25097.150007]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160

[25097.150009]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190

[25097.150011]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0

[25097.150012]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122

[25097.150013]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0

[25097.150015]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860

[25097.150016]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0

[25097.150018]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9

[25097.206563] 	0: (1 GPs behind) idle=845/140000000000001/0 softirq=1616166/1616167 
[25097.207170] 	 (t=6000 jiffies g=777862 c=777861 q=0)
[25097.207778] Task dump for CPU 0:
[25097.208382] trinity-c150    R  running task    13312  2940  29562 0x10000008
[25097.209002]  ffff8801bf3536b0 000000002e7fd874 ffff88024e203d78 ffffffff810a3932
[25097.209635]  0000000000000000 ffffffff81c4bf80 ffff88024e203d98 ffffffff810a76dd
[25097.210304]  ffff88024e20c440 0000000000000001 ffff88024e203dc8 ffffffff810ce920
[25097.210965] Call Trace:
[25097.211614]  <IRQ>  [<ffffffff810a3932>] sched_show_task+0xd2/0x140
[25097.212284]  [<ffffffff810a76dd>] dump_cpu_task+0x3d/0x50
[25097.212955]  [<ffffffff810ce920>] rcu_dump_cpu_stacks+0x90/0xd0
[25097.213628]  [<ffffffff810d4c33>] rcu_check_callbacks+0x4c3/0x730
[25097.214300]  [<ffffffff81120b4c>] ? acct_account_cputime+0x1c/0x20
[25097.214973]  [<ffffffff810a800e>] ? account_system_time+0x8e/0x190
[25097.215645]  [<ffffffff810da34b>] update_process_times+0x4b/0x80
[25097.216317]  [<ffffffff810e9cff>] tick_sched_timer+0x4f/0x160
[25097.216996]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25097.217676]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25097.218359]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25097.219040]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25097.219719]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25097.220415]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25097.221091]  <EOI>  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25097.221778]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25097.222454]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25097.223125]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25097.223793]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25097.224461] Task dump for CPU 1:
[25097.225124] trinity-c205    R  running task    13312   636  29562 0x10000008
[25097.225806]  00000000458261c0 ffff880066bae1b8 ffff880066bae1f0 ffff88021cbef200
[25097.226496]  ffff88024267bd20 0000000000000001 ffff88024267bd78 ffffffff811927cf
[25097.227190]  0000000000000000 0000000000000000 ffff880066bae1b8 0000000000000000
[25097.227889] Call Trace:
[25097.228578]  [<ffffffff811927cf>] ? unmap_mapping_range+0x12f/0x190
[25097.229279]  [<ffffffff8118193d>] ? shmem_fallocate+0x38d/0x4c0
[25097.230004]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25097.230708]  [<ffffffff811d38af>] ? do_fallocate+0x12f/0x1d0
[25097.231409]  [<ffffffff811a6995>] ? SyS_madvise+0x385/0x860
[25097.232106]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25097.232810]  [<ffffffff817994c4>] ? tracesys_phase2+0xd4/0xd9
[25121.945509] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636]
[25121.946236] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25121.950189] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25121.951854] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25121.952705] RIP: 0010:[<ffffffff810ee9ce>]  [<ffffffff810ee9ce>] generic_exec_single+0xee/0x1b0
[25121.953566] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25121.954428] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25121.955296] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25121.956199] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25121.957065] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25121.957937] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25121.958811] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25121.959692] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25121.960571] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25121.961462] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25121.962346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25121.963231] Stack:
[25121.964105]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25121.965010]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25121.965945]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25121.966838] Call Trace:
[25121.967723]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25121.968615]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25121.969494]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25121.970371]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25121.971239]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25121.972107]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25121.972968]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25121.973829]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25121.974690]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25121.975592]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25121.976454]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25121.977319]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25121.978178]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25121.979044]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25121.979905]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25121.980768]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25121.981636]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25121.982501]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25121.983362]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25121.984228]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25121.985086] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[25121.986993] sending NMI to other CPUs:
[25121.987891] NMI backtrace for cpu 0
[25121.988871] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25121.990863] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25121.991871] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25121.992886] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[25121.993898] RAX: 00000000cbd1340c RBX: 000000000051684a RCX: 0000000000000000
[25121.994911] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[25121.995923] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[25121.996942] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016dc7859e117
[25121.997961] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[25121.998975] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25122.000003] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25122.001032] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25122.002066] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25122.003094] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25122.004099] Stack:
[25122.005070]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[25122.006050]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[25122.007005]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[25122.007936] Call Trace:
[25122.008841]  <IRQ> 

[25122.009729]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25122.010594]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[25122.011453]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25122.012301]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25122.013148]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25122.013996]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25122.014843]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25122.015690]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25122.016534]  <EOI> 

[25122.017374]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25122.018207]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25122.019037]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25122.019859]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25122.020679]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25122.021490] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25122.023285] NMI backtrace for cpu 2
[25122.024138] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25122.025877] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25122.026772] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25122.027680] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25122.028590] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25122.029511] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25122.030427] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25122.031345] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005
[25122.032268] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25122.033189] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25122.034115] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25122.035042] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25122.035977] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25122.036911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25122.037844] Stack:
[25122.038768]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25122.039719]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25122.040671]  000016dc8fbb58a7 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25122.041629] Call Trace:
[25122.042576]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25122.043530]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25122.044462]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25122.045370]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25122.046275] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25122.048261] NMI backtrace for cpu 3
[25122.049177] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25122.050992] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25122.051911] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25122.052836] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25122.053757] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25122.054682] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25122.055628] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff
[25122.056550] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005
[25122.057466] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25122.058374] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25122.059282] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25122.060191] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25122.061107] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25122.062020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25122.062925] Stack:
[25122.063822]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25122.064744]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25122.065690]  000016dc8fbb74c1 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25122.066617] Call Trace:
[25122.067536]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25122.068470]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25122.069400]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25122.070329]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25122.071258] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25149.929286] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636]
[25149.930256] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25149.935393] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25149.937508] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25149.938555] RIP: 0010:[<ffffffff810ee9ce>]  [<ffffffff810ee9ce>] generic_exec_single+0xee/0x1b0
[25149.939652] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25149.940706] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25149.941751] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25149.942780] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25149.943803] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25149.944822] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25149.945839] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25149.946862] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25149.947891] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25149.948928] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25149.949997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25149.951027] Stack:
[25149.952045]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25149.953082]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25149.954119]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25149.955159] Call Trace:
[25149.956184]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25149.957219]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25149.958239]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25149.959258]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25149.960299]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25149.961305]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25149.962310]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25149.963313]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25149.964320]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25149.965312]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25149.966281]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25149.967249]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25149.968212]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25149.969163]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25149.970131]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25149.971056]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25149.971977]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25149.972895]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25149.973812]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25149.974735]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25149.975654] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[25149.977642] sending NMI to other CPUs:
[25149.978587] NMI backtrace for cpu 0
[25149.979615] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25149.981704] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25149.982766] RIP: 0010:[<ffffffff810cf1db>]  [<ffffffff810cf1db>] invoke_rcu_core+0x2b/0x50
[25149.983834] RSP: 0018:ffff88024e203db8  EFLAGS: 00000093
[25149.984894] RAX: ffffffff81cfe330 RBX: 0000000000000000 RCX: 00000000000bde85
[25149.985959] RDX: 00000000000bde86 RSI: ffff88024e20c440 RDI: ffffffff81c4bf80
[25149.987021] RBP: ffff88024e203dc8 R08: 0000000000000000 R09: 000000000000000f
[25149.988084] R10: 0000000000000526 R11: 0000000000aaaaaa R12: ffffffff81c4bf80
[25149.989155] R13: ffffffff81c4bf80 R14: 0000000000000000 R15: 000016eacc784674
[25149.990227] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25149.991310] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25149.992374] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25149.993420] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25149.994462] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25149.995477] Stack:
[25149.996463]  ffff8801bf3536b0 ffff88024e20c440 ffff88024e203e38 ffffffff810d48f3
[25149.997456]  ffff88024e203de8 ffffffff81120b4c ffff88024e203e28 ffffffff810a800e
[25149.998421]  ffff88024e203e08 ffffffff81cfe338 ffff88024e203e38 ffff8801bf3536b0
[25149.999364] Call Trace:
[25150.000280]  <IRQ> 

[25150.001179]  [<ffffffff810d48f3>] rcu_check_callbacks+0x183/0x730
[25150.002063]  [<ffffffff81120b4c>] ? acct_account_cputime+0x1c/0x20
[25150.002939]  [<ffffffff810a800e>] ? account_system_time+0x8e/0x190
[25150.003805]  [<ffffffff810da34b>] update_process_times+0x4b/0x80
[25150.004669]  [<ffffffff810e9cff>] tick_sched_timer+0x4f/0x160
[25150.005530]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25150.006391]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25150.007248]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25150.008106]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25150.008958]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25150.009803]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25150.010645]  <EOI> 

[25150.011487]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25150.012328]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25150.013162]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25150.013991]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25150.014809]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25150.015627] Code: 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 08 65 8b 1c 25 2c a0 00 00 3b 1d e8 02 c3 00 73 24 89 db 48 8b 05 31 3e 74 00 48 0f a3 18 <19> db 85 db 74 0a bf 09 00 00 00 e8 b5 be fa ff 48 83 c4 08 5b 
[25150.017421] NMI backtrace for cpu 2
[25150.018283] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25150.020063] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25150.020972] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25150.021897] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25150.022819] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25150.023745] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25150.024669] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25150.025595] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005
[25150.026523] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25150.027445] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25150.028378] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25150.029311] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25150.030250] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25150.031190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25150.032125] Stack:
[25150.033056]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25150.034012]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25150.034966]  000016e314a978e8 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25150.035903] Call Trace:
[25150.036807]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25150.037724]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25150.038637]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25150.039548]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25150.040438] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25150.042345] NMI backtrace for cpu 3
[25150.043231] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25150.045019] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25150.045926] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25150.046843] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25150.047755] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25150.048670] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25150.049598] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff
[25150.050500] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005
[25150.051406] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25150.052313] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25150.053216] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25150.054114] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25150.055021] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25150.055929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25150.056829] Stack:
[25150.057727]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25150.058649]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25150.059600]  000016e314a99619 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25150.060530] Call Trace:
[25150.061450]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25150.062380]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25150.063303]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25150.064231]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25150.065157] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25177.913063] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636]
[25177.914027] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25177.919143] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G             L 3.18.0+ #108
[25177.921249] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000
[25177.922289] RIP: 0010:[<ffffffff810ee9ca>]  [<ffffffff810ee9ca>] generic_exec_single+0xea/0x1b0
[25177.923382] RSP: 0018:ffff88024267b9c8  EFLAGS: 00000202
[25177.924433] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038
[25177.925476] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25177.926504] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000
[25177.927525] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938
[25177.928542] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60
[25177.929556] FS:  00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25177.930580] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25177.931606] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0
[25177.932638] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25177.933707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25177.934735] Stack:
[25177.935750]  ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000
[25177.936786]  ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768
[25177.937819]  ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0
[25177.938856] Call Trace:
[25177.939882]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25177.940911]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25177.941928]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25177.942941]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25177.943980]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25177.944982]  [<ffffffff81173f62>] ? release_pages+0x1c2/0x280
[25177.945985]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25177.946988]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25177.947993]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25177.948984]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25177.949951]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25177.950920]  [<ffffffff8116d377>] ? __free_pages+0x37/0x50
[25177.951886]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25177.952837]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25177.953805]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25177.954730]  [<ffffffff8178df51>] ? cmpxchg_double_slab.isra.49+0xd8/0x122
[25177.955653]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25177.956573]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25177.957491]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25177.958414]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25177.959332] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 
[25177.961321] sending NMI to other CPUs:
[25177.962268] NMI backtrace for cpu 0
[25177.963295] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G             L 3.18.0+ #108
[25177.965391] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000
[25177.966456] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25177.967525] RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
[25177.968586] RAX: 00000000fb9d303f RBX: 00000000005193fc RCX: 0000000000000000
[25177.969651] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
[25177.970712] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
[25177.971777] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016e982361117
[25177.972846] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674
[25177.973915] FS:  00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25177.974998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25177.976058] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0
[25177.977103] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25177.978139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25177.979151] Stack:
[25177.980134]  ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80
[25177.981123]  ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3
[25177.982089]  ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0
[25177.983029] Call Trace:
[25177.983943]  <IRQ> 

[25177.984839]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25177.985714]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[25177.986581]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[25177.987439]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[25177.988296]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[25177.989149]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[25177.990007]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[25177.990862]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[25177.991714]  <EOI> 

[25177.992558]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[25177.993396]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[25177.994233]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[25177.995063]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70
[25177.995891]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25177.996712] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25177.998518] NMI backtrace for cpu 2
[25177.999377] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25178.001121] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25178.002019] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25178.002932] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25178.003849] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25178.004771] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25178.005688] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff
[25178.006609] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005
[25178.007532] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25178.008453] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25178.009379] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25178.010307] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25178.011246] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25178.012181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25178.013112] Stack:
[25178.014036]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25178.014985]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25178.015936]  000016e99997935a ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25178.016893] Call Trace:
[25178.017836]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25178.018791]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25178.019719]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25178.020625]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25178.021526] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25178.023504] NMI backtrace for cpu 3
[25178.024417] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25178.026229] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25178.027145] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25178.028067] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25178.028983] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25178.029906] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25178.030826] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff
[25178.031744] R10: 0000000000002656 R11: 00000000000003ff R12: 0000000000000005
[25178.032657] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25178.033586] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25178.034496] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25178.035401] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25178.036319] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25178.037229] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25178.038130] Stack:
[25178.039023]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25178.039948]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25178.040867]  000016e99997b046 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25178.041793] Call Trace:
[25178.042713]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25178.043663]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25178.044589]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25178.045515]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25178.046441] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25183.090076] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 88.570 msecs
[25183.090087] sched: RT throttling activated
[25205.896861] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [trinity-c65:8051]
[25205.897959] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25205.904284] CPU: 1 PID: 8051 Comm: trinity-c65 Tainted: G             L 3.18.0+ #108
[25205.906737] task: ffff8800a1b98000 ti: ffff880071a70000 task.ti: ffff880071a70000
[25205.907986] RIP: 0010:[<ffffffff8178ecca>]  [<ffffffff8178ecca>] __slab_alloc+0x4e5/0x53b
[25205.909296] RSP: 0018:ffff880071a739f8  EFLAGS: 00000246
[25205.910532] RAX: 0000000000000001 RBX: ffff88022f966c40 RCX: 0000000180240022
[25205.911789] RDX: 0000000000000024 RSI: ffffea0008be5800 RDI: ffff880245827cc0
[25205.913018] RBP: ffff880071a73ac8 R08: ffff88022f965138 R09: 0000000000000000
[25205.914222] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810133af
[25205.915459] R13: ffff880071a73978 R14: 0000000100240021 R15: ffffffff8134f019
[25205.916728] FS:  00007f2805635740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25205.917955] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25205.919158] CR2: 0000000001a86a10 CR3: 0000000097832000 CR4: 00000000001407e0
[25205.920348] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25205.921514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25205.922678] Stack:
[25205.923828]  ffff880071a73a68 ffffffff811b7057 ffff8800a1b98000 ffff8800a1b98000
[25205.925006]  0000000000000000 ffff88022f9669f0 0000000200000000 ffffffff8134f019
[25205.926179]  ffffffff00000000 0000000000000246 0000000000000000 0000000000000000
[25205.927394] Call Trace:
[25205.928579]  [<ffffffff811b7057>] ? alloc_pages_vma+0x97/0x160
[25205.929762]  [<ffffffff8134f019>] ? __radix_tree_preload+0x49/0xc0
[25205.930972]  [<ffffffff8117ccde>] ? shmem_alloc_page+0x6e/0xc0
[25205.932185]  [<ffffffff8134fd05>] ? __radix_tree_create+0x85/0x220
[25205.933398]  [<ffffffff8134f019>] ? __radix_tree_preload+0x49/0xc0
[25205.934603]  [<ffffffff811c012b>] kmem_cache_alloc+0x1bb/0x1e0
[25205.935825]  [<ffffffff8134f019>] __radix_tree_preload+0x49/0xc0
[25205.937011]  [<ffffffff8134f101>] radix_tree_maybe_preload+0x21/0x30
[25205.938175]  [<ffffffff8117fa46>] shmem_getpage_gfp+0x466/0x810
[25205.939323]  [<ffffffff8109bd1a>] ? finish_task_switch+0x4a/0x100
[25205.940453]  [<ffffffff8117fe2f>] shmem_write_begin+0x3f/0x60
[25205.941549]  [<ffffffff811652e4>] generic_perform_write+0xd4/0x1f0
[25205.942627]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25205.943694]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25205.944754]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25205.945813]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.946885]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25205.948034]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25205.949150]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.950297]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.951463]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25205.952586]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25205.953678]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25205.954747]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25205.955854]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25205.956941] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 07 b4 9a ff 4c 89 e0 eb 0f e8 fd b4 9a ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 75 c8 65 48 33 34 25 28 00 00 00 74 32 e8 ff 79 
[25205.959247] sending NMI to other CPUs:
[25205.960438] NMI backtrace for cpu 2
[25205.961454] CPU: 2 PID: 7491 Comm: trinity-c14 Tainted: G             L 3.18.0+ #108
[25205.963519] task: ffff88022b98a0d0 ti: ffff880096408000 task.ti: ffff880096408000
[25205.964574] RIP: 0010:[<ffffffff811741ea>]  [<ffffffff811741ea>] __lru_cache_add+0x4a/0x90
[25205.965645] RSP: 0018:ffff88009640bb38  EFLAGS: 00000293
[25205.966690] RAX: 000000000000000c RBX: ffff88024e28e0c0 RCX: 000000000001602a
[25205.967724] RDX: 000000000000000d RSI: 0000000000000018 RDI: ffffea00042410c0
[25205.968745] RBP: ffff88009640bb48 R08: ffff8801020dae90 R09: 0000000000000000
[25205.969761] R10: 0000000000000001 R11: 0000000000000000 R12: ffffea00042410c0
[25205.970773] R13: 0000000000000000 R14: ffff880096339f18 R15: 0000000000000000
[25205.971763] FS:  00007f2805635740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25205.972737] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25205.973689] CR2: 0000003370219050 CR3: 000000009ab9e000 CR4: 00000000001407e0
[25205.974628] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25205.975543] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25205.976440] Stack:
[25205.977315]  0000000000000003 0000000000000000 ffff88009640bb58 ffffffff81174919
[25205.978198]  ffff88009640bc18 ffffffff8117fa80 ffff88009640bb98 ffffffff8109bdb0
[25205.979071]  ffff8802292c8980 ffff88009640bc80 0000000000000000 00000000000000d0
[25205.979939] Call Trace:
[25205.980791]  [<ffffffff81174919>] lru_cache_add_anon+0x19/0x20
[25205.981659]  [<ffffffff8117fa80>] shmem_getpage_gfp+0x4a0/0x810
[25205.982526]  [<ffffffff8109bdb0>] ? finish_task_switch+0xe0/0x100
[25205.983385]  [<ffffffff8117fe2f>] shmem_write_begin+0x3f/0x60
[25205.984245]  [<ffffffff811652e4>] generic_perform_write+0xd4/0x1f0
[25205.985098]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25205.985946]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25205.986787]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25205.987629]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.988479]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25205.989323]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25205.990156]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.990994]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25205.991822]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25205.992644]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25205.993461]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25205.994277]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25205.995089]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25205.995900] Code: 48 03 1c 25 68 bd 00 00 48 8b 07 f6 c4 80 75 47 41 8b 44 24 1c 85 c0 7e 4c f0 41 ff 44 24 1c 48 8b 03 83 f8 0e 74 1a 48 8d 50 01 <48> 89 13 4c 89 64 c3 10 5b 65 ff 0c 25 60 a8 00 00 41 5c 5d c3 
[25205.997698] NMI backtrace for cpu 3
[25205.998563] CPU: 3 PID: 10185 Comm: trinity-c162 Tainted: G             L 3.18.0+ #108
[25206.000332] task: ffff880197ea2bc0 ti: ffff8801c6410000 task.ti: ffff8801c6410000
[25206.001236] RIP: 0010:[<ffffffff810ee9ce>]  [<ffffffff810ee9ce>] generic_exec_single+0xee/0x1b0
[25206.002161] RSP: 0018:ffff8801c64139c8  EFLAGS: 00000202
[25206.003077] RAX: 0000000000000008 RBX: ffff8801c64139e0 RCX: 0000000000000038
[25206.004003] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000
[25206.004926] RBP: ffff8801c6413a28 R08: ffff8802444f47e0 R09: 0000000000000000
[25206.005848] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[25206.006769] R13: 0000000000000001 R14: ffff880229f27a00 R15: 0000000000000003
[25206.007683] FS:  00007f2805635740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25206.008612] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25206.009539] CR2: 00007f28048e2220 CR3: 000000021ca6c000 CR4: 00000000001407e0
[25206.010484] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25206.011426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25206.012367] Stack:
[25206.013300]  ffffffff8124055d ffff8801e640ddc0 ffffffff8124055d 0000000000000000
[25206.014263]  ffffffff81047bb0 ffff8801c6413ad8 0000000000000003 00000000e731055f
[25206.015228]  ffff8801c6413a48 00000000ffffffff 0000000000000001 ffffffff81047bb0
[25206.016172] Call Trace:
[25206.017090]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25206.018026]  [<ffffffff8124055d>] ? proc_alloc_inode+0x1d/0xb0
[25206.018958]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25206.019879]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25206.020766]  [<ffffffff810eeb30>] smp_call_function_single+0x70/0xd0
[25206.021637]  [<ffffffff81047bb0>] ? do_flush_tlb_all+0x60/0x60
[25206.022498]  [<ffffffff810ef229>] smp_call_function_many+0x2b9/0x320
[25206.023356]  [<ffffffff81184681>] ? zone_statistics+0x81/0xa0
[25206.024210]  [<ffffffff81047ed0>] flush_tlb_mm_range+0x90/0x1b0
[25206.025067]  [<ffffffff81190d62>] tlb_flush_mmu_tlbonly+0x42/0x50
[25206.025914]  [<ffffffff811922c8>] unmap_single_vma+0x6d8/0x900
[25206.026757]  [<ffffffff811925ec>] zap_page_range_single+0xfc/0x160
[25206.027601]  [<ffffffff811927cf>] unmap_mapping_range+0x12f/0x190
[25206.028439]  [<ffffffff8118193d>] shmem_fallocate+0x38d/0x4c0
[25206.029270]  [<ffffffff811d38af>] do_fallocate+0x12f/0x1d0
[25206.030089]  [<ffffffff811a6995>] SyS_madvise+0x385/0x860
[25206.030910]  [<ffffffff811c029e>] ? kmem_cache_free+0x14e/0x1e0
[25206.031733]  [<ffffffff811e6029>] ? putname+0x29/0x40
[25206.032546]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25206.033361]  [<ffffffff817994c4>] tracesys_phase2+0xd4/0xd9
[25206.034170] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 
[25206.035960] NMI backtrace for cpu 0
[25206.036822] CPU: 0 PID: 9587 Comm: trinity-c129 Tainted: G             L 3.18.0+ #108
[25206.038638] task: ffff88009a8695e0 ti: ffff880039f48000 task.ti: ffff880039f48000
[25206.039565] RIP: 0033:[<000000336ee891e5>]  [<000000336ee891e5>] 0x336ee891e5
[25206.040510] RSP: 002b:00007fffdd789658  EFLAGS: 00000202
[25206.041437] RAX: 0000000001b33250 RBX: 0000000001b33240 RCX: 000000000202fe00
[25206.042378] RDX: 0000000002533240 RSI: 0000000000000054 RDI: 0000000001b33250
[25206.043317] RBP: 0000000000a00010 R08: 0000000001b33240 R09: 000000000000002e
[25206.044255] R10: 0000000000003666 R11: 0000000000000246 R12: 0000000002533250
[25206.045186] R13: 0000000000020db0 R14: 0000000000000000 R15: 0000000000000000
[25206.046117] FS:  00007f2805635740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25206.047047] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25206.047954] CR2: 00000000019fcad0 CR3: 000000009edfa000 CR4: 00000000001407f0
[25206.048866] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25206.049779] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600

[25206.051604] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 91.159 msecs
[25233.880621] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c251:17822]
[25233.881750] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25233.887204] CPU: 1 PID: 17822 Comm: trinity-c251 Tainted: G             L 3.18.0+ #108
[25233.889315] task: ffff8801c65ec1a0 ti: ffff88009b7b0000 task.ti: ffff88009b7b0000
[25233.890406] RIP: 0010:[<ffffffff81356ec5>]  [<ffffffff81356ec5>] copy_user_enhanced_fast_string+0x5/0x10
[25233.891548] RSP: 0018:ffff88009b7b3be0  EFLAGS: 00010206
[25233.892666] RAX: 00007f95d93d2239 RBX: ffffffff8117faa1 RCX: 00000000000000a0
[25233.893801] RDX: 0000000000001000 RSI: 00007f95d93d3199 RDI: ffff88019049ef60
[25233.894943] RBP: ffff88009b7b3c28 R08: ffff88022f8d1668 R09: 0000000000000000
[25233.896102] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[25233.897252] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
[25233.898395] FS:  00007f95dc296740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25233.899559] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25233.900733] CR2: 00007f95da9e0bbe CR3: 0000000095a94000 CR4: 00000000001407e0
[25233.901878] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25233.903009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25233.904151] Stack:
[25233.905253]  ffffffff8118e606 ffff88009b7b3bf8 000000001bb22000 0000000000001000
[25233.906364]  000000001bb22000 0000000000001000 ffff88009b7b3d60 0000000000000000
[25233.907478]  ffff8800a2518d28 ffff88009b7b3cc8 ffffffff81165307 ffff88009b7b3c88
[25233.908613] Call Trace:
[25233.909759]  [<ffffffff8118e606>] ? iov_iter_copy_from_user_atomic+0x156/0x180
[25233.910938]  [<ffffffff81165307>] generic_perform_write+0xf7/0x1f0
[25233.912164]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25233.913376]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25233.914574]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25233.915730]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25233.916890]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25233.918061]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25233.919202]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25233.920376]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25233.921584]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25233.922742]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25233.923883]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25233.925034]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25233.926189]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25233.927320] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 <f3> a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 
[25233.929775] sending NMI to other CPUs:
[25233.930989] NMI backtrace for cpu 0
[25233.932037] CPU: 0 PID: 17755 Comm: trinity-c184 Tainted: G             L 3.18.0+ #108
[25233.934172] task: ffff88022d25cc90 ti: ffff880229f78000 task.ti: ffff880229f78000
[25233.935239] RIP: 0010:[<ffffffff81593482>]  [<ffffffff81593482>] usb_unanchor_urb+0x12/0x60
[25233.936302] RSP: 0018:ffff88024e203d50  EFLAGS: 00000082
[25233.937352] RAX: 0000000000000000 RBX: ffff880242a2ecb0 RCX: 0000000000000000
[25233.938412] RDX: 000000000000000e RSI: 0000000000000046 RDI: ffff880242a2ecb0
[25233.939464] RBP: ffff88024e203d68 R08: ffffffff81f5dea8 R09: 00000000000fffe2
[25233.940527] R10: fffffffff8000000 R11: 0000000000000004 R12: ffff88009f014f38
[25233.941592] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880242080000
[25233.942660] FS:  00007f95dc296740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25233.943738] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25233.944808] CR2: 0000003370219050 CR3: 000000022e902000 CR4: 00000000001407f0
[25233.945886] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25233.946955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25233.948024] Stack:
[25233.949088]  ffff88009f014f38 0000000000000000 0000000000000000 ffff88024e203d98
[25233.950171]  ffffffff8158fc9c ffff880241d1d6c0 ffff880242a2ecb0 ffff8802416a4558
[25233.951243]  ffff880241d1d6c0 ffff88024e203dc8 ffffffff8158fec0 ffff88024e203dc8
[25233.952305] Call Trace:
[25233.953331]  <IRQ> 

[25233.954339]  [<ffffffff8158fc9c>] __usb_hcd_giveback_urb+0x5c/0x120
[25233.955323]  [<ffffffff8158fec0>] usb_hcd_giveback_urb+0x40/0xf0
[25233.956290]  [<ffffffff815d276f>] xhci_irq+0xccf/0x1bc0
[25233.957234]  [<ffffffff815d3671>] xhci_msi_irq+0x11/0x20
[25233.958161]  [<ffffffff810c4d2e>] handle_irq_event_percpu+0x3e/0x1c0
[25233.959072]  [<ffffffff810c4ef2>] handle_irq_event+0x42/0x70
[25233.959958]  [<ffffffff810c801f>] handle_edge_irq+0x7f/0x150
[25233.960829]  [<ffffffff81005371>] handle_irq+0xb1/0x140
[25233.961691]  [<ffffffff8179bbb3>] do_IRQ+0x53/0x100
[25233.962544]  [<ffffffff81799daf>] common_interrupt+0x6f/0x6f
[25233.963393]  <EOI> 

[25233.964240]  [<ffffffff81165a08>] ? unlock_page+0x18/0x90
[25233.965088]  [<ffffffff81180d42>] shmem_undo_range+0x1b2/0x710
[25233.965941]  [<ffffffff811812b8>] shmem_truncate_range+0x18/0x40
[25233.966786]  [<ffffffff81181520>] shmem_setattr+0x110/0x1a0
[25233.967622]  [<ffffffff811f3361>] notify_change+0x241/0x390
[25233.968453]  [<ffffffff811d3323>] do_truncate+0x73/0xc0
[25233.969279]  [<ffffffff811d36cc>] do_sys_ftruncate.constprop.14+0x10c/0x160
[25233.970113]  [<ffffffff8135862e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[25233.970943]  [<ffffffff811d375e>] SyS_ftruncate+0xe/0x10
[25233.971761]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25233.972581] Code: 00 49 8d 7c 24 10 31 c9 ba 01 00 00 00 be 03 00 00 00 e8 42 50 b2 ff eb cd 0f 1f 44 00 00 55 48 85 ff 48 89 e5 41 56 41 55 41 54 <53> 48 89 fb 74 35 4c 8b 67 40 4d 85 e4 74 2c 4d 8d 6c 24 28 4c 
[25233.974377] NMI backtrace for cpu 3
[25233.975270] CPU: 3 PID: 17777 Comm: trinity-c206 Tainted: G             L 3.18.0+ #108
[25233.977090] task: ffff880228aa2bc0 ti: ffff88022936c000 task.ti: ffff88022936c000
[25233.978013] RIP: 0010:[<ffffffff81165a15>]  [<ffffffff81165a15>] unlock_page+0x25/0x90
[25233.978949] RSP: 0018:ffff88022936fbf8  EFLAGS: 00000202
[25233.979880] RAX: ffffea0008ee2d80 RBX: ffffea0008ee2d80 RCX: 00000000146c9000
[25233.980816] RDX: 002ffe000008001c RSI: 9e37fffffffc0001 RDI: ffffea0008ee2d80
[25233.981761] RBP: ffff88022936fbf8 R08: 0000000000001000 R09: ffffea0008ee2d80
[25233.982702] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000
[25233.983653] R13: ffff88022936fd60 R14: 0000000000000000 R15: ffff8800a251da08
[25233.984603] FS:  00007f95dc296740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25233.985565] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25233.986528] CR2: 0000000000000008 CR3: 000000022d223000 CR4: 00000000001407e0
[25233.987505] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25233.988484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25233.989436] Stack:
[25233.990361]  ffff88022936fc28 ffffffff8117f380 00000000146c8000 0000000000001000
[25233.991308]  ffff88022936fd60 0000000000000000 ffff88022936fcc8 ffffffff8116532a
[25233.992263]  ffff88022936fd68 0000000000001000 ffff880228aa2bc0 00000000811f1681
[25233.993212] Call Trace:
[25233.994140]  [<ffffffff8117f380>] shmem_write_end+0x40/0xf0
[25233.995056]  [<ffffffff8116532a>] generic_perform_write+0x11a/0x1f0
[25233.995969]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25233.996873]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25233.997776]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25233.998673]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25233.999570]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25234.000462]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25234.001357]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25234.002277]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25234.003155]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25234.004045]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25234.004942]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25234.005827]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25234.006705]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25234.007581] Code: 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 f8 48 8b 17 48 89 e5 83 e2 01 74 60 f0 80 27 fe 48 be 01 00 fc ff ff ff 37 9e 48 8b 17 <48> 0f af f7 48 89 d1 48 c1 ea 34 83 e2 03 48 c1 e9 36 48 8d 14 
[25234.009513] NMI backtrace for cpu 2
[25234.010438] CPU: 2 PID: 17723 Comm: trinity-c152 Tainted: G             L 3.18.0+ #108
[25234.012415] task: ffff880229a50af0 ti: ffff88022e918000 task.ti: ffff88022e918000
[25234.013417] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>] read_hpet+0x16/0x20
[25234.014445] RSP: 0018:ffff88022e91be58  EFLAGS: 00000046
[25234.015465] RAX: 000000002b67efe4 RBX: 000000000051c224 RCX: 0000000000000004
[25234.016505] RDX: 0000000000004949 RSI: ffff88022e91be98 RDI: ffffffff81c26f40
[25234.017547] RBP: ffff88022e91be58 R08: 0000000000000004 R09: 00000000004323f1
[25234.018587] R10: 0000000000000004 R11: 0000000000000202 R12: 000016f68c124117
[25234.019640] R13: 0000000000000000 R14: ffff880229825108 R15: 0000000000000000
[25234.020693] FS:  00007f95dc296740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25234.021740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25234.022761] CR2: 00007f95dc055220 CR3: 000000014fecd000 CR4: 00000000001407e0
[25234.023813] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25234.024844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25234.025871] Stack:
[25234.026890]  ffff88022e91be88 ffffffff810e0d3e ffff88022e91be98 ffff880229825108
[25234.027926]  000016f6dfac1843 ffff88022e91bf18 ffff88022e91beb8 ffffffff810da9da
[25234.028977]  0000000000000086 00000000749a29d5 ffff880229825108 ffff88022e91bf38
[25234.030035] Call Trace:
[25234.031074]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[25234.032113]  [<ffffffff810da9da>] hrtimer_get_remaining+0x3a/0x70
[25234.033154]  [<ffffffff810dbbf6>] itimer_get_remtime+0x16/0x60
[25234.034172]  [<ffffffff810dc242>] do_setitimer+0xb2/0x260
[25234.035175]  [<ffffffff810dc439>] alarm_setitimer+0x49/0x90
[25234.036173]  [<ffffffff810da3ae>] SyS_alarm+0xe/0x20
[25234.037171]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25234.038136] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 
[25234.040197] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 109.194 msecs
[25261.864386] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c24:30558]
[25261.865435] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25261.871111] CPU: 1 PID: 30558 Comm: trinity-c24 Tainted: G             L 3.18.0+ #108
[25261.873609] task: ffff88009b63abc0 ti: ffff8802283d4000 task.ti: ffff8802283d4000
[25261.874875] RIP: 0033:[<0000000000412fc8>]  [<0000000000412fc8>] 0x412fc8
[25261.876181] RSP: 002b:00007fff73bbaf70  EFLAGS: 00000202
[25261.877378] RAX: 00000000759aa432 RBX: ffffffff81799dfa RCX: 0000000000000038
[25261.878584] RDX: 000000000bc2aa05 RSI: 00007fff73bbaf4c RDI: 000000336f1b76e0
[25261.879738] RBP: 0000000000000002 R08: 000000336f1b70e8 R09: 000000336f1b7140
[25261.880885] R10: 0000000000000000 R11: 0000000000000206 R12: ffff8802283d7f78
[25261.882058] R13: 0000000000000000 R14: ffff8802283d4000 R15: ffff88009b63abc0
[25261.883248] FS:  00007f95dc296740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25261.884396] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25261.885546] CR2: 0000000000000001 CR3: 0000000235a45000 CR4: 00000000001407e0
[25261.886701] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25261.887903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600

[25261.890230] sending NMI to other CPUs:
[25261.891374] NMI backtrace for cpu 0
[25261.892452] CPU: 0 PID: 30551 Comm: trinity-c4 Tainted: G             L 3.18.0+ #108
[25261.894667] task: ffff880229a515e0 ti: ffff88000e014000 task.ti: ffff88000e014000
[25261.895791] RIP: 0010:[<ffffffff81799260>]  [<ffffffff81799260>] system_call+0x0/0x3
[25261.896927] RSP: 0018:00007fff73bbafa8  EFLAGS: 00000046
[25261.898061] RAX: 000000000000003d RBX: 0000000000007780 RCX: 000000336eebc2fc
[25261.899206] RDX: 000000000000000b RSI: 00007fff73bbafb0 RDI: 0000000000007780
[25261.900327] RBP: 0000000000000000 R08: 00007f95dc296740 R09: 0000000000000000
[25261.901427] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f95dc0cb000
[25261.902524] R13: 00007f95dc0cb068 R14: 0000000000000000 R15: 0000000000000000
[25261.903615] FS:  00007f95dc296740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25261.904695] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25261.905755] CR2: 000000336f1b7740 CR3: 0000000095b7d000 CR4: 00000000001407f0
[25261.906819] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25261.907870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25261.908922] Stack:
[25261.909964]  000000000041401b 00007f95dc216768 00007f95dc0cb000 0000000000000000
[25261.911029]  00007f95dc0cb07c 00007f95dc0cb000 0000000000000000 00007f95dc0cb07c
[25261.912090]  0000000000416c03 00000000000042bc 0000000000000017 00000000cccccccd
[25261.913152] Call Trace:
[25261.914193]  <UNK> 
[25261.914200] Code: 
[25261.915236] 8b 3c 24 4c 8b 74 24 08 4c 8b 6c 24 10 4c 8b 64 24 18 48 8b 6c 24 20 48 8b 5c 24 28 48 83 c4 30 e9 6f 02 00 00 66 0f 1f 44 00 00 <0f> 01 f8 65 48 89 24 25 80 a0 00 00 65 48 8b 24 25 88 a8 00 00 
[25261.917470] NMI backtrace for cpu 3
[25261.918589] CPU: 3 PID: 30572 Comm: trinity-c230 Tainted: G             L 3.18.0+ #108
[25261.920852] task: ffff8800967736b0 ti: ffff88022cbc0000 task.ti: ffff88022cbc0000
[25261.922001] RIP: 0010:[<ffffffff8118f1c4>]  [<ffffffff8118f1c4>] iov_iter_fault_in_readable+0x64/0x80
[25261.923165] RSP: 0018:ffff88022cbc3c18  EFLAGS: 00000206
[25261.924316] RAX: 0000000000000000 RBX: 00000000029ef000 RCX: 0000000000001007
[25261.925475] RDX: 00007f95d952b800 RSI: 0000000000001000 RDI: 000000000001e800
[25261.926633] RBP: ffff88022cbc3c28 R08: ffff880066aca360 R09: 0000000000094524
[25261.927794] R10: ffff88022cbc3af8 R11: 00000000000003ff R12: 0000000000001000
[25261.928954] R13: ffff88022cbc3d60 R14: 0000000000001000 R15: ffff8800a2518d28
[25261.930112] FS:  00007f95dc296740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25261.931280] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25261.932458] CR2: 00007f95d9a2036e CR3: 000000022b2d6000 CR4: 00000000001407e0
[25261.933618] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25261.934753] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25261.935873] Stack:
[25261.936974]  0000000000001000 00ff8800a2518d28 ffff88022cbc3cc8 ffffffff811652b8
[25261.938081]  ffff88022cbc3c98 0000000000001000 ffff8800967736b0 00000000811f1681
[25261.939164]  0000000000f60000 0000000000000000 ffff88009f3c9400 ffffffff81828380
[25261.940227] Call Trace:
[25261.941261]  [<ffffffff811652b8>] generic_perform_write+0xa8/0x1f0
[25261.942290]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25261.943300]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25261.944291]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25261.945270]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25261.946243]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25261.947210]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25261.948174]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25261.949141]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25261.950106]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25261.951068]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25261.952027]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25261.952982]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25261.953928]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25261.954865] Code: 10 0f 1f 00 40 8a 39 0f 1f 00 85 d2 40 88 7d ff 75 2d 48 63 f6 48 8d 54 31 ff 48 31 d1 48 f7 c1 00 f0 ff ff 74 0f 0f 1f 00 8a 12 <0f> 1f 00 88 55 ff 0f b6 55 ff c9 f3 c3 0f 1f 80 00 00 00 00 c9 
[25261.956947] NMI backtrace for cpu 2
[25261.957928] CPU: 2 PID: 30581 Comm: trinity-c42 Tainted: G             L 3.18.0+ #108
[25261.959978] task: ffff88022c1f2bc0 ti: ffff8800a2604000 task.ti: ffff8800a2604000
[25261.961020] RIP: 0033:[<00007fff73bd3bfe>]  [<00007fff73bd3bfe>] 0x7fff73bd3bfe
[25261.962065] RSP: 002b:00007fff73bbaf20  EFLAGS: 00000246
[25261.963081] RAX: 0000000043481221 RBX: 00007f95dbedd068 RCX: 000000336eeef4b9
[25261.964103] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 00007f95dbedd000
[25261.965115] RBP: 00007fff73bbaf20 R08: 002b34bd7fef2fdf R09: 00000000004d1e4c
[25261.966125] R10: 0000000000000002 R11: 0000000000000246 R12: 00007f95dbedd000
[25261.967135] R13: 00000000000001ff R14: 0000000000000000 R15: 00007f95dc2966a0
[25261.968161] FS:  00007f95dc296740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25261.969187] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25261.970198] CR2: 0000000000000008 CR3: 0000000040239000 CR4: 00000000001407e0
[25261.971236] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25261.972260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600

[25289.848160] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c27:31317]
[25289.849258] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25289.855194] CPU: 1 PID: 31317 Comm: trinity-c27 Tainted: G             L 3.18.0+ #108
[25289.857533] task: ffff8802415a0af0 ti: ffff8802290e4000 task.ti: ffff8802290e4000
[25289.858723] RIP: 0010:[<ffffffff81356c07>]  [<ffffffff81356c07>] clear_page_c_e+0x7/0x10
[25289.859915] RSP: 0000:ffff8802290e7ce0  EFLAGS: 00010246
[25289.861099] RAX: 0000000000000000 RBX: ffff8802415a0af0 RCX: 00000000000001c0
[25289.862267] RDX: 0000000000007a55 RSI: 0000000000000ead RDI: ffff880123d7ae40
[25289.863466] RBP: ffff8802290e7d18 R08: ffffea00048f0000 R09: 00000000000160f0
[25289.864660] R10: ffff88024e5d4d80 R11: 0000000000000000 R12: 0000000000000000
[25289.865808] R13: ffff8802415a0af0 R14: ffff88024e5d5b00 R15: 0000000200000000
[25289.866970] FS:  00007fd3c5236740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25289.868134] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25289.869294] CR2: 0000000001c00000 CR3: 0000000227866000 CR4: 00000000001407e0
[25289.870491] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25289.871683] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25289.872843] Stack:
[25289.874017]  ffffffff81196633 ffff880235814070 0000000001c00000 0000000000000000
[25289.875201]  ffff8800967dcc00 ffff880235814070 ffff8801c64b4600 ffff8802290e7d78
[25289.876372]  ffffffff811ca951 ffff8802290e7e18 ffffffff8115f002 ffffea00048f0000
[25289.877537] Call Trace:
[25289.878705]  [<ffffffff81196633>] ? clear_huge_page+0xa3/0x160
[25289.879878]  [<ffffffff811ca951>] do_huge_pmd_anonymous_page+0x151/0x3b0
[25289.881085]  [<ffffffff8115f002>] ? __perf_sw_event+0x1b2/0x200
[25289.882274]  [<ffffffff81194c1b>] handle_mm_fault+0x14b/0xe90
[25289.883439]  [<ffffffff810413cc>] __do_page_fault+0x20c/0x610
[25289.884554]  [<ffffffff8119b61d>] ? do_brk+0x24d/0x350
[25289.885663]  [<ffffffff8135866a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[25289.886786]  [<ffffffff810417dc>] do_page_fault+0xc/0x10
[25289.887911]  [<ffffffff8179ae62>] page_fault+0x22/0x30
[25289.889033] Code: bc 0f 1f 00 e8 fb fa d1 ff 90 90 90 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3 0f 1f 44 00 00 b9 00 10 00 00 31 c0 <f3> aa c3 66 0f 1f 44 00 00 eb ee 0f 1f 84 00 00 00 00 00 0f 1f 
[25289.891387] sending NMI to other CPUs:
[25289.892488] NMI backtrace for cpu 2
[25289.893575] CPU: 2 PID: 350 Comm: trinity-c163 Tainted: G             L 3.18.0+ #108
[25289.895771] task: ffff8802293236b0 ti: ffff88009ab34000 task.ti: ffff88009ab34000
[25289.896886] RIP: 0010:[<ffffffff8136544a>]  [<ffffffff8136544a>] __list_add+0x1a/0xc0
[25289.898020] RSP: 0000:ffff88009ab37b08  EFLAGS: 00000092
[25289.899149] RAX: 0000000000000030 RBX: ffffea00079e7c20 RCX: 000000000000000d
[25289.900265] RDX: ffffea00079e7c60 RSI: ffff88024e296120 RDI: ffffea00079e7c20
[25289.901361] RBP: ffff88009ab37b28 R08: ffff88024e296120 R09: 00000000001e79f0
[25289.902457] R10: 0000000000000002 R11: 0000000000000002 R12: ffffea00079e7c60
[25289.903549] R13: ffff88024e296120 R14: 0000000000000002 R15: 0000000000000202
[25289.904623] FS:  00007fd3c5236740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25289.905686] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25289.906746] CR2: 000000336ef629f2 CR3: 00000002359ea000 CR4: 00000000001407e0
[25289.907809] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25289.908869] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25289.909925] Stack:
[25289.910974]  002ffe000008000c ffffea00079e7c00 ffff88024e2960f0 0000000000000000
[25289.912045]  ffff88009ab37b78 ffffffff8116d218 00000000001e79f0 ffff88024e5d4d80
[25289.913122]  ffff880238414300 ffffea00079e7cc0 ffffea00079e7c00 0000000000000000
[25289.914193] Call Trace:
[25289.915250]  [<ffffffff8116d218>] free_hot_cold_page+0x148/0x1b0
[25289.916316]  [<ffffffff8116d2ce>] free_hot_cold_page_list+0x4e/0xc0
[25289.917375]  [<ffffffff81223d4f>] ? locks_alloc_lock+0x1f/0x70
[25289.918438]  [<ffffffff81173f62>] release_pages+0x1c2/0x280
[25289.919489]  [<ffffffff81174c93>] __pagevec_release+0x43/0x60
[25289.920534]  [<ffffffff81180ff0>] shmem_undo_range+0x460/0x710
[25289.921557]  [<ffffffff811812b8>] shmem_truncate_range+0x18/0x40
[25289.922553]  [<ffffffff81181520>] shmem_setattr+0x110/0x1a0
[25289.923527]  [<ffffffff811f3361>] notify_change+0x241/0x390
[25289.924478]  [<ffffffff811d3323>] do_truncate+0x73/0xc0
[25289.925401]  [<ffffffff811d36cc>] do_sys_ftruncate.constprop.14+0x10c/0x160
[25289.926315]  [<ffffffff8135862e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[25289.927214]  [<ffffffff811d3770>] compat_SyS_ftruncate+0x10/0x20
[25289.928093]  [<ffffffff8179b9e9>] ia32_do_call+0x13/0x13
[25289.928958] Code: ff ff e9 3b ff ff ff b8 f4 ff ff ff e9 31 ff ff ff 55 48 89 e5 41 55 49 89 f5 41 54 49 89 d4 53 48 89 fb 48 83 ec 08 4c 8b 42 08 <49> 39 f0 75 2e 4d 8b 45 00 4d 39 c4 75 6c 4c 39 e3 74 42 4c 39 
[25289.930848] NMI backtrace for cpu 3
[25289.931762] CPU: 3 PID: 31454 Comm: trinity-c164 Tainted: G             L 3.18.0+ #108
[25289.933619] task: ffff88009f3f95e0 ti: ffff880229bf4000 task.ti: ffff880229bf4000
[25289.934573] RIP: 0010:[<ffffffff811753f9>]  [<ffffffff811753f9>] cancel_dirty_page+0x39/0xc0
[25289.935536] RSP: 0000:ffff880229bf7c28  EFLAGS: 00000286
[25289.936482] RAX: 00000000fffffffb RBX: ffff8800a2518d28 RCX: 0000000000001fdd
[25289.937434] RDX: ffffea0008433d00 RSI: 0000000000001000 RDI: ffffea0008433d00
[25289.938386] RBP: ffff880229bf7c48 R08: 000000000001182a R09: ffff880121d39fc8
[25289.939345] R10: ffff880229bf7bd8 R11: 0000000000000220 R12: 0000000000001000
[25289.940295] R13: ffff880229bf7d40 R14: 0000000000000000 R15: 00000000000118ee
[25289.941237] FS:  00007fd3c5236740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25289.942185] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25289.943137] CR2: 00000000014f4fd8 CR3: 000000010750d000 CR4: 00000000001407e0
[25289.944088] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25289.945036] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25289.945982] Stack:
[25289.946922]  ffff880229bf7d40 ffffea0008433d00 ffff8800a2518d28 ffff880229bf7d40
[25289.947887]  ffff880229bf7c68 ffffffff811758fe 000000000000000a ffff880229bf7cd0
[25289.948859]  ffff880229bf7df8 ffffffff81180fab ffffea00027bbe00 ffff8800a2518be0
[25289.949826] Call Trace:
[25289.950786]  [<ffffffff811758fe>] truncate_inode_page+0x4e/0x90
[25289.951752]  [<ffffffff81180fab>] shmem_undo_range+0x41b/0x710
[25289.952721]  [<ffffffff811812b8>] shmem_truncate_range+0x18/0x40
[25289.953666]  [<ffffffff81181520>] shmem_setattr+0x110/0x1a0
[25289.954589]  [<ffffffff811f3361>] notify_change+0x241/0x390
[25289.955510]  [<ffffffff811d3323>] do_truncate+0x73/0xc0
[25289.956428]  [<ffffffff811d36cc>] do_sys_ftruncate.constprop.14+0x10c/0x160
[25289.957345]  [<ffffffff8135862e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[25289.958247]  [<ffffffff811d375e>] SyS_ftruncate+0xe/0x10
[25289.959129]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25289.960005] Code: 89 f4 53 48 83 ec 08 f0 0f ba 37 04 72 14 48 83 c4 08 5b 41 5c 41 5d 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 5f 08 48 85 db 74 e3 <48> 8b 83 88 00 00 00 f6 40 20 01 75 d6 be 0b 00 00 00 e8 f0 e9 
[25289.961909] NMI backtrace for cpu 0
[25289.962833] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #108
[25289.964723] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[25289.965679] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25289.966648] RSP: 0018:ffffffff81c03e58  EFLAGS: 00000046
[25289.967596] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25289.968548] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[25289.969493] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 00000000000006c9
[25289.970443] R10: 0000000000000fc9 R11: 0000000000000400 R12: 0000000000000005
[25289.971384] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[25289.972314] FS:  0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25289.973248] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25289.974183] CR2: 0000000000000008 CR3: 0000000001c11000 CR4: 00000000001407f0
[25289.975117] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25289.976047] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25289.976979] Stack:
[25289.977905]  0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000005
[25289.978865]  ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995
[25289.979820]  00001703ad828700 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25289.980783] Call Trace:
[25289.981737]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25289.982705]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25289.983669]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25289.984636]  [<ffffffff81787ac7>] rest_init+0x87/0x90
[25289.985602]  [<ffffffff81d1e032>] start_kernel+0x483/0x4a4
[25289.986565]  [<ffffffff81d1d99f>] ? set_init_arg+0x55/0x55
[25289.987521]  [<ffffffff81d1d581>] x86_64_start_reservations+0x2a/0x2c
[25289.988488]  [<ffffffff81d1d675>] x86_64_start_kernel+0xf2/0xf6
[25289.989425] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25317.831994] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [swapper/1:0]
[25317.833213] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25317.839376] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.18.0+ #108
[25317.841869] task: ffff8802444fed60 ti: ffff880244704000 task.ti: ffff880244704000
[25317.843147] RIP: 0010:[<ffffffff8163a9b9>]  [<ffffffff8163a9b9>] cpuidle_enter_state+0x79/0x190
[25317.844445] RSP: 0018:ffff880244707e48  EFLAGS: 00000246
[25317.845727] RAX: 0000170a2fdbc2c9 RBX: ffffffff81799e0d RCX: 0000000000000019
[25317.847008] RDX: 20c49ba5e353f7cf RSI: ffff880244707fd8 RDI: ffffffff81c26f40
[25317.848288] RBP: ffff880244707e88 R08: ffff88024e24ecf0 R09: 00000000ffffffff
[25317.849570] R10: 000000000000262f R11: 0000000000aaaaaa R12: ffff880244707db8
[25317.850860] R13: 0000000000000000 R14: ffff880244704000 R15: ffff8802444fed60
[25317.852157] FS:  0000000000000000(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25317.853393] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25317.854633] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0
[25317.855885] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25317.857121] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25317.858353] Stack:
[25317.859581]  0000170a2f44020c ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25317.860841]  ffffe8ffffc41ba8 ffff880244704000 ffffffff81c9e440 ffff880244704000
[25317.862108]  ffff880244707e98 ffffffff8163ab87 ffff880244707f08 ffffffff810b9095
[25317.863330] Call Trace:
[25317.864545]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25317.865774]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25317.866999]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25317.868218] Code: c8 48 89 df ff 50 48 41 89 c5 e8 63 63 aa ff 44 8b 63 04 49 89 c7 0f 1f 44 00 00 e8 12 f8 af ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 2b 7d c0 4c 89 f8 49 c1 ff 3f 48 f7 ea b8 ff ff ff 7f 48 c1 
[25317.870820] sending NMI to other CPUs:
[25317.872055] NMI backtrace for cpu 0
[25317.873204] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #108
[25317.875524] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[25317.876676] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25317.877832] RSP: 0018:ffffffff81c03e58  EFLAGS: 00000046
[25317.878979] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25317.880133] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[25317.881280] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 0000000000000401
[25317.882431] R10: 0000000000000f4c R11: 0000000000000000 R12: 0000000000000005
[25317.883583] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000
[25317.884735] FS:  0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25317.885886] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25317.887030] CR2: 00000000006392c8 CR3: 0000000001c11000 CR4: 00000000001407f0
[25317.888181] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25317.889331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25317.890475] Stack:
[25317.891613]  0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000005
[25317.892770]  ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995
[25317.893926]  0000170a323f875b ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25317.895082] Call Trace:
[25317.896217]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25317.897361]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25317.898503]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25317.899646]  [<ffffffff81787ac7>] rest_init+0x87/0x90
[25317.900788]  [<ffffffff81d1e032>] start_kernel+0x483/0x4a4
[25317.901931]  [<ffffffff81d1d99f>] ? set_init_arg+0x55/0x55
[25317.903073]  [<ffffffff81d1d581>] x86_64_start_reservations+0x2a/0x2c
[25317.904216]  [<ffffffff81d1d675>] x86_64_start_kernel+0xf2/0xf6
[25317.905357] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25317.907755] NMI backtrace for cpu 3
[25317.908965] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25317.911350] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25317.912485] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25317.913548] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25317.914602] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25317.915637] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25317.916642] RBP: ffff88024471be38 R08: ffff88024e2cecf0 R09: 00000000ffffffff
[25317.917629] R10: 000000000000263b R11: 0000000000000400 R12: 0000000000000005
[25317.918611] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000
[25317.919581] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25317.920557] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25317.921528] CR2: 0000000000000024 CR3: 0000000001c11000 CR4: 00000000001407e0
[25317.922527] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25317.923502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25317.924477] Stack:
[25317.925437]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005
[25317.926419]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25317.927423]  0000170a323e077e ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25317.928430] Call Trace:
[25317.929409]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25317.930397]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25317.931370]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25317.932373]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25317.933346] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25317.935489] NMI backtrace for cpu 2
[25317.936503] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25317.938500] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25317.939486] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25317.940485] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25317.941483] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[25317.942501] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25317.943457] RBP: ffff88024470fe38 R08: ffff88024e28ecf0 R09: 00000000ffffffff
[25317.944395] R10: 0000000000002639 R11: 0000000000000400 R12: 0000000000000005
[25317.945329] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000
[25317.946259] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25317.947190] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25317.948124] CR2: 0000000000639058 CR3: 0000000001c11000 CR4: 00000000001407e0
[25317.949063] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25317.950009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25317.950953] Stack:
[25317.951915]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005
[25317.952893]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25317.953885]  0000170a323df4ab ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330
[25317.954862] Call Trace:
[25317.955831]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25317.956807]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25317.957776]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25317.958741]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25317.959702] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25345.815706] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c103:12391]
[25345.816962] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25345.823837] CPU: 1 PID: 12391 Comm: trinity-c103 Tainted: G             L 3.18.0+ #108
[25345.826600] task: ffff88022911abc0 ti: ffff880143340000 task.ti: ffff880143340000
[25345.827931] RIP: 0010:[<ffffffff8178ecca>]  [<ffffffff8178ecca>] __slab_alloc+0x4e5/0x53b
[25345.829277] RSP: 0018:ffff880143343ca0  EFLAGS: 00000246
[25345.830607] RAX: 0000000000000001 RBX: ffff880143343c60 RCX: 000000018020001e
[25345.831928] RDX: 0000000000000020 RSI: ffffea00076ef700 RDI: ffff880245825680
[25345.833232] RBP: ffff880143343d70 R08: ffff8801dbbdc000 R09: 0000000000000000
[25345.834528] R10: 0000000000000082 R11: 00003ffffffff000 R12: 00000000000000bb
[25345.835836] R13: ffffffff810133af R14: ffff880143343c20 R15: 000000010020001d
[25345.837084] FS:  00007f5c9bc2f740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25345.838337] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25345.839599] CR2: 00000000020d4968 CR3: 00000002299b4000 CR4: 00000000001407e0
[25345.840857] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25345.842117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25345.843379] Stack:
[25345.844620]  01ffffff00000000 ffff88022ea27288 00007f5c97aa4fff 00007f5c97a24000
[25345.845915]  0000000000000000 ffff8801dbbdc600 0000000227852b98 ffffffff810750f5
[25345.847139]  ffffffff00000000 0000000000000246 00007f5c97aa5000 00007f5c97aa4fff
[25345.848385] Call Trace:
[25345.849599]  [<ffffffff810750f5>] ? copy_process.part.29+0x12a5/0x1970
[25345.850835]  [<ffffffff810750f5>] ? copy_process.part.29+0x12a5/0x1970
[25345.852052]  [<ffffffff811c012b>] kmem_cache_alloc+0x1bb/0x1e0
[25345.853258]  [<ffffffff810750f5>] copy_process.part.29+0x12a5/0x1970
[25345.854460]  [<ffffffff811d79ad>] ? get_empty_filp+0xdd/0x1f0
[25345.855664]  [<ffffffff81075981>] do_fork+0xe1/0x3d0
[25345.856837]  [<ffffffff81012047>] ? syscall_trace_enter_phase2+0xa7/0x1e0
[25345.858003]  [<ffffffff81075cf6>] SyS_clone+0x16/0x20
[25345.859172]  [<ffffffff81799609>] stub_clone+0x69/0x90
[25345.860312]  [<ffffffff817994c4>] ? tracesys_phase2+0xd4/0xd9
[25345.861429] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 07 b4 9a ff 4c 89 e0 eb 0f e8 fd b4 9a ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 75 c8 65 48 33 34 25 28 00 00 00 74 32 e8 ff 79 
[25345.863833] sending NMI to other CPUs:
[25345.864970] NMI backtrace for cpu 2
[25345.866051] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.18.0+ #108
[25345.868240] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000
[25345.869350] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25345.870475] RSP: 0018:ffff88024470fe08  EFLAGS: 00000046
[25345.871594] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[25345.872721] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002
[25345.873846] RBP: ffff88024470fe38 R08: 0000000000000018 R09: 0000000000001033
[25345.874967] R10: 000000000000331d R11: 0000000000000002 R12: 0000000000000002
[25345.876083] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88024470c000
[25345.877191] FS:  0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25345.878306] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25345.879419] CR2: 0000003370219050 CR3: 0000000229371000 CR4: 00000000001407e0
[25345.880537] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25345.881648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25345.882753] Stack:
[25345.883844]  000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000002
[25345.884958]  ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995
[25345.886071]  00001710b7b5ad88 ffffffff81c9e508 ffffffff81c9e440 ffffffff81cfe330
[25345.887189] Call Trace:
[25345.888301]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25345.889424]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25345.890517]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25345.891590]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25345.892652] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25345.894882] NMI backtrace for cpu 0
[25345.895925] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L 3.18.0+ #108
[25345.897992] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000
[25345.899014] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25345.900033] RSP: 0018:ffffffff81c03e58  EFLAGS: 00000046
[25345.901026] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[25345.902020] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000
[25345.903001] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 000000000000032a
[25345.903982] R10: 00000000000003f6 R11: 00000000000fff9d R12: 0000000000000001
[25345.904960] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff81c00000
[25345.905936] FS:  0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25345.906947] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25345.907962] CR2: 0000000000000001 CR3: 00000000973c9000 CR4: 00000000001407f0
[25345.909015] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25345.910072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25345.911128] Stack:
[25345.912132]  0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000001
[25345.913130]  ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995
[25345.914121]  00001710b7b5d674 ffffffff81c9e4b0 ffffffff81c9e440 ffffffff81cfe330
[25345.915112] Call Trace:
[25345.916097]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25345.917082]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25345.918056]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25345.919028]  [<ffffffff81787ac7>] rest_init+0x87/0x90
[25345.920015]  [<ffffffff81d1e032>] start_kernel+0x483/0x4a4
[25345.920988]  [<ffffffff81d1d99f>] ? set_init_arg+0x55/0x55
[25345.921960]  [<ffffffff81d1d581>] x86_64_start_reservations+0x2a/0x2c
[25345.922929]  [<ffffffff81d1d675>] x86_64_start_kernel+0xf2/0xf6
[25345.923900] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25345.926016] NMI backtrace for cpu 3
[25345.927071] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.18.0+ #108
[25345.929211] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000
[25345.930264] RIP: 0010:[<ffffffff813b84eb>]  [<ffffffff813b84eb>] intel_idle+0xdb/0x180
[25345.931337] RSP: 0018:ffff88024471be08  EFLAGS: 00000046
[25345.932420] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[25345.933563] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003
[25345.934692] RBP: ffff88024471be38 R08: 0000000000000018 R09: 0000000000000d4b
[25345.935804] R10: 0000000000002da4 R11: 0000000000000002 R12: 0000000000000002
[25345.936797] R13: 0000000000000001 R14: 0000000000000001 R15: ffff880244718000
[25345.937778] FS:  0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25345.938766] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25345.939752] CR2: 0000000001e81930 CR3: 00000001f9d90000 CR4: 00000000001407e0
[25345.940747] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25345.941745] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25345.942743] Stack:
[25345.943728]  000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000002
[25345.944753]  ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995
[25345.945764]  00001710b7b5d25c ffffffff81c9e508 ffffffff81c9e440 ffffffff81cfe330
[25345.946777] Call Trace:
[25345.947789]  [<ffffffff8163a995>] cpuidle_enter_state+0x55/0x190
[25345.948807]  [<ffffffff8163ab87>] cpuidle_enter+0x17/0x20
[25345.949812]  [<ffffffff810b9095>] cpu_startup_entry+0x355/0x410
[25345.950811]  [<ffffffff8102fd0a>] start_secondary+0x1aa/0x230
[25345.951826] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[25373.799486] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c17:29947]
[25373.800608] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc
[25373.806511] CPU: 1 PID: 29947 Comm: trinity-c17 Tainted: G             L 3.18.0+ #108
[25373.808876] task: ffff88022908abc0 ti: ffff8801bf164000 task.ti: ffff8801bf164000
[25373.810057] RIP: 0010:[<ffffffff810933ee>]  [<ffffffff810933ee>] find_get_pid+0x1e/0x30
[25373.811282] RSP: 0018:ffff8801bf167ee8  EFLAGS: 00000202
[25373.812535] RAX: ffff8801bf33a140 RBX: ffffffff810850e0 RCX: ffff8801bf33a170
[25373.813781] RDX: 0000000000000030 RSI: ffffffff81c44840 RDI: 0000000000007d64
[25373.815031] RBP: ffff8801bf167ee8 R08: 00007f0b34598740 R09: 0000000000000000
[25373.816283] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[25373.817515] R13: ffff8801bf167eb0 R14: ffff8801bf33a140 R15: ffff8801bf167e68
[25373.818738] FS:  00007f0b34598740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000
[25373.819941] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25373.821149] CR2: 000000336f1b7740 CR3: 0000000242a70000 CR4: 00000000001407e0
[25373.822379] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25373.823593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25373.824799] Stack:
[25373.825983]  ffff8801bf167f78 ffffffff81079935 0000000f00000000 ffff8801bf33a140
[25373.827203]  0000000000000000 00007fff74741110 0000000000000000 0000000000000000
[25373.828434]  ffff88022908abc0 ffffffff8135862e 0000000000000246 0000000000000000
[25373.829660] Call Trace:
[25373.830833]  [<ffffffff81079935>] SyS_wait4+0x55/0x110
[25373.832014]  [<ffffffff8135862e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[25373.833206]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25373.834409] Code: 80 00 00 00 00 31 c0 eb eb 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 65 ff 04 25 60 a8 00 00 e8 ca fe ff ff 48 85 c0 74 03 f0 ff 00 <65> ff 0c 25 60 a8 00 00 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 
[25373.837037] sending NMI to other CPUs:
[25373.838287] NMI backtrace for cpu 0
[25373.839429] CPU: 0 PID: 32111 Comm: modprobe Tainted: G             L 3.18.0+ #108
[25373.841745] task: ffff8802293be270 ti: ffff880103704000 task.ti: ffff880103704000
[25373.842890] RIP: 0010:[<ffffffff811f70b4>]  [<ffffffff811f70b4>] __lookup_mnt+0x64/0x80
[25373.844045] RSP: 0018:ffff880103707bb8  EFLAGS: 00000286
[25373.845190] RAX: ffff880241670780 RBX: ffff880103707da8 RCX: 000000000000000e
[25373.846327] RDX: ffff880244b56300 RSI: ffff880241b03ae8 RDI: ffff8802416707a0
[25373.847446] RBP: ffff880103707bb8 R08: 70646f6d2f6e7572 R09: ffff880103707bdc
[25373.848558] R10: ffff88014331e921 R11: 0000000000000003 R12: ffff880241b03ae8
[25373.849665] R13: ffff880103707c60 R14: ffff880103707c50 R15: ffff8802416707a0
[25373.850770] FS:  00007fdddcfb2740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000
[25373.851886] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25373.853000] CR2: 00000000016211d8 CR3: 00000001c6569000 CR4: 00000000001407f0
[25373.854123] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25373.855252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25373.856371] Stack:
[25373.857476]  ffff880103707c18 ffffffff811e082a 0000000000000081 ffff880200000001
[25373.858600]  0000000203707be8 00000000d85f4923 ffff880103707c08 ffff88014331e925
[25373.859730]  ffff8802293be270 6f636e6c2e6d7471 00000000ffffff9c ffff880103707da8
[25373.860865] Call Trace:
[25373.861983]  [<ffffffff811e082a>] lookup_fast+0xca/0x2e0
[25373.863101]  [<ffffffff811e28a5>] link_path_walk+0x1a5/0x850
[25373.864213]  [<ffffffff811e3424>] path_lookupat+0x64/0x770
[25373.865321]  [<ffffffff811c006f>] ? kmem_cache_alloc+0xff/0x1e0
[25373.866424]  [<ffffffff811e5dff>] ? getname_flags+0x4f/0x1a0
[25373.867522]  [<ffffffff811e3b5b>] filename_lookup+0x2b/0xc0
[25373.868622]  [<ffffffff811e6e23>] user_path_at_empty+0x63/0xc0
[25373.869721]  [<ffffffff811d7600>] ? fput+0xa0/0xa0
[25373.870814]  [<ffffffff812577da>] ? ext4_release_dir+0x2a/0x40
[25373.871890]  [<ffffffff811e6e91>] user_path_at+0x11/0x20
[25373.872942]  [<ffffffff811daa63>] vfs_fstatat+0x63/0xc0
[25373.873985]  [<ffffffff811d77e8>] ? __fput+0x188/0x1f0
[25373.875004]  [<ffffffff811daf8e>] SYSC_newstat+0x2e/0x60
[25373.876000]  [<ffffffff810941b4>] ? task_work_run+0xc4/0xe0
[25373.876971]  [<ffffffff81002bb9>] ? do_notify_resume+0x59/0x80
[25373.877921]  [<ffffffff8135862e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[25373.878846]  [<ffffffff811db26e>] SyS_newstat+0xe/0x10
[25373.879757]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25373.880651] Code: c2 48 8b 10 31 c0 48 85 d2 75 1c 5d c3 66 2e 0f 1f 84 00 00 00 00 00 48 39 72 18 74 23 48 8b 12 48 85 d2 74 17 66 90 48 8b 42 10 <48> 83 c0 20 48 39 c7 74 e3 48 8b 12 48 85 d2 75 eb 31 c0 5d c3 
[25373.882565] NMI backtrace for cpu 2
[25373.883509] CPU: 2 PID: 31661 Comm: trinity-c90 Tainted: G             L 3.18.0+ #108
[25373.885431] task: ffff88022e86a0d0 ti: ffff880039fc0000 task.ti: ffff880039fc0000
[25373.886415] RIP: 0010:[<ffffffff8118f1c4>]  [<ffffffff8118f1c4>] iov_iter_fault_in_readable+0x64/0x80
[25373.887422] RSP: 0018:ffff880039fc3c18  EFLAGS: 00000206
[25373.888422] RAX: 0000000000000000 RBX: 000000000468e000 RCX: 0000000000003001
[25373.889432] RDX: 00007f0b31a7ea00 RSI: 0000000000001000 RDI: 000000000026fa00
[25373.890454] RBP: ffff880039fc3c28 R08: ffff8800978c0cf0 R09: 00000000008b89e2
[25373.891462] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000
[25373.892470] R13: ffff880039fc3d60 R14: 0000000000001000 R15: ffff88022e9718b0
[25373.893546] FS:  00007f0b34598740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000
[25373.894591] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25373.895602] CR2: 00007f0b33620294 CR3: 000000009f3b0000 CR4: 00000000001407e0
[25373.896666] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25373.897705] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25373.898733] Stack:
[25373.899757]  0000000000001000 00ff88022e9718b0 ffff880039fc3cc8 ffffffff811652b8
[25373.900776]  0000000054940ba3 0000000000001000 ffff88022e86a0d0 0000000000000a01
[25373.901816]  000000000468dffa 0000000000000000 ffff880071a82580 ffffffff81828380
[25373.902857] Call Trace:
[25373.903836]  [<ffffffff811652b8>] generic_perform_write+0xa8/0x1f0
[25373.904819]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25373.905807]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25373.906776]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25373.907731]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.908671]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25373.909602]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25373.910526]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.911458]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.912427]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25373.913397]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25373.914320]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25373.915231]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25373.916128]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25373.917017] Code: 10 0f 1f 00 40 8a 39 0f 1f 00 85 d2 40 88 7d ff 75 2d 48 63 f6 48 8d 54 31 ff 48 31 d1 48 f7 c1 00 f0 ff ff 74 0f 0f 1f 00 8a 12 <0f> 1f 00 88 55 ff 0f b6 55 ff c9 f3 c3 0f 1f 80 00 00 00 00 c9 
[25373.918994] NMI backtrace for cpu 3
[25373.919961] CPU: 3 PID: 29575 Comm: trinity-c135 Tainted: G             L 3.18.0+ #108
[25373.921902] task: ffff88022d2fa0d0 ti: ffff88009aa0c000 task.ti: ffff88009aa0c000
[25373.922890] RIP: 0010:[<ffffffff81165a08>]  [<ffffffff81165a08>] unlock_page+0x18/0x90
[25373.923900] RSP: 0018:ffff88009aa0fbf8  EFLAGS: 00000202
[25373.924888] RAX: ffffea0001505d00 RBX: ffffea0001505d00 RCX: 007c8fc09347b000
[25373.925868] RDX: 0000000000000001 RSI: ffff88022e973770 RDI: ffffea0001505d00
[25373.926844] RBP: ffff88009aa0fbf8 R08: 0000000000001000 R09: ffffea0001505d00
[25373.927819] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000
[25373.928790] R13: ffff88009aa0fd60 R14: 0000000000000000 R15: ffff88022e973770
[25373.929770] FS:  00007f0b34598740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000
[25373.930759] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25373.931746] CR2: 00007f0b3320417d CR3: 0000000039ddb000 CR4: 00000000001407e0
[25373.932739] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000
[25373.933800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[25373.934848] Stack:
[25373.935836]  ffff88009aa0fc28 ffffffff8117f380 007c8fc09347a000 0000000000001000
[25373.936902]  ffff88009aa0fd60 0000000000000000 ffff88009aa0fcc8 ffffffff8116532a
[25373.938049]  ffff88009aa0fd68 0000000000001000 ffff88022d2fa0d0 0000000045806f40
[25373.939153] Call Trace:
[25373.940209]  [<ffffffff8117f380>] shmem_write_end+0x40/0xf0
[25373.941247]  [<ffffffff8116532a>] generic_perform_write+0x11a/0x1f0
[25373.942283]  [<ffffffff8116789f>] __generic_file_write_iter+0x15f/0x350
[25373.943317]  [<ffffffff811d5140>] ? new_sync_read+0xd0/0xd0
[25373.944323]  [<ffffffff81167acd>] generic_file_write_iter+0x3d/0xb0
[25373.945306]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.946298]  [<ffffffff811d5288>] do_iter_readv_writev+0x78/0xc0
[25373.947279]  [<ffffffff811d6a18>] do_readv_writev+0xd8/0x2a0
[25373.948237]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.949185]  [<ffffffff81167a90>] ? __generic_file_write_iter+0x350/0x350
[25373.950126]  [<ffffffff810dac62>] ? __hrtimer_start_range_ns+0x252/0x380
[25373.951061]  [<ffffffff810dadc8>] ? hrtimer_start+0x18/0x20
[25373.951985]  [<ffffffff811d6c69>] vfs_writev+0x39/0x50
[25373.952958]  [<ffffffff811d6dc9>] SyS_writev+0x59/0xf0
[25373.953919]  [<ffffffff817992d2>] system_call_fastpath+0x12/0x17
[25373.954946] Code: 41 f6 c4 01 0f 84 7d ff ff ff eb eb 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 f8 48 8b 17 48 89 e5 83 e2 01 74 60 f0 80 27 fe <48> be 01 00 fc ff ff ff 37 9e 48 8b 17 48 0f af f7 48 89 d1 48 
[25373.957052] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 118.758 msecs

Not quite the same, the spinlocks are gone.

-chris

Ok, so it's not spinlock debugging.

There are some interesting patters here, once again. Lookie:

     RIP: 0010:   generic_exec_single+0xea/0x1b0
     RIP: 0010:   generic_exec_single+0xee/0x1b0
     RIP: 0010:   generic_exec_single+0xea/0x1b0
     RIP: 0010:   generic_exec_single+0xea/0x1b0
     RIP: 0010:   generic_exec_single+0xee/0x1b0
     RIP: 0010:   generic_exec_single+0xee/0x1b0
     RIP: 0010:   generic_exec_single+0xea/0x1b0
     sched: RT throttling activated
     RIP: 0010:   __slab_alloc+0x4e5/0x53b
     RIP: 0010:   copy_user_enhanced_fast_string+0x5/0x10
     RIP: 0033:   0x412fc8
     RIP: 0010:   clear_page_c_e+0x7/0x10
     RIP: 0010:   cpuidle_enter_state+0x79/0x190
     RIP: 0010:   __slab_alloc+0x4e5/0x53b
     RIP: 0010:   find_get_pid+0x1e/0x30

so now copy_page_range() is gone, but every single case before the RT throttling is activated is that zap_page_range() followed by the TLB invalidate that we saw last time.

And after RT throttling, it's random (not even always trinity), but that's probably because the watchdog thread doesn't run reliably any more.

Another pattern in this one: it's CPU#1 that is stuck. Every single time. There are stack traces from other CPU's, but they are all the NMI broadcast *due* to the soft lockup on CPU#1.

And that is true even after the RT throttling thing.

And let's take another look at your previous one (with lock debugging, but that config detail is clearly not that important - it hasn't really changed anything major except make that lock very visible):

     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   generic_exec_single+0xee/0x1b0
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     RIP: 0010:   lock_acquire+0xb4/0x120
     sched: RT throttling activated
     RIP: 0010:   shmem_write_end+0x65/0xf0
     RIP: 0010:   _raw_spin_unlock_irqrestore+0x38/0x60
     RIP: 0010:   copy_user_enhanced_fast_string+0x5/0x10
     RIP: 0010:   copy_user_enhanced_fast_string+0x5/0x10
     RIP: 0010:   __slab_alloc+0x52f/0x58f
     RIP: 0010:   map_id_up+0x9/0x80
     RIP: 0010:   cpuidle_enter_state+0x79/0x190
     RIP: 0010:   unmap_single_vma+0x7d9/0x900
     RIP: 0010:   cpuidle_enter_state+0x79/0x190

same pattern: after the RT throttling, it's random because the watchdog is no longer sane, before that it's always reliably either the lock_acquire as part of copy_page_range(), or it's that TLB flush as part of zap_page_range().

And the CPU patterns are interesting too:

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [trinity-c154:22823] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c195:20128]

CPU#1 again, *except* for the third lockup, which happens to match the exact same pattern of copy_page_range() (CPU#1) vs zap_page_range() (CPU#2).

It's also the case that before the RT throttling, it really does seem to be one particular thread (ie trinity-c195.20128 does the copy_page_range on your previous one, and in the newer one it's trinity-c205:636 that does the zap_page_range(). So those threads really seem to be stuck for real. The fact that they *eventually* go away at all is interesting in itself.

And that "generic_exec_single()" place where it is stuck is the instruction after the "pause" (aka "cpu_relax()") in the final "csd_lock_wait()" once more. So it's waiting on some CPU to pick up the IPI, and that never happens.

Here's another pattern. In your latest thing, every single time that CPU1 is waiting for some other CPU to pick up the IPI, we have CPU0 doing this:

[24998.060963] NMI backtrace for cpu 0
[24998.061989] CPU: 0 PID: 2940 Comm: trinity-c150 Not tainted 3.18.0+ #108
[24998.064073] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti:
ffff880197e0c000
[24998.065137] RIP: 0010:[<ffffffff8103e006>]  [<ffffffff8103e006>]
read_hpet+0x16/0x20
[24998.083577]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[24998.084450]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[24998.085315]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[24998.086173]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[24998.087025]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[24998.087877]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[24998.088732]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[24998.089583]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[24998.090435]  <EOI>
[24998.091279]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[24998.092118]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[24998.092951]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[24998.093779]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70

and I think that's the smoking gun. The reason CPU0 isn't picking up any IPI's is because it is in some endless loop around read_hpet().

There is even time information in the register dump:

RAX: 0000000061fece8a RBX: 0000000000510792 RCX: 0000000000000000
RAX: 0000000079e588fc RBX: 0000000000511d6e RCX: 0000000000000000
RAX: 0000000091ca7f65 RBX: 0000000000513346 RCX: 0000000000000000
RAX: 00000000a9afbd0d RBX: 000000000051491e RCX: 0000000000000000
RAX: 00000000cbd1340c RBX: 000000000051684a RCX: 0000000000000000
RAX: 00000000fb9d303f RBX: 00000000005193fc RCX: 0000000000000000
RAX: 000000002b67efe4 RBX: 000000000051c224 RCX: 0000000000000004

That RAX value is the value we just read from the HPET, and RBX seems to be monotonically increasing too, so it's likely the sequence counter in ktime_get().

So it's not stuck *inside* read_hpet(), and it's almost certainly not the loop over the sequence counter in ktime_get() either (it's not increasing *that* quickly). But some basically infinite __run_hrtimer thing or something?

In your earlier trace (with spinlock debugging), the softlockup detection was in lock_acquire for copy_page_range(), but CPU2 was always in that "generic_exec_single" due to a TLB flush from that zap_page_range thing again. But there are no timer traces from that one, so I dunno.

Anyway, I do think we're getting somewhere. Your traces are interesting and have real patterns in them. Which is very different from the mess it used to be.

Linus

So if we want to shoot that RT throttling in the head you can do:

echo -1 > /proc/sys/kernel/sched_rt_runtime_us

That should completely disable that stuff, of course at that point a run-away RR/FIFO thread will hog your cpu indefinitely.

Ahh, and that's because the TLB flushing is done under the page table lock these days (see commit 1cf35d47712d: "mm: split 'tlb_flush_mmu()' into tlb flushing and memory freeing parts").

Which means that if the TLB flushing gets stuck on CPU#2, CPU#1 that is trying to get the page table lock will be locked up too.

So this is all very consistent, actually. The underlying bug in both cases seems to be that the IPI for the TLB flushing doesn't happen for some reason.

In your second trace, that's explained by the fact that CPU0 is in a timer interrupt. In the first trace with spinlock debugging, no such obvious explanation exists. It could be that an IPI has gotten lost for some reason.

However, the first trace does have this:

NMI backtrace for cpu 3
INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too
long to run: 66.180 msecs

   CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.18.0+ #107
   RIP: 0010:   intel_idle+0xdb/0x180
   Code: 31 d2 65 48 8b 34 ...
   INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too

long to run: 95.053 msecs

so something odd is happening (probably on CPU3). It took a long time to react to the NMI IPI too.

So there's definitely something screwy going on here in IPI-land.

I do note that we depend on the "new mwait" semantics where we do mwait with interrupts disabled and a non-zero RCX value. Are there possibly even any known CPU errata in that area? Not that it sounds likely, but still..

Linus

Remind me what CPU you have in that machine again? The %rax value for the mwait cases in question seems to be 0x32, which is either C7s-HSW or C7s-BDW, and in both cases has the "TLB flushed" flag set.

I'm pretty sure you have a Haswell, I'm just checking. Which model? I'm assuming it's family 6, model 60, stepping 3? I found you mentioning i5-4670T in a perf thread.. That the one?

Anyway, I don't actually believe in any CPU bugs, but you could try "intel_idle.max_cstate=0" and see if that makes any difference, for example.

Or perhaps just "intel_idle.max_cstate=1", which leaves intel_idle active, but gets rid of the deeper sleep states (that incidentally also play games with leave_mm() etc)

Linus

Yep.

vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i5-4670T CPU @ 2.30GHz
stepping : 3
microcode : 0x1a

So I'm leaving Red Hat on Tuesday, and can realistically only do one more experiment over the weekend before I give them this box back.

Right now I'm doing Chris' idea of "turn debugging back on, and try without serial console". Shall I try your suggestion on top of that ?

I *hate* for this to be "the one that got away", but we've at least gotten some good mileage out of this bug in the last two months. Who knows, maybe I'll find some new hardware that will exhibit the same behaviour in the new year.

Dave

Might as well. I doubt it really will make any difference, but I also don't think it will interact badly in any way, so..

Linus

It locked up. It's not even responding to icmp. It might be Monday before I can see what's on the console.

Dave

Really interesting.

So, we're calling __run_hrtimer in a loop:

                while ((node = timerqueue_getnext(&base->active))) {
				...
				__run_hrtimer(timer, &basenow);
				...
		}

The easy question is how often does trinity call nanosleep?

Looking at __run_hrtimer(), it drops the lock and runs the function and then takes the lock again, maybe enqueueing us again right away.

timer->state is supposed to protect us from other CPUs jumping in and doing something else with the timer, but it feels racey wrt remove_hrtimer(). Something like this, but I'm not sure how often __hrtimer_start_range_ns gets called

CPU 0 CPU 1
__run_hrtimer()
timer->state = HRTIMER_STATE_CALLBACK
removed from list
unlock cpu_base->lock
restrt = fn(timer)
__hrtimer_start_range_ns()
base = lock_hrtimer_base()
ret = remove_hrtimer()
finds timer->state = HRTIMER_STATE_CALLBACK does nothing
new_base = switch_hrtimer_base()
now we're on a different base, different lock
lock(cpu_base->lock)
enqueue the timer
enqueue the timer

It shouldn't call it directly. (syscalls/nanosleep.c)

/*
* SYSCALL_DEFINE2(nanosleep, struct timespec __user *, rqtp, struct timespec __user *, rmtp) */
#include "sanitise.h"

struct syscallentry syscall_nanosleep = {
.name = "nanosleep",
.num_args = 2,
.arg1name = "rqtp",
.arg1type = ARG_ADDRESS,
.arg2name = "rmtp",
.arg2type = ARG_ADDRESS,
.flags = AVOID_SYSCALL, // Boring. Can cause long sleeps.
};

That last line being the key one. We used to do it, but it's well.. boring. We could do something smarter, but given it's never triggered anything interesting in the past, focussing runtime on the more interesting syscalls seems to have been more fruitful.

Dave

But at the very end this would be detected by the runtime check of the hrtimer interrupt, which does not trigger. And it would trigger at some point as ALL cpus including CPU0 in that trace dump make progress.

Thanks,

tglx

I'll admit that at some point we should be hitting one of the WARN or BUG_ON, but it's possible to thread that needle and corrupt the timer list, without hitting a warning (CPU 1 in my example has to enqueue last). Once the rbtree is hosed, it can go forever. Probably not the bug we're looking for, but still suspect in general.

-chris

I surely have a close look at that, but in that case we get out of that state later on and I doubt that we have

A) a corruption of the rbtree
B) a self healing of the rbtree afterwards

I doubt it, but who knows.

Though even if A & B would happen we would still get the 'hrtimer interrupt took a gazillion of seconds' warning because CPU0 definitely leaves the timer interrupt at some point otherwise we would not see backtraces from usb, userspace and idle later on.

Thanks,

tglx

Here is the full diff of the first and the second splat for CPU0

task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000

 read_hpet+0x16/0x20
 RSP: 0018:ffff88024e203e38  EFLAGS: 00000046
-RAX: 0000000061fece8a RBX: 0000000000510792 RCX: 0000000000000000
+RAX: 0000000079e588fc RBX: 0000000000511d6e RCX: 0000000000000000

RAX:
(0x0000000079e588fc - 0x0000000061fece8a) / 14.318180e6 ~= 28.0061

So HPET @14.3MHz progressed by 28 seconds, which matches the splat delta between the first and the second one.

25026.001132 - 24998.017355 = 27.9838

RBX:
0x0000000000511d6e - 0x0000000000510792 = 5596

The sequence counter increments by 2 per tick. So:

28 / (5596/2) ~= 0.0025 s

==> HZ = 400

The sequence counter is even, so ktime_get() will succeed.

RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40
RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f
-R10: 0000000000000526 R11: 000000000000000f R12: 000016bf99600917 +R10: 0000000000000526 R11: 000000000000000f R12: 000016c61e4e2117

R12:
0x000016c61e4e2117 - 0x000016bf99600917 = 2.8e+10

That's the nanoseconds timestamp: 2.8e10/1e9 = 28

Now that all looks correct. So there is something else going on. After staring some more at it, I think we are looking at it from the wrong angle.

The watchdog always detects CPU1 as stuck and we got completely fixated on the csd_wait() in the stack trace on CPU1. Now we have stack traces which show a different picture, i.e. CPU1 makes progress after a gazillion of seconds.

I think we really need to look at CPU1 itself.

AFAICT all these 'stuck' events happen in fully interruptible context. So an undetected interrupt storm can cause that.

We only detect interrupt storms for unhandled interrupts, but for those where the handler returns IRQ_HANDLED, we just count them.

For directly handled vectors we do not even have a detection mechanism at all.

That also might explain the RT throttling. If that storm hits a high prio task, the throttler will trigger.

Just a theory, but worth to explore, IMO.

So adding a dump of the total interrupt counts to the watchdog trace might give us some insight.

Debug patch below.

Thanks,

tglx
---

index 501baa9ac1be..2021662663c7 100644
@@ -315,11 +315,21 @@ asmlinkage __visible void do_softirq(void)
 	local_irq_restore(flags);
 }
 
+static DEFINE_PER_CPU(unsigned long, irqcnt);
+
+void show_irqcnt(int cpu)
+{
+	pr_emerg("CPU#%d: IRQ %lu NMI %u\n", cpu, this_cpu_read(irqcnt),
+		 this_cpu_read(irq_stat.__nmi_count));
+}
+
 /*
  * Enter an interrupt context.
  */
 void irq_enter(void)
 {
+	this_cpu_inc(irqcnt);
+
 	rcu_irq_enter();
 	if (is_idle_task(current) && !in_interrupt()) {
 		/*

index 70bf11815f84..f505cc58d354 100644
@@ -306,6 +306,8 @@ static void watchdog_interrupt_count(void)
 static int watchdog_nmi_enable(unsigned int cpu);
 static void watchdog_nmi_disable(unsigned int cpu);
 
+extern void show_irqcnt(int cpu);
+
 /* watchdog kicker functions */
 static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 {

@@ -388,6 +390,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 			smp_processor_id(), duration,
 			current->comm, task_pid_nr(current));
 		__this_cpu_write(softlockup_task_ptr_saved, current);
+		show_irqcnt(smp_processor_id());
 		print_modules();
 		print_irqtrace_events(current);
 		if (regs)

.. but that doesn't explain why CPU0 ends up always being at that *exact* same instruction in the NMI backtrace.

While a fairly tight loop, together with "mmio read is very expensive and synchronizing" would explain it. An MMIO read can easily be as expensive as several thousand instructions.

Not so fast. Take another look at CPU0.

[24998.083577]  [<ffffffff810e0d3e>] ktime_get+0x3e/0xa0
[24998.084450]  [<ffffffff810e9cd3>] tick_sched_timer+0x23/0x160
[24998.085315]  [<ffffffff810daf96>] __run_hrtimer+0x76/0x1f0
[24998.086173]  [<ffffffff810e9cb0>] ? tick_init_highres+0x20/0x20
[24998.087025]  [<ffffffff810db2e7>] hrtimer_interrupt+0x107/0x260
[24998.087877]  [<ffffffff81031a4b>] local_apic_timer_interrupt+0x3b/0x70
[24998.088732]  [<ffffffff8179bca5>] smp_apic_timer_interrupt+0x45/0x60
[24998.089583]  [<ffffffff8179a0df>] apic_timer_interrupt+0x6f/0x80
[24998.090435]  <EOI>
[24998.091279]  [<ffffffff810da66e>] ? __remove_hrtimer+0x4e/0xa0
[24998.092118]  [<ffffffff812c7c7a>] ? ipcget+0x8a/0x1e0
[24998.092951]  [<ffffffff812c7c6c>] ? ipcget+0x7c/0x1e0
[24998.093779]  [<ffffffff812c8d6d>] SyS_msgget+0x4d/0x70

Really. None of that changed. NONE. The likelihood that we hit the exact same instruction every time? Over a timefraem of more than a minute?

The only way I see that happening is (a) NMI is completely buggered, and the backtrace is just random crap that is always the same. Or (b) it's really a fairly tight loop.

The fact that you had a hrtimer interrupt happen in the *middle* of __remove_hrtimer() is really another fairly strong hint. That smells like "__remove_hrtimer() has a race with hrtimer interrupts".

And that race results in a basically endless loop (which perhaps ends when the HRtimer overflows, in what, a few minutes?)

I really don't think you should look at CPU1. Not when CPU0 has such an interesting pattern that you dismissed just because the HPET is making progress.

Linus