BUG: soft lockup - CPU#6 stuck for 22s! [httpd2-event:15597]

DISCUSSION Cristian Rodríguez 3 participants 10 messages Aug 25, 2012 – Aug 30, 2012

Subsystems (3)

Networking Network drivers TCP

lore

Share & tools

lore

Cristian Rodríguez

author

2012-08-25 00:51 UTC #0 lore raw

Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..

Quoted 1 line

Aug 23 11:51:05 ex6 kernel: [55570.280878] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_co

nntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_
NFQUEUE xt_NFLOG nfnetlink_

Quoted 1 line

Aug 23 11:51:05 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet cachefiles fscache eeepc_wmi asus_wmi sparse_keymap acpi_cpufreq mperf rfkill pci_hotplug e1000e coretemp wmi kvm_intel i2c_i801 kvm mei pcspkr joydev ghash_clmulni_intel microcode crc32c_intel edd aesni_intel ablk_helper cryptd aes_x86_64 autofs4 hid_generic usbhid i915 xhci_hcd drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore usb_common video button scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh fan processor thermal

thermal_sys megaraid_sas

Quoted 22 lines

Aug 23 11:51:05 ex6 kernel: [55570.280987] CPU 6 
Aug 23 11:51:05 ex6 kernel: [55570.280991] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:05 ex6 kernel: [55570.280993] RIP: 0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281002] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:05 ex6 kernel: [55570.281004] RAX: 0000000000000014 RBX: ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:05 ex6 kernel: [55570.281006] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:05 ex6 kernel: [55570.281007] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:05 ex6 kernel: [55570.281009] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:05 ex6 kernel: [55570.281011] R13: ffff880407ba4cc0 R14: 0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:05 ex6 kernel: [55570.281013] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:05 ex6 kernel: [55570.281017] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:05 ex6 kernel: [55570.281019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:05 ex6 kernel: [55570.281023] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:05 ex6 kernel: [55570.281024] Stack:
Aug 23 11:51:05 ex6 kernel: [55570.281025]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:05 ex6 kernel: [55570.281029]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:05 ex6 kernel: [55570.281033]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:05 ex6 kernel: [55570.281037] Call Trace:
Aug 23 11:51:05 ex6 kernel: [55570.281046]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281052]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130

Quoted 8 lines

Aug 23 11:51:05 ex6 kernel: [55570.281058]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:05 ex6 kernel: [55570.281066]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:05 ex6 kernel: [55570.281075]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:05 ex6 kernel: [55570.281076] Code: 66 39 d0 75 f6 c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 
Aug 23 11:51:24 ex6 cachefilesd[5919]: Refilling cull table
Aug 23 11:51:24 ex6 cachefilesd[5919]: Scan complete
Aug 23 11:51:33 ex6 kernel: [55598.253247] BUG: soft lockup - CPU#6 stuck for 22s! [httpd2-event:15597]
Aug 23 11:51:33 ex6 kernel: [55598.253708] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_co

Quoted 1 line

Aug 23 11:51:33 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet cachefiles fscache eeepc_wmi asus_wmi sparse_keymap acpi_cpufreq mperf rfkill pci_hotplug e1000e coretemp wmi kvm_intel i2c_i801 kvm mei pcspkr joydev ghash_clmulni_intel microcode crc32c_intel edd aesni_intel ablk_helper cryptd aes_x86_64 autofs4 hid_generic usbhid i915 xhci_hcd drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore usb_common video button scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh fan processor thermal

thermal_sys megaraid_sas

Quoted 115 lines

Aug 23 11:51:33 ex6 kernel: [55598.253818] CPU 6 
Aug 23 11:51:33 ex6 kernel: [55598.253822] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:33 ex6 kernel: [55598.253824] RIP: 0010:[<ffffffff81562be5>]  [<ffffffff81562be5>] _raw_spin_lock_bh+0x25/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253833] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:33 ex6 kernel: [55598.253835] RAX: 0000000000000014 RBX: ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:33 ex6 kernel: [55598.253837] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:33 ex6 kernel: [55598.253839] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:33 ex6 kernel: [55598.253841] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:33 ex6 kernel: [55598.253842] R13: ffff880407ba4cc0 R14: 0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:33 ex6 kernel: [55598.253845] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253847] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:33 ex6 kernel: [55598.253849] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:33 ex6 kernel: [55598.253850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253852] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:33 ex6 kernel: [55598.253855] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:33 ex6 kernel: [55598.253856] Stack:
Aug 23 11:51:33 ex6 kernel: [55598.253857]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:33 ex6 kernel: [55598.253861]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:33 ex6 kernel: [55598.253865]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:33 ex6 kernel: [55598.253868] Call Trace:
Aug 23 11:51:33 ex6 kernel: [55598.253877]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253885]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130
Aug 23 11:51:33 ex6 kernel: [55598.253892]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:33 ex6 kernel: [55598.253900]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:33 ex6 kernel: [55598.253909]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:33 ex6 kernel: [55598.253910] Code: c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 f3 90 0f b7 03 <66> 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 a5 81 e9 34 d9 
Aug 23 11:51:46 ex6 kernel: [55610.629219] INFO: rcu_sched self-detected stall on CPU { 7}  (t=2580171 jiffies)
Aug 23 11:51:46 ex6 kernel: [55610.629225] sending NMI to all CPUs:
Aug 23 11:51:46 ex6 kernel: [55610.629231] NMI backtrace for cpu 6
Aug 23 11:51:46 ex6 kernel: [55610.629234] CPU 6 
Aug 23 11:51:46 ex6 kernel: [55610.629239] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629242] RIP: 0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629252] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:46 ex6 kernel: [55610.629255] RAX: 0000000000000014 RBX: ffff880194f4c090 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629257] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629260] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:46 ex6 kernel: [55610.629262] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff8804067d2580
Aug 23 11:51:46 ex6 kernel: [55610.629265] R13: 00000000ffffffea R14: 00007fcd60a664a0 R15: 00007fcd51db52b8
Aug 23 11:51:46 ex6 kernel: [55610.629268] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629271] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629274] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629277] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629282] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:46 ex6 kernel: [55610.629284] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629286]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:46 ex6 kernel: [55610.629292]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:46 ex6 kernel: [55610.629297]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:46 ex6 kernel: [55610.629303] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629314]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629323]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130
Aug 23 11:51:46 ex6 kernel: [55610.629330]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629340]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:46 ex6 kernel: [55610.629364]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:46 ex6 kernel: [55610.629366] Code: 66 39 d0 75 f6 c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 
Aug 23 11:51:46 ex6 kernel: [55610.629413] NMI backtrace for cpu 7
Aug 23 11:51:46 ex6 kernel: [55610.629415] CPU 7 
Aug 23 11:51:46 ex6 kernel: [55610.629420] Pid: 0, comm: swapper/7 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629422] RIP: 0010:[<ffffffff812df109>]  [<ffffffff812df109>] find_next_bit+0x39/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629432] RSP: 0018:ffff88041f3c3938  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629434] RAX: 0000000000000000 RBX: ffff88041f3ccfe0 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629437] RDX: 0000000000000001 RSI: 0000000000000200 RDI: 0000000000000200
Aug 23 11:51:46 ex6 kernel: [55610.629440] RBP: ffff88041f20d020 R08: ffff88041f3ccfe0 R09: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629442] R10: 0000000000000000 R11: 000000000000c2e2 R12: 000000000000d060
Aug 23 11:51:46 ex6 kernel: [55610.629445] R13: 0000000000080000 R14: 00000000000000ff R15: 0000000000000007
Aug 23 11:51:46 ex6 kernel: [55610.629448] FS:  0000000000000000(0000) GS:ffff88041f3c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629451] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629454] CR2: 00007fcd2c062ac0 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629459] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629462] Process swapper/7 (pid: 0, threadinfo ffff88040abc4000, task ffff88040abc2400)
Aug 23 11:51:46 ex6 kernel: [55610.629464] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629465]  ffffffff8102a4e7 0000000081a2cec0 000000000000d020 0000000200000007
Aug 23 11:51:46 ex6 kernel: [55610.629471]  0000000000000082 ffff88041f3c39b0 0000000000002710 ffffffff81a2ce00
Aug 23 11:51:46 ex6 kernel: [55610.629476]  ffff88040abc4000 0000000000000007 ffff88040abc4000 ffffffff81a2cec0
Aug 23 11:51:46 ex6 kernel: [55610.629482] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629494]  [<ffffffff8102a4e7>] __x2apic_send_IPI_mask+0xb7/0x1a0
Aug 23 11:51:46 ex6 kernel: [55610.629503]  [<ffffffff810260ff>] arch_trigger_all_cpu_backtrace+0x4f/0x90
Aug 23 11:51:46 ex6 kernel: [55610.629510]  [<ffffffff810d6d20>] rcu_pending+0x210/0x560
Aug 23 11:51:46 ex6 kernel: [55610.629519]  [<ffffffff810d78c0>] rcu_check_callbacks+0x90/0x110
Aug 23 11:51:46 ex6 kernel: [55610.629528]  [<ffffffff8105467f>] update_process_times+0x3f/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629536]  [<ffffffff810971fb>] tick_sched_timer+0x6b/0xe0
Aug 23 11:51:46 ex6 kernel: [55610.629544]  [<ffffffff810699a6>] __run_hrtimer+0x66/0x1d0
Aug 23 11:51:46 ex6 kernel: [55610.629552]  [<ffffffff8106a299>] hrtimer_interrupt+0xe9/0x210
Aug 23 11:51:46 ex6 kernel: [55610.629559]  [<ffffffff81024e93>] smp_apic_timer_interrupt+0x63/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629567]  [<ffffffff8156b0fa>] apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629578]  [<ffffffff814c2c78>] tcp_slow_start+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629585]  [<ffffffff814b6260>] tcp_ack+0x560/0x11e0
Aug 23 11:51:46 ex6 kernel: [55610.629593]  [<ffffffff814b715d>] tcp_rcv_established+0x27d/0x8e0
Aug 23 11:51:46 ex6 kernel: [55610.629601]  [<ffffffff814c045e>] tcp_v4_do_rcv+0xde/0x340
Aug 23 11:51:46 ex6 kernel: [55610.629609]  [<ffffffff814c173b>] tcp_v4_rcv+0x59b/0x850
Aug 23 11:51:46 ex6 kernel: [55610.629617]  [<ffffffff8149e156>] ip_local_deliver_finish+0xd6/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629625]  [<ffffffff8146eb5e>] __netif_receive_skb+0x67e/0x7e0
Aug 23 11:51:46 ex6 kernel: [55610.629633]  [<ffffffff8146ed61>] process_backlog+0xa1/0x180
Aug 23 11:51:46 ex6 kernel: [55610.629639]  [<ffffffff8146f830>] net_rx_action+0x140/0x230
Aug 23 11:51:46 ex6 kernel: [55610.629647]  [<ffffffff8104be2e>] __do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629655]  [<ffffffff8156b7ec>] call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629662]  [<ffffffff810045a5>] do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629669]  [<ffffffff8104c1e5>] irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629677]  [<ffffffff8156b2ba>] call_function_single_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629686]  [<ffffffff813303cc>] intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629695]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629702]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629710]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629713] Code: d1 49 89 d1 48 c1 e9 06 49 83 e1 c0 4c 8d 04 cf 48 89 f7 48 89 d1 4c 29 cf 83 e1 3f 74 40 48 c7 c0 ff ff ff ff 48 d3 e0 49 23 00 <48> 83 ff 3f 76 41 48 85 c0 75 52 49 83 c0 08 48 83 ef 40 49 83 
Aug 23 11:51:46 ex6 kernel: [55610.629761] NMI backtrace for cpu 1
Aug 23 11:51:46 ex6 kernel: [55610.629765] CPU 1 
Aug 23 11:51:46 ex6 kernel: [55610.629769] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629772] RIP: 0010:[<ffffffff815628dd>]  [<ffffffff815628dd>] _raw_spin_lock+0x1d/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629781] RSP: 0018:ffff88041f243e48  EFLAGS: 00000293
Aug 23 11:51:46 ex6 kernel: [55610.629784] RAX: 0000000000000014 RBX: ffff880194f4c040 RCX: 0000000100c76001
Aug 23 11:51:46 ex6 kernel: [55610.629787] RDX: 0000000000000016 RSI: 0000000000000286 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629789] RBP: ffff88040abd4000 R08: ffff88040701a29c R09: 0000000000000028
Aug 23 11:51:46 ex6 kernel: [55610.629792] R10: dead000000200200 R11: 0000000000000000 R12: 0000000000000100
Aug 23 11:51:46 ex6 kernel: [55610.629794] R13: ffffffff814bd160 R14: ffff880194f4c040 R15: 0000000000000000

Quoted 8 lines

Aug 23 11:51:46 ex6 kernel: [55610.629798] FS:  0000000000000000(0000) GS:ffff88041f240000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629800] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629803] CR2: 00007fa566135010 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629808] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629811] Process swapper/1 (pid: 0, threadinfo ffff88040aba6000, task ffff88040aba4280)
Aug 23 11:51:46 ex6 kernel: [55610.629813] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629815]  ffffffff814bd171 ffff88040abd4000 0000000000000100 ffff880194f4c200ug 23 11:51:05 ex6 kernel: [55570.280878] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_

conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntr
ack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_

Aug 23 11:51:05 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit

xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet cachefiles fscache eeepc_wmi asus_wmi sparse_keymap acpi_cpufreq mperf rfkill pci_hotplug e1000e coretemp wmi kvm_intel i2c_i801 kvm mei pcspkr joydev ghash_clmulni_intel microcode crc32c_intel edd aesni_intel ablk_helper cryptd aes_x86_64 autofs4 hid_generic usbhid i915 xhci_hcd drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore usb_common video button scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh fan processor thermal thermal_sys megaraid_sas

Aug 23 11:51:05 ex6 kernel: [55570.280987] CPU 6
Aug 23 11:51:05 ex6 kernel: [55570.280991] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:05 ex6 kernel: [55570.280993] RIP:
0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281002] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:05 ex6 kernel: [55570.281004] RAX: 0000000000000014 RBX:
ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:05 ex6 kernel: [55570.281006] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:05 ex6 kernel: [55570.281007] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:05 ex6 kernel: [55570.281009] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:05 ex6 kernel: [55570.281011] R13: ffff880407ba4cc0 R14:
0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:05 ex6 kernel: [55570.281013] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281015] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:05 ex6 kernel: [55570.281017] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:05 ex6 kernel: [55570.281019] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281021] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:05 ex6 kernel: [55570.281023] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:05 ex6 kernel: [55570.281024] Stack:
Aug 23 11:51:05 ex6 kernel: [55570.281025]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:05 ex6 kernel: [55570.281029]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:05 ex6 kernel: [55570.281033]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:05 ex6 kernel: [55570.281037] Call Trace:
Aug 23 11:51:05 ex6 kernel: [55570.281046]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281052]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:05 ex6 kernel: [55570.281058]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:05 ex6 kernel: [55570.281066]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:05 ex6 kernel: [55570.281075]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:05 ex6 kernel: [55570.281076] Code: 66 39 d0 75 f6 c3 0f 1f
44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1
ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3
90 90 90 90 48 c7 c7 60 74
Aug 23 11:51:24 ex6 cachefilesd[5919]: Refilling cull table
Aug 23 11:51:24 ex6 cachefilesd[5919]: Scan complete
Aug 23 11:51:33 ex6 kernel: [55598.253247] BUG: soft lockup - CPU#6
stuck for 22s! [httpd2-event:15597]
Aug 23 11:51:33 ex6 kernel: [55598.253708] Modules linked in: sg st

sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_

Aug 23 11:51:33 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit

Aug 23 11:51:33 ex6 kernel: [55598.253818] CPU 6
Aug 23 11:51:33 ex6 kernel: [55598.253822] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:33 ex6 kernel: [55598.253824] RIP:
0010:[<ffffffff81562be5>]  [<ffffffff81562be5>] _raw_spin_lock_bh+0x25/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253833] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:33 ex6 kernel: [55598.253835] RAX: 0000000000000014 RBX:
ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:33 ex6 kernel: [55598.253837] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:33 ex6 kernel: [55598.253839] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:33 ex6 kernel: [55598.253841] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:33 ex6 kernel: [55598.253842] R13: ffff880407ba4cc0 R14:
0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:33 ex6 kernel: [55598.253845] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253847] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:33 ex6 kernel: [55598.253849] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:33 ex6 kernel: [55598.253850] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253852] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:33 ex6 kernel: [55598.253855] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:33 ex6 kernel: [55598.253856] Stack:
Aug 23 11:51:33 ex6 kernel: [55598.253857]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:33 ex6 kernel: [55598.253861]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:33 ex6 kernel: [55598.253865]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:33 ex6 kernel: [55598.253868] Call Trace:
Aug 23 11:51:33 ex6 kernel: [55598.253877]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253885]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:33 ex6 kernel: [55598.253892]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:33 ex6 kernel: [55598.253900]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:33 ex6 kernel: [55598.253909]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:33 ex6 kernel: [55598.253910] Code: c3 0f 1f 44 00 00 53 48
89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2
74 0e 0f 1f 40 00 f3 90 0f b7 03 <66> 39 d0 75 f6 5b c3 90 90 90 90 48
c7 c7 60 74 a5 81 e9 34 d9
Aug 23 11:51:46 ex6 kernel: [55610.629219] INFO: rcu_sched self-detected
stall on CPU { 7}  (t=2580171 jiffies)
Aug 23 11:51:46 ex6 kernel: [55610.629225] sending NMI to all CPUs:
Aug 23 11:51:46 ex6 kernel: [55610.629231] NMI backtrace for cpu 6
Aug 23 11:51:46 ex6 kernel: [55610.629234] CPU 6
Aug 23 11:51:46 ex6 kernel: [55610.629239] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629242] RIP:
0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629252] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:46 ex6 kernel: [55610.629255] RAX: 0000000000000014 RBX:
ffff880194f4c090 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629257] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629260] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:46 ex6 kernel: [55610.629262] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff8804067d2580
Aug 23 11:51:46 ex6 kernel: [55610.629265] R13: 00000000ffffffea R14:
00007fcd60a664a0 R15: 00007fcd51db52b8
Aug 23 11:51:46 ex6 kernel: [55610.629268] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629271] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629274] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629277] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629279] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629282] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:46 ex6 kernel: [55610.629284] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629286]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:46 ex6 kernel: [55610.629292]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:46 ex6 kernel: [55610.629297]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:46 ex6 kernel: [55610.629303] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629314]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629323]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:46 ex6 kernel: [55610.629330]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629340]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:46 ex6 kernel: [55610.629364]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:46 ex6 kernel: [55610.629366] Code: 66 39 d0 75 f6 c3 0f 1f
44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1
ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3
90 90 90 90 48 c7 c7 60 74
Aug 23 11:51:46 ex6 kernel: [55610.629413] NMI backtrace for cpu 7
Aug 23 11:51:46 ex6 kernel: [55610.629415] CPU 7
Aug 23 11:51:46 ex6 kernel: [55610.629420] Pid: 0, comm: swapper/7 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629422] RIP:
0010:[<ffffffff812df109>]  [<ffffffff812df109>] find_next_bit+0x39/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629432] RSP: 0018:ffff88041f3c3938
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629434] RAX: 0000000000000000 RBX:
ffff88041f3ccfe0 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629437] RDX: 0000000000000001 RSI:
0000000000000200 RDI: 0000000000000200
Aug 23 11:51:46 ex6 kernel: [55610.629440] RBP: ffff88041f20d020 R08:
ffff88041f3ccfe0 R09: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629442] R10: 0000000000000000 R11:
000000000000c2e2 R12: 000000000000d060
Aug 23 11:51:46 ex6 kernel: [55610.629445] R13: 0000000000080000 R14:
00000000000000ff R15: 0000000000000007
Aug 23 11:51:46 ex6 kernel: [55610.629448] FS:  0000000000000000(0000)
GS:ffff88041f3c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629451] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629454] CR2: 00007fcd2c062ac0 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629456] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629459] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629462] Process swapper/7 (pid: 0,
threadinfo ffff88040abc4000, task ffff88040abc2400)
Aug 23 11:51:46 ex6 kernel: [55610.629464] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629465]  ffffffff8102a4e7
0000000081a2cec0 000000000000d020 0000000200000007
Aug 23 11:51:46 ex6 kernel: [55610.629471]  0000000000000082
ffff88041f3c39b0 0000000000002710 ffffffff81a2ce00
Aug 23 11:51:46 ex6 kernel: [55610.629476]  ffff88040abc4000
0000000000000007 ffff88040abc4000 ffffffff81a2cec0
Aug 23 11:51:46 ex6 kernel: [55610.629482] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629494]  [<ffffffff8102a4e7>]
__x2apic_send_IPI_mask+0xb7/0x1a0
Aug 23 11:51:46 ex6 kernel: [55610.629503]  [<ffffffff810260ff>]
arch_trigger_all_cpu_backtrace+0x4f/0x90
Aug 23 11:51:46 ex6 kernel: [55610.629510]  [<ffffffff810d6d20>]
rcu_pending+0x210/0x560
Aug 23 11:51:46 ex6 kernel: [55610.629519]  [<ffffffff810d78c0>]
rcu_check_callbacks+0x90/0x110
Aug 23 11:51:46 ex6 kernel: [55610.629528]  [<ffffffff8105467f>]
update_process_times+0x3f/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629536]  [<ffffffff810971fb>]
tick_sched_timer+0x6b/0xe0
Aug 23 11:51:46 ex6 kernel: [55610.629544]  [<ffffffff810699a6>]
__run_hrtimer+0x66/0x1d0
Aug 23 11:51:46 ex6 kernel: [55610.629552]  [<ffffffff8106a299>]
hrtimer_interrupt+0xe9/0x210
Aug 23 11:51:46 ex6 kernel: [55610.629559]  [<ffffffff81024e93>]
smp_apic_timer_interrupt+0x63/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629567]  [<ffffffff8156b0fa>]
apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629578]  [<ffffffff814c2c78>]
tcp_slow_start+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629585]  [<ffffffff814b6260>]
tcp_ack+0x560/0x11e0
Aug 23 11:51:46 ex6 kernel: [55610.629593]  [<ffffffff814b715d>]
tcp_rcv_established+0x27d/0x8e0
Aug 23 11:51:46 ex6 kernel: [55610.629601]  [<ffffffff814c045e>]
tcp_v4_do_rcv+0xde/0x340
Aug 23 11:51:46 ex6 kernel: [55610.629609]  [<ffffffff814c173b>]
tcp_v4_rcv+0x59b/0x850
Aug 23 11:51:46 ex6 kernel: [55610.629617]  [<ffffffff8149e156>]
ip_local_deliver_finish+0xd6/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629625]  [<ffffffff8146eb5e>]
__netif_receive_skb+0x67e/0x7e0
Aug 23 11:51:46 ex6 kernel: [55610.629633]  [<ffffffff8146ed61>]
process_backlog+0xa1/0x180
Aug 23 11:51:46 ex6 kernel: [55610.629639]  [<ffffffff8146f830>]
net_rx_action+0x140/0x230
Aug 23 11:51:46 ex6 kernel: [55610.629647]  [<ffffffff8104be2e>]
__do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629655]  [<ffffffff8156b7ec>]
call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629662]  [<ffffffff810045a5>]
do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629669]  [<ffffffff8104c1e5>]
irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629677]  [<ffffffff8156b2ba>]
call_function_single_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629686]  [<ffffffff813303cc>]
intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629695]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629702]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629710]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629713] Code: d1 49 89 d1 48 c1 e9 06
49 83 e1 c0 4c 8d 04 cf 48 89 f7 48 89 d1 4c 29 cf 83 e1 3f 74 40 48 c7
c0 ff ff ff ff 48 d3 e0 49 23 00 <48> 83 ff 3f 76 41 48 85 c0 75 52 49
83 c0 08 48 83 ef 40 49 83
Aug 23 11:51:46 ex6 kernel: [55610.629761] NMI backtrace for cpu 1
Aug 23 11:51:46 ex6 kernel: [55610.629765] CPU 1
Aug 23 11:51:46 ex6 kernel: [55610.629769] Pid: 0, comm: swapper/1 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629772] RIP:
0010:[<ffffffff815628dd>]  [<ffffffff815628dd>] _raw_spin_lock+0x1d/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629781] RSP: 0018:ffff88041f243e48
EFLAGS: 00000293
Aug 23 11:51:46 ex6 kernel: [55610.629784] RAX: 0000000000000014 RBX:
ffff880194f4c040 RCX: 0000000100c76001
Aug 23 11:51:46 ex6 kernel: [55610.629787] RDX: 0000000000000016 RSI:
0000000000000286 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629789] RBP: ffff88040abd4000 R08:
ffff88040701a29c R09: 0000000000000028
Aug 23 11:51:46 ex6 kernel: [55610.629792] R10: dead000000200200 R11:
0000000000000000 R12: 0000000000000100
Aug 23 11:51:46 ex6 kernel: [55610.629794] R13: ffffffff814bd160 R14:
ffff880194f4c040 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629798] FS:  0000000000000000(0000)
GS:ffff88041f240000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629800] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629803] CR2: 00007fa566135010 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629806] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629808] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629811] Process swapper/1 (pid: 0,
threadinfo ffff88040aba6000, task ffff88040aba4280)
Aug 23 11:51:46 ex6 kernel: [55610.629813] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629815]  ffffffff814bd171
ffff88040abd4000 0000000000000100 ffff880194f4c200
Aug 23 11:51:46 ex6 kernel: [55610.629820]  ffffffff81053ac1
0000000000000000 ffff88041f243ea0 ffff88040aba7fd8
Aug 23 11:51:46 ex6 kernel: [55610.629825]  ffff88040aba7fd8
ffff88040aba7fd8 ffff88040abd5028 ffff88041f243ea0
Aug 23 11:51:46 ex6 kernel: [55610.629831] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629841]  [<ffffffff814bd171>]
tcp_keepalive_timer+0x11/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629850]  [<ffffffff81053ac1>]
run_timer_softirq+0x151/0x350
Aug 23 11:51:46 ex6 kernel: [55610.629857]  [<ffffffff8104be2e>]
__do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629865]  [<ffffffff8156b7ec>]
call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629871]  [<ffffffff810045a5>]
do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629879]  [<ffffffff8104c1e5>]
irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629886]  [<ffffffff81024e98>]
smp_apic_timer_interrupt+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629893]  [<ffffffff8156b0fa>]
apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629902]  [<ffffffff813303cc>]
intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629911]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629918]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629925]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629929] Code: 90 90 90 90 90 90 90 90
90 90 90 90 90 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74
0f 0f 1f 44 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 c3 66 66 66 66 2e 0f
1f 84 00 00 00 00 00 66 83
Aug 23 11:51:46 ex6 kernel: [55610.629976] NMI backtrace for cpu 2
Aug 23 11:51:46 ex6 kernel: [55610.629979] CPU 2
Aug 23 11:51:46 ex6 kernel: [55610.629983] Pid: 0, comm: swapper/2 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629986] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629992] RSP: 0018:ffff88040ababe78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629994] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629997] RDX: 0000000000000000 RSI:
ffff88040ababfd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.629999] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e3c
Aug 23 11:51:46 ex6 kernel: [55610.630002] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630004] R13: 0000000000000020 R14:
12acf11f7efbdbfc R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630007] FS:  0000000000000000(0000)
GS:ffff88041f280000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630010] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630012] CR2: 00007fa566135010 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630015] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630017] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630020] Process swapper/2 (pid: 0,
threadinfo ffff88040abaa000, task ffff88040aba82c0)
Aug 23 11:51:46 ex6 kernel: [55610.630022] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630024]  0000000077359400
ffffffff81436a70 0000000000000000 0000000200f31e03
Aug 23 11:51:46 ex6 kernel: [55610.630029]  0000000000000000
0000000000f31e03 ffff88040ababfd8 ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630034]  0000000000000003
0000000000000002 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630039] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630048]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630055]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630062]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630066] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630113] NMI backtrace for cpu 5
Aug 23 11:51:46 ex6 kernel: [55610.630115] CPU 5
Aug 23 11:51:46 ex6 kernel: [55610.630120] Pid: 0, comm: swapper/5 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630122] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630128] RSP: 0018:ffff88040abb9e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630130] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630133] RDX: 0000000000000000 RSI:
ffff88040abb9fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630135] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000005634
Aug 23 11:51:46 ex6 kernel: [55610.630138] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630140] R13: 0000000000000020 R14:
12acf11f7e9e3c16 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630143] FS:  0000000000000000(0000)
GS:ffff88041f340000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630146] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630149] CR2: 00007fd29bfff000 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630151] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630154] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630157] Process swapper/5 (pid: 0,
threadinfo ffff88040abb8000, task ffff88040abb6380)
Aug 23 11:51:46 ex6 kernel: [55610.630158] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630160]  0000000077359400
ffffffff81436a70 0000000000000000 000000050150be3d
Aug 23 11:51:46 ex6 kernel: [55610.630165]  0000000000000000
000000000150be3d ffff88040abb9fd8 ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630170]  0000000000000003
0000000000000005 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630176] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630184]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630191]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630198]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630201] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630248] NMI backtrace for cpu 3
Aug 23 11:51:46 ex6 kernel: [55610.630251] CPU 3
Aug 23 11:51:46 ex6 kernel: [55610.630255] Pid: 0, comm: swapper/3 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630257] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630263] RSP: 0018:ffff88040abb1e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630266] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630268] RDX: 0000000000000000 RSI:
ffff88040abb1fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630271] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e43
Aug 23 11:51:46 ex6 kernel: [55610.630273] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630276] R13: 0000000000000020 R14:
12acf11f7efbc356 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630279] FS:  0000000000000000(0000)
GS:ffff88041f2c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630282] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630284] CR2: 00007fe7d784ba68 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630287] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630289] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630292] Process swapper/3 (pid: 0,
threadinfo ffff88040abb0000, task ffff88040abae300)
Aug 23 11:51:46 ex6 kernel: [55610.630294] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630296]  0000000077359400
ffffffff81436a70 0000000000000000 0000000300f3366a
Aug 23 11:51:46 ex6 kernel: [55610.630301]  0000000000000000
0000000000f3366a ffff88040abb1fd8 ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630306]  0000000000000003
0000000000000003 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630311] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630320]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630327]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630333]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630337] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630385] NMI backtrace for cpu 0
Aug 23 11:51:46 ex6 kernel: [55610.630388] CPU 0
Aug 23 11:51:46 ex6 kernel: [55610.630393] Pid: 0, comm: swapper/0 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630396] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630403] RSP: 0018:ffffffff81a01e88
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630406] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630409] RDX: 0000000000000000 RSI:
ffffffff81a01fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630411] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000005d52
Aug 23 11:51:46 ex6 kernel: [55610.630413] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630416] R13: 0000000000000020 R14:
12acf11f7f769680 R15: 0000000001fe8018
Aug 23 11:51:46 ex6 kernel: [55610.630419] FS:  0000000000000000(0000)
GS:ffff88041f200000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630422] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630425] CR2: 00007f03abb83000 CR3:
0000000001a0c000 CR4: 00000000000407f0
Aug 23 11:51:46 ex6 kernel: [55610.630427] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630430] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630433] Process swapper/0 (pid: 0,
threadinfo ffffffff81a00000, task ffffffff81a14420)
Aug 23 11:51:46 ex6 kernel: [55610.630435] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630436]  0000000077359400
ffffffff81436a70 0000000000000000 00000000016c88d5
Aug 23 11:51:46 ex6 kernel: [55610.630442]  0000000000000000
00000000016c88d5 ffffffff81a01fd8 ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630447]  0000000000000003
0000000000000000 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630453] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630464]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630471]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630480]  [<ffffffff81ac0ba2>]
start_kernel+0x391/0x39c
Aug 23 11:51:46 ex6 kernel: [55610.630488]  [<ffffffff81ac0436>]
x86_64_start_kernel+0x105/0x114
Aug 23 11:51:46 ex6 kernel: [55610.630492] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630539] NMI backtrace for cpu 4
Aug 23 11:51:46 ex6 kernel: [55610.630542] CPU 4
Aug 23 11:51:46 ex6 kernel: [55610.630546] Pid: 0, comm: swapper/4 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630549] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630555] RSP: 0018:ffff88040abb5e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630557] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630560] RDX: 0000000000000000 RSI:
ffff88040abb5fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630562] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e1e
Aug 23 11:51:46 ex6 kernel: [55610.630565] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630568] R13: 0000000000000020 R14:
12acf11f7efc52d2 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630571] FS:  0000000000000000(0000)
GS:ffff88041f300000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630573] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630576] CR2: 00007fd29bfff000 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630578] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630581] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630584] Process swapper/4 (pid: 0,
threadinfo ffff88040abb4000, task ffff88040abb2340)
Aug 23 11:51:46 ex6 kernel: [55610.630586] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630588]  0000000077359400
ffffffff81436a70 0000000000000000 0000000400f2a60d
Aug 23 11:51:46 ex6 kernel: [55610.630593]  0000000000000000
0000000000f2a60d ffff88040abb5fd8 ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630598]  0000000000000003
0000000000000004 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630604] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630614]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630622]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630629]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630633] Code: 28 e0 ff ff 83 e2 08 75

22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48

Quoted 134 lines

Aug 23 11:51:46 ex6 kernel: [55610.629820]  ffffffff81053ac1 0000000000000000 ffff88041f243ea0 ffff88040aba7fd8
Aug 23 11:51:46 ex6 kernel: [55610.629825]  ffff88040aba7fd8 ffff88040aba7fd8 ffff88040abd5028 ffff88041f243ea0
Aug 23 11:51:46 ex6 kernel: [55610.629831] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629841]  [<ffffffff814bd171>] tcp_keepalive_timer+0x11/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629850]  [<ffffffff81053ac1>] run_timer_softirq+0x151/0x350
Aug 23 11:51:46 ex6 kernel: [55610.629857]  [<ffffffff8104be2e>] __do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629865]  [<ffffffff8156b7ec>] call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629871]  [<ffffffff810045a5>] do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629879]  [<ffffffff8104c1e5>] irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629886]  [<ffffffff81024e98>] smp_apic_timer_interrupt+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629893]  [<ffffffff8156b0fa>] apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629902]  [<ffffffff813303cc>] intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629911]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629918]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629925]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629929] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 0f 0f 1f 44 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 83 
Aug 23 11:51:46 ex6 kernel: [55610.629976] NMI backtrace for cpu 2
Aug 23 11:51:46 ex6 kernel: [55610.629979] CPU 2 
Aug 23 11:51:46 ex6 kernel: [55610.629983] Pid: 0, comm: swapper/2 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629986] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629992] RSP: 0018:ffff88040ababe78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629994] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629997] RDX: 0000000000000000 RSI: ffff88040ababfd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.629999] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e3c
Aug 23 11:51:46 ex6 kernel: [55610.630002] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630004] R13: 0000000000000020 R14: 12acf11f7efbdbfc R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630007] FS:  0000000000000000(0000) GS:ffff88041f280000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630010] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630012] CR2: 00007fa566135010 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630020] Process swapper/2 (pid: 0, threadinfo ffff88040abaa000, task ffff88040aba82c0)
Aug 23 11:51:46 ex6 kernel: [55610.630022] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630024]  0000000077359400 ffffffff81436a70 0000000000000000 0000000200f31e03
Aug 23 11:51:46 ex6 kernel: [55610.630029]  0000000000000000 0000000000f31e03 ffff88040ababfd8 ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630034]  0000000000000003 0000000000000002 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630039] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630048]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630055]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630062]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630066] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630113] NMI backtrace for cpu 5
Aug 23 11:51:46 ex6 kernel: [55610.630115] CPU 5 
Aug 23 11:51:46 ex6 kernel: [55610.630120] Pid: 0, comm: swapper/5 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630122] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630128] RSP: 0018:ffff88040abb9e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630130] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630133] RDX: 0000000000000000 RSI: ffff88040abb9fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630135] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000005634
Aug 23 11:51:46 ex6 kernel: [55610.630138] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630140] R13: 0000000000000020 R14: 12acf11f7e9e3c16 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630143] FS:  0000000000000000(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630146] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630149] CR2: 00007fd29bfff000 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630154] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630157] Process swapper/5 (pid: 0, threadinfo ffff88040abb8000, task ffff88040abb6380)
Aug 23 11:51:46 ex6 kernel: [55610.630158] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630160]  0000000077359400 ffffffff81436a70 0000000000000000 000000050150be3d
Aug 23 11:51:46 ex6 kernel: [55610.630165]  0000000000000000 000000000150be3d ffff88040abb9fd8 ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630170]  0000000000000003 0000000000000005 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630176] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630184]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630191]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630198]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630201] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630248] NMI backtrace for cpu 3
Aug 23 11:51:46 ex6 kernel: [55610.630251] CPU 3 
Aug 23 11:51:46 ex6 kernel: [55610.630255] Pid: 0, comm: swapper/3 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630257] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630263] RSP: 0018:ffff88040abb1e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630266] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630268] RDX: 0000000000000000 RSI: ffff88040abb1fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630271] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e43
Aug 23 11:51:46 ex6 kernel: [55610.630273] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630276] R13: 0000000000000020 R14: 12acf11f7efbc356 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630279] FS:  0000000000000000(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630282] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630284] CR2: 00007fe7d784ba68 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630289] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630292] Process swapper/3 (pid: 0, threadinfo ffff88040abb0000, task ffff88040abae300)
Aug 23 11:51:46 ex6 kernel: [55610.630294] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630296]  0000000077359400 ffffffff81436a70 0000000000000000 0000000300f3366a
Aug 23 11:51:46 ex6 kernel: [55610.630301]  0000000000000000 0000000000f3366a ffff88040abb1fd8 ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630306]  0000000000000003 0000000000000003 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630311] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630320]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630327]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630333]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630337] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630385] NMI backtrace for cpu 0
Aug 23 11:51:46 ex6 kernel: [55610.630388] CPU 0 
Aug 23 11:51:46 ex6 kernel: [55610.630393] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630396] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630403] RSP: 0018:ffffffff81a01e88  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630406] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630409] RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630411] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000005d52
Aug 23 11:51:46 ex6 kernel: [55610.630413] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630416] R13: 0000000000000020 R14: 12acf11f7f769680 R15: 0000000001fe8018
Aug 23 11:51:46 ex6 kernel: [55610.630419] FS:  0000000000000000(0000) GS:ffff88041f200000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630422] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630425] CR2: 00007f03abb83000 CR3: 0000000001a0c000 CR4: 00000000000407f0
Aug 23 11:51:46 ex6 kernel: [55610.630427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630430] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630433] Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a14420)
Aug 23 11:51:46 ex6 kernel: [55610.630435] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630436]  0000000077359400 ffffffff81436a70 0000000000000000 00000000016c88d5
Aug 23 11:51:46 ex6 kernel: [55610.630442]  0000000000000000 00000000016c88d5 ffffffff81a01fd8 ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630447]  0000000000000003 0000000000000000 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630453] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630464]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630471]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630480]  [<ffffffff81ac0ba2>] start_kernel+0x391/0x39c
Aug 23 11:51:46 ex6 kernel: [55610.630488]  [<ffffffff81ac0436>] x86_64_start_kernel+0x105/0x114
Aug 23 11:51:46 ex6 kernel: [55610.630492] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630539] NMI backtrace for cpu 4
Aug 23 11:51:46 ex6 kernel: [55610.630542] CPU 4 
Aug 23 11:51:46 ex6 kernel: [55610.630546] Pid: 0, comm: swapper/4 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630549] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630555] RSP: 0018:ffff88040abb5e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630557] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630560] RDX: 0000000000000000 RSI: ffff88040abb5fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630562] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e1e
Aug 23 11:51:46 ex6 kernel: [55610.630565] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630568] R13: 0000000000000020 R14: 12acf11f7efc52d2 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630571] FS:  0000000000000000(0000) GS:ffff88041f300000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630573] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630576] CR2: 00007fd29bfff000 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630581] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630584] Process swapper/4 (pid: 0, threadinfo ffff88040abb4000, task ffff88040abb2340)
Aug 23 11:51:46 ex6 kernel: [55610.630586] Stack:

Quoted 8 lines

Aug 23 11:51:46 ex6 kernel: [55610.630588]  0000000077359400 ffffffff81436a70 0000000000000000 0000000400f2a60d
Aug 23 11:51:46 ex6 kernel: [55610.630593]  0000000000000000 0000000000f2a60d ffff88040abb5fd8 ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630598]  0000000000000003 0000000000000004 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630604] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630614]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630622]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630629]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630633] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48

Tested kernels with the problem 3.6.RC2 from SUSE, and today's linus vanilla tree.

Server needs reboot. (or let's say two completely different boxes have the same problem)

Eric Dumazet

Acked-by Yuchung Cheng Acked-by Neal Cardwell

2012-08-25 08:59 UTC #1 lore raw

Quoted 926 lines · Cristian Rodríguez

Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..

Quoted 1 line

Aug 23 11:51:05 ex6 kernel: [55570.280878] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_

NFQUEUE xt_NFLOG nfnetlink_

Quoted 23 lines

Aug 23 11:51:05 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet cachefiles fscache eeepc_wmi asus_wmi sparse_keymap acpi_cpufreq mperf rfkill pci_hotplug e1000e coretemp wmi kvm_intel i2c_i801 kvm mei pcspkr joydev ghash_clmulni_intel microcode crc32c_intel edd aesni_intel ablk_helper cryptd aes_x86_64 autofs4 hid_generic usbhid i915 xhci_hcd drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore usb_common video button scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh fan processor thermal thermal_sys megaraid_sas
Aug 23 11:51:05 ex6 kernel: [55570.280987] CPU 6 
Aug 23 11:51:05 ex6 kernel: [55570.280991] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:05 ex6 kernel: [55570.280993] RIP: 0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281002] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:05 ex6 kernel: [55570.281004] RAX: 0000000000000014 RBX: ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:05 ex6 kernel: [55570.281006] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:05 ex6 kernel: [55570.281007] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:05 ex6 kernel: [55570.281009] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:05 ex6 kernel: [55570.281011] R13: ffff880407ba4cc0 R14: 0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:05 ex6 kernel: [55570.281013] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:05 ex6 kernel: [55570.281017] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:05 ex6 kernel: [55570.281019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:05 ex6 kernel: [55570.281023] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:05 ex6 kernel: [55570.281024] Stack:
Aug 23 11:51:05 ex6 kernel: [55570.281025]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:05 ex6 kernel: [55570.281029]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:05 ex6 kernel: [55570.281033]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:05 ex6 kernel: [55570.281037] Call Trace:
Aug 23 11:51:05 ex6 kernel: [55570.281046]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281052]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130

Quoted 8 lines

Aug 23 11:51:05 ex6 kernel: [55570.281058]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:05 ex6 kernel: [55570.281066]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:05 ex6 kernel: [55570.281075]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:05 ex6 kernel: [55570.281076] Code: 66 39 d0 75 f6 c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 
Aug 23 11:51:24 ex6 cachefilesd[5919]: Refilling cull table
Aug 23 11:51:24 ex6 cachefilesd[5919]: Scan complete
Aug 23 11:51:33 ex6 kernel: [55598.253247] BUG: soft lockup - CPU#6 stuck for 22s! [httpd2-event:15597]
Aug 23 11:51:33 ex6 kernel: [55598.253708] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_

NFQUEUE xt_NFLOG nfnetlink_

Quoted 116 lines

Aug 23 11:51:33 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet cachefiles fscache eeepc_wmi asus_wmi sparse_keymap acpi_cpufreq mperf rfkill pci_hotplug e1000e coretemp wmi kvm_intel i2c_i801 kvm mei pcspkr joydev ghash_clmulni_intel microcode crc32c_intel edd aesni_intel ablk_helper cryptd aes_x86_64 autofs4 hid_generic usbhid i915 xhci_hcd drm_kms_helper drm ehci_hcd i2c_algo_bit usbcore usb_common video button scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh fan processor thermal thermal_sys megaraid_sas
Aug 23 11:51:33 ex6 kernel: [55598.253818] CPU 6 
Aug 23 11:51:33 ex6 kernel: [55598.253822] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:33 ex6 kernel: [55598.253824] RIP: 0010:[<ffffffff81562be5>]  [<ffffffff81562be5>] _raw_spin_lock_bh+0x25/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253833] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:33 ex6 kernel: [55598.253835] RAX: 0000000000000014 RBX: ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:33 ex6 kernel: [55598.253837] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:33 ex6 kernel: [55598.253839] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:33 ex6 kernel: [55598.253841] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:33 ex6 kernel: [55598.253842] R13: ffff880407ba4cc0 R14: 0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:33 ex6 kernel: [55598.253845] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253847] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:33 ex6 kernel: [55598.253849] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:33 ex6 kernel: [55598.253850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253852] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:33 ex6 kernel: [55598.253855] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:33 ex6 kernel: [55598.253856] Stack:
Aug 23 11:51:33 ex6 kernel: [55598.253857]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:33 ex6 kernel: [55598.253861]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:33 ex6 kernel: [55598.253865]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:33 ex6 kernel: [55598.253868] Call Trace:
Aug 23 11:51:33 ex6 kernel: [55598.253877]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253885]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130
Aug 23 11:51:33 ex6 kernel: [55598.253892]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:33 ex6 kernel: [55598.253900]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:33 ex6 kernel: [55598.253909]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:33 ex6 kernel: [55598.253910] Code: c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 f3 90 0f b7 03 <66> 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 a5 81 e9 34 d9 
Aug 23 11:51:46 ex6 kernel: [55610.629219] INFO: rcu_sched self-detected stall on CPU { 7}  (t=2580171 jiffies)
Aug 23 11:51:46 ex6 kernel: [55610.629225] sending NMI to all CPUs:
Aug 23 11:51:46 ex6 kernel: [55610.629231] NMI backtrace for cpu 6
Aug 23 11:51:46 ex6 kernel: [55610.629234] CPU 6 
Aug 23 11:51:46 ex6 kernel: [55610.629239] Pid: 15597, comm: httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629242] RIP: 0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629252] RSP: 0018:ffff88018fc79f08  EFLAGS: 00000297
Aug 23 11:51:46 ex6 kernel: [55610.629255] RAX: 0000000000000014 RBX: ffff880194f4c090 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629257] RDX: 0000000000000015 RSI: 0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629260] RBP: 0000000000000002 R08: ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:46 ex6 kernel: [55610.629262] R10: ffff88041e836800 R11: 0000000000000202 R12: ffff8804067d2580
Aug 23 11:51:46 ex6 kernel: [55610.629265] R13: 00000000ffffffea R14: 00007fcd60a664a0 R15: 00007fcd51db52b8
Aug 23 11:51:46 ex6 kernel: [55610.629268] FS:  00007fcd524e4700(0000) GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629271] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629274] CR2: 00007fcd51b59000 CR3: 0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629277] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629282] Process httpd2-event (pid: 15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:46 ex6 kernel: [55610.629284] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629286]  ffff880194f4c040 ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:46 ex6 kernel: [55610.629292]  ffff880194f5a300 ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:46 ex6 kernel: [55610.629297]  0004c7e9679f5fbf ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:46 ex6 kernel: [55610.629303] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629314]  [<ffffffff8145d45d>] lock_sock_nested+0xd/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629323]  [<ffffffff814d120e>] inet_shutdown+0x3e/0x130
Aug 23 11:51:46 ex6 kernel: [55610.629330]  [<ffffffff8145ac14>] sys_shutdown+0x74/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629340]  [<ffffffff8156a47d>] system_call_fastpath+0x1a/0x1f
Aug 23 11:51:46 ex6 kernel: [55610.629364]  [<00007fcd5fc42477>] 0x7fcd5fc42476
Aug 23 11:51:46 ex6 kernel: [55610.629366] Code: 66 39 d0 75 f6 c3 0f 1f 44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3 90 90 90 90 48 c7 c7 60 74 
Aug 23 11:51:46 ex6 kernel: [55610.629413] NMI backtrace for cpu 7
Aug 23 11:51:46 ex6 kernel: [55610.629415] CPU 7 
Aug 23 11:51:46 ex6 kernel: [55610.629420] Pid: 0, comm: swapper/7 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629422] RIP: 0010:[<ffffffff812df109>]  [<ffffffff812df109>] find_next_bit+0x39/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629432] RSP: 0018:ffff88041f3c3938  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629434] RAX: 0000000000000000 RBX: ffff88041f3ccfe0 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629437] RDX: 0000000000000001 RSI: 0000000000000200 RDI: 0000000000000200
Aug 23 11:51:46 ex6 kernel: [55610.629440] RBP: ffff88041f20d020 R08: ffff88041f3ccfe0 R09: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629442] R10: 0000000000000000 R11: 000000000000c2e2 R12: 000000000000d060
Aug 23 11:51:46 ex6 kernel: [55610.629445] R13: 0000000000080000 R14: 00000000000000ff R15: 0000000000000007
Aug 23 11:51:46 ex6 kernel: [55610.629448] FS:  0000000000000000(0000) GS:ffff88041f3c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629451] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629454] CR2: 00007fcd2c062ac0 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629459] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629462] Process swapper/7 (pid: 0, threadinfo ffff88040abc4000, task ffff88040abc2400)
Aug 23 11:51:46 ex6 kernel: [55610.629464] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629465]  ffffffff8102a4e7 0000000081a2cec0 000000000000d020 0000000200000007
Aug 23 11:51:46 ex6 kernel: [55610.629471]  0000000000000082 ffff88041f3c39b0 0000000000002710 ffffffff81a2ce00
Aug 23 11:51:46 ex6 kernel: [55610.629476]  ffff88040abc4000 0000000000000007 ffff88040abc4000 ffffffff81a2cec0
Aug 23 11:51:46 ex6 kernel: [55610.629482] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629494]  [<ffffffff8102a4e7>] __x2apic_send_IPI_mask+0xb7/0x1a0
Aug 23 11:51:46 ex6 kernel: [55610.629503]  [<ffffffff810260ff>] arch_trigger_all_cpu_backtrace+0x4f/0x90
Aug 23 11:51:46 ex6 kernel: [55610.629510]  [<ffffffff810d6d20>] rcu_pending+0x210/0x560
Aug 23 11:51:46 ex6 kernel: [55610.629519]  [<ffffffff810d78c0>] rcu_check_callbacks+0x90/0x110
Aug 23 11:51:46 ex6 kernel: [55610.629528]  [<ffffffff8105467f>] update_process_times+0x3f/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629536]  [<ffffffff810971fb>] tick_sched_timer+0x6b/0xe0
Aug 23 11:51:46 ex6 kernel: [55610.629544]  [<ffffffff810699a6>] __run_hrtimer+0x66/0x1d0
Aug 23 11:51:46 ex6 kernel: [55610.629552]  [<ffffffff8106a299>] hrtimer_interrupt+0xe9/0x210
Aug 23 11:51:46 ex6 kernel: [55610.629559]  [<ffffffff81024e93>] smp_apic_timer_interrupt+0x63/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629567]  [<ffffffff8156b0fa>] apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629578]  [<ffffffff814c2c78>] tcp_slow_start+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629585]  [<ffffffff814b6260>] tcp_ack+0x560/0x11e0
Aug 23 11:51:46 ex6 kernel: [55610.629593]  [<ffffffff814b715d>] tcp_rcv_established+0x27d/0x8e0
Aug 23 11:51:46 ex6 kernel: [55610.629601]  [<ffffffff814c045e>] tcp_v4_do_rcv+0xde/0x340
Aug 23 11:51:46 ex6 kernel: [55610.629609]  [<ffffffff814c173b>] tcp_v4_rcv+0x59b/0x850
Aug 23 11:51:46 ex6 kernel: [55610.629617]  [<ffffffff8149e156>] ip_local_deliver_finish+0xd6/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629625]  [<ffffffff8146eb5e>] __netif_receive_skb+0x67e/0x7e0
Aug 23 11:51:46 ex6 kernel: [55610.629633]  [<ffffffff8146ed61>] process_backlog+0xa1/0x180
Aug 23 11:51:46 ex6 kernel: [55610.629639]  [<ffffffff8146f830>] net_rx_action+0x140/0x230
Aug 23 11:51:46 ex6 kernel: [55610.629647]  [<ffffffff8104be2e>] __do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629655]  [<ffffffff8156b7ec>] call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629662]  [<ffffffff810045a5>] do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629669]  [<ffffffff8104c1e5>] irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629677]  [<ffffffff8156b2ba>] call_function_single_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629686]  [<ffffffff813303cc>] intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629695]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629702]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629710]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629713] Code: d1 49 89 d1 48 c1 e9 06 49 83 e1 c0 4c 8d 04 cf 48 89 f7 48 89 d1 4c 29 cf 83 e1 3f 74 40 48 c7 c0 ff ff ff ff 48 d3 e0 49 23 00 <48> 83 ff 3f 76 41 48 85 c0 75 52 49 83 c0 08 48 83 ef 40 49 83 
Aug 23 11:51:46 ex6 kernel: [55610.629761] NMI backtrace for cpu 1
Aug 23 11:51:46 ex6 kernel: [55610.629765] CPU 1 
Aug 23 11:51:46 ex6 kernel: [55610.629769] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629772] RIP: 0010:[<ffffffff815628dd>]  [<ffffffff815628dd>] _raw_spin_lock+0x1d/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629781] RSP: 0018:ffff88041f243e48  EFLAGS: 00000293
Aug 23 11:51:46 ex6 kernel: [55610.629784] RAX: 0000000000000014 RBX: ffff880194f4c040 RCX: 0000000100c76001
Aug 23 11:51:46 ex6 kernel: [55610.629787] RDX: 0000000000000016 RSI: 0000000000000286 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629789] RBP: ffff88040abd4000 R08: ffff88040701a29c R09: 0000000000000028
Aug 23 11:51:46 ex6 kernel: [55610.629792] R10: dead000000200200 R11: 0000000000000000 R12: 0000000000000100
Aug 23 11:51:46 ex6 kernel: [55610.629794] R13: ffffffff814bd160 R14: ffff880194f4c040 R15: 0000000000000000

Quoted 8 lines

Aug 23 11:51:46 ex6 kernel: [55610.629798] FS:  0000000000000000(0000) GS:ffff88041f240000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629800] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629803] CR2: 00007fa566135010 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629806] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629808] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629811] Process swapper/1 (pid: 0, threadinfo ffff88040aba6000, task ffff88040aba4280)
Aug 23 11:51:46 ex6 kernel: [55610.629813] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629815]  ffffffff814bd171 ffff88040abd4000 0000000000000100 ffff880194f4c200ug 23 11:51:05 ex6 kernel: [55570.280878] Modules linked in: sg st sr_mod cdrom act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq bridge stp llc xt_statistic xt_CT xt_LOG xt_realm xt_connlimit xt_addrtype iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp xt_time xt_TCPMSS xt_sctp nf_conntrack_amanda xt_policy nf_conntrack_sane ip6t_REJECT nf_conntrack_tftp nf_conntrack_ipv6 nf_conntrack_sip ip6table_raw nf_conntrack_proto_udplite ip6table_mangle nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntr

ack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core nf_defrag_ipv6 xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_

Aug 23 11:51:05 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit

Aug 23 11:51:05 ex6 kernel: [55570.280987] CPU 6
Aug 23 11:51:05 ex6 kernel: [55570.280991] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:05 ex6 kernel: [55570.280993] RIP:
0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281002] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:05 ex6 kernel: [55570.281004] RAX: 0000000000000014 RBX:
ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:05 ex6 kernel: [55570.281006] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:05 ex6 kernel: [55570.281007] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:05 ex6 kernel: [55570.281009] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:05 ex6 kernel: [55570.281011] R13: ffff880407ba4cc0 R14:
0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:05 ex6 kernel: [55570.281013] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281015] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:05 ex6 kernel: [55570.281017] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:05 ex6 kernel: [55570.281019] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:05 ex6 kernel: [55570.281021] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:05 ex6 kernel: [55570.281023] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:05 ex6 kernel: [55570.281024] Stack:
Aug 23 11:51:05 ex6 kernel: [55570.281025]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:05 ex6 kernel: [55570.281029]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:05 ex6 kernel: [55570.281033]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:05 ex6 kernel: [55570.281037] Call Trace:
Aug 23 11:51:05 ex6 kernel: [55570.281046]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:05 ex6 kernel: [55570.281052]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:05 ex6 kernel: [55570.281058]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:05 ex6 kernel: [55570.281066]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:05 ex6 kernel: [55570.281075]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:05 ex6 kernel: [55570.281076] Code: 66 39 d0 75 f6 c3 0f 1f
44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1
ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3
90 90 90 90 48 c7 c7 60 74
Aug 23 11:51:24 ex6 cachefilesd[5919]: Refilling cull table
Aug 23 11:51:24 ex6 cachefilesd[5919]: Scan complete
Aug 23 11:51:33 ex6 kernel: [55598.253247] BUG: soft lockup - CPU#6
stuck for 22s! [httpd2-event:15597]
Aug 23 11:51:33 ex6 kernel: [55598.253708] Modules linked in: sg st

Aug 23 11:51:33 ex6 kernel: log xt_multiport xt_mark xt_mac xt_limit

Aug 23 11:51:33 ex6 kernel: [55598.253818] CPU 6
Aug 23 11:51:33 ex6 kernel: [55598.253822] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:33 ex6 kernel: [55598.253824] RIP:
0010:[<ffffffff81562be5>]  [<ffffffff81562be5>] _raw_spin_lock_bh+0x25/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253833] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:33 ex6 kernel: [55598.253835] RAX: 0000000000000014 RBX:
ffff880407ba4cc0 RCX: 0000000000000001
Aug 23 11:51:33 ex6 kernel: [55598.253837] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:33 ex6 kernel: [55598.253839] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:33 ex6 kernel: [55598.253841] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff880407ba4d08
Aug 23 11:51:33 ex6 kernel: [55598.253842] R13: ffff880407ba4cc0 R14:
0000000000000292 R15: 0000293da5c4986a
Aug 23 11:51:33 ex6 kernel: [55598.253845] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253847] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:33 ex6 kernel: [55598.253849] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:33 ex6 kernel: [55598.253850] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:33 ex6 kernel: [55598.253852] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:33 ex6 kernel: [55598.253855] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:33 ex6 kernel: [55598.253856] Stack:
Aug 23 11:51:33 ex6 kernel: [55598.253857]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:33 ex6 kernel: [55598.253861]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:33 ex6 kernel: [55598.253865]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:33 ex6 kernel: [55598.253868] Call Trace:
Aug 23 11:51:33 ex6 kernel: [55598.253877]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:33 ex6 kernel: [55598.253885]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:33 ex6 kernel: [55598.253892]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:33 ex6 kernel: [55598.253900]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:33 ex6 kernel: [55598.253909]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:33 ex6 kernel: [55598.253910] Code: c3 0f 1f 44 00 00 53 48
89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2
74 0e 0f 1f 40 00 f3 90 0f b7 03 <66> 39 d0 75 f6 5b c3 90 90 90 90 48
c7 c7 60 74 a5 81 e9 34 d9
Aug 23 11:51:46 ex6 kernel: [55610.629219] INFO: rcu_sched self-detected
stall on CPU { 7}  (t=2580171 jiffies)
Aug 23 11:51:46 ex6 kernel: [55610.629225] sending NMI to all CPUs:
Aug 23 11:51:46 ex6 kernel: [55610.629231] NMI backtrace for cpu 6
Aug 23 11:51:46 ex6 kernel: [55610.629234] CPU 6
Aug 23 11:51:46 ex6 kernel: [55610.629239] Pid: 15597, comm:
httpd2-event Not tainted 3.6.0-rc2-1-default #1 System manufacturer
System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629242] RIP:
0010:[<ffffffff81562be0>]  [<ffffffff81562be0>] _raw_spin_lock_bh+0x20/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629252] RSP: 0018:ffff88018fc79f08
EFLAGS: 00000297
Aug 23 11:51:46 ex6 kernel: [55610.629255] RAX: 0000000000000014 RBX:
ffff880194f4c090 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629257] RDX: 0000000000000015 RSI:
0000000000000000 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629260] RBP: 0000000000000002 R08:
ffff8804089938f0 R09: 0000000000003ced
Aug 23 11:51:46 ex6 kernel: [55610.629262] R10: ffff88041e836800 R11:
0000000000000202 R12: ffff8804067d2580
Aug 23 11:51:46 ex6 kernel: [55610.629265] R13: 00000000ffffffea R14:
00007fcd60a664a0 R15: 00007fcd51db52b8
Aug 23 11:51:46 ex6 kernel: [55610.629268] FS:  00007fcd524e4700(0000)
GS:ffff88041f380000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629271] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629274] CR2: 00007fcd51b59000 CR3:
0000000208ab4000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629277] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629279] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629282] Process httpd2-event (pid:
15597, threadinfo ffff88018fc78000, task ffff880194f5a300)
Aug 23 11:51:46 ex6 kernel: [55610.629284] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629286]  ffff880194f4c040
ffffffff8145d45d ffff880194f4c040 ffffffff814d120e
Aug 23 11:51:46 ex6 kernel: [55610.629292]  ffff880194f5a300
ffff8804067d2580 0000000000000001 00007fcd60c8eaa0
Aug 23 11:51:46 ex6 kernel: [55610.629297]  0004c7e9679f5fbf
ffffffff8145ac14 00007fcd589c02e0 0000000100000000
Aug 23 11:51:46 ex6 kernel: [55610.629303] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629314]  [<ffffffff8145d45d>]
lock_sock_nested+0xd/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629323]  [<ffffffff814d120e>]
inet_shutdown+0x3e/0x130
Aug 23 11:51:46 ex6 kernel: [55610.629330]  [<ffffffff8145ac14>]
sys_shutdown+0x74/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629340]  [<ffffffff8156a47d>]
system_call_fastpath+0x1a/0x1f
Aug 23 11:51:46 ex6 kernel: [55610.629364]  [<00007fcd5fc42477>]
0x7fcd5fc42476
Aug 23 11:51:46 ex6 kernel: [55610.629366] Code: 66 39 d0 75 f6 c3 0f 1f
44 00 00 53 48 89 fb e8 67 88 ae ff b8 00 00 01 00 f0 0f c1 03 89 c2 c1
ea 10 66 39 c2 74 0e 0f 1f 40 00 <f3> 90 0f b7 03 66 39 d0 75 f6 5b c3
90 90 90 90 48 c7 c7 60 74
Aug 23 11:51:46 ex6 kernel: [55610.629413] NMI backtrace for cpu 7
Aug 23 11:51:46 ex6 kernel: [55610.629415] CPU 7
Aug 23 11:51:46 ex6 kernel: [55610.629420] Pid: 0, comm: swapper/7 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629422] RIP:
0010:[<ffffffff812df109>]  [<ffffffff812df109>] find_next_bit+0x39/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629432] RSP: 0018:ffff88041f3c3938
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629434] RAX: 0000000000000000 RBX:
ffff88041f3ccfe0 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629437] RDX: 0000000000000001 RSI:
0000000000000200 RDI: 0000000000000200
Aug 23 11:51:46 ex6 kernel: [55610.629440] RBP: ffff88041f20d020 R08:
ffff88041f3ccfe0 R09: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629442] R10: 0000000000000000 R11:
000000000000c2e2 R12: 000000000000d060
Aug 23 11:51:46 ex6 kernel: [55610.629445] R13: 0000000000080000 R14:
00000000000000ff R15: 0000000000000007
Aug 23 11:51:46 ex6 kernel: [55610.629448] FS:  0000000000000000(0000)
GS:ffff88041f3c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629451] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629454] CR2: 00007fcd2c062ac0 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629456] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629459] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629462] Process swapper/7 (pid: 0,
threadinfo ffff88040abc4000, task ffff88040abc2400)
Aug 23 11:51:46 ex6 kernel: [55610.629464] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629465]  ffffffff8102a4e7
0000000081a2cec0 000000000000d020 0000000200000007
Aug 23 11:51:46 ex6 kernel: [55610.629471]  0000000000000082
ffff88041f3c39b0 0000000000002710 ffffffff81a2ce00
Aug 23 11:51:46 ex6 kernel: [55610.629476]  ffff88040abc4000
0000000000000007 ffff88040abc4000 ffffffff81a2cec0
Aug 23 11:51:46 ex6 kernel: [55610.629482] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629494]  [<ffffffff8102a4e7>]
__x2apic_send_IPI_mask+0xb7/0x1a0
Aug 23 11:51:46 ex6 kernel: [55610.629503]  [<ffffffff810260ff>]
arch_trigger_all_cpu_backtrace+0x4f/0x90
Aug 23 11:51:46 ex6 kernel: [55610.629510]  [<ffffffff810d6d20>]
rcu_pending+0x210/0x560
Aug 23 11:51:46 ex6 kernel: [55610.629519]  [<ffffffff810d78c0>]
rcu_check_callbacks+0x90/0x110
Aug 23 11:51:46 ex6 kernel: [55610.629528]  [<ffffffff8105467f>]
update_process_times+0x3f/0x80
Aug 23 11:51:46 ex6 kernel: [55610.629536]  [<ffffffff810971fb>]
tick_sched_timer+0x6b/0xe0
Aug 23 11:51:46 ex6 kernel: [55610.629544]  [<ffffffff810699a6>]
__run_hrtimer+0x66/0x1d0
Aug 23 11:51:46 ex6 kernel: [55610.629552]  [<ffffffff8106a299>]
hrtimer_interrupt+0xe9/0x210
Aug 23 11:51:46 ex6 kernel: [55610.629559]  [<ffffffff81024e93>]
smp_apic_timer_interrupt+0x63/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629567]  [<ffffffff8156b0fa>]
apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629578]  [<ffffffff814c2c78>]
tcp_slow_start+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629585]  [<ffffffff814b6260>]
tcp_ack+0x560/0x11e0
Aug 23 11:51:46 ex6 kernel: [55610.629593]  [<ffffffff814b715d>]
tcp_rcv_established+0x27d/0x8e0
Aug 23 11:51:46 ex6 kernel: [55610.629601]  [<ffffffff814c045e>]
tcp_v4_do_rcv+0xde/0x340
Aug 23 11:51:46 ex6 kernel: [55610.629609]  [<ffffffff814c173b>]
tcp_v4_rcv+0x59b/0x850
Aug 23 11:51:46 ex6 kernel: [55610.629617]  [<ffffffff8149e156>]
ip_local_deliver_finish+0xd6/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629625]  [<ffffffff8146eb5e>]
__netif_receive_skb+0x67e/0x7e0
Aug 23 11:51:46 ex6 kernel: [55610.629633]  [<ffffffff8146ed61>]
process_backlog+0xa1/0x180
Aug 23 11:51:46 ex6 kernel: [55610.629639]  [<ffffffff8146f830>]
net_rx_action+0x140/0x230
Aug 23 11:51:46 ex6 kernel: [55610.629647]  [<ffffffff8104be2e>]
__do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629655]  [<ffffffff8156b7ec>]
call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629662]  [<ffffffff810045a5>]
do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629669]  [<ffffffff8104c1e5>]
irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629677]  [<ffffffff8156b2ba>]
call_function_single_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629686]  [<ffffffff813303cc>]
intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629695]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629702]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629710]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629713] Code: d1 49 89 d1 48 c1 e9 06
49 83 e1 c0 4c 8d 04 cf 48 89 f7 48 89 d1 4c 29 cf 83 e1 3f 74 40 48 c7
c0 ff ff ff ff 48 d3 e0 49 23 00 <48> 83 ff 3f 76 41 48 85 c0 75 52 49
83 c0 08 48 83 ef 40 49 83
Aug 23 11:51:46 ex6 kernel: [55610.629761] NMI backtrace for cpu 1
Aug 23 11:51:46 ex6 kernel: [55610.629765] CPU 1
Aug 23 11:51:46 ex6 kernel: [55610.629769] Pid: 0, comm: swapper/1 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629772] RIP:
0010:[<ffffffff815628dd>]  [<ffffffff815628dd>] _raw_spin_lock+0x1d/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629781] RSP: 0018:ffff88041f243e48
EFLAGS: 00000293
Aug 23 11:51:46 ex6 kernel: [55610.629784] RAX: 0000000000000014 RBX:
ffff880194f4c040 RCX: 0000000100c76001
Aug 23 11:51:46 ex6 kernel: [55610.629787] RDX: 0000000000000016 RSI:
0000000000000286 RDI: ffff880194f4c090
Aug 23 11:51:46 ex6 kernel: [55610.629789] RBP: ffff88040abd4000 R08:
ffff88040701a29c R09: 0000000000000028
Aug 23 11:51:46 ex6 kernel: [55610.629792] R10: dead000000200200 R11:
0000000000000000 R12: 0000000000000100
Aug 23 11:51:46 ex6 kernel: [55610.629794] R13: ffffffff814bd160 R14:
ffff880194f4c040 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629798] FS:  0000000000000000(0000)
GS:ffff88041f240000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629800] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.629803] CR2: 00007fa566135010 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.629806] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.629808] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.629811] Process swapper/1 (pid: 0,
threadinfo ffff88040aba6000, task ffff88040aba4280)
Aug 23 11:51:46 ex6 kernel: [55610.629813] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.629815]  ffffffff814bd171
ffff88040abd4000 0000000000000100 ffff880194f4c200
Aug 23 11:51:46 ex6 kernel: [55610.629820]  ffffffff81053ac1
0000000000000000 ffff88041f243ea0 ffff88040aba7fd8
Aug 23 11:51:46 ex6 kernel: [55610.629825]  ffff88040aba7fd8
ffff88040aba7fd8 ffff88040abd5028 ffff88041f243ea0
Aug 23 11:51:46 ex6 kernel: [55610.629831] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629841]  [<ffffffff814bd171>]
tcp_keepalive_timer+0x11/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629850]  [<ffffffff81053ac1>]
run_timer_softirq+0x151/0x350
Aug 23 11:51:46 ex6 kernel: [55610.629857]  [<ffffffff8104be2e>]
__do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629865]  [<ffffffff8156b7ec>]
call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629871]  [<ffffffff810045a5>]
do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629879]  [<ffffffff8104c1e5>]
irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629886]  [<ffffffff81024e98>]
smp_apic_timer_interrupt+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629893]  [<ffffffff8156b0fa>]
apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629902]  [<ffffffff813303cc>]
intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629911]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629918]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629925]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629929] Code: 90 90 90 90 90 90 90 90
90 90 90 90 90 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74
0f 0f 1f 44 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 c3 66 66 66 66 2e 0f
1f 84 00 00 00 00 00 66 83
Aug 23 11:51:46 ex6 kernel: [55610.629976] NMI backtrace for cpu 2
Aug 23 11:51:46 ex6 kernel: [55610.629979] CPU 2
Aug 23 11:51:46 ex6 kernel: [55610.629983] Pid: 0, comm: swapper/2 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629986] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629992] RSP: 0018:ffff88040ababe78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629994] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629997] RDX: 0000000000000000 RSI:
ffff88040ababfd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.629999] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e3c
Aug 23 11:51:46 ex6 kernel: [55610.630002] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630004] R13: 0000000000000020 R14:
12acf11f7efbdbfc R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630007] FS:  0000000000000000(0000)
GS:ffff88041f280000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630010] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630012] CR2: 00007fa566135010 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630015] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630017] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630020] Process swapper/2 (pid: 0,
threadinfo ffff88040abaa000, task ffff88040aba82c0)
Aug 23 11:51:46 ex6 kernel: [55610.630022] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630024]  0000000077359400
ffffffff81436a70 0000000000000000 0000000200f31e03
Aug 23 11:51:46 ex6 kernel: [55610.630029]  0000000000000000
0000000000f31e03 ffff88040ababfd8 ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630034]  0000000000000003
0000000000000002 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630039] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630048]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630055]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630062]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630066] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630113] NMI backtrace for cpu 5
Aug 23 11:51:46 ex6 kernel: [55610.630115] CPU 5
Aug 23 11:51:46 ex6 kernel: [55610.630120] Pid: 0, comm: swapper/5 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630122] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630128] RSP: 0018:ffff88040abb9e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630130] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630133] RDX: 0000000000000000 RSI:
ffff88040abb9fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630135] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000005634
Aug 23 11:51:46 ex6 kernel: [55610.630138] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630140] R13: 0000000000000020 R14:
12acf11f7e9e3c16 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630143] FS:  0000000000000000(0000)
GS:ffff88041f340000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630146] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630149] CR2: 00007fd29bfff000 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630151] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630154] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630157] Process swapper/5 (pid: 0,
threadinfo ffff88040abb8000, task ffff88040abb6380)
Aug 23 11:51:46 ex6 kernel: [55610.630158] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630160]  0000000077359400
ffffffff81436a70 0000000000000000 000000050150be3d
Aug 23 11:51:46 ex6 kernel: [55610.630165]  0000000000000000
000000000150be3d ffff88040abb9fd8 ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630170]  0000000000000003
0000000000000005 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630176] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630184]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630191]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630198]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630201] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630248] NMI backtrace for cpu 3
Aug 23 11:51:46 ex6 kernel: [55610.630251] CPU 3
Aug 23 11:51:46 ex6 kernel: [55610.630255] Pid: 0, comm: swapper/3 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630257] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630263] RSP: 0018:ffff88040abb1e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630266] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630268] RDX: 0000000000000000 RSI:
ffff88040abb1fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630271] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e43
Aug 23 11:51:46 ex6 kernel: [55610.630273] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630276] R13: 0000000000000020 R14:
12acf11f7efbc356 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630279] FS:  0000000000000000(0000)
GS:ffff88041f2c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630282] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630284] CR2: 00007fe7d784ba68 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630287] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630289] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630292] Process swapper/3 (pid: 0,
threadinfo ffff88040abb0000, task ffff88040abae300)
Aug 23 11:51:46 ex6 kernel: [55610.630294] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630296]  0000000077359400
ffffffff81436a70 0000000000000000 0000000300f3366a
Aug 23 11:51:46 ex6 kernel: [55610.630301]  0000000000000000
0000000000f3366a ffff88040abb1fd8 ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630306]  0000000000000003
0000000000000003 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630311] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630320]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630327]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630333]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630337] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630385] NMI backtrace for cpu 0
Aug 23 11:51:46 ex6 kernel: [55610.630388] CPU 0
Aug 23 11:51:46 ex6 kernel: [55610.630393] Pid: 0, comm: swapper/0 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630396] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630403] RSP: 0018:ffffffff81a01e88
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630406] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630409] RDX: 0000000000000000 RSI:
ffffffff81a01fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630411] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000005d52
Aug 23 11:51:46 ex6 kernel: [55610.630413] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630416] R13: 0000000000000020 R14:
12acf11f7f769680 R15: 0000000001fe8018
Aug 23 11:51:46 ex6 kernel: [55610.630419] FS:  0000000000000000(0000)
GS:ffff88041f200000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630422] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630425] CR2: 00007f03abb83000 CR3:
0000000001a0c000 CR4: 00000000000407f0
Aug 23 11:51:46 ex6 kernel: [55610.630427] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630430] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630433] Process swapper/0 (pid: 0,
threadinfo ffffffff81a00000, task ffffffff81a14420)
Aug 23 11:51:46 ex6 kernel: [55610.630435] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630436]  0000000077359400
ffffffff81436a70 0000000000000000 00000000016c88d5
Aug 23 11:51:46 ex6 kernel: [55610.630442]  0000000000000000
00000000016c88d5 ffffffff81a01fd8 ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630447]  0000000000000003
0000000000000000 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630453] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630464]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630471]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630480]  [<ffffffff81ac0ba2>]
start_kernel+0x391/0x39c
Aug 23 11:51:46 ex6 kernel: [55610.630488]  [<ffffffff81ac0436>]
x86_64_start_kernel+0x105/0x114
Aug 23 11:51:46 ex6 kernel: [55610.630492] Code: 28 e0 ff ff 83 e2 08 75
22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8
08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8
76 a8 d1 ff 48 89 04 24 48
Aug 23 11:51:46 ex6 kernel: [55610.630539] NMI backtrace for cpu 4
Aug 23 11:51:46 ex6 kernel: [55610.630542] CPU 4
Aug 23 11:51:46 ex6 kernel: [55610.630546] Pid: 0, comm: swapper/4 Not
tainted 3.6.0-rc2-1-default #1 System manufacturer System Product
Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630549] RIP:
0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630555] RSP: 0018:ffff88040abb5e78
EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630557] RAX: 0000000000000020 RBX:
0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630560] RDX: 0000000000000000 RSI:
ffff88040abb5fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630562] RBP: 0000000000000003 R08:
0000000000000003 R09: 0000000000003e1e
Aug 23 11:51:46 ex6 kernel: [55610.630565] R10: 0000000000000001 R11:
0000000000000000 R12: ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630568] R13: 0000000000000020 R14:
12acf11f7efc52d2 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630571] FS:  0000000000000000(0000)
GS:ffff88041f300000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630573] CS:  0010 DS: 0000 ES: 0000
CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630576] CR2: 00007fd29bfff000 CR3:
0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630578] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630581] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630584] Process swapper/4 (pid: 0,
threadinfo ffff88040abb4000, task ffff88040abb2340)
Aug 23 11:51:46 ex6 kernel: [55610.630586] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630588]  0000000077359400
ffffffff81436a70 0000000000000000 0000000400f2a60d
Aug 23 11:51:46 ex6 kernel: [55610.630593]  0000000000000000
0000000000f2a60d ffff88040abb5fd8 ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630598]  0000000000000003
0000000000000004 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630604] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630614]  [<ffffffff8143566d>]
cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630622]  [<ffffffff8100b715>]
cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630629]  [<ffffffff81550547>]
start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630633] Code: 28 e0 ff ff 83 e2 08 75

22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48

Quoted 134 lines

Aug 23 11:51:46 ex6 kernel: [55610.629820]  ffffffff81053ac1 0000000000000000 ffff88041f243ea0 ffff88040aba7fd8
Aug 23 11:51:46 ex6 kernel: [55610.629825]  ffff88040aba7fd8 ffff88040aba7fd8 ffff88040abd5028 ffff88041f243ea0
Aug 23 11:51:46 ex6 kernel: [55610.629831] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.629841]  [<ffffffff814bd171>] tcp_keepalive_timer+0x11/0x250
Aug 23 11:51:46 ex6 kernel: [55610.629850]  [<ffffffff81053ac1>] run_timer_softirq+0x151/0x350
Aug 23 11:51:46 ex6 kernel: [55610.629857]  [<ffffffff8104be2e>] __do_softirq+0xbe/0x1f0
Aug 23 11:51:46 ex6 kernel: [55610.629865]  [<ffffffff8156b7ec>] call_softirq+0x1c/0x30
Aug 23 11:51:46 ex6 kernel: [55610.629871]  [<ffffffff810045a5>] do_softirq+0x75/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629879]  [<ffffffff8104c1e5>] irq_exit+0xa5/0xb0
Aug 23 11:51:46 ex6 kernel: [55610.629886]  [<ffffffff81024e98>] smp_apic_timer_interrupt+0x68/0xa0
Aug 23 11:51:46 ex6 kernel: [55610.629893]  [<ffffffff8156b0fa>] apic_timer_interrupt+0x6a/0x70
Aug 23 11:51:46 ex6 kernel: [55610.629902]  [<ffffffff813303cc>] intel_idle+0xec/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629911]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.629918]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.629925]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.629929] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 0f 0f 1f 44 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 83 
Aug 23 11:51:46 ex6 kernel: [55610.629976] NMI backtrace for cpu 2
Aug 23 11:51:46 ex6 kernel: [55610.629979] CPU 2 
Aug 23 11:51:46 ex6 kernel: [55610.629983] Pid: 0, comm: swapper/2 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.629986] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.629992] RSP: 0018:ffff88040ababe78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.629994] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.629997] RDX: 0000000000000000 RSI: ffff88040ababfd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.629999] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e3c
Aug 23 11:51:46 ex6 kernel: [55610.630002] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630004] R13: 0000000000000020 R14: 12acf11f7efbdbfc R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630007] FS:  0000000000000000(0000) GS:ffff88041f280000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630010] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630012] CR2: 00007fa566135010 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630020] Process swapper/2 (pid: 0, threadinfo ffff88040abaa000, task ffff88040aba82c0)
Aug 23 11:51:46 ex6 kernel: [55610.630022] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630024]  0000000077359400 ffffffff81436a70 0000000000000000 0000000200f31e03
Aug 23 11:51:46 ex6 kernel: [55610.630029]  0000000000000000 0000000000f31e03 ffff88040ababfd8 ffff88041f298f70
Aug 23 11:51:46 ex6 kernel: [55610.630034]  0000000000000003 0000000000000002 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630039] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630048]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630055]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630062]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630066] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630113] NMI backtrace for cpu 5
Aug 23 11:51:46 ex6 kernel: [55610.630115] CPU 5 
Aug 23 11:51:46 ex6 kernel: [55610.630120] Pid: 0, comm: swapper/5 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630122] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630128] RSP: 0018:ffff88040abb9e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630130] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630133] RDX: 0000000000000000 RSI: ffff88040abb9fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630135] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000005634
Aug 23 11:51:46 ex6 kernel: [55610.630138] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630140] R13: 0000000000000020 R14: 12acf11f7e9e3c16 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630143] FS:  0000000000000000(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630146] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630149] CR2: 00007fd29bfff000 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630154] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630157] Process swapper/5 (pid: 0, threadinfo ffff88040abb8000, task ffff88040abb6380)
Aug 23 11:51:46 ex6 kernel: [55610.630158] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630160]  0000000077359400 ffffffff81436a70 0000000000000000 000000050150be3d
Aug 23 11:51:46 ex6 kernel: [55610.630165]  0000000000000000 000000000150be3d ffff88040abb9fd8 ffff88041f358f70
Aug 23 11:51:46 ex6 kernel: [55610.630170]  0000000000000003 0000000000000005 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630176] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630184]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630191]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630198]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630201] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630248] NMI backtrace for cpu 3
Aug 23 11:51:46 ex6 kernel: [55610.630251] CPU 3 
Aug 23 11:51:46 ex6 kernel: [55610.630255] Pid: 0, comm: swapper/3 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630257] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630263] RSP: 0018:ffff88040abb1e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630266] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630268] RDX: 0000000000000000 RSI: ffff88040abb1fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630271] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e43
Aug 23 11:51:46 ex6 kernel: [55610.630273] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630276] R13: 0000000000000020 R14: 12acf11f7efbc356 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630279] FS:  0000000000000000(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630282] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630284] CR2: 00007fe7d784ba68 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630289] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630292] Process swapper/3 (pid: 0, threadinfo ffff88040abb0000, task ffff88040abae300)
Aug 23 11:51:46 ex6 kernel: [55610.630294] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630296]  0000000077359400 ffffffff81436a70 0000000000000000 0000000300f3366a
Aug 23 11:51:46 ex6 kernel: [55610.630301]  0000000000000000 0000000000f3366a ffff88040abb1fd8 ffff88041f2d8f70
Aug 23 11:51:46 ex6 kernel: [55610.630306]  0000000000000003 0000000000000003 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630311] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630320]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630327]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630333]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630337] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630385] NMI backtrace for cpu 0
Aug 23 11:51:46 ex6 kernel: [55610.630388] CPU 0 
Aug 23 11:51:46 ex6 kernel: [55610.630393] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630396] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630403] RSP: 0018:ffffffff81a01e88  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630406] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630409] RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630411] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000005d52
Aug 23 11:51:46 ex6 kernel: [55610.630413] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630416] R13: 0000000000000020 R14: 12acf11f7f769680 R15: 0000000001fe8018
Aug 23 11:51:46 ex6 kernel: [55610.630419] FS:  0000000000000000(0000) GS:ffff88041f200000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630422] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630425] CR2: 00007f03abb83000 CR3: 0000000001a0c000 CR4: 00000000000407f0
Aug 23 11:51:46 ex6 kernel: [55610.630427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630430] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630433] Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a14420)
Aug 23 11:51:46 ex6 kernel: [55610.630435] Stack:
Aug 23 11:51:46 ex6 kernel: [55610.630436]  0000000077359400 ffffffff81436a70 0000000000000000 00000000016c88d5
Aug 23 11:51:46 ex6 kernel: [55610.630442]  0000000000000000 00000000016c88d5 ffffffff81a01fd8 ffff88041f218f70
Aug 23 11:51:46 ex6 kernel: [55610.630447]  0000000000000003 0000000000000000 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630453] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630464]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630471]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630480]  [<ffffffff81ac0ba2>] start_kernel+0x391/0x39c
Aug 23 11:51:46 ex6 kernel: [55610.630488]  [<ffffffff81ac0436>] x86_64_start_kernel+0x105/0x114
Aug 23 11:51:46 ex6 kernel: [55610.630492] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48 
Aug 23 11:51:46 ex6 kernel: [55610.630539] NMI backtrace for cpu 4
Aug 23 11:51:46 ex6 kernel: [55610.630542] CPU 4 
Aug 23 11:51:46 ex6 kernel: [55610.630546] Pid: 0, comm: swapper/4 Not tainted 3.6.0-rc2-1-default #1 System manufacturer System Product Name/P8B WS
Aug 23 11:51:46 ex6 kernel: [55610.630549] RIP: 0010:[<ffffffff8133039a>]  [<ffffffff8133039a>] intel_idle+0xba/0x160
Aug 23 11:51:46 ex6 kernel: [55610.630555] RSP: 0018:ffff88040abb5e78  EFLAGS: 00000046
Aug 23 11:51:46 ex6 kernel: [55610.630557] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Aug 23 11:51:46 ex6 kernel: [55610.630560] RDX: 0000000000000000 RSI: ffff88040abb5fd8 RDI: ffffffff81a17640
Aug 23 11:51:46 ex6 kernel: [55610.630562] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000003e1e
Aug 23 11:51:46 ex6 kernel: [55610.630565] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630568] R13: 0000000000000020 R14: 12acf11f7efc52d2 R15: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630571] FS:  0000000000000000(0000) GS:ffff88041f300000(0000) knlGS:0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630573] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 23 11:51:46 ex6 kernel: [55610.630576] CR2: 00007fd29bfff000 CR3: 0000000001a0c000 CR4: 00000000000407e0
Aug 23 11:51:46 ex6 kernel: [55610.630578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 23 11:51:46 ex6 kernel: [55610.630581] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 23 11:51:46 ex6 kernel: [55610.630584] Process swapper/4 (pid: 0, threadinfo ffff88040abb4000, task ffff88040abb2340)
Aug 23 11:51:46 ex6 kernel: [55610.630586] Stack:

Quoted 8 lines

Aug 23 11:51:46 ex6 kernel: [55610.630588]  0000000077359400 ffffffff81436a70 0000000000000000 0000000400f2a60d
Aug 23 11:51:46 ex6 kernel: [55610.630593]  0000000000000000 0000000000f2a60d ffff88040abb5fd8 ffff88041f318f70
Aug 23 11:51:46 ex6 kernel: [55610.630598]  0000000000000003 0000000000000004 ffffffff81a52300 ffffffff8143566d
Aug 23 11:51:46 ex6 kernel: [55610.630604] Call Trace:
Aug 23 11:51:46 ex6 kernel: [55610.630614]  [<ffffffff8143566d>] cpuidle_idle_call+0x9d/0x240
Aug 23 11:51:46 ex6 kernel: [55610.630622]  [<ffffffff8100b715>] cpu_idle+0x65/0xd0
Aug 23 11:51:46 ex6 kernel: [55610.630629]  [<ffffffff81550547>] start_secondary+0x1fe/0x203
Aug 23 11:51:46 ex6 kernel: [55610.630633] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <e8> e1 f9 d5 ff 48 89 c7 4c 29 f7 e8 76 a8 d1 ff 48 89 04 24 48

Quoted 5 lines

Tested kernels with the problem 3.6.RC2 from SUSE, and today's linus vanilla tree.
Server needs reboot. (or let's say two completely different boxes have the same problem)

This looks like an infinite loop in tcp_slow_start()

This could happen if tp->snd_cwnd is zero

Bug probably introduced with commit commit
9dc274151a548ffd215caecec5a8872db8799447

Author: Eric Dumazet <[email protected]>
Date:   Fri Jul 20 05:02:33 2012 +0000

tcp: fix ABC in tcp_slow_start()

When/if sysctl_tcp_abc > 1, we expect to increase cwnd by 2 if the received ACK acknowledges more than 2*MSS bytes, in tcp_slow_start()

Problem is this RFC 3465 statement is not correctly coded, as the while () loop increases snd_cwnd one by one.

Add a new variable to avoid this off-by one error.

Signed-off-by Eric Dumazet <[email protected]>

Cc Tom Herbert <[email protected]>

Cc Yuchung Cheng <[email protected]>

Cc Neal Cardwell <[email protected]>

Cc Nandita Dukkipati <[email protected]>

Cc John Heffner <[email protected]>

Cc Stephen Hemminger <[email protected]>

Acked-by Yuchung Cheng <[email protected]>

Acked-by Neal Cardwell <[email protected]>

Signed-off-by David S. Miller <[email protected]>

net/ipv4/tcp_cong.c +3 -2

index 04dbd7a..4d4db16 100644
@@ -307,6 +307,7 @@ EXPORT_SYMBOL_GPL(tcp_is_cwnd_limited);
 void tcp_slow_start(struct tcp_sock *tp)
 {
        int cnt; /* increase in packets */
+       unsigned int delta = 0;
 
        /* RFC3465: ABC Slow start
         * Increase only after a full MSS of bytes is acked

@@ -333,9 +334,9 @@ void tcp_slow_start(struct tcp_sock *tp)
        tp->snd_cwnd_cnt += cnt;
        while (tp->snd_cwnd_cnt >= tp->snd_cwnd) {
                tp->snd_cwnd_cnt -= tp->snd_cwnd;
-               if (tp->snd_cwnd < tp->snd_cwnd_clamp)
-                       tp->snd_cwnd++;
+               delta++;
        }
+       tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
 }
 EXPORT_SYMBOL_GPL(tcp_slow_start);

Oh, I now see the bug, I'll send a patch asap

Eric Dumazet

2012-08-25 09:14 UTC #2 lore raw

From: Eric Dumazet <[email protected]>

Quoted 5 lines · Eric Dumazet

Quoted 2 lines · Cristian Rodríguez
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap

Please try the following fix.

Thanks !

[PATCH] tcp: tcp_slow_start() should not decrease snd_cwnd

Cristian Rodríguez reported various lockups in TCP stack, introduced by commit 9dc274151a548 (tcp: fix ABC in tcp_slow_start())

We could exit tcp_slow_start() with a zeroed snd_cwnd, and next time we enter tcp_slow_start(), we run an infinite loop.

Reported-by Cristian Rodríguez <[email protected]>

Cc Yuchung Cheng <[email protected]>

Cc Neal Cardwell <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

---
 net/ipv4/tcp_cong.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

net/ipv4/tcp_cong.c +3 -1

index 1432cdb..f20a761 100644
@@ -309,6 +309,7 @@ void tcp_slow_start(struct tcp_sock *tp)
 {
 	int cnt; /* increase in packets */
 	unsigned int delta = 0;
+	u32 new_snd_cwnd;
 
 	/* RFC3465: ABC Slow start
 	 * Increase only after a full MSS of bytes is acked

@@ -337,7 +338,8 @@ void tcp_slow_start(struct tcp_sock *tp)
 		tp->snd_cwnd_cnt -= tp->snd_cwnd;
 		delta++;
 	}
-	tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+	new_snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+	tp->snd_cwnd = max(tp->snd_cwnd, new_snd_cwnd);
 }
 EXPORT_SYMBOL_GPL(tcp_slow_start);

Eric Dumazet

2012-08-25 11:47 UTC #3 lore raw

Quoted 4 lines · Eric Dumazet

From: Eric Dumazet <[email protected]>
Quoted 1 line · Eric Dumazet
On Fri, 2012-08-24 at 20:50 -0400, Cristian Rodríguez wrote:

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap
Please try the following fix.
Thanks !

Well, this v2 seems cleaner :

[PATCH v2] tcp: tcp_slow_start() should not decrease snd_cwnd

Cristian Rodríguez reported various lockups in TCP stack, introduced by commit 9dc274151a548 (tcp: fix ABC in tcp_slow_start())

We could exit tcp_slow_start() with a zeroed snd_cwnd, and next time we enter tcp_slow_start(), we run an infinite loop.

Reported-by Cristian Rodríguez <[email protected]>

Cc Yuchung Cheng <[email protected]>

Cc Neal Cardwell <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

---
 net/ipv4/tcp_cong.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

net/ipv4/tcp_cong.c +1 -1

index 1432cdb..e656c72 100644
@@ -337,7 +337,7 @@ void tcp_slow_start(struct tcp_sock *tp)
 		tp->snd_cwnd_cnt -= tp->snd_cwnd;
 		delta++;
 	}
-	tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+	tp->snd_cwnd = clamp(tp->snd_cwnd + delta, tp->snd_cwnd, tp->snd_cwnd_clamp);
 }
 EXPORT_SYMBOL_GPL(tcp_slow_start);

Cristian Rodríguez

author

2012-08-25 17:47 UTC #4 lore raw

Quoted 5 lines · Eric Dumazet

Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Quoted 1 line · Eric Dumazet
On Fri, 2012-08-24 at 20:50 -0400, Cristian Rodríguez wrote:

Quoted 8 lines

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap
Please try the following fix.
Thanks !

Eric, thanks for your work, I have deployed the fix in one machine and will keep an eye for any other anomaly.

As it happends around twice a day, it would take a while to determine if this issue has gone away.

Keep up great work :-)

Eric Dumazet

2012-08-25 17:52 UTC #5 lore raw

In reply to Eric Dumazet

Quoted 38 lines · Eric Dumazet

Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Quoted 1 line · Eric Dumazet
On Fri, 2012-08-24 at 20:50 -0400, Cristian Rodríguez wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap
Please try the following fix.
Thanks !
Well, this v2 seems cleaner :
[PATCH v2] tcp: tcp_slow_start() should not decrease snd_cwnd
Cristian Rodríguez reported various lockups in TCP stack, introduced by commit 9dc274151a548 (tcp: fix ABC in tcp_slow_start())
We could exit tcp_slow_start() with a zeroed snd_cwnd, and next time we enter tcp_slow_start(), we run an infinite loop.
Reported-by Cristian Rodríguez <[email protected]>
Cc Yuchung Cheng <[email protected]>
Cc Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp_cong.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/tcp_cong.c +0 -0
index 1432cdb..e656c72 100644

net/ipv4/tcp_cong.c void tcp_slow_start(struct tcp_sock *tp) · line 343

Quoted 8 lines

net/ipv4/tcp_cong.c +1 -1

@@ -337,7 +337,7 @@ void tcp_slow_start(struct tcp_sock *tp)
 		tp->snd_cwnd_cnt -= tp->snd_cwnd;
 		delta++;
 	}
-	tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+	tp->snd_cwnd = clamp(tp->snd_cwnd + delta, tp->snd_cwnd, tp->snd_cwnd_clamp);
 }
 EXPORT_SYMBOL_GPL(tcp_slow_start);

Hmm...

We probably have a bug in tcp_metrics.c, because snd_cwnd_clamp should not be zero.

With RCU, it seems following code in tcpm_new() is racy :

tm->tcpm_addr = *addr;
...
tcpm_suck_dst(tm, dst);

Coupled with the code in tcpm_suck_dst(tm, dst)

static void tcpm_suck_dst(struct tcp_metrics_block *tm, struct dst_entry *dst)

{
        u32 val;

        tm->tcpm_stamp = jiffies;

        val = 0;
        if (dst_metric_locked(dst, RTAX_RTT))
                val |= 1 << TCP_METRIC_RTT;
        if (dst_metric_locked(dst, RTAX_RTTVAR))
                val |= 1 << TCP_METRIC_RTTVAR;
        if (dst_metric_locked(dst, RTAX_SSTHRESH))
                val |= 1 << TCP_METRIC_SSTHRESH;
        if (dst_metric_locked(dst, RTAX_CWND))
                val |= 1 << TCP_METRIC_CWND;
        if (dst_metric_locked(dst, RTAX_REORDERING))
                val |= 1 << TCP_METRIC_REORDERING;
        tm->tcpm_lock = val;

// HERE we set tcpm_lock before the tcpm_vals[]

        tm->tcpm_vals[TCP_METRIC_RTT] = dst_metric_raw(dst, RTAX_RTT);
        tm->tcpm_vals[TCP_METRIC_RTTVAR] = dst_metric_raw(dst, RTAX_RTTVAR);
        tm->tcpm_vals[TCP_METRIC_SSTHRESH] = dst_metric_raw(dst, RTAX_SSTHRESH);
        tm->tcpm_vals[TCP_METRIC_CWND] = dst_metric_raw(dst, RTAX_CWND);
        tm->tcpm_vals[TCP_METRIC_REORDERING] = dst_metric_raw(dst, RTAX_REORDERING);
        tm->tcpm_ts = 0;
        tm->tcpm_ts_stamp = 0;
        tm->tcpm_fastopen.mss = 0;
        tm->tcpm_fastopen.syn_loss = 0;
        tm->tcpm_fastopen.cookie.len = 0;
}

Neal Cardwell

maintainer

2012-08-27 04:02 UTC #6 lore raw

In reply to Eric Dumazet

Quoted 37 lines · Eric Dumazet

Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Quoted 1 line · Eric Dumazet
On Fri, 2012-08-24 at 20:50 -0400, Cristian Rodríguez wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap
Please try the following fix.
Thanks !
Well, this v2 seems cleaner :
[PATCH v2] tcp: tcp_slow_start() should not decrease snd_cwnd
Cristian Rodríguez reported various lockups in TCP stack, introduced by commit 9dc274151a548 (tcp: fix ABC in tcp_slow_start())
We could exit tcp_slow_start() with a zeroed snd_cwnd, and next time we enter tcp_slow_start(), we run an infinite loop.
Reported-by Cristian Rodríguez <[email protected]>
Cc Yuchung Cheng <[email protected]>
Cc Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp_cong.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/tcp_cong.c +0 -0
index 1432cdb..e656c72 100644

net/ipv4/tcp_cong.c void tcp_slow_start(struct tcp_sock *tp) · line 340

Quoted 5 lines

net/ipv4/tcp_cong.c +1 -1

@@ -337,7 +337,7 @@ void tcp_slow_start(struct tcp_sock *tp)
                tp->snd_cwnd_cnt -= tp->snd_cwnd;
                delta++;
        }
-       tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+       tp->snd_cwnd = clamp(tp->snd_cwnd + delta, tp->snd_cwnd, tp->snd_cwnd_clamp);

AFAICT if tcp_slow_start() is changing snd_cwnd from non-zero to zero then this is because snd_cwnd_clamp is zero here, as you theorize may be happening to races somewhere.

However, AFAICT from reading the min() and clamp() macros, this code with clamp() will still have the same problem as the existing code that uses min: if snd_cwnd_clamp is 0 then snd_cwnd will end up 0 here. (This is because the clamp() macro implicitly assumes that the max value is above the min value, and filters agains the max last.)

neal

Eric Dumazet

2012-08-27 04:06 UTC #7 lore raw

Quoted 48 lines · Neal Cardwell

Quoted 37 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Quoted 1 line · Eric Dumazet
On Fri, 2012-08-24 at 20:50 -0400, Cristian Rodríguez wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Hi, the issue I reported with IPV6 few weeks ago seems to be gone, but now I am getting the following crash..
Oh, I now see the bug, I'll send a patch asap
Please try the following fix.
Thanks !
Well, this v2 seems cleaner :
[PATCH v2] tcp: tcp_slow_start() should not decrease snd_cwnd
Cristian Rodríguez reported various lockups in TCP stack, introduced by commit 9dc274151a548 (tcp: fix ABC in tcp_slow_start())
We could exit tcp_slow_start() with a zeroed snd_cwnd, and next time we enter tcp_slow_start(), we run an infinite loop.
Reported-by Cristian Rodríguez <[email protected]>
Cc Yuchung Cheng <[email protected]>
Cc Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp_cong.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/tcp_cong.c +0 -0
index 1432cdb..e656c72 100644
net/ipv4/tcp_cong.c void tcp_slow_start(struct tcp_sock *tp) · line 340
Quoted 5 lines
net/ipv4/tcp_cong.c +1 -1
@@ -337,7 +337,7 @@ void tcp_slow_start(struct tcp_sock *tp)
                tp->snd_cwnd_cnt -= tp->snd_cwnd;
                delta++;
        }
-       tp->snd_cwnd = min(tp->snd_cwnd + delta, tp->snd_cwnd_clamp);
+       tp->snd_cwnd = clamp(tp->snd_cwnd + delta, tp->snd_cwnd, tp->snd_cwnd_clamp);
AFAICT if tcp_slow_start() is changing snd_cwnd from non-zero to zero then this is because snd_cwnd_clamp is zero here, as you theorize may be happening to races somewhere.

Quoted 6 lines

However, AFAICT from reading the min() and clamp() macros, this code with clamp() will still have the same problem as the existing code that uses min: if snd_cwnd_clamp is 0 then snd_cwnd will end up 0 here. (This is because the clamp() macro implicitly assumes that the max value is above the min value, and filters agains the max last.)

Indeed, so the first patch was better...

Not sure I can investigate this problem this week, as I attend LKS/LPC in San Diego.

Could be that snd_cwnd is zero as well so we have this infinite loop...

        while (tp->snd_cwnd_cnt >= tp->snd_cwnd) {
                tp->snd_cwnd_cnt -= tp->snd_cwnd;
                delta++;
        }

Neal Cardwell

maintainer

2012-08-30 01:37 UTC #8 lore raw

Quoted 11 lines · Eric Dumazet

Indeed, so the first patch was better...
Not sure I can investigate this problem this week, as I attend LKS/LPC in San Diego.
Could be that snd_cwnd is zero as well so we have this infinite loop...
        while (tp->snd_cwnd_cnt >= tp->snd_cwnd) {
                tp->snd_cwnd_cnt -= tp->snd_cwnd;
                delta++;
        }

Yes, it seems that one way or another cwnd is getting to zero and turning this into an infinite loop.

Two questions for you, Cristian, if you have a second:

1) Are you still seeing this problem in your workload? If so, would you have time to try another small patch to add instrumentation to track down the cause?

2) Do you happen to run with the tcp_mtu_probing sysctl enabled? I have been looking for code paths that could cause the cwnd to go to zero, and that's the first possibility that I've been able to see (in tcp_mtup_probe_success() it seems like there are corner cases where snd_cwnd could become zero).

neal

Cristian Rodríguez

author

2012-08-30 03:06 UTC #9 lore raw

El mié 29 ago 2012 21:37:46 CLT, Neal Cardwell escribió:

Hi neal, thanks for taking a look at this..

Quoted 3 lines

1) Are you still seeing this problem in your workload? If so, would you have time to try another small patch to add instrumentation to track down the cause?

After Eric's patch I no longer see this backtrace or hang, however the machine does
hang apparently for another reason that I have been unable to capture no clue at all in the logs and the KVM-over-ip video does not respond after hang, needing a reset,
that's strange though, as it is usually able to display kernel messages after a crash.

Quoted 1 line

2) Do you happen to run with the tcp_mtu_probing sysctl enabled?

No, sysctl returns : net.ipv4.tcp_mtu_probing = 0