2.6.31.4 panic: CRED: put_cred_rcu() sees ffff880204e58c00 with usage 82150912

DISCUSSION Nikola Ciprich 4 participants 5 messages Oct 30, 2009 – Nov 3, 2009 KVM

Share & tools

Nikola Ciprich

author

2009-10-30 11:11 UTC #0 lore raw

Hi,
some time ago, I updated my KVM hosting machine to 2.6.31.1 and it just died horribly:

Oct 30 10:45:17 vbox [706369.133516] Kernel panic - not syncing: CRED: put_cred_rcu() sees ffff880204e58c00 with usage 82150912
Oct 30 10:45:17 vbox [706369.133519]
Oct 30 10:45:17 vbox [706369.144990] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.31lb.02 #1
Oct 30 10:45:17 vbox [706369.151554] Call Trace:
Oct 30 10:45:17 vbox [706369.154332]  <IRQ>
Oct 30 10:45:17 vbox [<ffffffff8104c1fa>] panic+0xaa/0x180
Oct 30 10:45:17 vbox [706369.160280]  [<ffffffff81322b90>] ? _spin_unlock+0x30/0x60
Oct 30 10:45:17 vbox [706369.166256]  [<ffffffff810f5671>] ? add_partial+0x21/0x90
Oct 30 10:45:17 vbox [706369.172155]  [<ffffffff810f6a92>] ? __slab_free+0x92/0x3c0
Oct 30 10:45:17 vbox [706369.178127]  [<ffffffff81102317>] ? file_free_rcu+0x37/0x50
Oct 30 10:45:17 vbox [706369.184198]  [<ffffffff8106c655>] put_cred_rcu+0x75/0x80
Oct 30 10:45:17 vbox [706369.190008]  [<ffffffff810a2525>] __rcu_process_callbacks+0x125/0x250
Oct 30 10:45:17 vbox [706369.197020]  [<ffffffff810a2689>] rcu_process_callbacks+0x39/0x60
Oct 30 10:45:17 vbox [706369.203624]  [<ffffffff81052a61>] __do_softirq+0xc1/0x250
Oct 30 10:45:17 vbox [706369.209506]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.215182]  [<ffffffff8100c4dc>] call_softirq+0x1c/0x30
Oct 30 10:45:17 vbox [706369.220986]  [<ffffffff8100e46d>] do_softirq+0x3d/0x80
Oct 30 10:45:17 vbox [706369.227317]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.233020]  [<ffffffff810538e4>] ksoftirqd+0x84/0x1a0
Oct 30 10:45:17 vbox [706369.238622]  [<ffffffff81066686>] kthread+0xa6/0xb0
Oct 30 10:45:17 vbox [706369.243956]  [<ffffffff8100c3da>] child_rip+0xa/0x20
Oct 30 10:45:17 vbox [706369.249390]  [<ffffffff810665e0>] ? kthread+0x0/0xb0
Oct 30 10:45:17 vbox [706369.254806]  [<ffffffff8100c3d0>] ? child_rip+0x0/0x20
Oct 30 10:45:17 vbox [706369.260454] Rebooting in 10 seconds..

(trace is obtained from netconsole, so hopefully it's not mangled).
The machine was running ~30 KVM guests, it's 8CPU 16GB x86_64, when it crashed, it was only moderately loaded. Never had this (or any other) kind of crash on it before. I know there's 2.6.31.5 already out, but I'm not sure if some related problem has been reported/fixed and I'm obviously not able to quicky test/reproduce it with latest kernel, so I'm rather reporting.
Should more information/testing/etc be required, I'll be glad to help regards
nik

Signature · 13 lines

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------

Nikola Ciprich

author

2009-10-30 11:14 UTC #1 lore raw

Ouch, typo in subject, it's 2.6.31.1 of course. sorry about that. also CCing kvm.
n.

Quoted 56 lines · Nikola Ciprich

Hi,
some time ago, I updated my KVM hosting machine to 2.6.31.1 and it just died horribly:

Oct 30 10:45:17 vbox [706369.133516] Kernel panic - not syncing: CRED: put_cred_rcu() sees ffff880204e58c00 with usage 82150912
Oct 30 10:45:17 vbox [706369.133519]
Oct 30 10:45:17 vbox [706369.144990] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.31lb.02 #1
Oct 30 10:45:17 vbox [706369.151554] Call Trace:
Oct 30 10:45:17 vbox [706369.154332]  <IRQ>
Oct 30 10:45:17 vbox [<ffffffff8104c1fa>] panic+0xaa/0x180
Oct 30 10:45:17 vbox [706369.160280]  [<ffffffff81322b90>] ? _spin_unlock+0x30/0x60
Oct 30 10:45:17 vbox [706369.166256]  [<ffffffff810f5671>] ? add_partial+0x21/0x90
Oct 30 10:45:17 vbox [706369.172155]  [<ffffffff810f6a92>] ? __slab_free+0x92/0x3c0
Oct 30 10:45:17 vbox [706369.178127]  [<ffffffff81102317>] ? file_free_rcu+0x37/0x50
Oct 30 10:45:17 vbox [706369.184198]  [<ffffffff8106c655>] put_cred_rcu+0x75/0x80
Oct 30 10:45:17 vbox [706369.190008]  [<ffffffff810a2525>] __rcu_process_callbacks+0x125/0x250
Oct 30 10:45:17 vbox [706369.197020]  [<ffffffff810a2689>] rcu_process_callbacks+0x39/0x60
Oct 30 10:45:17 vbox [706369.203624]  [<ffffffff81052a61>] __do_softirq+0xc1/0x250
Oct 30 10:45:17 vbox [706369.209506]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.215182]  [<ffffffff8100c4dc>] call_softirq+0x1c/0x30
Oct 30 10:45:17 vbox [706369.220986]  [<ffffffff8100e46d>] do_softirq+0x3d/0x80
Oct 30 10:45:17 vbox [706369.227317]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.233020]  [<ffffffff810538e4>] ksoftirqd+0x84/0x1a0
Oct 30 10:45:17 vbox [706369.238622]  [<ffffffff81066686>] kthread+0xa6/0xb0
Oct 30 10:45:17 vbox [706369.243956]  [<ffffffff8100c3da>] child_rip+0xa/0x20
Oct 30 10:45:17 vbox [706369.249390]  [<ffffffff810665e0>] ? kthread+0x0/0xb0
Oct 30 10:45:17 vbox [706369.254806]  [<ffffffff8100c3d0>] ? child_rip+0x0/0x20
Oct 30 10:45:17 vbox [706369.260454] Rebooting in 10 seconds..

Signature · 18 lines

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Signature · 13 lines

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------

Rafael J. Wysocki

2009-10-30 18:18 UTC #2 lore raw

Quoted 2 lines · Nikola Ciprich

Ouch, typo in subject, it's 2.6.31.1 of course. sorry about that. also CCing kvm.

What about later kernels? Mainline in particular?

Rafael

Marcelo Tosatti

2009-11-03 14:46 UTC #3 lore raw

In reply to Nikola Ciprich

Quoted 7 lines · Nikola Ciprich

Ouch, typo in subject, it's 2.6.31.1 of course. sorry about that. also CCing kvm.
n.
Quoted 2 lines · Nikola Ciprich
Hi,
some time ago, I updated my KVM hosting machine to 2.6.31.1 and it just died horribly:

Nikola,

Upgraded from what? Did you see experience the crash again?

Quoted 33 lines

Oct 30 10:45:17 vbox [706369.133516] Kernel panic - not syncing: CRED: put_cred_rcu() sees ffff880204e58c00 with usage 82150912
Oct 30 10:45:17 vbox [706369.133519]
Oct 30 10:45:17 vbox [706369.144990] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.31lb.02 #1
Oct 30 10:45:17 vbox [706369.151554] Call Trace:
Oct 30 10:45:17 vbox [706369.154332]  <IRQ>
Oct 30 10:45:17 vbox [<ffffffff8104c1fa>] panic+0xaa/0x180
Oct 30 10:45:17 vbox [706369.160280]  [<ffffffff81322b90>] ? _spin_unlock+0x30/0x60
Oct 30 10:45:17 vbox [706369.166256]  [<ffffffff810f5671>] ? add_partial+0x21/0x90
Oct 30 10:45:17 vbox [706369.172155]  [<ffffffff810f6a92>] ? __slab_free+0x92/0x3c0
Oct 30 10:45:17 vbox [706369.178127]  [<ffffffff81102317>] ? file_free_rcu+0x37/0x50
Oct 30 10:45:17 vbox [706369.184198]  [<ffffffff8106c655>] put_cred_rcu+0x75/0x80
Oct 30 10:45:17 vbox [706369.190008]  [<ffffffff810a2525>] __rcu_process_callbacks+0x125/0x250
Oct 30 10:45:17 vbox [706369.197020]  [<ffffffff810a2689>] rcu_process_callbacks+0x39/0x60
Oct 30 10:45:17 vbox [706369.203624]  [<ffffffff81052a61>] __do_softirq+0xc1/0x250
Oct 30 10:45:17 vbox [706369.209506]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.215182]  [<ffffffff8100c4dc>] call_softirq+0x1c/0x30
Oct 30 10:45:17 vbox [706369.220986]  [<ffffffff8100e46d>] do_softirq+0x3d/0x80
Oct 30 10:45:17 vbox [706369.227317]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.233020]  [<ffffffff810538e4>] ksoftirqd+0x84/0x1a0
Oct 30 10:45:17 vbox [706369.238622]  [<ffffffff81066686>] kthread+0xa6/0xb0
Oct 30 10:45:17 vbox [706369.243956]  [<ffffffff8100c3da>] child_rip+0xa/0x20
Oct 30 10:45:17 vbox [706369.249390]  [<ffffffff810665e0>] ? kthread+0x0/0xb0
Oct 30 10:45:17 vbox [706369.254806]  [<ffffffff8100c3d0>] ? child_rip+0x0/0x20
Oct 30 10:45:17 vbox [706369.260454] Rebooting in 10 seconds..

Nikola Ciprich

2009-11-03 21:12 UTC #4 lore raw

Hello Marcelo,
well, my report might have been a bit misleading, 2.6.31.2 has been running there for almost 3 weeks. So the problem didn't occur JUST after the upgrade. But it never occured before it, while we were using older kernels (varius 2.6.30.x, 2.6.29.x and older).
I've updated machine to 2.6.31.5, and it's been running without problem since then, but it's been just few days, so we'll see.
I'll report if the problem appears again.
Thanks.
regards
nik

Quoted 47 lines · Marcelo Tosatti

Quoted 7 lines · Nikola Ciprich
Ouch, typo in subject, it's 2.6.31.1 of course. sorry about that. also CCing kvm.
n.
Quoted 2 lines · Nikola Ciprich
Hi,
some time ago, I updated my KVM hosting machine to 2.6.31.1 and it just died horribly:
Nikola,
Upgraded from what? Did you see experience the crash again?
Quoted 33 lines
Quoted 33 lines
Oct 30 10:45:17 vbox [706369.133516] Kernel panic - not syncing: CRED: put_cred_rcu() sees ffff880204e58c00 with usage 82150912
Oct 30 10:45:17 vbox [706369.133519]
Oct 30 10:45:17 vbox [706369.144990] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.31lb.02 #1
Oct 30 10:45:17 vbox [706369.151554] Call Trace:
Oct 30 10:45:17 vbox [706369.154332]  <IRQ>
Oct 30 10:45:17 vbox [<ffffffff8104c1fa>] panic+0xaa/0x180
Oct 30 10:45:17 vbox [706369.160280]  [<ffffffff81322b90>] ? _spin_unlock+0x30/0x60
Oct 30 10:45:17 vbox [706369.166256]  [<ffffffff810f5671>] ? add_partial+0x21/0x90
Oct 30 10:45:17 vbox [706369.172155]  [<ffffffff810f6a92>] ? __slab_free+0x92/0x3c0
Oct 30 10:45:17 vbox [706369.178127]  [<ffffffff81102317>] ? file_free_rcu+0x37/0x50
Oct 30 10:45:17 vbox [706369.184198]  [<ffffffff8106c655>] put_cred_rcu+0x75/0x80
Oct 30 10:45:17 vbox [706369.190008]  [<ffffffff810a2525>] __rcu_process_callbacks+0x125/0x250
Oct 30 10:45:17 vbox [706369.197020]  [<ffffffff810a2689>] rcu_process_callbacks+0x39/0x60
Oct 30 10:45:17 vbox [706369.203624]  [<ffffffff81052a61>] __do_softirq+0xc1/0x250
Oct 30 10:45:17 vbox [706369.209506]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.215182]  [<ffffffff8100c4dc>] call_softirq+0x1c/0x30
Oct 30 10:45:17 vbox [706369.220986]  [<ffffffff8100e46d>] do_softirq+0x3d/0x80
Oct 30 10:45:17 vbox [706369.227317]  [<ffffffff81053860>] ? ksoftirqd+0x0/0x1a0
Oct 30 10:45:17 vbox [706369.233020]  [<ffffffff810538e4>] ksoftirqd+0x84/0x1a0
Oct 30 10:45:17 vbox [706369.238622]  [<ffffffff81066686>] kthread+0xa6/0xb0
Oct 30 10:45:17 vbox [706369.243956]  [<ffffffff8100c3da>] child_rip+0xa/0x20
Oct 30 10:45:17 vbox [706369.249390]  [<ffffffff810665e0>] ? kthread+0x0/0xb0
Oct 30 10:45:17 vbox [706369.254806]  [<ffffffff8100c3d0>] ? child_rip+0x0/0x20
Oct 30 10:45:17 vbox [706369.260454] Rebooting in 10 seconds..
(trace is obtained from netconsole, so hopefully it's not mangled).
The machine was running ~30 KVM guests, it's 8CPU 16GB x86_64, when it crashed, it was only moderately loaded. Never had this (or any other) kind of crash on it before. I know there's 2.6.31.5 already out, but I'm not sure if some related problem has been reported/fixed and I'm obviously not able to quicky test/reproduce it with latest kernel, so I'm rather reporting.
Should more information/testing/etc be required, I'll be glad to help regards
nik

Signature · 14 lines

-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------