New: overflow/panic on KVM hipervizor
Subsystems (3)
https://bugzilla.kernel.org/show_bug.cgi?id=40542
Summary: overflow/panic on KVM hipervizor
Product: Virtualization
Version: unspecified
Kernel Version: 3.0.0
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
Created an attachment (id=67522)
--> (https://bugzilla.kernel.org/attachment.cgi?id=67522)
some debug info like /proc/cpuinfo and .config from kernel
I have 2 machines virtualized with KVM on Gentoo host. There is a gentoo and win2k8r2 guests (there is apache2 on gentoo which connects with php to mssql server on the second guest).
During the apache benchmark (ab -c 50 -n 100000 http://address/), hipervisor TCP stack crashes (not entirely, but random anomalies are noticed for example iptables -Lnv makes ssh session crashed). Here is kern.log snap:
---
Aug 4 18:54:46 flu kernel: general protection fault: 0000 [#1] SMP Aug 4 18:54:46 flu kernel: CPU 0 Aug 4 18:54:46 flu kernel: Modules linked in: bridge stp llc kvm_intel kvm Aug 4 18:54:46 flu kernel: Aug 4 18:54:46 flu kernel: Pid: 4327, comm: bash Not tainted 3.0.0-xxxx-std-ipv4-64 #7 Supermicro X8STi/X8STi Aug 4 18:54:46 flu kernel: RIP: 0010:[<ffffffff810f8aaa>] [<ffffffff810f8aaa>] fget_light+0x6a/0xc0 Aug 4 18:54:46 flu kernel: RSP: 0018:ffff880579015f38 EFLAGS: 00010202 Aug 4 18:54:46 flu kernel: RAX: ffff8806196fa180 RBX: 00007fffdade98f0 RCX: 0000000000000008 Aug 4 18:54:46 flu kernel: RDX: 0000880618ffac80 RSI: ffff880579015f54 RDI: ffff880618fc77f8 Aug 4 18:54:46 flu kernel: RBP: ffff880579015f38 R08: 0000000000000000 R09: 0000000000000000 Aug 4 18:54:46 flu kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 00000000000000ff Aug 4 18:54:46 flu kernel: R13: 0000000000005410 R14: 00007fffdade985c R15: 00000000007b6ae0 Aug 4 18:54:46 flu kernel: FS: 00007f495700d700(0000) GS:ffff88063fc00000(0000) knlGS:0000000000000000 Aug 4 18:54:46 flu kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 4 18:54:46 flu kernel: CR2: 00000000004620d0 CR3: 000000057927b000 CR4: 00000000000026e0 Aug 4 18:54:46 flu kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 4 18:54:46 flu kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 4 18:54:46 flu kernel: Process bash (pid: 4327, threadinfo ffff880579014000, task ffff88061aacc750) Aug 4 18:54:46 flu kernel: Stack: Aug 4 18:54:46 flu kernel: ffff880579015f78 ffffffff8110786a 0000000000010000 0000000000000000 Aug 4 18:54:46 flu kernel: 00007fffdade98f0 0000000000000082 00007fffdade9990 0000000000000000 Aug 4 18:54:46 flu kernel: 00000000000010e7 ffffffff818fe2fb 0000000000000246 0000000000000008 Aug 4 18:54:46 flu kernel: Call Trace: Aug 4 18:54:46 flu kernel: [<ffffffff8110786a>] sys_ioctl+0x2a/0x80 Aug 4 18:54:46 flu kernel: [<ffffffff818fe2fb>] system_call_fastpath+0x16/0x1b Aug 4 18:54:46 flu kernel: Code: 74 37 31 d2 48 89 d0 c9 c3 0f 1f 84 00 00 00 00 00 48 8b 40 08 3b 38 73 e9 89 ff 48 c1 e7 03 48 03 78 08 48 8b 17 48 85 d2 74 d9 <f6> 42 3d 40 75 d1 48 89 d0 c9 c3 0f 1f 00 48 8b 7a 30 4c 8d 42 Aug 4 18:54:46 flu kernel: RIP [<ffffffff810f8aaa>] fget_light+0x6a/0xc0 Aug 4 18:54:46 flu kernel: RSP <ffff880579015f38> Aug 4 18:54:46 flu kernel: ---[ end trace 9de898b3d5b9d4c6 ]--- Aug 4 18:54:46 flu kernel: general protection fault: 0000 [#2] SMP Aug 4 18:54:46 flu kernel: CPU 0 Aug 4 18:54:46 flu kernel: Modules linked in: bridge stp llc kvm_intel kvm Aug 4 18:54:46 flu kernel: Aug 4 18:54:46 flu kernel: Pid: 4327, comm: bash Tainted: G D 3.0.0-xxxx-std-ipv4-64 #7 Supermicro X8STi/X8STi Aug 4 18:54:46 flu kernel: RIP: 0010:[<ffffffff810f5567>] [<ffffffff810f5567>] filp_close+0x17/0x90 Aug 4 18:54:46 flu kernel: RSP: 0018:ffff880579015cc8 EFLAGS: 00010282 Aug 4 18:54:46 flu kernel: RAX: ffff880618fc77f8 RBX: 0000880618ffac80 RCX: ffff8806196fa240 Aug 4 18:54:46 flu kernel: RDX: 0000000000000000 RSI: ffff880619648140 RDI: 0000880618ffac80 Aug 4 18:54:46 flu kernel: RBP: ffff880579015ce8 R08: 0000000000000000 R09: 0000000000000000 Aug 4 18:54:46 flu kernel: R10: ffff88061875c090 R11: 0000000000000446 R12: 0000000000000001 Aug 4 18:54:46 flu kernel: R13: ffff8806196fa180 R14: ffff880619648140 R15: 00000000000000ff Aug 4 18:54:46 flu kernel: FS: 0000000000000000(0000) GS:ffff88063fc00000(0000) knlGS:0000000000000000 Aug 4 18:54:46 flu kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 4 18:54:46 flu kernel: CR2: 00000000004620d0 CR3: 0000000001b93000 CR4: 00000000000026e0 Aug 4 18:54:46 flu kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 4 18:54:46 flu kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 4 18:54:46 flu kernel: Process bash (pid: 4327, threadinfo ffff880579014000, task ffff88061aacc750) Aug 4 18:54:46 flu kernel: Stack: Aug 4 18:54:46 flu kernel: ffff880617983840 0000000000000003 0000000000000001 ffff8806196fa180 Aug 4 18:54:46 flu kernel: ffff880579015d38 ffffffff8104feb0 ffff8806179838a0 0000000000000000 Aug 4 18:54:46 flu kernel: ffff880579015d58 ffff88061aacc750 ffff880619648140 ffff880617983840 Aug 4 18:54:46 flu kernel: Call Trace: Aug 4 18:54:46 flu kernel: [<ffffffff8104feb0>] put_files_struct+0xd0/0xf0 Aug 4 18:54:46 flu kernel: [<ffffffff8104ff15>] exit_files+0x45/0x50 Aug 4 18:54:46 flu kernel: [<ffffffff81051da8>] do_exit+0x718/0x830 Aug 4 18:54:46 flu kernel: [<ffffffff81005bfc>] oops_end+0x9c/0xe0 Aug 4 18:54:46 flu kernel: [<ffffffff81005d36>] die+0x56/0x90 Aug 4 18:54:46 flu kernel: [<ffffffff81003942>] do_general_protection+0x152/0x160 Aug 4 18:54:46 flu kernel: [<ffffffff818fdeaf>] general_protection+0x1f/0x30 Aug 4 18:54:46 flu kernel: [<ffffffff810f8aaa>] ? fget_light+0x6a/0xc0 Aug 4 18:54:46 flu kernel: [<ffffffff8105c4d1>] ? sys_rt_sigprocmask+0x71/0xc0 Aug 4 18:54:46 flu kernel: [<ffffffff8110786a>] sys_ioctl+0x2a/0x80 Aug 4 18:54:46 flu kernel: [<ffffffff818fe2fb>] system_call_fastpath+0x16/0x1b Aug 4 18:54:46 flu kernel: Code: 24 4c 8b 64 24 08 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 48 89 fb Aug 4 18:54:46 flu kernel: RIP [<ffffffff810f5567>] filp_close+0x17/0x90 Aug 4 18:54:46 flu kernel: RSP <ffff880579015cc8> Aug 4 18:54:46 flu kernel: ---[ end trace 9de898b3d5b9d4c7 ]--- Aug 4 18:54:46 flu kernel: Fixing recursive fault but reboot is needed!
---
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #1 from Slawek Rozbicki <[email protected]> 2011-08-04 17:24:03 --- moreover, hardware tests were done. Looks like cpu, hdd and ram aren't faulty.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #2 from Slawek Rozbicki <[email protected]> 2011-08-04 17:33:52 --- http port is DNATed by iptables to gentoo guest on tap virtio adapter.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
Re: [Bug 40542] overflow/panic on KVM hipervizor
Quoted 9 lines · On 05/08/11 01:33, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #2 from Slawek Rozbicki<[email protected]> 2011-08-04 17:33:52 ---
http port is DNATed by iptables to gentoo guest on tap virtio adapter.
This looks like the bug I've been fighting with on and off.
I'll wager if you can run your tests without using any form of NAT (and preferably with netfilter compiled out) you won't see the problem occur.
Re: [Bug 40542] overflow/panic on KVM hipervizor
Quoted 2 lines · Brad Campbell
This looks like the bug I've been fighting with on and off.
What's the bugzilla number for that?
(unfortunately, no great insight except for "CLOSED DUPLICATE")
hopefully someone from netdev can take a look, DNAT is seriously broken.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Bug 40542] overflow/panic on KVM hipervizor
Quoted 10 lines · Avi Kivity
Quoted 2 lines · Brad Campbell
This looks like the bug I've been fighting with on and off.
What's the bugzilla number for that?
(unfortunately, no great insight except for "CLOSED DUPLICATE")
hopefully someone from netdev can take a look, DNAT is seriously broken.
I can reproduce it at will, but it's on a live production machine. I've just ordered a second machine which I can use to reproduce and test against. From a bisection standpoint I'm about half way between 2.6.35 & 2.6.36, but until the second machine arrives I'm just unable to chase it any further.
Brad
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #3 from Slawek Rozbicki <[email protected]> 2011-08-04 17:58:18 --- Created an attachment (id=67532)
--> (https://bugzilla.kernel.org/attachment.cgi?id=67532)/etc/conf.d/net (network configuration)
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
Marcelo Tosatti <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #4 from Marcelo Tosatti <[email protected]> 2011-08-05 17:17:29 --- (In reply to comment #3)
Quoted 1 line
Created an attachment (id=67532)
--> (https://bugzilla.kernel.org/attachment.cgi?id=67532) [details]
Quoted 1 line
/etc/conf.d/net(network configuration)
Can you please enable SLUB debugging, with the "slub_debug=ZFPU" kernel parameter.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #5 from Slawek Rozbicki <[email protected]> 2011-08-06 13:09:43 --- I've tested if this problem exists even if no TLB and KSM is compiled in kernel. Moreover the kernel is now very small (i've removed lots of unused kernel features). Unfortunately, the problem is still present. slub debug is on its way:
CONFIG_SLUB_DEBUG=yCONFIG_SLUB=yCONFIG_SLUB_DEBUG_ON=y
# CONFIG_SLUB_STATS is not set
but there is no ZFPU option. I'll provide updated .config and kern.log when it's crashed again.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #6 from Slawek Rozbicki <[email protected]> 2011-08-06 16:21:48 --- Created an attachment (id=67842)
--> (https://bugzilla.kernel.org/attachment.cgi?id=67842)
kern.log with SLAB DEBUG = ON
Aug 6 18:10 - first occurance of SLAB debugged exception/crash. Please investigate. I'll provide updated .config of kernel, because ive recompiled it with less features.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #7 from Slawek Rozbicki <[email protected]> 2011-08-06 16:31:15 --- its probably irrelevant but after last crash, now my ext4 root partition is damaged and /forcefsck is not fixing it (in spite tune2fs shows me that the check was made). Something tells me its a netfilter bug. I am going to turn the virtio net adapter off, find out if this helps and if not switch to VDE just in order to avoid iptables dnat.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #8 from Slawek Rozbicki <[email protected]> 2011-08-06 16:33:22 --- Created an attachment (id=67852)
--> (https://bugzilla.kernel.org/attachment.cgi?id=67852)/proc/config.gz
updated .config from /proc
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #9 from Anonymous Emailer <[email protected]> 2011-08-07 14:55:13 ---
Reply-To: [email protected]
Quoted 9 lines · On 05/08/11 01:33, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #2 from Slawek Rozbicki<[email protected]> 2011-08-04 17:33:52 ---
http port is DNATed by iptables to gentoo guest on tap virtio adapter.
This looks like the bug I've been fighting with on and off.
I'll wager if you can run your tests without using any form of NAT (and preferably with netfilter compiled out) you won't see the problem occur.
http://permalink.gmane.org/gmane.linux.kernel.mm/64047
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #10 from Slawek Rozbicki <[email protected]> 2011-08-07 15:10:09 --- My environment needs apache port mapping from kvm guest to public IP address. I am able to run apache benchmark from KVM host just to investigate further this bug, but eventualy i need to have DNAT and masquerade.
Is there anything I can do to help you guys address this issue? At the end of the day, it is going to be production environment.
I suppose if everything fails, I'll buy another IP address and make bridged network.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #11 from Avi Kivity <[email protected]> 2011-08-07 16:01:51 ---
Quoted 2 lines · Brad Campbell
This looks like the bug I've been fighting with on and off.
What's the bugzilla number for that?
(unfortunately, no great insight except for "CLOSED DUPLICATE")
hopefully someone from netdev can take a look, DNAT is seriously broken.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #12 from Anonymous Emailer <[email protected]> 2011-08-08 01:41:09 ---
Reply-To: [email protected]
Quoted 10 lines · Avi Kivity
Quoted 2 lines · Brad Campbell
This looks like the bug I've been fighting with on and off.
What's the bugzilla number for that?
(unfortunately, no great insight except for "CLOSED DUPLICATE")
hopefully someone from netdev can take a look, DNAT is seriously broken.
I can reproduce it at will, but it's on a live production machine. I've just ordered a second
machine which I can use to reproduce and test against. From a bisection standpoint I'm about half
way between 2.6.35 & 2.6.36, but until the second machine arrives I'm just unable to chase it any
further.
Brad
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
--- Comment #13 from Slawek Rozbicki <[email protected]> 2011-08-19 10:37:39 --- i've changed my network layout to provide SNAT to the guests via iptables on 2 additional IPs. It seems that this bug doesn't exist anymore. Looks like static nat is a walkaround. It's stable after an hour of testing. I've updated qemu as well.
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 40542] overflow/panic on KVM hipervizor
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=40542
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #14 from [email protected] --- This bug is outdated, please test against a newer kernel. Cheers Nick
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.