Starting a VM reboots my machine

Hi,

I'm running an AMD

vendor_id : AuthenticAMD
cpu family : 15
model : 75
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4600+

with 4GB of RAM and Scientific Linux 5.0. I've compiled kvm-48 on it and the compile works and loading the modules works fine, the only dmesg line referring to kvm is SELinux: initialized (dev kvm_anon_inodefs, type kvm_anon_inodefs), not configured for labeling.

However, when I boot any KVM, the machine reboots. It works fine with the no-kvm flag. Has anyone else seen this behaviour?

Thanks,
Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

We have some instabilities with AMD. These issues are under debug these days. Can you please supply netconsole debug messages from the host. This can be very helpful. Also what's the guest OS?

These is a slight chance that kvm-49 and -no-kvm-irqchip will help. Dor.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Hi Dor,

I tried to hunt on my own, but I can't find much on netconsole set up. Do you have any tips or links for setting up netconsole? I'm running Scientific Linux (a RHEL5 clone).

Thanks very much,

Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

I added a wiki page for it:
http://kvm.qumranet.com/kvmwiki/NetConsole
Regards,
Dor.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Hi Dor,

Thanks very much for doing this. For the command

/sbin/modprobe netconsole netconsole=@/eth0,[email protected]/00:E0:81:2B:0C:C1

do both the IP and MAC have to be set to the listening machine? And to be clear - the nc listener is a machine on the same LAN?

Thanks,
Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Hi Cam,

The MAC should be of the default gateway, if the listener's IP is not on the same subnet as the client's.

Y.

-----Original Message-----
From: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org [mailto:kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of Cam Macdonell
Sent: Monday, November 05, 2007 4:40 PM
To: Dor Laor
Cc: kvm-devel
Subject: Re: [kvm-devel] Starting a VM reboots my machine

Hi Dor,

Thanks very much for doing this. For the command

/sbin/modprobe netconsole netconsole=@/eth0,[email protected]/00:E0:81:2B:0C:C1

do both the IP and MAC have to be set to the listening machine? And to be clear - the nc listener is a machine on the same LAN?

Thanks,
Cam

------------------------------------------------------------------------
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Hi Yaniv,

Well, I started netconsole as a module on "machine A". It loads fine and I get this in dmesg

netconsole: local port 6665
netconsole: interface eth0
netconsole: remote port 12345
netconsole: remote IP <machine B's IP>
netconsole: remote ethernet address <machine B's MAC>
netconsole: local IP <machine A's IP>
netconsole: network logging started

However, when I run "nc -dul 12345" on machine B I get nothing displayed, even as messages get added to dmesg on machine A. Are there any common problems with netconsole that I may not be handling?

Thanks,
Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Common configuration errors I've encountered (made):
- Firewall on Machine B
- Wrong MAC for Machine B (typo)
- eth0 on Machine A was not used
- They were not on the same subnet.

I'd verify with tcpdump that packets are indeed being sent & received. Use 'tcpdump -i eth0 -n port 12345' to catch those packets. You can create your own dmesg entries by running something like:
echo 'test' > /dev/kmsg

HTH,
Y.

-----Original Message-----
From: Cam Macdonell [mailto:[email protected]]
Sent: Monday, November 05, 2007 8:08 PM
To: Yaniv Kaul
Cc: Dor Laor; kvm-devel
Subject: Re: [kvm-devel] Starting a VM reboots my machine

on

Hi Yaniv,

Well, I started netconsole as a module on "machine A". It loads fine and I get this in dmesg

netconsole: local port 6665
netconsole: interface eth0
netconsole: remote port 12345
netconsole: remote IP <machine B's IP>
netconsole: remote ethernet address <machine B's MAC>
netconsole: local IP <machine A's IP>
netconsole: network logging started

However, when I run "nc -dul 12345" on machine B I get nothing displayed, even as messages get added to dmesg on machine A. Are there any common problems with netconsole that I may not be handling?

Thanks,
Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Thanks Yaniv, those are easy to follow tips. Unfortunately, nothing is being sent (or received) according to tcpdump. Is there a setting or configuration for the network card that may be necessary to get it to work?

Thanks,
Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Hi Dor,

Sorry for the delay, here's the netConsole dump. This is trying to boot FC6 on Scientific Linux 5. Let me know if you need more info. This is with kvm-50 with the flag -no-kvm-irqchip. Also, this time it didn't reboot, but froze.

Thanks,
Cam

Unable to handle kernel paging request at 0000000000008e5b RIP:
  [<ffffffff800b1ecd>] audit_syscall_exit+0x17d/0x2eb
PGD 10908d067 PUD 10908c067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /class/misc/kvm/dev
CPU 0
Modules linked in: kvm_amd(U) tun kvm(U) netconsole nfs lockd fscache 

nfs_acl autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_mod video sbs i2c_ec button battery asus_acpi acpi_memhotplug ac ipv6 lp snd_hda_intel snd_hda_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_mpu401 snd_mpu401_uart snd_rawmidi snd_seq_device parport_pc snd sg ide_cd parport floppy pcspkr cdrom i2c_nforce2 k8_edac snd_page_alloc shpchp fglrx(U) serio_raw forcedeth soundcore i2c_core edac_mc ns558 gameport sata_nv libata sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 3346, comm: icewm Tainted: P 2.6.18-8.1.15.el5 #1

RIP: 0010:[<ffffffff800b1ecd>]  [<ffffffff800b1ecd>] 
audit_syscall_exit+0x17d/0x2eb
RSP: 0018:ffff81010923bf00  EFLAGS: 00010206
RAX: ffffffff8006415f RBX: ffff810108022000 RCX: ffff81010923a000
RDX: 0000000000000080 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000008e53 R08: ffff81010923a000 R09: 000000000000003e
R10: ffff81012fd2c038 R11: ffffffff8002fbb9 R12: 0000000000000000
R13: 0000000000000000 R14: ffff81010849e040 R15: 0000000000000000
FS:  00002aaaaaade830(0000) GS:ffffffff80389000(0000) knlGS:00000000f7fd16c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000008e5b CR3: 0000000109257000 CR4: 00000000000006e0
Process icewm (pid: 3346, threadinfo ffff81010923a000, task 
ffff81010849e040)
Stack:  ffffffff8006415f ffff81010923bf58 0000000000000000 0000000000000020
  00007fff5e76cc70 00000000ffffffff 00007fff5e76cb80 ffffffff8006ade0
  00007fff5e76cc00 ffffffff8005b3ce 000000000000feff 00007fff5e76cb80
Call Trace:
  [<ffffffff8006415f>] do_page_fault+0x0/0x81d
  [<ffffffff8006ade0>] syscall_trace_leave+0x2c/0x87
  [<ffffffff8005b3ce>] int_very_careful+0x35/0x3f


Code: 81 7d 08 7a 05 00 00 75 1e 48 8b 7d 10 e8 8a af f5 ff 48 8b
RIP  [<ffffffff800b1ecd>] audit_syscall_exit+0x17d/0x2eb
  RSP <ffff81010923bf00>

CR2: 0000000000008e5b
<0>Kernel panic - not syncing: Fatal exception

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Does running the vm with 'taskset 1' help?

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

It runs a few seconds longer and then reboots. With netconsole, I don't always get a full dump, here's what I have.

Unable to handle kernel NULL pointer dereference at 0000000000000041 RIP:
  [<ffffffff800285cd>] sock_poll+0x11/0x18
PGD 10fcf0067 PUD 114b53067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /class/misc/kvm/dev
CPU 1
Modules linked in: kvm_amd(U) kvm(U) netconsole nfs lockd fscache 

nfs_acl autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_mod video sbs i2c_ec button battery asus_acpi acpi_memhotplug ac ipv6 lp snd_hda_intel snd_hda_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss parport_pc snd_mixer_oss shpchp parport sg snd_pcm i2c_nforce2 snd_mpu401 snd_mpu401_uart snd_rawmidi snd_seq_device snd_timer fglrx

Cam

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Cam: KVM-AMD is unstable now, unless you're a developer, I would suggest you to downgrade to KVM-36, which was the latest AMD-friendly version. We're working on a fix.

Dor: kvm-amd crashes host, without leaving any netconsole messages.

-Alexey

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

With x2 stepping G1 and with the below two fixes applied on top of kvm and kvm-userland at least the host is certainly solid for me (the opensuse 10.3 guest seems ok too). The x86 hack is mandatory with last git kvm kernel, without it it's an immediate crash.

index b78c81a..68f6929 100644
@@ -32,6 +32,7 @@ sync:
 	$(call hack, mmu.c)
 	$(call hack, vmx.c)
 	$(call hack, svm.c)
+	$(call hack, x86.c)
 
 install:
 	mkdir -p $(DESTDIR)/$(INSTALLDIR)

index 9584d0f..95a3489 100644

@@ -1459,11 +1459,6 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 
 	local_irq_enable();
 
-	vcpu->guest_mode = 1;
-	if (vcpu->requests)
-		if (test_and_clear_bit(KVM_TLB_FLUSH, &vcpu->requests))
-		    svm_flush_tlb(vcpu);
-
 	asm volatile (
 #ifdef CONFIG_X86_64
 		"push %%rbp; \n\t"

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

thanks!

I use the bleeding edge for userland and kernel, so kvm.git with "make sync".

Doesn't the common layer already take care of test_and_clearing that bitflag and calling kvm_x86_ops->tlb_flush before kvm_x86_ops->run in __vcpu_run? I thought it was an obsolete piece of code (besides it doesn't compile anyway it would need to be changed KVM_REQ_TLB_FLUSH) and it got re-introduced by mistake with a merging error in the last commit.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Well, I can't find anything like that it my tree. Maybe something's stale?

It does.

We're definitely looking at different trees.

Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Could be, this is why I don't like git that much, with hg there would be no way that we could be looking at different trees. Anyway I agree it's mandatory for merging with Linus so...

I started with "git clone
git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git" some time ago and I kept pulling over and over again. Sometime I had to run 'git reset --hard', sometime 'git checkout' followed by 'git checkout master', and the below is what I got right now.

andrea@v2 ~/kernel/kvm $ git reset --hard

HEAD is now at dfe6652... Merge branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm
andrea@v2 ~/kernel/kvm $ patch -p1 </tmp/z patching file drivers/kvm/svm.c
andrea@v2 ~/kernel/kvm $ git reset --hard

HEAD is now at dfe6652... Merge branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm
andrea@v2 ~/kernel/kvm $ patch -p1 </tmp/z patching file drivers/kvm/svm.c
andrea@v2 ~/kernel/kvm $

See, it applies cleanly here on top of kvm.git master.

Now I'm re-downloding the entire kvm.git and see if what I get is a different tree and if my previous kvm.git bitrotten.

Ok.

here my current tip.

commit dfe665260be338e3a7fe59172220ccadd2d1b7e7
Merge: 8c37564... c388ba8...
Author: Andrea Arcangeli <[email protected]>
Date: Sun Nov 4 13:00:47 2007 +0100

Merge branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm

commit c388ba81841a1ed30a5dd4f6029db2263ad1aca3

Author: Hollis Blanchard <hollisb-r/[email protected]>
Date:   Thu Nov 1 14:16:10 2007 -0500

KVM: Portability: Move x86 vcpu ioctl handlers to x86.c

commit a9acc2800d8676d8a9a91aeaedd16ae4f75c05df

Author: Hollis Blanchard <hollisb-r/[email protected]>
Date:   Wed Oct 31 17:24:25 2007 -0500

KVM: Portability: Move x86 FPU handling to x86.c

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

git bitrotten. What concerns me is that "git pull + git reset --hard" can't bring my old kvm.git tree in sync with the newly downloaded one, that diff I just posted remains between the two. I wonder ho to translate "hg pull + hg update -C" in git terms, "git reset --hard" clearly isn't good enough (or git isn't good enough in the first place, one of the two).

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

I'm not sure if it will fit you but you can do git reset --hard OLD_COMMIT_HASH
using a commit hash in the past before any damaged merge point. Good luck.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

git fetch git:.../kvm.git +master:some_branch_name

will force the local branch some_branch_name to be equivalent to kvm.git's master.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

btw, the easiest way to keep sync like that is 'git fetch origin'. That gives you a branch 'origin/master' that is equivalent to the state of kvm.git's master at the time of the fetch.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

I kept the bitrotten tree for tests like this, and specifying an older commit to reset --hard fixed the problem, the above probably would have worked too, thanks for the tips. Still I wonder how it went out of sync.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

Perhaps I did a non-fast-forward update (I try not to do these, but sometimes yield to temptation) and your pull merged kvm.git with an old version of itself.

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

That's a wild exaggeration. On some hosts kvm-amd hangs, and some guests (Windows XP w/ ACPI HAL) it is unstable, but other hosts and workloads are stable.

Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/