Suggested Parameters for SLES 10 64-bit
Hi,
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Has anyone experience with suitable kvm and/or kernel parameters to get this stable?
Thanks,
Peter
Quoted 6 lines · Peter Lieven
Hi,
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Could you please post the panics you get? What does hang during boot mean?
The easiest way to get them is probably to start the guest with -serial stdio and pass "console=ttyS0" on the grub command line.
Quoted 2 lines
Has anyone experience with suitable kvm and/or kernel parameters to get this stable?
It should be reasonably stable already. The only issue I'm aware of is the missing kvm-clock, so don't expect your timekeeping inside the guest to be 100% sane.
Alex
Quoted 5 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
Hi,
Quoted 8 lines
Quoted 5 lines
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Could you please post the panics you get? What does hang during boot mean?
with acpi=off it hangs after starting powersaved
Quoted 3 lines
The easiest way to get them is probably to start the guest with -serial stdio and pass "console=ttyS0" on the grub command line.
without acpi=off it hangs always with a lookup. one time it happened directly after initializing hpet0. btw, is it safe to turn of hpet for linux guests in genereal?
regarding missing kvm-clock. i made the experience that some guests crash after live migration with clocksource=kvm_clock while they don't with clocksource=acpi_pm. this is off topic here, but its also something i would like to debug with someone.
Starting cupsd done
Starting ZENworks Management Daemon done
IA-32 Microcode Update Driver v1.14 unregistered
Starting INET services. (xinetd) done
Checking/updating CPU microcode done
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
..dead
Try to get initial date and time via NTP from 212.110.100.1 done Starting network time protocol daemon (NTPD) done Starting Name Service Cache Daemon done
NetBackup SAN Client Fibre Transport daemon started.
Starting powersaved: done
Starting mail service (Postfix)
NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in: ipv6 button battery ac apparmor aamatch_pcre loop usbhid dm_mod 8139cp mii uhci_hcd e1000 i2c_piix4 usbcore ide_cd i2c_core cdrom parport_pc lp parport ext3 jbd sg sym53c8xx scsi_transport_spi edd fan thermal processor piix sd_mod scsi_mod ide_disk ide_core
Pid: 3134, comm: powersaved Not tainted 2.6.16.46-0.12-smp #1
RIP: 0010:[<ffffffff802dabf1>] <ffffffff802dabf1>{paranoid_restore+81}
RSP: 0000:ffffffff8041afd8 EFLAGS: 00010086
RAX: ffffffff88021d50 RBX: ffffffff88021e18 RCX: 00000000b4c562b6
RDX: 00000000000001f7 RSI: ffffffff88021e18 RDI: 00000000000001f7
RBP: 0000000000000000 R08: 00000000000004e2 R09: ffffffff80417d60
R10: 000000000000001f R11: ffffffff8800752c R12: ffffffff88021d00
R13: 00000000000004e2 R14: ffffffff80417dac R15: 0000000000000040
FS: 00002ae1d26ee760(0000) GS:ffffffff803be000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000555b30 CR3: 0000000210fe9000 CR4: 00000000000006e0
Process powersaved (pid: 3134, threadinfo ffff810210ff2000, task ffff810211f83850)
Stack: ffffffff802dabf1 0000000000000010 0000000000010086 ffffffff8041afd8
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace: <NMI> <ffffffff802dabf1>{paranoid_restore+81} <EOE>
<ffffffff802dabf1>{paranoid_restore+81}
Code: 48 cf 65 48 8b 0c 25 10 00 00 00 48 81 e9 d8 1f 00 00 8b 59console shuts up ...
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
Quoted 11 lines
Quoted 3 lines
Has anyone experience with suitable kvm and/or kernel parameters to get this stable?
It should be reasonably stable already. The only issue I'm aware of is the missing kvm-clock, so don't expect your timekeeping inside the guest to be 100% sane.
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Quoted 35 lines · Peter Lieven
Quoted 5 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
Hi,
Quoted 8 lines
Quoted 5 lines
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Could you please post the panics you get? What does hang during boot mean?
with acpi=off it hangs after starting powersaved
Quoted 3 lines
The easiest way to get them is probably to start the guest with -serial stdio and pass "console=ttyS0" on the grub command line.
without acpi=off it hangs always with a lookup. one time it happened directly after initializing hpet0. btw, is it safe to turn of hpet for linux guests in genereal?
regarding missing kvm-clock. i made the experience that some guests crash after live migration with clocksource=kvm_clock while they don't with clocksource=acpi_pm. this is off topic here, but its also something i would like to debug with someone.
Starting cupsd done
Starting ZENworks Management Daemon done
IA-32 Microcode Update Driver v1.14 unregistered
Starting INET services. (xinetd) done
Checking/updating CPU microcode done
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
..dead
Quoted 7 lines
Try to get initial date and time via NTP from 212.110.100.1 done Starting network time protocol daemon (NTPD) done Starting Name Service Cache Daemon doneNetBackup SAN Client Fibre Transport daemon started.
Starting powersaved: done
Starting mail service (Postfix)
NMI Watchdog detected LOCKUP on CPU 0
What host kernel are you running on? Or rather, what host kvm module version?
Quoted 3 lines
CPU 0
Modules linked in: ipv6 button battery ac apparmor aamatch_pcre loop usbhid dm_mod 8139cp mii uhci_hcd e1000 i2c_piix4 usbcore ide_cd i2c_core cdrom parport_pc lp parport ext3 jbd sg sym53c8xx scsi_transport_spi edd fan thermal processor piix sd_mod scsi_mod ide_disk ide_corePid: 3134, comm: powersaved Not tainted 2.6.16.46-0.12-smp #1
This looks old. The SP3 GM version is 2.6.16.60-0.54.5. This release is definitely out of support :(.
After some searching I found the RPM. That's the SLES10 SP1 GM kernel version. I don't think anyone ever tested that one on kvm.
Quoted 1 line
RIP: 0010:[<ffffffff802dabf1>] <ffffffff802dabf1>{paranoid_restore+81}Yeah, great. That one's not helpful at all:
0xffffffff802dabf1 <paranoid_restore+81>: iretq
Quoted 17 lines
RSP: 0000:ffffffff8041afd8 EFLAGS: 00010086
RAX: ffffffff88021d50 RBX: ffffffff88021e18 RCX: 00000000b4c562b6
RDX: 00000000000001f7 RSI: ffffffff88021e18 RDI: 00000000000001f7
RBP: 0000000000000000 R08: 00000000000004e2 R09: ffffffff80417d60
R10: 000000000000001f R11: ffffffff8800752c R12: ffffffff88021d00
R13: 00000000000004e2 R14: ffffffff80417dac R15: 0000000000000040
FS: 00002ae1d26ee760(0000) GS:ffffffff803be000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000555b30 CR3: 0000000210fe9000 CR4: 00000000000006e0
Process powersaved (pid: 3134, threadinfo ffff810210ff2000, task ffff810211f83850)
Stack: ffffffff802dabf1 0000000000000010 0000000000010086 ffffffff8041afd8
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace: <NMI> <ffffffff802dabf1>{paranoid_restore+81} <EOE>
<ffffffff802dabf1>{paranoid_restore+81}
Code: 48 cf 65 48 8b 0c 25 10 00 00 00 48 81 e9 d8 1f 00 00 8b 59Quoted 2 lines
console shuts up ...
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
Please try out different values for -cpu. I think powersaved gets confused by the invalid CPU type we're emulating.
Alex
hi alex,
what 64-bit -cpu types do you suggest?
when i boot the kernel with "nohpet" it simply hangs shortly after powersaved...
br,
peter
Quoted 103 lines · Alexander Graf
Quoted 40 lines · Peter Lieven
Quoted 6 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
Hi,
Quoted 9 lines
Quoted 6 lines
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Could you please post the panics you get? What does hang during boot mean?
with acpi=off it hangs after starting powersaved
Quoted 4 lines
The easiest way to get them is probably to start the guest with -serial stdio and pass "console=ttyS0" on the grub command line.
without acpi=off it hangs always with a lookup. one time it happened directly after initializing hpet0. btw, is it safe to turn of hpet for linux guests in genereal?
regarding missing kvm-clock. i made the experience that some guests crash after live migration with clocksource=kvm_clock while they don't with clocksource=acpi_pm. this is off topic here, but its also something i would like to debug with someone.
Starting cupsd done
Starting ZENworks Management Daemon done
IA-32 Microcode Update Driver v1.14 unregistered
Starting INET services. (xinetd) done
Checking/updating CPU microcode done
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
..deadQuoted 8 lines
Try to get initial date and time via NTP from 212.110.100.1 done Starting network time protocol daemon (NTPD) done Starting Name Service Cache Daemon doneNetBackup SAN Client Fibre Transport daemon started.
Starting powersaved: done
Starting mail service (Postfix)
NMI Watchdog detected LOCKUP on CPU 0What host kernel are you running on? Or rather, what host kvm module version?
Quoted 4 lines
CPU 0
Modules linked in: ipv6 button battery ac apparmor aamatch_pcre loop usbhid dm_mod 8139cp mii uhci_hcd e1000 i2c_piix4 usbcore ide_cd i2c_core cdrom parport_pc lp parport ext3 jbd sg sym53c8xx scsi_transport_spi edd fan thermal processor piix sd_mod scsi_mod ide_disk ide_corePid: 3134, comm: powersaved Not tainted 2.6.16.46-0.12-smp #1
This looks old. The SP3 GM version is 2.6.16.60-0.54.5. This release is definitely out of support :(.
After some searching I found the RPM. That's the SLES10 SP1 GM kernel version. I don't think anyone ever tested that one on kvm.
Quoted 2 lines
RIP: 0010:[<ffffffff802dabf1>] <ffffffff802dabf1>{paranoid_restore+81}Yeah, great. That one's not helpful at all:
0xffffffff802dabf1<paranoid_restore+81>: iretqQuoted 17 lines
RSP: 0000:ffffffff8041afd8 EFLAGS: 00010086 RAX: ffffffff88021d50 RBX: ffffffff88021e18 RCX: 00000000b4c562b6 RDX: 00000000000001f7 RSI: ffffffff88021e18 RDI: 00000000000001f7 RBP: 0000000000000000 R08: 00000000000004e2 R09: ffffffff80417d60 R10: 000000000000001f R11: ffffffff8800752c R12: ffffffff88021d00 R13: 00000000000004e2 R14: ffffffff80417dac R15: 0000000000000040 FS: 00002ae1d26ee760(0000) GS:ffffffff803be000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000555b30 CR3: 0000000210fe9000 CR4: 00000000000006e0 Process powersaved (pid: 3134, threadinfo ffff810210ff2000, task ffff810211f83850) Stack: ffffffff802dabf1 0000000000000010 0000000000010086 ffffffff8041afd8 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Call Trace: <NMI> <ffffffff802dabf1>{paranoid_restore+81} <EOE> <ffffffff802dabf1>{paranoid_restore+81} Code: 48 cf 65 48 8b 0c 25 10 00 00 00 48 81 e9 d8 1f 00 00 8b 59Quoted 3 lines
console shuts up ...
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!Please try out different values for -cpu. I think powersaved gets confused by the invalid CPU type we're emulating.
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Quoted 3 lines · Peter Lieven
hi alex,
what 64-bit -cpu types do you suggest?
For starters I'd say try -cpu host.
Alex
hi alex,
unfortunately -cpu host, -cpu qemu64, -cpu core2duo, -cpu kvm64 (which should be default) doesn't help all other cpus available are 32-bit afaik.
as i said if i boot with with kernel parameter "nohpet", but acpi on the guest just hangs at 100% cpu.
would a backtrace help?
br,
peter
Quoted 15 lines · Alexander Graf
Quoted 4 lines · Peter Lieven
hi alex,
what 64-bit -cpu types do you suggest?
For starters I'd say try -cpu host.
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Quoted 8 lines · Peter Lieven
hi alex,
unfortunately -cpu host, -cpu qemu64, -cpu core2duo, -cpu kvm64 (which should be default) doesn't help all other cpus available are 32-bit afaik.
as i said if i boot with with kernel parameter "nohpet", but acpi on the guest just hangs at 100% cpu.
would a backtrace help?
Sigh. Is there any way to upgrade the system to SP3? Preferably incl. the latest maintenance update?
Alex
Quoted 8 lines · Alexander Graf
Quoted 5 lines · Peter Lieven
hi alex,
unfortunately -cpu host, -cpu qemu64, -cpu core2duo, -cpu kvm64 (which should be default) doesn't help all other cpus available are 32-bit afaik.
Quoted 7 lines
Quoted 4 lines
as i said if i boot with with kernel parameter "nohpet", but acpi on the guest just hangs at 100% cpu.
would a backtrace help?
Sigh. Is there any way to upgrade the system to SP3? Preferably incl. the latest maintenance update?
yes, we will try this. if you don't suspect it is a generall problem with qemu-kvm that should be fixed. keep you posted.
thanks for you help,
peter
Quoted 5 lines
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Quoted 11 lines · Peter Lieven
Quoted 10 lines · Alexander Graf
Quoted 7 lines · Peter Lieven
hi alex,
unfortunately -cpu host, -cpu qemu64, -cpu core2duo, -cpu kvm64 (which should be default) doesn't help all other cpus available are 32-bit afaik.
as i said if i boot with with kernel parameter "nohpet", but acpi on the guest just hangs at 100% cpu.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
would a backtrace help?
Sigh. Is there any way to upgrade the system to SP3? Preferably incl. the latest maintenance update?
yes, we will try this. if you don't suspect it is a generall problem with qemu-kvm that should be fixed. keep you posted.
Alright. Installing SLES10 SP1 in a VM in parallel.
Alex
hi alex,
here is a backtrace of the sles 10 sp1 vm running on qemu-kvm 0.12.4. hanging at boot.
a colleguage is meanwhile upgraded a cloned system to sp3 to see if it is also not working.
^C
Program received signal SIGINT, Interrupt. [Switching to Thread 0x7f0b753c76f0 (LWP 25532)] 0x00007f0b73e13742 in select () from /lib/libc.so.6 (gdb) thread apply all bt
Thread 5 (Thread 0x7f0b72115950 (LWP 25546)):
#0 0x00007f0b73e12cd7 in ioctl () from /lib/libc.so.6 #1 0x000000000042b945 in kvm_run (env=0x201cb10) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:921 #2 0x000000000042cea2 in kvm_cpu_exec (env=0x201cb10) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1651 #3 0x000000000042d62c in kvm_main_loop_cpu (env=0x201cb10) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1893 #4 0x000000000042d76d in ap_main_loop (_env=0x201cb10) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943 #5 0x00007f0b74d983ba in start_thread () from /lib/libpthread.so.0 #6 0x00007f0b73e1afcd in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f0b72916950 (LWP 25545)):
#0 0x00007f0b73d69461 in sigtimedwait () from /lib/libc.so.6 #1 0x000000000042d1e5 in kvm_main_loop_wait (env=0x200ed90, timeout=1000) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1752 #2 0x000000000042d64a in kvm_main_loop_cpu (env=0x200ed90) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1896 #3 0x000000000042d76d in ap_main_loop (_env=0x200ed90) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943 #4 0x00007f0b74d983ba in start_thread () from /lib/libpthread.so.0 #5 0x00007f0b73e1afcd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f0b73117950 (LWP 25544)):
#0 0x00007f0b73d69461 in sigtimedwait () from /lib/libc.so.6 #1 0x000000000042d1e5 in kvm_main_loop_wait (env=0x2001010, timeout=1000) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1752 #2 0x000000000042d64a in kvm_main_loop_cpu (env=0x2001010) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1896 #3 0x000000000042d76d in ap_main_loop (_env=0x2001010) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943 #4 0x00007f0b74d983ba in start_thread () from /lib/libpthread.so.0 #5 0x00007f0b73e1afcd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f0b73918950 (LWP 25543)):
#0 0x00007f0b73d69461 in sigtimedwait () from /lib/libc.so.6 #1 0x000000000042d1e5 in kvm_main_loop_wait (env=0x1fe75b0, timeout=1000) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1752 #2 0x000000000042d64a in kvm_main_loop_cpu (env=0x1fe75b0) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1896 #3 0x000000000042d76d in ap_main_loop (_env=0x1fe75b0) at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943 #4 0x00007f0b74d983ba in start_thread () from /lib/libpthread.so.0 #5 0x00007f0b73e1afcd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f0b753c76f0 (LWP 25532)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007f0b73e13742 in select () from /lib/libc.so.6 #1 0x000000000040c25a in main_loop_wait (timeout=1000) at /usr/src/qemu-kvm-0.12.4/vl.c:3994 #2 0x000000000042dcf1 in kvm_main_loop () at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:2126 #3 0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.4/vl.c:4212 #4 0x000000000041054b in main (argc=43, argv=0x7fffcc0c3398, envp=0x7fffcc0c34f8) at /usr/src/qemu-kvm-0.12.4/vl.c:6252
Quoted 32 lines · Alexander Graf
Quoted 13 lines · Peter Lieven
Quoted 11 lines · Alexander Graf
Quoted 7 lines · Peter Lieven
hi alex,
unfortunately -cpu host, -cpu qemu64, -cpu core2duo, -cpu kvm64 (which should be default) doesn't help all other cpus available are 32-bit afaik.
as i said if i boot with with kernel parameter "nohpet", but acpi on the guest just hangs at 100% cpu.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
would a backtrace help?
Sigh. Is there any way to upgrade the system to SP3? Preferably incl. the latest maintenance update?
yes, we will try this. if you don't suspect it is a generall problem with qemu-kvm that should be fixed. keep you posted.
Alright. Installing SLES10 SP1 in a VM in parallel.
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Peter,
Quoted 4 lines · Peter Lieven
hi alex,
here is a backtrace of the sles 10 sp1 vm running on qemu-kvm 0.12.4. hanging at boot.
Please do not top post! Seriously. One more time and I'll stop responding.
I tried to reproduce this locally on an openSUSE 11.1 system using latest qemu-kvm.git/stable-0.12 which should basically be 0.12.4:
$ /sbin/modinfo kvm-intel ... version: kvm-kmod-78.2.6.30.1-20.4 ...
agraf@busu:~/git/qemu-kvm> cat /proc/cpuinfo
processor : 0vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : AMD Phenom(tm) 9550 Quad-Core Processor
stepping : 3
cpu MHz : 1100.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips : 4409.11
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
The freshly installed sles10 sp1 image works just fine.
Quoted 52 lines · Alexander Graf
Peter,
Quoted 5 lines · Peter Lieven
hi alex,
here is a backtrace of the sles 10 sp1 vm running on qemu-kvm 0.12.4. hanging at boot.
Please do not top post! Seriously. One more time and I'll stop responding.
I tried to reproduce this locally on an openSUSE 11.1 system using latest qemu-kvm.git/stable-0.12 which should basically be 0.12.4:
$ /sbin/modinfo kvm-intel ... version: kvm-kmod-78.2.6.30.1-20.4 ...agraf@busu:~/git/qemu-kvm> cat
/proc/cpuinfo
processor : 0vendor_id: AuthenticAMD
cpu family : 16
model : 2
model name : AMD Phenom(tm) 9550 Quad-Core Processor
stepping : 3
cpu MHz : 1100.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception: yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxextfxsr_optpdpe1gb rdtscp lm 3dnowext 3dnowconstant_tscrep_goodnopl pni monitor cx16 popcntlahf_lmcmp_legacysvm extapiccr8_legacyabm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips : 4409.11
TLB size : 1024 4K pages
clflush size : 64cache_alignment: 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
Quoted 4 lines
The freshly installed sles10 sp1 image works just fine.
we are running on intel xeons here:
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
kvm-kmod is 2.6.32.7
...
which commandline parameters do you supply to qemu-kvm?
which kernel parameters do you use for the sles 10 guest?
Quoted 1 line · Peter Lieven
we are running on intel xeons here:
That might be the reason. Does it break when passing -no-kvm?
Quoted 31 lines
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception: yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologytsc_reliablenonstop_tscpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmtpr_shadowvnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment: 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Quoted 4 lines
kvm-kmod is 2.6.32.7
...which commandline parameters do you supply to qemu-kvm?
None :)
Quoted 2 lines
which kernel parameters do you use for the sles 10 guest?
Just the default. I took the SP1 DVD, installed it and booted it up. All is fine:
$ ./x86_64-softmmu/qemu-system-x86_64 -snapshot /media/studio/images/SUSE/sles10/sles10sp1.x86_64.raw -vnc :10
Running on Intel might be the culprit though. I'll try and check if booting the image on an Intel box works.
Alex
Quoted 38 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
we are running on intel xeons here:
That might be the reason. Does it break when passing -no-kvm?
Quoted 30 lines
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception: yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologytsc_reliablenonstop_tscpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmtpr_shadowvnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment: 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Quoted 8 lines
Quoted 5 lines
kvm-kmod is 2.6.32.7
...which commandline parameters do you supply to qemu-kvm?
None :)
It seems to stop working if i supply -no-kvm-irqchip. Can you try to reproduce this?
We introduced that parameter because we encountered some problems with the e1000 kernel driver stopped to work in some guests after live migration with a "nobody cared about interupt" (i don't know the exact error anymore). supplying -no-kvm-irqchip made live migration of these guests possible...
Sounds that familiar to someone?
Peter
Quoted 18 lines
Quoted 3 lines
which kernel parameters do you use for the sles 10 guest?
Just the default. I took the SP1 DVD, installed it and booted it up. All is fine:
$ ./x86_64-softmmu/qemu-system-x86_64 -snapshot /media/studio/images/SUSE/sles10/sles10sp1.x86_64.raw -vnc :10Running on Intel might be the culprit though. I'll try and check if booting the image on an Intel box works.
Alex
Quoted 50 lines · Peter Lieven
Quoted 38 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
we are running on intel xeons here:
That might be the reason. Does it break when passing -no-kvm?
Quoted 30 lines
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception: yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologytsc_reliablenonstop_tscpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmtpr_shadowvnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment: 64
address sizes : 40 bits physical, 48 bits virtual
power management:Quoted 8 lines
Quoted 5 lines
kvm-kmod is 2.6.32.7
...which commandline parameters do you supply to qemu-kvm?
None :)
It seems to stop working if i supply -no-kvm-irqchip. Can you try to reproduce this?
Quoted 6 lines
We introduced that parameter because we encountered some problems with the e1000 kernel driver stopped to work in some
guests after live migration with a "nobody cared about interupt" (i don't know the exact error anymore). supplying -no-kvm-irqchip made live migration of these guests possible... Sounds that familiar to someone?
So it works with the in-kernel irqchip? That's the normally supported configuration anyways. If migration fails with that, that's a different thing and definitely needs to be addressed.
Alex
Quoted 64 lines · Alexander Graf
Quoted 54 lines · Peter Lieven
Quoted 40 lines · Alexander Graf
Quoted 3 lines · Peter Lieven
we are running on intel xeons here:
That might be the reason. Does it break when passing -no-kvm?
Quoted 30 lines
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception: yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologytsc_reliablenonstop_tscpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmtpr_shadowvnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment: 64
address sizes : 40 bits physical, 48 bits virtual
power management:Quoted 9 lines
Quoted 6 lines
kvm-kmod is 2.6.32.7
...which commandline parameters do you supply to qemu-kvm?
None :)
It seems to stop working if i supply -no-kvm-irqchip. Can you try to reproduce this?
Quoted 7 lines
We introduced that parameter because we encountered some problems with the e1000 kernel driver stopped to work in some
guests after live migration with a "nobody cared about interupt" (i don't know the exact error anymore). supplying -no-kvm-irqchip made live migration of these guests possible... Sounds that familiar to someone?
Quoted 4 lines
So it works with the in-kernel irqchip? That's the normally supported configuration anyways. If migration fails with that, that's a different thing and definitely needs to be addressed.
Yes, it seems, I got the machine booting ~10 times without any problems.
I will try to reproduce the error condition after migration and come back to the list.
Thanks for your help,
Peter
Quoted 5 lines
Alex
Am 18.05.2010 um 15:51 schrieb Alexander Graf:
Quoted 58 lines
Quoted 50 lines · Peter Lieven
Quoted 38 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
we are running on intel xeons here:
That might be the reason. Does it break when passing -no-kvm?
Quoted 30 lines
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU L5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.403
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception: yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologytsc_reliablenonstop_tscpni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmtpr_shadowvnmi flexpriority ept vpid
bogomips : 4788.80
clflush size : 64cache_alignment: 64
address sizes : 40 bits physical, 48 bits virtual
power management:Quoted 8 lines
Quoted 5 lines
kvm-kmod is 2.6.32.7
...which commandline parameters do you supply to qemu-kvm?
None :)
It seems to stop working if i supply -no-kvm-irqchip. Can you try to reproduce this?
Quoted 6 lines
We introduced that parameter because we encountered some problems with the e1000 kernel driver stopped to work in some
guests after live migration with a "nobody cared about interupt" (i don't know the exact error anymore). supplying -no-kvm-irqchip made live migration of these guests possible... Sounds that familiar to someone?
Quoted 3 lines
So it works with the in-kernel irqchip? That's the normally supported configuration anyways. If migration fails with that, that's a different thing and definitely needs to be addressed.
I got the error reproduced in ubuntu 9.10 server 64-bit and Suse Linux 10.1. I will file a seperate report.
Peter
Quoted 5 lines
Alex
during my tests i also got the following error:
Freeing initrd memory: 2860k freed
not found!
..MP-BIOS bug: 8254 timer not connected to IO-APIC failed.
timer doesn't work through the IO-APIC - disabling NMI Watchdog!
Sorry, ommitted your questions. This particular system is still runnung on
/kernel: /2.6.31-14-server, /bin: /qemu-kvm-0.12.2, /mod: /kvm-kmod-2.6.32.7
If there where any fixes improvements I can try on:
/kernel: /2.6.33.3, /bin: /qemu-kvm-0.12.4, /mod: /2.6.33.3
BR,
Peter
Quoted 103 lines · Alexander Graf
Quoted 40 lines · Peter Lieven
Quoted 6 lines · Alexander Graf
Quoted 2 lines · Peter Lieven
Hi,
Quoted 9 lines
Quoted 6 lines
we try to migrate some Suse Linux Enterprise 10 64-bit guests from abandoned Virtual Iron by Iron Port to qemu-kvm 0.12.4. Unfortunately the guests are not very stable by now. With ACPI they end up in a kernel panic at boot time and without they occasionally hang during boot or shortly after.
Could you please post the panics you get? What does hang during boot mean?
with acpi=off it hangs after starting powersaved
Quoted 4 lines
The easiest way to get them is probably to start the guest with -serial stdio and pass "console=ttyS0" on the grub command line.
without acpi=off it hangs always with a lookup. one time it happened directly after initializing hpet0. btw, is it safe to turn of hpet for linux guests in genereal?
regarding missing kvm-clock. i made the experience that some guests crash after live migration with clocksource=kvm_clock while they don't with clocksource=acpi_pm. this is off topic here, but its also something i would like to debug with someone.
Starting cupsd done
Starting ZENworks Management Daemon done
IA-32 Microcode Update Driver v1.14 unregistered
Starting INET services. (xinetd) done
Checking/updating CPU microcode done
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
..deadQuoted 8 lines
Try to get initial date and time via NTP from 212.110.100.1 done Starting network time protocol daemon (NTPD) done Starting Name Service Cache Daemon doneNetBackup SAN Client Fibre Transport daemon started.
Starting powersaved: done
Starting mail service (Postfix)
NMI Watchdog detected LOCKUP on CPU 0What host kernel are you running on? Or rather, what host kvm module version?
Quoted 4 lines
CPU 0
Modules linked in: ipv6 button battery ac apparmor aamatch_pcre loop usbhid dm_mod 8139cp mii uhci_hcd e1000 i2c_piix4 usbcore ide_cd i2c_core cdrom parport_pc lp parport ext3 jbd sg sym53c8xx scsi_transport_spi edd fan thermal processor piix sd_mod scsi_mod ide_disk ide_corePid: 3134, comm: powersaved Not tainted 2.6.16.46-0.12-smp #1
This looks old. The SP3 GM version is 2.6.16.60-0.54.5. This release is definitely out of support :(.
After some searching I found the RPM. That's the SLES10 SP1 GM kernel version. I don't think anyone ever tested that one on kvm.
Quoted 2 lines
RIP: 0010:[<ffffffff802dabf1>] <ffffffff802dabf1>{paranoid_restore+81}Yeah, great. That one's not helpful at all:
0xffffffff802dabf1<paranoid_restore+81>: iretqQuoted 17 lines
RSP: 0000:ffffffff8041afd8 EFLAGS: 00010086 RAX: ffffffff88021d50 RBX: ffffffff88021e18 RCX: 00000000b4c562b6 RDX: 00000000000001f7 RSI: ffffffff88021e18 RDI: 00000000000001f7 RBP: 0000000000000000 R08: 00000000000004e2 R09: ffffffff80417d60 R10: 000000000000001f R11: ffffffff8800752c R12: ffffffff88021d00 R13: 00000000000004e2 R14: ffffffff80417dac R15: 0000000000000040 FS: 00002ae1d26ee760(0000) GS:ffffffff803be000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000555b30 CR3: 0000000210fe9000 CR4: 00000000000006e0 Process powersaved (pid: 3134, threadinfo ffff810210ff2000, task ffff810211f83850) Stack: ffffffff802dabf1 0000000000000010 0000000000010086 ffffffff8041afd8 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Call Trace: <NMI> <ffffffff802dabf1>{paranoid_restore+81} <EOE> <ffffffff802dabf1>{paranoid_restore+81} Code: 48 cf 65 48 8b 0c 25 10 00 00 00 48 81 e9 d8 1f 00 00 8b 59Quoted 3 lines
console shuts up ...
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!Please try out different values for -cpu. I think powersaved gets confused by the invalid CPU type we're emulating.
Alex
Signature · 16 lines
Mit freundlichen Grüßen/Kind Regards Peter Lieven .......................................................................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 mailto:[email protected] | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 .........................................................................................................
Hi Peter,
Quoted 2 lines · Peter Lieven
Sorry, ommitted your questions. This particular system is still runnung on
Please don't top post.
Quoted 2 lines
/kernel: /2.6.31-14-server, /bin: /qemu-kvm-0.12.2, /mod: /kvm-kmod-2.6.32.7
This looks reasonable.
Alex
Quoted 2 lines · Peter Lieven
Starting mail service (Postfix)
NMI Watchdog detected LOCKUP on CPU 0
You could simply turn off the NMI watchdog (nmi_watchdog=0 at the kernel command line)
Perhaps the PMU emulation is not complete and nmi watchdog needs PMU. It's not really needed normally.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.