FreeBSD boot hangs on qemu-kvm on AMD host
Hello.
Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running
kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.
The guest unpacks kernel, draws a menu (apparently in real kernel already, not in the boot loader) and starts a 10-second countdown. At this point it just stops (either at 10 or 9, ie, right at the beginning of the cycle) and kvm process starts using 100% of the CPU core given to it. With this typical kvm_stats from qemu-kvm 0.14:
kvm statistics
exits 43397783 1400960
halt_exits 42862622 1400769
irq_exits 5487 172
host_state_reload 300074 101
io_exits 292212 18
irq_injections 768 18
halt_wakeup 257 3 mmu_pte_write 7702163 0 insn_emulation 378273 0 mmio_exits 15786 0 pf_fixed 4817 0 mmu_shadow_zapped 170 0 mmu_cache_miss 151 0 remote_tlb_flush 2 0 signal_exits 1 0 fpu_reload 1 0
I tried i386 and amd64 guest images, and tried the same thing with qemu-kvm 0.12.5 and 0.14.0 - the effect is exactly the same.
So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.
Running it with -no-kvm lets the guest to go (but obviously very slow). Using -no-kvm-irqchip helps too.
Any guess what it can be?
Thanks!
/mjt
Quoted 38 lines · Michael Tokarev
Hello.
Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running
kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.
The guest unpacks kernel, draws a menu (apparently in real kernel already, not in the boot loader) and starts a 10-second countdown. At this point it just stops (either at 10 or 9, ie, right at the beginning of the cycle) and kvm process starts using 100% of the CPU core given to it. With this typical
kvm_statsfrom qemu-kvm 0.14:kvm statistics
exits 43397783 1400960
halt_exits42862622 1400769
irq_exits5487 172
host_state_reload300074 101
io_exits292212 18
irq_injections768 18halt_wakeup 257 3 mmu_pte_write 7702163 0 insn_emulation 378273 0 mmio_exits 15786 0 pf_fixed 4817 0 mmu_shadow_zapped 170 0 mmu_cache_miss 151 0 remote_tlb_flush 2 0 signal_exits 1 0 fpu_reload 1 0
Quoted 8 lines
I tried i386 and amd64 guest images, and tried the same thing with qemu-kvm 0.12.5 and 0.14.0 - the effect is exactly the same.
So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.
Can't reproduce. What host kernel version are you using?
Please post the output of
info register
x/50i $eip-30
on the qemu monitor.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Michael Tokarev
Hello.
Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running
kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.
[]
Quoted 3 lines
Quoted 3 lines
So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.
I also tried on another AMD machine, with TurionI, -- the same effect.
Quoted 1 line
Can't reproduce. What host kernel version are you using?
And this is a very good question, this is what I forgot to mention.
It hangs here - in all cases - with 2.6.37+ kernels. I tried 2.6.35 on one of my boxes, and there it works just fine. We're testing various kernels now, or trying to.
So I'm not sure anymore it's actually AMD-specific, but at least one person tried debian 2.6.37-2-amd64 kernel on a core-i7 box and that combination works.
Quoted 3 lines
Please post the output of
info register
And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...
Quoted 1 line
x/50i $eip-30
...so I can't get this information anymore. Here it goes with an explicit "stop", fwiw:
$ kvm -monitor stdio -cdrom /stage/tmp/FreeBSD-8.2-RELEASE-i386-bootonly.iso QEMU 0.14.0 monitor - type 'help' for more information
(qemu) stop
(qemu) info registers
EAX=00000042 EBX=00009f8b ECX=00000042 EDX=00005dd0
ESI=00000001 EDI=0000c350 EBP=00000000 ESP=00005d9c
EIP=0000f3b4 EFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =9f40 0009f400 ffffffff 00c09300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 ffffffff 00c09300
DS =0000 00000000 ffffffff 00c09300
FS =0000 00000000 ffffffff 00c0f300
GS =0000 00000000 ffffffff 00c0f300
LDT=0000 00000000 0000ffff 00008200
TR =0038 00005f98 00002067 00008b00
GDT= 00009590 0000003f
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
(qemu) x/50i $eip-30
0x000000000000f396: add 0xc(%di),%ax 0x000000000000f399: add -0xb0(%di),%ax 0x000000000000f39d: (bad) 0x000000000000f39e: decw 0x6885(%bx,%di) 0x000000000000f3a2: (bad) 0x000000000000f3a3: (bad) 0x000000000000f3a4: ljmp *<internal disassembler error> 0x000000000000f3a6: out %al,$0xfe 0x000000000000f3a8: (bad) 0x000000000000f3a9: decw 0x856(%bp,%di) 0x000000000000f3ad: mov %dx,-0xb0(%di) 0x000000000000f3b1: (bad) 0x000000000000f3b2: ljmp *<internal disassembler error> 0x000000000000f3b4: jnp 0xf3b5 0x000000000000f3b6: (bad) 0x000000000000f3b7: ljmp *(%bx,%di) 0x000000000000f3b9: rorw %cl,0xc45(%bp,%di) 0x000000000000f3bd: mov %bx,0x24(%si) 0x000000000000f3c0: add $0x3,%al 0x000000000000f3c2: inc %si 0x000000000000f3c3: or %al,(%bx,%di) 0x000000000000f3c5: rorb 0x2404(%bx,%di) 0x000000000000f3c9: call 0xd58e 0x000000000000f3cc: (bad) 0x000000000000f3cd: decw 0x1446(%bp,%di) 0x000000000000f3d1: mov %ax,-0xb4(%di) 0x000000000000f3d5: (bad) 0x000000000000f3d6: ljmp *<internal disassembler error> 0x000000000000f3d8: and %bh,%bh 0x000000000000f3da: (bad) 0x000000000000f3db: decw 0x29d8(%bx,%di) 0x000000000000f3df: fiaddl (%bp,%di) 0x000000000000f3e1: inc %si 0x000000000000f3e2: add $0xc7,%al 0x000000000000f3e4: inc %sp 0x000000000000f3e5: and $0x10,%al 0x000000000000f3e7: add %al,(%bx,%si) 0x000000000000f3e9: add %al,(%bx,%si) 0x000000000000f3eb: mov %dx,0x24(%si) 0x000000000000f3ee: or %cl,0x2444(%bx,%di) 0x000000000000f3f2: or $0x8b,%al 0x000000000000f3f4: inc %bp 0x000000000000f3f5: or $0x3,%al 0x000000000000f3f7: inc %si 0x000000000000f3f8: or %al,(%bx,%di) 0x000000000000f3fa: fmuls 0x2444(%bx,%di) 0x000000000000f3fe: add $0x8b,%al 0x000000000000f400: inc %bp 0x000000000000f401: movsb %ds:(%si),%es:(%di) 0x000000000000f402: mov %ax,(%si)
(qemu) _
Thanks!
/mjt
Quoted 7 lines · Michael Tokarev
Quoted 3 lines
Please post the output of
info register
And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...
Probably one of the KVM_REQ_EVENT changes. Please try kvm.git and report, it's likely fixed.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Michael Tokarev
Quoted 3 lines
Please post the output of
info register
Quoted 8 lines
Quoted 3 lines
And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...
Probably one of the
KVM_REQ_EVENTchanges. Please try kvm.git and report, it's likely fixed.
Or try this patch from kvm.git: 09493165180e69a1dcdb44e049015e31c6de57e2
--
Gleb.
09.03.2011 17:24, Gleb Natapov пишет:
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Michael Tokarev
Quoted 4 lines
Quoted 3 lines
Please post the output of
info register
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...
Probably one of the
KVM_REQ_EVENTchanges. Please try kvm.git and report, it's likely fixed.Or try this patch from kvm.git:
09493165180e69a1dcdb44e049015e31c6de57e2
Nope, that's not it. Applying this commit has no
(visible) effect on this hang.
I'll try kvm.git later today, hopefully.
Thanks!
/mjt
Quoted 2 lines
Please try kvm.git and
report, it's likely fixed.
With kvm.git kernel it works.
Here's what I did:
git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config
# set LOCALVERSION to -kvm.git.20110309 make modules bzImage make modules_install install # build initrd reboot /usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom /root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso vncviewer localhost:0 # visually verify that it does not hang as it did before
Regards
Dominik
Quoted 10 lines · Dominik Klein
Quoted 2 lines
Please try kvm.git and
report, it's likely fixed.With kvm.git kernel it works.
Here's what I did:
git clone
git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config
Quoted 9 lines
# set LOCALVERSION to -kvm.git.20110309 make modules bzImage make modules_install install # build initrd reboot /usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom /root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso vncviewer localhost:0 # visually verify that it does not hang as it did before
Please, on this kernel, run a trace per http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.
Signature · 1 line
error compiling committee.c: too many arguments to function
09.03.2011 18:38, Avi Kivity пишет:
Quoted 21 lines · Dominik Klein
Quoted 10 lines
Quoted 2 lines
Please try kvm.git and
report, it's likely fixed.With kvm.git kernel it works.
Here's what I did:
git clone
git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .configQuoted 9 lines
# set LOCALVERSION to -kvm.git.20110309 make modules bzImage make modules_install install # build initrd reboot /usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom /root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso vncviewer localhost:0 # visually verify that it does not hang as it did before
Quoted 3 lines
Please, on this kernel, run a trace per
http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.
http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz
(Yes, this kernel works for me too).
Thanks!
/mjt
Quoted 22 lines · Michael Tokarev
09.03.2011 18:38, Avi Kivity пишет:
Quoted 21 lines · Dominik Klein
Quoted 19 lines
Quoted 2 lines
Please try kvm.git and
report, it's likely fixed.With kvm.git kernel it works.
Here's what I did:
git clone
git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config# set LOCALVERSION to -kvm.git.20110309 make modules bzImage make modules_install install # build initrd reboot /usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom /root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso vncviewer localhost:0 # visually verify that it does not hang as it did before
Quoted 8 lines
Quoted 3 lines
Please, on this kernel, run a trace per
http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz
(Yes, this kernel works for me too).
So the trace is for a working kernel? That's not very interesting (well maybe later to compare against).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines
http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz
(Yes, this kernel works for me too).
Quoted 1 line
# visually verify that it does not hang as it did beforeSo the trace is for a working kernel? That's not very interesting (well maybe later to compare against).
Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.
What's the last kernel that *didn't* work for you?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.
What's the last kernel that *didn't* work for you?
2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.
I know it warrants a very nice bisection :)
/mjt
Quoted 11 lines · Michael Tokarev
Quoted 4 lines · Avi Kivity
Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.
What's the last kernel that *didn't* work for you?
2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.I know it warrants a very nice bisection :)
There's some chance it's e5d135f80b98b0.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Michael Tokarev
Quoted 1 line · Avi Kivity
Sorry, I misread. So kvm.git works, we just have to identify what
patch
Quoted 3 lines
fixed it.
What's the last kernel that *didn't* work for you?
Quoted 8 lines
Quoted 5 lines
2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.I know it warrants a very nice bisection :)
There's some chance it's
e5d135f80b98b0.
If you apply it, also apply the following commit 2607b0533353c since it fixes a bug that tends to be exposed by the previous patch.
Signature · 1 line
error compiling committee.c: too many arguments to function
09.03.2011 19:34, Avi Kivity wrote:
[]
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.
2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.There's some chance it's
e5d135f80b98b0.If you apply it, also apply the following commit
2607b0533353csince it fixes a bug that tends to be exposed by the previous patch.
So, the two mentioned patches on top of 2.6.37.3 makes the problem go away, -- guest runs flawlessly with the two patches applied. I had to apply kvm_host.h change manually since there's one extra #define (KVM_REQ_APF_HALT) which is not present in 2.6.37.
So.. -stable candidates, after applying them for 2.6.38? :)
Thanks!
/mjt
Quoted 12 lines · Michael Tokarev
09.03.2011 19:34, Avi Kivity wrote:
[]Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.
2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.There's some chance it's
e5d135f80b98b0.If you apply it, also apply the following commit
2607b0533353csince it fixes a bug that tends to be exposed by the previous patch.
Quoted 8 lines
So, the two mentioned patches on top of 2.6.37.3 makes the problem go away, -- guest runs flawlessly with the two patches applied. I had to apply
kvm_host.hchange manually since there's one extra #define (KVM_REQ_APF_HALT) which is not present in 2.6.37.So.. -stable candidates, after applying them for 2.6.38? :)
Certainly. Thanks for testing.
Signature · 1 line
error compiling committee.c: too many arguments to function