Skip to contentSkip to conversation

FreeBSD boot hangs on qemu-kvm on AMD host

DISCUSSION Michael Tokarev 4 participants 16 messages Mar 9, 2011 – Mar 10, 2011 KVM

lore
Share & tools
lore
#0 lore raw

Hello.

Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running

kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.

The guest unpacks kernel, draws a menu (apparently in real kernel already, not in the boot loader) and starts a 10-second countdown. At this point it just stops (either at 10 or 9, ie, right at the beginning of the cycle) and kvm process starts using 100% of the CPU core given to it. With this typical kvm_stats from qemu-kvm 0.14:

kvm statistics

exits 43397783 1400960
halt_exits 42862622 1400769
irq_exits 5487 172
host_state_reload 300074 101
io_exits 292212 18
irq_injections 768 18

 halt_wakeup                                    257       3
 mmu_pte_write                              7702163       0
 insn_emulation                              378273       0
 mmio_exits                                   15786       0
 pf_fixed                                      4817       0
 mmu_shadow_zapped                              170       0
 mmu_cache_miss                                 151       0
 remote_tlb_flush                                 2       0
 signal_exits                                     1       0
 fpu_reload                                       1       0

I tried i386 and amd64 guest images, and tried the same thing with qemu-kvm 0.12.5 and 0.14.0 - the effect is exactly the same.

So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.

Running it with -no-kvm lets the guest to go (but obviously very slow). Using -no-kvm-irqchip helps too.

Any guess what it can be?

Thanks!

/mjt

#1 lore raw
Quoted 38 lines · Michael Tokarev

Hello.

Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running

kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.

The guest unpacks kernel, draws a menu (apparently in real kernel already, not in the boot loader) and starts a 10-second countdown. At this point it just stops (either at 10 or 9, ie, right at the beginning of the cycle) and kvm process starts using 100% of the CPU core given to it. With this typical kvm_stats from qemu-kvm 0.14:

kvm statistics

exits 43397783 1400960
halt_exits 42862622 1400769
irq_exits 5487 172
host_state_reload 300074 101
io_exits 292212 18
irq_injections 768 18

  halt_wakeup                                    257       3
  mmu_pte_write                              7702163       0
  insn_emulation                              378273       0
  mmio_exits                                   15786       0
  pf_fixed                                      4817       0
  mmu_shadow_zapped                              170       0
  mmu_cache_miss                                 151       0
  remote_tlb_flush                                 2       0
  signal_exits                                     1       0
  fpu_reload                                       1       0
Quoted 8 lines

I tried i386 and amd64 guest images, and tried the same thing with qemu-kvm 0.12.5 and 0.14.0 - the effect is exactly the same.

So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.

Can't reproduce. What host kernel version are you using?

Please post the output of

info register

x/50i $eip-30

on the qemu monitor.

Signature · 1 line
error compiling committee.c: too many arguments to function
#2 lore raw
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Michael Tokarev

Hello.

Dominik Klein (Cc'd) reported on irc that current FreeBSD release hangs at boot in qemu-kvm. It is trivially verifiable by d/loading the bootonly FreeBSD 8.2 CD image from ftp.freebsd.org and running

kvm -cdrom FreeBSD-8.2-RELEASE-amd64-bootonly.iso.

[]

Quoted 3 lines
Quoted 3 lines

So far the 3 affected systems where we tried this are all AMD machines. My home box is AthlonII, my work box is PhenomI 9750, and Dominik tries it on Opteron 6174.

I also tried on another AMD machine, with TurionI, -- the same effect.

Quoted 1 line

Can't reproduce. What host kernel version are you using?

And this is a very good question, this is what I forgot to mention.

It hangs here - in all cases - with 2.6.37+ kernels. I tried 2.6.35 on one of my boxes, and there it works just fine. We're testing various kernels now, or trying to.

So I'm not sure anymore it's actually AMD-specific, but at least one person tried debian 2.6.37-2-amd64 kernel on a core-i7 box and that combination works.

Quoted 3 lines

Please post the output of

info register

And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...

Quoted 1 line

x/50i $eip-30

...so I can't get this information anymore. Here it goes with an explicit "stop", fwiw:

$ kvm -monitor stdio -cdrom /stage/tmp/FreeBSD-8.2-RELEASE-i386-bootonly.iso
QEMU 0.14.0 monitor - type 'help' for more information

(qemu) stop

(qemu) info registers
EAX=00000042 EBX=00009f8b ECX=00000042 EDX=00005dd0
ESI=00000001 EDI=0000c350 EBP=00000000 ESP=00005d9c
EIP=0000f3b4 EFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =9f40 0009f400 ffffffff 00c09300
CS =f000 000f0000 0000ffff 00009b00
SS =0000 00000000 ffffffff 00c09300
DS =0000 00000000 ffffffff 00c09300
FS =0000 00000000 ffffffff 00c0f300
GS =0000 00000000 ffffffff 00c0f300
LDT=0000 00000000 0000ffff 00008200
TR =0038 00005f98 00002067 00008b00
GDT= 00009590 0000003f
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

(qemu) x/50i $eip-30

0x000000000000f396:  add    0xc(%di),%ax
0x000000000000f399:  add    -0xb0(%di),%ax
0x000000000000f39d:  (bad)
0x000000000000f39e:  decw   0x6885(%bx,%di)
0x000000000000f3a2:  (bad)
0x000000000000f3a3:  (bad)
0x000000000000f3a4:  ljmp   *<internal disassembler error>
0x000000000000f3a6:  out    %al,$0xfe
0x000000000000f3a8:  (bad)
0x000000000000f3a9:  decw   0x856(%bp,%di)
0x000000000000f3ad:  mov    %dx,-0xb0(%di)
0x000000000000f3b1:  (bad)
0x000000000000f3b2:  ljmp   *<internal disassembler error>
0x000000000000f3b4:  jnp    0xf3b5
0x000000000000f3b6:  (bad)
0x000000000000f3b7:  ljmp   *(%bx,%di)
0x000000000000f3b9:  rorw   %cl,0xc45(%bp,%di)
0x000000000000f3bd:  mov    %bx,0x24(%si)
0x000000000000f3c0:  add    $0x3,%al
0x000000000000f3c2:  inc    %si
0x000000000000f3c3:  or     %al,(%bx,%di)
0x000000000000f3c5:  rorb   0x2404(%bx,%di)
0x000000000000f3c9:  call   0xd58e
0x000000000000f3cc:  (bad)
0x000000000000f3cd:  decw   0x1446(%bp,%di)
0x000000000000f3d1:  mov    %ax,-0xb4(%di)
0x000000000000f3d5:  (bad)
0x000000000000f3d6:  ljmp   *<internal disassembler error>
0x000000000000f3d8:  and    %bh,%bh
0x000000000000f3da:  (bad)
0x000000000000f3db:  decw   0x29d8(%bx,%di)
0x000000000000f3df:  fiaddl (%bp,%di)
0x000000000000f3e1:  inc    %si
0x000000000000f3e2:  add    $0xc7,%al
0x000000000000f3e4:  inc    %sp
0x000000000000f3e5:  and    $0x10,%al
0x000000000000f3e7:  add    %al,(%bx,%si)
0x000000000000f3e9:  add    %al,(%bx,%si)
0x000000000000f3eb:  mov    %dx,0x24(%si)
0x000000000000f3ee:  or     %cl,0x2444(%bx,%di)
0x000000000000f3f2:  or     $0x8b,%al
0x000000000000f3f4:  inc    %bp
0x000000000000f3f5:  or     $0x3,%al
0x000000000000f3f7:  inc    %si
0x000000000000f3f8:  or     %al,(%bx,%di)
0x000000000000f3fa:  fmuls  0x2444(%bx,%di)
0x000000000000f3fe:  add    $0x8b,%al
0x000000000000f400:  inc    %bp
0x000000000000f401:  movsb  %ds:(%si),%es:(%di)
0x000000000000f402:  mov    %ax,(%si)

(qemu) _

Thanks!

/mjt

#3 lore raw
Quoted 7 lines · Michael Tokarev
Quoted 3 lines

Please post the output of

info register

And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...

Probably one of the KVM_REQ_EVENT changes. Please try kvm.git and report, it's likely fixed.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Michael Tokarev
Quoted 3 lines

Please post the output of

info register

Quoted 8 lines
Quoted 3 lines

And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...

Probably one of the KVM_REQ_EVENT changes. Please try kvm.git and report, it's likely fixed.

Or try this patch from kvm.git: 09493165180e69a1dcdb44e049015e31c6de57e2

--
Gleb.

#5 lore raw

09.03.2011 17:24, Gleb Natapov пишет:

Quoted 6 lines · Avi Kivity
Quoted 5 lines · Michael Tokarev
Quoted 4 lines
Quoted 3 lines

Please post the output of

info register

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

And this is where the most fun begins. Right after `info registers' it "unhangs" and continues as if nothing happened, ...

Probably one of the KVM_REQ_EVENT changes. Please try kvm.git and report, it's likely fixed.

Or try this patch from kvm.git: 09493165180e69a1dcdb44e049015e31c6de57e2

Nope, that's not it. Applying this commit has no
(visible) effect on this hang.

I'll try kvm.git later today, hopefully.

Thanks!

/mjt

#6 lore raw
In reply to Avi Kivity
Quoted 2 lines

Please try kvm.git and
report, it's likely fixed.

With kvm.git kernel it works.

Here's what I did:

git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config

# set LOCALVERSION to -kvm.git.20110309
make modules bzImage
make modules_install install
# build initrd
reboot
/usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom
/root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso
vncviewer localhost:0
# visually verify that it does not hang as it did before

Regards
Dominik

#7 lore raw
Quoted 10 lines · Dominik Klein
Quoted 2 lines

Please try kvm.git and
report, it's likely fixed.

With kvm.git kernel it works.

Here's what I did:

git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config

Quoted 9 lines
# set LOCALVERSION to -kvm.git.20110309
make modules bzImage
make modules_install install
# build initrd
reboot
/usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom
/root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso
vncviewer localhost:0
# visually verify that it does not hang as it did before

Please, on this kernel, run a trace per http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw

09.03.2011 18:38, Avi Kivity пишет:

Quoted 21 lines · Dominik Klein
Quoted 10 lines
Quoted 2 lines

Please try kvm.git and
report, it's likely fixed.

With kvm.git kernel it works.

Here's what I did:

git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config

Quoted 9 lines
# set LOCALVERSION to -kvm.git.20110309
make modules bzImage
make modules_install install
# build initrd
reboot
/usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom
/root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso
vncviewer localhost:0
# visually verify that it does not hang as it did before
Quoted 3 lines

Please, on this kernel, run a trace per
http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.

http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz

(Yes, this kernel works for me too).

Thanks!

/mjt

#9 lore raw
Quoted 22 lines · Michael Tokarev

09.03.2011 18:38, Avi Kivity пишет:

Quoted 21 lines · Dominik Klein
Quoted 19 lines
Quoted 2 lines

Please try kvm.git and
report, it's likely fixed.

With kvm.git kernel it works.

Here's what I did:

git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd kvm
cp /boot/config-$(uname -r) .config

 # set LOCALVERSION to -kvm.git.20110309
 make modules bzImage
 make modules_install install
 # build initrd
 reboot
 /usr/bin/qemu-kvm -vnc 127.0.0.1:0 -cdrom
 /root/isos/FreeBSD-8.2-RELEASE-amd64-disc1.iso
 vncviewer localhost:0
 # visually verify that it does not hang as it did before
Quoted 8 lines
Quoted 3 lines

Please, on this kernel, run a trace per
http://www.linux-kvm.org/page/Tracing. Stop trace-cmd early so the trace file doesn't blow up.

http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz

(Yes, this kernel works for me too).

So the trace is for a working kernel? That's not very interesting (well maybe later to compare against).

Signature · 1 line
error compiling committee.c: too many arguments to function
#10 lore raw
Quoted 9 lines · Avi Kivity
Quoted 6 lines

http://www.corpit.ru/mjt/tmp/kvm-fbsd-boot-trace.txt.gz

(Yes, this kernel works for me too).

Quoted 1 line
 # visually verify that it does not hang as it did before

So the trace is for a working kernel? That's not very interesting (well maybe later to compare against).

Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.

What's the last kernel that *didn't* work for you?

Signature · 1 line
error compiling committee.c: too many arguments to function
#11 lore raw
Quoted 4 lines · Avi Kivity

Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.

What's the last kernel that *didn't* work for you?

2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.

I know it warrants a very nice bisection :)

/mjt

#12 lore raw
Quoted 11 lines · Michael Tokarev
Quoted 4 lines · Avi Kivity

Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.

What's the last kernel that *didn't* work for you?

2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.

I know it warrants a very nice bisection :)

There's some chance it's e5d135f80b98b0.

Signature · 1 line
error compiling committee.c: too many arguments to function
#13 lore raw
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Michael Tokarev
Quoted 1 line · Avi Kivity

Sorry, I misread. So kvm.git works, we just have to identify what

patch

Quoted 3 lines

fixed it.

What's the last kernel that *didn't* work for you?

Quoted 8 lines
Quoted 5 lines

2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.

I know it warrants a very nice bisection :)

There's some chance it's e5d135f80b98b0.

If you apply it, also apply the following commit 2607b0533353c since it fixes a bug that tends to be exposed by the previous patch.

Signature · 1 line
error compiling committee.c: too many arguments to function
#14 lore raw

09.03.2011 19:34, Avi Kivity wrote:
[]

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.

2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.

There's some chance it's e5d135f80b98b0.

If you apply it, also apply the following commit 2607b0533353c since it fixes a bug that tends to be exposed by the previous patch.

So, the two mentioned patches on top of 2.6.37.3 makes the problem go away, -- guest runs flawlessly with the two patches applied. I had to apply kvm_host.h change manually since there's one extra #define (KVM_REQ_APF_HALT) which is not present in 2.6.37.

So.. -stable candidates, after applying them for 2.6.38? :)

Thanks!

/mjt

#15 lore raw
Quoted 12 lines · Michael Tokarev

09.03.2011 19:34, Avi Kivity wrote:
[]

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Sorry, I misread. So kvm.git works, we just have to identify what patch fixed it.

2.6.38-rc8 does not work - verified by Dominik.
But kvm.git works.

There's some chance it's e5d135f80b98b0.

If you apply it, also apply the following commit 2607b0533353c since it fixes a bug that tends to be exposed by the previous patch.

Quoted 8 lines

So, the two mentioned patches on top of 2.6.37.3 makes the problem go away, -- guest runs flawlessly with the two patches applied. I had to apply kvm_host.h change manually since there's one extra #define (KVM_REQ_APF_HALT) which is not present in 2.6.37.

So.. -stable candidates, after applying them for 2.6.38? :)

Certainly. Thanks for testing.

Signature · 1 line
error compiling committee.c: too many arguments to function