FreeBSD-amd64 fails to start with SMP on quemu-kvm

DISCUSSION Artur Samborski 4 participants 11 messages Jan 7, 2013 – Jan 17, 2013 KVM

Share & tools

Artur Samborski

author

2013-01-07 17:23 UTC #0 lore raw

Hello,

When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:

qemu-kvm -m 1024m -cpu host -smp 2 -cdrom /storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso

it freezes KVM with:

KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A]
SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00

Freeze occurs immediately after FreeBSD kernel messages:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012
[email protected]:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz (2925.91-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a
Stepping = 5

Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
real memory = 1073741824 (1024 MB)
avail memory = 1011343360 (964 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BOCHS BXPCAPIC>

so just prior to probing of SMP.

This also applies to FreeBSD-7.3-RELEASE-amd64 and FreeBSD-9.1-RC3-amd64 (other releases not tested).

When quemu-kvm is started without SMP (1 vpcu) amd64 FreeBSD kernel boots correctly. I did not notice this problem (SMP) for the i386 versions (FreeBSD-7.3-RELEASE-i386, FreeBSD-9.0-RELEASE-i386, FreeBSD-9.1-RC3-i386).

Additional info:

- KVM Host OS:
Fedora Core 17

- CPUs on my KVM host -- Xeons X5570

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping        : 5
microcode       : 0x11
cpu MHz         : 2926.183
cache size      : 8192 KB
physical id     : 1
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 16
initial apicid  : 16
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 5852.36
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

- kernel (from FC17 repo):
3.6.9 (kernel-3.6.9-2.fc17.x86_64)
- quemu version:
qemu-kvm 1.0.1 (qemu-kvm-1.0.1-2.fc17.x86_64)
- neither the -no-kvm-irqchip nor -no-kvm-pit switch helps
- with he -no-kvm switch FreeBSD boots correctly
- linux guest (x86_64 with SMP) works perfectly ok

I suspect that this bug is related in some way with the hardware. I tested the same KVM-host system (exact clone) with the same guest (FreeBSD-amd64) on another machine (i3-2120 workstation) and have not noticed a similar problems witch SMP.

I will be grateful for any hints.

Regards,
Artur Samborski

Marcelo Tosatti

2013-01-07 23:03 UTC #1 lore raw

Quoted 31 lines · Artur Samborski

Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff

Quoted 8 lines

CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00

Artur,

Can you check whether

https://patchwork-mail.kernel.org/patch/1942681/

fixes your problem

TIA

Artur Samborski

author

2013-01-08 20:45 UTC #2 lore raw

W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:

Quoted 41 lines · Artur Samborski

Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00

Quoted 8 lines

Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem

Hi, thanks for the reply.

Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.

The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.

Do I need some additional patches?

Signature · 1 line

Artur

Gleb Natapov

2013-01-09 10:15 UTC #3 lore raw

Quoted 53 lines · Artur Samborski

W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.

Quoted 9 lines

Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?

Can you try queue branch from kvm.git?

--
Gleb.

Artur Samborski

author

2013-01-11 14:57 UTC #4 lore raw

W dniu 09.01.2013 11:15, Gleb Natapov pisze:

Quoted 58 lines · Artur Samborski

Quoted 57 lines
W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.
Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.

Quoted 7 lines

Quoted 5 lines
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?
Can you try queue branch from kvm.git?

Thanks for the advice - I used kernel from kvm.git(queue) and was able to run FreeBSD-amd guest with SMP on my kvm-host server without previous problems. Unfortunately, after few hours server was hung up. Case requires further investigations, but generally speaking we went forward. Unfortunately, experiments on the server are difficult, because it is used for everyday work.

Signature · 1 line

Artur

Gleb Natapov

2013-01-13 09:23 UTC #5 lore raw

Quoted 67 lines · Artur Samborski

W dniu 09.01.2013 11:15, Gleb Natapov pisze:
Quoted 58 lines · Artur Samborski
Quoted 57 lines
W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.
Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.
Quoted 7 lines
Quoted 5 lines
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?
Can you try queue branch from kvm.git?

Quoted 7 lines

Thanks for the advice - I used kernel from kvm.git(queue) and was able to run FreeBSD-amd guest with SMP on my kvm-host server without previous problems. Unfortunately, after few hours server was hung up. Case requires further investigations, but generally speaking we went forward. Unfortunately, experiments on the server are difficult, because it is used for everyday work.

Thanks for testing. This hang indeed looks like completely different problem. What workload the VM was running?

--
Gleb.

Artur Samborski

author

2013-01-15 11:19 UTC #6 lore raw

W dniu 13.01.2013 10:23, Gleb Natapov pisze:

Quoted 75 lines · Artur Samborski

Quoted 67 lines
W dniu 09.01.2013 11:15, Gleb Natapov pisze:
Quoted 58 lines · Artur Samborski
Quoted 57 lines
W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.
Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.
Quoted 7 lines
Quoted 5 lines
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?
Can you try queue branch from kvm.git?
Quoted 7 lines
Thanks for the advice - I used kernel from kvm.git(queue) and was able to run FreeBSD-amd guest with SMP on my kvm-host server without previous problems. Unfortunately, after few hours server was hung up. Case requires further investigations, but generally speaking we went forward. Unfortunately, experiments on the server are difficult, because it is used for everyday work.

Quoted 3 lines

Thanks for testing. This hang indeed looks like completely different problem. What workload the VM was running?

First test:

- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world and
kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:

kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! [vhost-3686:3700]

Second test:

- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:

BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>] skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task ffff880421151740)
Stack:
 ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
 000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
 0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
 [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
 [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
 [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
 [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
 [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
 [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
 [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
 [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
 [<ffffffff81081750>] kthread+0xc0/0xd0
 [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49 03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb

repeated several times with similar content

Signature · 1 line

Artur

Gleb Natapov

2013-01-16 13:06 UTC #7 lore raw

Copying Michael.

Quoted 151 lines · Artur Samborski

W dniu 13.01.2013 10:23, Gleb Natapov pisze:
Quoted 75 lines · Artur Samborski
Quoted 67 lines
W dniu 09.01.2013 11:15, Gleb Natapov pisze:
Quoted 58 lines · Artur Samborski
Quoted 57 lines
W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.
Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.
Quoted 7 lines
Quoted 5 lines
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?
Can you try queue branch from kvm.git?
Quoted 7 lines
Thanks for the advice - I used kernel from kvm.git(queue) and was able to run FreeBSD-amd guest with SMP on my kvm-host server without previous problems. Unfortunately, after few hours server was hung up. Case requires further investigations, but generally speaking we went forward. Unfortunately, experiments on the server are difficult, because it is used for everyday work.
Quoted 3 lines
Thanks for testing. This hang indeed looks like completely different problem. What workload the VM was running?
First test:
- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
and kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:
kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! [vhost-3686:3700]
Second test:
- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:
BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables
ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>]
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
ffff880421151740)
Stack:
 ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
 000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
 0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
 [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
 [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
 [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
 [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
 [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
 [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
 [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
 [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
 [<ffffffff81081750>] kthread+0xc0/0xd0
 [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49

Quoted 7 lines

03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb
repeated several times with similar content
Signature · 1 line
Artur

--
Gleb.

Michael S. Tsirkin

2013-01-16 14:38 UTC #8 lore raw

Quoted 162 lines · Gleb Natapov

Copying Michael.
Quoted 158 lines · Artur Samborski
W dniu 13.01.2013 10:23, Gleb Natapov pisze:
Quoted 75 lines · Artur Samborski
Quoted 67 lines
W dniu 09.01.2013 11:15, Gleb Natapov pisze:
Quoted 58 lines · Artur Samborski
Quoted 57 lines
W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
Quoted 41 lines · Artur Samborski
Quoted 31 lines
Hello,
When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
/storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
it freezes KVM with:
KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080 RDX=0000000000000000
RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A] SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000000009f080 00000020
IDT= 0000000000000000 0000ffff
Quoted 8 lines
CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000501
Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80 <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 99 20 00 ff ff 00 00
Quoted 8 lines
Artur,
Can you check whether
https://patchwork-mail.kernel.org/patch/1942681/
fixes your problem
Hi, thanks for the reply.
Unfortunately, the patch does not help. Attempt to start FreeBSD amd64 via quemu-kvm with -smp parameter greater than 1 fails in exactly the same way as before.
Quoted 7 lines
Quoted 5 lines
The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
Do I need some additional patches?
Can you try queue branch from kvm.git?
Quoted 7 lines
Thanks for the advice - I used kernel from kvm.git(queue) and was able to run FreeBSD-amd guest with SMP on my kvm-host server without previous problems. Unfortunately, after few hours server was hung up. Case requires further investigations, but generally speaking we went forward. Unfortunately, experiments on the server are difficult, because it is used for everyday work.
Quoted 3 lines
Thanks for testing. This hang indeed looks like completely different problem. What workload the VM was running?
First test:
- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
and kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:
kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! [vhost-3686:3700]
Second test:
- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:
BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables
ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>]
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
ffff880421151740)
Stack:
 ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
 000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
 0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
 [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
 [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
 [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
 [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
 [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
 [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
 [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
 [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
 [<ffffffff81081750>] kthread+0xc0/0xd0
 [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49
03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb
repeated several times with similar content
Signature · 1 line
Artur

Looks like this?
https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145

Quoted 2 lines

--
Gleb.

Gleb Natapov

2013-01-17 06:37 UTC #9 lore raw

Quoted 74 lines · Michael S. Tsirkin

Quoted 74 lines
Quoted 74 lines
First test:
- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
and kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:
kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! [vhost-3686:3700]
Second test:
- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:
BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables
ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>]
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
ffff880421151740)
Stack:
 ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
 000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
 0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
 [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
 [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
 [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
 [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
 [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
 [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
 [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
 [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
 [<ffffffff81081750>] kthread+0xc0/0xd0
 [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49
03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb

Quoted 9 lines

Quoted 4 lines
Quoted 4 lines
repeated several times with similar content
Signature · 1 line
Artur
Looks like this?
https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145

Artur,

Can you please apply this fix on top of kvm.git/queue and try again.

--
Gleb.

Artur Samborski

author

2013-01-17 07:53 UTC #10 lore raw

W dniu 17.01.2013 07:37, Gleb Natapov pisze:

Quoted 81 lines · Michael S. Tsirkin

Quoted 80 lines
Quoted 78 lines
Quoted 78 lines
First test:
- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
and kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:
kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! [vhost-3686:3700]
Second test:
- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:
BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables
ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>]
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
ffff880421151740)
Stack:
  ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
  000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
  0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
  [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
  [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
  [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
  [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
  [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
  [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
  [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
  [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
  [<ffffffff81081750>] kthread+0xc0/0xd0
  [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
  [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
  [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
  [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49
03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb
repeated several times with similar content
--
Artur

Quoted 7 lines

Quoted 3 lines
Looks like this?
https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145
Artur,
Can you please apply this fix on top of kvm.git/queue and try again.

Hi, thanks for the advice. I applied the patch created from commit 76fe45812a3b134c39170ca32dfd4b7217d33145 to branch queue previous evening. Fortunately, it seems that it works now. During the whole night (about 10 hours) I tested two virtual machines (Linux and FreeBSD-amd64) under high load and nothing bad happened.

From the first observation it seems to me that the performance of the VM with FreeBSD-amd64 is much worse than the VM with FreeBSD-i386 -- but, it requires further testing.

Signature · 1 line

Artur