Two cpus are not brought up normally in SLES11 sp3 VM after reboot

BUG/RFC zhanghailiang 3 participants 8 messages Jul 6, 2015 – Jul 7, 2015 KVM

lore

Share & tools

lore

zhanghailiang

author

2015-07-06 07:54 UTC #0 lore raw

Hi,

Recently we encountered a problem in our project: 2 CPUs in VM are not brought up normally after reboot.

Our host is using KVM kmod 3.6 and QEMU 2.1.
A SLES 11 sp3 VM configured with 8 vcpus,
cpu model is configured with 'host-passthrough'.

After VM's first time started up, everything seems to be OK.
and then VM is paniced and rebooted.
After reboot, only 6 cpus are brought up in VM, cpu1 and cpu7 are not online.

This is the only message we can get from VM:
VM dmesg shows:

[    0.069867] Booting Node   0, Processors  #1
[    5.060042] CPU1: Stuck ??
[    5.060499]  #2
[    5.088322] kvm-clock: cpu 2, msr 6:3fc90901, secondary cpu clock
[    5.088335] KVM setup async PF for cpu 2
[    5.092967] NMI watchdog enabled, takes one hw-pmu counter.
[    5.094405]  #3
[    5.108324] kvm-clock: cpu 3, msr 6:3fcd0901, secondary cpu clock
[    5.108333] KVM setup async PF for cpu 3
[    5.113553] NMI watchdog enabled, takes one hw-pmu counter.
[    5.114970]  #4
[    5.128325] kvm-clock: cpu 4, msr 6:3fd10901, secondary cpu clock
[    5.128336] KVM setup async PF for cpu 4
[    5.134576] NMI watchdog enabled, takes one hw-pmu counter.
[    5.135998]  #5
[    5.152324] kvm-clock: cpu 5, msr 6:3fd50901, secondary cpu clock
[    5.152334] KVM setup async PF for cpu 5
[    5.154764] NMI watchdog enabled, takes one hw-pmu counter.
[    5.156467]  #6
[    5.172327] kvm-clock: cpu 6, msr 6:3fd90901, secondary cpu clock
[    5.172341] KVM setup async PF for cpu 6
[    5.180738] NMI watchdog enabled, takes one hw-pmu counter.
[    5.182173]  #7 Ok.
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs
[   10.172394] Total of 6 processors activated (28799.97 BogoMIPS).

From host, we found that QEMU vcpu1 thread and vcpu7 thread were not consuming any cpu (Should be in idle state), All of VCPUs' stacks in host is like bellow:

[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b

[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff

We looked into the kernel codes that could leading to the above 'Stuck' warning,
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong. But since we can’t reproduce this problem, we are not quite sure.
Is there any possible that the cupid emulation in kvm/qemu has some bug ?

Has anyone come across these problem before? Or any idea?

Thanks,
zhanghailiang

Paolo Bonzini

maintainer

2015-07-06 08:45 UTC #1 lore raw

Quoted 15 lines · zhanghailiang

From host, we found that QEMU vcpu1 thread and vcpu7 thread were not consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff

Quoted 6 lines

We looked into the kernel codes that could leading to the above 'Stuck' warning,
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?

Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?

Paolo

zhanghailiang

author

2015-07-06 09:59 UTC #2 lore raw

Quoted 25 lines · Paolo Bonzini

Quoted 15 lines · zhanghailiang
From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff
Quoted 6 lines
We looked into the kernel codes that could leading to the above 'Stuck' warning,
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?

Quoted 2 lines

Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?

OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'.

If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). The follow is the starup process of BSP and AP.
BSP:
start_kernel()
->smp_init()

     ->smp_boot_cpus()
       ->do_boot_cpu()
           ->start_ip = trampoline_address(); //set the address that AP will go to execute
           ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
           ->for (timeout = 0; timeout < 50000; timeout++)
               if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not

APs:
ENTRY(trampoline_data) (trampoline_64.S)

      ->ENTRY(secondary_startup_64) (head_64.S)
         ->start_secondary() (smpboot.c)
            ->cpu_init();
            ->smp_callin();
                ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message.

From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. It is located in trampoline_data():

ENTRY(trampoline_data)
...

call verify_cpu # Verify the cpu supports long mode
testl %eax, %eax # Check for return code
jnz no_longmode

...

no_longmode:
hlt
jmp no_longmode

For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu.

From the message in VM, we know vcpu1 and vcpu7 is something wrong.

[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs

Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584

Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in host.

Thanks,
zhanghailiang

Paolo Bonzini

maintainer

2015-07-06 11:03 UTC #3 lore raw

Quoted 14 lines · zhanghailiang

Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570 CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573 CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575 CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576 CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577 CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578 CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583 CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584

Quoted 3 lines

Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in
host.

Can you capture a trace on the host (trace-cmd record -e kvm) and send it privately? Please note which CPUs get stuck, since I guess it's not always 1 and 7.

Paolo

Igor Mammedov

2015-07-07 11:23 UTC #4 lore raw

In reply to zhanghailiang

Quoted 19 lines · On Mon, 6 Jul 2015 17:59:10 +0800 zhanghailiang…

Quoted 18 lines · Paolo Bonzini
Quoted 15 lines · zhanghailiang
From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
We looked into the kernel codes that could leading to the above 'Stuck' warning,

in current upstream there isn't any printk(...Stuck...) left since that code path has been reworked.
I've often seen this on over-committed host during guest CPUs up/down torture test. Could you update guest kernel to upstream and see if issue reproduces?

Quoted 84 lines

Quoted 7 lines
Quoted 4 lines
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?
Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?
OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'.
If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). The follow is the starup process of BSP and AP.
BSP:
start_kernel()
->smp_init()
      ->smp_boot_cpus()
        ->do_boot_cpu()
            ->start_ip = trampoline_address(); //set the address that AP will go to execute
            ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
            ->for (timeout = 0; timeout < 50000; timeout++)
                if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not
APs:
ENTRY(trampoline_data) (trampoline_64.S)
       ->ENTRY(secondary_startup_64) (head_64.S)
          ->start_secondary() (smpboot.c)
             ->cpu_init();
             ->smp_callin();
                 ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message.
From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in
smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. It is located in trampoline_data():
ENTRY(trampoline_data)
...
call verify_cpu # Verify the cpu supports long mode
testl %eax, %eax # Check for return code
jnz no_longmode
...
no_longmode:
hlt
jmp no_longmode
For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu.
From the message in VM, we know vcpu1 and vcpu7 is something wrong.
[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs
Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584
Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in host.
Thanks,
zhanghailiang
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

zhanghailiang

author

2015-07-07 11:44 UTC #5 lore raw

Quoted 22 lines · Igor Mammedov

Quoted 19 lines · On Mon, 6 Jul 2015 17:59:10 +0800 zhanghailiang…
Quoted 18 lines · Paolo Bonzini
Quoted 15 lines · zhanghailiang
From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff

Quoted 7 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We looked into the kernel codes that could leading to the above 'Stuck' warning,
in current upstream there isn't any printk(...Stuck...) left since that code path has been reworked.
I've often seen this on over-committed host during guest CPUs up/down torture test. Could you update guest kernel to upstream and see if issue reproduces?

Hmm, Unfortunately, it is very hard to reproduce, and we are still trying to reproduce it.

For your test case, is it a kernel bug?
Or is there any related patch could solve your test problem been merged into upstream ?

Thanks,
zhanghailiang

Quoted 88 lines

Quoted 80 lines
Quoted 7 lines
Quoted 4 lines
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?
Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?
OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'.
If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). The follow is the starup process of BSP and AP.
BSP:
start_kernel()
    ->smp_init()
       ->smp_boot_cpus()
         ->do_boot_cpu()
             ->start_ip = trampoline_address(); //set the address that AP will go to execute
             ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
             ->for (timeout = 0; timeout < 50000; timeout++)
                 if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not
APs:
ENTRY(trampoline_data) (trampoline_64.S)
        ->ENTRY(secondary_startup_64) (head_64.S)
           ->start_secondary() (smpboot.c)
              ->cpu_init();
              ->smp_callin();
                  ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message.
From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in
smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. It is located in trampoline_data():
ENTRY(trampoline_data)
...
call verify_cpu # Verify the cpu supports long mode
testl %eax, %eax # Check for return code
jnz no_longmode
...
no_longmode:
hlt
jmp no_longmode
For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu.
From the message in VM, we know vcpu1 and vcpu7 is something wrong.
[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs
Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
    CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
    CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
    CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
    CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
    CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
    CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
    CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584
Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in host.
Thanks,
zhanghailiang
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.

Igor Mammedov

2015-07-07 12:21 UTC #6 lore raw

Quoted 31 lines · On Tue, 7 Jul 2015 19:43:35 +0800 zhanghailiang…

Quoted 22 lines · Igor Mammedov
Quoted 19 lines · On Mon, 6 Jul 2015 17:59:10 +0800 zhanghailiang…
Quoted 18 lines · Paolo Bonzini
Quoted 15 lines · zhanghailiang
From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We looked into the kernel codes that could leading to the above 'Stuck' warning,
in current upstream there isn't any printk(...Stuck...) left since that code path has been reworked.
I've often seen this on over-committed host during guest CPUs up/down torture test. Could you update guest kernel to upstream and see if issue reproduces?

Quoted 5 lines

Hmm, Unfortunately, it is very hard to reproduce, and we are still trying to reproduce it.
For your test case, is it a kernel bug?
Or is there any related patch could solve your test problem been merged into upstream ?

I don't remember all prerequisite patches but you should be able to find
http://marc.info/?l=linux-kernel&m=140326703108009&w=2
"x86/smpboot: Initialize secondary CPU only if master CPU will wait for it"
and then look for dependencies.

Quoted 95 lines

Thanks,
zhanghailiang
Quoted 88 lines
Quoted 80 lines
Quoted 7 lines
Quoted 4 lines
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?
Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?
OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'.
If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). The follow is the starup process of BSP and AP.
BSP:
start_kernel()
    ->smp_init()
       ->smp_boot_cpus()
         ->do_boot_cpu()
             ->start_ip = trampoline_address(); //set the address that AP will go to execute
             ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
             ->for (timeout = 0; timeout < 50000; timeout++)
                 if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not
APs:
ENTRY(trampoline_data) (trampoline_64.S)
        ->ENTRY(secondary_startup_64) (head_64.S)
           ->start_secondary() (smpboot.c)
              ->cpu_init();
              ->smp_callin();
                  ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message.
From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in
smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. It is located in trampoline_data():
ENTRY(trampoline_data)
...
call verify_cpu # Verify the cpu supports long mode
testl %eax, %eax # Check for return code
jnz no_longmode
...
no_longmode:
hlt
jmp no_longmode
For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu.
From the message in VM, we know vcpu1 and vcpu7 is something wrong.
[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs
Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
    CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
    CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
    CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
    CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
    CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
    CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
    CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584
Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in host.
Thanks,
zhanghailiang
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.

zhanghailiang

author

2015-07-07 12:40 UTC #7 lore raw

Quoted 36 lines · Igor Mammedov

Quoted 33 lines · On Tue, 7 Jul 2015 19:43:35 +0800 zhanghailiang…
Quoted 22 lines · Igor Mammedov
Quoted 19 lines · On Mon, 6 Jul 2015 17:59:10 +0800 zhanghailiang…
Quoted 18 lines · Paolo Bonzini
Quoted 15 lines · zhanghailiang
From host, we found that QEMU vcpu1 thread and vcpu7 thread were not
consuming any cpu (Should be in idle state),
All of VCPUs' stacks in host is like bellow:
[<ffffffffa07089b5>] kvm_vcpu_block+0x65/0xa0 [kvm]
[<ffffffffa071c7c1>] __vcpu_run+0xd1/0x260 [kvm]
[<ffffffffa071d508>] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 [kvm]
[<ffffffffa0709cee>] kvm_vcpu_ioctl+0x38e/0x580 [kvm]
[<ffffffff8116be8b>] do_vfs_ioctl+0x8b/0x3b0
[<ffffffff8116c251>] sys_ioctl+0xa1/0xb0
[<ffffffff81468092>] system_call_fastpath+0x16/0x1b
[<00002ab9fe1f99a7>] 0x2ab9fe1f99a7
[<ffffffffffffffff>] 0xffffffffffffffff
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
We looked into the kernel codes that could leading to the above 'Stuck' warning,
in current upstream there isn't any printk(...Stuck...) left since that code path has been reworked.
I've often seen this on over-committed host during guest CPUs up/down torture test. Could you update guest kernel to upstream and see if issue reproduces?
Hmm, Unfortunately, it is very hard to reproduce, and we are still trying to reproduce it.

Quoted 8 lines

Quoted 3 lines
For your test case, is it a kernel bug?
Or is there any related patch could solve your test problem been merged into upstream ?
I don't remember all prerequisite patches but you should be able to find
http://marc.info/?l=linux-kernel&m=140326703108009&w=2
"x86/smpboot: Initialize secondary CPU only if master CPU will wait for it"
and then look for dependencies.

Er, we have investigated this patch, and it is not related to our problem, :)

Thanks.

Quoted 100 lines

Quoted 96 lines
Thanks,
zhanghailiang
Quoted 88 lines
Quoted 80 lines
Quoted 7 lines
Quoted 4 lines
and found that the only possible is the emulation of 'cpuid' instruct in kvm/qemu has something wrong.
But since we can’t reproduce this problem, we are not quite sure. Is there any possible that the cupid emulation in kvm/qemu has some bug ?
Can you explain the relationship to the cpuid emulation? What do the traces say about vcpus 1 and 7?
OK, we searched the VM's kernel codes with the 'Stuck' message, and it is located in
do_boot_cpu(). It's in BSP context, the call process is:
BSP executes start_kernel() -> smp_init() -> smp_boot_cpus() -> do_boot_cpu() -> wakeup_secondary_via_INIT() to trigger APs. It will wait 5s for APs to startup, if some AP not startup normally, it will print 'CPU%d Stuck' or 'CPU%d: Not responding'.
If it prints 'Stuck', it means the AP has received the SIPI interrupt and begins to execute the code 'ENTRY(trampoline_data)' (trampoline_64.S) , but be stuck in some places before smp_callin()(smpboot.c). The follow is the starup process of BSP and AP.
BSP:
start_kernel()
     ->smp_init()
        ->smp_boot_cpus()
          ->do_boot_cpu()
              ->start_ip = trampoline_address(); //set the address that AP will go to execute
              ->wakeup_secondary_cpu_via_init(); // kick the secondary CPU
              ->for (timeout = 0; timeout < 50000; timeout++)
                  if (cpumask_test_cpu(cpu, cpu_callin_mask)) break;// check if AP startup or not
APs:
ENTRY(trampoline_data) (trampoline_64.S)
         ->ENTRY(secondary_startup_64) (head_64.S)
            ->start_secondary() (smpboot.c)
               ->cpu_init();
               ->smp_callin();
                   ->cpumask_set_cpu(cpuid, cpu_callin_mask); ->Note: if AP comes here, the BSP will not prints the error message.
From above call process, we can be sure that, the AP has been stuck between trampoline_data and the cpumask_set_cpu() in
smp_callin(), we look through these codes path carefully, and only found a 'hlt' instruct that could block the process. It is located in trampoline_data():
ENTRY(trampoline_data)
...
call verify_cpu # Verify the cpu supports long mode
testl %eax, %eax # Check for return code
jnz no_longmode
...
no_longmode:
hlt
jmp no_longmode
For the process verify_cpu(),
we can only find the 'cpuid' sensitive instruct that could lead VM exit from No-root mode. This is why we doubt if cpuid emulation is wrong in KVM/QEMU that leading to the fail in verify_cpu.
From the message in VM, we know vcpu1 and vcpu7 is something wrong.
[    5.060042] CPU1: Stuck ??
[   10.170815] CPU7: Stuck ??
[   10.171648] Brought up 6 CPUs
Besides, the follow is the cpus message got from host. 80FF72F5-FF6D-E411-A8C8-000000821800:/home/fsp/hrg # virsh qemu-monitor-command instance-0000000
* CPU #0: pc=0x00007f64160c683d thread_id=68570
     CPU #1: pc=0xffffffff810301f1 (halted) thread_id=68573
     CPU #2: pc=0xffffffff810301e2 (halted) thread_id=68575
     CPU #3: pc=0xffffffff810301e2 (halted) thread_id=68576
     CPU #4: pc=0xffffffff810301e2 (halted) thread_id=68577
     CPU #5: pc=0xffffffff810301e2 (halted) thread_id=68578
     CPU #6: pc=0xffffffff810301e2 (halted) thread_id=68583
     CPU #7: pc=0xffffffff810301f1 (halted) thread_id=68584
Oh, i also forgot to mention in the above message that, we have bond each vCPU to different physical CPU in host.
Thanks,
zhanghailiang
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.
.