KVM deadlock with CONFIG_PREEMPT host

Hi,

I'm seeing lock-ups of the QEMU process on kvm.git as well as current upstream kernels. This is a backtrace of the hanging VCPU thread:

[<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
[<ffffffff810a286f>] try_stop_cpus+0x40/0x59
[<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
[<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
[<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
[<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
[<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
[<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
[<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
[<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
[<ffffffff81143cbd>] sys_ioctl+0x56/0x79
[<ffffffff81002df2>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

This issue disappears when disabling CONFIG_PREEMPT on the host. According to some rough bisecting, it was imported into kvm.git with merge 146d3bb06b. Given that RCU is involved, I also tried force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no difference as long as PREEMPT is on.

Can anyone confirm this or does someone have an idea what goes wrong? Of course, .config will be provided if required.

Quoted 25 lines · Jan Kiszka

Hi,

I'm seeing lock-ups of the QEMU process on kvm.git as well as current upstream kernels. This is a backtrace of the hanging VCPU thread:

[<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
[<ffffffff810a286f>] try_stop_cpus+0x40/0x59
[<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
[<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
[<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
[<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
[<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
[<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
[<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
[<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
[<ffffffff81143cbd>] sys_ioctl+0x56/0x79
[<ffffffff81002df2>] system_call_fastpath+0x16/0x1b

[<ffffffffffffffff>] 0xffffffffffffffff

This issue disappears when disabling CONFIG_PREEMPT on the host. According to some rough bisecting, it was imported into kvm.git with merge 146d3bb06b. Given that RCU is involved, I also tried force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no difference as long as PREEMPT is on.

Quoted 2 lines

Can anyone confirm this or does someone have an idea what goes wrong? Of course, .config will be provided if required.

This patch should help (,it fixes the problem in my case):
http://article.gmane.org/gmane.linux.kernel/1058018

Quoted 29 lines · Markus Trippelsdorf

Quoted 27 lines · Jan Kiszka

Hi,

I'm seeing lock-ups of the QEMU process on kvm.git as well as current upstream kernels. This is a backtrace of the hanging VCPU thread:

 [<ffffffff810a27bc>] __stop_cpus+0x184/0x1a7
 [<ffffffff810a286f>] try_stop_cpus+0x40/0x59
 [<ffffffff8103a30c>] synchronize_sched_expedited+0x84/0x9d
 [<ffffffff810715a3>] __synchronize_srcu+0x33/0x72
 [<ffffffff810715f7>] synchronize_srcu_expedited+0x15/0x17
 [<ffffffffa06602ae>] __kvm_set_memory_region+0x6a3/0x782 [kvm]
 [<ffffffffa06603c4>] kvm_set_memory_region+0x37/0x50 [kvm]
 [<ffffffffa0661c30>] kvm_vm_ioctl_set_memory_region+0x18/0x1a [kvm]
 [<ffffffffa0661e5f>] kvm_vm_ioctl+0x22d/0x3b1 [kvm]
 [<ffffffff81143c26>] do_vfs_ioctl+0x5a1/0x5e2
 [<ffffffff81143cbd>] sys_ioctl+0x56/0x79
 [<ffffffff81002df2>] system_call_fastpath+0x16/0x1b
 [<ffffffffffffffff>] 0xffffffffffffffff

This issue disappears when disabling CONFIG_PREEMPT on the host. According to some rough bisecting, it was imported into kvm.git with merge 146d3bb06b. Given that RCU is involved, I also tried force-enabling non-preemptible CONFIG_TREE_RCU again, but that made no difference as long as PREEMPT is on.

Can anyone confirm this or does someone have an idea what goes wrong? Of course, .config will be provided if required.

Quoted 2 lines

This patch should help (,it fixes the problem in my case):
http://article.gmane.org/gmane.linux.kernel/1058018

Yeah, that works here as well.

Would be a nice-to-have in kvm.git until it's pull via the next upstream merge. At least for me this bug triggers at every VM boot.