RISC-V: KVM: Fix SRCU deadlock caused by kvm_riscv_check_vcpu_requests()
Subsystems (3)
PATCH 1/1 RISC-V: KVM: Fix SRCU deadlock caused by kvm_riscv_check_vcpu_requests()
The kvm_riscv_check_vcpu_requests() is called with SRCU read lock held and for KVM_REQ_SLEEP request it will block the VCPU without releasing SRCU read lock. This causes KVM ioctls (such as KVM_IOEVENTFD) from other VCPUs of the same Guest/VM to hang/deadlock if there is any synchronize_srcu() or synchronize_srcu_expedited() in the path.
To fix the above in kvm_riscv_check_vcpu_requests(), we should do SRCU read unlock before blocking the VCPU and do SRCU read lock after VCPU wakeup.
requests handling")
--- arch/riscv/kvm/vcpu.c | 2 ++ 1 file changed, 2 insertions(+)
arch/riscv/kvm/vcpu.c +2 -0
@@ -845,9 +845,11 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_request_pending (vcpu)) {if (kvm_check_request (KVM_REQ_SLEEP , vcpu)) { +kvm_vcpu_srcu_read_unlock (vcpu);rcuwait_wait_event (wait, (!vcpu->arch.power_off) && (!vcpu->arch.pause),TASK_INTERRUPTIBLE ); +kvm_vcpu_srcu_read_lock (vcpu);if (vcpu->arch.power_off || vcpu->arch.pause) {/*
Signature · 1 line
2.34.1
Quoted 36 lines · Anup Patel
The
kvm_riscv_check_vcpu_requests()is called with SRCU read lock held and forKVM_REQ_SLEEPrequest it will block the VCPU without releasing SRCU read lock. This causes KVM ioctls (such asKVM_IOEVENTFD) from other VCPUs of the same Guest/VM to hang/deadlock if there is anysynchronize_srcu()orsynchronize_srcu_expedited()in the path.To fix the above in
kvm_riscv_check_vcpu_requests(), we should do SRCU read unlock before blocking the VCPU and do SRCU read lock after VCPU wakeup.Fixes cce69aff689e ("RISC-V: KVM: Implement VCPU interrupts andrequests handling")
Reported-by Bin Meng <[email protected]>Signed-off-by Anup Patel <[email protected]>--- arch/riscv/kvm/vcpu.c | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/kvm/vcpu.c +2
-1@@ -845,9 +845,11 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_request_pending (vcpu)) {if (kvm_check_request (KVM_REQ_SLEEP , vcpu)) { +kvm_vcpu_srcu_read_unlock (vcpu);rcuwait_wait_event (wait, (!vcpu->arch.power_off) && (!vcpu->arch.pause),TASK_INTERRUPTIBLE ); +kvm_vcpu_srcu_read_lock (vcpu);if (vcpu->arch.power_off || vcpu->arch.pause) {/* --
Quoted 2 lines
2.34.1
Signature · 2 lines
Regards, Atish
Quoted 6 lines · Anup Patel
The
kvm_riscv_check_vcpu_requests()is called with SRCU read lock held and forKVM_REQ_SLEEPrequest it will block the VCPU without releasing SRCU read lock. This causes KVM ioctls (such asKVM_IOEVENTFD) from other VCPUs of the same Guest/VM to hang/deadlock if there is anysynchronize_srcu()orsynchronize_srcu_expedited()in the path.
Quoted 7 lines
To fix the above in
kvm_riscv_check_vcpu_requests(), we should do SRCU read unlock before blocking the VCPU and do SRCU read lock after VCPU wakeup.Fixes cce69aff689e ("RISC-V: KVM: Implement VCPU interrupts andrequests handling")
Reported-by Bin Meng <[email protected]>
Thanks Anup for resolving the problem originally reported in
https://lore.kernel.org/all/5df27902-9009-afb9-68d3-186fdb4e4067 at canonical.com/
Thanks to Bin for his analysis.
Quoted 1 line
Signed-off-by Anup Patel <[email protected]>
With this patch applied to Linux v5.19-rc5 I am able to run U-Boot qemu-riscv64_smode_defconfig on QEMU 7.0 with
qemu-system-riscv64 \
-M virt -accel kvm -m 2G -smp 2 \
-nographic \
-kernel u-boot \
-drive file=kinetic-server-cloudimg-riscv64.raw,format=raw,if=virtio \
-device virtio-net-device,netdev=eth0 \
-netdev user,id=eth0,hostfwd=tcp::8022-:22
and load files from the virtio drive.
Without the patch virtio access blocks:
[ +0.102462] INFO: task qemu-system-ris:1254 blocked for more than 120 seconds.
[ +0.004034] Not tainted 5.19.0-rc5 #4
[ +0.001145] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.002189] task:qemu-system-ris state:D stack: 0 pid: 1254 ppid:
1068 flags:0x00000000
[ +0.001546] Call Trace: [ +0.000389] [<ffffffff806b1340>] schedule+0x42/0xaa [ +0.008026] [<ffffffff806b6164>] schedule_timeout+0xa0/0xd4 [ +0.000086] [<ffffffff806b1c0a>] __wait_for_common+0x9a/0x19a [ +0.000057] [<ffffffff806b1d24>] wait_for_completion+0x1a/0x22 [ +0.000053] [<ffffffff80063a88>] __synchronize_srcu.part.0+0x78/0xce [ +0.000049] [<ffffffff80063b00>] synchronize_srcu_expedited+0x22/0x2c [ +0.000474] [<ffffffff01417560>] kvm_swap_active_memslots+0x12e/0x170 [kvm] [ +0.000864] [<ffffffff01419ad2>] kvm_set_memslot+0x1e8/0x388 [kvm] [ +0.000267] [<ffffffff01419da6>] __kvm_set_memory_region+0x134/0x2f8 [kvm] [ +0.000439] [<ffffffff0141d412>] kvm_vm_ioctl+0x1fc/0xba0 [kvm] [ +0.000232] [<ffffffff80176af0>] sys_ioctl+0x80/0x96 [ +0.000129] [<ffffffff800032d2>] ret_from_syscall+0x0/0x2
Quoted 20 lines
--- arch/riscv/kvm/vcpu.c | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/kvm/vcpu.c +2
-0@@ -845,9 +845,11 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_request_pending (vcpu)) {if (kvm_check_request (KVM_REQ_SLEEP , vcpu)) { +kvm_vcpu_srcu_read_unlock (vcpu);rcuwait_wait_event (wait, (!vcpu->arch.power_off) && (!vcpu->arch.pause),TASK_INTERRUPTIBLE ); +kvm_vcpu_srcu_read_lock (vcpu);if (vcpu->arch.power_off || vcpu->arch.pause) {/*
Quoted 7 lines · Anup Patel
The
kvm_riscv_check_vcpu_requests()is called with SRCU read lock held and forKVM_REQ_SLEEPrequest it will block the VCPU without releasing SRCU read lock. This causes KVM ioctls (such asKVM_IOEVENTFD) from other VCPUs of the same Guest/VM to hang/deadlock if there is anysynchronize_srcu()orsynchronize_srcu_expedited()in the path.
Quoted 6 lines
To fix the above in
kvm_riscv_check_vcpu_requests(), we should do SRCU read unlock before blocking the VCPU and do SRCU read lock after VCPU wakeup.Fixes cce69aff689e ("RISC-V: KVM: Implement VCPU interrupts andrequests handling")
nites: the "Fixes" tag should be put in a single line to avoid breaking scripts that parse the "Fixes" tag
Quoted 13 lines
Reported-by Bin Meng <[email protected]>Signed-off-by Anup Patel <[email protected]>--- arch/riscv/kvm/vcpu.c | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/kvm/vcpu.c +0
-0@@ -845,9 +845,11 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_request_pending (vcpu)) {
arch/riscv/kvm/vcpu.c static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)
Quoted 10 lines
arch/riscv/kvm/vcpu.c +2
-1if (kvm_check_request (KVM_REQ_SLEEP , vcpu)) { +kvm_vcpu_srcu_read_unlock (vcpu);rcuwait_wait_event (wait, (!vcpu->arch.power_off) && (!vcpu->arch.pause),TASK_INTERRUPTIBLE ); +kvm_vcpu_srcu_read_lock (vcpu);if (vcpu->arch.power_off || vcpu->arch.pause) {/* --
Quoted 7 lines · Anup Patel
The
kvm_riscv_check_vcpu_requests()is called with SRCU read lock held and forKVM_REQ_SLEEPrequest it will block the VCPU without releasing SRCU read lock. This causes KVM ioctls (such asKVM_IOEVENTFD) from other VCPUs of the same Guest/VM to hang/deadlock if there is anysynchronize_srcu()orsynchronize_srcu_expedited()in the path.
Quoted 8 lines
To fix the above in
kvm_riscv_check_vcpu_requests(), we should do SRCU read unlock before blocking the VCPU and do SRCU read lock after VCPU wakeup.Fixes cce69aff689e ("RISC-V: KVM: Implement VCPU interrupts andrequests handling")
Reported-by Bin Meng <[email protected]>Signed-off-by Anup Patel <[email protected]>
Thanks everyone for providing Tested-by and Reviewed-by.
I have queued this patch for 5.19-rcX fixes.
Regards,
Anup
Quoted 23 lines
--- arch/riscv/kvm/vcpu.c | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/kvm/vcpu.c +2
-1@@ -845,9 +845,11 @@ static void kvm_riscv_check_vcpu_requests(struct kvm_vcpu *vcpu)if (kvm_request_pending (vcpu)) {if (kvm_check_request (KVM_REQ_SLEEP , vcpu)) { +kvm_vcpu_srcu_read_unlock (vcpu);rcuwait_wait_event (wait, (!vcpu->arch.power_off) && (!vcpu->arch.pause),TASK_INTERRUPTIBLE ); +kvm_vcpu_srcu_read_lock (vcpu);if (vcpu->arch.power_off || vcpu->arch.pause) {/* -- 2.34.1