Skip to contentSkip to conversation

arm64: KVM: Initialize vGIC before preempt-disabled section in kvm_reset_vcpu()

PATCH In review Deepanshu Kartikey 2 participants 4 messages Apr 12 – Apr 21

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm64: KVM: Initialize vGIC before preempt-disabled section in kvm_reset_vcpu()

kvm_reset_vcpu() calls kvm_timer_vcpu_reset() inside a preempt-disabled section to avoid races with preempt notifiers that also call vcpu put/load.

However, kvm_timer_vcpu_reset() eventually calls kvm_vgic_inject_irq() which triggers vgic_lazy_init() if the vGIC has not been initialized yet. vgic_lazy_init() acquires a mutex and calls vgic_init() which invokes synchronize_srcu_expedited() -- both of which may sleep. Sleeping inside a preempt-disabled section is illegal and causes:

  BUG: scheduling while atomic: syz.1.49/3699/0x00000002

Fix this by calling vgic_lazy_init() before preempt_disable(). On the second call inside kvm_vgic_inject_irq(), vgic_initialized() will return true and vgic_lazy_init() will return immediately without sleeping.

Fixes e761a927bc9a ("KVM: arm/arm64: Reset the VCPU without preemption and vcpu state loaded")
Reported-by [email protected]
---
 arch/arm64/kvm/reset.c | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/arm64/kvm/reset.c +9 -0
index b963fd975aac..4ee16b4a37b5 100644
@@ -25,6 +25,7 @@
 #include <asm/ptrace.h>
 #include <asm/kvm_arm.h>
 #include <asm/kvm_asm.h>
+#include "vgic/vgic.h"
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_nested.h>
@@ -198,6 +199,14 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	vcpu->arch.reset_state.reset = false;
 	spin_unlock(&vcpu->arch.mp_state_lock);
 
+
+	/*
+	 * Initialize vGIC before entering preempt-disabled section.
+	 * vgic_lazy_init() may sleep via mutex_lock, which is illegal
+	 * inside preempt_disable(). Second call inside kvm_vgic_inject_irq
+	 * will find vGIC already initialized and return immediately.
+	 */
+	vgic_lazy_init(vcpu->kvm);
 	preempt_disable();
 	loaded = (vcpu->cpu != -1);
 	if (loaded)
Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 10 lines · Deepanshu Kartikey

kvm_reset_vcpu() calls kvm_timer_vcpu_reset() inside a preempt-disabled section to avoid races with preempt notifiers that also call vcpu put/load.

However, kvm_timer_vcpu_reset() eventually calls kvm_vgic_inject_irq() which triggers vgic_lazy_init() if the vGIC has not been initialized yet. vgic_lazy_init() acquires a mutex and calls vgic_init() which invokes synchronize_srcu_expedited() -- both of which may sleep. Sleeping inside a preempt-disabled section is illegal and causes:

Quoted 6 lines
  BUG: scheduling while atomic: syz.1.49/3699/0x00000002

Fix this by calling vgic_lazy_init() before preempt_disable(). On the second call inside kvm_vgic_inject_irq(), vgic_initialized() will return true and vgic_lazy_init() will return immediately without sleeping.

I think this really goes in the wrong direction. Forcing the vgic (a global resource) to initialise when the vcpu's timer (a local resource) is reset feels at best bizarre. Now you are promoting it to be forced at vcpu reset. This makes things worse.

You probably want to take a step back and look at *why* we end-up here. The core reason seems to be that the timer emulation caches the level in a per-timer structure, and tries hard not call into the vgic unless the level changes. Which means that unless the vgic is initialised and is able to latch that state, the initial pending state will not be propagated to the guest.

But do we need this optimisation? I don't think so. Other emulated devices don't require it. We can let the vgic know the state of the timer at every vcpu entry, just like we do for other virtual interrupts that the kernel injects (PMU, vgic MI).

Once you remove the this cache and the need for the vgic to buffer things outside of normal execution, you can also drop the magic init from the interrupt injection path, because the injection will happen on the run path, just like any other PPI.

That'd be a much better approach IMO.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#2 lore raw
Quoted 42 lines · Marc Zyngier
Quoted 10 lines · Deepanshu Kartikey

kvm_reset_vcpu() calls kvm_timer_vcpu_reset() inside a preempt-disabled section to avoid races with preempt notifiers that also call vcpu put/load.

However, kvm_timer_vcpu_reset() eventually calls kvm_vgic_inject_irq() which triggers vgic_lazy_init() if the vGIC has not been initialized yet. vgic_lazy_init() acquires a mutex and calls vgic_init() which invokes synchronize_srcu_expedited() -- both of which may sleep. Sleeping inside a preempt-disabled section is illegal and causes:

Quoted 6 lines
  BUG: scheduling while atomic: syz.1.49/3699/0x00000002

Fix this by calling vgic_lazy_init() before preempt_disable(). On the second call inside kvm_vgic_inject_irq(), vgic_initialized() will return true and vgic_lazy_init() will return immediately without sleeping.

I think this really goes in the wrong direction. Forcing the vgic (a global resource) to initialise when the vcpu's timer (a local resource) is reset feels at best bizarre. Now you are promoting it to be forced at vcpu reset. This makes things worse.

You probably want to take a step back and look at *why* we end-up here. The core reason seems to be that the timer emulation caches the level in a per-timer structure, and tries hard not call into the vgic unless the level changes. Which means that unless the vgic is initialised and is able to latch that state, the initial pending state will not be propagated to the guest.

But do we need this optimisation? I don't think so. Other emulated devices don't require it. We can let the vgic know the state of the timer at every vcpu entry, just like we do for other virtual interrupts that the kernel injects (PMU, vgic MI).

Once you remove the this cache and the need for the vgic to buffer things outside of normal execution, you can also drop the magic init from the interrupt injection path, because the injection will happen on the run path, just like any other PPI.

Quoted 8 lines

That'd be a much better approach IMO.

Thanks,

        M.

--
Without deviation from the norm, progress is not possible.

Hi Marc,

Thank you for the detailed feedback! I apologize for the delayed response — I was away on holiday.

I understand your point. My fix addresses the symptom rather than the root cause. Forcing vGIC (a global resource) to initialize during timer (a local resource) reset is not the right approach.

I will take your suggestion and work on:

I will send a v2 once I have something ready.

Thanks again for the guidance!

Best regards,
Deeanshu Kartikey

maintainer
#3 lore raw
Quoted 50 lines · Deepanshu Kartikey
Quoted 48 lines · Marc Zyngier
Quoted 10 lines · Deepanshu Kartikey

kvm_reset_vcpu() calls kvm_timer_vcpu_reset() inside a preempt-disabled section to avoid races with preempt notifiers that also call vcpu put/load.

However, kvm_timer_vcpu_reset() eventually calls kvm_vgic_inject_irq() which triggers vgic_lazy_init() if the vGIC has not been initialized yet. vgic_lazy_init() acquires a mutex and calls vgic_init() which invokes synchronize_srcu_expedited() -- both of which may sleep. Sleeping inside a preempt-disabled section is illegal and causes:

Quoted 6 lines
  BUG: scheduling while atomic: syz.1.49/3699/0x00000002

Fix this by calling vgic_lazy_init() before preempt_disable(). On the second call inside kvm_vgic_inject_irq(), vgic_initialized() will return true and vgic_lazy_init() will return immediately without sleeping.

I think this really goes in the wrong direction. Forcing the vgic (a global resource) to initialise when the vcpu's timer (a local resource) is reset feels at best bizarre. Now you are promoting it to be forced at vcpu reset. This makes things worse.

You probably want to take a step back and look at *why* we end-up here. The core reason seems to be that the timer emulation caches the level in a per-timer structure, and tries hard not call into the vgic unless the level changes. Which means that unless the vgic is initialised and is able to latch that state, the initial pending state will not be propagated to the guest.

But do we need this optimisation? I don't think so. Other emulated devices don't require it. We can let the vgic know the state of the timer at every vcpu entry, just like we do for other virtual interrupts that the kernel injects (PMU, vgic MI).

Once you remove the this cache and the need for the vgic to buffer things outside of normal execution, you can also drop the magic init from the interrupt injection path, because the injection will happen on the run path, just like any other PPI.

That'd be a much better approach IMO.

Thanks,

        M.
Quoted 8 lines
Quoted 2 lines

--
Without deviation from the norm, progress is not possible.

Hi Marc,

Thank you for the detailed feedback! I apologize for the delayed response — I was away on holiday.

No need to apologise. I hadn't spotted this report (syzkaller badly triaged it), so thanks for bring it to my attention.

Quoted 8 lines

I understand your point. My fix addresses the symptom rather than the root cause. Forcing vGIC (a global resource) to initialize during timer (a local resource) reset is not the right approach.

I will take your suggestion and work on:

I will send a v2 once I have something ready.

I've posted my own take on this at [1], for which you are on Cc. Feel free to review it if you have the time (I'll post a v2 anyway, as I have accumulated a couple of additional fixes).

Cheers,

	M.

[1] https://lore.kernel.org/r/[email protected]

Signature · 1 line
Without deviation from the norm, progress is not possible.