accel/kvm: Fix BQL lock imbalance in kvm_cpu_exec
PATCH 1/1 accel/kvm: Fix BQL lock imbalance in kvm_cpu_exec
When kvm_cpu_exec() returns EXCP_HLT due to kvm_arch_process_async_events() returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back to kvm_cpu_exec() while still holding the BQL, leading to deadlocks.
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call kvm_cpu_exec() which returns EXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.
The fix has two parts:
1. In kvm_cpu_exec() (kvm-all.c):
Release the BQL before returning EXCP_HLT in the early return path, matching the behavior of the normal execution path where bql_unlock() is called before entering the main KVM execution loop.
2. In kvm_vcpu_thread_fn() (kvm-accel-ops.c):
Re-acquire the BQL after kvm_cpu_exec() returns EXCP_HLT, since the loop expects to hold the BQL when calling kvm_cpu_exec() again.
This ensures proper BQL lock/unlock pairing:
- kvm_vcpu_thread_fn() holds BQL before calling kvm_cpu_exec()
- kvm_cpu_exec() releases BQL before returning (for EXCP_HLT)
- kvm_vcpu_thread_fn() re-acquires BQL if EXCP_HLT was returned
- Next iteration has BQL held as expected
This is a regression introduced by commit 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change") which refactored kvm_irqchip_create() and changed the initialization timing, exposing this lock imbalance issue.
--- accel/kvm/kvm-accel-ops.c | 4 ++++ accel/kvm/kvm-all.c | 1 + 2 files changed, 5 insertions(+)
accel/kvm/kvm-accel-ops.c +4 -0
@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }
accel/kvm/kvm-all.c +1 -0
@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }
Signature · 1 line
2.52.0
Quoted 8 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.
I am not sure I understand this. Seems kvm_cpu_exec() does expect that the caller holds bql before calling the function. Where is the lock imbalance?
Quoted 23 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expected
Quoted 6 lines
This is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed.
Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Quoted 38 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Hi Ani,
Quoted 10 lines · Ani Sinha
Quoted 8 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?
The issue is not that kvm_cpu_exec() doesn't expect the caller to hold the BQL - it does. The problem is that kvm_cpu_exec() has inconsistent BQL handling across its return paths.
Normal execution path:
int kvm_cpu_exec(CPUState *cpu)
{
// BQL held on entry (from caller)
if (kvm_arch_process_async_events(cpu)) {
return EXCP_HLT; // ← Returns with BQL STILL HELD
}
bql_unlock(); // ← Normal path unlocks here
// ... KVM execution loop ...
bql_lock(); // ← Re-acquires before returning
return ret;
}The lock imbalance:
When kvm_arch_process_async_events() returns true, the function returns EXCP_HLT before the bql_unlock() call. This means the early return path keeps the BQL held, while the normal execution path releases and re-acquires it. The caller (kvm_vcpu_thread_fn()) loops back and calls kvm_cpu_exec() again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations when EXCP_HLT is returned.
Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly call kvm_cpu_exec() which returns EXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.
Quoted 29 lines
Quoted 29 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.
Quoted 5 lines
Quoted 1 line
Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed.
Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit 98884e0cc1:
Reverting the kvm_irqchip_create refactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance in kvm_cpu_exec(). We can either:
Remove the "Fixes:" tag entirely, or
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returning EXCP_HLT) is correct and addresses the actual deadlock mechanism.
Quoted 40 lines
Quoted 39 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 20 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)
Quoted 12 lines
{
// BQL held on entry (from caller)
if (kvm_arch_process_async_events(cpu)) {
return EXCP_HLT; // ← Returns with BQL STILL HELD
}
bql_unlock(); // ← Normal path unlocks here
// ... KVM execution loop ...
bql_lock(); // ← Re-acquires before returning
return ret;
}Yes the semantics of the function kvm_cpu_exec() is that it should always return with bql in locked state. This is because the caller kvm_vcpu_thread_fn() calls this function with bql locked and if you see the end of kvm_vcpu_thread_fn(), it releases the lock.
So if kvm_cpu_exec() unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.
Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.
Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
Quoted 5 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.
This seems like some kind of architectural issue with PowerPC. Shouldn’t qemu_process_cpu_events() -> qemu_cond_wait(cpu->halt_cond, &bql) block other secondary cpus? Then the main cpu does a qemu_cpu_kick() to make them active again at some point?
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")
Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 49 lines · Ani Sinha
Quoted 28 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)
{
// BQL held on entry (from caller)if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD }Quoted 5 lines
bql_unlock(); // ← Normal path unlocks here
// ... KVM execution loop ...
bql_lock(); // ← Re-acquires before returning
return ret;
}Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock.So if
kvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 7 lines
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked.
Another likely reason is that we cannot have large atomic regions for performance reasons. So a general principle is to acquire the lock narrowly only for specific regions of the code that requires atomicity.
Quoted 45 lines · Ani Sinha
Quoted 20 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock.So if
kvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 7 lines
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
Quoted 8 lines
Quoted 6 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?
KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquire bql_lock in qemu_default_main() whereas secondary vcpu is spinning with BQL held returning EXCP_HLT. This is causing deadlock.
Quoted 85 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 42 lines · Harsh Prateek Bora
Quoted 35 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 3 lines
The lock imbalance:
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
Quoted 9 lines
Quoted 7 lines
Quoted 6 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.
Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
Quoted 80 lines
Quoted 80 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 44 lines · Ani Sinha
Quoted 44 lines · Harsh Prateek Bora
Quoted 37 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become a bql_unlock followed by a bql_lock in the caller for EXCP_HLT, which might look a bit odd as well.
Paolo, suggestions?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
I did explain the race condition observed above. Other architectures may have different timing that masks the issue.
Quoted 83 lines
Quoted 81 lines
Quoted 81 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 51 lines · Harsh Prateek Bora
Quoted 44 lines · Ani Sinha
Quoted 44 lines · Harsh Prateek Bora
Quoted 39 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.
But you are doing exactly that across a function call.
Quoted 7 lines
Paolo, suggestions?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
I did explain the race condition observed above.
So why not fix it properly?
Quoted 82 lines
Other architectures may have different timing that masks the issue.
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 54 lines · Ani Sinha
Quoted 54 lines · Harsh Prateek Bora
Quoted 47 lines · Ani Sinha
Quoted 47 lines · Harsh Prateek Bora
Quoted 41 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.But you are doing exactly that across a function call.
I am fine with either/or maintainer's choice.
Quoted 10 lines
Quoted 10 lines
Paolo, suggestions?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
I did explain the race condition observed above.
So why not fix it properly?
I think the suggested fix is appropriate for this scenario.
Keeping BQL held forever in a loop for EXCP_HLT case isnt the right thing to do.
Quoted 85 lines
Quoted 83 lines
Other architectures may have different timing that masks the issue.
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Quoted 8 lines
Quoted 2 lines
I do not think this is the right reference. The above commit may have exposed some underlying issue but is certainly not the cause of it. Further, as we have discussed in the other thread, the changes in that commit are not even getting executed. Personally I think the core issue is somewhere else. I am not convinced this is the proper fix.
Regarding commit
98884e0cc1:Reverting the
kvm_irqchip_createrefactoring makes the problem go away. This commit may have changed timing that exposed the issue, but the root cause is the pre-existing BQL lock imbalance inkvm_cpu_exec(). We can either:Remove the "Fixes:" tag entirely, or
Lets remove fixes tag entirely unless you can pin point the exact commit that introduced the architectural issues.
Quoted 40 lines
Add a note that this is a pre-existing issue exposed by timing changes
The core fix (releasing BQL before returningEXCP_HLT) is correct and addresses the actual deadlock mechanism.Quoted 37 lines
Quoted 37 lines
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>---
accel/kvm/kvm-accel-ops.c| 4 ++++accel/kvm/kvm-all.c| 1 +2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }Signature · 1 line
2.52.0
Quoted 79 lines · Harsh Prateek Bora
Quoted 50 lines · Ani Sinha
Quoted 50 lines · Harsh Prateek Bora
Quoted 45 lines · Ani Sinha
Quoted 45 lines · Harsh Prateek Bora
Quoted 40 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.But you are doing exactly that across a function call.
I am fine with either/or maintainer's choice.
Quoted 11 lines
Quoted 11 lines
Paolo, suggestions?
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
I did explain the race condition observed above.
So why not fix it properly?
I think the suggested fix is appropriate for this scenario.
Keeping BQL held forever in a loop forEXCP_HLTcase isnt the right thing to do.
IMHO that loop is a symptom of a deeper architectural issue. The fix proposed here feels more like a incorrect hack.
Quoted 82 lines · Ani Sinha
Quoted 82 lines · Harsh Prateek Bora
Quoted 50 lines · Ani Sinha
Quoted 50 lines · Harsh Prateek Bora
Quoted 45 lines · Ani Sinha
Quoted 45 lines · Harsh Prateek Bora
Quoted 40 lines · Ani Sinha
Quoted 18 lines · Harsh Prateek Bora
Hi Ani,
Quoted 7 lines · Ani Sinha
Quoted 6 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.I am not sure I understand this. Seems
kvm_cpu_exec()does expect that the caller holds bql before calling the function. Where is the lock imbalance?The issue is not that
kvm_cpu_exec()doesn't expect the caller to hold the BQL - it does. The problem is thatkvm_cpu_exec()has inconsistent BQL handling across its return paths.Normal execution path:
int
kvm_cpu_exec(CPUState *cpu)Quoted 12 lines
{ // BQL held on entry (from caller) if (kvm_arch_process_async_events(cpu)) { return EXCP_HLT; // ← Returns with BQL STILL HELD } bql_unlock(); // ← Normal path unlocks here // ... KVM execution loop ... bql_lock(); // ← Re-acquires before returning return ret; }Yes the semantics of the function
kvm_cpu_exec()is that it should always return with bql in locked state. This is because the callerkvm_vcpu_thread_fn()calls this function with bql locked and if you see the end ofkvm_vcpu_thread_fn(), it releases the lock. So ifkvm_cpu_exec()unlocks bql internally, it has the responsibility to lock it again before returning. This makes the locking and unlocking symmetric.Quoted 4 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.But you are doing exactly that across a function call.
I am fine with either/or maintainer's choice.
Quoted 14 lines
Quoted 14 lines
Paolo, suggestions?
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 6 lines
The caller (
kvm_vcpu_thread_fn()) loops back and callskvm_cpu_exec()again, but now the BQL is already held from the previous iteration This creates a situation where the BQL is never released between iterations whenEXCP_HLTis returned.Why this matters:
On PowerPC pseries with halted secondary vCPUs (start-powered-off=true), these vCPUs repeatedly callkvm_cpu_exec()which returnsEXCP_HLT. Each iteration accumulates BQL holds, preventing other threads (including CPU 0) from making progress.This seems like some kind of architectural issue with PowerPC. Shouldn’t
qemu_process_cpu_events()->qemu_cond_wait(cpu->halt_cond, &bql)block other secondary cpus? Then the main cpu does aqemu_cpu_kick()to make them active again at some point?KVM vCPUs need to enter the kernel to handle the halted state and therefore can run. On spapr, it is handled via start-cpu rtas call for which the handler in qemu does a
qemu_cpu_kick(). However CPU 0 needs to be able to proceed before that stage is reached, but it hangs while trying to acquirebql_lockinqemu_default_main()whereas secondary vcpu is spinning with BQL held returningEXCP_HLT. This is causing deadlock.Without looking at the code, it seems there is a race condition in the way the vcpu threads are initialised in spapr. I think that needs fixing.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
I did explain the race condition observed above.
So why not fix it properly?
I think the suggested fix is appropriate for this scenario.
Keeping BQL held forever in a loop forEXCP_HLTcase isnt the right thing to do.IMHO that loop is a symptom of a deeper architectural issue. The fix proposed here feels more like a incorrect hack.
I do not think so. Irrespective of the issue, there's no reason to keep holding BQL forever in a loop like that for EXCP_HLT case.
Quoted 1 line
Quoted 7 lines · Harsh Prateek Bora
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
The lock imbalance:
When
kvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.
So why not do
bql_unlock()
/* comment explaining why this is needed */bql_lock()
return EXCP_HLT;
That should keep the function return locked and fix the problem without having to hack the caller for this case.
Quoted 7 lines · Ani Sinha
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
I did explain the race condition observed above.
So why not fix it properly?
I think the suggested fix is appropriate for this scenario.
Keeping BQL held forever in a loop forEXCP_HLTcase isnt the right thing to do.IMHO that loop is a symptom of a deeper architectural issue. The fix proposed here feels more like a incorrect hack.
Even if this is not a complete fix of an underlying problem but only a work around maybe it's safer to do this just before a release than trying to rewrite something deeper now.
Regards,
BALATON Zoltan
Quoted 18 lines · BALATON Zoltan
Quoted 4 lines · Harsh Prateek Bora
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
The lock imbalance:
Whenkvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.
Quoted 6 lines
So why not do
bql_unlock()
/* comment explaining why this is needed */bql_lock()
returnEXCP_HLT;
If we go down this path, what would be wrong with
bql_unlock()
sleep(100);bql_lock();
This would make the window even larger for the other thread to make progress … How about sleeping for 1 min? 10 min?
Quoted 14 lines
That should keep the function return locked and fix the problem without having to hack the caller for this case.
Quoted 5 lines · Ani Sinha
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
I did explain the race condition observed above.
So why not fix it properly?
I think the suggested fix is appropriate for this scenario.Keeping BQL held forever in a loop for
EXCP_HLTcase isnt the right thing to do.IMHO that loop is a symptom of a deeper architectural issue. The fix proposed here feels more like a incorrect hack.
Even if this is not a complete fix of an underlying problem but only a work around maybe it's safer to do this just before a release than trying to rewrite something deeper now.
Regards,
BALATON Zoltan
Quoted 22 lines · Ani Sinha
Quoted 15 lines · BALATON Zoltan
Quoted 4 lines · Harsh Prateek Bora
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
The lock imbalance:
Whenkvm_arch_process_async_events()returns true, the function returnsEXCP_HLTbefore thebql_unlock()call.Why should it unlock it before returning? In fact it’s opposite. If the function had unlocked bql, it should lock it again before returning.
Quoted 1 line
This means the early return path keeps the BQL held,
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
This would be the correct thing to do.
Quoted 1 line
while the normal execution path releases and re-acquires it.
Because the functions it calls after unlocking requires bql to be unlocked. Since it had to unlock it, it locks it again before returning.
It had to unlock it for the same reason - to give others a chance to lock. We need to handle failure/exception cases for the same purpose as well.
But by unlocking and returning you are breaking the semantics of the function and introducing imbalance.
I think it is better to unlock bql early in failure cases.
Even Otherwise, it would become abql_unlockfollowed by abql_lockin the caller forEXCP_HLT, which might look a bit odd as well.Quoted 6 lines
So why not do
bql_unlock()
/* comment explaining why this is needed */bql_lock()
returnEXCP_HLT;
Quoted 7 lines
If we go down this path, what would be wrong with
bql_unlock()
sleep(100);bql_lock();This would make the window even larger for the other thread to make progress … How about sleeping for 1 min? 10 min?
If the reason to unlock is to give a chance to other waiting threads to acquire the lock then no need to sleep long as we'll wait in bql_lock if another thread got the lock so less than a second sleep or maybe just a few milliseconds should be plenty of time for other threads to get a chance to run and avoid a deadlock.
I understand that you say that maybe there could be a better fix elsewhere but we don't seem to fully understand the issue and trying a bigger rewrite now is more likely to break something than this hack that should give suffucient work around for the relase at least.
But I'm not involved in this just tried to at least resolve the deadlock between you if I can't resolve the deadlock in BQL. :-)
Regards,
BALATON Zoltan
Hi,
I've tested the patch on PowerPC pseries machine and it resolves the boot hang issue seen on ppc when booting KVM guest with >1 smp value.
Test Environment:
- Host Arch: ppc64le
- Host and Guest OS: Fedora 42
- Machine Type: pseries with KVM acceleration
- QEMU: Latest master with this patch applied
Test Results:
All the following SMP topologies now boot successfully:
Single and simple multi-CPU:
- -smp 1
- -smp 2
- -smp 4
- -smp 32
Various socket/core/thread combinations (8 vCPUs):
- -smp 8,sockets=8,cores=1,threads=1
- -smp 8,sockets=1,cores=8,threads=1
- -smp 8,sockets=1,cores=1,threads=8
- -smp 8,sockets=2,cores=4,threads=1
- -smp 8,sockets=1,cores=4,threads=2
- -smp 8,sockets=2,cores=1,threads=4
- -smp 8,sockets=2,cores=2,threads=2
Higher vCPU count:
- -smp 16,sockets=2,cores=4,threads=2
- -smp 32,sockets=1,cores=8,threads=4
Thanks,
Misbah Anjum N <[email protected]>
Quoted 70 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()
is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VMfile descriptor change")
Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>--- accel/kvm/kvm-accel-ops.c | 4 ++++ accel/kvm/kvm-all.c | 1 + 2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +2
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */
accel/kvm/kvm-accel-ops.c +2
-0+bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }
Quoted 10 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.
AFAIU, with halted=1, the thread should be waiting at the qemu_process_cpu_events() qemu_cond_wait invocation which will release the BQL during the wait.
What is your irqchip setting? on/off/split? Aren't you just hitting the early return at do_kvm_irqchip_create()? The refactoring from commit 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change") made it so a few lines that would have been skipped are now executed.
In this case, probably kvm_halt_in_kernel_allowed=true is making cpu_thread_is_idle() return false and skip the wait at qemu_process_cpu_events().
Quoted 57 lines
The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>--- accel/kvm/kvm-accel-ops.c | 4 ++++ accel/kvm/kvm-all.c | 1 + 2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }
Hi Fabiano, Balaton,
Thanks for pitching in and the suggestions.
Quoted 23 lines · Fabiano Rosas
Quoted 10 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.AFAIU, with halted=1, the thread should be waiting at the
qemu_process_cpu_events()qemu_cond_waitinvocation which will release the BQL during the wait.What is your irqchip setting? on/off/split? Aren't you just hitting the early return at
do_kvm_irqchip_create()? The refactoring from commit98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") made it so a few lines that would have been skipped are now executed.
Quoted 3 lines
In this case, probably kvm_halt_in_kernel_allowed=true is making
cpu_thread_is_idle()return false and skip the wait atqemu_process_cpu_events().
You caught it right, the early return changed the behaviour with the
commit mentioned. I just tried with below change which retains the behaviour prior to the commit mentioned and it works as expected (no more BQL deadlocks).
Ani, would you like to post the patch fixing the behaviour change introduced with commit 98884e0cc1 or I can send below patch if looks fine?
accel/kvm/kvm-all.c +2 -2
@@ -2575,7 +2575,7 @@ void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)g_hash_table_insert (s->gsimap, irq,GINT_TO_POINTER (gsi)); } -static void do_kvm_irqchip_create (KVMState *s) +static int do_kvm_irqchip_create (KVMState *s) {int ret;if (kvm_check_extension (s,KVM_CAP_IRQCHIP )) {
@@ -2587,7 +2587,7 @@ static void do_kvm_irqchip_create(KVMState *s)exit (1 ); } }else { -return ; +return -1 ; }if (kvm_check_extension (s,KVM_CAP_IRQFD ) <=0 ) {
@@ -2610,13 +2610,17 @@ static void do_kvm_irqchip_create(KVMState *s) fprintf(stderr, "Create kernel irqchip failed: %s\n",
strerror(-ret));
exit(1);
}+ return 0;
}
static void kvm_irqchip_create(KVMState *s)
{
assert(s->kernel_irqchip_split != ON_OFF_AUTO_AUTO);-
- do_kvm_irqchip_create(s);
+ int ret = 0;
+
+ ret = do_kvm_irqchip_create(s);
+ if (ret < 0)
+ return;
kvm_kernel_irqchip = true;
/* If we have an in-kernel IRQ chip then we must have asynchronous
* interrupt delivery (though the reverse is not necessarily true)Quoted 58 lines
Quoted 58 lines
The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>--- accel/kvm/kvm-accel-ops.c | 4 ++++ accel/kvm/kvm-all.c | 1 + 2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }
Quoted 32 lines · Harsh Prateek Bora
Hi Fabiano, Balaton,
Thanks for pitching in and the suggestions.
Quoted 11 lines · Fabiano Rosas
Quoted 10 lines · Harsh Prateek Bora
When
kvm_cpu_exec()returnsEXCP_HLTdue tokvm_arch_process_async_events()returning true, it was returning before releasing the BQL (Big QEMU Lock). This caused a lock imbalance where the vCPU thread would loop back tokvm_cpu_exec()while still holding the BQL, leading to deadlocks.The issue manifests as boot hangs on PowerPC pseries machines with multiple vCPUs, where secondary vCPUs with start-powered-off=true remain halted and repeatedly call
kvm_cpu_exec()which returnsEXCP_HLT. Each iteration held the BQL, preventing other operations from proceeding.Quoted 11 lines
AFAIU, with halted=1, the thread should be waiting at the
qemu_process_cpu_events()qemu_cond_waitinvocation which will release the BQL during the wait.
What is your irqchip setting? on/off/split? Aren't you just hitting the early return atdo_kvm_irqchip_create()? The refactoring from commit98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") made it so a few lines that would have been skipped are now executed.
In this case, probably kvm_halt_in_kernel_allowed=true is makingcpu_thread_is_idle()return false and skip the wait atqemu_process_cpu_events().
Quoted 4 lines
You caught it right, the early return changed the behaviour with the
commit mentioned. I just tried with below change which retains the behaviour prior to the commit mentioned and it works as expected (no more BQL deadlocks).
This makes more sense. Btw if you had followed through with the suggestion I made here
https://yhbt.net/lore/all/[email protected]/
It would have pointed straight at this change and then figuring out the rest would have been straightforward.
Quoted 2 lines
Ani, would you like to post the patch fixing the behaviour change introduced with commit
98884e0cc1or I can send below patch if looks fine?
I will post something soon.
Quoted 100 lines
accel/kvm/kvm-all.c +8
-4@@ -2575,7 +2575,7 @@ void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)g_hash_table_insert (s->gsimap, irq,GINT_TO_POINTER (gsi)); } -static void do_kvm_irqchip_create (KVMState *s) +static int do_kvm_irqchip_create (KVMState *s) {int ret;if (kvm_check_extension (s,KVM_CAP_IRQCHIP )) {@@ -2587,7 +2587,7 @@ static void do_kvm_irqchip_create(KVMState *s)exit (1 ); } }else { -return ; +return -1 ; }if (kvm_check_extension (s,KVM_CAP_IRQFD ) <=0 ) {@@ -2610,13 +2610,17 @@ static void do_kvm_irqchip_create(KVMState *s)fprintf (stderr,"Create kernel irqchip failed: %s\n" ,strerror (-ret));exit (1 ); } +return 0 ; }static void kvm_irqchip_create (KVMState *s) {assert (s->kernel_irqchip_split !=ON_OFF_AUTO_AUTO ); - -do_kvm_irqchip_create (s); +int ret =0 ; + + ret =do_kvm_irqchip_create (s); +if (ret <0 ) +return ; kvm_kernel_irqchip = true;/* If we have an in-kernel IRQ chip then we must have asynchronous * interrupt delivery (though the reverse is not necessarily true) Quoted 57 lines
Quoted 57 lines
The fix has two parts:
1. In
kvm_cpu_exec()(kvm-all.c):
Release the BQL before returningEXCP_HLTin the early return path, matching the behavior of the normal execution path wherebql_unlock()is called before entering the main KVM execution loop.2. In
kvm_vcpu_thread_fn()(kvm-accel-ops.c):
Re-acquire the BQL afterkvm_cpu_exec()returnsEXCP_HLT, since the loop expects to hold the BQL when callingkvm_cpu_exec()again.This ensures proper BQL lock/unlock pairing:
-kvm_vcpu_thread_fn()holds BQL before callingkvm_cpu_exec()
-kvm_cpu_exec()releases BQL before returning (forEXCP_HLT)
-kvm_vcpu_thread_fn()re-acquires BQL ifEXCP_HLTwas returned
- Next iteration has BQL held as expectedThis is a regression introduced by commit
98884e0cc1("accel/kvm: add changes required to support KVM VM file descriptor change") which refactoredkvm_irqchip_create()and changed the initialization timing, exposing this lock imbalance issue.Fixes 98884e0cc1 ("accel/kvm: add changes required to support KVM VM file descriptor change")Reported-by Misbah Anjum N <[email protected]>Reported-by Gautam Menghani <[email protected]>Signed-off-by Harsh Prateek Bora <[email protected]>--- accel/kvm/kvm-accel-ops.c | 4 ++++ accel/kvm/kvm-all.c | 1 + 2 files changed, 5 insertions(+)accel/kvm/kvm-accel-ops.c +4
-0@@ -52,6 +52,10 @@ static void *kvm_vcpu_thread_fn(void *arg)if (cpu_can_run (cpu)) { r =kvm_cpu_exec (cpu); +if (r ==EXCP_HLT ) { +/* kvm_cpu_exec() released BQL, re-acquire for next iteration */ +bql_lock (); + }if (r ==EXCP_DEBUG ) {cpu_handle_guest_debug (cpu); }accel/kvm/kvm-all.c +1
-0@@ -3439,6 +3439,7 @@ int kvm_cpu_exec(CPUState *cpu)trace_kvm_cpu_exec ();if (kvm_arch_process_async_events (cpu)) { +bql_unlock ();return EXCP_HLT ; }
Quoted 3 lines · Ani Sinha
Quoted 2 lines
Ani, would you like to post the patch fixing the behaviour change introduced with commit
98884e0cc1or I can send below patch if looks fine?I will post something soon.
Thanks, Ani. Let's get it fixed in -rc4!