VM: Fix a benign race in kicking vCPUs
Sean Christopherson
reviewercover VM: Fix a benign race in kicking vCPUs
Fix benign races when kicking vCPUs where the task doing the kicking can consume a stale vcpu->cpu. The races are benign because of the impliciations of task migration with respect to interrupts and being in guest mode, but IMO they're worth fixing if only as an excuse to document the flows.
Patch 2 is a tangentially related cleanup to prevent future me from trying to get rid of the NULL check on the cpumask parameters, which _looks_ like it can't ever be NULL, but has a subtle edge case due to the way CONFIG_CPUMASK_OFFSTACK=y handles cpumasks.
Sean Christopherson (2):
KVM: Clean up benign vcpu->cpu data races when kicking vCPUs
KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
virt/kvm/kvm_main.c | 46 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 37 insertions(+), 9 deletions(-)
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Sean Christopherson
reviewerPATCH 1/2 KVM: Clean up benign vcpu->cpu data races when kicking vCPUs
Fix a benign data race reported by syzbot+KCSAN[*] by ensuring vcpu->cpu is read exactly once, and by ensuring the vCPU is booted from guest mode if kvm_arch_vcpu_should_kick() returns true. Fix a similar race in kvm_make_vcpus_request_mask() by ensuring the vCPU is interrupted if kvm_request_needs_ipi() returns true.
Reading vcpu->cpu before vcpu->mode (via kvm_arch_vcpu_should_kick() or kvm_request_needs_ipi()) means the target vCPU could get migrated (change vcpu->cpu) and enter !OUTSIDE_GUEST_MODE between reading vcpu->cpud and reading vcpu->mode. If that happens, the kick/IPI will be sent to the old pCPU, not the new pCPU that is now running the vCPU or reading SPTEs.
Although failing to kick the vCPU is not exactly ideal, practically speaking it cannot cause a functional issue unless there is also a bug in the caller, and any such bug would exist regardless of kvm_vcpu_kick()'s behavior.
The purpose of sending an IPI is purely to get a vCPU into the host (or out of reading SPTEs) so that the vCPU can recognize a change in state, e.g. a KVM_REQ_* request. If vCPU's handling of the state change is required for correctness, KVM must ensure either the vCPU sees the change before entering the guest, or that the sender sees the vCPU as running in guest mode. All architectures handle this by (a) sending the request before calling kvm_vcpu_kick() and (b) checking for requests _after_ setting vcpu->mode.
x86's READING_SHADOW_PAGE_TABLES has similar requirements; KVM needs to ensure it kicks and waits for vCPUs that started reading SPTEs _before_ MMU changes were finalized, but any vCPU that starts reading after MMU changes were finalized will see the new state and can continue on uninterrupted.
For uses of kvm_vcpu_kick() that are not paired with a KVM_REQ_*, e.g. x86's kvm_arch_sync_dirty_log(), the order of the kick must not be relied upon for functional correctness, e.g. in the dirty log case, userspace cannot assume it has a 100% complete log if vCPUs are still running.
All that said, eliminate the benign race since the cost of doing so is an "extra" atomic cmpxchg() in the case where the target vCPU is loaded by the current pCPU or is not loaded at all. I.e. the kick will be skipped due to kvm_vcpu_exiting_guest_mode() seeing a compatible vcpu->mode as opposed to the kick being skipped because of the cpu checks.
Keep the "cpu != me" checks even though they appear useless/impossible at first glance. x86 processes guest IPI writes in a fast path that runs in IN_GUEST_MODE, i.e. can call kvm_vcpu_kick() from IN_GUEST_MODE. And calling kvm_vm_bugged()->kvm_make_vcpus_request_mask() from IN_GUEST or READING_SHADOW_PAGE_TABLES is perfectly reasonable.
Note, a race with the cpu_online() check in kvm_vcpu_kick() likely persists, e.g. the vCPU could exit guest mode and get offlined between the cpu_online() check and the sending of smp_send_reschedule(). But, the online check appears to exist only to avoid a WARN in x86's native_smp_send_reschedule() that fires if the target CPU is not online. The reschedule WARN exists because CPU offlining takes the CPU out of the scheduling pool, i.e. the WARN is intended to detect the case where the kernel attempts to schedule a task on an offline CPU. The actual sending of the IPI is a non-issue as at worst it will simpy be dropped on the floor. In other words, KVM's usurping of the reschedule IPI could theoretically trigger a WARN if the stars align, but there will be no loss of functionality.
[*] https://syzkaller.appspot.com/bug?extid=cd4154e502f43f10808a
--- virt/kvm/kvm_main.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-)
virt/kvm/kvm_main.c +28 -8
@@ -273,14 +273,26 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,continue ;kvm_make_request (req, vcpu); - cpu = vcpu->cpu;if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; -if (tmp !=NULL && cpu !=-1 && cpu != me && -kvm_request_needs_ipi (vcpu, req)) -__cpumask_set_cpu (cpu, tmp); +/* +* Note, the vCPU could get migrated to a different pCPU at any +* point after kvm_request_needs_ipi(), which could result in +* sending an IPI to the previous pCPU. But, that's ok because +* the purpose of the IPI is to ensure the vCPU returns to +* OUTSIDE_GUEST_MODE, which is satisfied if the vCPU migrates. +* Entering READING_SHADOW_PAGE_TABLES after this point is also +* ok, as the requirement is only that KVM wait for vCPUs that +* were reading SPTEs _before_ any changes were finalized. See +* kvm_vcpu_kick() for more details on handling requests. +*/ +if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { + cpu =READ_ONCE (vcpu->cpu); +if (cpu !=-1 && cpu != me) +__cpumask_set_cpu (cpu, tmp); + } } called =kvm_kick_many_cpus (tmp, !!(req &KVM_REQUEST_WAIT ));
@@ -3309,16 +3321,24 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_wake_up);*/ void kvm_vcpu_kick (struct kvm_vcpu *vcpu) { -int me; -int cpu = vcpu->cpu; +int me, cpu;if (kvm_vcpu_wake_up (vcpu))return ; +/* +* Note, the vCPU could get migrated to a different pCPU at any point +* after kvm_arch_vcpu_should_kick(), which could result in sending an +* IPI to the previous pCPU. But, that's ok because the purpose of the +* IPI is to force the vCPU to leave IN_GUEST_MODE, and migrating the +* vCPU also requires it to leave IN_GUEST_MODE. +*/ me =get_cpu (); -if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu)) -if (kvm_arch_vcpu_should_kick (vcpu)) +if (kvm_arch_vcpu_should_kick (vcpu)) { + cpu =READ_ONCE (vcpu->cpu); +if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu))smp_send_reschedule (cpu); + }put_cpu (); }EXPORT_SYMBOL_GPL (kvm_vcpu_kick);
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH 1/2] KVM: Clean up benign vcpu->cpu data races when kicking vCPUs
Quoted 82 lines · Sean Christopherson
Fix a benign data race reported by syzbot+KCSAN[*] by ensuring
vcpu->cpuis read exactly once, and by ensuring the vCPU is booted from guest mode ifkvm_arch_vcpu_should_kick()returns true. Fix a similar race inkvm_make_vcpus_request_mask()by ensuring the vCPU is interrupted ifkvm_request_needs_ipi()returns true.Reading
vcpu->cpubeforevcpu->mode(viakvm_arch_vcpu_should_kick()orkvm_request_needs_ipi()) means the target vCPU could get migrated (changevcpu->cpu) and enter !OUTSIDE_GUEST_MODE between readingvcpu->cpudand readingvcpu->mode. If that happens, the kick/IPI will be sent to the old pCPU, not the new pCPU that is now running the vCPU or reading SPTEs.Although failing to kick the vCPU is not exactly ideal, practically speaking it cannot cause a functional issue unless there is also a bug in the caller, and any such bug would exist regardless of kvm_vcpu_kick()'s behavior.
The purpose of sending an IPI is purely to get a vCPU into the host (or out of reading SPTEs) so that the vCPU can recognize a change in state, e.g. a KVM_REQ_* request. If vCPU's handling of the state change is required for correctness, KVM must ensure either the vCPU sees the change before entering the guest, or that the sender sees the vCPU as running in guest mode. All architectures handle this by (a) sending the request before calling
kvm_vcpu_kick()and (b) checking for requests _after_ settingvcpu->mode.x86's
READING_SHADOW_PAGE_TABLEShas similar requirements; KVM needs to ensure it kicks and waits for vCPUs that started reading SPTEs _before_ MMU changes were finalized, but any vCPU that starts reading after MMU changes were finalized will see the new state and can continue on uninterrupted.For uses of
kvm_vcpu_kick()that are not paired with a KVM_REQ_*, e.g. x86'skvm_arch_sync_dirty_log(), the order of the kick must not be relied upon for functional correctness, e.g. in the dirty log case, userspace cannot assume it has a 100% complete log if vCPUs are still running.All that said, eliminate the benign race since the cost of doing so is an "extra" atomic
cmpxchg()in the case where the target vCPU is loaded by the current pCPU or is not loaded at all. I.e. the kick will be skipped due tokvm_vcpu_exiting_guest_mode()seeing a compatiblevcpu->modeas opposed to the kick being skipped because of the cpu checks.Keep the "cpu != me" checks even though they appear useless/impossible at first glance. x86 processes guest IPI writes in a fast path that runs in
IN_GUEST_MODE, i.e. can callkvm_vcpu_kick()fromIN_GUEST_MODE. And calling kvm_vm_bugged()->kvm_make_vcpus_request_mask() fromIN_GUESTorREADING_SHADOW_PAGE_TABLESis perfectly reasonable.Note, a race with the
cpu_online()check inkvm_vcpu_kick()likely persists, e.g. the vCPU could exit guest mode and get offlined between thecpu_online()check and the sending ofsmp_send_reschedule(). But, the online check appears to exist only to avoid a WARN in x86'snative_smp_send_reschedule()that fires if the target CPU is not online. The reschedule WARN exists because CPU offlining takes the CPU out of the scheduling pool, i.e. the WARN is intended to detect the case where the kernel attempts to schedule a task on an offline CPU. The actual sending of the IPI is a non-issue as at worst it will simpy be dropped on the floor. In other words, KVM's usurping of the reschedule IPI could theoretically trigger a WARN if the stars align, but there will be no loss of functionality.[*] https://syzkaller.appspot.com/bug?extid=cd4154e502f43f10808a
Cc Venkatesh Srinivas <[email protected]>Cc Vitaly Kuznetsov <[email protected]>Fixes 97222cc83163 ("KVM: Emulate local APIC in kernel")Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/kvm_main.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-)virt/kvm/kvm_main.c +0
-1@@ -273,14 +273,26 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,continue ;kvm_make_request (req, vcpu); - cpu = vcpu->cpu;
virt/kvm/kvm_main.c bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 283
Quoted 10 lines
virt/kvm/kvm_main.c +4
-3if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; -if (tmp !=NULL && cpu !=-1 && cpu != me && -kvm_request_needs_ipi (vcpu, req)) -__cpumask_set_cpu (cpu, tmp); +/* +* Note, the vCPU could get migrated to a different pCPU at any +* point after kvm_request_needs_ipi(), which could result in +* sending an IPI to the previous pCPU. But, that's ok because
"OK" (unless there's risk someone will think of Oklahoma and take it as an offense) :-)
virt/kvm/kvm_main.c bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 287
Quoted 4 lines
virt/kvm/kvm_main.c +4
-0+* the purpose of the IPI is to ensure the vCPU returns to +* OUTSIDE_GUEST_MODE, which is satisfied if the vCPU migrates. +* Entering READING_SHADOW_PAGE_TABLES after this point is also +* ok, as the requirement is only that KVM wait for vCPUs that
"OK"
Quoted 16 lines
virt/kvm/kvm_main.c +8
-1+* were reading SPTEs _before_ any changes were finalized. See +* kvm_vcpu_kick() for more details on handling requests. +*/ +if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { + cpu =READ_ONCE (vcpu->cpu); +if (cpu !=-1 && cpu != me) +__cpumask_set_cpu (cpu, tmp); + } } called =kvm_kick_many_cpus (tmp, !!(req &KVM_REQUEST_WAIT ));@@ -3309,16 +3321,24 @@ EXPORT_SYMBOL_GPL(kvm_vcpu_wake_up);*/ void kvm_vcpu_kick (struct kvm_vcpu *vcpu) { -int me;
virt/kvm/kvm_main.c EXPORT_SYMBOL_GPL(kvm_vcpu_wake_up); · line 3332
Quoted 10 lines
virt/kvm/kvm_main.c +5
-1-int cpu = vcpu->cpu; +int me, cpu;if (kvm_vcpu_wake_up (vcpu))return ; +/* +* Note, the vCPU could get migrated to a different pCPU at any point +* after kvm_arch_vcpu_should_kick(), which could result in sending an +* IPI to the previous pCPU. But, that's ok because the purpose of the
"OK"
virt/kvm/kvm_main.c EXPORT_SYMBOL_GPL(kvm_vcpu_wake_up); · line 3339
Quoted 9 lines
virt/kvm/kvm_main.c +6
-2+* IPI is to force the vCPU to leave IN_GUEST_MODE, and migrating the +* vCPU also requires it to leave IN_GUEST_MODE. +*/ me =get_cpu (); -if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu)) -if (kvm_arch_vcpu_should_kick (vcpu)) +if (kvm_arch_vcpu_should_kick (vcpu)) { + cpu =READ_ONCE (vcpu->cpu); +if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu))
It seems it was Marcelo who wrote "(unsigned)cpu < nr_cpu_ids" back in 2009 but isn't it the same as "cpu != -1" or are there any other possible negative values? I don't think vcpu->cpu can go above nr_cpu_ids somehow but maybe it can?
virt/kvm/kvm_main.c EXPORT_SYMBOL_GPL(kvm_vcpu_wake_up); · line 3344
Quoted 5 lines
virt/kvm/kvm_main.c +1
-0smp_send_reschedule (cpu); + }put_cpu (); }EXPORT_SYMBOL_GPL (kvm_vcpu_kick);
Signature · 1 line
Vitaly
Sean Christopherson
reviewerPATCH 2/2 KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
In reply to Sean ChristophersonCheck for a NULL cpumask_var_t when kicking multiple vCPUs if and only if cpumasks are configured to be allocated off-stack. This is a meaningless optimization, e.g. avoids a TEST+Jcc and TEST+CMOV on x86, but more importantly helps document that the NULL check is necessary even though all callers pass in a local variable.
No functional change intended.
--- virt/kvm/kvm_main.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +10 -2
@@ -247,7 +247,7 @@ static void ack_flush(void *_completed)static inline bool kvm_kick_many_cpus (const struct cpumask *cpus,bool wait) { -if (unlikely (!cpus)) +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) &&unlikely (!cpus)) cpus = cpu_online_mask;if (cpumask_empty (cpus))
@@ -277,6 +277,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; +/* +* tmp can be NULL if cpumasks are allocated off stack, as +* allocation of the mask is deliberately not fatal and is +* handled by falling back to kicking all online CPUs. +*/ +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) && !tmp) +continue ; +/* * Note, the vCPU could get migrated to a different pCPU at any * point after kvm_request_needs_ipi(), which could result in
@@ -288,7 +296,7 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,* were reading SPTEs _before_ any changes were finalized. See * kvm_vcpu_kick() for more details on handling requests. */ -if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { +if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu);if (cpu !=-1 && cpu != me)__cpumask_set_cpu (cpu, tmp);
Signature · 1 line
2.33.0.rc2.250.ged5fa647cd-goog
Re: [PATCH 2/2] KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
Quoted 42 lines · Sean Christopherson
Check for a NULL
cpumask_var_twhen kicking multiple vCPUs if and only if cpumasks are configured to be allocated off-stack. This is a meaningless optimization, e.g. avoids a TEST+Jcc and TEST+CMOV on x86, but more importantly helps document that the NULL check is necessary even though all callers pass in a local variable.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/kvm_main.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +9
-1@@ -247,7 +247,7 @@ static void ack_flush(void *_completed)static inline bool kvm_kick_many_cpus (const struct cpumask *cpus,bool wait) { -if (unlikely (!cpus)) +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) &&unlikely (!cpus)) cpus = cpu_online_mask;if (cpumask_empty (cpus))@@ -277,6 +277,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; +/* +* tmp can be NULL if cpumasks are allocated off stack, as +* allocation of the mask is deliberately not fatal and is +* handled by falling back to kicking all online CPUs. +*/ +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) && !tmp) +continue ; +/* * Note, the vCPU could get migrated to a different pCPU at any * point after kvm_request_needs_ipi(), which could result in
virt/kvm/kvm_main.c bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 302
Quoted 8 lines
virt/kvm/kvm_main.c +1
-1@@ -288,7 +296,7 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,* were reading SPTEs _before_ any changes were finalized. See * kvm_vcpu_kick() for more details on handling requests. */ -if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { +if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu);if (cpu !=-1 && cpu != me)__cpumask_set_cpu (cpu, tmp);
In case MM people don't like us poking into CONFIG_CPUMASK_OFFSTACK details we can probably get away with a comment. Otherwise
Signature · 1 line
Vitaly
Re: [PATCH 2/2] KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
In reply to Sean ChristophersonQuoted 29 lines · Sean Christopherson
Check for a NULL
cpumask_var_twhen kicking multiple vCPUs if and only if cpumasks are configured to be allocated off-stack. This is a meaningless optimization, e.g. avoids a TEST+Jcc and TEST+CMOV on x86, but more importantly helps document that the NULL check is necessary even though all callers pass in a local variable.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- virt/kvm/kvm_main.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +1
-1@@ -247,7 +247,7 @@ static void ack_flush(void *_completed)static inline bool kvm_kick_many_cpus (const struct cpumask *cpus,bool wait) { -if (unlikely (!cpus)) +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) &&unlikely (!cpus)) cpus = cpu_online_mask;if (cpumask_empty (cpus))@@ -277,6 +277,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))
virt/kvm/kvm_main.c bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 287
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0continue ; +/* +* tmp can be NULL if cpumasks are allocated off stack, as +* allocation of the mask is deliberately not fatal and is +* handled by falling back to kicking all online CPUs. +*/ +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) && !tmp) +continue ; +
Hello, Sean
I don't think it is a good idea to reinvent the cpumask_available(). You can rework the patch as the following code if cpumask_available() fits for you.
Thanks
Lai
virt/kvm/kvm_main.c +4 -2
@@ -245,9 +245,11 @@ static void ack_flush(void *_completed) { } -static inline bool kvm_kick_many_cpus (const struct cpumask *cpus,bool wait) +static inline bool kvm_kick_many_cpus (cpumask_var_t tmp,bool wait) { -if (unlikely (!cpus)) +const struct cpumask *cpus = tmp; + +if (unlikely (!cpumask_available (tmp))) cpus = cpu_online_mask;if (cpumask_empty (cpus))
@@ -278,7 +280,7 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm,
unsigned int req,
if (!(req & KVM_REQUEST_NO_WAKEUP) && kvm_vcpu_wake_up(vcpu))
continue;- if (tmp != NULL && cpu != -1 && cpu != me &&
+ if (cpumask_available(tmp) && cpu != -1 && cpu != me &&
kvm_request_needs_ipi(vcpu, req))
__cpumask_set_cpu(cpu, tmp);
}Quoted 15 lines
virt/kvm/kvm_main.c +1
-2/* * Note, the vCPU could get migrated to a different pCPU at any * point after kvm_request_needs_ipi(), which could result in @@ -288,7 +296,7 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,* were reading SPTEs _before_ any changes were finalized. See * kvm_vcpu_kick() for more details on handling requests. */ -if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { +if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu);if (cpu !=-1 && cpu != me)__cpumask_set_cpu (cpu, tmp); --2.33.0.rc2.250.ged5fa647cd-goog
Sean Christopherson
reviewerRe: [PATCH 2/2] KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
Quoted 16 lines · Lai Jiangshan
Review bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 287
Quoted 12 lines · Sean Christopherson
(quoted diff) +8
-0@@ -277,6 +277,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, if (!(req & KVM_REQUEST_NO_WAKEUP) && kvm_vcpu_wake_up(vcpu)) continue; + /* + * tmp can be NULL if cpumasks are allocated off stack, as + * allocation of the mask is deliberately not fatal and is + * handled by falling back to kicking all online CPUs. + */ + if (IS_ENABLED(CONFIG_CPUMASK_OFFSTACK) && !tmp) + continue; +Hello, Sean
Quoted 1 line
I don't think it is a good idea to reinvent the
cpumask_available().
Using cpumask_available() is waaaay better, thanks!
Vitaly / Paolo, take this one instead?
From deff3e168c0612a2947d1ef29e488282631a788c Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Fri, 20 Aug 2021 13:36:21 -0700
Subject: [PATCH] KVM: Use cpumask_available() to check for NULL cpumask when
kicking vCPUs
Check for a NULL cpumask_var_t when kicking multiple vCPUs via cpumask_available(), which performs a !NULL check if and only if cpumasks are configured to be allocated off-stack. This is a meaningless optimization, e.g. avoids a TEST+Jcc and TEST+CMOV on x86, but more importantly helps document that the NULL check is necessary even though all callers pass in a local variable.
No functional change intended.
--- virt/kvm/kvm_main.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +15 -3
@@ -245,9 +245,13 @@ static void ack_flush(void *_completed) { } -static inline bool kvm_kick_many_cpus (const struct cpumask *cpus,bool wait) +static inline bool kvm_kick_many_cpus (cpumask_var_t tmp,bool wait) { -if (unlikely (!cpus)) +const struct cpumask *cpus; + +if (likely (cpumask_available (tmp))) + cpus = tmp; +else cpus = cpu_online_mask;if (cpumask_empty (cpus))
@@ -277,6 +281,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; +/* +* tmp can be "unavailable" if cpumasks are allocated off stack +* as allocation of the mask is deliberately not fatal and is +* handled by falling back to kicking all online CPUs. +*/ +if (!cpumask_available (tmp)) +continue ; +/* * Note, the vCPU could get migrated to a different pCPU at any * point after kvm_request_needs_ipi(), which could result in
@@ -288,7 +300,7 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,* were reading SPTEs _before_ any changes were finalized. See * kvm_vcpu_kick() for more details on handling requests. */ -if (tmp !=NULL &&kvm_request_needs_ipi (vcpu, req)) { +if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu);if (cpu !=-1 && cpu != me)__cpumask_set_cpu (cpu, tmp);
--
Re: [PATCH 2/2] KVM: Guard cpusmask NULL check with CONFIG_CPUMASK_OFFSTACK
Quoted 17 lines · Sean Christopherson
Quoted 16 lines · Lai Jiangshan
virt/kvm/kvm_main.c bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req, · line 287
Quoted 12 lines · Sean Christopherson
virt/kvm/kvm_main.c +8
-0@@ -277,6 +277,14 @@ bool kvm_make_vcpus_request_mask(struct kvm *kvm, unsigned int req,if (!(req &KVM_REQUEST_NO_WAKEUP ) &&kvm_vcpu_wake_up (vcpu))continue ; +/* +* tmp can be NULL if cpumasks are allocated off stack, as +* allocation of the mask is deliberately not fatal and is +* handled by falling back to kicking all online CPUs. +*/ +if (IS_ENABLED (CONFIG_CPUMASK_OFFSTACK ) && !tmp) +continue ; +Hello, Sean
Quoted 6 lines
Quoted 1 line
I don't think it is a good idea to reinvent the
cpumask_available().Using
cpumask_available()is waaaay better, thanks!Vitaly / Paolo, take this one instead?
Sure, putting this to my v3, thanks!
Signature · 1 line
Vitaly