KVM: fix rcu usage warning in kvm_arch_vcpu_ioctl_set_sregs()
PATCH 1/3 KVM: fix rcu usage warning in kvm_arch_vcpu_ioctl_set_sregs()
Fix:
[ 1001.499596] =================================================== [ 1001.499599] [ INFO: suspicious rcu_dereference_check() usage. ] [ 1001.499601] --------------------------------------------------- ...... [ 1001.499633] stack backtrace: [ 1001.499636] Pid: 6035, comm: qemu-system-x86 Not tainted 2.6.37-rc6+ #62 [ 1001.499638] Call Trace: [ 1001.499644] [<ffffffff81076cf6>] lockdep_rcu_dereference+0x9d/0xa5 [ 1001.499653] [<ffffffffa02567a6>] gfn_to_memslot+0x8d/0xc8 [kvm] [ 1001.499661] [<ffffffffa0256824>] gfn_to_hva+0x16/0x3f [kvm] [ 1001.499669] [<ffffffffa02568f5>] kvm_read_guest_page+0x1e/0x5e [kvm] [ 1001.499681] [<ffffffffa026c24a>] kvm_read_guest_page_mmu+0x53/0x5e [kvm] [ 1001.499699] [<ffffffffa026c294>] load_pdptrs+0x3f/0x9c [kvm] [ 1001.499705] [<ffffffffa00d825e>] ? vmx_set_cr0+0x507/0x517 [kvm_intel] [ 1001.499717] [<ffffffffa026ca56>] kvm_arch_vcpu_ioctl_set_sregs+0x1f3/0x3c0 [kvm] [ 1001.499727] [<ffffffffa025a410>] kvm_vcpu_ioctl+0x6a5/0xbc5 [kvm]
--- arch/x86/kvm/x86.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
arch/x86/kvm/x86.c +4 -1
@@ -5571,7 +5571,7 @@ int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) {int mmu_reset_needed =0 ; -int pending_vec, max_bits; +int pending_vec, max_bits, idx;struct desc_ptr dt; dt.size = sregs->idt.limit;
@@ -5600,10 +5600,13 @@ int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, kvm_x86_ops->set_cr4 (vcpu, sregs->cr4);if (sregs->cr4 &X86_CR4_OSXSAVE )update_cpuid (vcpu); + + idx =srcu_read_lock (&vcpu->kvm->srcu);if (!is_long_mode (vcpu) &&is_pae (vcpu)) {load_pdptrs (vcpu, vcpu->arch.walk_mmu,kvm_read_cr3 (vcpu)); mmu_reset_needed =1 ; } +srcu_read_unlock (&vcpu->kvm->srcu, idx);if (mmu_reset_needed)kvm_mmu_reset_context (vcpu);
Signature · 1 line
1.7.3.4
PATCH 2/3 KVM: send IPI to vcpu only when it's in guest mode
We can interrupt the vcpu only when it's running in guest mode to reduce IPI.
It looks like only ia64 and x86 need to send IPI to other vcpus, so i only add the implementation of 'vcpu->guest_mode' in ia64, but i don't know ia64 well, please point out the right way for me if the implementation is incorrect
And ia64 is not tested since i don't have ia64 box
--- arch/ia64/kvm/kvm-ia64.c | 2 ++ virt/kvm/kvm_main.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletions(-)
arch/ia64/kvm/kvm-ia64.c +2 -0
@@ -662,6 +662,7 @@ again: goto vcpu_run_fail;srcu_read_unlock (&vcpu->kvm->srcu, idx); +atomic_set (&vcpu->guest_mode,1 );kvm_guest_enter ();/*
@@ -683,6 +684,7 @@ again:*/ barrier ();kvm_guest_exit (); +atomic_set (&vcpu->guest_mode,0 );preempt_enable (); idx =srcu_read_lock (&vcpu->kvm->srcu);
virt/kvm/kvm_main.c +2 -1
@@ -153,7 +153,8 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)if (kvm_make_check_request (req, vcpu))continue ; cpu = vcpu->cpu; -if (cpus !=NULL && cpu !=-1 && cpu != me) +if (cpus !=NULL && cpu !=-1 && cpu != me && +atomic_read (&vcpu->guest_mode))cpumask_set_cpu (cpu, cpus); }if (unlikely (cpus ==NULL ))
Signature · 1 line
1.7.3.4
Re: [PATCH 2/3] KVM: send IPI to vcpu only when it's in guest mode
Quoted 21 lines · Xiao Guangrong
We can interrupt the vcpu only when it's running in guest mode to reduce IPI.
It looks like only ia64 and x86 need to send IPI to other vcpus, so i only add the implementation of '
vcpu->guest_mode' in ia64, but i don't know ia64 well, please point out the right way for me if the implementation is incorrectAnd ia64 is not tested since i don't have ia64 box
Signed-off-by Xiao Guangrong<[email protected]>--- arch/ia64/kvm/kvm-ia64.c | 2 ++ virt/kvm/kvm_main.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletions(-)arch/ia64/kvm/kvm-ia64.c +0
-0
arch/ia64/kvm/kvm-ia64.c again: · line 666
Quoted 5 lines
arch/ia64/kvm/kvm-ia64.c +1
-0@@ -662,6 +662,7 @@ again: goto vcpu_run_fail;srcu_read_unlock (&vcpu->kvm->srcu, idx); +atomic_set (&vcpu->guest_mode,1 );kvm_guest_enter ();
I think it needs an smp_wmb() (technically x86 needs it too, but x86 is strongly ordered)
Quoted 15 lines
arch/ia64/kvm/kvm-ia64.c +1
-0/* @@ -683,6 +684,7 @@ again:*/ barrier ();kvm_guest_exit (); +atomic_set (&vcpu->guest_mode,0 );preempt_enable (); idx =srcu_read_lock (&vcpu->kvm->srcu);virt/kvm/kvm_main.c +0
-0
virt/kvm/kvm_main.c static bool make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 158
Quoted 7 lines
virt/kvm/kvm_main.c +2
-1@@ -153,7 +153,8 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)if (kvm_make_check_request (req, vcpu))continue ; cpu = vcpu->cpu; -if (cpus !=NULL && cpu !=-1 && cpu != me) +if (cpus !=NULL && cpu !=-1 && cpu != me&& +atomic_read (&vcpu->guest_mode))cpumask_set_cpu (cpu, cpus);
and smp_rmb() before the atomic_read().
virt/kvm/kvm_main.c static bool make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 160
Quoted 2 lines
virt/kvm/kvm_main.c +0
-0}if (unlikely (cpus ==NULL ))
Not sure if this is an optimization. On one hand it removes an expensive IPI for the fraction of time the cpu is out of guest mode. On the other hand it adds an unconditional cacheline bounce (and bounce back).
Hm. I see that ->guest_mode and ->requests are in fact in the same cache line. So this is likely really an optimization. We should probably reorganize kvm_vcpu so that this is made explicit.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 2/3] KVM: send IPI to vcpu only when it's in guest mode
Quoted 5 lines · Avi Kivity
Quoted 3 lines
srcu_read_unlock(&vcpu->kvm->srcu, idx); + atomic_set(&vcpu->guest_mode, 1); kvm_guest_enter();I think it needs an
smp_wmb()(technically x86 needs it too, but x86 is strongly ordered)
......
Quoted 6 lines
virt/kvm/kvm_main.c static bool make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 158
Quoted 4 lines
virt/kvm/kvm_main.c +2
-1-if (cpus !=NULL && cpu !=-1 && cpu != me) +if (cpus !=NULL && cpu !=-1 && cpu != me&& +atomic_read (&vcpu->guest_mode))cpumask_set_cpu (cpu, cpus);and
smp_rmb()before theatomic_read().
Yeah, you're right, i'll fix these in the next version.
Quoted 8 lines
Quoted 3 lines
} if (unlikely(cpus == NULL))Not sure if this is an optimization. On one hand it removes an expensive IPI for the fraction of time the cpu is out of guest mode. On the other hand it adds an unconditional cacheline bounce (and bounce back).
Hm. I see that ->guest_mode and ->requests are in fact in the same cache line. So this is likely really an optimization. We should probably reorganize
kvm_vcpuso that this is made explicit.
OK, will do it in the separate patch in the future, thanks!
Re: [PATCH 2/3] KVM: send IPI to vcpu only when it's in guest mode
In reply to Avi KivityQuoted 24 lines · Avi Kivity
Quoted 23 lines · Xiao Guangrong
We can interrupt the vcpu only when it's running in guest mode to reduce IPI.
It looks like only ia64 and x86 need to send IPI to other vcpus, so i only add the implementation of '
vcpu->guest_mode' in ia64, but i don't know ia64 well, please point out the right way for me if the implementation is incorrectAnd ia64 is not tested since i don't have ia64 box
Signed-off-by Xiao Guangrong<[email protected]>--- arch/ia64/kvm/kvm-ia64.c | 2 ++ virt/kvm/kvm_main.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletions(-)arch/ia64/kvm/kvm-ia64.c +0
-0@@ -662,6 +662,7 @@ again: goto vcpu_run_fail;
Quoted 6 lines
arch/ia64/kvm/kvm-ia64.c again: · line 666
Quoted 3 lines
arch/ia64/kvm/kvm-ia64.c +1
-0srcu_read_unlock (&vcpu->kvm->srcu, idx); +atomic_set (&vcpu->guest_mode,1 );kvm_guest_enter ();I think it needs an
smp_wmb()(technically x86 needs it too, but x86 is strongly ordered)
Hi Avi,
After read the code of ia64 more carefully, i noticed ia64 doesn't check 'vcpu->requests' before go to guest mode, so it means we no need memory barrier here? right?
Certainly, we need an smb_mb between setting vcpu->mode and checking vcpu->request in x86.
PATCH 3/3 KVM: make make_all_cpus_request() lockless
In reply to Xiao GuangrongNow, we have 'vcpu->guest_mode' to judge whether need to send ipi to other cpus, this way is very exact, so checking request bit is needless, then we can drop the spinlock let it's collateral
There is the simple test result of kernbench:
before patch:
real 5m5.100s user 3m58.051s sys 9m4.742s
real 5m3.443s user 3m58.837s sys 9m6.007s
after patch:
real 5m0.769s user 3m59.666s sys 8m41.406s
real 4m57.980s user 3m59.302s sys 8m38.916s
--- include/linux/kvm_host.h | 6 ------ virt/kvm/kvm_main.c | 9 +++------ 2 files changed, 3 insertions(+), 12 deletions(-)
include/linux/kvm_host.h +0 -6
@@ -212,7 +212,6 @@ struct kvm_memslots {struct kvm {spinlock_t mmu_lock; -raw_spinlock_t requests_lock;struct mutex slots_lock;struct mm_struct *mm;/* userspace tied to this vm */ struct kvm_memslots *memslots;
@@ -719,11 +718,6 @@ static inline void kvm_make_request(int req, struct kvm_vcpu *vcpu)set_bit (req, &vcpu->requests); } -static inline bool kvm_make_check_request (int req,struct kvm_vcpu *vcpu) -{ -return test_and_set_bit (req, &vcpu->requests); -} -static inline bool kvm_check_request (int req,struct kvm_vcpu *vcpu) {if (test_bit (req, &vcpu->requests)) {
virt/kvm/kvm_main.c +3 -6
@@ -147,11 +147,9 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)zalloc_cpumask_var (&cpus,GFP_ATOMIC ); -raw_spin_lock (&kvm->requests_lock); - me =smp_processor_id (); + me =get_cpu ();kvm_for_each_vcpu (i, vcpu, kvm) { -if (kvm_make_check_request (req, vcpu)) -continue ; +kvm_make_request (req, vcpu); cpu = vcpu->cpu;if (cpus !=NULL && cpu !=-1 && cpu != me &&atomic_read (&vcpu->guest_mode))
@@ -163,7 +161,7 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)smp_call_function_many (cpus, ack_flush,NULL ,1 ); else called = false; -raw_spin_unlock (&kvm->requests_lock); +put_cpu ();free_cpumask_var (cpus);return called; }
@@ -429,7 +427,6 @@ static struct kvm *kvm_create_vm(void) kvm->mm = current->mm;atomic_inc (&kvm->mm->mm_count);spin_lock_init (&kvm->mmu_lock); -raw_spin_lock_init (&kvm->requests_lock);kvm_eventfd_init (kvm);mutex_init (&kvm->lock);mutex_init (&kvm->irq_lock);
Signature · 1 line
1.7.3.4
Re: [PATCH 3/3] KVM: make make_all_cpus_request() lockless
Quoted 3 lines · Xiao Guangrong
Now, we have '
vcpu->guest_mode' to judge whether need to send ipi to other cpus, this way is very exact, so checking request bit is needless, then we can drop the spinlock let it's collateral
Clever.
Quoted 15 lines
virt/kvm/kvm_main.c +2
-4@@ -147,11 +147,9 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)zalloc_cpumask_var (&cpus,GFP_ATOMIC ); -raw_spin_lock (&kvm->requests_lock); - me =smp_processor_id (); + me =get_cpu ();kvm_for_each_vcpu (i, vcpu, kvm) { -if (kvm_make_check_request (req, vcpu)) -continue ; +kvm_make_request (req, vcpu); cpu = vcpu->cpu;if (cpus !=NULL && cpu !=-1 && cpu != me&&atomic_read (&vcpu->guest_mode))
virt/kvm/kvm_main.c static bool make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 167
Quoted 8 lines
virt/kvm/kvm_main.c +1
-1@@ -163,7 +161,7 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)smp_call_function_many (cpus, ack_flush,NULL ,1 ); else called = false; -raw_spin_unlock (&kvm->requests_lock); +put_cpu ();free_cpumask_var (cpus);return called; }
Maybe we can drop 'cpu != me' and then we don't need to disable preemption?
Can be done in a later patch.
Anyway, I really like this, requests_lock is an ugly wart.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 3/3] KVM: make make_all_cpus_request() lockless
Quoted 7 lines · Avi Kivity
Quoted 7 lines
virt/kvm/kvm_main.c +1
-0+kvm_make_request (req, vcpu); cpu = vcpu->cpu;if (cpus !=NULL && cpu !=-1 && cpu != me&&atomic_read (&vcpu->guest_mode))@@ -163,7 +161,7 @@ static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)smp_call_function_many (cpus, ack_flush,NULL ,1 ); else
Quoted 9 lines
virt/kvm/kvm_main.c static bool make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 167
Quoted 6 lines
virt/kvm/kvm_main.c +1
-1called = false; -raw_spin_unlock (&kvm->requests_lock); +put_cpu ();free_cpumask_var (cpus);return called; }Maybe we can drop 'cpu != me' and then we don't need to disable preemption?
Preemption should be disabled in the caller site of smp_call_function_many():
......
* You must not call this function with disabled interrupts or from a
* hardware interrupt handler or from a bottom half handler. Preemption
* must be disabled when calling this function.
*/
void smp_call_function_many(const struct cpumask *mask,
smp_call_func_t func, void *info, bool wait)
{
......