KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
PATCH 1/1 KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
Acquire the per-VM slots_lock when zapping all shadow pages as part of toggling nx_huge_pages. The fast zap algorithm relies on exclusivity (via slots_lock) to identify obsolete vs. valid shadow pages, e.g. it uses a single bit for its generation number. Holding slots_lock also obviates the need to acquire a read lock on the VM's srcu.
Failing to take slots_lock when toggling nx_huge_pages allows multiple instances of kvm_mmu_zap_all_fast() to run concurrently, as the other user, KVM_SET_USER_MEMORY_REGION, does not take the global kvm_lock. Concurrent fast zap instances causes obsolete shadow pages to be incorrectly identified as valid due to the single bit generation number wrapping, which results in stale shadow pages being left in KVM's MMU and leads to all sorts of undesirable behavior.
The bug is easily confirmed by running with CONFIG_PROVE_LOCKING and toggling nx_huge_pages via its module param.
Note, the fast zap algorithm could use a 64-bit generation instead of relying on exclusivity for correctness, but all callers except the recently added set_nx_huge_pages() need to hold slots_lock anyways. Given that toggling nx_huge_pages is by no means a fast path, force it to conform to the current approach instead of reworking the algorithm to support concurrent calls.
--- arch/x86/kvm/mmu.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu.c +2 -3
@@ -6285,14 +6285,13 @@ static int set_nx_huge_pages(const char *val, const struct kernel_param *kp)if (new_val != old_val) {struct kvm *kvm; -int idx;mutex_lock (&kvm_lock);list_for_each_entry (kvm, &vm_list, vm_list) { - idx =srcu_read_lock (&kvm->srcu); +mutex_lock (&kvm->slots_lock);kvm_mmu_zap_all_fast (kvm); -srcu_read_unlock (&kvm->srcu, idx); +mutex_unlock (&kvm->slots_lock);wake_up_process (kvm->arch.nx_lpage_recovery_thread); }
Signature · 1 line
2.24.0
Quoted 42 lines · Sean Christopherson
Acquire the per-VM
slots_lockwhen zapping all shadow pages as part of togglingnx_huge_pages. The fast zap algorithm relies on exclusivity (viaslots_lock) to identify obsolete vs. valid shadow pages, e.g. it uses a single bit for its generation number. Holdingslots_lockalso obviates the need to acquire a read lock on the VM's srcu.Failing to take
slots_lockwhen togglingnx_huge_pagesallows multiple instances ofkvm_mmu_zap_all_fast()to run concurrently, as the other user,KVM_SET_USER_MEMORY_REGION, does not take the globalkvm_lock. Concurrent fast zap instances causes obsolete shadow pages to be incorrectly identified as valid due to the single bit generation number wrapping, which results in stale shadow pages being left in KVM's MMU and leads to all sorts of undesirable behavior.The bug is easily confirmed by running with
CONFIG_PROVE_LOCKINGand togglingnx_huge_pagesvia its module param.Note, the fast zap algorithm could use a 64-bit generation instead of relying on exclusivity for correctness, but all callers except the recently added
set_nx_huge_pages()need to holdslots_lockanyways. Given that togglingnx_huge_pagesis by no means a fast path, force it to conform to the current approach instead of reworking the algorithm to support concurrent calls.Fixes b8e8c8303ff28 ("kvm: mmu: ITLB_MULTIHIT mitigation")Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)arch/x86/kvm/mmu.c +0
-1@@ -6285,14 +6285,13 @@ static int set_nx_huge_pages(const char *val, const struct kernel_param *kp)if (new_val != old_val) {struct kvm *kvm; -int idx;mutex_lock (&kvm_lock);
arch/x86/kvm/mmu.c static int set_nx_huge_pages(const char *val, const struct kernel_param *kp) · line 6297
Quoted 10 lines
arch/x86/kvm/mmu.c +2
-2list_for_each_entry (kvm, &vm_list, vm_list) { - idx =srcu_read_lock (&kvm->srcu); +mutex_lock (&kvm->slots_lock);kvm_mmu_zap_all_fast (kvm); -srcu_read_unlock (&kvm->srcu, idx); +mutex_unlock (&kvm->slots_lock);wake_up_process (kvm->arch.nx_lpage_recovery_thread); }
Queued, thanks.
Paolo
Paolo Bonzini
maintainerQuoted 7 lines · Sean Christopherson
Failing to take
slots_lockwhen togglingnx_huge_pagesallows multiple instances ofkvm_mmu_zap_all_fast()to run concurrently, as the other user,KVM_SET_USER_MEMORY_REGION, does not take the globalkvm_lock. Concurrent fast zap instances causes obsolete shadow pages to be incorrectly identified as valid due to the single bit generation number wrapping, which results in stale shadow pages being left in KVM's MMU and leads to all sorts of undesirable behavior.
Indeed the current code fails lockdep miserably, but isn't the whole body of kvm_mmu_zap_all_fast() covered by kvm->mmu_lock? What kind of badness can happen if kvm->slots_lock isn't taken?
Paolo
Quoted 9 lines · Paolo Bonzini
Quoted 7 lines · Sean Christopherson
Failing to take
slots_lockwhen togglingnx_huge_pagesallows multiple instances ofkvm_mmu_zap_all_fast()to run concurrently, as the other user,KVM_SET_USER_MEMORY_REGION, does not take the globalkvm_lock. Concurrent fast zap instances causes obsolete shadow pages to be incorrectly identified as valid due to the single bit generation number wrapping, which results in stale shadow pages being left in KVM's MMU and leads to all sorts of undesirable behavior.
Quoted 3 lines
Indeed the current code fails lockdep miserably, but isn't the whole body of
kvm_mmu_zap_all_fast()covered bykvm->mmu_lock? What kind of badness can happen ifkvm->slots_lockisn't taken?
kvm_zap_obsolete_pages() temporarily drops mmu_lock and reschedules so that it doesn't block other vCPUS from inserting shadow pages into the new generation of the mmu.
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Paolo Bonzini
Quoted 7 lines · Sean Christopherson
Failing to take
slots_lockwhen togglingnx_huge_pagesallows multiple instances ofkvm_mmu_zap_all_fast()to run concurrently, as the other user,KVM_SET_USER_MEMORY_REGION, does not take the globalkvm_lock. Concurrent fast zap instances causes obsolete shadow pages to be incorrectly identified as valid due to the single bit generation number wrapping, which results in stale shadow pages being left in KVM's MMU and leads to all sorts of undesirable behavior.
Quoted 7 lines
Quoted 3 lines
Indeed the current code fails lockdep miserably, but isn't the whole body of
kvm_mmu_zap_all_fast()covered bykvm->mmu_lock? What kind of badness can happen ifkvm->slots_lockisn't taken?
kvm_zap_obsolete_pages()temporarily dropsmmu_lockand reschedules so that it doesn't block other vCPUS from inserting shadow pages into the new generation of the mmu.
Oh, of course. I've worked on all this on the pre-5.4 MMU and that does not have commit ca333add693 ("KVM: x86/mmu: Explicitly track only a single invalid mmu generation").
I queued this patch with a small tweak to the commit message, to explain why it doesn't need a stable backport.
Paolo