KVM: x86/mmu: Fix per-cpu counter corruption on 32-bit builds
Sean Christopherson
maintainerPATCH 1/1 KVM: x86/mmu: Fix per-cpu counter corruption on 32-bit builds
Take a signed 'long' instead of an 'unsigned long' for the number of pages to add/subtract to the total number of pages used by the MMU. This fixes a zero-extension bug on 32-bit kernels that effectively corrupts the per-cpu counter used by the shrinker.
Per-cpu counters take a signed 64-bit value on both 32-bit and 64-bit kernels, whereas kvm_mod_used_mmu_pages() takes an unsigned long and thus an unsigned 32-bit value on 32-bit kernels. As a result, the value used to adjust the per-cpu counter is zero-extended (unsigned -> signed), not sign-extended (signed -> signed), and so KVM's intended -1 gets morphed to 4294967295 and effectively corrupts the counter.
This was found by a staggering amount of sheer dumb luck when running kvm-unit-tests on a 32-bit KVM build. The shrinker just happened to kick in while running tests and do_shrink_slab() logged an error about trying to free a negative number of objects. The truly lucky part is that the kernel just happened to be a slightly stale build, as the shrinker no longer yells about negative objects as of commit 18bb473e5031 ("mm:
vmscan: shrink deferred objects proportional to priority").
vmscan: shrink_slab: mmu_shrink_scan+0x0/0x210 [kvm] negative objects to delete nr=-858993460
--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -1700,7 +1700,7 @@ static int is_empty_shadow_page(u64 *spt)* aggregate version in order to make the slab shrinker * faster */ -static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,unsigned long nr) +static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,long nr) { kvm->arch.n_used_mmu_pages += nr;percpu_counter_add (&kvm_total_used_mmu_pages, nr);
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
Quoted 21 lines · Sean Christopherson
Take a signed 'long' instead of an 'unsigned long' for the number of pages to add/subtract to the total number of pages used by the MMU. This fixes a zero-extension bug on 32-bit kernels that effectively corrupts the per-cpu counter used by the shrinker.
Per-cpu counters take a signed 64-bit value on both 32-bit and 64-bit kernels, whereas
kvm_mod_used_mmu_pages()takes an unsigned long and thus an unsigned 32-bit value on 32-bit kernels. As a result, the value used to adjust the per-cpu counter is zero-extended (unsigned -> signed), not sign-extended (signed -> signed), and so KVM's intended -1 gets morphed to 4294967295 and effectively corrupts the counter.This was found by a staggering amount of sheer dumb luck when running kvm-unit-tests on a 32-bit KVM build. The shrinker just happened to kick in while running tests and
do_shrink_slab()logged an error about trying to free a negative number of objects. The truly lucky part is that the kernel just happened to be a slightly stale build, as the shrinker no longer yells about negative objects as of commit18bb473e5031("mm:
vmscan: shrink deferred objects proportional to priority").
Quoted 6 lines
vmscan:
shrink_slab:mmu_shrink_scan+0x0/0x210[kvm] negative objects to delete nr=-858993460Fixes bc8a3d8925a8 ("kvm: mmu: Fix overflow on kvm mmu page limit calculation")Cc Ben Gardon <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Ouch!
Quoted 36 lines · Sean Christopherson
Take a signed 'long' instead of an 'unsigned long' for the number of pages to add/subtract to the total number of pages used by the MMU. This fixes a zero-extension bug on 32-bit kernels that effectively corrupts the per-cpu counter used by the shrinker.
Per-cpu counters take a signed 64-bit value on both 32-bit and 64-bit kernels, whereas
kvm_mod_used_mmu_pages()takes an unsigned long and thus an unsigned 32-bit value on 32-bit kernels. As a result, the value used to adjust the per-cpu counter is zero-extended (unsigned -> signed), not sign-extended (signed -> signed), and so KVM's intended -1 gets morphed to 4294967295 and effectively corrupts the counter.This was found by a staggering amount of sheer dumb luck when running kvm-unit-tests on a 32-bit KVM build. The shrinker just happened to kick in while running tests and
do_shrink_slab()logged an error about trying to free a negative number of objects. The truly lucky part is that the kernel just happened to be a slightly stale build, as the shrinker no longer yells about negative objects as of commit18bb473e5031("mm:
vmscan: shrink deferred objects proportional to priority").vmscan:
shrink_slab:mmu_shrink_scan+0x0/0x210[kvm] negative objects to delete nr=-858993460Fixes bc8a3d8925a8 ("kvm: mmu: Fix overflow on kvm mmu page limit calculation")Cc Ben Gardon <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static int is_empty_shadow_page(u64 *spt) · line 1706
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +1
-1@@ -1700,7 +1700,7 @@ static int is_empty_shadow_page(u64 *spt)* aggregate version in order to make the slab shrinker * faster */ -static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,unsigned long nr) +static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,long nr) { kvm->arch.n_used_mmu_pages += nr;percpu_counter_add (&kvm_total_used_mmu_pages, nr);
Queued, thanks.
Paolo
Quoted 35 lines · Sean Christopherson
Take a signed 'long' instead of an 'unsigned long' for the number of pages to add/subtract to the total number of pages used by the MMU. This fixes a zero-extension bug on 32-bit kernels that effectively corrupts the per-cpu counter used by the shrinker.
Per-cpu counters take a signed 64-bit value on both 32-bit and 64-bit kernels, whereas
kvm_mod_used_mmu_pages()takes an unsigned long and thus an unsigned 32-bit value on 32-bit kernels. As a result, the value used to adjust the per-cpu counter is zero-extended (unsigned -> signed), not sign-extended (signed -> signed), and so KVM's intended -1 gets morphed to 4294967295 and effectively corrupts the counter.This was found by a staggering amount of sheer dumb luck when running kvm-unit-tests on a 32-bit KVM build. The shrinker just happened to kick in while running tests and
do_shrink_slab()logged an error about trying to free a negative number of objects. The truly lucky part is that the kernel just happened to be a slightly stale build, as the shrinker no longer yells about negative objects as of commit18bb473e5031("mm:
vmscan: shrink deferred objects proportional to priority").vmscan:
shrink_slab:mmu_shrink_scan+0x0/0x210[kvm] negative objects to delete nr=-858993460Fixes bc8a3d8925a8 ("kvm: mmu: Fix overflow on kvm mmu page limit calculation")Cc Ben Gardon <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static int is_empty_shadow_page(u64 *spt) · line 1706
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +1
-1@@ -1700,7 +1700,7 @@ static int is_empty_shadow_page(u64 *spt)* aggregate version in order to make the slab shrinker * faster */ -static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,unsigned long nr) +static inline void kvm_mod_used_mmu_pages (struct kvm *kvm,long nr) { kvm->arch.n_used_mmu_pages += nr;percpu_counter_add (&kvm_total_used_mmu_pages, nr);
I am almost sure that I seen this bug as well (I do test 32 bit KVM hosts, even with nested 32 bit guests once in a while), but I didn't dare to investigate it due to the fact the 32 bit KVM host is a very rare thing these days.
Best regards,
Maxim Levitsky