KVM: x86: Increase KVM_MAX_VCPUS to 4096
PATCH 1/1 KVM: x86: Increase KVM_MAX_VCPUS to 4096
Increase KVM_MAX_VCPUS to 4096 when MAXSMP is enabled.
Notable changes (when MAXSMP is enabled):
* KMV_MAX_VCPUS will increase from 1024 to 4096.
* KVM_MAX_VCPU_IDS will increase from 4096 to 16384.
* KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 64.
* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (0x40000005)].EAX will now be 4096.
* struct kvm will increase from 39408 B to 39792 B.
* struct kvm_ioapic will increase from 5240 B to 19064 B.
* The following (on-stack) bitmaps will increase from 128 B to 512 B:
* dest_vcpu_bitmap in kvm_irq_delivery_to_apic.
* vcpu_mask in kvm_hv_flush_tlb.
* vcpu_bitmap in ioapic_write_indirect.
* vp_bitmap in sparse_set_to_vcpu_mask.
---
Virtual machines with 4096 virtual CPUs have been created on 32 socket Cascade Lake and Sapphire Rapids systems.
4096 is the current maximum value because of the Hyper-V TLFS. See BUILD_BUG_ON in arch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].
arch/x86/include/asm/kvm_host.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -39,7 +39,11 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#ifdef CONFIG_MAXSMP +#define KVM_MAX_VCPUS 4096 +#else #define KVM_MAX_VCPUS 1024 +#endif /* * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
Signature · 1 line
2.35.3
Sean Christopherson
maintainerQuoted 18 lines · Kyle Meyer
Increase
KVM_MAX_VCPUSto 4096 when MAXSMP is enabled.Notable changes (when MAXSMP is enabled):
*
KMV_MAX_VCPUSwill increase from 1024 to 4096.
*KVM_MAX_VCPU_IDSwill increase from 4096 to 16384.
*KVM_HV_MAX_SPARSE_VCPU_SET_BITSwill increase from 16 to 64.
* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (0x40000005)].EAX will now be 4096.* struct kvm will increase from 39408 B to 39792 B.
* structkvm_ioapicwill increase from 5240 B to 19064 B.* The following (on-stack) bitmaps will increase from 128 B to 512 B:
*dest_vcpu_bitmapinkvm_irq_delivery_to_apic.
*vcpu_maskinkvm_hv_flush_tlb.
*vcpu_bitmapinioapic_write_indirect.
*vp_bitmapinsparse_set_to_vcpu_mask.
Quoted 8 lines
Signed-off-by Kyle Meyer <[email protected]>---
Virtual machines with 4096 virtual CPUs have been created on 32 socket Cascade Lake and Sapphire Rapids systems.4096 is the current maximum value because of the Hyper-V TLFS. See
BUILD_BUG_ONinarch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].
Mostly out of curiosity, do you care about Hyper-V support? If not, at some
point it'd probably be worth exploring a CONFIG_KVM_HYPERV option to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS.
Quoted 8 lines
arch/x86/include/asm/kvm_host.h | 4 ++++ 1 file changed, 4 insertions(+)arch/x86/include/asm/kvm_host.h +0
-0
arch/x86/include/asm/kvm_host.h line 46
Quoted 8 lines
arch/x86/include/asm/kvm_host.h +4
-0@@ -39,7 +39,11 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#ifdef CONFIG_MAXSMP +#define KVM_MAX_VCPUS 4096 +#else #define KVM_MAX_VCPUS 1024 +#endif
Rather than tightly couple this to MAXSMP, what if we add a Kconfig? I know of at least one scenario, SVM's AVIC/x2AVIC, where it would be desirable to configure KVM to a much smaller maximum. The biggest downside I can think of is that KVM selftests would need to be updated (they assume the max is >=512), and some of the tests might be completely invalid if KVM_MAX_VCPUS is too low (<256?).
E.g.
arch/x86/include/asm/kvm_host.h +1 -1
@@ -39,7 +39,7 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 1024 +#define KVM_MAX_VCPUS CONFIG_KVM_MAX_NR_VCPUS/* * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
arch/x86/kvm/Kconfig +11 -0
@@ -151,6 +151,17 @@ config KVM_PROVE_MMU If in doubt, say "N". +config KVM_MAX_NR_VCPUS + int "Maximum vCPUs per VM" + default "4096" if MAXSMP + default "1024" + range 1 4096 + depends on KVM + help + Set the maximum number of vCPUs for a single VM. Larger values + increase the memory footprint of each VM regardless of how many vCPUs + are actually created (though the memory increase is relatively small). + config KVM_EXTERNAL_WRITE_TRACKING bool
Quoted 8 lines
Quoted 3 lines
4096 is the current maximum value because of the Hyper-V TLFS. See
BUILD_BUG_ONinarch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].Mostly out of curiosity, do you care about Hyper-V support? If not, at some
point it'd probably be worth exploring aCONFIG_KVM_HYPERVoption to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS.
Yes, I care about Hyper-V support. I would like this limitation to be addressed in the future.
Quoted 5 lines
Rather than tightly couple this to MAXSMP, what if we add a
Kconfig? I know of at least one scenario, SVM's AVIC/x2AVIC, where it would be desirable to configure KVM to a much smaller maximum. The biggest downside I can think of is that KVM selftests would need to be updated (they assume the max is >=512), and some of the tests might be completely invalid ifKVM_MAX_VCPUSis too low (<256?).
That sounds good to me. I would prefer to set the range from 1024 to 4096 in this patch.
Sean Christopherson
maintainerQuoted 12 lines · Kyle Meyer
Quoted 8 lines
Quoted 3 lines
4096 is the current maximum value because of the Hyper-V TLFS. See
BUILD_BUG_ONinarch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].Mostly out of curiosity, do you care about Hyper-V support? If not, at some
point it'd probably be worth exploring aCONFIG_KVM_HYPERVoption to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS.Yes, I care about Hyper-V support. I would like this limitation to be addressed in the future.
Quoted 8 lines
Quoted 5 lines
Rather than tightly couple this to MAXSMP, what if we add a
Kconfig? I know of at least one scenario, SVM's AVIC/x2AVIC, where it would be desirable to configure KVM to a much smaller maximum. The biggest downside I can think of is that KVM selftests would need to be updated (they assume the max is >=512), and some of the tests might be completely invalid ifKVM_MAX_VCPUSis too low (<256?).That sounds good to me. I would prefer to set the range from 1024 to 4096 in this patch.
Yeah, that's probably for the best.
Quoted 24 lines · Sean Christopherson
Quoted 23 lines · Kyle Meyer
Increase
KVM_MAX_VCPUSto 4096 when MAXSMP is enabled.Notable changes (when MAXSMP is enabled):
*
KMV_MAX_VCPUSwill increase from 1024 to 4096.
*KVM_MAX_VCPU_IDSwill increase from 4096 to 16384.
*KVM_HV_MAX_SPARSE_VCPU_SET_BITSwill increase from 16 to 64.
* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (0x40000005)].EAX will now be 4096.* struct kvm will increase from 39408 B to 39792 B.
* structkvm_ioapicwill increase from 5240 B to 19064 B.* The following (on-stack) bitmaps will increase from 128 B to 512 B:
*dest_vcpu_bitmapinkvm_irq_delivery_to_apic.
*vcpu_maskinkvm_hv_flush_tlb.
*vcpu_bitmapinioapic_write_indirect.
*vp_bitmapinsparse_set_to_vcpu_mask.Signed-off-by Kyle Meyer <[email protected]>---
Virtual machines with 4096 virtual CPUs have been created on 32 socket Cascade Lake and Sapphire Rapids systems.
Quoted 9 lines
Quoted 3 lines
4096 is the current maximum value because of the Hyper-V TLFS. See
BUILD_BUG_ONinarch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].Mostly out of curiosity, do you care about Hyper-V support? If not, at some
point it'd probably be worth exploring aCONFIG_KVM_HYPERVoption to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS.
(sorry for necroposting)
While adding CONFIG_KVM_HYPERV to disable all-things-Hyper-V may make sense for some deployments (and as we already have CONFIG_KVM_XEN), I don't think we should forbid KVM_MAX_VCPUS > 4096 when it is enabled:
'general purpose' (distro) kernels are used both for hosting large Linux guests and Windows guests. Instead, I'd suggest we define KVM_MAX_HV_VCPUS as MIN(KVM_MAX_VCPUS, 4096) and then e.g. fail KVM_SET_CPUID[,2] if we already have > 4096 vCPUs + fail kvm_arch_vcpu_create() if we already have something-hyperv enabled on the already created vCPUs.
Signature · 1 line
Vitaly
Sean Christopherson
maintainerQuoted 30 lines · Vitaly Kuznetsov
Quoted 28 lines · Sean Christopherson
Quoted 26 lines · Kyle Meyer
Increase
KVM_MAX_VCPUSto 4096 when MAXSMP is enabled.Notable changes (when MAXSMP is enabled):
*
KMV_MAX_VCPUSwill increase from 1024 to 4096.
*KVM_MAX_VCPU_IDSwill increase from 4096 to 16384.
*KVM_HV_MAX_SPARSE_VCPU_SET_BITSwill increase from 16 to 64.
* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (0x40000005)].EAX will now be 4096.* struct kvm will increase from 39408 B to 39792 B.
* structkvm_ioapicwill increase from 5240 B to 19064 B.* The following (on-stack) bitmaps will increase from 128 B to 512 B:
*dest_vcpu_bitmapinkvm_irq_delivery_to_apic.
*vcpu_maskinkvm_hv_flush_tlb.
*vcpu_bitmapinioapic_write_indirect.
*vp_bitmapinsparse_set_to_vcpu_mask.Signed-off-by Kyle Meyer <[email protected]>---
Virtual machines with 4096 virtual CPUs have been created on 32 socket Cascade Lake and Sapphire Rapids systems.4096 is the current maximum value because of the Hyper-V TLFS. See
BUILD_BUG_ONinarch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's comment on https://lore.kernel.org/all/[email protected].
Quoted 7 lines
Quoted 5 lines
Mostly out of curiosity, do you care about Hyper-V support? If not, at some
point it'd probably be worth exploring aCONFIG_KVM_HYPERVoption to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS.(sorry for necroposting)
There's a big difference between mostly dead and all dead. Mostly dead is slightly alive. ;-)
Quoted 9 lines
While adding
CONFIG_KVM_HYPERVto disable all-things-Hyper-V may make sense for some deployments (and as we already haveCONFIG_KVM_XEN), I don't think we should forbidKVM_MAX_VCPUS> 4096 when it is enabled:
'general purpose' (distro) kernels are used both for hosting large Linux guests and Windows guests. Instead, I'd suggest we defineKVM_MAX_HV_VCPUSasMIN(KVM_MAX_VCPUS, 4096)and then e.g. fail KVM_SET_CPUID[,2] if we already have > 4096 vCPUs + failkvm_arch_vcpu_create()if we already have something-hyperv enabled on the already created vCPUs.
Ah, bummer. I'd still like to add CONFIG_KVM_HYPERV at some point, but I agree it should be orthogonal to KVM_MAX_VCPUS.