kvm: x86: Increase MAX_VCPUS to 710
PATCH 1/1 kvm: x86: Increase MAX_VCPUS to 710
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.
For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied
--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710 +#define KVM_SOFT_MAX_VCPUS 710#define KVM_MAX_VCPU_ID 1023/* memory slots that are not exposed to userspace */ #define KVM_PRIVATE_MEM_SLOTS 3
Signature · 1 line
2.31.1
Quoted 25 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0
arch/x86/include/asm/kvm_host.h line 40
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-2@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710
Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
arch/x86/include/asm/kvm_host.h line 41
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710
Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?
Reading
commit 8c3ba334f8588e1d5099f8602cf01897720e0eca
Author: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300
KVM: x86: Raise the hard VCPU count limit
the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.
I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)
arch/x86/include/asm/kvm_host.h line 42
Quoted 1 line
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 1023
1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:
https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0
-0/* memory slots that are not exposed to userspace */ #define KVM_PRIVATE_MEM_SLOTS 3
Signature · 1 line
Vitaly
Quoted 59 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h line 40
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-2@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
arch/x86/include/asm/kvm_host.h line 41
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)arch/x86/include/asm/kvm_host.h line 42
Quoted 1 line
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 1023
Quoted 8 lines
1023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS
VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.
Quoted 3 lines
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0
-0/* memory slots that are not exposed to userspace */ #define KVM_PRIVATE_MEM_SLOTS 3
Quoted 66 lines · Igor Mammedov
Quoted 63 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h line 40
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-2@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
arch/x86/include/asm/kvm_host.h line 41
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)arch/x86/include/asm/kvm_host.h line 42
Quoted 1 line
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 10231023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:
Quoted 7 lines
Quoted 4 lines
https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS
VCPU_ID(sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal toVCPU_ID.
Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?
KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():
static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)
{
return apic->vcpu->vcpu_id;
}So I'm pretty certain this is actually APIC id and it has topology in it.
Signature · 1 line
Vitaly
Quoted 70 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 68 lines · Igor Mammedov
Quoted 65 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h line 40
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-2@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
arch/x86/include/asm/kvm_host.h line 41
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)arch/x86/include/asm/kvm_host.h line 42
Quoted 1 line
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 10231023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS
VCPU_ID(sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal toVCPU_ID.Why do we even have
KVM_MAX_VCPU_IDwhich is != KVM[_SOFT]_MAX_VCPUS then?
I'd say for compat reasons (8c3ba334f85 KVM: x86: Raise the hard VCPU count limit)
qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
See soft_vcpus_limit in qemu.
Quoted 11 lines
KVM_MAX_VCPU_IDis only checked inkvm_vm_ioctl_create_vcpu()which passes 'id' down tokvm_vcpu_init()which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned bykvm_x2apic_id():static inline u32
kvm_x2apic_id(struct kvm_lapic *apic){ return apic->vcpu->vcpu_id; }So I'm pretty certain this is actually APIC id and it has topology in it.
Yep, I mixed it up with cpu_index on QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.
It looks like KVM_MAX_VCPU_ID (KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).
Quoted 79 lines · Igor Mammedov
Quoted 76 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 73 lines · Igor Mammedov
Quoted 59 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h line 40
Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1
-2@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
arch/x86/include/asm/kvm_host.h line 41
Quoted 1 line
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)arch/x86/include/asm/kvm_host.h line 42
Quoted 1 line
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 1023Quoted 8 lines
1023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS
VCPU_ID(sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal toVCPU_ID.
Quoted 8 lines
Quoted 2 lines
Why do we even have
KVM_MAX_VCPU_IDwhich is != KVM[_SOFT]_MAX_VCPUS then?I'd say for compat reasons (
8c3ba334f85KVM: x86: Raise the hard VCPU count limit)qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
Seesoft_vcpus_limitin qemu.
That's the reason why we have KVM_SOFT_MAX_VCPUS in addition to KVM_MAX_VCPUS, not why we have KVM_MAX_VCPU_ID :-)
Quoted 12 lines
Quoted 5 lines
KVM_MAX_VCPU_IDis only checked inkvm_vm_ioctl_create_vcpu()which passes 'id' down tokvm_vcpu_init()which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned bykvm_x2apic_id():Quoted 7 lines
static inline u32
kvm_x2apic_id(struct kvm_lapic *apic){ return apic->vcpu->vcpu_id; }So I'm pretty certain this is actually APIC id and it has topology in it.
Quoted 7 lines
Yep, I mixed it up with
cpu_indexon QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.It looks like
KVM_MAX_VCPU_ID(KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).
Yes. To avoid the confusion, I'd suggest we re-define KVM_MAX_VCPU_ID as something like:
#define KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO 4
#define KVM_MAX_VCPU_ID (KVM_MAX_VCPUS * KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)
and add a comment about sparse APIC IDs/topology.
Signature · 1 line
Vitaly
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 33 lines · Igor Mammedov
Quoted 30 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 28 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
arch/x86/include/asm/kvm_host.h line 40
Quoted 3 lines
arch/x86/include/asm/kvm_host.h +1
-2-#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).
I would be more than happy to set it to 1024 or 2048 if the KVM maintainers agree. :)
For reference, RHEL-8.4 is compiled with KVM_MAX_VCPUS=2048, but userspace enforces a 710 VCPU limit.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/kvm_host.h line 41
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)
I will send a RFC later proposing we make KVM_MAX_VCPUS configurable by Kconfig, and dropping KVM_SOFT_MAX_VCPUS.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/kvm_host.h line 42
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 10231023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS
1024 seems to be enough on all the CPU topologies I have seen, but I can happily implement the 4x rule below, just to be sure.
Quoted 36 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
VCPU_ID(sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal toVCPU_ID.Quoted 8 lines
Quoted 2 lines
Why do we even have
KVM_MAX_VCPU_IDwhich is != KVM[_SOFT]_MAX_VCPUS then?I'd say for compat reasons (
8c3ba334f85KVM: x86: Raise the hard VCPU count limit)qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
Seesoft_vcpus_limitin qemu.That's the reason why we have
KVM_SOFT_MAX_VCPUSin addition toKVM_MAX_VCPUS, not why we haveKVM_MAX_VCPU_ID:-)Quoted 12 lines
Quoted 5 lines
KVM_MAX_VCPU_IDis only checked inkvm_vm_ioctl_create_vcpu()which passes 'id' down tokvm_vcpu_init()which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned bykvm_x2apic_id():Quoted 7 lines
static inline u32
kvm_x2apic_id(struct kvm_lapic *apic){ return apic->vcpu->vcpu_id; }So I'm pretty certain this is actually APIC id and it has topology in it.
Quoted 7 lines
Yep, I mixed it up with
cpu_indexon QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.It looks like
KVM_MAX_VCPU_ID(KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).
Quoted 7 lines
Yes. To avoid the confusion, I'd suggest we re-define
KVM_MAX_VCPU_IDas something like:#define
KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO4
#defineKVM_MAX_VCPU_ID(KVM_MAX_VCPUS*KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)and add a comment about sparse APIC IDs/topology.
I will submit a new version of this patch with a rule like the above.
A 4x ratio is very generous, but the only impact of a large KVM_MAX_VCPU_ID is a larger struct kvm_ioapic. Changing KVM_MAX_VCPU_ID from 1024 to 4096 will make struct kvm_ioapic grow from 1628 bytes to 5084 bytes, which I assume is OK.
Signature · 1 line
Eduardo
Quoted 3 lines · Eduardo Habkost
710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).
Oops, I mean VMs _larger_ than 710 VCPUs weren't tested. 710 was tested with real VMs, and was the higher we could go before hitting the SMBIOS 2.1 table size limits.
Signature · 1 line
Eduardo
Quoted 141 lines · Eduardo Habkost
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 33 lines · Igor Mammedov
Quoted 30 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 28 lines · Eduardo Habkost
Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase
KVM_MAX_VCPUSandKVM_SOFT_MAX_VCPUSto 710.For reference, visible effects of changing
KVM_MAX_VCPUSare:
-KVM_CAP_MAX_VCPUSandKVM_CAP_NR_VCPUSwill now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
-vcpu_bitmapwill be 92 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedSigned-off-by Eduardo Habkost <[email protected]>--- arch/x86/include/asm/kvm_host.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +0
-0@@ -37,8 +37,8 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFSQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
arch/x86/include/asm/kvm_host.h line 40
Quoted 3 lines
arch/x86/include/asm/kvm_host.h +1
-2-#define KVM_MAX_VCPUS 288 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_MAX_VCPUS 710Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)
710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).
I would be more than happy to set it to 1024 or 2048 if the KVM maintainers agree. :)
For reference, RHEL-8.4 is compiled with KVM_MAX_VCPUS=2048, but userspace enforces a 710 VCPU limit.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/kvm_host.h line 41
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +1
-0+#define KVM_SOFT_MAX_VCPUS 710Do we really need
KVM_SOFT_MAX_VCPUSwhich is equal toKVM_MAX_VCPUS?Reading
commit
8c3ba334f8588e1d5099f8602cf01897720e0ecaAuthor: Sasha Levin <[email protected]> Date: Mon Jul 18 17:17:15 2011 +0300Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
KVM: x86: Raise the hard VCPU count limit
the idea behind
KVM_SOFT_MAX_VCPUSwas to allow developers to test high vCPU numbers without claiming such configurations as supported.I have two alternative suggestions:
1) DropKVM_SOFT_MAX_VCPUScompletely.
2) Raise it to a higher number (e.g. 2048)I will send a RFC later proposing we make
KVM_MAX_VCPUSconfigurable byKconfig, and droppingKVM_SOFT_MAX_VCPUS.Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/include/asm/kvm_host.h line 42
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0
-0#define KVM_MAX_VCPU_ID 10231023 may not be enough now. I rememeber there was a suggestion to make
max_vcpusconfigurable via module parameter and this question was raised:https://lore.kernel.org/lkml/[email protected]/
TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID= 4 *KVM_MAX_VCPUS1024 seems to be enough on all the CPU topologies I have seen, but I can happily implement the 4x rule below, just to be sure.
Quoted 36 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
VCPU_ID(sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal toVCPU_ID.Quoted 8 lines
Quoted 2 lines
Why do we even have
KVM_MAX_VCPU_IDwhich is != KVM[_SOFT]_MAX_VCPUS then?I'd say for compat reasons (
8c3ba334f85KVM: x86: Raise the hard VCPU count limit)qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
Seesoft_vcpus_limitin qemu.That's the reason why we have
KVM_SOFT_MAX_VCPUSin addition toKVM_MAX_VCPUS, not why we haveKVM_MAX_VCPU_ID:-)Quoted 12 lines
Quoted 5 lines
KVM_MAX_VCPU_IDis only checked inkvm_vm_ioctl_create_vcpu()which passes 'id' down tokvm_vcpu_init()which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned bykvm_x2apic_id():Quoted 7 lines
static inline u32
kvm_x2apic_id(struct kvm_lapic *apic){ return apic->vcpu->vcpu_id; }So I'm pretty certain this is actually APIC id and it has topology in it.
Quoted 7 lines
Yep, I mixed it up with
cpu_indexon QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.It looks like
KVM_MAX_VCPU_ID(KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).Quoted 7 lines
Yes. To avoid the confusion, I'd suggest we re-define
KVM_MAX_VCPU_IDas something like:#define
KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO4
#defineKVM_MAX_VCPU_ID(KVM_MAX_VCPUS*KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)and add a comment about sparse APIC IDs/topology.
Quoted 8 lines
I will submit a new version of this patch with a rule like the above.
A 4x ratio is very generous, but the only impact of a large
KVM_MAX_VCPU_IDis a larger structkvm_ioapic. ChangingKVM_MAX_VCPU_IDfrom 1024 to 4096 will make structkvm_ioapicgrow from 1628 bytes to 5084 bytes, which I assume is OK.
I'm just about to send V2 of my series [1] to support specifying the max vcpu-id and max number of vcpus of a guest via command line parameters.
Juergen