Skip to contentSkip to conversation

kvm: x86: Increase MAX_VCPUS to 710

PATCH In review Eduardo Habkost 4 participants 9 messages Aug 31, 2021 – Sep 3, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvm: x86: Increase MAX_VCPUS to 710

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
index af6ce8d4c86a..f76fae42bf45 100644
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710
+#define KVM_SOFT_MAX_VCPUS 710
 #define KVM_MAX_VCPU_ID 1023
 /* memory slots that are not exposed to userspace */
 #define KVM_PRIVATE_MEM_SLOTS 3
Signature · 1 line
2.31.1
#1 lore raw
Quoted 25 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644

arch/x86/include/asm/kvm_host.h line 40

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -2
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

arch/x86/include/asm/kvm_host.h line 41

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

arch/x86/include/asm/kvm_host.h line 42

Quoted 1 line
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
 /* memory slots that are not exposed to userspace */
 #define KVM_PRIVATE_MEM_SLOTS 3
Signature · 1 line
Vitaly
#2 lore raw
Quoted 59 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644

arch/x86/include/asm/kvm_host.h line 40

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -2
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710  

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

arch/x86/include/asm/kvm_host.h line 41

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710  

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

arch/x86/include/asm/kvm_host.h line 42

Quoted 1 line
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023  
Quoted 8 lines

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Quoted 3 lines
Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
 /* memory slots that are not exposed to userspace */
 #define KVM_PRIVATE_MEM_SLOTS 3  
#3 lore raw
Quoted 66 lines · Igor Mammedov
Quoted 63 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644

arch/x86/include/asm/kvm_host.h line 40

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -2
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710  

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

arch/x86/include/asm/kvm_host.h line 41

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710  

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

arch/x86/include/asm/kvm_host.h line 42

Quoted 1 line
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023  

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

Quoted 7 lines
Quoted 4 lines

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?

KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():

static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)

{
        return apic->vcpu->vcpu_id;
}

So I'm pretty certain this is actually APIC id and it has topology in it.

Signature · 1 line
Vitaly
#4 lore raw
Quoted 70 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 68 lines · Igor Mammedov
Quoted 65 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644

arch/x86/include/asm/kvm_host.h line 40

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -2
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710    

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

arch/x86/include/asm/kvm_host.h line 41

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710    

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

arch/x86/include/asm/kvm_host.h line 42

Quoted 1 line
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023    

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?

I'd say for compat reasons (8c3ba334f85 KVM: x86: Raise the hard VCPU count limit)

qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
See soft_vcpus_limit in qemu.

Quoted 11 lines

KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():

static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)

{
        return apic->vcpu->vcpu_id;
}

So I'm pretty certain this is actually APIC id and it has topology in it.

Yep, I mixed it up with cpu_index on QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.

It looks like KVM_MAX_VCPU_ID (KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).

#5 lore raw
Quoted 79 lines · Igor Mammedov
Quoted 76 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 73 lines · Igor Mammedov
Quoted 59 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 25 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644

arch/x86/include/asm/kvm_host.h line 40

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -2
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710    

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

arch/x86/include/asm/kvm_host.h line 41

Quoted 1 line
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710    

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

arch/x86/include/asm/kvm_host.h line 42

Quoted 1 line
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023    
Quoted 8 lines

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Quoted 8 lines
Quoted 2 lines

Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?

I'd say for compat reasons (8c3ba334f85 KVM: x86: Raise the hard VCPU count limit)

qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
See soft_vcpus_limit in qemu.

That's the reason why we have KVM_SOFT_MAX_VCPUS in addition to KVM_MAX_VCPUS, not why we have KVM_MAX_VCPU_ID :-)

Quoted 12 lines
Quoted 5 lines

KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():

Quoted 7 lines

static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)

{
        return apic->vcpu->vcpu_id;
}

So I'm pretty certain this is actually APIC id and it has topology in it.

Quoted 7 lines

Yep, I mixed it up with cpu_index on QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.

It looks like KVM_MAX_VCPU_ID (KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).

Yes. To avoid the confusion, I'd suggest we re-define KVM_MAX_VCPU_ID as something like:

#define KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO 4
#define KVM_MAX_VCPU_ID (KVM_MAX_VCPUS * KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)

and add a comment about sparse APIC IDs/topology.

Signature · 1 line
Vitaly
#6 lore raw
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 33 lines · Igor Mammedov
Quoted 30 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 28 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644
@@ -37,8 +37,8 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

arch/x86/include/asm/kvm_host.h line 40

Quoted 3 lines
arch/x86/include/asm/kvm_host.h +1 -2
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710    

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).

I would be more than happy to set it to 1024 or 2048 if the KVM maintainers agree. :)

For reference, RHEL-8.4 is compiled with KVM_MAX_VCPUS=2048, but userspace enforces a 710 VCPU limit.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

arch/x86/include/asm/kvm_host.h line 41

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710    

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

I will send a RFC later proposing we make KVM_MAX_VCPUS configurable by Kconfig, and dropping KVM_SOFT_MAX_VCPUS.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

arch/x86/include/asm/kvm_host.h line 42

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
 #define KVM_MAX_VCPU_ID 1023    

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

1024 seems to be enough on all the CPU topologies I have seen, but I can happily implement the 4x rule below, just to be sure.

Quoted 36 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Quoted 8 lines
Quoted 2 lines

Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?

I'd say for compat reasons (8c3ba334f85 KVM: x86: Raise the hard VCPU count limit)

qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
See soft_vcpus_limit in qemu.

That's the reason why we have KVM_SOFT_MAX_VCPUS in addition to KVM_MAX_VCPUS, not why we have KVM_MAX_VCPU_ID :-)

Quoted 12 lines
Quoted 5 lines

KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():

Quoted 7 lines

static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)

{
        return apic->vcpu->vcpu_id;
}

So I'm pretty certain this is actually APIC id and it has topology in it.

Quoted 7 lines

Yep, I mixed it up with cpu_index on QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.

It looks like KVM_MAX_VCPU_ID (KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).

Quoted 7 lines

Yes. To avoid the confusion, I'd suggest we re-define KVM_MAX_VCPU_ID as something like:

#define KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO 4
#define KVM_MAX_VCPU_ID (KVM_MAX_VCPUS * KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)

and add a comment about sparse APIC IDs/topology.

I will submit a new version of this patch with a rule like the above.

A 4x ratio is very generous, but the only impact of a large KVM_MAX_VCPU_ID is a larger struct kvm_ioapic. Changing KVM_MAX_VCPU_ID from 1024 to 4096 will make struct kvm_ioapic grow from 1628 bytes to 5084 bytes, which I assume is OK.

Signature · 1 line
Eduardo
#7 lore raw
Quoted 3 lines · Eduardo Habkost

710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).

Oops, I mean VMs _larger_ than 710 VCPUs weren't tested. 710 was tested with real VMs, and was the higher we could go before hitting the SMBIOS 2.1 table size limits.

Signature · 1 line
Eduardo
#8 lore raw
In reply to Eduardo Habkost
Quoted 141 lines · Eduardo Habkost
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Wed, 01 Sep 2021 12:13:55 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 33 lines · Igor Mammedov
Quoted 30 lines · On Wed, 01 Sep 2021 10:02:18 +0200 Vitaly Kuznetsov <[email protected]> wrote:
Quoted 28 lines · Eduardo Habkost

Support for 710 VCPUs has been tested by Red Hat since RHEL-8.4. Increase KVM_MAX_VCPUS and KVM_SOFT_MAX_VCPUS to 710.

For reference, visible effects of changing KVM_MAX_VCPUS are:
- KVM_CAP_MAX_VCPUS and KVM_CAP_NR_VCPUS will now return 710 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 710
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi():
- Sparse VCPU bitmap (vp_bitmap) will be 96 bytes long
- vcpu_bitmap will be 92 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 92 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied

---
  arch/x86/include/asm/kvm_host.h | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -0
index af6ce8d4c86a..f76fae42bf45 100644
@@ -37,8 +37,8 @@
 
  #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

arch/x86/include/asm/kvm_host.h line 40

Quoted 3 lines
arch/x86/include/asm/kvm_host.h +1 -2
-#define KVM_MAX_VCPUS 288
-#define KVM_SOFT_MAX_VCPUS 240
+#define KVM_MAX_VCPUS 710

Out of pure curiosity, where did 710 came from? Is this some particular hardware which was used for testing (weird number btw). Should we maybe go to e.g. 1024 for the sake of the beauty of powers of two? :-)

710 wasn't tested with real VMs yet due to userspace limitations that still need to be addressed (specifically, due to SMBIOS 2.1 table size limits).

I would be more than happy to set it to 1024 or 2048 if the KVM maintainers agree. :)

For reference, RHEL-8.4 is compiled with KVM_MAX_VCPUS=2048, but userspace enforces a 710 VCPU limit.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

arch/x86/include/asm/kvm_host.h line 41

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +1 -0
+#define KVM_SOFT_MAX_VCPUS 710

Do we really need KVM_SOFT_MAX_VCPUS which is equal to KVM_MAX_VCPUS?

Reading

commit 8c3ba334f8588e1d5099f8602cf01897720e0eca

Author: Sasha Levin <[email protected]>
Date:   Mon Jul 18 17:17:15 2011 +0300
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

KVM: x86: Raise the hard VCPU count limit

the idea behind KVM_SOFT_MAX_VCPUS was to allow developers to test high vCPU numbers without claiming such configurations as supported.

I have two alternative suggestions:
1) Drop KVM_SOFT_MAX_VCPUS completely.
2) Raise it to a higher number (e.g. 2048)

I will send a RFC later proposing we make KVM_MAX_VCPUS configurable by Kconfig, and dropping KVM_SOFT_MAX_VCPUS.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

arch/x86/include/asm/kvm_host.h line 42

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
  #define KVM_MAX_VCPU_ID 1023

1023 may not be enough now. I rememeber there was a suggestion to make max_vcpus configurable via module parameter and this question was raised:

https://lore.kernel.org/lkml/[email protected]/

TL;DR: to support EPYC-like topologies we need to keep
KVM_MAX_VCPU_ID = 4 * KVM_MAX_VCPUS

1024 seems to be enough on all the CPU topologies I have seen, but I can happily implement the 4x rule below, just to be sure.

Quoted 36 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

VCPU_ID (sequential 0-n range) is not APIC ID (sparse distribution), so topology encoded in the later should be orthogonal to VCPU_ID.

Quoted 8 lines
Quoted 2 lines

Why do we even have KVM_MAX_VCPU_ID which is != KVM[_SOFT]_MAX_VCPUS then?

I'd say for compat reasons (8c3ba334f85 KVM: x86: Raise the hard VCPU count limit)

qemu warns users that they are out of recommended (tested) limit when it sees requested maxcpus over soft limit.
See soft_vcpus_limit in qemu.

That's the reason why we have KVM_SOFT_MAX_VCPUS in addition to KVM_MAX_VCPUS, not why we have KVM_MAX_VCPU_ID :-)

Quoted 12 lines
Quoted 5 lines

KVM_MAX_VCPU_ID is only checked in kvm_vm_ioctl_create_vcpu() which passes 'id' down to kvm_vcpu_init() which, in its turn, sets 'vcpu->vcpu_id'. This is, for example, returned by kvm_x2apic_id():

Quoted 7 lines

static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)

{
         return apic->vcpu->vcpu_id;
}

So I'm pretty certain this is actually APIC id and it has topology in it.

Quoted 7 lines

Yep, I mixed it up with cpu_index on QEMU side,
for x86 it fetches actual apic id and feeds that to kvm when creating vCPU.

It looks like KVM_MAX_VCPU_ID (KVM_SOFT_MAX_VCPUS) is essentially MAX_[SOFT_]APIC_ID which in some places is treated as max number of vCPUs, so actual max count of vCPUs could be less than that (in case of sparse APIC IDs /non power of 2 thread|core|whatever count/).

Quoted 7 lines

Yes. To avoid the confusion, I'd suggest we re-define KVM_MAX_VCPU_ID as something like:

#define KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO 4
#define KVM_MAX_VCPU_ID (KVM_MAX_VCPUS * KVM_MAX_VCPU_ID_TO_MAX_VCPUS_RATIO)

and add a comment about sparse APIC IDs/topology.

Quoted 8 lines

I will submit a new version of this patch with a rule like the above.

A 4x ratio is very generous, but the only impact of a large KVM_MAX_VCPU_ID is a larger struct kvm_ioapic. Changing KVM_MAX_VCPU_ID from 1024 to 4096 will make struct kvm_ioapic grow from 1628 bytes to 5084 bytes, which I assume is OK.

I'm just about to send V2 of my series [1] to support specifying the max vcpu-id and max number of vcpus of a guest via command line parameters.

Juergen

[1]: https://lore.kernel.org/kvm/[email protected]/