kvm: x86: Set KVM_MAX_VCPUS=1024, KVM_SOFT_MAX_VCPUS=710
v2 cover kvm: x86: Set KVM_MAX_VCPUS=1024, KVM_SOFT_MAX_VCPUS=710
This:
- Increases KVM_MAX_VCPUS 288 to 1024.
- Increases KVM_MAX_VCPU_ID from 1023 to 4096.
- Increases KVM_SOFT_MAX_VCPUS from 240 to 710.
Note that this conflicts with:
https://lore.kernel.org/lkml/[email protected]
Date: Fri, 3 Sep 2021 15:08:01 +0200
From: Juergen Gross <[email protected]>
Subject: [PATCH v2 0/6] x86/kvm: add boot parameters for max vcpu configs
Message-Id: <[email protected]>
I don't intend to block Juergen's work. I will be happy to rebase and resubmit in case Juergen's series is merged first. However, I do propose that we set the values above as the default even if Juergen's series is merged.
The additional overhead (on x86_64) of the new defaults will be:
- struct kvm_ioapic will grow from 1628 to 5084 bytes.
- struct kvm will grow from 19328 to 22272 bytes.
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() & kvm_hv_send_ipi(), vp_bitmap[] and vcpu_bitmap[] will now be 128 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 128 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied
Changes v1 -> v2:
* KVM_MAX_VCPUS is now 1024 (v1 set it to 710)
* KVM_MAX_VCPU_ID is now 4096 (v1 left it unchanged, at 1023)
v1 of this series was:
https://lore.kernel.org/lkml/[email protected]
Date: Tue, 31 Aug 2021 16:45:35 -0400
From: Eduardo Habkost <[email protected]>
Subject: [PATCH] kvm: x86: Increase MAX_VCPUS to 710
Message-Id: <[email protected]>
Eduardo Habkost (3):
kvm: x86: Set KVM_MAX_VCPU_ID to 4*KVM_MAX_VCPUS
kvm: x86: Increase MAX_VCPUS to 1024
kvm: x86: Increase KVM_SOFT_MAX_VCPUS to 710
arch/x86/include/asm/kvm_host.h | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
Signature · 1 line
2.31.1
PATCH v2 1/3 kvm: x86: Set KVM_MAX_VCPU_ID to 4*KVM_MAX_VCPUS
Instead of requiring KVM_MAX_VCPU_ID to be manually increased every time we increase KVM_MAX_VCPUS, set it to 4*KVM_MAX_VCPUS. This should be enough for CPU topologies where Cores-per-Package and Packages-per-Socket are not powers of 2.
In practice, this increases KVM_MAX_VCPU_ID from 1023 to 1152. The only side effect of this change is making some fields in struct kvm_ioapic larger, increasing the struct size from 1628 to 1780 bytes (in x86_64).
---
Note: this conflicts with:
https://lore.kernel.org/lkml/[email protected]
Date: Fri, 3 Sep 2021 15:08:03 +0200
From: Juergen Gross <[email protected]>
Subject: [PATCH v2 2/6] x86/kvm: add boot parameter for adding vcpu-id bits
Message-Id: <[email protected]>
I would be happy to drop this patch and resubmit the series if Juergen's series gets merged first. This is part of this series only because I'm not sure how the final version of Juergen's changes will look like and how long they will take to be merged.
--- arch/x86/include/asm/kvm_host.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +13 -1
@@ -39,7 +39,19 @@#define KVM_MAX_VCPUS 288#define KVM_SOFT_MAX_VCPUS 240 -#define KVM_MAX_VCPU_ID 1023 + +/* +* In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs +* might be larger than the actual number of VCPUs because the +* APIC ID encodes CPU topology information. +* +* In the worst case, we'll need less than one extra bit for the +* Core ID, and less than one extra bit for the Package (Die) ID, +* so ratio of 4 should be enough. +*/ +#define KVM_VCPU_ID_RATIO 4 +#define KVM_MAX_VCPU_ID (KVM_MAX_VCPUS * KVM_VCPU_ID_RATIO) +/* memory slots that are not exposed to userspace */ #define KVM_PRIVATE_MEM_SLOTS 3
Signature · 1 line
2.31.1
PATCH v2 2/3 kvm: x86: Increase MAX_VCPUS to 1024
In reply to Eduardo HabkostIncrease KVM_MAX_VCPUS to 1024, so we can test larger VMs.
I'm not changing KVM_SOFT_MAX_VCPUS yet because I'm afraid it might involve complicated questions around the meaning of "supported" and "recommended" in the upstream tree. KVM_SOFT_MAX_VCPUS will be changed in a separate patch.
For reference, visible effects of this change are:
- KVM_CAP_MAX_VCPUS will now return 1024 (of course)
- Default value for CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 1024
- KVM_MAX_VCPU_ID will change from 1151 to 4096
- Size of struct kvm will increase from 19328 to 22272 bytes (in x86_64)
- Size of struct kvm_ioapic will increase from 1780 to 5084 bytes (in x86_64)
- Bitmap stack variables that will grow:
- At kvm_hv_flush_tlb() kvm_hv_send_ipi(),
vp_bitmap[] and vcpu_bitmap[] will now be 128 bytes long
- vcpu_bitmap at bioapic_write_indirect() will be 128 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access from ioapic_write_indirect()" is applied
---
Note: this conflicts with:
https://lore.kernel.org/lkml/[email protected]
Date: Fri, 3 Sep 2021 15:08:07 +0200
From: Juergen Gross <[email protected]>
Subject: [PATCH v2 6/6] x86/kvm: add boot parameter for setting max number of vcpus per
guest
Message-Id: <[email protected]>
I don't intend this to block Juergen's work. If Juergen's series is merged first, I can redo this patch to change KVM_DEFAULT_MAX_VCPUS instead.
--- arch/x86/include/asm/kvm_host.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -37,7 +37,7 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 288 +#define KVM_MAX_VCPUS 1024#define KVM_SOFT_MAX_VCPUS 240/*
Signature · 1 line
2.31.1
PATCH v2 3/3 kvm: x86: Increase KVM_SOFT_MAX_VCPUS to 710
In reply to Eduardo HabkostSupport for 710 VCPUs was tested by Red Hat since RHEL-8.4, so increase KVM_SOFT_MAX_VCPUS to 710.
---
Note: I would like to get rid of KVM_SOFT_MAX_VCPUS eventually, but I am simply bumping it to 710 by now, so we can at least declare 710 VCPUs as supported while we discuss alternatives to KVM_SOFT_MAX_VCPUS.
--- arch/x86/include/asm/kvm_host.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -38,7 +38,7 @@#define __KVM_HAVE_ARCH_VCPU_DEBUGFS#define KVM_MAX_VCPUS 1024 -#define KVM_SOFT_MAX_VCPUS 240 +#define KVM_SOFT_MAX_VCPUS 710/* * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
Signature · 1 line
2.31.1
Quoted 39 lines · Eduardo Habkost
This:
- IncreasesKVM_MAX_VCPUS288 to 1024.
- IncreasesKVM_MAX_VCPU_IDfrom 1023 to 4096.
- IncreasesKVM_SOFT_MAX_VCPUSfrom 240 to 710.Note that this conflicts with:
https://lore.kernel.org/lkml/[email protected] Date: Fri, 3 Sep 2021 15:08:01 +0200 From: Juergen Gross <[email protected]> Subject: [PATCH v2 0/6] x86/kvm: add boot parameters for max vcpu configs Message-Id: <[email protected]>I don't intend to block Juergen's work. I will be happy to rebase and resubmit in case Juergen's series is merged first. However, I do propose that we set the values above as the default even if Juergen's series is merged.
The additional overhead (on
x86_64) of the new defaults will be:
- structkvm_ioapicwill grow from 1628 to 5084 bytes.
- struct kvm will grow from 19328 to 22272 bytes.
- Bitmap stack variables that will grow:
- Atkvm_hv_flush_tlb()&kvm_hv_send_ipi(), vp_bitmap[] and vcpu_bitmap[] will now be 128 bytes long
-vcpu_bitmapatbioapic_write_indirect()will be 128 bytes long once patch "KVM: x86: Fix stack-out-of-bounds memory access fromioapic_write_indirect()" is appliedChanges v1 -> v2:
*KVM_MAX_VCPUSis now 1024 (v1 set it to 710)
*KVM_MAX_VCPU_IDis now 4096 (v1 left it unchanged, at 1023)v1 of this series was:
https://lore.kernel.org/lkml/[email protected] Date: Tue, 31 Aug 2021 16:45:35 -0400 From: Eduardo Habkost <[email protected]> Subject: [PATCH] kvm: x86: Increase MAX_VCPUS to 710 Message-Id: <[email protected]>
Quoted 8 lines
Eduardo Habkost (3):
kvm: x86: SetKVM_MAX_VCPU_IDto 4*KVM_MAX_VCPUS
kvm: x86: IncreaseMAX_VCPUSto 1024
kvm: x86: IncreaseKVM_SOFT_MAX_VCPUSto 710arch/x86/include/asm/kvm_host.h | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
Queued, thanks.
Paolo
Sean Christopherson
maintainerQuoted 2 lines · Paolo Bonzini
Quoted 1 line · Eduardo Habkost
- Increases
KVM_MAX_VCPU_IDfrom 1023 to 4096.
...
Quoted 2 lines
Quoted 2 lines
Eduardo Habkost (3):
kvm: x86: SetKVM_MAX_VCPU_IDto 4*KVM_MAX_VCPUS
...
Quoted 8 lines
Quoted 6 lines
kvm: x86: Increase
MAX_VCPUSto 1024
kvm: x86: IncreaseKVM_SOFT_MAX_VCPUSto 710arch/x86/include/asm/kvm_host.h | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)Queued, thanks.
Before we commit to this, can we sort out the off-by-one mess that is KVM_MAX_VCPU_ID? As Eduardo pointed out[*], Juergen's commit 76b4f357d0e7 ("x86/kvm: fix vcpu-id indexed array sizes") _shouldn't_ be necessary because kvm_vm_ioctl_create_vcpu() rejects attempts to create id==KVM_MAX_VCPU_ID
if (id >= KVM_MAX_VCPU_ID) return -EINVAL;
which aligns with the docs for KVM_CREATE_VCPU:
The vcpu id is an integer in the range [0, max_vcpu_id)
but the fix is "needed" because rtc_irq_eoi_tracking_reset() does
bitmap_zero(ioapic->rtc_status.dest_map.map, KVM_MAX_VCPU_ID + 1);
and now we have this
DECLARE_BITMAP(map, KVM_MAX_VCPU_ID + 1); u8 vectors[KVM_MAX_VCPU_ID + 1];
which is wrong as well. The "right" fix would have been to change rtc_irq_eoi_tracking_reset(), but that looks all kinds of wrong on the surface.
Non-x86 really mucks it up because generic code does:
#ifndef KVM_MAX_VCPU_ID #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS #endif
which means pretty much everything else can create more vCPUs than vCPU IDs.
Maybe fix KVM's internal KVM_MAX_VCPU_ID so that it's incluse, and handle the backwards compability mess in KVM_CAP_MAX_VCPU_ID? Then have the max ID for x86 be (4*KVM_MAX_VCPUS - 1). E.g. something like:
arch/x86/kvm/x86.c +2 -1
@@ -4061,7 +4061,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =KVM_MAX_VCPUS ;break ;case KVM_CAP_MAX_VCPU_ID : - r =KVM_MAX_VCPU_ID ; +/* KVM's ABI is stupid. */ + r =KVM_MAX_VCPU_ID -1 ;break ;case KVM_CAP_PV_MMU :/* obsolete */ r =0 ;
virt/kvm/kvm_main.c +1 -1
@@ -3460,7 +3460,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)struct kvm_vcpu *vcpu;struct page *page; -if (id >=KVM_MAX_VCPU_ID ) +if (id >KVM_MAX_VCPU_ID )return -EINVAL ;mutex_lock (&kvm->lock);
17:23:40 ✔ ~/go/src/kernel.org/host $ gdd
arch/powerpc/include/asm/kvm_book3s.h +1 -1
@@ -434,7 +434,7 @@ extern int kvmppc_h_logical_ci_store(struct kvm_vcpu *vcpu);#define SPLIT_HACK_OFFS 0xfb000000/* -* This packs a VCPU ID from the [0..KVM_MAX_VCPU_ID) space down to the +* This packs a VCPU ID from the [0..KVM_MAX_VCPU_ID] space down to the * [0..KVM_MAX_VCPUS) space, using knowledge of the guest's core stride * (but not its actual threading mode, which is not available) to avoid * collisions.
arch/powerpc/include/asm/kvm_host.h +2 -2
@@ -33,11 +33,11 @@#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE #include <asm/kvm_book3s_asm.h> /* for MAX_SMT_THREADS */ -#define KVM_MAX_VCPU_ID (MAX_SMT_THREADS * KVM_MAX_VCORES) +#define KVM_MAX_VCPU_ID (MAX_SMT_THREADS * KVM_MAX_VCORES) - 1#define KVM_MAX_NESTED_GUESTS KVMPPC_NR_LPIDS#else -#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS +#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS - 1#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ #define __KVM_HAVE_ARCH_INTC_INITIALIZED
arch/x86/kvm/x86.c +2 -1
@@ -4061,7 +4061,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =KVM_MAX_VCPUS ;break ;case KVM_CAP_MAX_VCPU_ID : - r =KVM_MAX_VCPU_ID ; +/* KVM's ABI is stupid. */ + r =KVM_MAX_VCPU_ID +1 ;break ;case KVM_CAP_PV_MMU :/* obsolete */ r =0 ;
include/linux/kvm_host.h +1 -1
@@ -40,7 +40,7 @@#include <linux/kvm_dirty_ring.h> #ifndef KVM_MAX_VCPU_ID -#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS +#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS - 1#endif /*
virt/kvm/kvm_main.c +1 -1
@@ -3460,7 +3460,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)struct kvm_vcpu *vcpu;struct page *page; -if (id >=KVM_MAX_VCPU_ID ) +if (id >KVM_MAX_VCPU_ID )return -EINVAL ;mutex_lock (&kvm->lock);
Quoted 51 lines · Sean Christopherson
Quoted 2 lines · Paolo Bonzini
Quoted 1 line · Eduardo Habkost
- Increases
KVM_MAX_VCPU_IDfrom 1023 to 4096....
Quoted 2 lines
Quoted 2 lines
Eduardo Habkost (3):
kvm: x86: SetKVM_MAX_VCPU_IDto 4*KVM_MAX_VCPUS...
Quoted 8 lines
Quoted 6 lines
kvm: x86: Increase
MAX_VCPUSto 1024
kvm: x86: IncreaseKVM_SOFT_MAX_VCPUSto 710arch/x86/include/asm/kvm_host.h | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)Queued, thanks.
Before we commit to this, can we sort out the off-by-one mess that is
KVM_MAX_VCPU_ID? As Eduardo pointed out[*], Juergen's commit76b4f357d0e7("x86/kvm: fix vcpu-id indexed array sizes") _shouldn't_ be necessary becausekvm_vm_ioctl_create_vcpu()rejects attempts to create id==KVM_MAX_VCPU_IDif (id >= KVM_MAX_VCPU_ID) return -EINVAL;which aligns with the docs for
KVM_CREATE_VCPU:The vcpu id is an integer in the range [0,
max_vcpu_id)but the fix is "needed" because
rtc_irq_eoi_tracking_reset()doesbitmap_zero(ioapic->rtc_status.dest_map.map, KVM_MAX_VCPU_ID + 1);and now we have this
DECLARE_BITMAP(map, KVM_MAX_VCPU_ID + 1); u8 vectors[KVM_MAX_VCPU_ID + 1];which is wrong as well. The "right" fix would have been to change
rtc_irq_eoi_tracking_reset(), but that looks all kinds of wrong on the surface.Non-x86 really mucks it up because generic code does:
#ifndef KVM_MAX_VCPU_ID #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS #endif
Quoted 5 lines
which means pretty much everything else can create more vCPUs than vCPU IDs.
Maybe fix KVM's internal
KVM_MAX_VCPU_IDso that it's incluse, and handle the backwards compability mess inKVM_CAP_MAX_VCPU_ID? Then have the max ID for x86 be (4*KVM_MAX_VCPUS - 1). E.g. something like:
Wouldn't it be simpler to just revert 76b4f357d0e7 and rename KVM_MAX_VCPU_ID to KVM_MAX_VCPU_IDS?
As far as I can see, every single line of code in KVM (except the 3 lines touched by 76b4f357d0e7) treats both KVM_MAX_VCPU_ID and KVM_CAP_MAX_VCPU_ID as exclusive. Including the API documentation.
Quoted 102 lines
arch/x86/kvm/x86.c +2
-1@@ -4061,7 +4061,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =KVM_MAX_VCPUS ;break ;case KVM_CAP_MAX_VCPU_ID : - r =KVM_MAX_VCPU_ID ; +/* KVM's ABI is stupid. */ + r =KVM_MAX_VCPU_ID -1 ;break ;case KVM_CAP_PV_MMU :/* obsolete */ r =0 ;virt/kvm/kvm_main.c +1
-1@@ -3460,7 +3460,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)struct kvm_vcpu *vcpu;struct page *page; -if (id >=KVM_MAX_VCPU_ID ) +if (id >KVM_MAX_VCPU_ID )return -EINVAL ;mutex_lock (&kvm->lock);17:23:40 ✔ ~/go/src/kernel.org/host $ gdd
arch/powerpc/include/asm/kvm_book3s.h +1
-1@@ -434,7 +434,7 @@ extern int kvmppc_h_logical_ci_store(struct kvm_vcpu *vcpu);#define SPLIT_HACK_OFFS 0xfb000000/* -* This packs a VCPU ID from the [0..KVM_MAX_VCPU_ID) space down to the +* This packs a VCPU ID from the [0..KVM_MAX_VCPU_ID] space down to the * [0..KVM_MAX_VCPUS) space, using knowledge of the guest's core stride * (but not its actual threading mode, which is not available) to avoid * collisions. arch/powerpc/include/asm/kvm_host.h +2
-2@@ -33,11 +33,11 @@#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE #include <asm/kvm_book3s_asm.h> /* for MAX_SMT_THREADS */ -#define KVM_MAX_VCPU_ID (MAX_SMT_THREADS * KVM_MAX_VCORES) +#define KVM_MAX_VCPU_ID (MAX_SMT_THREADS * KVM_MAX_VCORES) - 1#define KVM_MAX_NESTED_GUESTS KVMPPC_NR_LPIDS#else -#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS +#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS - 1#endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ #define __KVM_HAVE_ARCH_INTC_INITIALIZEDarch/x86/kvm/x86.c +2
-1@@ -4061,7 +4061,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =KVM_MAX_VCPUS ;break ;case KVM_CAP_MAX_VCPU_ID : - r =KVM_MAX_VCPU_ID ; +/* KVM's ABI is stupid. */ + r =KVM_MAX_VCPU_ID +1 ;break ;case KVM_CAP_PV_MMU :/* obsolete */ r =0 ;include/linux/kvm_host.h +1
-1@@ -40,7 +40,7 @@#include <linux/kvm_dirty_ring.h> #ifndef KVM_MAX_VCPU_ID -#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS +#define KVM_MAX_VCPU_ID KVM_MAX_VCPUS - 1#endif /* virt/kvm/kvm_main.c +1
-1@@ -3460,7 +3460,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)struct kvm_vcpu *vcpu;struct page *page; -if (id >=KVM_MAX_VCPU_ID ) +if (id >KVM_MAX_VCPU_ID )return -EINVAL ;mutex_lock (&kvm->lock);
Signature · 1 line
Eduardo
Quoted 6 lines · Eduardo Habkost
Wouldn't it be simpler to just revert
76b4f357d0e7and renameKVM_MAX_VCPU_IDtoKVM_MAX_VCPU_IDS?As far as I can see, every single line of code in KVM (except the 3 lines touched by
76b4f357d0e7) treats bothKVM_MAX_VCPU_IDandKVM_CAP_MAX_VCPU_IDas exclusive. Including the API documentation.
I agree.
Will send patches.
Juergen