Skip to content

KVM: Refactor vCPU creation

Sean Christopherson kvm kvm-ppc kvmarm linux-arm-kernel linux-mips lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest45 patches 29 repliesIn review
  2. v145 patches 5 repliesSuperseded

Changes in v2 from the cover letter

- Rebase to commit e41a90be9659 ("KVM: x86/mmu: WARN if root_hpa is invalid when handling a page fault"). A few minor x86 (VMX) conflicts, and one straightforward arm conflict with commit 8564d6372a7d ("KVM:
arm64: Support stolen time reporting via shared structure") in patch "KVM: ARM: Move all vcpu init code into kvm_arch_vcpu_create()".
- Collect Reviews and Acks [Christoffer, Greg].
- Fix a warning in "KVM: arm64: Free sve_state via arm specific hook" by using a "void" return type [Christoffer].
- Fix a bug in "KVM: PPC: Move kvm_vcpu_init() invocation to common code" where the call in kvmppc_core_vcpu_create_e500mc() was inadvertantly left behind.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/45KVM: PPC: Book3S HV: Uninit vCPU if vcore creation failsarch/powerpc/kvm/book3s_hv.c+3 -11 reply
2/45KVM: PPC: Book3S PR: Free shared page if mmu initialization failsarch/powerpc/kvm/book3s_pr.c+3 -11 reply
3/45KVM: x86: Free wbinvd_dirty_mask if vCPU creation failsarch/x86/kvm/x86.c+1 -10 replies
4/45KVM: VMX: Allocate VPID after initializing VCPUarch/x86/kvm/vmx/vmx.c+3 -30 replies
5/45KVM: VMX: Use direct vcpu pointer during vCPU create/freearch/x86/kvm/vmx/vmx.c+22 -200 replies
6/45KVM: SVM: Use direct vcpu pointer during vCPU create/freearch/x86/kvm/svm.c+16 -140 replies
7/45KVM: x86: Allocate vcpu struct in common x86 codearch/x86/include/asm/kvm_host.h arch/x86/kvm/svm.c arch/x86/kvm/vmx/vmx.c +1 more+30 -400 replies
8/45KVM: x86: Move FPU allocation to common x86 codearch/x86/kvm/svm.c arch/x86/kvm/vmx/vmx.c arch/x86/kvm/x86.c+23 -480 replies
9/45KVM: x86: Move allocation of pio_data page down a few linesarch/x86/kvm/x86.c+13 -200 replies
10/45KVM: x86: Move kvm_vcpu_init() invocation to common codearch/x86/include/asm/kvm_host.h arch/x86/kvm/svm.c arch/x86/kvm/vmx/vmx.c +1 more+25 -290 replies
11/45KVM: PPC: e500mc: Add build-time assert that vcpu is at offset 0arch/powerpc/kvm/e500mc.c+2 -00 replies
12/45KVM: PPC: Allocate vcpu struct in common PPC codearch/powerpc/include/asm/kvm_ppc.h arch/powerpc/kvm/book3s.c arch/powerpc/kvm/book3s_hv.c +5 more+49 -753 replies
13/45KVM: PPC: Book3S PR: Allocate book3s and shadow vcpu after common initarch/powerpc/kvm/book3s_pr.c+11 -110 replies
14/45KVM: PPC: e500mc: Move reset of oldpir below call to kvm_vcpu_init()arch/powerpc/kvm/e500mc.c+3 -30 replies
15/45KVM: PPC: Move kvm_vcpu_init() invocation to common codearch/powerpc/include/asm/kvm_ppc.h arch/powerpc/kvm/book3s.c arch/powerpc/kvm/book3s_hv.c +5 more+28 -533 replies
16/45KVM: MIPS: Use kvm_vcpu_cache to allocate vCPUsarch/mips/kvm/mips.c+3 -30 replies
17/45KVM: MIPS: Drop kvm_arch_vcpu_free()arch/mips/kvm/mips.c+2 -70 replies
18/45KVM: PPC: Drop kvm_arch_vcpu_free()arch/powerpc/kvm/powerpc.c+2 -70 replies
19/45KVM: arm: Drop kvm_arch_vcpu_free()virt/kvm/arm/arm.c+2 -70 replies
20/45KVM: x86: Remove spurious kvm_mmu_unload() from vcpu destruction patharch/x86/kvm/x86.c+0 -40 replies
21/45KVM: x86: Remove spurious clearing of async #PF MSRarch/x86/kvm/x86.c+0 -20 replies
22/45KVM: x86: Drop kvm_arch_vcpu_free()arch/x86/kvm/x86.c+11 -160 replies
23/45KVM: Remove kvm_arch_vcpu_free() declarationinclude/linux/kvm_host.h+0 -10 replies
24/45KVM: Add kvm_arch_vcpu_precreate() to handle pre-allocation issuesarch/mips/kvm/mips.c arch/powerpc/kvm/powerpc.c arch/s390/kvm/kvm-s390.c +4 more+43 -191 reply
25/45KVM: s390: Move guts of kvm_arch_vcpu_init() into kvm_arch_vcpu_create()arch/s390/kvm/kvm-s390.c+34 -281 reply
26/45KVM: s390: Invoke kvm_vcpu_init() before allocating sie_pagearch/s390/kvm/kvm-s390.c+10 -81 reply
27/45KVM: MIPS: Invoke kvm_vcpu_uninit() immediately prior to freeing vcpuarch/mips/kvm/mips.c+2 -20 replies
28/45KVM: x86: Invoke kvm_vcpu_uninit() immediately prior to freeing vcpuarch/x86/kvm/x86.c+2 -20 replies
29/45KVM: Introduce kvm_vcpu_destroy()arch/mips/kvm/mips.c arch/powerpc/kvm/powerpc.c arch/s390/kvm/kvm-s390.c +4 more+12 -51 reply
30/45KVM: Move vcpu alloc and init invocation to common codearch/mips/kvm/mips.c arch/powerpc/kvm/powerpc.c arch/s390/kvm/kvm-s390.c +4 more+38 -1334 replies
31/45KVM: Unexport kvm_vcpu_cache and kvm_vcpu_{un}init()include/linux/kvm_host.h virt/kvm/kvm_main.c+3 -101 reply
32/45KVM: Move initialization of preempt notifier to kvm_vcpu_init()virt/kvm/kvm_main.c+1 -21 reply
33/45KVM: x86: Move guts of kvm_arch_vcpu_setup() into kvm_arch_vcpu_create()arch/x86/kvm/x86.c+10 -40 replies
34/45KVM: MIPS: Move .vcpu_setup() call to kvm_arch_vcpu_create()arch/mips/kvm/mips.c+8 -20 replies
35/45KVM: s390: Manually invoke vcpu setup during kvm_arch_vcpu_create()arch/s390/kvm/kvm-s390.c+11 -04 replies
36/45KVM: PPC: BookE: Setup vcpu during kvmppc_core_vcpu_create()arch/powerpc/kvm/booke.c+33 -270 replies
37/45KVM: Drop kvm_arch_vcpu_setup()arch/arm/kvm/guest.c arch/arm64/kvm/guest.c arch/mips/kvm/mips.c +6 more+0 -411 reply
38/45KVM: x86: Move all vcpu init code into kvm_arch_vcpu_create()arch/x86/kvm/x86.c+100 -980 replies
39/45KVM: MIPS: Move all vcpu init code into kvm_arch_vcpu_create()arch/mips/kvm/mips.c+14 -120 replies
40/45KVM: ARM: Move all vcpu init code into kvm_arch_vcpu_create()virt/kvm/arm/arm.c+20 -140 replies
41/45KVM: PPC: Move all vcpu init code into kvm_arch_vcpu_create()arch/powerpc/kvm/powerpc.c+22 -142 replies
42/45KVM: arm64: Free sve_state via arm specific hookarch/arm/include/asm/kvm_host.h arch/arm64/include/asm/kvm_host.h arch/arm64/kvm/reset.c +1 more+9 -00 replies
43/45KVM: Drop kvm_arch_vcpu_init() and kvm_arch_vcpu_uninit()arch/arm/include/asm/kvm_host.h arch/arm64/include/asm/kvm_host.h arch/arm64/kvm/reset.c +8 more+2 -651 reply
44/45KVM: Move putting of vcpu->pid to kvm_vcpu_destroy()virt/kvm/kvm_main.c+7 -61 reply
45/45KVM: Move vcpu->run page allocation out of kvm_vcpu_init()virt/kvm/kvm_main.c+13 -211 reply

Reviews trailers per patch

16 of 45 patches have a Reviewed-by or Acked-by.

PatchGreg KurzChristoffer DallCornelia HuckPaul Mackerras
1/45R-b A-b
2/45R-b A-b
3/45
4/45
5/45
6/45
7/45
8/45
9/45
10/45
11/45
12/45
13/45
14/45
15/45
16/45
17/45
18/45
19/45A-b
20/45
21/45
22/45
23/45A-b
24/45A-b R-b
25/45R-b
26/45R-b
27/45
28/45
29/45A-b R-b
30/45A-b R-b
31/45A-b R-b
32/45A-b R-b
33/45
34/45
35/45R-b
36/45
37/45A-b R-b
38/45
39/45
40/45
41/45
42/45
43/45A-b R-b
44/45A-b R-b
45/45R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v2 thread

Cover letter [PATCH v2 00/45] KVM: Refactor vCPU creation

The end goal of this series is to strip down the interface between common KVM code and arch specific code so that there is precisely one arch hook for creating a vCPU and one hook for destroying a vCPU. In addition to cleaning up the code base, simplifying the interface gives architectures more freedom to organize their vCPU creation code. Details below.

v2:
- Rebase to commit e41a90be9659 ("KVM: x86/mmu: WARN if root_hpa is invalid when handling a page fault"). A few minor x86 (VMX) conflicts, and one straightforward arm conflict with commit 8564d6372a7d ("KVM:
arm64: Support stolen time reporting via shared structure") in patch "KVM: ARM: Move all vcpu init code into kvm_arch_vcpu_create()".
- Collect Reviews and Acks [Christoffer, Greg].
- Fix a warning in "KVM: arm64: Free sve_state via arm specific hook" by using a "void" return type [Christoffer].
- Fix a bug in "KVM: PPC: Move kvm_vcpu_init() invocation to common code" where the call in kvmppc_core_vcpu_create_e500mc() was inadvertantly left behind.

KVM's vCPU creation code is comically messy. kvm_vm_ioctl_create_vcpu() calls three separate arch hooks: init(), create() and setup(). The init() call is especially nasty as it's hidden away in a common KVM function, kvm_init_vcpu(), that for all intents and purposes must be immediately invoked after the vcpu object is allocated.

Not to be outdone, vCPU destruction also has three arch hooks: uninit(), destroy() and free(), the latter of which isn't actually invoked by common KVM code, but the hook declaration still exists because architectures are relying on its forward declaration.

Eliminating the extra arch hooks is relatively straightforward, just tedious. For the most part, there is no fundamental constraint that necessitated the proliferation of arch hooks, rather they crept in over time, usually when x86-centric code was moved out of generic KVM and into x86 code.

E.g. kvm_arch_vcpu_setup() was added to allow x86 to do vcpu_load(), which can only be done after preempt_notifier initialization, but adding setup() overlooked the fact that the preempt_notifier was only initialized after kvm_arch_vcpu_create() because preemption support was added when x86's MMU setup (the vcpu_load() user) was called from common KVM code.

For all intents and purposes, there is no true functional change in this series. The order of some allocations will change, and a few memory leaks are fixed, but the actual functionality of a guest should be unaffected.

Patches 01-03 are bug fixes in error handling paths that were found by inspection when refactoring the associated code.

Patches 04-43 refactor each arch implementation so that the unwanted arch hooks can be dropped without a functional change, e.g. move code out of kvm_arch_vcpu_setup() so that all implementations are empty, then drop the functions and caller.

Patches 44-45 are minor clean up to eliminate kvm_vcpu_uninit().

The net result is to go from this:

        vcpu = kvm_arch_vcpu_create(kvm, id);
               |
               |-> kvm_vcpu_init()
                   |
                   |-> kvm_arch_vcpu_init()

        if (IS_ERR(vcpu)) {
                r = PTR_ERR(vcpu);
                goto vcpu_decrement;
        }

        preempt_notifier_init(&vcpu->preempt_notifier, &kvm_preempt_ops);

        r = kvm_arch_vcpu_setup(vcpu);
        if (r)
                goto vcpu_destroy;

Sean Christopherson (45):
KVM: PPC: Book3S HV: Uninit vCPU if vcore creation fails
KVM: PPC: Book3S PR: Free shared page if mmu initialization fails
KVM: x86: Free wbinvd_dirty_mask if vCPU creation fails
KVM: VMX: Allocate VPID after initializing VCPU
KVM: VMX: Use direct vcpu pointer during vCPU create/free
KVM: SVM: Use direct vcpu pointer during vCPU create/free
KVM: x86: Allocate vcpu struct in common x86 code
KVM: x86: Move FPU allocation to common x86 code
KVM: x86: Move allocation of pio_data page down a few lines
KVM: x86: Move kvm_vcpu_init() invocation to common code
KVM: PPC: e500mc: Add build-time assert that vcpu is at offset 0
KVM: PPC: Allocate vcpu struct in common PPC code
KVM: PPC: Book3S PR: Allocate book3s and shadow vcpu after common init
KVM: PPC: e500mc: Move reset of oldpir below call to kvm_vcpu_init()
KVM: PPC: Move kvm_vcpu_init() invocation to common code
KVM: MIPS: Use kvm_vcpu_cache to allocate vCPUs
KVM: MIPS: Drop kvm_arch_vcpu_free()
KVM: PPC: Drop kvm_arch_vcpu_free()
KVM: arm: Drop kvm_arch_vcpu_free()
KVM: x86: Remove spurious kvm_mmu_unload() from vcpu destruction path
KVM: x86: Remove spurious clearing of async #PF MSR
KVM: x86: Drop kvm_arch_vcpu_free()
KVM: Remove kvm_arch_vcpu_free() declaration
KVM: Add kvm_arch_vcpu_precreate() to handle pre-allocation issues
KVM: s390: Move guts of kvm_arch_vcpu_init() into
kvm_arch_vcpu_create()
KVM: s390: Invoke kvm_vcpu_init() before allocating sie_page
KVM: MIPS: Invoke kvm_vcpu_uninit() immediately prior to freeing vcpu
KVM: x86: Invoke kvm_vcpu_uninit() immediately prior to freeing vcpu
KVM: Introduce kvm_vcpu_destroy()
KVM: Move vcpu alloc and init invocation to common code
KVM: Unexport kvm_vcpu_cache and kvm_vcpu_{un}init()
KVM: Move initialization of preempt notifier to kvm_vcpu_init()
KVM: x86: Move guts of kvm_arch_vcpu_setup() into
kvm_arch_vcpu_create()
KVM: MIPS: Move .vcpu_setup() call to kvm_arch_vcpu_create()
KVM: s390: Manually invoke vcpu setup during kvm_arch_vcpu_create()
KVM: PPC: BookE: Setup vcpu during kvmppc_core_vcpu_create()
KVM: Drop kvm_arch_vcpu_setup()
KVM: x86: Move all vcpu init code into kvm_arch_vcpu_create()
KVM: MIPS: Move all vcpu init code into kvm_arch_vcpu_create()
KVM: ARM: Move all vcpu init code into kvm_arch_vcpu_create()
KVM: PPC: Move all vcpu init code into kvm_arch_vcpu_create()
KVM: arm64: Free sve_state via arm specific hook
KVM: Drop kvm_arch_vcpu_init() and kvm_arch_vcpu_uninit()
KVM: Move putting of vcpu->pid to kvm_vcpu_destroy()
KVM: Move vcpu->run page allocation out of kvm_vcpu_init()

 arch/arm/include/asm/kvm_host.h    |   2 +-
 arch/arm/kvm/guest.c               |   5 -
 arch/arm64/include/asm/kvm_host.h  |   2 +-
 arch/arm64/kvm/guest.c             |   5 -
 arch/arm64/kvm/reset.c             |   2 +-
 arch/mips/kvm/mips.c               |  84 ++++-------
 arch/powerpc/include/asm/kvm_ppc.h |   6 +-
 arch/powerpc/kvm/book3s.c          |   9 +-
 arch/powerpc/kvm/book3s_hv.c       |  27 +---
 arch/powerpc/kvm/book3s_pr.c       |  33 ++---
 arch/powerpc/kvm/booke.c           |  65 ++++----
 arch/powerpc/kvm/e500.c            |  34 +----
 arch/powerpc/kvm/e500mc.c          |  28 +---
 arch/powerpc/kvm/powerpc.c         |  70 ++++-----
 arch/s390/include/asm/kvm_host.h   |   1 -
 arch/s390/kvm/kvm-s390.c           | 110 +++++++-------
 arch/x86/include/asm/kvm_host.h    |   2 +-
 arch/x86/kvm/svm.c                 |  52 +------
 arch/x86/kvm/vmx/vmx.c             |  72 +++------
 arch/x86/kvm/x86.c                 | 230 ++++++++++++++---------------
 include/linux/kvm_host.h           |  13 +-
 virt/kvm/arm/arm.c                 |  80 ++++------
 virt/kvm/kvm_main.c                |  71 +++++----
 23 files changed, 385 insertions(+), 618 deletions(-)
Signature · 1 line
2.24.1