KVM: move struct kvm_vcpu * array to the bottom of struct kvm
PATCH 1/1 KVM: move struct kvm_vcpu * array to the bottom of struct kvm
Increasing the max VCPUs on powerpc makes the kvm_arch member offset great enough that some assembly breaks due to addressing constants overflowing field widths.
Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.
---
It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic, however x86 kvm_svm uses its own scheme rather than kvm_arch for some reason.
Thanks,
Nick
include/linux/kvm_host.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +3 -1
@@ -553,7 +553,6 @@ struct kvm {struct mutex slots_arch_lock;struct mm_struct *mm;/* userspace tied to this vm */ struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ]; -struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ];/* Used to wait for completion of MMU notifiers. */ spinlock_t mn_invalidate_lock;
@@ -623,6 +622,9 @@ struct kvm {struct notifier_block pm_notifier;#endif char stats_id[KVM_STATS_NAME_SIZE ]; + +/* This array can be very large, so keep it at the bottom */ +struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ]; };#define kvm_err (fmt, ...) \
Signature · 1 line
2.23.0
Sean Christopherson
reviewer+Juergen and Marc
Quoted 9 lines · Nicholas Piggin
Increasing the max VCPUs on powerpc makes the
kvm_archmember offset great enough that some assembly breaks due to addressing constants overflowing field widths.Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.
Quoted 5 lines
Signed-off-by Nicholas Piggin <[email protected]>---
It would next be possible to now make this a dynamically sized array, and make the
KVM_MAX_VCPUSmore dynamic
Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]
Quoted 1 line
however x86
kvm_svmuses its own scheme rather thankvm_archfor some reason.
What's the problem in kvm_svm?
Quoted 31 lines
Thanks,
Nickinclude/linux/kvm_host.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)include/linux/kvm_host.h +3
-1@@ -553,7 +553,6 @@ struct kvm {struct mutex slots_arch_lock;struct mm_struct *mm;/* userspace tied to this vm */ struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ]; -struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ];/* Used to wait for completion of MMU notifiers. */ spinlock_t mn_invalidate_lock;@@ -623,6 +622,9 @@ struct kvm {struct notifier_block pm_notifier;#endif char stats_id[KVM_STATS_NAME_SIZE ]; + +/* This array can be very large, so keep it at the bottom */ +struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ]; };#define kvm_err (fmt, ...) \Signature · 1 line
2.23.0
Quoted 16 lines · Sean Christopherson
+Juergen and Marc
Quoted 12 lines · Nicholas Piggin
Increasing the max VCPUs on powerpc makes the
kvm_archmember offset great enough that some assembly breaks due to addressing constants overflowing field widths.Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.
Signed-off-by Nicholas Piggin <[email protected]>---
Quoted 6 lines
Quoted 2 lines
It would next be possible to now make this a dynamically sized array, and make the
KVM_MAX_VCPUSmore dynamicMarc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?
I've just revived it, as it needed some RISC-V changes. I'll post it in a jiffy.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Excerpts from Sean Christopherson's message of November 6, 2021 12:49 am:
Quoted 18 lines
+Juergen and Marc
Quoted 9 lines · Nicholas Piggin
Increasing the max VCPUs on powerpc makes the
kvm_archmember offset great enough that some assembly breaks due to addressing constants overflowing field widths.Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.
Quoted 5 lines
Signed-off-by Nicholas Piggin <[email protected]>---
It would next be possible to now make this a dynamically sized array, and make the
KVM_MAX_VCPUSmore dynamic
Quoted 6 lines
Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]
Seems like a good idea if it can allow vcpu structs to be allocated to preferred nodes.
Quoted 3 lines
Quoted 1 line
however x86
kvm_svmuses its own scheme rather thankvm_archfor some reason.What's the problem in
kvm_svm?
It embeds a struct kvm so it couldn't be variable sized.
Thanks,
Nick
Sean Christopherson
reviewerQuoted 11 lines · Nicholas Piggin
Excerpts from Sean Christopherson's message of November 6, 2021 12:49 am:
Quoted 9 lines
Quoted 2 lines
It would next be possible to now make this a dynamically sized array, and make the
KVM_MAX_VCPUSmore dynamicMarc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]
Quoted 8 lines
Seems like a good idea if it can allow vcpu structs to be allocated to preferred nodes.
Quoted 3 lines
Quoted 1 line
however x86
kvm_svmuses its own scheme rather thankvm_archfor some reason.What's the problem in
kvm_svm?It embeds a struct kvm so it couldn't be variable sized.
Oooh, when you said "dynamically sized" I thought you meant
struct kvm_vcpu *vcpus;
...
kvm->vcpus = kcalloc(...);
Anyways, SVM and VMX are quite different despited both being x86, to keep them separated without requiring an extra allocation, kvm_svm and kvm_vmx embed kvm. Ditto for vcpu_vmx and vmx_svm. Obviously not a hard requirement, but there also hasn't been a reason not to do embed kvm/kvm_vcpu.