Skip to contentSkip to conversation

KVM: move struct kvm_vcpu * array to the bottom of struct kvm

PATCH In review Nicholas Piggin 3 participants 5 messages Nov 5, 2021 – Nov 8, 2021 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: move struct kvm_vcpu * array to the bottom of struct kvm

Increasing the max VCPUs on powerpc makes the kvm_arch member offset great enough that some assembly breaks due to addressing constants overflowing field widths.

Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.

---

It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic, however x86 kvm_svm uses its own scheme rather than kvm_arch for some reason.

Thanks,
Nick

 include/linux/kvm_host.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +3 -1
index 0f18df7fe874..78cd9b63a6a5 100644
@@ -553,7 +553,6 @@ struct kvm {
 	struct mutex slots_arch_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
-	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 
 	/* Used to wait for completion of MMU notifiers.  */
 	spinlock_t mn_invalidate_lock;
@@ -623,6 +622,9 @@ struct kvm {
 	struct notifier_block pm_notifier;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
+
+	/* This array can be very large, so keep it at the bottom */
+	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 };
 
 #define kvm_err(fmt, ...) \
Signature · 1 line
2.23.0
#1 lore raw

+Juergen and Marc

Quoted 9 lines · Nicholas Piggin

Increasing the max VCPUs on powerpc makes the kvm_arch member offset great enough that some assembly breaks due to addressing constants overflowing field widths.

Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.

Quoted 5 lines

---

It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic

Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]

Quoted 1 line

however x86 kvm_svm uses its own scheme rather than kvm_arch for some reason.

What's the problem in kvm_svm?

Quoted 31 lines

Thanks,
Nick

 include/linux/kvm_host.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +3 -1
index 0f18df7fe874..78cd9b63a6a5 100644
@@ -553,7 +553,6 @@ struct kvm {
 	struct mutex slots_arch_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
-	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 
 	/* Used to wait for completion of MMU notifiers.  */
 	spinlock_t mn_invalidate_lock;
@@ -623,6 +622,9 @@ struct kvm {
 	struct notifier_block pm_notifier;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
+
+	/* This array can be very large, so keep it at the bottom */
+	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 };
 
 #define kvm_err(fmt, ...) \
Signature · 1 line
2.23.0
#2 lore raw
Quoted 16 lines · Sean Christopherson

+Juergen and Marc

Quoted 12 lines · Nicholas Piggin

Increasing the max VCPUs on powerpc makes the kvm_arch member offset great enough that some assembly breaks due to addressing constants overflowing field widths.

Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.

---

Quoted 6 lines
Quoted 2 lines

It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic

Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?

I've just revived it, as it needed some RISC-V changes. I'll post it in a jiffy.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw
In reply to Sean Christopherson

Excerpts from Sean Christopherson's message of November 6, 2021 12:49 am:

Quoted 18 lines

+Juergen and Marc

Quoted 9 lines · Nicholas Piggin

Increasing the max VCPUs on powerpc makes the kvm_arch member offset great enough that some assembly breaks due to addressing constants overflowing field widths.

Moving the vcpus array to the end of struct kvm prevents this from happening. It has the side benefit that moving the large array out from the middle of the structure should help keep other commonly accessed fields in the same or adjacent cache lines.

Quoted 5 lines

---

It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic

Quoted 6 lines

Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]

Seems like a good idea if it can allow vcpu structs to be allocated to preferred nodes.

Quoted 3 lines
Quoted 1 line

however x86 kvm_svm uses its own scheme rather than kvm_arch for some reason.

What's the problem in kvm_svm?

It embeds a struct kvm so it couldn't be variable sized.

Thanks,
Nick

#4 lore raw
Quoted 11 lines · Nicholas Piggin

Excerpts from Sean Christopherson's message of November 6, 2021 12:49 am:

Quoted 9 lines
Quoted 2 lines

It would next be possible to now make this a dynamically sized array, and make the KVM_MAX_VCPUS more dynamic

Marc has a mostly-baked series to use an xarray[1][2] that AFAICT would be well received. That has my vote, assuming it can get into 5.16. Marc or Juergen, are either of you actively working on that?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/vcpu-xarray
[2] https://lkml.kernel.org/r/[email protected]

Quoted 8 lines

Seems like a good idea if it can allow vcpu structs to be allocated to preferred nodes.

Quoted 3 lines
Quoted 1 line

however x86 kvm_svm uses its own scheme rather than kvm_arch for some reason.

What's the problem in kvm_svm?

It embeds a struct kvm so it couldn't be variable sized.

Oooh, when you said "dynamically sized" I thought you meant

	struct kvm_vcpu *vcpus;

...

	kvm->vcpus = kcalloc(...);

Anyways, SVM and VMX are quite different despited both being x86, to keep them separated without requiring an extra allocation, kvm_svm and kvm_vmx embed kvm. Ditto for vcpu_vmx and vmx_svm. Obviously not a hard requirement, but there also hasn't been a reason not to do embed kvm/kvm_vcpu.