KVM: arm64: Cap default IPA size to the host's own size
Subsystems (4)
Marc Zyngier
maintainerPATCH 1/1 KVM: arm64: Cap default IPA size to the host's own size
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).
Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
--- Documentation/virt/kvm/api.rst | 13 +++++++------ arch/arm64/kvm/reset.c | 12 ++++++++---- 2 files changed, 15 insertions(+), 10 deletions(-)
Documentation/virt/kvm/api.rst +7 -6
@@ -158,12 +158,13 @@ flag KVM_VM_MIPS_VZ. On arm64, the physical address size for a VM (IPA Size limit) is limited -to 40bits by default. The limit can be configured if the host supports the -extension KVM_CAP_ARM_VM_IPA_SIZE. When supported, use +to 40bits by default, though capped to the host's limit. The VM's own +limit can be configured if the host supports the extension +KVM_CAP_ARM_VM_IPA_SIZE. When supported, use KVM_VM_TYPE_ARM_IPA_SIZE(IPA_Bits) to set the size in the machine type -identifier, where IPA_Bits is the maximum width of any physical -address used by the VM. The IPA_Bits is encoded in bits[7-0] of the -machine type identifier. +identifier, where IPA_Bits is the maximum width of any physical address +used by the VM. The IPA_Bits is encoded in bits[7-0] of the machine type +identifier. e.g, to configure a guest to use 48bit physical address size::
@@ -172,7 +173,7 @@ e.g, to configure a guest to use 48bit physical address size:: The requested size (IPA_Bits) must be: == ========================================================= - 0 Implies default size, 40bits (for backward compatibility) + 0 Implies default size, 40bits or less (for backward compatibility) N Implies N bits, where N is a positive integer such that, 32 <= N <= Host_IPA_Limit == =========================================================
arch/arm64/kvm/reset.c +8 -4
@@ -324,10 +324,9 @@ int kvm_set_ipa_limit(void) } kvm_ipa_limit =id_aa64mmfr0_parange_to_phys_shift (parange); -WARN (kvm_ipa_limit <KVM_PHYS_SHIFT , -"KVM IPA Size Limit (%d bits) is smaller than default size\n" , - kvm_ipa_limit); -kvm_info ("IPA Size Limit: %d bits\n" , kvm_ipa_limit); +kvm_info ("IPA Size Limit: %d bits%s\n" , kvm_ipa_limit, + ((kvm_ipa_limit <KVM_PHYS_SHIFT ) ? +" (Reduced IPA size, limited VM compatibility)" :"" ));return 0 ; }
@@ -356,6 +355,11 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type)return -EINVAL ; }else { phys_shift =KVM_PHYS_SHIFT ; +if (phys_shift > kvm_ipa_limit) { +pr_warn_once ("Userspace using unsupported default IPA limit, capping to %d bits\n" , + kvm_ipa_limit); + phys_shift = kvm_ipa_limit; + } } mmfr0 =read_sanitised_ftr_reg (SYS_ID_AA64MMFR0_EL1 );
Signature · 6 lines
2.30.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 28 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
Quoted 6 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Signed-off-by Marc Zyngier <[email protected]>---
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerHi Suzuki,
Quoted 29 lines · Suzuki Poulose
Quoted 29 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
Quoted 8 lines
Quoted 6 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Signed-off-by Marc Zyngier <[email protected]>---
Reviewed-by Suzuki K Poulose <[email protected]>
Thanks for that. Whilst I have your attention and given that you are responsible for most of the variable IPA stuff... ;-)
I think we have another issue around the handling of our IPA size. Let's say I create a VM with a 32bit IPA space. If I register a 2GB memslot at 0x8000000, I'm getting an error, which I think is bogus.
I came to the conclusion that kvm_arch_prepare_memory_region() is a bit overzealous when rejecting the memslot, and I used the following patchlet to address it.
Does this seem sensible to you?
M.
arch/arm64/kvm/mmu.c +1 -2
@@ -1312,8 +1312,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,* Prevent userspace from creating a memory region outside of the IPA * space addressable by the KVM guest IPA space. */ -if (memslot->base_gfn + memslot->npages >= - (kvm_phys_size (kvm) >>PAGE_SHIFT )) +if ((memslot->base_gfn + memslot->npages) > (kvm_phys_size (kvm) >>PAGE_SHIFT ))return -EFAULT ;mmap_read_lock (current->mm);
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Will Deacon
maintainerQuoted 20 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
Quoted 7 lines
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Is there a way for userspace to discover the default IPA size, or does it have to try setting values until it finds one that sticks?
Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerQuoted 26 lines · Will Deacon
Quoted 24 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
Quoted 6 lines
Quoted 3 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Is there a way for userspace to discover the default IPA size, or does it have to try setting values until it finds one that sticks?
Yes, since 233a7cb23531 ("kvm: arm64: Allow tuning the physical address size for VM").
The VMM can issue a KVM_CAP_ARM_VM_IPA_SIZE ioctl(), and get in return the maximum IPA size (I have a patch for kvmtool that does this).
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Will Deacon
maintainerQuoted 32 lines · Marc Zyngier
Quoted 30 lines · Will Deacon
Quoted 27 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Quoted 8 lines
Quoted 2 lines
Is there a way for userspace to discover the default IPA size, or does it have to try setting values until it finds one that sticks?
Yes, since
233a7cb23531("kvm: arm64: Allow tuning the physical address size for VM").The VMM can issue a
KVM_CAP_ARM_VM_IPA_SIZEioctl(), and get in return the maximum IPA size (I have a patch for kvmtool that does this).
Great, thanks -- that's exactly what I was thinking about when I asked the question!
Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Marc,
Quoted 17 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
Quoted 6 lines
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
I'm not sure deferring the misconfiguration error until memslot request time is better than just failing to create a VM. If userspace doesn't use KVM_CAP_ARM_VM_IPA_SIZE to determine the limit (which it hasn't been obliged to do) and it is able to successfully create a VM, then it will assume up to 40-bit IPAs are supported. Later, when it tries to add memslots and fails it may be confused, especially if that later is much, much later with memory hotplug.
Quoted 4 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Failing to create any VM which hasn't selected a valid IPA limit should be pretty convincing :-)
Thanks,
drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerHi Andrew,
Quoted 28 lines · Andrew Jones
Hi Marc,
Quoted 17 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
Quoted 6 lines
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
Quoted 8 lines
I'm not sure deferring the misconfiguration error until memslot request time is better than just failing to create a VM. If userspace doesn't use
KVM_CAP_ARM_VM_IPA_SIZEto determine the limit (which it hasn't been obliged to do) and it is able to successfully create a VM, then it will assume up to 40-bit IPAs are supported. Later, when it tries to add memslots and fails it may be confused, especially if that later is much, much later with memory hotplug.
That's a fair point. However, no existing userspace will work on these systems. Is that what we want to do? I don't care much, but having non-usable defaults feel a bit... odd. I do spit out a warning, but I agree this isn't great either.
Quoted 6 lines
Quoted 3 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Failing to create any VM which hasn't selected a valid IPA limit should be pretty convincing :-)
I'll make sure to redirect the reports your way! :D
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 41 lines · Marc Zyngier
Hi Andrew,
Quoted 28 lines · Andrew Jones
Hi Marc,
Quoted 17 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
Quoted 6 lines
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
Quoted 8 lines
I'm not sure deferring the misconfiguration error until memslot request time is better than just failing to create a VM. If userspace doesn't use
KVM_CAP_ARM_VM_IPA_SIZEto determine the limit (which it hasn't been obliged to do) and it is able to successfully create a VM, then it will assume up to 40-bit IPAs are supported. Later, when it tries to add memslots and fails it may be confused, especially if that later is much, much later with memory hotplug.
Quoted 4 lines
That's a fair point. However, no existing userspace will work on these systems. Is that what we want to do? I don't care much, but having non-usable defaults feel a bit... odd. I do spit out a warning, but I agree this isn't great either.
I can send patches for QEMU, KVM selftests, and maybe even rust-vmm. Can you point me to something about these systems I can reference in my postings? Or I can just reference this mail thread.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Failing to create any VM which hasn't selected a valid IPA limit should be pretty convincing :-)
I'll make sure to redirect the reports your way! :D
What's the current error message when this occurs? Is it good enough, or should we improve it to help provide people hints? Please don't change it to "Invalid IPA limit, please mail Andrew Jones" :-)
Thanks,
drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerQuoted 43 lines · Andrew Jones
Quoted 41 lines · Marc Zyngier
Hi Andrew,
Quoted 36 lines · Andrew Jones
Hi Marc,
Quoted 17 lines · Marc Zyngier
KVM/arm64 has forever used a 40bit default IPA space, partially due to its 32bit heritage (where the only choice is 40bit).
However, there are implementations in the wild that have a *cough* much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access if userspace dares to ask for the default IPA setting (which most VMMs do).Instead, cap the default IPA size to what the host can actually do, and spit out a one-off message on the console. The boot warning is turned into a more meaningfull message, and the new behaviour is also documented.
Although this is a userspace ABI change, it doesn't really change much for userspace:
Quoted 6 lines
- the guest couldn't run before this change, while it now has a chance to if the memory range fits the reduced IPA space
- a memory slot that was accepted because it did fit the default IPA space but didn't fit the HW constraints is now properly rejected
I'm not sure deferring the misconfiguration error until memslot request time is better than just failing to create a VM. If userspace doesn't use
KVM_CAP_ARM_VM_IPA_SIZEto determine the limit (which it hasn't been obliged to do) and it is able to successfully create a VM, then it will assume up to 40-bit IPAs are supported. Later, when it tries to add memslots and fails it may be confused, especially if that later is much, much later with memory hotplug.
Quoted 8 lines
Quoted 4 lines
That's a fair point. However, no existing userspace will work on these systems. Is that what we want to do? I don't care much, but having non-usable defaults feel a bit... odd. I do spit out a warning, but I agree this isn't great either.
I can send patches for QEMU, KVM selftests, and maybe even rust-vmm. Can you point me to something about these systems I can reference in my postings? Or I can just reference this mail thread.
The system of choice to see this is an Apple M1 box. Not supported in mainline yet, but things are progressing pretty quickly.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
The other thing that's left doing is to convince userspace to actually use the IPA space setting instead of relying on the antiquated default.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Failing to create any VM which hasn't selected a valid IPA limit should be pretty convincing :-)
I'll make sure to redirect the reports your way! :D
What's the current error message when this occurs? Is it good enough, or should we improve it to help provide people hints? Please don't change it to "Invalid IPA limit, please mail Andrew Jones" :-)
Well, that's part of the problem. Currently, you don't get a message, and the guest faults on its first memory access forever (level 0 translation fault), as the VTCR_EL2.T0SZ value is bogus.
I can change this patch to reject 40bit IPA when requested as a default with something saying "Userspace using unsupported default IPA limit, upgrade your VMM".
Now, there is another nit[1] which I just found with my kvmtool setup that computes the optimal IPA space for a given VM. And that one is even more problematic...
Thanks,
M.
[1] https://lore.kernel.org/r/[email protected]
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm