target/arm: Reduced-IPA space and highmem=off fixes
Marc Zyngier
maintainercover target/arm: Reduced-IPA space and highmem=off fixes
With the availability of a fruity range of arm64 systems, it becomes obvious that QEMU doesn't deal very well with limited IPA ranges when used as a front-end for KVM.
This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the highmem option
This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
Marc Zyngier (3):
hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
scratch VM
hw/arm/virt: Honor highmem setting when computing highest_gpa docs/system/arm/virt: Fix documentation for the 'highmem' option
docs/system/arm/virt.rst | 6 +++--- hw/arm/virt.c | 10 +++++++--- target/arm/kvm.c | 7 ++++++- 3 files changed, 16 insertions(+), 7 deletions(-)
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH 1/3 hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM
Although we probe for the IPA limits imposed by KVM (and the hardware) when computing the memory map, we still use the old style '0' when creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
On systems that are severely IPA challenged (such as the Apple M1), this results in a failure as KVM cannot use the default 40bit that '0' represents.
Instead, probe for the extension and use the reported IPA limit if available.
--- target/arm/kvm.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
target/arm/kvm.c +6 -1
@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try, struct kvm_vcpu_init *init) {int ret =0 , kvmfd =-1 , vmfd =-1 , cpufd =-1 ; +int max_vm_pa_size; kvmfd =qemu_open_old ("/dev/kvm" ,O_RDWR );if (kvmfd <0 ) { goto err; } - vmfd =ioctl (kvmfd,KVM_CREATE_VM ,0 ); + max_vm_pa_size =ioctl (kvmfd,KVM_CHECK_EXTENSION ,KVM_CAP_ARM_VM_IPA_SIZE ); +if (max_vm_pa_size <0 ) { + max_vm_pa_size =0 ; + } + vmfd =ioctl (kvmfd,KVM_CREATE_VM , max_vm_pa_size);if (vmfd <0 ) { goto err; }
Signature · 1 line
2.30.2
Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM
Quoted 45 lines · Marc Zyngier
Although we probe for the IPA limits imposed by KVM (and the hardware) when computing the memory map, we still use the old style '0' when creating a scratch VM in
kvm_arm_create_scratch_host_vcpu().On systems that are severely IPA challenged (such as the Apple M1), this results in a failure as KVM cannot use the default 40bit that '0' represents.
Instead, probe for the extension and use the reported IPA limit if available.
Cc Andrew Jones <[email protected]>Cc Eric Auger <[email protected]>Cc Peter Maydell <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- target/arm/kvm.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)target/arm/kvm.c +6
-1@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try, struct kvm_vcpu_init *init) {int ret =0 , kvmfd =-1 , vmfd =-1 , cpufd =-1 ; +int max_vm_pa_size; kvmfd =qemu_open_old ("/dev/kvm" ,O_RDWR );if (kvmfd <0 ) { goto err; } - vmfd =ioctl (kvmfd,KVM_CREATE_VM ,0 ); + max_vm_pa_size =ioctl (kvmfd,KVM_CHECK_EXTENSION ,KVM_CAP_ARM_VM_IPA_SIZE ); +if (max_vm_pa_size <0 ) { + max_vm_pa_size =0 ; + } + vmfd =ioctl (kvmfd,KVM_CREATE_VM , max_vm_pa_size);if (vmfd <0 ) { goto err; }Signature · 1 line
2.30.2
Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM
In reply to Marc ZyngierHi Marc,
Quoted 8 lines · Marc Zyngier
Although we probe for the IPA limits imposed by KVM (and the hardware) when computing the memory map, we still use the old style '0' when creating a scratch VM in
kvm_arm_create_scratch_host_vcpu().On systems that are severely IPA challenged (such as the Apple M1), this results in a failure as KVM cannot use the default 40bit that '0' represents.
Quoted 7 lines
Instead, probe for the extension and use the reported IPA limit if available.
Cc Andrew Jones <[email protected]>Cc Eric Auger <[email protected]>Cc Peter Maydell <[email protected]>Signed-off-by Marc Zyngier <[email protected]>
Eric
Quoted 27 lines
--- target/arm/kvm.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)target/arm/kvm.c +6
-1@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try, struct kvm_vcpu_init *init) {int ret =0 , kvmfd =-1 , vmfd =-1 , cpufd =-1 ; +int max_vm_pa_size; kvmfd =qemu_open_old ("/dev/kvm" ,O_RDWR );if (kvmfd <0 ) { goto err; } - vmfd =ioctl (kvmfd,KVM_CREATE_VM ,0 ); + max_vm_pa_size =ioctl (kvmfd,KVM_CHECK_EXTENSION ,KVM_CAP_ARM_VM_IPA_SIZE ); +if (max_vm_pa_size <0 ) { + max_vm_pa_size =0 ; + } + vmfd =ioctl (kvmfd,KVM_CREATE_VM , max_vm_pa_size);if (vmfd <0 ) { goto err; }
Marc Zyngier
maintainerPATCH 2/3 hw/arm/virt: Honor highmem setting when computing highest_gpa
In reply to Marc ZyngierEven when the VM is configured with highmem=off, the highest_gpa field includes devices that are above the 4GiB limit, which is what highmem=off is supposed to enforce. This leads to failures in virt_kvm_type() on systems that have a crippled IPA range, as the reported IPA space is larger than what it should be.
Instead, honor the user-specified limit to only use the devices at the lowest end of the spectrum.
Note that this doesn't affect memory, which is still allowed to go beyond 4GiB with highmem=on configurations.
--- hw/arm/virt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
hw/arm/virt.c +7 -3
@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) {MachineState *ms =MACHINE (vms); -hwaddr base, device_memory_base, device_memory_size; +hwaddr base, device_memory_base, device_memory_size, ceiling;int i; vms->memmap = extended_memmap;
@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms) device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;/* Base address of the high IO region */ - base = device_memory_base +ROUND_UP (device_memory_size, GiB); + ceiling = base = device_memory_base +ROUND_UP (device_memory_size, GiB);if (base < device_memory_base) {error_report ("maxmem/slots too huge" );exit (EXIT_FAILURE );
@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = base -1 ; +if (vms->highmem) { +/* If we have highmem, move the IPA limit to the top */ + ceiling = base; + } + vms->highest_gpa = ceiling -1 ;if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;
Signature · 1 line
2.30.2
Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa
Quoted 44 lines · Marc Zyngier
Even when the VM is configured with highmem=off, the
highest_gpafield includes devices that are above the 4GiB limit, which is what highmem=off is supposed to enforce. This leads to failures invirt_kvm_type()on systems that have a crippled IPA range, as the reported IPA space is larger than what it should be.Instead, honor the user-specified limit to only use the devices at the lowest end of the spectrum.
Note that this doesn't affect memory, which is still allowed to go beyond 4GiB with highmem=on configurations.
Cc Andrew Jones <[email protected]>Cc Eric Auger <[email protected]>Cc Peter Maydell <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- hw/arm/virt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)hw/arm/virt.c +2
-2@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) {MachineState *ms =MACHINE (vms); -hwaddr base, device_memory_base, device_memory_size; +hwaddr base, device_memory_base, device_memory_size, ceiling;int i; vms->memmap = extended_memmap;@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms) device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;/* Base address of the high IO region */ - base = device_memory_base +ROUND_UP (device_memory_size, GiB); + ceiling = base = device_memory_base +ROUND_UP (device_memory_size, GiB);if (base < device_memory_base) {error_report ("maxmem/slots too huge" );exit (EXIT_FAILURE );
hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1649
Quoted 9 lines
hw/arm/virt.c +5
-1@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = base -1 ; +if (vms->highmem) { +/* If we have highmem, move the IPA limit to the top */ + ceiling = base; + } + vms->highest_gpa = ceiling -1 ;
This doesn't look right to me. If highmem is false and the high IO region would be above the 4GB mark then we should not create the high IO region at all, surely? This code looks like it goes ahead and puts the high IO region above 4GB and then lies in the highest_gpa value about what the highest used GPA is.
-- PMM
Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa
Hi
Quoted 59 lines · Peter Maydell
Quoted 43 lines · Marc Zyngier
Even when the VM is configured with highmem=off, the
highest_gpafield includes devices that are above the 4GiB limit, which is what highmem=off is supposed to enforce. This leads to failures invirt_kvm_type()on systems that have a crippled IPA range, as the reported IPA space is larger than what it should be.Instead, honor the user-specified limit to only use the devices at the lowest end of the spectrum.
Note that this doesn't affect memory, which is still allowed to go beyond 4GiB with highmem=on configurations.
Cc Andrew Jones <[email protected]>Cc Eric Auger <[email protected]>Cc Peter Maydell <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- hw/arm/virt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)hw/arm/virt.c +2
-2@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) {MachineState *ms =MACHINE (vms); -hwaddr base, device_memory_base, device_memory_size; +hwaddr base, device_memory_base, device_memory_size, ceiling;int i; vms->memmap = extended_memmap;@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms) device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;/* Base address of the high IO region */ - base = device_memory_base +ROUND_UP (device_memory_size, GiB); + ceiling = base = device_memory_base +ROUND_UP (device_memory_size, GiB);if (base < device_memory_base) {error_report ("maxmem/slots too huge" );exit (EXIT_FAILURE );hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1649
Quoted 9 lines
hw/arm/virt.c +5
-1@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = base -1 ; +if (vms->highmem) { +/* If we have highmem, move the IPA limit to the top */ + ceiling = base; + } + vms->highest_gpa = ceiling -1 ;This doesn't look right to me. If highmem is false and the high IO region would be above the 4GB mark then we should not create the high IO region at all, surely? This code looks like it goes ahead and puts the high IO region above 4GB and then lies in the
highest_gpavalue about what the highest used GPA is.
Quoted 2 lines
-- PMM
Doesn't the problem come from "if maxram_size is < 255GiB we keep the legacy memory map" and set base = vms->memmap[VIRT_MEM].base + LEGACY_RAMLIMIT_BYTES; leading to IO regions allocated above? Instead shouldn't we condition this to highmem=on only then?
But by the way do we need to added extended_memmap IO regions at all if highmem=off?
I am not wrong the VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO only are used if highmem=on. In create_pcie(), base_mmio_high/size_mmio_high are used if vms->highmem and we have ecam_id = VIRT_ECAM_ID(vms->highmem_ecam); with vms->highmem_ecam &= vms->highmem && (!firmware_loaded || aarch64);
So if I do not miss anything maybe we could skip the allocation of the extended_memmap IO regions if highmem=off?
And doesn't it look reasonable to limit the number of vcpus if highmem=off?
Thanks
Eric
Marc Zyngier
maintainerPATCH 3/3 docs/system/arm/virt: Fix documentation for the 'highmem' option
In reply to Marc ZyngierThe documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
--- docs/system/arm/virt.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
docs/system/arm/virt.rst +3 -3
@@ -86,9 +86,9 @@ mte Arm Memory Tagging Extensions. The default is ``off``. highmem - Set ``on``/``off`` to enable/disable placing devices and RAM in physical - address space above 32 bits. The default is ``on`` for machine types - later than ``virt-2.12``. + Set ``on``/``off`` to enable/disable placing devices in physical address + space above 32 bits. RAM in excess of 3GiB will always be placed above + 32 bits. The default is ``on`` for machine types later than ``virt-2.12``. gic-version Specify the version of the Generic Interrupt Controller (GIC) to provide.
Signature · 1 line
2.30.2
Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option
Quoted 7 lines · Marc Zyngier
The documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
I think it would be better to align the behaviour with the documentation.
The intent of 'highmem' is to allow a configuration for use with guests that can't address more than 32 bits (originally, 32-bit guests without LPAE support compiled in). It seems like a bug that we allow the user to specify more RAM than will fit into that 32-bit range. We should instead make QEMU exit with an error if the user tries to specify both highmem=off and a memory size that's too big to fit.
thanks
-- PMM
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option
Hi Peter,
Quoted 10 lines · Peter Maydell
Quoted 7 lines · Marc Zyngier
The documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
Quoted 8 lines
I think it would be better to align the behaviour with the documentation.
The intent of 'highmem' is to allow a configuration for use with guests that can't address more than 32 bits (originally, 32-bit guests without LPAE support compiled in). It seems like a bug that we allow the user to specify more RAM than will fit into that 32-bit range. We should instead make QEMU exit with an error if the user tries to specify both highmem=off and a memory size that's too big to fit.
I'm happy to address this if you are OK with the change in user visible behaviour.
However, I am still struggling with my original goal, which is to allow QEMU to create a usable KVM_based VM on systems with a small IPA space (36 bits on the system I have). What would an acceptable way to convey this to the code that deals with the virt memory map so that it falls back to something that actually works?
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option
Quoted 24 lines · Marc Zyngier
Hi Peter,
Quoted 10 lines · Peter Maydell
Quoted 7 lines · Marc Zyngier
The documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
Quoted 8 lines
I think it would be better to align the behaviour with the documentation.
The intent of 'highmem' is to allow a configuration for use with guests that can't address more than 32 bits (originally, 32-bit guests without LPAE support compiled in). It seems like a bug that we allow the user to specify more RAM than will fit into that 32-bit range. We should instead make QEMU exit with an error if the user tries to specify both highmem=off and a memory size that's too big to fit.
Quoted 8 lines
I'm happy to address this if you are OK with the change in user visible behaviour.
However, I am still struggling with my original goal, which is to allow QEMU to create a usable
KVM_basedVM on systems with a small IPA space (36 bits on the system I have). What would an acceptable way to convey this to the code that deals with the virt memory map so that it falls back to something that actually works?
Hmm, so at the moment we can either do "fits in 32 bits" or "assumes at least 40 bits" but not 36 ?
thanks
-- PMM
Marc Zyngier
maintainerRe: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option
Quoted 29 lines · Peter Maydell
Quoted 27 lines · Marc Zyngier
Hi Peter,
Quoted 10 lines · Peter Maydell
Quoted 7 lines · Marc Zyngier
The documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
Quoted 8 lines
I think it would be better to align the behaviour with the documentation.
The intent of 'highmem' is to allow a configuration for use with guests that can't address more than 32 bits (originally, 32-bit guests without LPAE support compiled in). It seems like a bug that we allow the user to specify more RAM than will fit into that 32-bit range. We should instead make QEMU exit with an error if the user tries to specify both highmem=off and a memory size that's too big to fit.
I'm happy to address this if you are OK with the change in user visible behaviour.
Quoted 8 lines
Quoted 5 lines
However, I am still struggling with my original goal, which is to allow QEMU to create a usable
KVM_basedVM on systems with a small IPA space (36 bits on the system I have). What would an acceptable way to convey this to the code that deals with the virt memory map so that it falls back to something that actually works?Hmm, so at the moment we can either do "fits in 32 bits" or "assumes at least 40 bits" but not 36 ?
Exactly. I have the gut feeling that we need a 'gpa_bits' option that would limit the guest physical range and generalise highmem. High IO ranges would simply not be available if the GPA range isn't big enough.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option
In reply to Peter MaydellHi,
Quoted 7 lines · Peter Maydell
Quoted 6 lines · Marc Zyngier
The documentation for the 'highmem' option indicates that it controls the placement of both devices and RAM. The actual behaviour of QEMU seems to be that RAM is allowed to go beyond the 4GiB limit, and that only devices are constraint by this option.
Align the documentation with the actual behaviour.
Quoted 8 lines
I think it would be better to align the behaviour with the documentation.
The intent of 'highmem' is to allow a configuration for use with guests that can't address more than 32 bits (originally, 32-bit guests without LPAE support compiled in). It seems like a bug that we allow the user to specify more RAM than will fit into that 32-bit range. We should instead make QEMU exit with an error if the user tries to specify both highmem=off and a memory size that's too big to fit.
That's my opinion too
Thanks
Eric
Quoted 4 lines
thanks
-- PMM
Marc Zyngier
maintainerQuoted 4 lines · Marc Zyngier
With the availability of a fruity range of arm64 systems, it becomes obvious that QEMU doesn't deal very well with limited IPA ranges when used as a front-end for KVM.
Quoted 8 lines
This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the highmem optionThis has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
I realise I haven't been very clear in my description of the above. With this series, using 'highmem=off' results in a usable VM, while sticking to the default 'highmem=on' still generates an error.
M.
Signature · 5 lines
Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Peter Maydell
AppliedQuoted 12 lines · Marc Zyngier
With the availability of a fruity range of arm64 systems, it becomes obvious that QEMU doesn't deal very well with limited IPA ranges when used as a front-end for KVM.
This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the highmem option
Quoted 7 lines
This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
Marc Zyngier (3):
hw/arm/virt: KVM: Probe forKVM_CAP_ARM_VM_IPA_SIZEwhen creating
scratch VM
hw/arm/virt: Honor highmem setting when computinghighest_gpadocs/system/arm/virt: Fix documentation for the 'highmem' option
I've applied patch 1 to target-arm.next; thanks.
-- PMM