target/arm: Reduced-IPA space and highmem fixes
Marc Zyngier
maintainerv5 cover target/arm: Reduced-IPA space and highmem fixes
Here's yet another stab at enabling QEMU on systems with pathologically reduced IPA ranges such as the Apple M1 (previous version at [1]). Eventually, we're able to run a KVM guest with more than just 3GB of RAM on a system with a 36bit IPA space, and at most 123 vCPUs.
This also addresses some pathological QEMU behaviours, where the highmem property is used as a flag allowing exposure of devices that can't possibly fit in the PA space of the VM, resulting in a guest failure.
In the end, we generalise the notion of PA space when exposing individual devices in the expanded memory map, and treat highmem as another flavour of PA space restriction.
This series does a few things:
- introduce new attributes to control the enabling of the highmem GICv3 redistributors and the highmem PCIe MMIO range
- correctly cap the PA range with highmem is off
- generalise the highmem behaviour to any PA range
- disable each highmem device region that doesn't fit in the PA range
- cleanup uses of highmem outside of virt_set_memmap()
This has been tested on an M1-based Mac-mini running Linux v5.16-rc6 with both KVM and TCG.
* From v4: [1]
- Moved cpu_type_valid() check before we compute the memory map
- Drop useless MAX() when computing highest_gpa
- Fixed more deviations from the QEMU coding style
- Collected Eric's RBs, with thanks
[1]: https://lore.kernel.org/r/[email protected]
Marc Zyngier (6):
hw/arm/virt: Add a control for the the highmem PCIe MMIO hw/arm/virt: Add a control for the the highmem redistributors hw/arm/virt: Honor highmem setting when computing the memory map hw/arm/virt: Use the PA range to compute the memory map hw/arm/virt: Disable highmem devices that don't fit in the PA range hw/arm/virt: Drop superfluous checks against highmem
hw/arm/virt-acpi-build.c | 10 ++-- hw/arm/virt.c | 98 ++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 5 +- 3 files changed, 91 insertions(+), 22 deletions(-)
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH v5 1/6 hw/arm/virt: Add a control for the the highmem PCIe MMIO
Just like we can control the enablement of the highmem PCIe ECAM region using highmem_ecam, let's add a control for the highmem PCIe MMIO region.
Similarily to highmem_ecam, this region is disabled when highmem is off.
--- hw/arm/virt-acpi-build.c | 10 ++++------ hw/arm/virt.c | 7 +++++-- include/hw/arm/virt.h | 1 + 3 files changed, 10 insertions(+), 8 deletions(-)
hw/arm/virt-acpi-build.c +4 -6
@@ -158,10 +158,9 @@ static void acpi_dsdt_add_virtio(Aml *scope, }static void acpi_dsdt_add_pci (Aml *scope,const MemMapEntry *memmap, -uint32_t irq,bool use_highmem,bool highmem_ecam, -VirtMachineState *vms) +uint32_t irq,VirtMachineState *vms) { -int ecam_id =VIRT_ECAM_ID (highmem_ecam); +int ecam_id =VIRT_ECAM_ID (vms->highmem_ecam);struct GPEXConfig cfg = { .mmio32 = memmap[VIRT_PCIE_MMIO ], .pio = memmap[VIRT_PCIE_PIO ],
@@ -170,7 +169,7 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, .bus = vms->bus, }; -if (use_highmem) { +if (vms->highmem_mmio) { cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; }
@@ -869,8 +868,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)acpi_dsdt_add_fw_cfg (scope, &memmap[VIRT_FW_CFG ]);acpi_dsdt_add_virtio (scope, &memmap[VIRT_MMIO ], (irqmap[VIRT_MMIO ] +ARM_SPI_BASE ),NUM_VIRTIO_TRANSPORTS ); -acpi_dsdt_add_pci (scope, memmap, (irqmap[VIRT_PCIE ] +ARM_SPI_BASE ), - vms->highmem, vms->highmem_ecam, vms); +acpi_dsdt_add_pci (scope, memmap, irqmap[VIRT_PCIE ] +ARM_SPI_BASE , vms);if (vms->acpi_dev) { build_ged_aml(scope, "\\_SB."GED_DEVICE, HOTPLUG_HANDLER(vms->acpi_dev),
hw/arm/virt.c +5 -2
@@ -1412,7 +1412,7 @@ static void create_pcie(VirtMachineState *vms) mmio_reg, base_mmio, size_mmio);memory_region_add_subregion (get_system_memory (), base_mmio, mmio_alias); -if (vms->highmem) { +if (vms->highmem_mmio) {/* Map high MMIO space */ MemoryRegion *high_mmio_alias =g_new0 (MemoryRegion,1 );
@@ -1466,7 +1466,7 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_sized_cells (ms->fdt, nodename,"reg" ,2 , base_ecam,2 , size_ecam); -if (vms->highmem) { +if (vms->highmem_mmio) { qemu_fdt_setprop_sized_cells(ms->fdt, nodename, "ranges", 1, FDT_PCI_RANGE_IOPORT, 2, 0, 2, base_pio, 2, size_pio,
@@ -2105,6 +2105,8 @@ static void machvirt_init(MachineState *machine)virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem); + vms->highmem_mmio &= vms->highmem; +create_gic (vms, sysmem);virt_cpu_post_init (vms, sysmem);
@@ -2802,6 +2804,7 @@ static void virt_instance_init(Object *obj) vms->gic_version =VIRT_GIC_VERSION_NOSEL ; vms->highmem_ecam = !vmc->no_highmem_ecam; + vms->highmem_mmio = true;if (vmc->no_its) { vms->its = false;
include/hw/arm/virt.h +1 -0
@@ -143,6 +143,7 @@ struct VirtMachineState {bool secure;bool highmem;bool highmem_ecam; +bool highmem_mmio;bool its;bool tcg_its;bool virt;
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v5 1/6] hw/arm/virt: Add a control for the the highmem PCIe MMIO
Hi Marc,
Quoted 8 lines · Marc Zyngier
Just like we can control the enablement of the highmem PCIe ECAM region using
highmem_ecam, let's add a control for the highmem PCIe MMIO region.Similarily to
highmem_ecam, this region is disabled when highmem is off.Signed-off-by Marc Zyngier <[email protected]>
Eric
Quoted 93 lines
--- hw/arm/virt-acpi-build.c | 10 ++++------ hw/arm/virt.c | 7 +++++-- include/hw/arm/virt.h | 1 + 3 files changed, 10 insertions(+), 8 deletions(-)hw/arm/virt-acpi-build.c +4
-6@@ -158,10 +158,9 @@ static void acpi_dsdt_add_virtio(Aml *scope, }static void acpi_dsdt_add_pci (Aml *scope,const MemMapEntry *memmap, -uint32_t irq,bool use_highmem,bool highmem_ecam, -VirtMachineState *vms) +uint32_t irq,VirtMachineState *vms) { -int ecam_id =VIRT_ECAM_ID (highmem_ecam); +int ecam_id =VIRT_ECAM_ID (vms->highmem_ecam);struct GPEXConfig cfg = { .mmio32 = memmap[VIRT_PCIE_MMIO ], .pio = memmap[VIRT_PCIE_PIO ],@@ -170,7 +169,7 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, .bus = vms->bus, }; -if (use_highmem) { +if (vms->highmem_mmio) { cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO ]; }@@ -869,8 +868,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)acpi_dsdt_add_fw_cfg (scope, &memmap[VIRT_FW_CFG ]);acpi_dsdt_add_virtio (scope, &memmap[VIRT_MMIO ], (irqmap[VIRT_MMIO ] +ARM_SPI_BASE ),NUM_VIRTIO_TRANSPORTS ); -acpi_dsdt_add_pci (scope, memmap, (irqmap[VIRT_PCIE ] +ARM_SPI_BASE ), - vms->highmem, vms->highmem_ecam, vms); +acpi_dsdt_add_pci (scope, memmap, irqmap[VIRT_PCIE ] +ARM_SPI_BASE , vms);if (vms->acpi_dev) { build_ged_aml(scope, "\\_SB."GED_DEVICE, HOTPLUG_HANDLER(vms->acpi_dev),hw/arm/virt.c +5
-2@@ -1412,7 +1412,7 @@ static void create_pcie(VirtMachineState *vms) mmio_reg, base_mmio, size_mmio);memory_region_add_subregion (get_system_memory (), base_mmio, mmio_alias); -if (vms->highmem) { +if (vms->highmem_mmio) {/* Map high MMIO space */ MemoryRegion *high_mmio_alias =g_new0 (MemoryRegion,1 );@@ -1466,7 +1466,7 @@ static void create_pcie(VirtMachineState *vms)qemu_fdt_setprop_sized_cells (ms->fdt, nodename,"reg" ,2 , base_ecam,2 , size_ecam); -if (vms->highmem) { +if (vms->highmem_mmio) { qemu_fdt_setprop_sized_cells(ms->fdt, nodename, "ranges", 1, FDT_PCI_RANGE_IOPORT, 2, 0, 2, base_pio, 2, size_pio,@@ -2105,6 +2105,8 @@ static void machvirt_init(MachineState *machine)virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem); + vms->highmem_mmio &= vms->highmem; +create_gic (vms, sysmem);virt_cpu_post_init (vms, sysmem);@@ -2802,6 +2804,7 @@ static void virt_instance_init(Object *obj) vms->gic_version =VIRT_GIC_VERSION_NOSEL ; vms->highmem_ecam = !vmc->no_highmem_ecam; + vms->highmem_mmio = true;if (vmc->no_its) { vms->its = false;include/hw/arm/virt.h +1
-0@@ -143,6 +143,7 @@ struct VirtMachineState {bool secure;bool highmem;bool highmem_ecam; +bool highmem_mmio;bool its;bool tcg_its;bool virt;
Marc Zyngier
maintainerPATCH v5 2/6 hw/arm/virt: Add a control for the the highmem redistributors
In reply to Marc ZyngierJust like we can control the enablement of the highmem PCIe region using highmem_ecam, let's add a control for the highmem GICv3 redistributor region.
Similarily to highmem_ecam, these redistributors are disabled when highmem is off.
--- hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ include/hw/arm/virt.h | 4 +++- 3 files changed, 7 insertions(+), 1 deletion(-)
hw/arm/virt-acpi-build.c +2 -0
@@ -947,6 +947,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)acpi_add_table (table_offsets, tables_blob);build_fadt_rev5 (tables_blob, tables->linker, vms, dsdt); + vms->highmem_redists &= vms->highmem; +acpi_add_table (table_offsets, tables_blob);build_madt (tables_blob, tables->linker, vms);
hw/arm/virt.c +2 -0
@@ -2106,6 +2106,7 @@ static void machvirt_init(MachineState *machine)virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem); vms->highmem_mmio &= vms->highmem; + vms->highmem_redists &= vms->highmem;create_gic (vms, sysmem);
@@ -2805,6 +2806,7 @@ static void virt_instance_init(Object *obj) vms->highmem_ecam = !vmc->no_highmem_ecam; vms->highmem_mmio = true; + vms->highmem_redists = true;if (vmc->no_its) { vms->its = false;
include/hw/arm/virt.h +3 -1
@@ -144,6 +144,7 @@ struct VirtMachineState {bool highmem;bool highmem_ecam;bool highmem_mmio; +bool highmem_redists;bool its;bool tcg_its;bool virt;
@@ -190,7 +191,8 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms)assert (vms->gic_version ==VIRT_GIC_VERSION_3 ); - return MACHINE(vms)->smp.cpus > redist0_capacity ? 2 : 1; +return (MACHINE (vms)->smp.cpus > redist0_capacity && + vms->highmem_redists) ? 2 : 1; }#endif /* QEMU_ARM_VIRT_H */
Signature · 1 line
2.30.2
Re: [PATCH v5 2/6] hw/arm/virt: Add a control for the the highmem redistributors
Hi Marc,
Quoted 9 lines · Marc Zyngier
Just like we can control the enablement of the highmem PCIe region using
highmem_ecam, let's add a control for the highmem GICv3 redistributor region.Similarily to
highmem_ecam, these redistributors are disabled when highmem is off.Reviewed-by Andrew Jones <[email protected]>Signed-off-by Marc Zyngier <[email protected]>
Eric
Quoted 61 lines
--- hw/arm/virt-acpi-build.c | 2 ++ hw/arm/virt.c | 2 ++ include/hw/arm/virt.h | 4 +++- 3 files changed, 7 insertions(+), 1 deletion(-)hw/arm/virt-acpi-build.c +2
-0@@ -947,6 +947,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)acpi_add_table (table_offsets, tables_blob);build_fadt_rev5 (tables_blob, tables->linker, vms, dsdt); + vms->highmem_redists &= vms->highmem; +acpi_add_table (table_offsets, tables_blob);build_madt (tables_blob, tables->linker, vms);hw/arm/virt.c +2
-0@@ -2106,6 +2106,7 @@ static void machvirt_init(MachineState *machine)virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem); vms->highmem_mmio &= vms->highmem; + vms->highmem_redists &= vms->highmem;create_gic (vms, sysmem);@@ -2805,6 +2806,7 @@ static void virt_instance_init(Object *obj) vms->highmem_ecam = !vmc->no_highmem_ecam; vms->highmem_mmio = true; + vms->highmem_redists = true;if (vmc->no_its) { vms->its = false;include/hw/arm/virt.h +3
-1@@ -144,6 +144,7 @@ struct VirtMachineState {bool highmem;bool highmem_ecam;bool highmem_mmio; +bool highmem_redists;bool its;bool tcg_its;bool virt;@@ -190,7 +191,8 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms)assert (vms->gic_version ==VIRT_GIC_VERSION_3 ); - return MACHINE(vms)->smp.cpus > redist0_capacity ? 2 : 1; +return (MACHINE (vms)->smp.cpus > redist0_capacity && + vms->highmem_redists) ? 2 : 1; }#endif /* QEMU_ARM_VIRT_H */
Marc Zyngier
maintainerPATCH v5 3/6 hw/arm/virt: Honor highmem setting when computing the memory map
In reply to Marc ZyngierEven when the VM is configured with highmem=off, the highest_gpa field includes devices that are above the 4GiB limit. Similarily, nothing seem to check that the memory is within the limit set by the highmem=off option.
This leads to failures in virt_kvm_type() on systems that have a crippled IPA range, as the reported IPA space is larger than what it should be.
Instead, honor the user-specified limit to only use the devices at the lowest end of the spectrum, and fail if we have memory crossing the 4GiB limit.
--- hw/arm/virt.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
hw/arm/virt.c +7 -3
@@ -1663,7 +1663,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) {MachineState *ms =MACHINE (vms); -hwaddr base, device_memory_base, device_memory_size; +hwaddr base, device_memory_base, device_memory_size, memtop;int i; vms->memmap = extended_memmap;
@@ -1690,7 +1690,11 @@ static void virt_set_memmap(VirtMachineState *vms) device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;/* Base address of the high IO region */ - base = device_memory_base +ROUND_UP (device_memory_size, GiB); + memtop = base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (!vms->highmem && memtop >4 * GiB) { +error_report ("highmem=off, but memory crosses the 4GiB limit\n" ); +exit (EXIT_FAILURE ); + }if (base < device_memory_base) {error_report ("maxmem/slots too huge" );exit (EXIT_FAILURE );
@@ -1707,7 +1711,7 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = base -1 ; + vms->highest_gpa = (vms->highmem ? base : memtop) -1 ;if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v5 4/6 hw/arm/virt: Use the PA range to compute the memory map
In reply to Marc ZyngierThe highmem attribute is nothing but another way to express the PA range of a VM. To support HW that has a smaller PA range then what QEMU assumes, pass this PA range to the virt_set_memmap() function, allowing it to correctly exclude highmem devices if they are outside of the PA range.
--- hw/arm/virt.c | 64 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 12 deletions(-)
hw/arm/virt.c +52 -12
@@ -1660,7 +1660,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } -static void virt_set_memmap (VirtMachineState *vms) +static void virt_set_memmap (VirtMachineState *vms,int pa_bits) {MachineState *ms =MACHINE (vms);hwaddr base, device_memory_base, device_memory_size, memtop;
@@ -1678,6 +1678,14 @@ static void virt_set_memmap(VirtMachineState *vms)exit (EXIT_FAILURE ); } +/* +* !highmem is exactly the same as limiting the PA space to 32bit, +* irrespective of the underlying capabilities of the HW. +*/ +if (!vms->highmem) { + pa_bits =32 ; + } +/* * We compute the base of the high IO region depending on the * amount of initial and device memory. The device memory start/size
@@ -1691,8 +1699,9 @@ static void virt_set_memmap(VirtMachineState *vms)/* Base address of the high IO region */ memtop = base = device_memory_base +ROUND_UP (device_memory_size, GiB); -if (!vms->highmem && memtop >4 * GiB) { -error_report ("highmem=off, but memory crosses the 4GiB limit\n" ); +if (memtop >BIT_ULL (pa_bits)) { +error_report ("Addressing limited to %d bits, but memory exceeds it by %llu bytes\n" , + pa_bits, memtop -BIT_ULL (pa_bits));exit (EXIT_FAILURE ); }if (base < device_memory_base) {
@@ -1711,7 +1720,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = (vms->highmem ? base : memtop) -1 ; + +/* +* If base fits within pa_bits, all good. If it doesn't, limit it +* to the end of RAM, which is guaranteed to fit within pa_bits. +*/ + vms->highest_gpa = (base <=BIT_ULL (pa_bits) ? base : memtop) -1 ; +if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;
@@ -1902,12 +1917,43 @@ static void machvirt_init(MachineState *machine)unsigned int smp_cpus = machine->smp.cpus;unsigned int max_cpus = machine->smp.max_cpus; +if (!cpu_type_valid (machine->cpu_type)) { +error_report ("mach-virt: CPU type %s not supported" , machine->cpu_type); +exit (1 ); + } + + possible_cpus = mc->possible_cpu_arch_ids (machine); +/* * In accelerated mode, the memory map is computed earlier in kvm_type() * to create a VM with the right number of IPA bits. */ if (!vms->memmap) { -virt_set_memmap (vms); +Object *cpuobj; +ARMCPU *armcpu; +int pa_bits; + +/* +* Instanciate a temporary CPU object to find out about what +* we are about to deal with. Once this is done, get rid of +* the object. +*/ + cpuobj =object_new (possible_cpus->cpus[0 ].type); + armcpu =ARM_CPU (cpuobj); + +if (object_property_get_bool (cpuobj,"aarch64" ,NULL )) { + pa_bits =arm_pamax (armcpu); + }else if (arm_feature (&armcpu->env,ARM_FEATURE_LPAE )) { +/* v7 with LPAE */ + pa_bits =40 ; + }else { +/* Anything else */ + pa_bits =32 ; + } + +object_unref (cpuobj); + +virt_set_memmap (vms, pa_bits); }/* We can probe only here because during property set
@@ -1915,11 +1961,6 @@ static void machvirt_init(MachineState *machine)*/ finalize_gic_version (vms); -if (!cpu_type_valid (machine->cpu_type)) { -error_report ("mach-virt: CPU type %s not supported" , machine->cpu_type); -exit (1 ); - } -if (vms->secure) {/* * The Secure view of the world is the same as the NonSecure,
@@ -1989,7 +2030,6 @@ static void machvirt_init(MachineState *machine)create_fdt (vms); - possible_cpus = mc->possible_cpu_arch_ids (machine);assert (possible_cpus->len == max_cpus);for (n =0 ; n < possible_cpus->len; n++) {Object *cpuobj;
@@ -2646,7 +2686,7 @@ static int virt_kvm_type(MachineState *ms, const char *type_str) max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms, &fixed_ipa);/* we freeze the memory map to compute the highest gpa */ -virt_set_memmap (vms); +virt_set_memmap (vms, max_vm_pa_size); requested_pa_size =64 -clz64 (vms->highest_gpa);
Signature · 1 line
2.30.2
Re: [PATCH v5 4/6] hw/arm/virt: Use the PA range to compute the memory map
Quoted 7 lines · Marc Zyngier
The highmem attribute is nothing but another way to express the PA range of a VM. To support HW that has a smaller PA range then what QEMU assumes, pass this PA range to the
virt_set_memmap()function, allowing it to correctly exclude highmem devices if they are outside of the PA range.Signed-off-by Marc Zyngier <[email protected]>
Thanks
Eric
Quoted 133 lines
--- hw/arm/virt.c | 64 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 12 deletions(-)hw/arm/virt.c +52
-12@@ -1660,7 +1660,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } -static void virt_set_memmap (VirtMachineState *vms) +static void virt_set_memmap (VirtMachineState *vms,int pa_bits) {MachineState *ms =MACHINE (vms);hwaddr base, device_memory_base, device_memory_size, memtop;@@ -1678,6 +1678,14 @@ static void virt_set_memmap(VirtMachineState *vms)exit (EXIT_FAILURE ); } +/* +* !highmem is exactly the same as limiting the PA space to 32bit, +* irrespective of the underlying capabilities of the HW. +*/ +if (!vms->highmem) { + pa_bits =32 ; + } +/* * We compute the base of the high IO region depending on the * amount of initial and device memory. The device memory start/size @@ -1691,8 +1699,9 @@ static void virt_set_memmap(VirtMachineState *vms)/* Base address of the high IO region */ memtop = base = device_memory_base +ROUND_UP (device_memory_size, GiB); -if (!vms->highmem && memtop >4 * GiB) { -error_report ("highmem=off, but memory crosses the 4GiB limit\n" ); +if (memtop >BIT_ULL (pa_bits)) { +error_report ("Addressing limited to %d bits, but memory exceeds it by %llu bytes\n" , + pa_bits, memtop -BIT_ULL (pa_bits));exit (EXIT_FAILURE ); }if (base < device_memory_base) {@@ -1711,7 +1720,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } - vms->highest_gpa = (vms->highmem ? base : memtop) -1 ; + +/* +* If base fits within pa_bits, all good. If it doesn't, limit it +* to the end of RAM, which is guaranteed to fit within pa_bits. +*/ + vms->highest_gpa = (base <=BIT_ULL (pa_bits) ? base : memtop) -1 ; +if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;@@ -1902,12 +1917,43 @@ static void machvirt_init(MachineState *machine)unsigned int smp_cpus = machine->smp.cpus;unsigned int max_cpus = machine->smp.max_cpus; +if (!cpu_type_valid (machine->cpu_type)) { +error_report ("mach-virt: CPU type %s not supported" , machine->cpu_type); +exit (1 ); + } + + possible_cpus = mc->possible_cpu_arch_ids (machine); +/* * In accelerated mode, the memory map is computed earlier in kvm_type() * to create a VM with the right number of IPA bits. */ if (!vms->memmap) { -virt_set_memmap (vms); +Object *cpuobj; +ARMCPU *armcpu; +int pa_bits; + +/* +* Instanciate a temporary CPU object to find out about what +* we are about to deal with. Once this is done, get rid of +* the object. +*/ + cpuobj =object_new (possible_cpus->cpus[0 ].type); + armcpu =ARM_CPU (cpuobj); + +if (object_property_get_bool (cpuobj,"aarch64" ,NULL )) { + pa_bits =arm_pamax (armcpu); + }else if (arm_feature (&armcpu->env,ARM_FEATURE_LPAE )) { +/* v7 with LPAE */ + pa_bits =40 ; + }else { +/* Anything else */ + pa_bits =32 ; + } + +object_unref (cpuobj); + +virt_set_memmap (vms, pa_bits); }/* We can probe only here because during property set @@ -1915,11 +1961,6 @@ static void machvirt_init(MachineState *machine)*/ finalize_gic_version (vms); -if (!cpu_type_valid (machine->cpu_type)) { -error_report ("mach-virt: CPU type %s not supported" , machine->cpu_type); -exit (1 ); - } -if (vms->secure) {/* * The Secure view of the world is the same as the NonSecure, @@ -1989,7 +2030,6 @@ static void machvirt_init(MachineState *machine)create_fdt (vms); - possible_cpus = mc->possible_cpu_arch_ids (machine);assert (possible_cpus->len == max_cpus);for (n =0 ; n < possible_cpus->len; n++) {Object *cpuobj;@@ -2646,7 +2686,7 @@ static int virt_kvm_type(MachineState *ms, const char *type_str) max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms, &fixed_ipa);/* we freeze the memory map to compute the highest gpa */ -virt_set_memmap (vms); +virt_set_memmap (vms, max_vm_pa_size); requested_pa_size =64 -clz64 (vms->highest_gpa);
Marc Zyngier
maintainerPATCH v5 5/6 hw/arm/virt: Disable highmem devices that don't fit in the PA range
In reply to Marc ZyngierIn order to only keep the highmem devices that actually fit in the PA range, check their location against the range and update highest_gpa if they fit. If they don't, mark them as disabled.
--- hw/arm/virt.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-)
hw/arm/virt.c +28 -6
@@ -1712,21 +1712,43 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits) base = vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ; } +/* We know for sure that at least the memory fits in the PA space */ + vms->highest_gpa = memtop -1 ; +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {hwaddr size = extended_memmap[i].size; +bool fits; base =ROUND_UP (base, size); vms->memmap[i].base = base; vms->memmap[i].size = size; + +/* +* Check each device to see if they fit in the PA space, +* moving highest_gpa as we go. +* +* For each device that doesn't fit, disable it. +*/ + fits = (base + size) <=BIT_ULL (pa_bits); +if (fits) { + vms->highest_gpa = base + size -1 ; + } + +switch (i) { +case VIRT_HIGH_GIC_REDIST2 : + vms->highmem_redists &= fits; +break ; +case VIRT_HIGH_PCIE_ECAM : + vms->highmem_ecam &= fits; +break ; +case VIRT_HIGH_PCIE_MMIO : + vms->highmem_mmio &= fits; +break ; + } + base += size; } -/* -* If base fits within pa_bits, all good. If it doesn't, limit it -* to the end of RAM, which is guaranteed to fit within pa_bits. -*/ - vms->highest_gpa = (base <=BIT_ULL (pa_bits) ? base : memtop) -1 ; -if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;
Signature · 1 line
2.30.2
Re: [PATCH v5 5/6] hw/arm/virt: Disable highmem devices that don't fit in the PA range
Hi Marc,
Quoted 5 lines · Marc Zyngier
In order to only keep the highmem devices that actually fit in the PA range, check their location against the range and update
highest_gpaif they fit. If they don't, mark them as disabled.Signed-off-by Marc Zyngier <[email protected]>
Eric
Quoted 58 lines
--- hw/arm/virt.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-)hw/arm/virt.c +28
-6@@ -1712,21 +1712,43 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits) base = vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ; } +/* We know for sure that at least the memory fits in the PA space */ + vms->highest_gpa = memtop -1 ; +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {hwaddr size = extended_memmap[i].size; +bool fits; base =ROUND_UP (base, size); vms->memmap[i].base = base; vms->memmap[i].size = size; + +/* +* Check each device to see if they fit in the PA space, +* moving highest_gpa as we go. +* +* For each device that doesn't fit, disable it. +*/ + fits = (base + size) <=BIT_ULL (pa_bits); +if (fits) { + vms->highest_gpa = base + size -1 ; + } + +switch (i) { +case VIRT_HIGH_GIC_REDIST2 : + vms->highmem_redists &= fits; +break ; +case VIRT_HIGH_PCIE_ECAM : + vms->highmem_ecam &= fits; +break ; +case VIRT_HIGH_PCIE_MMIO : + vms->highmem_mmio &= fits; +break ; + } + base += size; } -/* -* If base fits within pa_bits, all good. If it doesn't, limit it -* to the end of RAM, which is guaranteed to fit within pa_bits. -*/ - vms->highest_gpa = (base <=BIT_ULL (pa_bits) ? base : memtop) -1 ; -if (device_memory_size >0 ) { ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); ms->device_memory->base = device_memory_base;
Marc Zyngier
maintainerPATCH v5 6/6 hw/arm/virt: Drop superfluous checks against highmem
In reply to Marc ZyngierNow that the devices present in the extended memory map are checked against the available PA space and disabled when they don't fit, there is no need to keep the same checks against highmem, as highmem really is a shortcut for the PA space being 32bit.
--- hw/arm/virt-acpi-build.c | 2 -- hw/arm/virt.c | 5 +---- 2 files changed, 1 insertion(+), 6 deletions(-)
hw/arm/virt-acpi-build.c +0 -2
@@ -947,8 +947,6 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)acpi_add_table (table_offsets, tables_blob);build_fadt_rev5 (tables_blob, tables->linker, vms, dsdt); - vms->highmem_redists &= vms->highmem; -acpi_add_table (table_offsets, tables_blob);build_madt (tables_blob, tables->linker, vms);
hw/arm/virt.c +1 -4
@@ -2171,9 +2171,6 @@ static void machvirt_init(MachineState *machine)virt_flash_fdt (vms, sysmem, secure_sysmem ?: sysmem); - vms->highmem_mmio &= vms->highmem; - vms->highmem_redists &= vms->highmem; -create_gic (vms, sysmem);virt_cpu_post_init (vms, sysmem);
@@ -2192,7 +2189,7 @@ static void machvirt_init(MachineState *machine) machine->ram_size, "mach-virt.tag"); } - vms->highmem_ecam &= vms->highmem && (!firmware_loaded || aarch64); + vms->highmem_ecam &= (!firmware_loaded || aarch64);create_rtc (vms);
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Peter Maydell
AppliedQuoted 7 lines · Marc Zyngier
Here's yet another stab at enabling QEMU on systems with pathologically reduced IPA ranges such as the Apple M1 (previous version at [1]). Eventually, we're able to run a KVM guest with more than just 3GB of RAM on a system with a 36bit IPA space, and at most 123 vCPUs.
Quoted 8 lines
This also addresses some pathological QEMU behaviours, where the highmem property is used as a flag allowing exposure of devices that can't possibly fit in the PA space of the VM, resulting in a guest failure.
In the end, we generalise the notion of PA space when exposing individual devices in the expanded memory map, and treat highmem as another flavour of PA space restriction.
Applied to target-arm.next, thanks.
-- PMM