ARM virt: Initial RAM expansion and extended memory map
v11 cover ARM virt: Initial RAM expansion and extended memory map
This series aims to bump the 255GB RAM limit in machvirt and paves the way to device memory addition.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size ram_size and then we provision sufficient space for the looming device memory (dimensionned using ,maxmem and slots options).
IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on ram_size, slots, maxram_size. Size alignment is enforced.
In case maxmem value is inferior to 255GB, the legacy memory map still is used.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it will be possible to get access to this feature while keeping a 1TB address map.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-extended-memmap-v11
History:
v10 -> v11:
- remove vms->high_io_base
- remove RAM_BASE
- see individual change logs for other adjustments
v9 -> v10:
- only the keep the first 10 patches which move to the extended memmap
- remove the 1GB alignment check on maxram_size
v8 -> v9:
- fixed make check SIGSEV
- fixed TCG issue (restored virt_set_memmap in machvirt_init)
- removed Igor's R-b on "hw/arm/boot: Expose the PC-DIMM nodes in the DT"
v7 -> v8:
- addressed Igor's comments"
- remove no_extended_memmap knob
- moved acpi_nvdimm_state into MachineState
- separate "hw/arm/virt: Check the VCPU PA range in TCG mode" patch
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more details
v5 -> v6:
- mingw compilation issue fix
- kvm_arm_get_max_vm_phys_shift always returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization into machvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their size
v4 -> v5:
- change in the memory map
- see individual logs
v3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm: pre_plug "slot" and "addr" assignment"
- kvm-type option not used anymore. We directly use maxram_size and ram_size machine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
- device_memory_start and device_memory_size not stored anymore in vms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring of arm_load_dtb and the one related to the generation of the dtb after system_reset which would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machine ram_size, maxram_size and ram_slots earlier"
v2 -> v3:
- fix pc_q35 and pc_piix compilation error
- kwangwoo's email being not valid anymore, remove his address
v1 -> v2:
- kvm_get_max_vm_phys_shift moved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoring
v1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"
Best Regards
Eric
Eric Auger (9):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter to kvm_type callback
kvm: add kvm_arm_get_max_vm_ipa_size
vl: Set machine ram_size, maxram_size and ram_slots earlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implement kvm_type function for 4.0 machine hw/arm/virt: Check the VCPU PA range in TCG mode hw/arm/virt: Bump the 255GB initial RAM limit
Shameer Kolothum (1):
hw/arm/boot: introduce fdt_add_memory_node helper
accel/kvm/kvm-all.c | 2 +- hw/arm/boot.c | 54 +++++++---- hw/arm/virt-acpi-build.c | 10 +- hw/arm/virt.c | 197 +++++++++++++++++++++++++++++++-------- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 16 +++- include/hw/boards.h | 5 +- target/arm/kvm.c | 10 ++ target/arm/kvm_arm.h | 13 +++ vl.c | 6 +- 12 files changed, 242 insertions(+), 78 deletions(-)
Signature · 1 line
2.20.1
PATCH v11 1/10 hw/arm/boot: introduce fdt_add_memory_node helper
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
---
v7 -> v8:
- Added Igor's R-b
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID
--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +34 -20
@@ -423,6 +423,32 @@ static void set_kernel_args_old(const struct arm_boot_info *info, } } +static int fdt_add_memory_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -502,7 +528,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,void *fdt =NULL ;int size, rc, n =0 ;uint32_t acells, scells; -char *nodename;unsigned int i;hwaddr mem_base, mem_len;char **node_path;
@@ -576,35 +601,24 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start;for (i =0 ; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; - nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, mem_base, - scells, mem_len); + rc =fdt_add_memory_node (fdt, acells, mem_base, + scells, mem_len, i);if (rc <0 ) { -fprintf (stderr,"couldn't set %s/reg for node %d\n" , nodename, - i); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + mem_base); goto fail; } -qemu_fdt_setprop_cell (fdt, nodename,"numa-node-id" , i); mem_base += mem_len; -g_free (nodename); } }else { - nodename =g_strdup_printf ("/memory@%" PRIx64, binfo->loader_start); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, binfo->loader_start, - scells, binfo->ram_size); + rc =fdt_add_memory_node (fdt, acells, binfo->loader_start, + scells, binfo->ram_size,-1 );if (rc <0 ) { -fprintf (stderr,"couldn't set %s reg\n" , nodename); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + binfo->loader_start); goto fail; } -g_free (nodename); } rc =fdt_path_offset (fdt,"/chosen" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [Qemu-devel] [PATCH v11 01/10] hw/arm/boot: introduce fdt_add_memory_node helper
Quoted 12 lines · Eric Auger
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>Reviewed-by Igor Mammedov <[email protected]>---
v7 -> v8:
- Added Igor's R-b
Quoted 6 lines
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)
r~
PATCH v11 2/10 hw/arm/virt: Rename highmem IO regions
In reply to Eric AugerIn preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM
---
v8:
- added Igor's R-b and fixed "line over 80 characters" warnings
- nit: potential indent issue reported by Igor looks a false positive to me
v7: added Peter's R-b
v6: creation
--- hw/arm/virt-acpi-build.c | 10 ++++++---- hw/arm/virt.c | 33 ++++++++++++++++++--------------- include/hw/arm/virt.h | 8 ++++---- 3 files changed, 28 insertions(+), 23 deletions(-)
hw/arm/virt-acpi-build.c +6 -4
@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, size_pio));if (use_highmem) { -hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].size; aml_append(rbuf, aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -663,8 +663,10 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicr =acpi_data_push (table_data,sizeof (*gicr)); gicr->type =ACPI_APIC_GENERIC_REDISTRIBUTOR ; gicr->length =sizeof (*gicr); - gicr->base_address =cpu_to_le64 (memmap[VIRT_GIC_REDIST2 ].base); - gicr->range_length =cpu_to_le32 (memmap[VIRT_GIC_REDIST2 ].size); + gicr->base_address = +cpu_to_le64 (memmap[VIRT_HIGH_GIC_REDIST2 ].base); + gicr->range_length = +cpu_to_le32 (memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (its_class_name () && !vmc->no_its) {
hw/arm/virt.c +18 -15
@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM ] = {0x40000000 ,RAMLIMIT_BYTES }, /* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_PCIE_ECAM_HIGH ] = {0x4010000000ULL ,0x10000000 }, + [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, + [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_PCIE_MMIO_HIGH] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, };static const int a15irqmap[] = {
@@ -431,12 +431,12 @@ static void fdt_add_gic_node(VirtMachineState *vms) 2, vms->memmap[VIRT_GIC_REDIST].size); }else {qemu_fdt_setprop_sized_cells (vms->fdt, nodename,"reg" , -2 , vms->memmap[VIRT_GIC_DIST ].base, -2 , vms->memmap[VIRT_GIC_DIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST ].base, -2 , vms->memmap[VIRT_GIC_REDIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST2 ].base, -2 , vms->memmap[VIRT_GIC_REDIST2 ].size); +2 , vms->memmap[VIRT_GIC_DIST ].base, +2 , vms->memmap[VIRT_GIC_DIST ].size, +2 , vms->memmap[VIRT_GIC_REDIST ].base, +2 , vms->memmap[VIRT_GIC_REDIST ].size, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (vms->virt) {
@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (nb_redist_regions ==2 ) {uint32_t redist1_capacity = - vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ;qdev_prop_set_uint32 (gicdev,"redist-region-count[1]" ,MIN (smp_cpus - redist0_count, redist1_capacity));
@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (type ==3 ) {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_REDIST ].base);if (nb_redist_regions ==2 ) { -sysbus_mmio_map (gicbusdev,2 , vms->memmap[VIRT_GIC_REDIST2 ].base); +sysbus_mmio_map (gicbusdev,2 , + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base); } }else {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_CPU ].base);
@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO ].size; -hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].size;hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO ].base;hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO ].size;hwaddr base_ecam, size_ecam;
@@ -1417,8 +1418,10 @@ static void machvirt_init(MachineState *machine)* many redistributors we can fit into the memory map. */ if (vms->gic_version ==3 ) { - virt_max_cpus = vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; - virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + virt_max_cpus = + vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; + virt_max_cpus += + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; }else { virt_max_cpus =GIC_NCPU ; }
include/hw/arm/virt.h +4 -4
@@ -64,7 +64,7 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_GIC_REDIST2 , +VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,9 +74,9 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_PCIE_ECAM_HIGH , +VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_PCIE_MMIO_HIGH , +VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART , VIRT_SECURE_MEM,
@@ -128,7 +128,7 @@ typedef struct {int psci_conduit; } VirtMachineState; -#define VIRT_ECAM_ID (high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM) +#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)#define TYPE_VIRT_MACHINE MACHINE_TYPE_NAME("virt")#define VIRT_MACHINE (obj) \
Signature · 1 line
2.20.1
PATCH v11 3/10 hw/arm/virt: Split the memory map description
In reply to Eric AugerIn the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap, renamed base_memmap, contains regions below and including the RAM. MemMapEntries initialized in this array have a static size and base address.
- extended_memmap, only initialized with entries located after the RAM. MemMapEntries initialized in this array only get their size initialized. Their base address is dynamically computed depending on the the top of the RAM, with same alignment as their size.
Eventually base_memmap entries are copied into the extended_memmap array. Using two separate arrays however clarifies which entries are statically allocated and those which are dynamically allocated.
This new split will allow to grow the RAM size without changing the description of the high IO entries.
We introduce a new virt_set_memmap() helper function which "freezes" the memory map. We call it in machvirt_init as memory attributes of the machine are not yet set when virt_instance_init() gets called.
The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
---
v10 -> v11:
- remove vms->high_io_base
- initialize vms->memmamp to NULL in the instance_init
v8 -> v9:
- restore virt_set_memmap call in machvirt_init as otherwise it does not work in TCG mode
v7 -> v8:
- removed Peter's R-b due to the changes induced by Igor's comments
- call set_memmap back in virt_instance_init
- add comments about sizing of extended_memmap
- s/region/entries as suggested by Igor
- rewording of the commit message
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-b
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments
--- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 13 +++++++---- 2 files changed, 54 insertions(+), 10 deletions(-)
hw/arm/virt.c +45 -6
@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h"
@@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
@@ -149,11 +150,24 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each MemMapEntry base (GPA) will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB entry will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +* Note the extended_memmap is sized so that it eventually also includes the +* base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last +* index of base_memmap). +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
@@ -1354,6 +1368,29 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + base =256 * GiB;/* Top of the legacy initial RAM region */ + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);
@@ -1368,6 +1405,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */
@@ -1845,7 +1884,7 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; + vms->memmap =NULL ; vms->irqmap = a15irqmap; }
include/hw/arm/virt.h +9 -4
@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {
@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v11 03/10] hw/arm/virt: Split the memory map description
Quoted 23 lines · On Fri, 1 Mar 2019 14:47:02 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap, renamed
base_memmap, contains regions below and including the RAM. MemMapEntries initialized in this array have a static size and base address.
-extended_memmap, only initialized with entries located after the RAM. MemMapEntries initialized in this array only get their size initialized. Their base address is dynamically computed depending on the the top of the RAM, with same alignment as their size.Eventually
base_memmapentries are copied into theextended_memmaparray. Using two separate arrays however clarifies which entries are statically allocated and those which are dynamically allocated.This new split will allow to grow the RAM size without changing the description of the high IO entries.
We introduce a new
virt_set_memmap()helper function which "freezes" the memory map. We call it inmachvirt_initas memory attributes of the machine are not yet set whenvirt_instance_init()gets called.
Quoted 4 lines
The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>
with nits fixed or not:
Quoted 57 lines
---
v10 -> v11:
- removevms->high_io_base
- initializevms->memmampto NULL in theinstance_initv8 -> v9:
- restorevirt_set_memmapcall inmachvirt_initas otherwise it does not work in TCG modev7 -> v8:
- removed Peter's R-b due to the changes induced by Igor's comments
- callset_memmapback invirt_instance_init
- add comments about sizing ofextended_memmap
- s/region/entries as suggested by Igor
- rewording of the commit messagev6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 13 +++++++---- 2 files changed, 54 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 152
Quoted 3 lines
hw/arm/virt.c +0
-0@@ -149,11 +150,24 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000,
},
Quoted 31 lines
hw/arm/virt.c +19
-4+}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each MemMapEntry base (GPA) will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB entry will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +* Note the extended_memmap is sized so that it eventually also includes the +* base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last +* index of base_memmap). +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {@@ -1354,6 +1368,29 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 10 lines
hw/arm/virt.c +10
-0+hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + base =256 * GiB;/* Top of the legacy initial RAM region */
nit:
like you suggested, maybe use here [VIRT_MEM] + RAMLIMIT_BYTES
Quoted 24 lines
hw/arm/virt.c +13
-0+ +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1405,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */
hw/arm/virt.c static void virt_instance_init(Object *obj) · line 1887
Quoted 5 lines
hw/arm/virt.c +1
-1@@ -1845,7 +1884,7 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; + vms->memmap =NULL ;
nit:
it's sufficient to just remove this line as objects a guarantied to be 0 initialized
Quoted 45 lines
hw/arm/virt.c +0
-0vms->irqmap = a15irqmap; }include/hw/arm/virt.h +9
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;
PATCH v11 4/10 hw/boards: Add a MachineState parameter to kvm_type callback
In reply to Eric AugerOn ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.
---
v7 -> v8: added Igor's R-b
v6 -> v7:
- add a comment for kvm_type
- use machine instead of ms in the declaration
- add Peter's R-b
--- accel/kvm/kvm-all.c | 2 +- hw/ppc/mac_newworld.c | 3 +-- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/boards.h | 5 ++++- 5 files changed, 8 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +1 -1
@@ -1593,7 +1593,7 @@ static int kvm_init(MachineState *ms) kvm_type =qemu_opt_get (qemu_get_machine_opts (),"kvm-type" );if (mc->kvm_type) { - type = mc->kvm_type (kvm_type); + type = mc->kvm_type (ms, kvm_type); }else if (kvm_type) { ret = -EINVAL ;fprintf (stderr,"Invalid argument kvm-type=%s\n" , kvm_type);
hw/ppc/mac_newworld.c +1 -2
@@ -564,8 +564,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } - -static int core99_kvm_type (const char *arg) +static int core99_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/mac_oldworld.c +1 -1
@@ -420,7 +420,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } -static int heathrow_kvm_type (const char *arg) +static int heathrow_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/spapr.c +1 -1
@@ -3023,7 +3023,7 @@ static void spapr_machine_init(MachineState *machine) } } -static int spapr_kvm_type (const char *vm_type) +static int spapr_kvm_type (MachineState *machine,const char *vm_type) {if (!vm_type) {return 0 ;
include/hw/boards.h +4 -1
@@ -157,6 +157,9 @@ typedef struct {* should instead use "unimplemented-device" for all memory ranges where * the guest will attempt to probe for a device that QEMU doesn't * implement and a stub device is required. +* @kvm_type: +* Return the type of KVM corresponding to the kvm-type string option or +* computed based on other criteria such as the host kernel capabilities. */ struct MachineClass {/*< private >*/
@@ -172,7 +175,7 @@ struct MachineClass {void (*init)(MachineState *state);void (*reset)(void );void (*hot_add_cpu)(const int64_t id,Error **errp); -int (*kvm_type)(const char *arg); +int (*kvm_type)(MachineState *machine,const char *arg);BlockInterfaceType block_default_type;int units_per_default_bus;
Signature · 1 line
2.20.1
PATCH v11 5/10 kvm: add kvm_arm_get_max_vm_ipa_size
In reply to Eric AugerAdd the kvm_arm_get_max_vm_ipa_size() helper that returns the number of bits in the IPA address space supported by KVM.
This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.
---
v10 -> v11:
- added Igor's R-b
v6 -> v7:
- s/kvm_arm_get_max_vm_phys_shift/kvm_arm_get_max_vm_ipa_size
- reword the comment
v4 -> v5:
- return 40 if the host does not support the capability
v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension
v1 -> v2:
- put this in ARM specific code
--- target/arm/kvm.c | 10 ++++++++++ target/arm/kvm_arm.h | 13 +++++++++++++ 2 files changed, 23 insertions(+)
target/arm/kvm.c +10 -0
@@ -18,6 +18,7 @@#include "qemu/error-report.h" #include "sysemu/sysemu.h" #include "sysemu/kvm.h" +#include "sysemu/kvm_int.h" #include "kvm_arm.h" #include "cpu.h" #include "trace.h"
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) env->features = arm_host_cpu_features.features; } +int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +KVMState *s =KVM_STATE (ms->accelerator); +int ret; + + ret =kvm_check_extension (s,KVM_CAP_ARM_VM_IPA_SIZE ); +return ret >0 ? ret :40 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +13 -0
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);*/ void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu); +/** +* kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the +* IPA address space supported by KVM +* +* @ms: Machine state handle +*/ +int kvm_arm_get_max_vm_ipa_size (MachineState *ms); +/** * kvm_arm_sync_mpstate_to_kvm * @cpu: ARMCPU
@@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) cpu->host_cpu_probe_failed = true; } +static inline int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +return -ENOENT ; +} +static inline int kvm_arm_vgic_probe (void ) {return 0 ;
Signature · 1 line
2.20.1
PATCH v11 6/10 vl: Set machine ram_size, maxram_size and ram_slots earlier
In reply to Eric AugerThe machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.
---
v7 -> v8:
- added Igor's R-b
v6 -> v7:
- add Peter's R-b
v4: new
--- vl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +3 -3
@@ -4239,6 +4239,9 @@ int main(int argc, char **argv, char **envp) machine_opts =qemu_get_machine_opts ();qemu_opt_foreach (machine_opts, machine_set_property, current_machine, &error_fatal); + current_machine->ram_size = ram_size; + current_machine->maxram_size = maxram_size; + current_machine->ram_slots = ram_slots;configure_accelerator (current_machine, argv[0 ]);
@@ -4434,9 +4437,6 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); - current_machine->ram_size = ram_size; - current_machine->maxram_size = maxram_size; - current_machine->ram_slots = ram_slots; current_machine->boot_order = boot_order;/* parse features once if machine provides default cpu_type */
Signature · 1 line
2.20.1
PATCH v11 7/10 hw/arm/virt: Dynamic memory map depending on RAM requirements
In reply to Eric AugerUp to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the virt_set_memmap() function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.
The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.
Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB.
---
v10 -> v11:
- remove RAMBASE
- s/256GiB/vms->memmap[VIRT_MEM].base + LEGACY_RAMLIMIT_BYTES while checking if high io region base < 256GB
- added a comment in base_memmap about the actual RAM size
v9 -> v10:
- remove check of maxram_size 1GB alignment
v8 -> v9:
- add if (machine->device_memory) check
v7 -> v8:
- allocate ms->device_memory and removes vms->device_memory_base and vms->device_memory_storage
- remove (ms->maxram_size > ms->ram_size || ms->ram_slots > 0) and (ms->ram_size > (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check and maxram_size alignment checks in this patch
--- hw/arm/virt.c | 52 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 1 + 2 files changed, 47 insertions(+), 6 deletions(-)
hw/arm/virt.c +46 -6
@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -107,8 +108,8 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
@@ -149,7 +150,8 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM ] = {0x3f000000 ,0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + /* Actual RAM size depends on initial RAM and device memory settings */ + [VIRT_MEM] = { GiB, LEGACY_RAMLIMIT_BYTES }, };/*
@@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;
@@ -1379,7 +1382,32 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base = +ROUND_UP (vms->memmap[VIRT_MEM ].base + ms->ram_size, GiB); + device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + +/* Base address of the high IO region */ + base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (base < vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ) { + base = vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ; + }for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {hwaddr size = extended_memmap[i].size;
@@ -1389,6 +1417,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)
@@ -1475,7 +1510,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }
@@ -1569,6 +1605,10 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (machine->device_memory) { +memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr); + }create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
include/hw/arm/virt.h +1 -0
@@ -131,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v11 07/10] hw/arm/virt: Dynamic memory map depending on RAM requirements
Quoted 25 lines · On Fri, 1 Mar 2019 14:47:06 +0100 Eric Auger <[email protected]> wrote:
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Quoted 3 lines
Still the user cannot request an initial RAM size greater than 255GB.
Signed-off-by Eric Auger <[email protected]>
Quoted 151 lines
---
v10 -> v11:
- remove RAMBASE
- s/256GiB/vms->memmap[VIRT_MEM].base +LEGACY_RAMLIMIT_BYTESwhile checking if high io region base < 256GB
- added a comment inbase_memmapabout the actual RAM sizev9 -> v10:
- remove check ofmaxram_size1GB alignmentv8 -> v9:
- add if (machine->device_memory) checkv7 -> v8:
- allocatems->device_memoryand removesvms->device_memory_baseandvms->device_memory_storage
- remove (ms->maxram_size>ms->ram_size||ms->ram_slots> 0) and (ms->ram_size> (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check andmaxram_sizealignment checks in this patch--- hw/arm/virt.c | 52 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 1 + 2 files changed, 47 insertions(+), 6 deletions(-)hw/arm/virt.c +46
-6@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -107,8 +108,8 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +150,8 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM ] = {0x3f000000 ,0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + /* Actual RAM size depends on initial RAM and device memory settings */ + [VIRT_MEM] = { GiB, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;@@ -1379,7 +1382,32 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base = +ROUND_UP (vms->memmap[VIRT_MEM ].base + ms->ram_size, GiB); + device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + +/* Base address of the high IO region */ + base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (base < vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ) { + base = vms->memmap[VIRT_MEM ].base +LEGACY_RAMLIMIT_BYTES ; + }for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {hwaddr size = extended_memmap[i].size;@@ -1389,6 +1417,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)@@ -1475,7 +1510,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }@@ -1569,6 +1605,10 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (machine->device_memory) { +memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr); + }create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);include/hw/arm/virt.h +1
-0@@ -131,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v11 8/10 hw/arm/virt: Implement kvm_type function for 4.0 machine
In reply to Eric AugerThis patch implements the machine class kvm_type() callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.
To compute the highest GPA used in the memory map, kvm_type() must freeze the memory map by calling virt_set_memmap().
---
v10 -> v11:
- replace !kvm_enabled check by !vms->memmap
v7 -> v8:
- remove vmc->no_extended_memmap and vms->extended_memmap
v6 -> v7:
- Introduce RAMBASE and rename add LEGACY_ prefix in that patch
- use local variables with explicit names in virt_set_memmap:
device_memory_base, device_memory_size
- add an extended_memmap field in the class
v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB
--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
hw/arm/virt.c +38 -1
@@ -1440,7 +1440,13 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed earlier in kvm_type() +* to create a VM with the right number of IPA bits. +*/ +if (!vms->memmap) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet
@@ -1829,6 +1835,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -1853,6 +1889,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v11 08/10] hw/arm/virt: Implement kvm_type function for 4.0 machine
Quoted 4 lines · On Fri, 1 Mar 2019 14:47:07 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and
s/in/is/
Quoted 6 lines
this allows KVM to set the stage2 tables dynamically.
To compute the highest GPA used in the memory map,
kvm_type()must freeze the memory map by callingvirt_set_memmap().Signed-off-by Eric Auger <[email protected]>
with fixed commit:
Quoted 85 lines
---
v10 -> v11:
- replace !kvm_enabled check by !vms->memmapv7 -> v8:
- removevmc->no_extended_memmapandvms->extended_memmapv6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)hw/arm/virt.c +38
-1@@ -1440,7 +1440,13 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed earlier in kvm_type() +* to create a VM with the right number of IPA bits. +*/ +if (!vms->memmap) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1829,6 +1835,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -1853,6 +1889,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
PATCH v11 9/10 hw/arm/virt: Check the VCPU PA range in TCG mode
In reply to Eric AugerWe are about to allow the memory map to grow beyond 1TB and potentially overshoot the VCPU AA64MMFR0.PARANGE.
In aarch64 mode and when highmem is set, let's check the VCPU PA range is sufficient to address the highest GPA of the memory map.
--- hw/arm/virt.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
hw/arm/virt.c +17 -0
@@ -60,6 +60,7 @@#include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" #include "hw/acpi/acpi.h" +#include "target/arm/internals.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1608,6 +1609,22 @@ static void machvirt_init(MachineState *machine)fdt_add_timer_nodes (vms);fdt_add_cpu_nodes (vms); +if (!kvm_enabled ()) { +ARMCPU *cpu =ARM_CPU (first_cpu); +bool aarch64 =object_property_get_bool (OBJECT (cpu),"aarch64" ,NULL ); + +if (aarch64 && vms->highmem) { +int requested_pa_size, pamax =arm_pamax (cpu); + + requested_pa_size =64 -clz64 (vms->highest_gpa); +if (pamax < requested_pa_size) { +error_report ("VCPU supports less PA bits (%d) than requested " +"by the memory map (%d)" , pamax, requested_pa_size); +exit (1 ); + } + } + } +memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v11 09/10] hw/arm/virt: Check the VCPU PA range in TCG mode
Quoted 8 lines · On Fri, 1 Mar 2019 14:47:08 +0100 Eric Auger <[email protected]> wrote:
We are about to allow the memory map to grow beyond 1TB and potentially overshoot the VCPU AA64MMFR0.PARANGE.
In aarch64 mode and when highmem is set, let's check the VCPU PA range is sufficient to address the highest GPA of the memory map.
Signed-off-by Eric Auger <[email protected]>
Quoted 39 lines
--- hw/arm/virt.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)hw/arm/virt.c +17
-0@@ -60,6 +60,7 @@#include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" #include "hw/acpi/acpi.h" +#include "target/arm/internals.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -1608,6 +1609,22 @@ static void machvirt_init(MachineState *machine)fdt_add_timer_nodes (vms);fdt_add_cpu_nodes (vms); +if (!kvm_enabled ()) { +ARMCPU *cpu =ARM_CPU (first_cpu); +bool aarch64 =object_property_get_bool (OBJECT (cpu),"aarch64" ,NULL ); + +if (aarch64 && vms->highmem) { +int requested_pa_size, pamax =arm_pamax (cpu); + + requested_pa_size =64 -clz64 (vms->highest_gpa); +if (pamax < requested_pa_size) { +error_report ("VCPU supports less PA bits (%d) than requested " +"by the memory map (%d)" , pamax, requested_pa_size); +exit (1 ); + } + } + } +memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
PATCH v11 10/10 hw/arm/virt: Bump the 255GB initial RAM limit
In reply to Eric AugerNow we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
---
v7 -> v8:
- TCG PAMAX check moved in a separate patch
v6 -> v7
- handle TCG case
- set_memmap modifications moved to previous patches
--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
hw/arm/virt.c +1 -20
@@ -95,20 +95,7 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces -* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
@@ -1516,12 +1503,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [Qemu-devel] [PATCH v11 10/10] hw/arm/virt: Bump the 255GB initial RAM limit
Quoted 8 lines · On Fri, 1 Mar 2019 14:47:09 +0100 Eric Auger <[email protected]> wrote:
Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
Signed-off-by Eric Auger <[email protected]>
Quoted 51 lines
---
v7 -> v8:
- TCG PAMAX check moved in a separate patchv6 -> v7
- handle TCG case
-set_memmapmodifications moved to previous patches--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)hw/arm/virt.c +1
-20@@ -95,20 +95,7 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces -* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)@@ -1516,12 +1503,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );