ARM virt: Initial RAM expansion and extended memory map
v10 cover ARM virt: Initial RAM expansion and extended memory map
This series aims to bump the 255GB RAM limit in machvirt and paves the way to device memory addition.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size ram_size and then we provision sufficient space for the looming device memory (dimensionned using ,maxmem and slots options).
IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on ram_size, slots, maxram_size. Size alignment is enforced.
In case maxmem value is inferior to 255GB, the legacy memory map still is used.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it will be possible to get access to this feature while keeping a 1TB address map.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-extended-memmap-v10
History:
v9 -> v10:
- only the keep the first 10 patches which move to the extended memmap
- remove the 1GB alignment check on maxram_size
v8 -> v9:
- fixed make check SIGSEV
- fixed TCG issue (restored virt_set_memmap in machvirt_init)
- removed Igor's R-b on "hw/arm/boot: Expose the PC-DIMM nodes in the DT"
v7 -> v8:
- addressed Igor's comments"
- remove no_extended_memmap knob
- moved acpi_nvdimm_state into MachineState
- separate "hw/arm/virt: Check the VCPU PA range in TCG mode" patch
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more details
v5 -> v6:
- mingw compilation issue fix
- kvm_arm_get_max_vm_phys_shift always returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization into machvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their size
v4 -> v5:
- change in the memory map
- see individual logs
v3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm: pre_plug "slot" and "addr" assignment"
- kvm-type option not used anymore. We directly use maxram_size and ram_size machine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
- device_memory_start and device_memory_size not stored anymore in vms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring of arm_load_dtb and the one related to the generation of the dtb after system_reset which would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machine ram_size, maxram_size and ram_slots earlier"
v2 -> v3:
- fix pc_q35 and pc_piix compilation error
- kwangwoo's email being not valid anymore, remove his address
v1 -> v2:
- kvm_get_max_vm_phys_shift moved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoring
v1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"
Best Regards
Eric
Eric Auger (9):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter to kvm_type callback
kvm: add kvm_arm_get_max_vm_ipa_size
vl: Set machine ram_size, maxram_size and ram_slots earlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implement kvm_type function for 4.0 machine hw/arm/virt: Check the VCPU PA range in TCG mode hw/arm/virt: Bump the 255GB initial RAM limit
Shameer Kolothum (1):
hw/arm/boot: introduce fdt_add_memory_node helper
accel/kvm/kvm-all.c | 2 +- hw/arm/boot.c | 54 +++++++---- hw/arm/virt-acpi-build.c | 10 +- hw/arm/virt.c | 195 +++++++++++++++++++++++++++++++-------- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 17 +++- include/hw/boards.h | 5 +- target/arm/kvm.c | 10 ++ target/arm/kvm_arm.h | 13 +++ vl.c | 6 +- 12 files changed, 241 insertions(+), 78 deletions(-)
Signature · 1 line
2.20.1
PATCH v10 1/10 hw/arm/boot: introduce fdt_add_memory_node helper
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
---
v7 -> v8:
- Added Igor's R-b
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID
--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +34 -20
@@ -423,6 +423,32 @@ static void set_kernel_args_old(const struct arm_boot_info *info, } } +static int fdt_add_memory_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -502,7 +528,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,void *fdt =NULL ;int size, rc, n =0 ;uint32_t acells, scells; -char *nodename;unsigned int i;hwaddr mem_base, mem_len;char **node_path;
@@ -576,35 +601,24 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start;for (i =0 ; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; - nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, mem_base, - scells, mem_len); + rc =fdt_add_memory_node (fdt, acells, mem_base, + scells, mem_len, i);if (rc <0 ) { -fprintf (stderr,"couldn't set %s/reg for node %d\n" , nodename, - i); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + mem_base); goto fail; } -qemu_fdt_setprop_cell (fdt, nodename,"numa-node-id" , i); mem_base += mem_len; -g_free (nodename); } }else { - nodename =g_strdup_printf ("/memory@%" PRIx64, binfo->loader_start); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, binfo->loader_start, - scells, binfo->ram_size); + rc =fdt_add_memory_node (fdt, acells, binfo->loader_start, + scells, binfo->ram_size,-1 );if (rc <0 ) { -fprintf (stderr,"couldn't set %s reg\n" , nodename); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + binfo->loader_start); goto fail; } -g_free (nodename); } rc =fdt_path_offset (fdt,"/chosen" );
Signature · 1 line
2.20.1
PATCH v10 2/10 hw/arm/virt: Rename highmem IO regions
In reply to Eric AugerIn preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM
---
v8:
- added Igor's R-b and fixed "line over 80 characters" warnings
- nit: potential indent issue reported by Igor looks a false positive to me
v7: added Peter's R-b
v6: creation
--- hw/arm/virt-acpi-build.c | 10 ++++++---- hw/arm/virt.c | 33 ++++++++++++++++++--------------- include/hw/arm/virt.h | 8 ++++---- 3 files changed, 28 insertions(+), 23 deletions(-)
hw/arm/virt-acpi-build.c +6 -4
@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, size_pio));if (use_highmem) { -hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].size; aml_append(rbuf, aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -663,8 +663,10 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicr =acpi_data_push (table_data,sizeof (*gicr)); gicr->type =ACPI_APIC_GENERIC_REDISTRIBUTOR ; gicr->length =sizeof (*gicr); - gicr->base_address =cpu_to_le64 (memmap[VIRT_GIC_REDIST2 ].base); - gicr->range_length =cpu_to_le32 (memmap[VIRT_GIC_REDIST2 ].size); + gicr->base_address = +cpu_to_le64 (memmap[VIRT_HIGH_GIC_REDIST2 ].base); + gicr->range_length = +cpu_to_le32 (memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (its_class_name () && !vmc->no_its) {
hw/arm/virt.c +18 -15
@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM ] = {0x40000000 ,RAMLIMIT_BYTES }, /* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_PCIE_ECAM_HIGH ] = {0x4010000000ULL ,0x10000000 }, + [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, + [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_PCIE_MMIO_HIGH] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, };static const int a15irqmap[] = {
@@ -431,12 +431,12 @@ static void fdt_add_gic_node(VirtMachineState *vms) 2, vms->memmap[VIRT_GIC_REDIST].size); }else {qemu_fdt_setprop_sized_cells (vms->fdt, nodename,"reg" , -2 , vms->memmap[VIRT_GIC_DIST ].base, -2 , vms->memmap[VIRT_GIC_DIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST ].base, -2 , vms->memmap[VIRT_GIC_REDIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST2 ].base, -2 , vms->memmap[VIRT_GIC_REDIST2 ].size); +2 , vms->memmap[VIRT_GIC_DIST ].base, +2 , vms->memmap[VIRT_GIC_DIST ].size, +2 , vms->memmap[VIRT_GIC_REDIST ].base, +2 , vms->memmap[VIRT_GIC_REDIST ].size, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (vms->virt) {
@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (nb_redist_regions ==2 ) {uint32_t redist1_capacity = - vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ;qdev_prop_set_uint32 (gicdev,"redist-region-count[1]" ,MIN (smp_cpus - redist0_count, redist1_capacity));
@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (type ==3 ) {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_REDIST ].base);if (nb_redist_regions ==2 ) { -sysbus_mmio_map (gicbusdev,2 , vms->memmap[VIRT_GIC_REDIST2 ].base); +sysbus_mmio_map (gicbusdev,2 , + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base); } }else {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_CPU ].base);
@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO ].size; -hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].size;hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO ].base;hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO ].size;hwaddr base_ecam, size_ecam;
@@ -1417,8 +1418,10 @@ static void machvirt_init(MachineState *machine)* many redistributors we can fit into the memory map. */ if (vms->gic_version ==3 ) { - virt_max_cpus = vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; - virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + virt_max_cpus = + vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; + virt_max_cpus += + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; }else { virt_max_cpus =GIC_NCPU ; }
include/hw/arm/virt.h +4 -4
@@ -64,7 +64,7 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_GIC_REDIST2 , +VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,9 +74,9 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_PCIE_ECAM_HIGH , +VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_PCIE_MMIO_HIGH , +VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART , VIRT_SECURE_MEM,
@@ -128,7 +128,7 @@ typedef struct {int psci_conduit; } VirtMachineState; -#define VIRT_ECAM_ID (high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM) +#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)#define TYPE_VIRT_MACHINE MACHINE_TYPE_NAME("virt")#define VIRT_MACHINE (obj) \
Signature · 1 line
2.20.1
PATCH v10 3/10 hw/arm/virt: Split the memory map description
In reply to Eric AugerIn the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap, renamed base_memmap, contains regions below and including the RAM. MemMapEntries initialized in this array have a static size and base address.
- extended_memmap, only initialized with entries located after the RAM. MemMapEntries initialized in this array only get their size initialized. Their base address is dynamically computed depending on the the top of the RAM, with same alignment as their size.
Eventually base_memmap entries are copied into the extended_memmap array. Using two separate arrays however clarifies which entries are statically allocated and those which are dynamically allocated.
This new split will allow to grow the RAM size without changing the description of the high IO entries.
We introduce a new virt_set_memmap() helper function which "freezes" the memory map. We call it in machvirt_init as memory attributes of the machine are not yet set when virt_instance_init() gets called.
The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
---
v8 -> v9:
- restore virt_set_memmap call in machvirt_init as otherwise it does not work in TCG mode
v7 -> v8:
- removed Peter's R-b due to the changes induced by Igor's comments
- call set_memmap back in virt_instance_init
- add comments about sizing of extended_memmap
- s/region/entries as suggested by Igor
- rewording of the commit message
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-b
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments
--- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 14 ++++++++---- 2 files changed, 55 insertions(+), 10 deletions(-)
hw/arm/virt.c +45 -6
@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h"
@@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
@@ -149,11 +150,24 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each MemMapEntry base (GPA) will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB entry will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +* Note the extended_memmap is sized so that it eventually also includes the +* base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last +* index of base_memmap). +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
@@ -1354,6 +1368,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);
@@ -1368,6 +1406,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */
@@ -1845,7 +1885,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }
include/hw/arm/virt.h +10 -4
@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {
@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;
@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v10 03/10] hw/arm/virt: Split the memory map description
Quoted 114 lines · On Thu, 28 Feb 2019 16:03:17 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap, renamed
base_memmap, contains regions below and including the RAM. MemMapEntries initialized in this array have a static size and base address.
-extended_memmap, only initialized with entries located after the RAM. MemMapEntries initialized in this array only get their size initialized. Their base address is dynamically computed depending on the the top of the RAM, with same alignment as their size.Eventually
base_memmapentries are copied into theextended_memmaparray. Using two separate arrays however clarifies which entries are statically allocated and those which are dynamically allocated.This new split will allow to grow the RAM size without changing the description of the high IO entries.
We introduce a new
virt_set_memmap()helper function which "freezes" the memory map. We call it inmachvirt_initas memory attributes of the machine are not yet set whenvirt_instance_init()gets called.The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>---
v8 -> v9:
- restorevirt_set_memmapcall inmachvirt_initas otherwise it does not work in TCG modev7 -> v8:
- removed Peter's R-b due to the changes induced by Igor's comments
- callset_memmapback invirt_instance_init
- add comments about sizing ofextended_memmap
- s/region/entries as suggested by Igor
- rewording of the commit messagev6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 14 ++++++++---- 2 files changed, 55 insertions(+), 10 deletions(-)hw/arm/virt.c +21
-5@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },@@ -149,11 +150,24 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each MemMapEntry base (GPA) will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB entry will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +* Note the extended_memmap is sized so that it eventually also includes the +* base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last +* index of base_memmap). +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {@@ -1354,6 +1368,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 10 lines
hw/arm/virt.c +10
-0+hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */
Why do we need to carry this state in vms?
it looks like it is used as local variable
and then it's possible to access that memmap[VIRT_HIGH_GIC_REDIST2] outside of this function, hence it rises duplication alarm
Quoted 82 lines
hw/arm/virt.c +14
-1+ base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1406,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1845,7 +1885,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v10 4/10 hw/boards: Add a MachineState parameter to kvm_type callback
In reply to Eric AugerOn ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.
---
v7 -> v8: added Igor's R-b
v6 -> v7:
- add a comment for kvm_type
- use machine instead of ms in the declaration
- add Peter's R-b
--- accel/kvm/kvm-all.c | 2 +- hw/ppc/mac_newworld.c | 3 +-- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/boards.h | 5 ++++- 5 files changed, 8 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +1 -1
@@ -1593,7 +1593,7 @@ static int kvm_init(MachineState *ms) kvm_type =qemu_opt_get (qemu_get_machine_opts (),"kvm-type" );if (mc->kvm_type) { - type = mc->kvm_type (kvm_type); + type = mc->kvm_type (ms, kvm_type); }else if (kvm_type) { ret = -EINVAL ;fprintf (stderr,"Invalid argument kvm-type=%s\n" , kvm_type);
hw/ppc/mac_newworld.c +1 -2
@@ -564,8 +564,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } - -static int core99_kvm_type (const char *arg) +static int core99_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/mac_oldworld.c +1 -1
@@ -420,7 +420,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } -static int heathrow_kvm_type (const char *arg) +static int heathrow_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/spapr.c +1 -1
@@ -2920,7 +2920,7 @@ static void spapr_machine_init(MachineState *machine) } } -static int spapr_kvm_type (const char *vm_type) +static int spapr_kvm_type (MachineState *machine,const char *vm_type) {if (!vm_type) {return 0 ;
include/hw/boards.h +4 -1
@@ -156,6 +156,9 @@ typedef struct {* should instead use "unimplemented-device" for all memory ranges where * the guest will attempt to probe for a device that QEMU doesn't * implement and a stub device is required. +* @kvm_type: +* Return the type of KVM corresponding to the kvm-type string option or +* computed based on other criteria such as the host kernel capabilities. */ struct MachineClass {/*< private >*/
@@ -171,7 +174,7 @@ struct MachineClass {void (*init)(MachineState *state);void (*reset)(void );void (*hot_add_cpu)(const int64_t id,Error **errp); -int (*kvm_type)(const char *arg); +int (*kvm_type)(MachineState *machine,const char *arg);BlockInterfaceType block_default_type;int units_per_default_bus;
Signature · 1 line
2.20.1
PATCH v10 5/10 kvm: add kvm_arm_get_max_vm_ipa_size
In reply to Eric AugerAdd the kvm_arm_get_max_vm_ipa_size() helper that returns the number of bits in the IPA address space supported by KVM.
This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.
---
v6 -> v7:
- s/kvm_arm_get_max_vm_phys_shift/kvm_arm_get_max_vm_ipa_size
- reword the comment
v4 -> v5:
- return 40 if the host does not support the capability
v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension
v1 -> v2:
- put this in ARM specific code
--- target/arm/kvm.c | 10 ++++++++++ target/arm/kvm_arm.h | 13 +++++++++++++ 2 files changed, 23 insertions(+)
target/arm/kvm.c +10 -0
@@ -18,6 +18,7 @@#include "qemu/error-report.h" #include "sysemu/sysemu.h" #include "sysemu/kvm.h" +#include "sysemu/kvm_int.h" #include "kvm_arm.h" #include "cpu.h" #include "trace.h"
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) env->features = arm_host_cpu_features.features; } +int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +KVMState *s =KVM_STATE (ms->accelerator); +int ret; + + ret =kvm_check_extension (s,KVM_CAP_ARM_VM_IPA_SIZE ); +return ret >0 ? ret :40 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +13 -0
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);*/ void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu); +/** +* kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the +* IPA address space supported by KVM +* +* @ms: Machine state handle +*/ +int kvm_arm_get_max_vm_ipa_size (MachineState *ms); +/** * kvm_arm_sync_mpstate_to_kvm * @cpu: ARMCPU
@@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) cpu->host_cpu_probe_failed = true; } +static inline int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +return -ENOENT ; +} +static inline int kvm_arm_vgic_probe (void ) {return 0 ;
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v10 05/10] kvm: add kvm_arm_get_max_vm_ipa_size
Quoted 7 lines · On Thu, 28 Feb 2019 16:03:19 +0100 Eric Auger <[email protected]> wrote:
Add the
kvm_arm_get_max_vm_ipa_size()helper that returns the number of bits in the IPA address space supported by KVM.This capability needs to be known to create the VM with a specific IPA max size (
kvm_typepassed alongKVM_CREATE_VMioctl.Signed-off-by Eric Auger <[email protected]>
Quoted 79 lines
---
v6 -> v7:
- s/kvm_arm_get_max_vm_phys_shift/kvm_arm_get_max_vm_ipa_size
- reword the commentv4 -> v5:
- return 40 if the host does not support the capabilityv3 -> v4:
- s/s/ms inkvm_arm_get_max_vm_phys_shiftfunction comment
- checkKVM_CAP_ARM_VM_IPA_SIZEextensionv1 -> v2:
- put this in ARM specific code--- target/arm/kvm.c | 10 ++++++++++ target/arm/kvm_arm.h | 13 +++++++++++++ 2 files changed, 23 insertions(+)target/arm/kvm.c +10
-0@@ -18,6 +18,7 @@#include "qemu/error-report.h" #include "sysemu/sysemu.h" #include "sysemu/kvm.h" +#include "sysemu/kvm_int.h" #include "kvm_arm.h" #include "cpu.h" #include "trace.h" @@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) env->features = arm_host_cpu_features.features; } +int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +KVMState *s =KVM_STATE (ms->accelerator); +int ret; + + ret =kvm_check_extension (s,KVM_CAP_ARM_VM_IPA_SIZE ); +return ret >0 ? ret :40 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* For ARM interrupt delivery is always asynchronous, target/arm/kvm_arm.h +13
-0@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);*/ void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu); +/** +* kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the +* IPA address space supported by KVM +* +* @ms: Machine state handle +*/ +int kvm_arm_get_max_vm_ipa_size (MachineState *ms); +/** * kvm_arm_sync_mpstate_to_kvm * @cpu: ARMCPU @@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) cpu->host_cpu_probe_failed = true; } +static inline int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +return -ENOENT ; +} +static inline int kvm_arm_vgic_probe (void ) {return 0 ;
PATCH v10 6/10 vl: Set machine ram_size, maxram_size and ram_slots earlier
In reply to Eric AugerThe machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.
---
v7 -> v8:
- added Igor's R-b
v6 -> v7:
- add Peter's R-b
v4: new
--- vl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +3 -3
@@ -4239,6 +4239,9 @@ int main(int argc, char **argv, char **envp) machine_opts =qemu_get_machine_opts ();qemu_opt_foreach (machine_opts, machine_set_property, current_machine, &error_fatal); + current_machine->ram_size = ram_size; + current_machine->maxram_size = maxram_size; + current_machine->ram_slots = ram_slots;configure_accelerator (current_machine, argv[0 ]);
@@ -4434,9 +4437,6 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); - current_machine->ram_size = ram_size; - current_machine->maxram_size = maxram_size; - current_machine->ram_slots = ram_slots; current_machine->boot_order = boot_order;/* parse features once if machine provides default cpu_type */
Signature · 1 line
2.20.1
PATCH v10 7/10 hw/arm/virt: Dynamic memory map depending on RAM requirements
In reply to Eric AugerUp to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the virt_set_memmap() function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.
The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.
Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB.
---
v9 -> v10:
- remove check of maxram_size 1GB alignment
v8 -> v9:
- add if (machine->device_memory) check
v7 -> v8:
- allocate ms->device_memory and removes vms->device_memory_base and vms->device_memory_storage
- remove (ms->maxram_size > ms->ram_size || ms->ram_slots > 0) and (ms->ram_size > (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check and maxram_size alignment checks in this patch
--- hw/arm/virt.c | 50 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 1 + 2 files changed, 45 insertions(+), 6 deletions(-)
hw/arm/virt.c +44 -6
@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -107,8 +108,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
@@ -149,7 +151,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/*
@@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;
@@ -1379,7 +1382,30 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + + vms->high_io_base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (vms->high_io_base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {
@@ -1390,6 +1416,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)
@@ -1476,7 +1509,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }
@@ -1570,6 +1604,10 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (machine->device_memory) { +memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr); + }create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
include/hw/arm/virt.h +1 -0
@@ -132,6 +132,7 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v10 07/10] hw/arm/virt: Dynamic memory map depending on RAM requirements
Quoted 101 lines · On Thu, 28 Feb 2019 16:03:21 +0100 Eric Auger <[email protected]> wrote:
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB.
Signed-off-by Eric Auger <[email protected]>---
v9 -> v10:
- remove check ofmaxram_size1GB alignmentv8 -> v9:
- add if (machine->device_memory) checkv7 -> v8:
- allocatems->device_memoryand removesvms->device_memory_baseandvms->device_memory_storage
- remove (ms->maxram_size>ms->ram_size||ms->ram_slots> 0) and (ms->ram_size> (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check andmaxram_sizealignment checks in this patch--- hw/arm/virt.c | 50 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 1 + 2 files changed, 45 insertions(+), 6 deletions(-)hw/arm/virt.c +11
-5@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -107,8 +108,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +151,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;@@ -1379,7 +1382,30 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE );
hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1398
Quoted 10 lines
hw/arm/virt.c +10
-0+ } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB);
s/RAMBASE/vms->memmap[VIRT_MEM].base/
Quoted 60 lines
hw/arm/virt.c +23
-1+ device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + + vms->high_io_base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (vms->high_io_base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {@@ -1390,6 +1416,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)@@ -1476,7 +1509,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }@@ -1570,6 +1604,10 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (machine->device_memory) { +memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr); + }create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);include/hw/arm/virt.h +1
-0@@ -132,6 +132,7 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v10 8/10 hw/arm/virt: Implement kvm_type function for 4.0 machine
In reply to Eric AugerThis patch implements the machine class kvm_type() callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.
To compute the highest GPA used in the memory map, kvm_type() must freeze the memory map by calling virt_set_memmap().
---
v7 -> v8:
- remove vmc->no_extended_memmap and vms->extended_memmap
v6 -> v7:
- Introduce RAMBASE and rename add LEGACY_ prefix in that patch
- use local variables with explicit names in virt_set_memmap:
device_memory_base, device_memory_size
- add an extended_memmap field in the class
v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB
--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
hw/arm/virt.c +38 -1
@@ -1439,7 +1439,13 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type() +* to create a VM with the right number of IPA bits. +*/ +if (!kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet
@@ -1828,6 +1834,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -1852,6 +1888,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v10 08/10] hw/arm/virt: Implement kvm_type function for 4.0 machine
Quoted 33 lines · On Thu, 28 Feb 2019 16:03:22 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.To compute the highest GPA used in the memory map,
kvm_type()must freeze the memory map by callingvirt_set_memmap().Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- removevmc->no_extended_memmapandvms->extended_memmapv6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)hw/arm/virt.c +0
-0
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1443
Quoted 6 lines
hw/arm/virt.c +2
-1@@ -1439,7 +1439,13 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type()
s/in/earlier in/
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1446
Quoted 3 lines
hw/arm/virt.c +3
-0+* to create a VM with the right number of IPA bits. +*/ +if (!kvm_enabled ()) {
maybe check not for KVM but for vms->memmap == NULL
so it would match with comment and be clear that we initializing memmap because nobody else did it yet.
Quoted 22 lines
hw/arm/virt.c +15
-0+virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1828,6 +1834,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); +
hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1859
Quoted 10 lines
hw/arm/virt.c +9
-0+ requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + }
this part is recurring pattern, is it worth using a helper function?
Quoted 19 lines
hw/arm/virt.c +8
-0+/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc = MACHINE_CLASS(oc);hw/arm/virt.c +1
-0@@ -1852,6 +1888,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
Re: [Qemu-devel] [PATCH v10 08/10] hw/arm/virt: Implement kvm_type function for 4.0 machine
Hi Igor,
Quoted 83 lines · Igor Mammedov
Quoted 33 lines · On Thu, 28 Feb 2019 16:03:22 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.To compute the highest GPA used in the memory map,
kvm_type()must freeze the memory map by callingvirt_set_memmap().Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- removevmc->no_extended_memmapandvms->extended_memmapv6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)hw/arm/virt.c +0
-0hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1443
Quoted 6 lines
hw/arm/virt.c +2
-1@@ -1439,7 +1439,13 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type() s/in/earlier in/
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1446
Quoted 3 lines
hw/arm/virt.c +3
-0+* to create a VM with the right number of IPA bits. +*/ +if (!kvm_enabled ()) {maybe check not for KVM but for
vms->memmap== NULL
so it would match with comment and be clear that we initializing memmap because nobody else did it yet.Quoted 22 lines
hw/arm/virt.c +15
-0+virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1828,6 +1834,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); +hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1859
Quoted 10 lines
hw/arm/virt.c +9
-0+ requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + }
Quoted 1 line
this part is recurring pattern, is it worth using a helper function?
in TCG mode the message is different because this issue stems from the fact the VCPU does not support a large enough PA range whereas here the host kernel does not support large enough IPA. So I tend to keep the code as is.
Thanks
Eric
Quoted 21 lines
Quoted 20 lines
hw/arm/virt.c +8
-0+/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc = MACHINE_CLASS(oc);hw/arm/virt.c +1
-0@@ -1852,6 +1888,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
PATCH v10 9/10 hw/arm/virt: Check the VCPU PA range in TCG mode
In reply to Eric AugerWe are about to allow the memory map to grow beyond 1TB and potentially overshoot the VCPU AA64MMFR0.PARANGE.
In aarch64 mode and when highmem is set, let's check the VCPU PA range is sufficient to address the highest GPA of the memory map.
--- hw/arm/virt.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
hw/arm/virt.c +17 -0
@@ -60,6 +60,7 @@#include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" #include "hw/acpi/acpi.h" +#include "target/arm/internals.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1607,6 +1608,22 @@ static void machvirt_init(MachineState *machine)fdt_add_timer_nodes (vms);fdt_add_cpu_nodes (vms); +if (!kvm_enabled ()) { +ARMCPU *cpu =ARM_CPU (first_cpu); +bool aarch64 =object_property_get_bool (OBJECT (cpu),"aarch64" ,NULL ); + +if (aarch64 && vms->highmem) { +int requested_pa_size, pamax =arm_pamax (cpu); + + requested_pa_size =64 -clz64 (vms->highest_gpa); +if (pamax < requested_pa_size) { +error_report ("VCPU supports less PA bits (%d) than requested " +"by the memory map (%d)" , pamax, requested_pa_size); +exit (1 ); + } + } + } +memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
Signature · 1 line
2.20.1
PATCH v10 10/10 hw/arm/virt: Bump the 255GB initial RAM limit
In reply to Eric AugerNow we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
---
v7 -> v8:
- TCG PAMAX check moved in a separate patch
v6 -> v7
- handle TCG case
- set_memmap modifications moved to previous patches
--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
hw/arm/virt.c +1 -20
@@ -95,21 +95,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces -* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
@@ -1515,12 +1502,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v10 10/10] hw/arm/virt: Bump the 255GB initial RAM limit
Quoted 36 lines · On Thu, 28 Feb 2019 16:03:24 +0100 Eric Auger <[email protected]> wrote:
Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- TCG PAMAX check moved in a separate patchv6 -> v7
- handle TCG case
-set_memmapmodifications moved to previous patches--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)hw/arm/virt.c +0
-8@@ -95,21 +95,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces
hw/arm/virt.c line 101
Quoted 10 lines
hw/arm/virt.c +1
-6-* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
do we need to keep these couple around?
it's used only in
[VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES },
and doesn't have any effect whatsoever.
I'd set initial VIRT_MEM.size to 0 and drop LEGACY_RAMLIMIT_* maybe add comment above entry that size is defined by ram_size
Quoted 14 lines
hw/arm/virt.c +0
-6@@ -1515,12 +1502,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
Re: [Qemu-arm] [PATCH v10 10/10] hw/arm/virt: Bump the 255GB initial RAM limit
Hi Igor,
Quoted 51 lines · Igor Mammedov
Quoted 36 lines · On Thu, 28 Feb 2019 16:03:24 +0100 Eric Auger <[email protected]> wrote:
Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- TCG PAMAX check moved in a separate patchv6 -> v7
- handle TCG case
-set_memmapmodifications moved to previous patches--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)hw/arm/virt.c +0
-8@@ -95,21 +95,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces hw/arm/virt.c line 101
Quoted 10 lines
hw/arm/virt.c +1
-6-* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)do we need to keep these couple around?
Quoted 5 lines
it's used only in
[VIRT_MEM] = { RAMBASE,LEGACY_RAMLIMIT_BYTES},
and doesn't have any effect whatsoever.
I'd set initial VIRT_MEM.size to 0 and drop LEGACY_RAMLIMIT_* maybe add comment above entry that size is defined byram_size
in virt_set_memmap I was checking if (high_io_base < 256 GiB) then high_io_base = 256GiB. Maybe this 256GiB value comes out of the blue and I should also replace it with vms->memmap[VIRT_MEM].base + LEGACY_RAMLIMIT_BYTES.
We maintain some kind of compatibility with the old memmap so I prefer to keep this info somewhere.
I added a comment though:
/* Actual RAM size depends on initial RAM and device memory options */
Thanks
Eric
Quoted 16 lines
Quoted 15 lines
hw/arm/virt.c +0
-6@@ -1515,12 +1502,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
Re: [Qemu-arm] [PATCH v10 10/10] hw/arm/virt: Bump the 255GB initial RAM limit
Quoted 60 lines · On Fri, 1 Mar 2019 11:08:17 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 51 lines · Igor Mammedov
Quoted 36 lines · On Thu, 28 Feb 2019 16:03:24 +0100 Eric Auger <[email protected]> wrote:
Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- TCG PAMAX check moved in a separate patchv6 -> v7
- handle TCG case
-set_memmapmodifications moved to previous patches--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)hw/arm/virt.c +0
-8@@ -95,21 +95,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces hw/arm/virt.c line 101
Quoted 10 lines
hw/arm/virt.c +1
-6-* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)do we need to keep these couple around?
Quoted 5 lines
it's used only in
[VIRT_MEM] = { RAMBASE,LEGACY_RAMLIMIT_BYTES},
and doesn't have any effect whatsoever.
I'd set initial VIRT_MEM.size to 0 and drop LEGACY_RAMLIMIT_* maybe add comment above entry that size is defined byram_size
Quoted 4 lines
in
virt_set_memmapI was checking if (high_io_base< 256 GiB) thenhigh_io_base= 256GiB. Maybe this 256GiB value comes out of the blue and I should also replace it with vms->memmap[VIRT_MEM].base +LEGACY_RAMLIMIT_BYTES.
I'd go for it + comment on top of it.
Quoted 27 lines
We maintain some kind of compatibility with the old memmap so I prefer to keep this info somewhere.
I added a comment though:
/* Actual RAM size depends on initial RAM and device memory options */Thanks
Eric
Quoted 16 lines
Quoted 15 lines
hw/arm/virt.c +0
-6@@ -1515,12 +1502,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );