ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support
v8 cover ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size ram_size and then comes the device memory (,maxmem) of size maxram_size - ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on ram_size and maxram_size. Size alignment is enforced.
In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 10] and change in the memory map
2) Support of PC-DIMM [11 - 13]
3) Support of NV-DIMM [14 - 18]
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.
So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
DIMM hotplug in not supported in the series.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/
[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.html
This series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v8
History:
v7 -> v8:
- addressed Igor's comments
- remove no_extended_memmap knob
- moved acpi_nvdimm_state into MachineState
- separate "hw/arm/virt: Check the VCPU PA range in TCG mode" patch
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more details
v5 -> v6:
- mingw compilation issue fix
- kvm_arm_get_max_vm_phys_shift always returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization into machvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their size
v4 -> v5:
- change in the memory map
- see individual logs
v3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm: pre_plug "slot" and "addr" assignment"
- kvm-type option not used anymore. We directly use maxram_size and ram_size machine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
- device_memory_start and device_memory_size not stored anymore in vms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring of arm_load_dtb and the one related to the generation of the dtb after system_reset which would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machine ram_size, maxram_size and ram_slots earlier"
v2 -> v3:
- fix pc_q35 and pc_piix compilation error
- kwangwoo's email being not valid anymore, remove his address
v1 -> v2:
- kvm_get_max_vm_phys_shift moved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoring
v1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"
Best Regards
Eric
Eric Auger (13):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter to kvm_type callback
kvm: add kvm_arm_get_max_vm_ipa_size
vl: Set machine ram_size, maxram_size and ram_slots earlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implement kvm_type function for 4.0 machine hw/arm/virt: Check the VCPU PA range in TCG mode hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework
machine: Move acpi_nvdimm_state into struct MachineState hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Allow nvdimm instantiation
Kwangwoo Lee (2):
nvdimm: Use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructure
Shameer Kolothum (3):
hw/arm/boot: introduce fdt_add_memory_node helper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRAT
accel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 30 +++- hw/arm/boot.c | 136 ++++++++++++--- hw/arm/virt-acpi-build.c | 25 ++- hw/arm/virt.c | 285 +++++++++++++++++++++++++++----- hw/core/machine.c | 57 +++++++ hw/i386/acpi-build.c | 12 +- hw/i386/acpi-build.h | 3 + hw/i386/pc.c | 56 +------ hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 19 ++- include/hw/boards.h | 7 +- include/hw/i386/pc.h | 4 - include/hw/mem/nvdimm.h | 3 + include/hw/mem/pc-dimm.h | 1 - target/arm/kvm.c | 10 ++ target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 23 files changed, 540 insertions(+), 152 deletions(-)
Signature · 1 line
2.20.1
PATCH v8 1/18 hw/arm/boot: introduce fdt_add_memory_node helper
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
---
v7 -> v8:
- Added Igor's R-b
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID
--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +34 -20
@@ -423,6 +423,32 @@ static void set_kernel_args_old(const struct arm_boot_info *info, } } +static int fdt_add_memory_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -502,7 +528,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,void *fdt =NULL ;int size, rc, n =0 ;uint32_t acells, scells; -char *nodename;unsigned int i;hwaddr mem_base, mem_len;char **node_path;
@@ -576,35 +601,24 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start;for (i =0 ; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; - nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, mem_base, - scells, mem_len); + rc =fdt_add_memory_node (fdt, acells, mem_base, + scells, mem_len, i);if (rc <0 ) { -fprintf (stderr,"couldn't set %s/reg for node %d\n" , nodename, - i); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + mem_base); goto fail; } -qemu_fdt_setprop_cell (fdt, nodename,"numa-node-id" , i); mem_base += mem_len; -g_free (nodename); } }else { - nodename =g_strdup_printf ("/memory@%" PRIx64, binfo->loader_start); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, binfo->loader_start, - scells, binfo->ram_size); + rc =fdt_add_memory_node (fdt, acells, binfo->loader_start, + scells, binfo->ram_size,-1 );if (rc <0 ) { -fprintf (stderr,"couldn't set %s reg\n" , nodename); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + binfo->loader_start); goto fail; } -g_free (nodename); } rc =fdt_path_offset (fdt,"/chosen" );
Signature · 1 line
2.20.1
PATCH v8 2/18 hw/arm/virt: Rename highmem IO regions
In reply to Eric AugerIn preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM
---
v8:
- added Igor's R-b and fixed "line over 80 characters" warnings
- nit: potential indent issue reported by Igor looks a false positive to me
v7: added Peter's R-b
v6: creation
--- hw/arm/virt-acpi-build.c | 10 ++++++---- hw/arm/virt.c | 33 ++++++++++++++++++--------------- include/hw/arm/virt.h | 8 ++++---- 3 files changed, 28 insertions(+), 23 deletions(-)
hw/arm/virt-acpi-build.c +6 -4
@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, size_pio));if (use_highmem) { -hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].size; aml_append(rbuf, aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -663,8 +663,10 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicr =acpi_data_push (table_data,sizeof (*gicr)); gicr->type =ACPI_APIC_GENERIC_REDISTRIBUTOR ; gicr->length =sizeof (*gicr); - gicr->base_address =cpu_to_le64 (memmap[VIRT_GIC_REDIST2 ].base); - gicr->range_length =cpu_to_le32 (memmap[VIRT_GIC_REDIST2 ].size); + gicr->base_address = +cpu_to_le64 (memmap[VIRT_HIGH_GIC_REDIST2 ].base); + gicr->range_length = +cpu_to_le32 (memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (its_class_name () && !vmc->no_its) {
hw/arm/virt.c +18 -15
@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM ] = {0x40000000 ,RAMLIMIT_BYTES }, /* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_PCIE_ECAM_HIGH ] = {0x4010000000ULL ,0x10000000 }, + [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, + [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_PCIE_MMIO_HIGH] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, };static const int a15irqmap[] = {
@@ -431,12 +431,12 @@ static void fdt_add_gic_node(VirtMachineState *vms) 2, vms->memmap[VIRT_GIC_REDIST].size); }else {qemu_fdt_setprop_sized_cells (vms->fdt, nodename,"reg" , -2 , vms->memmap[VIRT_GIC_DIST ].base, -2 , vms->memmap[VIRT_GIC_DIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST ].base, -2 , vms->memmap[VIRT_GIC_REDIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST2 ].base, -2 , vms->memmap[VIRT_GIC_REDIST2 ].size); +2 , vms->memmap[VIRT_GIC_DIST ].base, +2 , vms->memmap[VIRT_GIC_DIST ].size, +2 , vms->memmap[VIRT_GIC_REDIST ].base, +2 , vms->memmap[VIRT_GIC_REDIST ].size, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base, +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (vms->virt) {
@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (nb_redist_regions ==2 ) {uint32_t redist1_capacity = - vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ;qdev_prop_set_uint32 (gicdev,"redist-region-count[1]" ,MIN (smp_cpus - redist0_count, redist1_capacity));
@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (type ==3 ) {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_REDIST ].base);if (nb_redist_regions ==2 ) { -sysbus_mmio_map (gicbusdev,2 , vms->memmap[VIRT_GIC_REDIST2 ].base); +sysbus_mmio_map (gicbusdev,2 , + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base); } }else {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_CPU ].base);
@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO ].size; -hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].size;hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO ].base;hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO ].size;hwaddr base_ecam, size_ecam;
@@ -1417,8 +1418,10 @@ static void machvirt_init(MachineState *machine)* many redistributors we can fit into the memory map. */ if (vms->gic_version ==3 ) { - virt_max_cpus = vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; - virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + virt_max_cpus = + vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; + virt_max_cpus += + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; }else { virt_max_cpus =GIC_NCPU ; }
include/hw/arm/virt.h +4 -4
@@ -64,7 +64,7 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_GIC_REDIST2 , +VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,9 +74,9 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_PCIE_ECAM_HIGH , +VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_PCIE_MMIO_HIGH , +VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART , VIRT_SECURE_MEM,
@@ -128,7 +128,7 @@ typedef struct {int psci_conduit; } VirtMachineState; -#define VIRT_ECAM_ID (high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM) +#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)#define TYPE_VIRT_MACHINE MACHINE_TYPE_NAME("virt")#define VIRT_MACHINE (obj) \
Signature · 1 line
2.20.1
PATCH v8 3/18 hw/arm/virt: Split the memory map description
In reply to Eric AugerIn the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap, renamed base_memmap, contains regions below and including the RAM. MemMapEntries initialized in this array have a static size and base address.
- extended_memmap, only initialized with entries located after the RAM. MemMapEntries initialized in this array only get their size initialized. Their base address is dynamically computed depending on the the top of the RAM, with same alignment as their size.
Eventually base_memmap entries are copied into the extended_memmap array. Using two separate arrays however clarifies which entries are statically allocated and those which are dynamically allocated.
This new split will allow to grow the RAM size without changing the description of the high IO entries.
We introduce a new virt_set_memmap() helper function which "freezes" the memory map.
The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
---
v7 -> v8:
- removed Peter's R-b due to the changes induced by Igor's comments
- call set_memmap back in virt_instance_init
- add comments about sizing of extended_memmap
- s/region/entries as suggested by Igor
- rewording of the commit message
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-b
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments
--- hw/arm/virt.c | 51 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 14 ++++++++---- 2 files changed, 55 insertions(+), 10 deletions(-)
hw/arm/virt.c +45 -6
@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h"
@@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
@@ -149,11 +150,24 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each MemMapEntry base (GPA) will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB entry will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +* Note the extended_memmap is sized so that it eventually also includes the +* base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last +* index of base_memmap). +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
@@ -1354,6 +1368,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);
@@ -1845,7 +1883,8 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; +virt_set_memmap (vms); + vms->irqmap = a15irqmap; }
include/hw/arm/virt.h +10 -4
@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {
@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;
@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
PATCH v8 4/18 hw/boards: Add a MachineState parameter to kvm_type callback
In reply to Eric AugerOn ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.
---
v7 -> v8: added Igor's R-b
v6 -> v7:
- add a comment for kvm_type
- use machine instead of ms in the declaration
- add Peter's R-b
--- accel/kvm/kvm-all.c | 2 +- hw/ppc/mac_newworld.c | 3 +-- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/boards.h | 5 ++++- 5 files changed, 8 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +1 -1
@@ -1593,7 +1593,7 @@ static int kvm_init(MachineState *ms) kvm_type =qemu_opt_get (qemu_get_machine_opts (),"kvm-type" );if (mc->kvm_type) { - type = mc->kvm_type (kvm_type); + type = mc->kvm_type (ms, kvm_type); }else if (kvm_type) { ret = -EINVAL ;fprintf (stderr,"Invalid argument kvm-type=%s\n" , kvm_type);
hw/ppc/mac_newworld.c +1 -2
@@ -564,8 +564,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } - -static int core99_kvm_type (const char *arg) +static int core99_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/mac_oldworld.c +1 -1
@@ -420,7 +420,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } -static int heathrow_kvm_type (const char *arg) +static int heathrow_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/spapr.c +1 -1
@@ -2920,7 +2920,7 @@ static void spapr_machine_init(MachineState *machine) } } -static int spapr_kvm_type (const char *vm_type) +static int spapr_kvm_type (MachineState *machine,const char *vm_type) {if (!vm_type) {return 0 ;
include/hw/boards.h +4 -1
@@ -156,6 +156,9 @@ typedef struct {* should instead use "unimplemented-device" for all memory ranges where * the guest will attempt to probe for a device that QEMU doesn't * implement and a stub device is required. +* @kvm_type: +* Return the type of KVM corresponding to the kvm-type string option or +* computed based on other criteria such as the host kernel capabilities. */ struct MachineClass {/*< private >*/
@@ -171,7 +174,7 @@ struct MachineClass {void (*init)(MachineState *state);void (*reset)(void );void (*hot_add_cpu)(const int64_t id,Error **errp); -int (*kvm_type)(const char *arg); +int (*kvm_type)(MachineState *machine,const char *arg);BlockInterfaceType block_default_type;int units_per_default_bus;
Signature · 1 line
2.20.1
PATCH v8 5/18 kvm: add kvm_arm_get_max_vm_ipa_size
In reply to Eric AugerAdd the kvm_arm_get_max_vm_ipa_size() helper that returns the number of bits in the IPA address space supported by KVM.
This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.
---
v6 -> v7:
- s/kvm_arm_get_max_vm_phys_shift/kvm_arm_get_max_vm_ipa_size
- reword the comment
v4 -> v5:
- return 40 if the host does not support the capability
v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension
v1 -> v2:
- put this in ARM specific code
--- target/arm/kvm.c | 10 ++++++++++ target/arm/kvm_arm.h | 13 +++++++++++++ 2 files changed, 23 insertions(+)
target/arm/kvm.c +10 -0
@@ -18,6 +18,7 @@#include "qemu/error-report.h" #include "sysemu/sysemu.h" #include "sysemu/kvm.h" +#include "sysemu/kvm_int.h" #include "kvm_arm.h" #include "cpu.h" #include "trace.h"
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) env->features = arm_host_cpu_features.features; } +int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +KVMState *s =KVM_STATE (ms->accelerator); +int ret; + + ret =kvm_check_extension (s,KVM_CAP_ARM_VM_IPA_SIZE ); +return ret >0 ? ret :40 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +13 -0
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);*/ void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu); +/** +* kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the +* IPA address space supported by KVM +* +* @ms: Machine state handle +*/ +int kvm_arm_get_max_vm_ipa_size (MachineState *ms); +/** * kvm_arm_sync_mpstate_to_kvm * @cpu: ARMCPU
@@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) cpu->host_cpu_probe_failed = true; } +static inline int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +return -ENOENT ; +} +static inline int kvm_arm_vgic_probe (void ) {return 0 ;
Signature · 1 line
2.20.1
PATCH v8 6/18 vl: Set machine ram_size, maxram_size and ram_slots earlier
In reply to Eric AugerThe machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.
---
v7 -> v8:
- added Igor's R-b
v6 -> v7:
- add Peter's R-b
v4: new
--- vl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +3 -3
@@ -4239,6 +4239,9 @@ int main(int argc, char **argv, char **envp) machine_opts =qemu_get_machine_opts ();qemu_opt_foreach (machine_opts, machine_set_property, current_machine, &error_fatal); + current_machine->ram_size = ram_size; + current_machine->maxram_size = maxram_size; + current_machine->ram_slots = ram_slots;configure_accelerator (current_machine, argv[0 ]);
@@ -4434,9 +4437,6 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); - current_machine->ram_size = ram_size; - current_machine->maxram_size = maxram_size; - current_machine->ram_slots = ram_slots; current_machine->boot_order = boot_order;/* parse features once if machine provides default cpu_type */
Signature · 1 line
2.20.1
PATCH v8 7/18 hw/arm/virt: Dynamic memory map depending on RAM requirements
In reply to Eric AugerUp to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the virt_set_memmap() function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.
The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.
Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB.
---
v7 -> v8:
- allocate ms->device_memory and removes vms->device_memory_base and vms->device_memory_storage
- remove (ms->maxram_size > ms->ram_size || ms->ram_slots > 0) and (ms->ram_size > (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check and maxram_size alignment checks in this patch
--- hw/arm/virt.c | 52 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 1 + 2 files changed, 47 insertions(+), 6 deletions(-)
hw/arm/virt.c +46 -6
@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -107,8 +108,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
@@ -149,7 +151,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/*
@@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;
@@ -1379,7 +1382,34 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); + } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + + vms->high_io_base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (vms->high_io_base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {
@@ -1390,6 +1420,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)
@@ -1474,7 +1511,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }
@@ -1568,6 +1606,8 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr);create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
include/hw/arm/virt.h +1 -0
@@ -132,6 +132,7 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [Qemu-devel] [PATCH v8 07/18] hw/arm/virt: Dynamic memory map depending on RAM requirements
Hi,
Quoted 148 lines · Eric Auger
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory,
- the high IO region base
- the highest GPA used in the memory map.Entries of the high IO region are assigned a base address. The device memory is initialized.
The highest GPA used in the memory map will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB.
Signed-off-by Eric Auger <[email protected]>---
v7 -> v8:
- allocatems->device_memoryand removesvms->device_memory_baseandvms->device_memory_storage
- remove (ms->maxram_size>ms->ram_size||ms->ram_slots> 0) and (ms->ram_size> (ram_addr_t)LEGACY_RAMLIMIT_BYTES) checks
- initialize the device memory
- move the slots nb check andmaxram_sizealignment checks in this patch--- hw/arm/virt.c | 52 ++++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 1 + 2 files changed, 47 insertions(+), 6 deletions(-)hw/arm/virt.c +44
-6@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -107,8 +108,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +151,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1370,7 +1372,8 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { -hwaddr base; +MachineState *ms =MACHINE (vms); +hwaddr base, device_memory_base, device_memory_size;int i; vms->memmap = extended_memmap;@@ -1379,7 +1382,34 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); + } + +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; + + vms->high_io_base = device_memory_base +ROUND_UP (device_memory_size, GiB); +if (vms->high_io_base < device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {@@ -1390,6 +1420,13 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; +if (device_memory_size >0 ) { + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = device_memory_base; +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , device_memory_size); + } }static void machvirt_init (MachineState *machine)@@ -1474,7 +1511,8 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus;if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB );exit (1 ); }
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1608
Quoted 3 lines
hw/arm/virt.c +0
-0@@ -1568,6 +1606,8 @@ static void machvirt_init(MachineState *machine)memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
Argh, missing a
if (machine->device_memory) {}
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1610
Quoted 2 lines
hw/arm/virt.c +2
-0+memory_region_add_subregion (sysmem, machine->device_memory->base, + &machine->device_memory->mr);
Respinning ...
Thanks
Eric
Quoted 16 lines
hw/arm/virt.c +0
-0create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);include/hw/arm/virt.h +1
-0@@ -132,6 +132,7 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v8 8/18 hw/arm/virt: Implement kvm_type function for 4.0 machine
In reply to Eric AugerThis patch implements the machine class kvm_type() callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.
To compute the highest GPA used in the memory map, kvm_type() must freeze the memory map by calling virt_set_memmap().
---
v7 -> v8:
- remove vmc->no_extended_memmap and vms->extended_memmap
v6 -> v7:
- Introduce RAMBASE and rename add LEGACY_ prefix in that patch
- use local variables with explicit names in virt_set_memmap:
device_memory_base, device_memory_size
- add an extended_memmap field in the class
v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB
--- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
hw/arm/virt.c +38 -1
@@ -1828,6 +1828,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -1852,6 +1882,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
@@ -1923,7 +1954,13 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type() +* to create a VM with the right number of IPA bits. +*/ +if (!kvm_enabled ()) { +virt_set_memmap (vms); + } vms->irqmap = a15irqmap; }
Signature · 1 line
2.20.1
PATCH v8 9/18 hw/arm/virt: Check the VCPU PA range in TCG mode
In reply to Eric AugerWe are about to allow the memory map to grow beyond 1TB and potentially overshoot the VCPU AA64MMFR0.PARANGE.
In aarch64 mode and when highmem is set, let's check the VCPU PA range is sufficient to address the highest GPA of the memory map.
--- hw/arm/virt.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
hw/arm/virt.c +17 -0
@@ -60,6 +60,7 @@#include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" #include "hw/acpi/acpi.h" +#include "target/arm/internals.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1603,6 +1604,22 @@ static void machvirt_init(MachineState *machine)fdt_add_timer_nodes (vms);fdt_add_cpu_nodes (vms); +if (!kvm_enabled ()) { +ARMCPU *cpu =ARM_CPU (first_cpu); +bool aarch64 =object_property_get_bool (OBJECT (cpu),"aarch64" ,NULL ); + +if (aarch64 && vms->highmem) { +int requested_pa_size, pamax =arm_pamax (cpu); + + requested_pa_size =64 -clz64 (vms->highest_gpa); +if (pamax < requested_pa_size) { +error_report ("VCPU supports less PA bits (%d) than requested " +"by the memory map (%d)" , pamax, requested_pa_size); +exit (1 ); + } + } + } +memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
Signature · 1 line
2.20.1
PATCH v8 10/18 hw/arm/virt: Bump the 255GB initial RAM limit
In reply to Eric AugerNow we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel, in accelerated mode. In TCG mode, it depends on the VCPU AA64MMFR0.PARANGE.
---
v7 -> v8:
- TCG PAMAX check moved in a separate patch
v6 -> v7
- handle TCG case
- set_memmap modifications moved to previous patches
--- hw/arm/virt.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
hw/arm/virt.c +1 -20
@@ -95,21 +95,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces -* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
@@ -1511,12 +1498,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
Signature · 1 line
2.20.1
PATCH v8 11/18 hw/arm/virt: Add memory hotplug framework
In reply to Eric AugerThis patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.
---
v7 -> v8:
- fix indent issue reported by Igor and added his R-b
v4 -> v5:
- change in pc_dimm_pre_plug signature
- CONFIG_MEM_HOTPLUG replaced by CONFIG_MEM_DEVICE and CONFIG_DIMM
v3 -> v4:
- check the memory device is not hotplugged
v2 -> v3:
- change in pc_dimm_plug()'s signature
- add pc_dimm_pre_plug call
v1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- added pre_plug cb
- don't handle get_memory_region failure anymore
--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt.c | 64 ++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
default-configs/arm-softmmu.mak +2 -0
@@ -164,3 +164,5 @@ CONFIG_PCI_EXPRESS_DESIGNWARE=y CONFIG_STRONGARM=y CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y +CONFIG_MEM_DEVICE=y +CONFIG_DIMM=y
hw/arm/virt.c +63 -1
94 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v8 12/18 hw/arm/boot: Expose the PC-DIMM nodes in the DT
In reply to Eric AugerFrom: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
NVDIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NVDIMM at this stage.
---
v7 -> v8:
- s/NV_DIMM/NVDIMM
- fix indent
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes
v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop
--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
hw/arm/boot.c +42 -0
@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h"
@@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {
@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail; + } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v8 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT
Quoted 10 lines · On Tue, 26 Feb 2019 14:50:08 +0100 Eric Auger <[email protected]> wrote:
From: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
NVDIMM and
ACPI_NVDIMMconfigs are not yet set for ARM so we don't need to care about NVDIMM at this stage.Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Igor Mammedov <[email protected]>
I'm having second thoughts on this one,
maybe we should drop this patch or at least not to do this by default. In the later case it would mean some kind of property that user could use to turn this on which in its own turn adds more clutter to user tunables (which is not desirable). So I'm leaning more towards dropping this patch for now.
Quoted 83 lines
---
v7 -> v8:
- s/NV_DIMM/NVDIMM
- fix indent
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len infdt_add_hotpluggable_memory_nodesv1 -> v2:
- addedqapi_free_MemoryDeviceInfoListand simplify the loop--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)hw/arm/boot.c +42
-0@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h" @@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail; + } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
Re: [Qemu-arm] [PATCH v8 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT
Hi Igor,
Quoted 7 lines · Igor Mammedov
Quoted 4 lines · On Tue, 26 Feb 2019 14:50:08 +0100 Eric Auger <[email protected]> wrote:
From: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
Quoted 8 lines
Quoted 6 lines
NVDIMM and
ACPI_NVDIMMconfigs are not yet set for ARM so we don't need to care about NVDIMM at this stage.Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Igor Mammedov <[email protected]>I'm having second thoughts on this one,
maybe we should drop this patch or at least not to do this by default.
See my reply on your cover letter remark.
Quoted 3 lines
In the later case it would mean some kind of property that user could use to turn this on which in its own turn adds more clutter to user tunables (which is not desirable). So I'm leaning more towards dropping this patch for now.
So what about DT support? Doesn't it make sense to support cold-pluggable DIMMS?
Thanks
Eric
Quoted 85 lines
Quoted 84 lines
---
v7 -> v8:
- s/NV_DIMM/NVDIMM
- fix indent
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len infdt_add_hotpluggable_memory_nodesv1 -> v2:
- addedqapi_free_MemoryDeviceInfoListand simplify the loop--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)hw/arm/boot.c +42
-0@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h" @@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail; + } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
Re: [Qemu-arm] [PATCH v8 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT
Quoted 13 lines · On Tue, 26 Feb 2019 18:59:49 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 10 lines · Igor Mammedov
Quoted 7 lines · On Tue, 26 Feb 2019 14:50:08 +0100 Eric Auger <[email protected]> wrote:
From: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
NVDIMM and
ACPI_NVDIMMconfigs are not yet set for ARM so we don't need to care about NVDIMM at this stage.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Eric Auger <[email protected]>Reviewed-by Igor Mammedov <[email protected]>I'm having second thoughts on this one,
maybe we should drop this patch or at least not to do this by default.See my reply on your cover letter remark.
Replied there.
Quoted 5 lines
Quoted 3 lines
In the later case it would mean some kind of property that user could use to turn this on which in its own turn adds more clutter to user tunables (which is not desirable). So I'm leaning more towards dropping this patch for now.
So what about DT support? Doesn't it make sense to support cold-pluggable DIMMS?
It would have a sense if initial memory was also dimms, in that case for static cases it would be possible to describe numa mapping using only -device, but we are far from being ale to do it yet.
So right now, I'm not sure if is there are a compelling reasons for cold-plugged dimms, at startup time initial RAM is sufficient to do the job. Main goal was to introduce most of infrastructure so that it would be easier to add hotplug on top.
Quoted 89 lines
Thanks
Eric
Quoted 85 lines
Quoted 84 lines
---
v7 -> v8:
- s/NV_DIMM/NVDIMM
- fix indent
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len infdt_add_hotpluggable_memory_nodesv1 -> v2:
- addedqapi_free_MemoryDeviceInfoListand simplify the loop--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)hw/arm/boot.c +42
-0@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h" @@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail; + } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
PATCH v8 13/18 hw/arm/virt-acpi-build: Add PC-DIMM in SRAT
In reply to Eric AugerFrom: Shameer Kolothum <[email protected]>
Generate Memory Affinity Structures for PC-DIMM ranges.
---
v6 -> v7:
- add Igor's R-b
v5 -> v6:
- fix mingw compil issue
v4 -> v5:
- Align to x86 code and especially
"pc: acpi: revert back to 1 SRAT entry for hotpluggable area"
v3 -> v4:
- do not use vms->bootinfo.device_memory_start/device_memory_size anymore
v1 -> v2:
- build_srat_hotpluggable_memory movedc to aml-build
--- hw/arm/virt-acpi-build.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/arm/virt-acpi-build.c +9 -0
@@ -516,6 +516,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)int i, srat_start;uint64_t mem_base;MachineClass *mc =MACHINE_GET_CLASS (vms); +MachineState *ms =MACHINE (vms);const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids (MACHINE (vms)); srat_start = table_data->len;
@@ -541,6 +542,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } } +if (ms->device_memory) { + numamem =acpi_data_push (table_data,sizeof *numamem); +build_srat_memory (numamem, ms->device_memory->base, +memory_region_size (&ms->device_memory->mr), + nb_numa_nodes -1 , +MEM_AFFINITY_HOTPLUGGABLE |MEM_AFFINITY_ENABLED ); + } +build_header (linker, table_data, (void *)(table_data->data + srat_start),"SRAT" , table_data->len - srat_start,3 ,NULL ,NULL ); }
Signature · 1 line
2.20.1
PATCH v8 14/18 nvdimm: Use configurable ACPI IO base and size
In reply to Eric AugerFrom: Kwangwoo Lee <[email protected]>
This patch makes IO base and size configurable to create NPIO AML for ACPI NFIT. Since a different architecture like AArch64 does not use port-mapped IO, a configurable IO base is required to create correct mapping of ACPI IO address and size.
---
v7 -> v8:
- s/NvdimmDsmIO/x86_nvdimm_acpi_dsmio
- nvdimm_init_acpi_state() now takes a AcpiGenericAddress arg
v6 -> v7:
- Use NvdimmDsmIO constant
- use AcpiGenericAddress instead of AcpiNVDIMMIOEntry
v2 -> v3:
- s/size/len in pc_piix.c and pc_q35.c
--- hw/acpi/nvdimm.c | 30 +++++++++++++++++++++--------- hw/i386/acpi-build.c | 6 ++++++ hw/i386/acpi-build.h | 3 +++ hw/i386/pc_piix.c | 2 ++ hw/i386/pc_q35.c | 2 ++ include/hw/mem/nvdimm.h | 3 +++ 6 files changed, 37 insertions(+), 9 deletions(-)
hw/acpi/nvdimm.c +21 -9
@@ -926,11 +926,13 @@ void nvdimm_acpi_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev) }void nvdimm_init_acpi_state (AcpiNVDIMMState *state,MemoryRegion *io, +struct AcpiGenericAddress dsm_io,FWCfgState *fw_cfg,Object *owner) { + state->dsm_io = dsm_io;memory_region_init_io (&state->io_mr, owner, &nvdimm_dsm_ops, state, -"nvdimm-acpi-io" ,NVDIMM_ACPI_IO_LEN ); -memory_region_add_subregion (io,NVDIMM_ACPI_IO_BASE , &state->io_mr); +"nvdimm-acpi-io" , dsm_io.bit_width >>3 ); +memory_region_add_subregion (io, dsm_io.address, &state->io_mr); state->dsm_mem =g_array_new (false, true/* clear */ ,1 );acpi_data_push (state->dsm_mem,sizeof (NvdimmDsmIn));
@@ -959,12 +961,14 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,#define NVDIMM_QEMU_RSVD_UUID "648B9CF2-CDA1-4312-8AD9-49C4AF32BD62" -static void nvdimm_build_common_dsm (Aml *dev) +static void nvdimm_build_common_dsm (Aml *dev, +AcpiNVDIMMState *acpi_nvdimm_state) {Aml *method, *ifctx, *function, *handle, *uuid, *dsm_mem, *elsectx2;Aml *elsectx, *unsupport, *unpatched, *expected_uuid, *uuid_invalid;Aml *pckg, *pckg_index, *pckg_buf, *field, *dsm_out_buf, *dsm_out_buf_size;uint8_t byte_list[1 ]; +AmlRegionSpace rs; method =aml_method (NVDIMM_COMMON_DSM ,5 ,AML_SERIALIZED ); uuid =aml_arg (0 );
@@ -975,9 +979,16 @@ static void nvdimm_build_common_dsm(Aml *dev)aml_append (method,aml_store (aml_name (NVDIMM_ACPI_MEM_ADDR ), dsm_mem)); +if (acpi_nvdimm_state->dsm_io.space_id ==AML_AS_SYSTEM_IO ) { + rs =AML_SYSTEM_IO ; + }else { + rs =AML_SYSTEM_MEMORY ; + } +/* map DSM memory and IO into ACPI namespace. */ -aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT ,AML_SYSTEM_IO , -aml_int (NVDIMM_ACPI_IO_BASE ),NVDIMM_ACPI_IO_LEN )); +aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT , rs, +aml_int (acpi_nvdimm_state->dsm_io.address), + acpi_nvdimm_state->dsm_io.bit_width >>3 ));aml_append (method,aml_operation_region (NVDIMM_DSM_MEMORY ,AML_SYSTEM_MEMORY , dsm_mem,sizeof (NvdimmDsmIn)));
@@ -1260,7 +1271,8 @@ static void nvdimm_build_nvdimm_devices(Aml *root_dev, uint32_t ram_slots) }static void nvdimm_build_ssdt (GArray *table_offsets,GArray *table_data, -BIOSLinker *linker,GArray *dsm_dma_arrea, +BIOSLinker *linker, +AcpiNVDIMMState *acpi_nvdimm_state,uint32_t ram_slots) {Aml *ssdt, *sb_scope, *dev;
@@ -1288,7 +1300,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,*/ aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0012" ))); -nvdimm_build_common_dsm (dev); +nvdimm_build_common_dsm (dev, acpi_nvdimm_state);/* 0 is reserved for root device. */ nvdimm_build_device_dsm (dev,0 );
@@ -1307,7 +1319,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data, NVDIMM_ACPI_MEM_ADDR);bios_linker_loader_alloc (linker, -NVDIMM_DSM_MEM_FILE , dsm_dma_arrea, +NVDIMM_DSM_MEM_FILE , acpi_nvdimm_state->dsm_mem,sizeof (NvdimmDsmIn), false/* high memory */ ); bios_linker_loader_add_pointer(linker, ACPI_BUILD_TABLE_FILE, mem_addr_offset, sizeof(uint32_t),
@@ -1329,7 +1341,7 @@ void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data,return ; } -nvdimm_build_ssdt (table_offsets, table_data, linker, state->dsm_mem, +nvdimm_build_ssdt (table_offsets, table_data, linker, state, ram_slots); device_list =nvdimm_get_device_list ();
hw/i386/acpi-build.c +6 -0
@@ -125,6 +125,12 @@ typedef struct FwCfgTPMConfig {uint8_t tpmppi_version; }QEMU_PACKED FwCfgTPMConfig; +const struct AcpiGenericAddress x86_nvdimm_acpi_dsmio = { + .space_id =AML_AS_SYSTEM_IO , + .address =NVDIMM_ACPI_IO_BASE , + .bit_width =NVDIMM_ACPI_IO_LEN <<3 +}; +static void init_common_fadt_data (Object *o,AcpiFadtData *data) {uint32_t io =object_property_get_uint (o,ACPI_PM_PROP_PM_IO_BASE ,NULL );
hw/i386/acpi-build.h +3 -0
@@ -1,6 +1,9 @@#ifndef HW_I386_ACPI_BUILD_H #define HW_I386_ACPI_BUILD_H +#include "hw/acpi/acpi-defs.h" + +extern const struct AcpiGenericAddress x86_nvdimm_acpi_dsmio;void acpi_setup (void );
hw/i386/pc_piix.c +2 -0
@@ -58,6 +58,7 @@#include "migration/misc.h" #include "kvm_i386.h" #include "sysemu/numa.h" +#include "hw/i386/acpi-build.h" #define MAX_IDE_BUS 2
@@ -299,6 +300,7 @@ static void pc_init1(MachineState *machine,if (pcms->acpi_nvdimm_state.is_enabled) {nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, + x86_nvdimm_acpi_dsmio, pcms->fw_cfg,OBJECT (pcms)); } }
hw/i386/pc_q35.c +2 -0
@@ -54,6 +54,7 @@#include "qapi/error.h" #include "qemu/error-report.h" #include "sysemu/numa.h" +#include "hw/i386/acpi-build.h" /* ICH9 AHCI has 6 ports */ #define MAX_SATA_PORTS 6
@@ -331,6 +332,7 @@ static void pc_q35_init(MachineState *machine)if (pcms->acpi_nvdimm_state.is_enabled) {nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, + x86_nvdimm_acpi_dsmio, pcms->fw_cfg,OBJECT (pcms)); } }
include/hw/mem/nvdimm.h +3 -0
@@ -25,6 +25,7 @@#include "hw/mem/pc-dimm.h" #include "hw/acpi/bios-linker-loader.h" +#include "hw/acpi/aml-build.h" #define NVDIMM_DEBUG 0#define nvdimm_debug (fmt, ...) \
@@ -140,10 +141,12 @@ struct AcpiNVDIMMState {*/ int32_t persistence;char *persistence_string; +struct AcpiGenericAddress dsm_io; };typedef struct AcpiNVDIMMState AcpiNVDIMMState ;void nvdimm_init_acpi_state (AcpiNVDIMMState *state,MemoryRegion *io, +struct AcpiGenericAddress dsm_io,FWCfgState *fw_cfg,Object *owner); void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data, BIOSLinker *linker, AcpiNVDIMMState *state,
Signature · 1 line
2.20.1
PATCH v8 15/18 machine: Move acpi_nvdimm_state into struct MachineState
In reply to Eric AugerAs NVDIMM support is looming for ARM and SPAPR, let's move the acpi_nvdimm_state to the generic machine struct instead of duplicating the same code in several machines.
nvdimm and nvdimm-persistence becomes generic machine options. We also add a description for those options.
--- hw/core/machine.c | 57 ++++++++++++++++++++++++++++++++++++++++ hw/i386/acpi-build.c | 6 ++--- hw/i386/pc.c | 56 +++------------------------------------ hw/i386/pc_piix.c | 4 +-- hw/i386/pc_q35.c | 4 +-- include/hw/boards.h | 2 ++ include/hw/i386/pc.h | 4 --- include/hw/mem/pc-dimm.h | 1 - 8 files changed, 70 insertions(+), 64 deletions(-)
hw/core/machine.c +57 -0
@@ -479,6 +479,47 @@ static void machine_set_memory_encryption(Object *obj, const char *value, ms->memory_encryption =g_strdup (value); } +static bool machine_get_nvdimm (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->acpi_nvdimm_state.is_enabled; +} + +static void machine_set_nvdimm (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->acpi_nvdimm_state.is_enabled = value; +} + +static char *machine_get_nvdimm_persistence (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return g_strdup (ms->acpi_nvdimm_state.persistence_string); +} + +static void machine_set_nvdimm_persistence (Object *obj,const char *value, +Error **errp) +{ +MachineState *ms =MACHINE (obj); +AcpiNVDIMMState *nvdimm_state = &ms->acpi_nvdimm_state; + +if (strcmp (value,"cpu" ) ==0 ) + nvdimm_state->persistence =3 ; +else if (strcmp (value,"mem-ctrl" ) ==0 ) + nvdimm_state->persistence =2 ; +else { +error_setg (errp,"-machine nvdimm-persistence=%s: unsupported option" , + value); +return ; + } + +g_free (nvdimm_state->persistence_string); + nvdimm_state->persistence_string =g_strdup (value); +} +void machine_class_allow_dynamic_sysbus_dev (MachineClass *mc,const char *type) {strList *item =g_new0 (strList,1 );
@@ -763,6 +804,20 @@ static void machine_class_init(ObjectClass *oc, void *data) &error_abort);object_class_property_set_description (oc,"memory-encryption" ,"Set memory encryption object to use" , &error_abort); + +object_class_property_add_bool (oc,"nvdimm" , + machine_get_nvdimm, machine_set_nvdimm, &error_abort); +object_class_property_set_description (oc,"nvdimm" , +"Set on/off to enable/disable NVDIMM " +"instantiation" ,NULL ); + +object_class_property_add_str (oc,"nvdimm-persistence" , + machine_get_nvdimm_persistence, + machine_set_nvdimm_persistence, &error_abort); +object_class_property_set_description (oc,"nvdimm-persistence" , +"Set NVDIMM persistence" +"Valid values are cpu and mem-ctrl" , +NULL ); }static void machine_class_base_init (ObjectClass *oc,void *data)
@@ -788,6 +843,8 @@ static void machine_initfn(Object *obj) ms->dump_guest_core = true; ms->mem_merge = true; ms->enable_graphics = true; +/* nvdimm is disabled on default. */ + ms->acpi_nvdimm_state.is_enabled = false;/* Register notifier when init is done for sysbus sanity checks */ ms->sysbus_notifier.notify = machine_init_notify;
hw/i386/acpi-build.c +3 -3
@@ -1873,7 +1873,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,aml_append (scope, method); } -if (pcms->acpi_nvdimm_state.is_enabled) { +if (machine->acpi_nvdimm_state.is_enabled) { method =aml_method ("_E04" ,0 ,AML_NOTSERIALIZED );aml_append (method,aml_notify (aml_name ("\\_SB.NVDR" ),aml_int (0x80 )));
@@ -2710,9 +2710,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)build_dmar_q35 (tables_blob, tables->linker); } } -if (pcms->acpi_nvdimm_state.is_enabled) { +if (machine->acpi_nvdimm_state.is_enabled) {nvdimm_build_acpi (table_offsets, tables_blob, tables->linker, - &pcms->acpi_nvdimm_state, machine->ram_slots); + &machine->acpi_nvdimm_state, machine->ram_slots); }/* Add tables supplied by user (if any) */
hw/i386/pc.c +4 -52
98 lines not on this page · raw message
hw/i386/pc_piix.c +2 -2
@@ -298,8 +298,8 @@ static void pc_init1(MachineState *machine, PC_MACHINE_ACPI_DEVICE_PROP, &error_abort); } -if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +if (machine->acpi_nvdimm_state.is_enabled) { +nvdimm_init_acpi_state (&machine->acpi_nvdimm_state, system_io, x86_nvdimm_acpi_dsmio, pcms->fw_cfg,OBJECT (pcms)); }
hw/i386/pc_q35.c +2 -2
@@ -330,8 +330,8 @@ static void pc_q35_init(MachineState *machine)pc_vga_init (isa_bus, host_bus);pc_nic_init (pcmc, isa_bus, host_bus); -if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +if (machine->acpi_nvdimm_state.is_enabled) { +nvdimm_init_acpi_state (&machine->acpi_nvdimm_state, system_io, x86_nvdimm_acpi_dsmio, pcms->fw_cfg,OBJECT (pcms)); }
include/hw/boards.h +2 -0
@@ -8,6 +8,7 @@#include "hw/qdev.h" #include "qom/object.h" #include "qom/cpu.h" +#include "hw/mem/nvdimm.h" /** * memory_region_allocate_system_memory - Allocate a board's main memory
@@ -271,6 +272,7 @@ struct MachineState {const char *cpu_type;AccelState *accelerator;CPUArchIdList *possible_cpus; +AcpiNVDIMMState acpi_nvdimm_state; };#define DEFINE_MACHINE (namestr, machine_initfn) \
include/hw/i386/pc.h +0 -4
@@ -45,8 +45,6 @@ struct PCMachineState {OnOffAuto vmport;OnOffAuto smm; -AcpiNVDIMMState acpi_nvdimm_state; -bool acpi_build_enabled;bool smbus_enabled;bool sata_enabled;
@@ -74,8 +72,6 @@ struct PCMachineState {#define PC_MACHINE_MAX_RAM_BELOW_4G "max-ram-below-4g"#define PC_MACHINE_VMPORT "vmport"#define PC_MACHINE_SMM "smm" -#define PC_MACHINE_NVDIMM "nvdimm" -#define PC_MACHINE_NVDIMM_PERSIST "nvdimm-persistence"#define PC_MACHINE_SMBUS "smbus"#define PC_MACHINE_SATA "sata"#define PC_MACHINE_PIT "pit"
include/hw/mem/pc-dimm.h +0 -1
@@ -19,7 +19,6 @@#include "exec/memory.h" #include "sysemu/hostmem.h" #include "hw/qdev.h" -#include "hw/boards.h" #define TYPE_PC_DIMM "pc-dimm"#define PC_DIMM (obj) \
Signature · 1 line
2.20.1
PATCH v8 16/18 hw/arm/virt: Add nvdimm hot-plug infrastructure
In reply to Eric AugerFrom: Kwangwoo Lee <[email protected]>
Pre-plug and plug handlers are prepared for NVDIMM support.
---
v7 -> v8:
- s/VIRT_ACPI_IO_BASE/VIRT_NVDIMM_ACPI_IO_BASE
- use const AcpiGenericAddress arm_virt_nvdimm_acpi_dsmio
--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt-acpi-build.c | 6 ++++++ hw/arm/virt.c | 27 +++++++++++++++++++++++++-- include/hw/arm/virt.h | 2 ++ 4 files changed, 35 insertions(+), 2 deletions(-)
default-configs/arm-softmmu.mak +2 -0
@@ -166,3 +166,5 @@ CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y CONFIG_MEM_DEVICE=y CONFIG_DIMM=y +CONFIG_NVDIMM=y +CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
@@ -786,6 +786,7 @@ staticvoid virt_acpi_build (VirtMachineState *vms,AcpiBuildTables *tables) {VirtMachineClass *vmc =VIRT_MACHINE_GET_CLASS (vms); +MachineState *ms =MACHINE (vms);GArray *table_offsets;unsigned dsdt, xsdt;GArray *tables_blob = tables->table_data;
@@ -826,6 +827,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables) } } +if (ms->acpi_nvdimm_state.is_enabled) { +nvdimm_build_acpi (table_offsets, tables_blob, tables->linker, + &ms->acpi_nvdimm_state, ms->ram_slots); + } +if (its_class_name () && !vmc->no_its) {acpi_add_table (table_offsets, tables_blob);build_iort (tables_blob, tables->linker, vms);
hw/arm/virt.c +25 -2
@@ -134,6 +134,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, [VIRT_SMMU] = { 0x09050000, 0x00020000 }, + [VIRT_NVDIMM_ACPI_IO] = { 0x09070000, 0x00010000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 },
@@ -1641,6 +1642,19 @@ static void machvirt_init(MachineState *machine)create_platform_bus (vms, pic); +if (machine->acpi_nvdimm_state.is_enabled) { +const struct AcpiGenericAddress arm_virt_nvdimm_acpi_dsmio = { + .space_id =AML_AS_SYSTEM_MEMORY , + .address = vms->memmap[VIRT_NVDIMM_ACPI_IO ].base + +NVDIMM_ACPI_IO_BASE , + .bit_width =NVDIMM_ACPI_IO_LEN <<3 + }; + +nvdimm_init_acpi_state (&machine->acpi_nvdimm_state, sysmem, + arm_virt_nvdimm_acpi_dsmio, + vms->fw_cfg,OBJECT (vms)); + } + vms->bootinfo.ram_size = machine->ram_size; vms->bootinfo.kernel_filename = machine->kernel_filename; vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1825,11 +1839,20 @@ static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,static void virt_memory_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) { -VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +MachineState *ms =MACHINE (hotplug_dev); +bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM );Error *local_err =NULL ; -pc_dimm_plug (PC_DIMM (dev),MACHINE (vms), &local_err); +pc_dimm_plug (PC_DIMM (dev), ms, &local_err); +if (local_err) { + goto out; + } +if (is_nvdimm) { +nvdimm_plug (&ms->acpi_nvdimm_state); + } + +out:error_propagate (errp, local_err); }
include/hw/arm/virt.h +2 -0
@@ -37,6 +37,7 @@#include "hw/arm/arm.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/mem/nvdimm.h" #define NUM_GICV2M_SPIS 64#define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , +VIRT_NVDIMM_ACPI_IO , VIRT_LOWMEMMAP_LAST, };
Signature · 1 line
2.20.1
PATCH v8 17/18 hw/arm/boot: Expose the pmem nodes in the DT
In reply to Eric AugerIn case of NV-DIMM slots, let's add /pmem DT nodes.
---
v6 -> v7
- does the same rework as for fdt_add_memory_node
--- hw/arm/boot.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
hw/arm/boot.c +40 -0
@@ -450,6 +450,32 @@ out:return ret; } +static int fdt_add_pmem_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/pmem@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"compatible" ,"pmem-region" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -546,6 +572,20 @@ static int fdt_add_hotpluggable_memory_nodes(void *fdt, }break ; } +case MEMORY_DEVICE_INFO_KIND_NVDIMM : + { +PCDIMMDeviceInfo *di = mi->u.nvdimm.data; + + ret =fdt_add_pmem_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add NVDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + }default :fprintf (stderr,"%s memory nodes are not yet supported\n" ,MemoryDeviceInfoKind_str (mi->type));
Signature · 1 line
2.20.1
PATCH v8 18/18 hw/arm/virt: Allow nvdimm instantiation
In reply to Eric AugerIf NVDIMM option is enabled at machine level, let's allow NVDIMM plug.
--- hw/arm/virt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
hw/arm/virt.c +3 -2
@@ -1823,13 +1823,14 @@ static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) {const bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM ); +MachineState *ms =MACHINE (hotplug_dev);if (dev->hotplugged) {error_setg (errp,"memory hotplug is not supported" ); } -if (is_nvdimm) { -error_setg (errp,"nvdimm is not yet supported" ); +if (is_nvdimm && !ms->acpi_nvdimm_state.is_enabled) { +error_setg (errp,"nvdimm is not enabled: missing 'nvdimm' in '-M'" );return ; }
Signature · 1 line
2.20.1