ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support
v7 cover ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size ram_size and then comes the device memory (,maxmem) of size maxram_size - ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on ram_size and maxram_size. Size alignment is enforced.
In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory map
2) Support of PC-DIMM [10 - 13]
3) Support of NV-DIMM [14 - 17]
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.
So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/
[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.html
This series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7
History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more details
v5 -> v6:
- mingw compilation issue fix
- kvm_arm_get_max_vm_phys_shift always returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization into machvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their size
v4 -> v5:
- change in the memory map
- see individual logs
v3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm: pre_plug "slot" and "addr" assignment"
- kvm-type option not used anymore. We directly use maxram_size and ram_size machine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
- device_memory_start and device_memory_size not stored anymore in vms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring of arm_load_dtb and the one related to the generation of the dtb after system_reset which would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machine ram_size, maxram_size and ram_slots earlier"
v2 -> v3:
- fix pc_q35 and pc_piix compilation error
- kwangwoo's email being not valid anymore, remove his address
v1 -> v2:
- kvm_get_max_vm_phys_shift moved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoring
v1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"
Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter to kvm_type callback
kvm: add kvm_arm_get_max_vm_ipa_size
vl: Set machine ram_size, maxram_size and ram_slots earlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implement kvm_type function for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocate device_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence options
Kwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructure
Shameer Kolothum (3):
hw/arm/boot: introduce fdt_add_memory_node helper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRAT
accel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Signature · 1 line
2.20.1
PATCH v7 1/17 hw/arm/boot: introduce fdt_add_memory_node helper
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
---
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID
--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +34 -20
@@ -423,6 +423,32 @@ static void set_kernel_args_old(const struct arm_boot_info *info, } } +static int fdt_add_memory_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -502,7 +528,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,void *fdt =NULL ;int size, rc, n =0 ;uint32_t acells, scells; -char *nodename;unsigned int i;hwaddr mem_base, mem_len;char **node_path;
@@ -576,35 +601,24 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start;for (i =0 ; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; - nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, mem_base, - scells, mem_len); + rc =fdt_add_memory_node (fdt, acells, mem_base, + scells, mem_len, i);if (rc <0 ) { -fprintf (stderr,"couldn't set %s/reg for node %d\n" , nodename, - i); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + mem_base); goto fail; } -qemu_fdt_setprop_cell (fdt, nodename,"numa-node-id" , i); mem_base += mem_len; -g_free (nodename); } }else { - nodename =g_strdup_printf ("/memory@%" PRIx64, binfo->loader_start); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, binfo->loader_start, - scells, binfo->ram_size); + rc =fdt_add_memory_node (fdt, acells, binfo->loader_start, + scells, binfo->ram_size,-1 );if (rc <0 ) { -fprintf (stderr,"couldn't set %s reg\n" , nodename); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + binfo->loader_start); goto fail; } -g_free (nodename); } rc =fdt_path_offset (fdt,"/chosen" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 01/17] hw/arm/boot: introduce fdt_add_memory_node helper
Quoted 7 lines · On Wed, 20 Feb 2019 23:39:47 +0100 Eric Auger <[email protected]> wrote:
From: Shameer Kolothum <[email protected]>
We introduce an helper to create a memory node.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>
Quoted 98 lines
---
v6 -> v7:
- msg error in the caller
- add comment about NUMA ID--- hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 20 deletions(-)hw/arm/boot.c +34
-20@@ -423,6 +423,32 @@ static void set_kernel_args_old(const struct arm_boot_info *info, } } +static int fdt_add_memory_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;@@ -502,7 +528,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,void *fdt =NULL ;int size, rc, n =0 ;uint32_t acells, scells; -char *nodename;unsigned int i;hwaddr mem_base, mem_len;char **node_path;@@ -576,35 +601,24 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start;for (i =0 ; i < nb_numa_nodes; i++) { mem_len = numa_info[i].node_mem; - nodename =g_strdup_printf ("/memory@%" PRIx64, mem_base); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, mem_base, - scells, mem_len); + rc =fdt_add_memory_node (fdt, acells, mem_base, + scells, mem_len, i);if (rc <0 ) { -fprintf (stderr,"couldn't set %s/reg for node %d\n" , nodename, - i); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + mem_base); goto fail; } -qemu_fdt_setprop_cell (fdt, nodename,"numa-node-id" , i); mem_base += mem_len; -g_free (nodename); } }else { - nodename =g_strdup_printf ("/memory@%" PRIx64, binfo->loader_start); -qemu_fdt_add_subnode (fdt, nodename); -qemu_fdt_setprop_string (fdt, nodename,"device_type" ,"memory" ); - - rc =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , - acells, binfo->loader_start, - scells, binfo->ram_size); + rc =fdt_add_memory_node (fdt, acells, binfo->loader_start, + scells, binfo->ram_size,-1 );if (rc <0 ) { -fprintf (stderr,"couldn't set %s reg\n" , nodename); +fprintf (stderr,"couldn't add /memory@%" PRIx64" node\n" , + binfo->loader_start); goto fail; } -g_free (nodename); } rc =fdt_path_offset (fdt,"/chosen" );
PATCH v7 2/17 hw/arm/virt: Rename highmem IO regions
In reply to Eric AugerIn preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM
---
v7: added Peter's R-b
v6: creation
--- hw/arm/virt-acpi-build.c | 8 ++++---- hw/arm/virt.c | 21 +++++++++++---------- include/hw/arm/virt.h | 8 ++++---- 3 files changed, 19 insertions(+), 18 deletions(-)
hw/arm/virt-acpi-build.c +4 -4
@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, size_pio));if (use_highmem) { -hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].size; aml_append(rbuf, aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -663,8 +663,8 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicr =acpi_data_push (table_data,sizeof (*gicr)); gicr->type =ACPI_APIC_GENERIC_REDISTRIBUTOR ; gicr->length =sizeof (*gicr); - gicr->base_address =cpu_to_le64 (memmap[VIRT_GIC_REDIST2 ].base); - gicr->range_length =cpu_to_le32 (memmap[VIRT_GIC_REDIST2 ].size); + gicr->base_address =cpu_to_le64 (memmap[VIRT_HIGH_GIC_REDIST2 ].base); + gicr->range_length =cpu_to_le32 (memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (its_class_name () && !vmc->no_its) {
hw/arm/virt.c +11 -10
@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM ] = {0x40000000 ,RAMLIMIT_BYTES }, /* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_PCIE_ECAM_HIGH ] = {0x4010000000ULL ,0x10000000 }, + [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, + [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_PCIE_MMIO_HIGH] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, };static const int a15irqmap[] = {
@@ -435,8 +435,8 @@ static void fdt_add_gic_node(VirtMachineState *vms)2 , vms->memmap[VIRT_GIC_DIST ].size,2 , vms->memmap[VIRT_GIC_REDIST ].base,2 , vms->memmap[VIRT_GIC_REDIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST2 ].base, - 2, vms->memmap[VIRT_GIC_REDIST2].size); +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base, + 2, vms->memmap[VIRT_HIGH_GIC_REDIST2].size); }if (vms->virt) {
@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (nb_redist_regions ==2 ) {uint32_t redist1_capacity = - vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ;qdev_prop_set_uint32 (gicdev,"redist-region-count[1]" ,MIN (smp_cpus - redist0_count, redist1_capacity));
@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (type ==3 ) {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_REDIST ].base);if (nb_redist_regions ==2 ) { -sysbus_mmio_map (gicbusdev,2 , vms->memmap[VIRT_GIC_REDIST2 ].base); +sysbus_mmio_map (gicbusdev,2 , + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base); } }else {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_CPU ].base);
@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO ].size; -hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].size;hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO ].base;hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO ].size;hwaddr base_ecam, size_ecam;
@@ -1418,7 +1419,7 @@ static void machvirt_init(MachineState *machine)*/ if (vms->gic_version ==3 ) { virt_max_cpus = vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; - virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + virt_max_cpus += vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; }else { virt_max_cpus =GIC_NCPU ; }
include/hw/arm/virt.h +4 -4
@@ -64,7 +64,7 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_GIC_REDIST2 , +VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,9 +74,9 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_PCIE_ECAM_HIGH , +VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_PCIE_MMIO_HIGH , +VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART , VIRT_SECURE_MEM,
@@ -128,7 +128,7 @@ typedef struct {int psci_conduit; } VirtMachineState; -#define VIRT_ECAM_ID (high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM) +#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)#define TYPE_VIRT_MACHINE MACHINE_TYPE_NAME("virt")#define VIRT_MACHINE (obj) \
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 02/17] hw/arm/virt: Rename highmem IO regions
Quoted 6 lines · On Wed, 20 Feb 2019 23:39:48 +0100 Eric Auger <[email protected]> wrote:
In preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>
with indent and checkpatch warnings fixed
Quoted 66 lines
---
v7: added Peter's R-b
v6: creation--- hw/arm/virt-acpi-build.c | 8 ++++---- hw/arm/virt.c | 21 +++++++++++---------- include/hw/arm/virt.h | 8 ++++---- 3 files changed, 19 insertions(+), 18 deletions(-)hw/arm/virt-acpi-build.c +4
-4@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, size_pio));if (use_highmem) { -hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO ].size; aml_append(rbuf, aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,@@ -663,8 +663,8 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) gicr =acpi_data_push (table_data,sizeof (*gicr)); gicr->type =ACPI_APIC_GENERIC_REDISTRIBUTOR ; gicr->length =sizeof (*gicr); - gicr->base_address =cpu_to_le64 (memmap[VIRT_GIC_REDIST2 ].base); - gicr->range_length =cpu_to_le32 (memmap[VIRT_GIC_REDIST2 ].size); + gicr->base_address =cpu_to_le64 (memmap[VIRT_HIGH_GIC_REDIST2 ].base); + gicr->range_length =cpu_to_le32 (memmap[VIRT_HIGH_GIC_REDIST2 ].size); }if (its_class_name () && !vmc->no_its) {hw/arm/virt.c +5
-5@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM ] = {0x40000000 ,RAMLIMIT_BYTES }, /* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_PCIE_ECAM_HIGH ] = {0x4010000000ULL ,0x10000000 }, + [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, + [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_PCIE_MMIO_HIGH] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, };static const int a15irqmap[] = {@@ -435,8 +435,8 @@ static void fdt_add_gic_node(VirtMachineState *vms)2 , vms->memmap[VIRT_GIC_DIST ].size,2 , vms->memmap[VIRT_GIC_REDIST ].base,2 , vms->memmap[VIRT_GIC_REDIST ].size, -2 , vms->memmap[VIRT_GIC_REDIST2 ].base, - 2, vms->memmap[VIRT_GIC_REDIST2].size); +2 , vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base, + 2, vms->memmap[VIRT_HIGH_GIC_REDIST2].size); }if (vms->virt) {
hw/arm/virt.c static void create_gic(VirtMachineState *vms, qemu_irq *pic) · line 587
Quoted 5 lines
hw/arm/virt.c +1
-1@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (nb_redist_regions ==2 ) {uint32_t redist1_capacity = - vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ;
is indent correct here (it didn't look correct to begin with).
Strangle checkpatch didn't complain about it, but since I've run it does complain about a bunch of "line over 80 characters" on this patch
Quoted 67 lines
hw/arm/virt.c +5
-4qdev_prop_set_uint32 (gicdev,"redist-region-count[1]" ,MIN (smp_cpus - redist0_count, redist1_capacity));@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)if (type ==3 ) {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_REDIST ].base);if (nb_redist_regions ==2 ) { -sysbus_mmio_map (gicbusdev,2 , vms->memmap[VIRT_GIC_REDIST2 ].base); +sysbus_mmio_map (gicbusdev,2 , + vms->memmap[VIRT_HIGH_GIC_REDIST2 ].base); } }else {sysbus_mmio_map (gicbusdev,1 , vms->memmap[VIRT_GIC_CPU ].base);@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic) {hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO ].base;hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO ].size; -hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].base; -hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH ].size; +hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].base; +hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO ].size;hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO ].base;hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO ].size;hwaddr base_ecam, size_ecam;@@ -1418,7 +1419,7 @@ static void machvirt_init(MachineState *machine)*/ if (vms->gic_version ==3 ) { virt_max_cpus = vms->memmap[VIRT_GIC_REDIST ].size /GICV3_REDIST_SIZE ; - virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; + virt_max_cpus += vms->memmap[VIRT_HIGH_GIC_REDIST2 ].size /GICV3_REDIST_SIZE ; }else { virt_max_cpus =GIC_NCPU ; }include/hw/arm/virt.h +4
-4@@ -64,7 +64,7 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_GIC_REDIST2 , +VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,9 +74,9 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_PCIE_ECAM_HIGH , +VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_PCIE_MMIO_HIGH , +VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART , VIRT_SECURE_MEM,@@ -128,7 +128,7 @@ typedef struct {int psci_conduit; } VirtMachineState; -#define VIRT_ECAM_ID (high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM) +#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)#define TYPE_VIRT_MACHINE MACHINE_TYPE_NAME("virt")#define VIRT_MACHINE (obj) \
PATCH v7 3/17 hw/arm/virt: Split the memory map description
In reply to Eric AugerIn the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap contains regions below and including the RAM
- extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup into machvirt_init(). The rationale is the memory map will be soon affected by the kvm_type() call that happens after virt_instance_init() and before machvirt_init().
The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-b
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments
--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)
hw/arm/virt.c +42 -6
@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h"
@@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);
@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */
@@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }
include/hw/arm/virt.h +10 -4
@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,
@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {
@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;
@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Quoted 9 lines · On Wed, 20 Feb 2019 23:39:49 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap contains regions below and including the RAM
-
extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.
can't parse this part and pinpoint what is 'their', care to rephrase?
Quoted 4 lines
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup
s/moves/makes/
s/$/dynamic and moves it/
Quoted 5 lines
into
machvirt_init().The rationale is the memory map will be soon affected by the
kvm_type()call that happens aftervirt_instance_init()and
is dependency on kvm_type() still valid, shouldn't split memmap work for TCG just fine as well?
Quoted 49 lines
before
machvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },
hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 157
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the
s/IO region offset/MemMapEntry base (GPA)/
hw/arm/virt.c static const MemMapEntry a15memmap[] = {
Quoted 3 lines
hw/arm/virt.c +2
-0+* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is
s/region/entry/
Quoted 16 lines
hw/arm/virt.c +7
-4+* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1373
Quoted 9 lines
hw/arm/virt.c +6
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap;
I probably don't see something but ...
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1376
Quoted 3 lines
hw/arm/virt.c +3
-0+ +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i];
ARRAY_SIZE(base_memmap) > 3 ARRAY_SIZE(extended_memmap) == 3
as result shouldn't we observe OOB at vms->memmap[i] access starting from i==3 ?
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 6 lines
hw/arm/virt.c +6
-0+ } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {
not sure why VIRT_LOWMEMMAP_LAST is needed at all, one could just continue with current 'i' value, provided extended_memmap wasn't corrupted by previous loop.
And does this loop ever executes? VIRT_LOWMEMMAP_LAST > ARRAY_SIZE(extended_memmap)
Quoted 79 lines
hw/arm/virt.c +11
-1+hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Hi Igor,
Quoted 6 lines · Igor Mammedov
Quoted 3 lines · On Wed, 20 Feb 2019 23:39:49 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
Quoted 7 lines
Quoted 6 lines
- the former a15memmap contains regions below and including the RAM
-
extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.can't parse this part and pinpoint what is 'their', care to rephrase?
Only the size of the High IO region entries is initialized (there are currently 3 entries: VIRT_HIGH_GIC_REDIST2, VIRT_HIGH_PCIE_ECAM, VIRT_HIGH_PCIE_MMIO). The base address is dynamically computed so it is not initialized.
Quoted 9 lines
Quoted 6 lines
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup
s/moves/makes/
s/$/dynamic and moves it/
Quoted 7 lines
Quoted 5 lines
into
machvirt_init().The rationale is the memory map will be soon affected by the
kvm_type()call that happens aftervirt_instance_init()andis dependency on
kvm_type()still valid, shouldn't split memmap work for TCG just fine as well?
See in 08/17: in TCG mode the memory map will be "frozen" (set_memmap) in machvirt_init. Otherwise set_memmap is called from kvm_type().
Split memmap works both in TCG and in accelerated mode.
I will rephrase the commit message.
Quoted 91 lines
Quoted 50 lines
before
machvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 157
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the s/IO region offset/MemMapEntry base (GPA)/
hw/arm/virt.c static const MemMapEntry a15memmap[] = {
Quoted 3 lines
hw/arm/virt.c +2
-0+* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is s/region/entry/
Quoted 16 lines
hw/arm/virt.c +7
-4+* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1373
Quoted 9 lines
hw/arm/virt.c +6
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap;I probably don't see something but ...
Quoted 8 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1376
Quoted 3 lines
hw/arm/virt.c +3
-0+ +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i];
ARRAY_SIZE(base_memmap)> 3ARRAY_SIZE(extended_memmap)== 3
as result shouldn't we observe OOB at vms->memmap[i] access starting from i==3 ?
ARRAY_SIZE(extended_memmap) = ARRAY_SIZE(base_memmap) + 3
VIRT_HIGH_GIC_REDIST2 = VIRT_LOWMEMMAP_LAST is what you miss.
/* indices of IO regions located after the RAM */
enum {
VIRT_HIGH_GIC_REDIST2 = VIRT_LOWMEMMAP_LAST,
VIRT_HIGH_PCIE_ECAM,
VIRT_HIGH_PCIE_MMIO,
};
Quoted 10 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 7 lines
hw/arm/virt.c +6
-0+ } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {not sure why
VIRT_LOWMEMMAP_LASTis needed at all, one could just continue with current 'i' value, providedextended_memmapwasn't corrupted by previous loop.
Yep maybe. But I think it is less error prone like this if someone later on adds some intermediate manipulation on i.
Quoted 1 line
And does this loop ever executes?
VIRT_LOWMEMMAP_LAST>ARRAY_SIZE(extended_memmap)
yes it does
Thanks
Eric
Quoted 82 lines
Quoted 80 lines
hw/arm/virt.c +11
-1+hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Quoted 136 lines · On Thu, 21 Feb 2019 18:21:11 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Igor Mammedov
Quoted 3 lines · On Wed, 20 Feb 2019 23:39:49 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
Quoted 7 lines
Quoted 6 lines
- the former a15memmap contains regions below and including the RAM
-
extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.can't parse this part and pinpoint what is 'their', care to rephrase?
Only the size of the High IO region entries is initialized (there are currently 3 entries:
VIRT_HIGH_GIC_REDIST2,VIRT_HIGH_PCIE_ECAM,VIRT_HIGH_PCIE_MMIO). The base address is dynamically computed so it is not initialized.Quoted 9 lines
Quoted 6 lines
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup
s/moves/makes/
s/$/dynamic and moves it/Quoted 7 lines
Quoted 5 lines
into
machvirt_init().The rationale is the memory map will be soon affected by the
kvm_type()call that happens aftervirt_instance_init()andis dependency on
kvm_type()still valid, shouldn't split memmap work for TCG just fine as well?See in 08/17: in TCG mode the memory map will be "frozen" (
set_memmap) inmachvirt_init. Otherwiseset_memmapis called fromkvm_type().Split memmap works both in TCG and in accelerated mode.
I will rephrase the commit message.
Quoted 95 lines
Quoted 50 lines
before
machvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 157
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the s/IO region offset/MemMapEntry base (GPA)/
hw/arm/virt.c static const MemMapEntry a15memmap[] = {
Quoted 3 lines
hw/arm/virt.c +2
-0+* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is s/region/entry/
Quoted 16 lines
hw/arm/virt.c +7
-4+* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1373
Quoted 9 lines
hw/arm/virt.c +6
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap;I probably don't see something but ...
Quoted 3 lines
hw/arm/virt.c +3
-0+ +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i];
Quoted 6 lines
Quoted 4 lines
ARRAY_SIZE(base_memmap)> 3ARRAY_SIZE(extended_memmap)== 3
as result shouldn't we observe OOB at vms->memmap[i] access starting from i==3 ?
ARRAY_SIZE(extended_memmap)=ARRAY_SIZE(base_memmap)+ 3VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LASTis what you miss.
Yep, that's the trick.
It is too much subtle for my taste,
is it possible to make extended_memmap sizing more explicit/trivial,
so one could see it right away without figuring out how indexes influence array size?
Quoted 107 lines
/* indices of IO regions located after the RAM */
enum {
VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LAST,
VIRT_HIGH_PCIE_ECAM,
VIRT_HIGH_PCIE_MMIO,
};Quoted 10 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 7 lines
hw/arm/virt.c +6
-0+ } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {not sure why
VIRT_LOWMEMMAP_LASTis needed at all, one could just continue with current 'i' value, providedextended_memmapwasn't corrupted by previous loop.Yep maybe. But I think it is less error prone like this if someone later on adds some intermediate manipulation on i.
Quoted 1 line
And does this loop ever executes?
VIRT_LOWMEMMAP_LAST>ARRAY_SIZE(extended_memmap)yes it does
Thanks
Eric
Quoted 82 lines
Quoted 80 lines
hw/arm/virt.c +11
-1+hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Hi Igor,
Quoted 143 lines · Igor Mammedov
Quoted 140 lines · On Thu, 21 Feb 2019 18:21:11 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Igor Mammedov
Quoted 3 lines · On Wed, 20 Feb 2019 23:39:49 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
Quoted 7 lines
Quoted 6 lines
- the former a15memmap contains regions below and including the RAM
-
extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.can't parse this part and pinpoint what is 'their', care to rephrase?
Only the size of the High IO region entries is initialized (there are currently 3 entries:
VIRT_HIGH_GIC_REDIST2,VIRT_HIGH_PCIE_ECAM,VIRT_HIGH_PCIE_MMIO). The base address is dynamically computed so it is not initialized.Quoted 9 lines
Quoted 6 lines
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup
s/moves/makes/
s/$/dynamic and moves it/Quoted 7 lines
Quoted 5 lines
into
machvirt_init().The rationale is the memory map will be soon affected by the
kvm_type()call that happens aftervirt_instance_init()andis dependency on
kvm_type()still valid, shouldn't split memmap work for TCG just fine as well?See in 08/17: in TCG mode the memory map will be "frozen" (
set_memmap) inmachvirt_init. Otherwiseset_memmapis called fromkvm_type().Split memmap works both in TCG and in accelerated mode.
I will rephrase the commit message.
Quoted 99 lines
Quoted 50 lines
before
machvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 157
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the s/IO region offset/MemMapEntry base (GPA)/
hw/arm/virt.c static const MemMapEntry a15memmap[] = {
Quoted 3 lines
hw/arm/virt.c +2
-0+* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is s/region/entry/
Quoted 16 lines
hw/arm/virt.c +7
-4+* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1373
Quoted 9 lines
hw/arm/virt.c +6
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap;I probably don't see something but ...
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1376
Quoted 3 lines
hw/arm/virt.c +3
-0+ +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i];
ARRAY_SIZE(base_memmap)> 3ARRAY_SIZE(extended_memmap)== 3
as result shouldn't we observe OOB at vms->memmap[i] access starting from i==3 ?
Quoted 6 lines
Quoted 2 lines
ARRAY_SIZE(extended_memmap)=ARRAY_SIZE(base_memmap)+ 3VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LASTis what you miss.Yep, that's the trick.
It is too much subtle for my taste,
is it possible to makeextended_memmapsizing more explicit/trivial,
so one could see it right away without figuring out how indexes influence array size?
The issue is if we explicitly size the array is will need to change the size whenever we add a new entry. I don't see any better solution to be honest. I can definitively add comments in the code about this sizing aspect. Another solution is to merge the 2 arrays as suggested by Heyi but I dislike the fact one part is initialized in one way and the other is initialized another way?
Thanks
Eric
Quoted 111 lines
Quoted 109 lines
/* indices of IO regions located after the RAM */
enum {
VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LAST,
VIRT_HIGH_PCIE_ECAM,
VIRT_HIGH_PCIE_MMIO,
};Quoted 10 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 7 lines
hw/arm/virt.c +6
-0+ } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {not sure why
VIRT_LOWMEMMAP_LASTis needed at all, one could just continue with current 'i' value, providedextended_memmapwasn't corrupted by previous loop.Yep maybe. But I think it is less error prone like this if someone later on adds some intermediate manipulation on i.
Quoted 1 line
And does this loop ever executes?
VIRT_LOWMEMMAP_LAST>ARRAY_SIZE(extended_memmap)yes it does
Thanks
Eric
Quoted 82 lines
Quoted 80 lines
hw/arm/virt.c +11
-1+hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Quoted 152 lines · On Fri, 22 Feb 2019 15:28:20 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 143 lines · Igor Mammedov
Quoted 140 lines · On Thu, 21 Feb 2019 18:21:11 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Igor Mammedov
Quoted 3 lines · On Wed, 20 Feb 2019 23:39:49 +0100 Eric Auger <[email protected]> wrote:
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
Quoted 7 lines
Quoted 6 lines
- the former a15memmap contains regions below and including the RAM
-
extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.can't parse this part and pinpoint what is 'their', care to rephrase?
Only the size of the High IO region entries is initialized (there are currently 3 entries:
VIRT_HIGH_GIC_REDIST2,VIRT_HIGH_PCIE_ECAM,VIRT_HIGH_PCIE_MMIO). The base address is dynamically computed so it is not initialized.Quoted 9 lines
Quoted 6 lines
This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup
s/moves/makes/
s/$/dynamic and moves it/Quoted 7 lines
Quoted 5 lines
into
machvirt_init().The rationale is the memory map will be soon affected by the
kvm_type()call that happens aftervirt_instance_init()andis dependency on
kvm_type()still valid, shouldn't split memmap work for TCG just fine as well?See in 08/17: in TCG mode the memory map will be "frozen" (
set_memmap) inmachvirt_init. Otherwiseset_memmapis called fromkvm_type().Split memmap works both in TCG and in accelerated mode.
I will rephrase the commit message.
Quoted 99 lines
Quoted 50 lines
before
machvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 157
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the s/IO region offset/MemMapEntry base (GPA)/
hw/arm/virt.c static const MemMapEntry a15memmap[] = {
Quoted 3 lines
hw/arm/virt.c +2
-0+* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is s/region/entry/
Quoted 16 lines
hw/arm/virt.c +7
-4+* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1373
Quoted 9 lines
hw/arm/virt.c +6
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap;I probably don't see something but ...
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1376
Quoted 3 lines
hw/arm/virt.c +3
-0+ +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i];
ARRAY_SIZE(base_memmap)> 3ARRAY_SIZE(extended_memmap)== 3
as result shouldn't we observe OOB at vms->memmap[i] access starting from i==3 ?Quoted 6 lines
Quoted 2 lines
ARRAY_SIZE(extended_memmap)=ARRAY_SIZE(base_memmap)+ 3VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LASTis what you miss.Yep, that's the trick.
It is too much subtle for my taste,
is it possible to makeextended_memmapsizing more explicit/trivial,
so one could see it right away without figuring out how indexes influence array size?
Quoted 6 lines
The issue is if we explicitly size the array is will need to change the size whenever we add a new entry. I don't see any better solution to be honest. I can definitively add comments in the code about this sizing aspect. Another solution is to merge the 2 arrays as suggested by Heyi but I dislike the fact one part is initialized in one way and the other is initialized another way?
I'd go with dynamically allocated memmap where we would copy base map and add extended.
But considering how memmap is used throughout code (treated like a map structure, instead of an array) it probably deserves it's own series.
For now it's fine with extra comments and maybe more explanation in commit, so that inattentive patch reader won't have to wonder what's going on now and later doing archaeology studies on QEMU :)
Quoted 115 lines
Thanks
Eric
Quoted 111 lines
Quoted 109 lines
/* indices of IO regions located after the RAM */
enum {
VIRT_HIGH_GIC_REDIST2=VIRT_LOWMEMMAP_LAST,
VIRT_HIGH_PCIE_ECAM,
VIRT_HIGH_PCIE_MMIO,
};Quoted 10 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1382
Quoted 7 lines
hw/arm/virt.c +6
-0+ } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {not sure why
VIRT_LOWMEMMAP_LASTis needed at all, one could just continue with current 'i' value, providedextended_memmapwasn't corrupted by previous loop.Yep maybe. But I think it is less error prone like this if someone later on adds some intermediate manipulation on i.
Quoted 1 line
And does this loop ever executes?
VIRT_LOWMEMMAP_LAST>ARRAY_SIZE(extended_memmap)yes it does
Thanks
Eric
Quoted 82 lines
Quoted 80 lines
hw/arm/virt.c +11
-1+hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
In reply to Eric AugerHi Eric,
Can't we still use one single memory map and update the base of every entry following VIRT_MEM? So that we don't need to split memory map or the enumeration definition, neither do we need to copy a15memmap into the extended memmap.
Thanks,
Heyi
Quoted 187 lines · Eric Auger
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap contains regions below and including the RAM
-extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup into
machvirt_init(). The rationale is the memory map will be soon affected by thekvm_type()call that happens aftervirt_instance_init()and beforemachvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments--- hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------ include/hw/arm/virt.h | 14 +++++++++---- 2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +42
-6@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h" @@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = {/* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH ] = {0 ,0x08000000 }, [VIRT_CPUPERIPHS ] = {0x08000000 ,0x00020000 },@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM. +* Each IO region offset will be dynamically computed, depending on the +* top of the RAM, so that its base get the same alignment as the size, +* ie. a 512GiB region will be aligned on a 512GiB boundary. If there is +* less than 256GiB of RAM, the floating area starts at the 256GiB mark. +*/ +static MemMapEntry extended_memmap[] = {/* Additional 64 MB redist region (can contain up to 512 redistributors) */ - [VIRT_HIGH_GIC_REDIST2] = { 0x4000000000ULL, 0x4000000 }, - [VIRT_HIGH_PCIE_ECAM ] = {0x4010000000ULL ,0x10000000 }, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = { 0x8000000000ULL, 0x8000000000ULL }, + [VIRT_HIGH_GIC_REDIST2 ] = {0x0 ,64 * MiB }, + [VIRT_HIGH_PCIE_ECAM ] = {0x0 ,256 * MiB }, +/* Second PCIe window */ + [VIRT_HIGH_PCIE_MMIO ] = {0x0 ,512 * GiB }, };static const int a15irqmap[] = {@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)return arm_cpu_mp_affinity (idx, clustersz); } +static void virt_set_memmap (VirtMachineState *vms) +{ +hwaddr base; +int i; + + vms->memmap = extended_memmap; + +for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ + base = vms->high_io_base; + +for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) { +hwaddr size = extended_memmap[i].size; + + base =ROUND_UP (base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void machvirt_init (MachineState *machine) {VirtMachineState *vms =VIRT_MACHINE (machine);@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; +virt_set_memmap (vms); +/* We can probe only here because during property set * KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum {VIRT_GIC_VCPU ,VIRT_GIC_ITS ,VIRT_GIC_REDIST , -VIRT_HIGH_GIC_REDIST2 ,VIRT_SMMU ,VIRT_UART , VIRT_MMIO,@@ -74,12 +73,18 @@ enum {VIRT_PCIE_MMIO ,VIRT_PCIE_PIO ,VIRT_PCIE_ECAM , -VIRT_HIGH_PCIE_ECAM ,VIRT_PLATFORM_BUS , -VIRT_HIGH_PCIE_MMIO ,VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { +VIRT_HIGH_GIC_REDIST2 =VIRT_LOWMEMMAP_LAST , +VIRT_HIGH_PCIE_ECAM , +VIRT_HIGH_PCIE_MMIO , }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct {int32_t gic_version;VirtIOMMUType iommu;struct arm_boot_info bootinfo; -const MemMapEntry *memmap; +MemMapEntry *memmap;const int *irqmap;int smp_cpus;void *fdt;@@ -126,6 +131,7 @@ typedef struct {uint32_t msi_phandle;uint32_t iommu_phandle;int psci_conduit; +hwaddr high_io_base; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-devel] [PATCH v7 03/17] hw/arm/virt: Split the memory map description
Hi Heyi,
Quoted 6 lines · Heyi Guo
Hi Eric,
Can't we still use one single memory map and update the base of every entry following
VIRT_MEM? So that we don't need to split memory map or the enumeration definition, neither do we need to copy a15memmap into the extended memmap.
I made the decision to have 2 separate arrays since we have
- one array with static base address entries (which are initialized with base address and size)
- and another array with floating base address entries (only the size is initialized)
To me it makes things clearer with respect to what is static and what is dynamically allocated (same for the enum).
Thanks
Eric
Quoted 195 lines
Thanks,
Heyi
Quoted 186 lines · Eric Auger
In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:
- the former a15memmap contains regions below and including the RAM
-extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.This new split will allow to grow the RAM size without changing the description of the high regions.
The patch also moves the memory map setup into
machvirt_init(). The rationale is the memory map will be soon affected by thekvm_type()call that happens aftervirt_instance_init()and beforemachvirt_init().The memory map is unchanged (the top of the initial RAM still is 256GiB). Then come the high IO regions with same layout as before.
Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>---
v6 -> v7:
- s/a15memmap/base_memmap
- slight rewording of the commit message
- add "if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark" in the comment associated to the floating memory map
- Added Peter's R-bv5 -> v6
- removal of many macros inunits.h
- introduce thevirt_set_memmaphelper
- new computation for offsets of high IO regions
- add comments
---
hw/arm/virt.c | 48 +++++++++++++++++++++++++++++++++++++------include/hw/arm/virt.h| 14 +++++++++----2 files changed, 52 insertions(+), 10 deletions(-)hw/arm/virt.c +2
-1@@ -29,6 +29,7 @@*/ #include "qemu/osdep.h" +#include "qemu/units.h" #include "qapi/error.h" #include "hw/sysbus.h" #include "hw/arm/arm.h"@@ -121,7 +122,7 @@* Note that devices should generally be placed at multiples of 0x10000, * to accommodate guests using 64K pages. */ -static const MemMapEntry a15memmap[] = { +static const MemMapEntry base_memmap[] = { /* Space up to 0x8000000 is reserved for a boot ROM */ [VIRT_FLASH] = { 0, 0x08000000 },[
VIRT_CPUPERIPHS] = {0x08000000,0x00020000},hw/arm/virt.c +4
-0@@ -149,11 +150,21 @@ static const MemMapEntry a15memmap[] = { [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, +}; + +/* +* Highmem IO Regions: This memory map is floating, located after the RAM.
+ * Each IO region offset will be dynamically computed, depending on the
+ * top of the RAM, so that its base get the same alignment as the size,
+ * ie. a 512GiB region will be aligned on a 512GiB boundary. If there is
+ * less than 256GiB of RAM, the floating area starts at the 256GiB mark.
+ */
+static MemMapEntry extended_memmap[] = {
/* Additional 64 MB redist region (can contain up to 512 redistributors) */
- [VIRT_HIGH_GIC_REDIST2] = {0x4000000000ULL,0x4000000}, - [VIRT_HIGH_PCIE_ECAM] = {0x4010000000ULL,0x10000000}, - /* Second PCIe window, 512GB wide at the 512GB boundary */ - [VIRT_HIGH_PCIE_MMIO] = {0x8000000000ULL,0x8000000000ULL}, + [VIRT_HIGH_GIC_REDIST2] = {0x0, 64 * MiB },
+ [VIRT_HIGH_PCIE_ECAM] = {0x0, 256 * MiB },
+ /* Second PCIe window */
+ [VIRT_HIGH_PCIE_MMIO] = {0x0, 512 * GiB },
};
static const int a15irqmap[] = {hw/arm/virt.c +11
-0@@ -1354,6 +1365,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) return arm_cpu_mp_affinity(idx, clustersz); } +static void virt_set_memmap(VirtMachineState *vms) +{ + hwaddr base; + int i; + + vms->memmap = extended_memmap; + + for (i = 0; i < ARRAY_SIZE(base_memmap); i++) { + vms->memmap[i] = base_memmap[i]; + } + + vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAMregion */
+ base =vms->high_io_base;
+
+ for (i =VIRT_LOWMEMMAP_LAST; i <ARRAY_SIZE(extended_memmap); i++) {
+ hwaddr size = extended_memmap[i].size;hw/arm/virt.c +8
-0+ + base = ROUND_UP(base, size); + vms->memmap[i].base = base; + vms->memmap[i].size = size; + base += size; + } +} +static void
machvirt_init(MachineState *machine){
VirtMachineState *vms =VIRT_MACHINE(machine);hw/arm/virt.c +1
-0@@ -1368,6 +1403,8 @@ static void machvirt_init(MachineState *machine) bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0); bool aarch64 = true; + virt_set_memmap(vms); + /* We can probe only here because during property set* KVM is not available yet */ @@ -1843,7 +1880,6 @@ static void virt_instance_init(Object *obj) "Valid values are none and smmuv3", NULL); - vms->memmap = a15memmap; vms->irqmap = a15irqmap; }diff --git
a/include/hw/arm/virt.hb/include/hw/arm/virt.hindexa27086d524..3dc7a6c5d5100644include/hw/arm/virt.h +10
-4@@ -64,7 +64,6 @@ enum { VIRT_GIC_VCPU, VIRT_GIC_ITS, VIRT_GIC_REDIST, - VIRT_HIGH_GIC_REDIST2, VIRT_SMMU, VIRT_UART, VIRT_MMIO,@@ -74,12 +73,18 @@ enum { VIRT_PCIE_MMIO, VIRT_PCIE_PIO, VIRT_PCIE_ECAM, - VIRT_HIGH_PCIE_ECAM, VIRT_PLATFORM_BUS, - VIRT_HIGH_PCIE_MMIO, VIRT_GPIO, VIRT_SECURE_UART, VIRT_SECURE_MEM, + VIRT_LOWMEMMAP_LAST, +}; + +/* indices of IO regions located after the RAM */ +enum { + VIRT_HIGH_GIC_REDIST2 = VIRT_LOWMEMMAP_LAST, + VIRT_HIGH_PCIE_ECAM, + VIRT_HIGH_PCIE_MMIO, }; typedef enum VirtIOMMUType {@@ -116,7 +121,7 @@ typedef struct { int32_t gic_version; VirtIOMMUType iommu; struct arm_boot_info bootinfo; - const MemMapEntry *memmap; + MemMapEntry *memmap; const int *irqmap; int smp_cpus; void *fdt;@@ -126,6 +131,7 @@ typedef struct { uint32_t msi_phandle; uint32_t iommu_phandle; int psci_conduit; + hwaddr high_io_base; } VirtMachineState; #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v7 4/17 hw/boards: Add a MachineState parameter to kvm_type callback
In reply to Eric AugerOn ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.
---
v6 -> v7:
- add a comment for kvm_type
- use machine instead of ms in the declaration
- add Peter's R-b
--- accel/kvm/kvm-all.c | 2 +- hw/ppc/mac_newworld.c | 3 +-- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/boards.h | 5 ++++- 5 files changed, 8 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +1 -1
@@ -1593,7 +1593,7 @@ static int kvm_init(MachineState *ms) kvm_type =qemu_opt_get (qemu_get_machine_opts (),"kvm-type" );if (mc->kvm_type) { - type = mc->kvm_type (kvm_type); + type = mc->kvm_type (ms, kvm_type); }else if (kvm_type) { ret = -EINVAL ;fprintf (stderr,"Invalid argument kvm-type=%s\n" , kvm_type);
hw/ppc/mac_newworld.c +1 -2
@@ -564,8 +564,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } - -static int core99_kvm_type (const char *arg) +static int core99_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/mac_oldworld.c +1 -1
@@ -420,7 +420,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } -static int heathrow_kvm_type (const char *arg) +static int heathrow_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;
hw/ppc/spapr.c +1 -1
@@ -2920,7 +2920,7 @@ static void spapr_machine_init(MachineState *machine) } } -static int spapr_kvm_type (const char *vm_type) +static int spapr_kvm_type (MachineState *machine,const char *vm_type) {if (!vm_type) {return 0 ;
include/hw/boards.h +4 -1
@@ -156,6 +156,9 @@ typedef struct {* should instead use "unimplemented-device" for all memory ranges where * the guest will attempt to probe for a device that QEMU doesn't * implement and a stub device is required. +* @kvm_type: +* Return the type of KVM corresponding to the kvm-type string option or +* computed based on other criteria such as the host kernel capabilities. */ struct MachineClass {/*< private >*/
@@ -171,7 +174,7 @@ struct MachineClass {void (*init)(MachineState *state);void (*reset)(void );void (*hot_add_cpu)(const int64_t id,Error **errp); -int (*kvm_type)(const char *arg); +int (*kvm_type)(MachineState *machine,const char *arg);BlockInterfaceType block_default_type;int units_per_default_bus;
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 04/17] hw/boards: Add a MachineState parameter to kvm_type callback
Quoted 9 lines · On Wed, 20 Feb 2019 23:39:50 +0100 Eric Auger <[email protected]> wrote:
On ARM, the
kvm_typewill be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. inkvm_init, when the callback is called, thekvm_statevariable is not yet set.Signed-off-by Eric Auger <[email protected]>Acked-by David Gibson <[email protected]>[ppc parts]Reviewed-by Peter Maydell <[email protected]>
Quoted 90 lines
---
v6 -> v7:
- add a comment forkvm_type
- use machine instead of ms in the declaration
- add Peter's R-b--- accel/kvm/kvm-all.c | 2 +- hw/ppc/mac_newworld.c | 3 +-- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/boards.h | 5 ++++- 5 files changed, 8 insertions(+), 6 deletions(-)accel/kvm/kvm-all.c +1
-1@@ -1593,7 +1593,7 @@ static int kvm_init(MachineState *ms) kvm_type =qemu_opt_get (qemu_get_machine_opts (),"kvm-type" );if (mc->kvm_type) { - type = mc->kvm_type (kvm_type); + type = mc->kvm_type (ms, kvm_type); }else if (kvm_type) { ret = -EINVAL ;fprintf (stderr,"Invalid argument kvm-type=%s\n" , kvm_type);hw/ppc/mac_newworld.c +1
-2@@ -564,8 +564,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } - -static int core99_kvm_type (const char *arg) +static int core99_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;hw/ppc/mac_oldworld.c +1
-1@@ -420,7 +420,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,return NULL ; } -static int heathrow_kvm_type (const char *arg) +static int heathrow_kvm_type (MachineState *machine,const char *arg) {/* Always force PR KVM */ return 2 ;hw/ppc/spapr.c +1
-1@@ -2920,7 +2920,7 @@ static void spapr_machine_init(MachineState *machine) } } -static int spapr_kvm_type (const char *vm_type) +static int spapr_kvm_type (MachineState *machine,const char *vm_type) {if (!vm_type) {return 0 ;include/hw/boards.h +4
-1@@ -156,6 +156,9 @@ typedef struct {* should instead use "unimplemented-device" for all memory ranges where * the guest will attempt to probe for a device that QEMU doesn't * implement and a stub device is required. +* @kvm_type: +* Return the type of KVM corresponding to the kvm-type string option or +* computed based on other criteria such as the host kernel capabilities. */ struct MachineClass {/*< private >*/ @@ -171,7 +174,7 @@ struct MachineClass {void (*init)(MachineState *state);void (*reset)(void );void (*hot_add_cpu)(const int64_t id,Error **errp); -int (*kvm_type)(const char *arg); +int (*kvm_type)(MachineState *machine,const char *arg);BlockInterfaceType block_default_type;int units_per_default_bus;
PATCH v7 5/17 kvm: add kvm_arm_get_max_vm_ipa_size
In reply to Eric AugerAdd the kvm_arm_get_max_vm_ipa_size() helper that returns the number of bits in the IPA address space supported by KVM.
This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.
---
v6 -> v7:
- s/kvm_arm_get_max_vm_phys_shift/kvm_arm_get_max_vm_ipa_size
- reword the comment
v4 -> v5:
- return 40 if the host does not support the capability
v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension
v1 -> v2:
- put this in ARM specific code
--- target/arm/kvm.c | 10 ++++++++++ target/arm/kvm_arm.h | 13 +++++++++++++ 2 files changed, 23 insertions(+)
target/arm/kvm.c +10 -0
@@ -18,6 +18,7 @@#include "qemu/error-report.h" #include "sysemu/sysemu.h" #include "sysemu/kvm.h" +#include "sysemu/kvm_int.h" #include "kvm_arm.h" #include "cpu.h" #include "trace.h"
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) env->features = arm_host_cpu_features.features; } +int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +KVMState *s =KVM_STATE (ms->accelerator); +int ret; + + ret =kvm_check_extension (s,KVM_CAP_ARM_VM_IPA_SIZE ); +return ret >0 ? ret :40 ; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +13 -0
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);*/ void kvm_arm_set_cpu_features_from_host (ARMCPU *cpu); +/** +* kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the +* IPA address space supported by KVM +* +* @ms: Machine state handle +*/ +int kvm_arm_get_max_vm_ipa_size (MachineState *ms); +/** * kvm_arm_sync_mpstate_to_kvm * @cpu: ARMCPU
@@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) cpu->host_cpu_probe_failed = true; } +static inline int kvm_arm_get_max_vm_ipa_size (MachineState *ms) +{ +return -ENOENT ; +} +static inline int kvm_arm_vgic_probe (void ) {return 0 ;
Signature · 1 line
2.20.1
PATCH v7 6/17 vl: Set machine ram_size, maxram_size and ram_slots earlier
In reply to Eric AugerThe machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.
---
v6 -> v7:
- add Peter's R-b
v4: new
--- vl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +3 -3
@@ -4239,6 +4239,9 @@ int main(int argc, char **argv, char **envp) machine_opts =qemu_get_machine_opts ();qemu_opt_foreach (machine_opts, machine_set_property, current_machine, &error_fatal); + current_machine->ram_size = ram_size; + current_machine->maxram_size = maxram_size; + current_machine->ram_slots = ram_slots;configure_accelerator (current_machine, argv[0 ]);
@@ -4434,9 +4437,6 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); - current_machine->ram_size = ram_size; - current_machine->maxram_size = maxram_size; - current_machine->ram_slots = ram_slots; current_machine->boot_order = boot_order;/* parse features once if machine provides default cpu_type */
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 06/17] vl: Set machine ram_size, maxram_size and ram_slots earlier
Quoted 8 lines · On Wed, 20 Feb 2019 23:39:52 +0100 Eric Auger <[email protected]> wrote:
The machine RAM attributes will need to be analyzed during the
configure_accelerator()process. especiallykvm_type()arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before callingconfigure_accelerator.Signed-off-by Eric Auger <[email protected]>Reviewed-by Peter Maydell <[email protected]>
Quoted 34 lines
---
v6 -> v7:
- add Peter's R-bv4: new
--- vl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)vl.c +3
-3@@ -4239,6 +4239,9 @@ int main(int argc, char **argv, char **envp) machine_opts =qemu_get_machine_opts ();qemu_opt_foreach (machine_opts, machine_set_property, current_machine, &error_fatal); + current_machine->ram_size = ram_size; + current_machine->maxram_size = maxram_size; + current_machine->ram_slots = ram_slots;configure_accelerator (current_machine, argv[0 ]);@@ -4434,9 +4437,6 @@ int main(int argc, char **argv, char **envp)replay_checkpoint (CHECKPOINT_INIT );qdev_machine_init (); - current_machine->ram_size = ram_size; - current_machine->maxram_size = maxram_size; - current_machine->ram_slots = ram_slots; current_machine->boot_order = boot_order;/* parse features once if machine provides default cpu_type */
PATCH v7 7/17 hw/arm/virt: Dynamic memory map depending on RAM requirements
In reply to Eric AugerUp to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the virt_set_memmap() function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.
The function computes
- the base of the device memory,
- the size of the device memory and
- the highest GPA used in the memory map.
The two former will be used when defining the device memory region while the latter will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB. Also we handle the case where maxmem or slots options are passed, although no device memory is usable at the moment. In this case, we just ignore those settings.
--- hw/arm/virt.c | 47 ++++++++++++++++++++++++++++++++++--------- include/hw/arm/virt.h | 3 +++ 2 files changed, 41 insertions(+), 9 deletions(-)
hw/arm/virt.c +38 -9
@@ -107,8 +107,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
@@ -149,7 +150,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/*
@@ -1367,16 +1368,48 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { +MachineState *ms =MACHINE (vms);hwaddr base;int i; +if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) { +error_report ("mach-virt: does not support device memory: " +"ignore maxmem and slots options" ); + ms->maxram_size = ms->ram_size; + ms->ram_slots =0 ; + } +if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB ); +exit (1 ); + } + vms->memmap = extended_memmap;for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + vms->device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + vms->device_memory_size = ms->maxram_size - ms->ram_size + + ms->ram_slots * GiB; + + vms->high_io_base = vms->device_memory_base + +ROUND_UP (vms->device_memory_size, GiB); +if (vms->high_io_base < vms->device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {
@@ -1387,6 +1420,7 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; }static void machvirt_init (MachineState *machine)
@@ -1470,11 +1504,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );
include/hw/arm/virt.h +3 -0
@@ -132,6 +132,9 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; +hwaddr device_memory_base; +hwaddr device_memory_size; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 07/17] hw/arm/virt: Dynamic memory map depending on RAM requirements
Quoted 67 lines · On Wed, 20 Feb 2019 23:39:53 +0100 Eric Auger <[email protected]> wrote:
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory and
- the highest GPA used in the memory map.The two former will be used when defining the device memory region while the latter will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB. Also we handle the case where maxmem or slots options are passed, although no device memory is usable at the moment. In this case, we just ignore those settings.
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 47 ++++++++++++++++++++++++++++++++++--------- include/hw/arm/virt.h | 3 +++ 2 files changed, 41 insertions(+), 9 deletions(-)hw/arm/virt.c +6
-3@@ -107,8 +107,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +150,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1367,16 +1368,48 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { +MachineState *ms =MACHINE (vms);hwaddr base;int i; +if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) {
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1385
Quoted 10 lines
hw/arm/virt.c +10
-0+error_report ("mach-virt: does not support device memory: " +"ignore maxmem and slots options" ); + ms->maxram_size = ms->ram_size; + ms->ram_slots =0 ; + } +if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB ); +exit (1 ); + }
I'd drop these checks and amend below code so that it would init device_memory logic when ram_slots > 0. It should simplify follow up patches by dropping all machine version specific parts.
It shouldn't break old machines as layout stays the same but would allow to start old machine with pc-dimms which is fine from migration pov as target also should be able to start QEMU with the same options for migration to start.
Quoted 65 lines
hw/arm/virt.c +22
-6+ vms->memmap = extended_memmap;for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + vms->device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + vms->device_memory_size = ms->maxram_size - ms->ram_size + + ms->ram_slots * GiB; + + vms->high_io_base = vms->device_memory_base + +ROUND_UP (vms->device_memory_size, GiB); +if (vms->high_io_base < vms->device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {@@ -1387,6 +1420,7 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; }static void machvirt_init (MachineState *machine)@@ -1470,11 +1504,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );include/hw/arm/virt.h +3
-0@@ -132,6 +132,9 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; +hwaddr device_memory_base; +hwaddr device_memory_size; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [PATCH v7 07/17] hw/arm/virt: Dynamic memory map depending on RAM requirements
Hi Igor,
Quoted 69 lines · Igor Mammedov
Quoted 66 lines · On Wed, 20 Feb 2019 23:39:53 +0100 Eric Auger <[email protected]> wrote:
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory and
- the highest GPA used in the memory map.The two former will be used when defining the device memory region while the latter will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB. Also we handle the case where maxmem or slots options are passed, although no device memory is usable at the moment. In this case, we just ignore those settings.
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 47 ++++++++++++++++++++++++++++++++++--------- include/hw/arm/virt.h | 3 +++ 2 files changed, 41 insertions(+), 9 deletions(-)hw/arm/virt.c +5
-3@@ -107,8 +107,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +150,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1367,16 +1368,48 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { +MachineState *ms =MACHINE (vms);hwaddr base;int i;
Quoted 14 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1385
Quoted 11 lines
hw/arm/virt.c +11
-0+if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) { +error_report ("mach-virt: does not support device memory: " +"ignore maxmem and slots options" ); + ms->maxram_size = ms->ram_size; + ms->ram_slots =0 ; + } +if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB ); +exit (1 ); + }I'd drop these checks and amend below code so that it would init
device_memorylogic whenram_slots> 0. It should simplify follow up patches by dropping all machine version specific parts.
I don't have sufficient knowledge of virtio-mem/virtio-pmem. Do they also use slots?
Quoted 23 lines
It shouldn't break old machines as layout stays the same but would allow to start old machine with pc-dimms which is fine from migration pov as target also should be able to start QEMU with the same options for migration to start.
Quoted 18 lines
hw/arm/virt.c +11
-1+ vms->memmap = extended_memmap;for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + vms->device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + vms->device_memory_size = ms->maxram_size - ms->ram_size + + ms->ram_slots * GiB;
So does everyone agree on this device memory size computation? I would like to make sure this is future proof. Do I need to add a machine option like on x86 to enforce slot alignment or is it OK?
Thanks
Eric
Quoted 48 lines
Quoted 47 lines
hw/arm/virt.c +11
-5+ + vms->high_io_base = vms->device_memory_base + +ROUND_UP (vms->device_memory_size, GiB); +if (vms->high_io_base < vms->device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {@@ -1387,6 +1420,7 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; }static void machvirt_init (MachineState *machine)@@ -1470,11 +1504,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );include/hw/arm/virt.h +3
-0@@ -132,6 +132,9 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; +hwaddr device_memory_base; +hwaddr device_memory_size; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [PATCH v7 07/17] hw/arm/virt: Dynamic memory map depending on RAM requirements
Quoted 86 lines · On Fri, 22 Feb 2019 15:06:14 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 69 lines · Igor Mammedov
Quoted 66 lines · On Wed, 20 Feb 2019 23:39:53 +0100 Eric Auger <[email protected]> wrote:
Up to now the memory map has been static and the high IO region base has always been 256GiB.
This patch modifies the
virt_set_memmap()function, which freezes the memory map, so that the high IO range base becomes floating, located after the initial RAM and the device memory.The function computes
- the base of the device memory,
- the size of the device memory and
- the highest GPA used in the memory map.The two former will be used when defining the device memory region while the latter will be used at VM creation to choose the requested IPA size.
Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.
Still the user cannot request an initial RAM size greater than 255GB. Also we handle the case where maxmem or slots options are passed, although no device memory is usable at the moment. In this case, we just ignore those settings.
Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 47 ++++++++++++++++++++++++++++++++++--------- include/hw/arm/virt.h | 3 +++ 2 files changed, 41 insertions(+), 9 deletions(-)hw/arm/virt.c +5
-3@@ -107,8 +107,9 @@* of a terabyte of RAM will be doing it on a host with more than a * terabyte of physical address space.) */ -#define RAMLIMIT_GB 255 -#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024) +#define RAMBASE GiB +#define LEGACY_RAMLIMIT_GB 255 +#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)/* Addresses and sizes of our components. * 0..128MB is space for a flash device so we can run bootrom code such as UEFI. @@ -149,7 +150,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PCIE_MMIO] = { 0x10000000, 0x2eff0000 }, [VIRT_PCIE_PIO] = { 0x3eff0000, 0x00010000 }, [VIRT_PCIE_ECAM] = { 0x3f000000, 0x01000000 }, - [VIRT_MEM] = { 0x40000000, RAMLIMIT_BYTES }, + [VIRT_MEM] = { RAMBASE, LEGACY_RAMLIMIT_BYTES }, };/* @@ -1367,16 +1368,48 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)static void virt_set_memmap (VirtMachineState *vms) { +MachineState *ms =MACHINE (vms);hwaddr base;int i;Quoted 14 lines
hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1385
Quoted 11 lines
hw/arm/virt.c +11
-0+if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) { +error_report ("mach-virt: does not support device memory: " +"ignore maxmem and slots options" ); + ms->maxram_size = ms->ram_size; + ms->ram_slots =0 ; + } +if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB ); +exit (1 ); + }I'd drop these checks and amend below code so that it would init
device_memorylogic whenram_slots> 0. It should simplify follow up patches by dropping all machine version specific parts.
Quoted 2 lines
I don't have sufficient knowledge of virtio-mem/virtio-pmem. Do they also use slots?
if they don't then they will have to take care of it for every machine anyway so I'd dismiss that from consideration.
Quoted 25 lines
Quoted 25 lines
It shouldn't break old machines as layout stays the same but would allow to start old machine with pc-dimms which is fine from migration pov as target also should be able to start QEMU with the same options for migration to start.
Quoted 18 lines
hw/arm/virt.c +11
-1+ vms->memmap = extended_memmap;for (i =0 ; i <ARRAY_SIZE (base_memmap); i++) { vms->memmap[i] = base_memmap[i]; } - vms->high_io_base =256 * GiB;/* Top of the legacy initial RAM region */ +/* +* We compute the base of the high IO region depending on the +* amount of initial and device memory. The device memory start/size +* is aligned on 1GiB. We never put the high IO region below 256GiB +* so that if maxram_size is < 255GiB we keep the legacy memory map. +* The device region size assumes 1GiB page max alignment per slot. +*/ + vms->device_memory_base =ROUND_UP (RAMBASE + ms->ram_size, GiB); + vms->device_memory_size = ms->maxram_size - ms->ram_size + + ms->ram_slots * GiB;
Quoted 3 lines
So does everyone agree on this device memory size computation? I would like to make sure this is future proof. Do I need to add a machine option like on x86 to enforce slot alignment or is it OK?
computation looks fine to me,
Though, I'd try not duplicate data in vms->device_memory_base and vms->device_memory_size as vms->device_memory is sufficient, see comment in 13/17
Quoted 52 lines
Thanks
Eric
Quoted 48 lines
Quoted 47 lines
hw/arm/virt.c +11
-5+ + vms->high_io_base = vms->device_memory_base + +ROUND_UP (vms->device_memory_size, GiB); +if (vms->high_io_base < vms->device_memory_base) { +error_report ("maxmem/slots too huge" ); +exit (EXIT_FAILURE ); + } +if (vms->high_io_base <256 * GiB) { + vms->high_io_base =256 * GiB; + } base = vms->high_io_base;for (i =VIRT_LOWMEMMAP_LAST ; i <ARRAY_SIZE (extended_memmap); i++) {@@ -1387,6 +1420,7 @@ static void virt_set_memmap(VirtMachineState *vms) vms->memmap[i].size = size; base += size; } + vms->highest_gpa = base -1 ; }static void machvirt_init (MachineState *machine)@@ -1470,11 +1504,6 @@ static void machvirt_init(MachineState *machine) vms->smp_cpus = smp_cpus; -if (machine->ram_size > vms->memmap[VIRT_MEM ].size) { -error_report ("mach-virt: cannot model more than %dGB RAM" ,RAMLIMIT_GB ); -exit (1 ); - } -if (vms->virt &&kvm_enabled ()) {error_report ("mach-virt: KVM does not support providing " "Virtualization extensions to the guest CPU" );include/hw/arm/virt.h +3
-0@@ -132,6 +132,9 @@ typedef struct {uint32_t iommu_phandle;int psci_conduit;hwaddr high_io_base; +hwaddr highest_gpa; +hwaddr device_memory_base; +hwaddr device_memory_size; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v7 8/17 hw/arm/virt: Implement kvm_type function for 4.0 machine
In reply to Eric AugerThis patch implements the machine class kvm_type() callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.
---
v6 -> v7:
- Introduce RAMBASE and rename add LEGACY_ prefix in that patch
- use local variables with explicit names in virt_set_memmap:
device_memory_base, device_memory_size
- add an extended_memmap field in the class
v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB
--- hw/arm/virt.c | 50 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 2 ++ 2 files changed, 51 insertions(+), 1 deletion(-)
hw/arm/virt.c +49 -1
@@ -1437,7 +1437,14 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type(), +* if set, to create a VM with the right number of IPA bits. +*/ + +if (!mc->kvm_type || !kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet
@@ -1814,6 +1821,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -1838,6 +1875,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;
@@ -1909,6 +1947,12 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); +if (vmc->no_extended_memmap) { + vms->extended_memmap = false; + }else { + vms->extended_memmap = true; + } + vms->irqmap = a15irqmap; }
@@ -1939,8 +1983,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(4, 0)static void virt_machine_3_1_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc));virt_machine_4_0_options (mc);compat_props_add (mc->compat_props, hw_compat_3_1, hw_compat_3_1_len); + +/* extended memory map is enabled from 4.0 onwards */ + vmc->no_extended_memmap = true; }DEFINE_VIRT_MACHINE (3 ,1 )
include/hw/arm/virt.h +2 -0
@@ -106,6 +106,7 @@ typedef struct {bool claim_edge_triggered_timers;bool smbios_old_sys_ver;bool no_highmem_ecam; +bool no_extended_memmap; } VirtMachineClass; typedef struct {
@@ -135,6 +136,7 @@ typedef struct {hwaddr highest_gpa;hwaddr device_memory_base;hwaddr device_memory_size; +bool extended_memmap; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 08/17] hw/arm/virt: Implement kvm_type function for 4.0 machine
Quoted 103 lines · On Wed, 20 Feb 2019 23:39:54 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.Signed-off-by Eric Auger <[email protected]>---
v6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 50 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 2 ++ 2 files changed, 51 insertions(+), 1 deletion(-)hw/arm/virt.c +45
-1@@ -1437,7 +1437,14 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type(), +* if set, to create a VM with the right number of IPA bits. +*/ + +if (!mc->kvm_type || !kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1814,6 +1821,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -1838,6 +1875,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;@@ -1909,6 +1947,12 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); +if (vmc->no_extended_memmap) { + vms->extended_memmap = false; + }else { + vms->extended_memmap = true; + } + vms->irqmap = a15irqmap; }
hw/arm/virt.c DEFINE_VIRT_MACHINE_AS_LATEST(4, 0) · line 1991
Quoted 9 lines
hw/arm/virt.c +4
-0@@ -1939,8 +1983,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(4, 0)static void virt_machine_3_1_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc));virt_machine_4_0_options (mc);compat_props_add (mc->compat_props, hw_compat_3_1, hw_compat_3_1_len); + +/* extended memory map is enabled from 4.0 onwards */ + vmc->no_extended_memmap = true;
That's probably was asked in v6,
do we really need this knob?
Quoted 23 lines
hw/arm/virt.c +0
-0}DEFINE_VIRT_MACHINE (3 ,1 )include/hw/arm/virt.h +2
-0@@ -106,6 +106,7 @@ typedef struct {bool claim_edge_triggered_timers;bool smbios_old_sys_ver;bool no_highmem_ecam; +bool no_extended_memmap; } VirtMachineClass; typedef struct {@@ -135,6 +136,7 @@ typedef struct {hwaddr highest_gpa;hwaddr device_memory_base;hwaddr device_memory_size; +bool extended_memmap; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-devel] [PATCH v7 08/17] hw/arm/virt: Implement kvm_type function for 4.0 machine
Hi Igor,
Quoted 107 lines · Igor Mammedov
Quoted 104 lines · On Wed, 20 Feb 2019 23:39:54 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.Signed-off-by Eric Auger <[email protected]>---
v6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 50 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 2 ++ 2 files changed, 51 insertions(+), 1 deletion(-)hw/arm/virt.c +45
-1@@ -1437,7 +1437,14 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type(), +* if set, to create a VM with the right number of IPA bits. +*/ + +if (!mc->kvm_type || !kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1814,6 +1821,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -1838,6 +1875,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;@@ -1909,6 +1947,12 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); +if (vmc->no_extended_memmap) { + vms->extended_memmap = false; + }else { + vms->extended_memmap = true; + } + vms->irqmap = a15irqmap; }@@ -1939,8 +1983,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(4, 0)
Quoted 10 lines
hw/arm/virt.c DEFINE_VIRT_MACHINE_AS_LATEST(4, 0) · line 1991
Quoted 8 lines
hw/arm/virt.c +4
-0static void virt_machine_3_1_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc));virt_machine_4_0_options (mc);compat_props_add (mc->compat_props, hw_compat_3_1, hw_compat_3_1_len); + +/* extended memory map is enabled from 4.0 onwards */ + vmc->no_extended_memmap = true;That's probably was asked in v6,
do we really need this knob?
Yes the point was raised by Peter and I replied by another question ;-) " But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?
"
without this knob:
- pre-4.0 machines will gain the capability to support more than 255GB initial RAM if the kernel supports dynamic IPA setting
- pre-4.0 machines will gain PCDIMM/NVDIMM support
- another concern, maxmem and slots were not checked previously. If for some reason - without instantiating the actual slots -, the user specified it this was previously ignored. Now this is not anymore as both parameters allow to compute the requested IPA range. So this may fail now.
So I thought this was clearer to disable all the above for pre-4.0 machines. However if both of you agree, I will remove it.
thanks
Eric
Quoted 25 lines
Quoted 24 lines
hw/arm/virt.c +0
-0}DEFINE_VIRT_MACHINE (3 ,1 )include/hw/arm/virt.h +2
-0@@ -106,6 +106,7 @@ typedef struct {bool claim_edge_triggered_timers;bool smbios_old_sys_ver;bool no_highmem_ecam; +bool no_extended_memmap; } VirtMachineClass; typedef struct {@@ -135,6 +136,7 @@ typedef struct {hwaddr highest_gpa;hwaddr device_memory_base;hwaddr device_memory_size; +bool extended_memmap; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 08/17] hw/arm/virt: Implement kvm_type function for 4.0 machine
Quoted 120 lines · On Fri, 22 Feb 2019 15:01:25 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 107 lines · Igor Mammedov
Quoted 104 lines · On Wed, 20 Feb 2019 23:39:54 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.Signed-off-by Eric Auger <[email protected]>---
v6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 50 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 2 ++ 2 files changed, 51 insertions(+), 1 deletion(-)hw/arm/virt.c +45
-1@@ -1437,7 +1437,14 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type(), +* if set, to create a VM with the right number of IPA bits. +*/ + +if (!mc->kvm_type || !kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1814,6 +1821,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -1838,6 +1875,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;@@ -1909,6 +1947,12 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); +if (vmc->no_extended_memmap) { + vms->extended_memmap = false; + }else { + vms->extended_memmap = true; + } + vms->irqmap = a15irqmap; }@@ -1939,8 +1983,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(4, 0)Quoted 10 lines
hw/arm/virt.c DEFINE_VIRT_MACHINE_AS_LATEST(4, 0) · line 1991
Quoted 8 lines
hw/arm/virt.c +4
-0static void virt_machine_3_1_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc));virt_machine_4_0_options (mc);compat_props_add (mc->compat_props, hw_compat_3_1, hw_compat_3_1_len); + +/* extended memory map is enabled from 4.0 onwards */ + vmc->no_extended_memmap = true;That's probably was asked in v6,
do we really need this knob?
Quoted 7 lines
Yes the point was raised by Peter and I replied by another question ;-) " But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?
"
without this knob:
- pre-4.0 machines will gain the capability to support more than 255GB initial RAM if the kernel supports dynamic IPA setting
should be fine, you shouldn't be able to start old QEMU with more than 255Gb
Quoted 1 line
- pre-4.0 machines will gain PCDIMM/NVDIMM support
ditto
Quoted 5 lines
- another concern, maxmem and slots were not checked previously. If for some reason - without instantiating the actual slots -, the user specified it this was previously ignored. Now this is not anymore as both parameters allow to compute the requested IPA range. So this may fail now.
well that's in category of a broken setup even if QEMU didn't complain about it before. User should fix it instead of QEMU supporting madness. (I think that such invariant doesn't even deserve deprecation process)
Quoted 32 lines
So I thought this was clearer to disable all the above for pre-4.0 machines. However if both of you agree, I will remove it.
thanks
Eric
Quoted 25 lines
Quoted 24 lines
hw/arm/virt.c +0
-0}DEFINE_VIRT_MACHINE (3 ,1 )include/hw/arm/virt.h +2
-0@@ -106,6 +106,7 @@ typedef struct {bool claim_edge_triggered_timers;bool smbios_old_sys_ver;bool no_highmem_ecam; +bool no_extended_memmap; } VirtMachineClass; typedef struct {@@ -135,6 +136,7 @@ typedef struct {hwaddr highest_gpa;hwaddr device_memory_base;hwaddr device_memory_size; +bool extended_memmap; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Re: [Qemu-devel] [PATCH v7 08/17] hw/arm/virt: Implement kvm_type function for 4.0 machine
Hi Igor,
Quoted 135 lines · Igor Mammedov
Quoted 120 lines · On Fri, 22 Feb 2019 15:01:25 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 107 lines · Igor Mammedov
Quoted 104 lines · On Wed, 20 Feb 2019 23:39:54 +0100 Eric Auger <[email protected]> wrote:
This patch implements the machine class
kvm_type()callback. It returns the number of bits requested to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though theKVM_CREATE_VMioctl and this allows KVM to set the stage2 tables dynamically.Signed-off-by Eric Auger <[email protected]>---
v6 -> v7:
- Introduce RAMBASE and rename addLEGACY_prefix in that patch
- use local variables with explicit names invirt_set_memmap:
device_memory_base,device_memory_size
- add anextended_memmapfield in the classv5 -> v6:
- add some comments
- high IO region cannot start before 256GiB--- hw/arm/virt.c | 50 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 2 ++ 2 files changed, 51 insertions(+), 1 deletion(-)hw/arm/virt.c +45
-1@@ -1437,7 +1437,14 @@ static void machvirt_init(MachineState *machine)bool firmware_loaded = bios_name ||drive_get (IF_PFLASH ,0 ,0 );bool aarch64 = true; -virt_set_memmap (vms); +/* +* In accelerated mode, the memory map is computed in kvm_type(), +* if set, to create a VM with the right number of IPA bits. +*/ + +if (!mc->kvm_type || !kvm_enabled ()) { +virt_set_memmap (vms); + }/* We can probe only here because during property set * KVM is not available yet @@ -1814,6 +1821,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,return NULL ; } +/* +* for arm64 kvm_type [7-0] encodes the requested number of bits +* in the IPA address space +*/ +static int virt_kvm_type (MachineState *ms,const char *type_str) +{ +VirtMachineState *vms =VIRT_MACHINE (ms); +int max_vm_pa_size =kvm_arm_get_max_vm_ipa_size (ms); +int requested_pa_size; + +/* we freeze the memory map to compute the highest gpa */ +virt_set_memmap (vms); + + requested_pa_size =64 -clz64 (vms->highest_gpa); + +if (requested_pa_size > max_vm_pa_size) { +error_report ("-m and ,maxmem option values " +"require an IPA range (%d bits) larger than " +"the one supported by the host (%d bits)" , + requested_pa_size, max_vm_pa_size); +exit (1 ); + } +/* +* By default we return 0 which corresponds to an implicit legacy +* 40b IPA setting. Otherwise we return the actual requested PA +* logsize +*/ +return requested_pa_size >40 ? requested_pa_size :0 ; +} +static void virt_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -1838,6 +1875,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_instance_props = virt_cpu_index_to_props; mc->default_cpu_type =ARM_CPU_TYPE_NAME ("cortex-a15" ); mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; + mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->plug = virt_machine_device_plug_cb;@@ -1909,6 +1947,12 @@ static void virt_instance_init(Object *obj)"Valid values are none and smmuv3" , NULL); +if (vmc->no_extended_memmap) { + vms->extended_memmap = false; + }else { + vms->extended_memmap = true; + } + vms->irqmap = a15irqmap; }@@ -1939,8 +1983,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(4, 0)Quoted 10 lines
hw/arm/virt.c DEFINE_VIRT_MACHINE_AS_LATEST(4, 0) · line 1991
Quoted 8 lines
hw/arm/virt.c +4
-0static void virt_machine_3_1_options (MachineClass *mc) { +VirtMachineClass *vmc =VIRT_MACHINE_CLASS (OBJECT_CLASS (mc));virt_machine_4_0_options (mc);compat_props_add (mc->compat_props, hw_compat_3_1, hw_compat_3_1_len); + +/* extended memory map is enabled from 4.0 onwards */ + vmc->no_extended_memmap = true;That's probably was asked in v6,
do we really need this knob?Quoted 7 lines
Yes the point was raised by Peter and I replied by another question ;-) " But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?
"
without this knob:
- pre-4.0 machines will gain the capability to support more than 255GB initial RAM if the kernel supports dynamic IPA settingshould be fine, you shouldn't be able to start old QEMU with more than 255Gb
Quoted 1 line
- pre-4.0 machines will gain PCDIMM/NVDIMM support
ditto
Quoted 8 lines
Quoted 5 lines
- another concern, maxmem and slots were not checked previously. If for some reason - without instantiating the actual slots -, the user specified it this was previously ignored. Now this is not anymore as both parameters allow to compute the requested IPA range. So this may fail now.
well that's in category of a broken setup even if QEMU didn't complain about it before. User should fix it instead of QEMU supporting madness. (I think that such invariant doesn't even deserve deprecation process)
OK. So I will remove the knob in the next release.
thanks
Eric
Quoted 34 lines
Quoted 33 lines
So I thought this was clearer to disable all the above for pre-4.0 machines. However if both of you agree, I will remove it.
thanks
Eric
Quoted 25 lines
Quoted 24 lines
hw/arm/virt.c +0
-0}DEFINE_VIRT_MACHINE (3 ,1 )include/hw/arm/virt.h +2
-0@@ -106,6 +106,7 @@ typedef struct {bool claim_edge_triggered_timers;bool smbios_old_sys_ver;bool no_highmem_ecam; +bool no_extended_memmap; } VirtMachineClass; typedef struct {@@ -135,6 +136,7 @@ typedef struct {hwaddr highest_gpa;hwaddr device_memory_base;hwaddr device_memory_size; +bool extended_memmap; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v7 9/17 hw/arm/virt: Bump the 255GB initial RAM limit
In reply to Eric AugerNow we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel.
---
v6 -> v7
- handle TCG case
- set_memmap modifications moved to previous patches
--- hw/arm/virt.c | 54 ++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 24 deletions(-)
hw/arm/virt.c +30 -24
@@ -59,6 +59,7 @@#include "qapi/visitor.h" #include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" +#include "target/arm/internals.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -93,21 +94,8 @@#define PLATFORM_BUS_NUM_IRQS 64 -/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means -* RAM can go up to the 256GB mark, leaving 256GB of the physical -* address space unallocated and free for future use between 256G and 512G. -* If we need to provide more RAM to VMs in the future then we need to: -* * allocate a second bank of RAM starting at 2TB and working up -* * fix the DT and ACPI table generation code in QEMU to correctly -* report two split lumps of RAM to the guest -* * fix KVM in the host kernel to allow guests with >40 bit address spaces -* (We don't want to fill all the way up to 512GB with RAM because -* we might want it for non-RAM purposes later. Conversely it seems -* reasonable to assume that anybody configuring a VM with a quarter -* of a terabyte of RAM will be doing it on a host with more than a -* terabyte of physical address space.) -*/ #define RAMBASE GiB +/* Legacy RAM limit in GB (< version 4.0) */ #define LEGACY_RAMLIMIT_GB 255#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
@@ -1372,16 +1360,18 @@ static void virt_set_memmap(VirtMachineState *vms)hwaddr base;int i; -if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) { -error_report ("mach-virt: does not support device memory: " -"ignore maxmem and slots options" ); - ms->maxram_size = ms->ram_size; - ms->ram_slots =0 ; - } -if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { -error_report ("mach-virt: cannot model more than %dGB RAM" , -LEGACY_RAMLIMIT_GB ); -exit (1 ); +if (!vms->extended_memmap) { +if (ms->maxram_size > ms->ram_size || ms->ram_slots >0 ) { +error_report ("mach-virt: does not support device memory: " +"ignore maxmem and slots options" ); + ms->maxram_size = ms->ram_size; + ms->ram_slots =0 ; + } +if (ms->ram_size > (ram_addr_t )LEGACY_RAMLIMIT_BYTES ) { +error_report ("mach-virt: cannot model more than %dGB RAM" , +LEGACY_RAMLIMIT_GB ); +exit (1 ); + } } vms->memmap = extended_memmap;
@@ -1598,6 +1588,22 @@ static void machvirt_init(MachineState *machine)fdt_add_timer_nodes (vms);fdt_add_cpu_nodes (vms); +if (!kvm_enabled ()) { +ARMCPU *cpu =ARM_CPU (first_cpu); +bool aarch64 =object_property_get_bool (OBJECT (cpu),"aarch64" ,NULL ); + +if (aarch64 && vms->highmem) { +int requested_pa_size, pamax =arm_pamax (cpu); + + requested_pa_size =64 -clz64 (vms->highest_gpa); +if (pamax < requested_pa_size) { +error_report ("VCPU supports less PA bits (%d) than requested " +"by the memory map (%d)" , pamax, requested_pa_size); +exit (1 ); + } + } + } +memory_region_allocate_system_memory (ram,NULL ,"mach-virt.ram" , machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram);
Signature · 1 line
2.20.1
PATCH v7 10/17 hw/arm/virt: Add memory hotplug framework
In reply to Eric AugerThis patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.
---
v4 -> v5:
- change in pc_dimm_pre_plug signature
- CONFIG_MEM_HOTPLUG replaced by CONFIG_MEM_DEVICE and CONFIG_DIMM
v3 -> v4:
- check the memory device is not hotplugged
v2 -> v3:
- change in pc_dimm_plug()'s signature
- add pc_dimm_pre_plug call
v1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- added pre_plug cb
- don't handle get_memory_region failure anymore
--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt.c | 64 ++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
default-configs/arm-softmmu.mak +2 -0
@@ -163,3 +163,5 @@ CONFIG_PCI_EXPRESS_DESIGNWARE=y CONFIG_STRONGARM=y CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y +CONFIG_MEM_DEVICE=y +CONFIG_DIMM=y
hw/arm/virt.c +63 -1
94 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v7 10/17] hw/arm/virt: Add memory hotplug framework
Quoted 6 lines · On Wed, 20 Feb 2019 23:39:56 +0100 Eric Auger <[email protected]> wrote:
This patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Kwangwoo Lee <[email protected]>
with nit below fixed
Quoted 98 lines
---
v4 -> v5:
- change inpc_dimm_pre_plugsignature
-CONFIG_MEM_HOTPLUGreplaced byCONFIG_MEM_DEVICEandCONFIG_DIMMv3 -> v4:
- check the memory device is not hotpluggedv2 -> v3:
- change in pc_dimm_plug()'s signature
- addpc_dimm_pre_plugcallv1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- addedpre_plugcb
- don't handleget_memory_regionfailure anymore--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt.c | 64 ++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)default-configs/arm-softmmu.mak +2
-0@@ -163,3 +163,5 @@ CONFIG_PCI_EXPRESS_DESIGNWARE=y CONFIG_STRONGARM=y CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y +CONFIG_MEM_DEVICE=y +CONFIG_DIMM=yhw/arm/virt.c +45
-0@@ -60,6 +60,8 @@#include "standard-headers/linux/input.h" #include "hw/arm/smmuv3.h" #include "target/arm/internals.h" +#include "hw/mem/pc-dimm.h" +#include "hw/mem/nvdimm.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -1804,6 +1806,49 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)return ms->possible_cpus; } +static void virt_memory_pre_plug (HotplugHandler *hotplug_dev,DeviceState *dev, +Error **errp) +{ +const bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM ); + +if (dev->hotplugged) { +error_setg (errp,"memory hotplug is not supported" ); + } + +if (is_nvdimm) { +error_setg (errp,"nvdimm is not yet supported" ); +return ; + } + +pc_dimm_pre_plug (PC_DIMM (dev),MACHINE (hotplug_dev),NULL , errp); +} + +static void virt_memory_plug (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +Error *local_err =NULL ; + +pc_dimm_plug (PC_DIMM (dev),MACHINE (vms), &local_err); + +error_propagate (errp, local_err); +} + +static void virt_memory_unplug (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +pc_dimm_unplug (PC_DIMM (dev),MACHINE (hotplug_dev)); +object_unparent (OBJECT (dev)); +} + +static void virt_machine_device_pre_plug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) { +virt_memory_pre_plug (hotplug_dev, dev, errp); + } +} +static void virt_machine_device_plug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {
hw/arm/virt.c static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, · line 1864
Quoted 5 lines
hw/arm/virt.c +2
-0@@ -1815,12 +1860,27 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, SYS_BUS_DEVICE(dev)); } } +if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) { +virt_memory_plug (hotplug_dev, dev, errp);
wrong indent
Quoted 33 lines
hw/arm/virt.c +16
-1+ } +} + +static void virt_machine_device_unplug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM )) { +virt_memory_unplug (hotplug_dev, dev, errp); + }else { +error_setg (errp,"device unplug request for unsupported device" +" type: %s" ,object_get_typename (OBJECT (dev))); + } }static HotplugHandler *virt_machine_get_hotplug_handler (MachineState *machine,DeviceState *dev) { -if (object_dynamic_cast (OBJECT (dev),TYPE_SYS_BUS_DEVICE )) { +if (object_dynamic_cast (OBJECT (dev),TYPE_SYS_BUS_DEVICE ) || + (object_dynamic_cast (OBJECT (dev),TYPE_PC_DIMM ))) {return HOTPLUG_HANDLER (machine); }@@ -1884,7 +1944,9 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->kvm_type = virt_kvm_type;assert (!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; + hc->pre_plug = virt_machine_device_pre_plug_cb; hc->plug = virt_machine_device_plug_cb; + hc->unplug = virt_machine_device_unplug_cb; }static void virt_instance_init (Object *obj)
PATCH v7 11/17 hw/arm/boot: Expose the PC-DIMM nodes in the DT
In reply to Eric AugerFrom: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
NV_DIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NV-DIMM at this stage.
---
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes
v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop
--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
hw/arm/boot.c +42 -0
@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h"
@@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {
@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail; + } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 11/17] hw/arm/boot: Expose the PC-DIMM nodes in the DT
Quoted 5 lines · On Wed, 20 Feb 2019 23:39:57 +0100 Eric Auger <[email protected]> wrote:
From: Shameer Kolothum <[email protected]>
This patch adds memory nodes corresponding to PC-DIMM regions.
NV_DIMMandACPI_NVDIMMconfigs are not yet set for ARM so we
^^^
git grep says it doesn't exists
Quoted 1 line
don't need to care about NV-DIMM at this stage.
we use NVDIMM term everywhere in QEMU so fix it up just for consistency
Quoted 3 lines
Signed-off-by Shameer Kolothum <[email protected]>Signed-off-by Eric Auger <[email protected]>
with all comments addressed
Quoted 68 lines
---
v6 -> v7:
- rework the error messages, use a switch/case
v3 -> v4:
- git rid of @base and @len infdt_add_hotpluggable_memory_nodesv1 -> v2:
- addedqapi_free_MemoryDeviceInfoListand simplify the loop--- hw/arm/boot.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)hw/arm/boot.c +36
-0@@ -19,6 +19,7 @@#include "sysemu/numa.h" #include "hw/boards.h" #include "hw/loader.h" +#include "hw/mem/memory-device.h" #include "elf.h" #include "sysemu/device_tree.h" #include "qemu/config-file.h" @@ -522,6 +523,41 @@ static void fdt_add_psci_node(void *fdt)qemu_fdt_setprop_cell (fdt,"/psci" ,"migrate" , migrate_fn); } +static int fdt_add_hotpluggable_memory_nodes (void *fdt, +uint32_t acells,uint32_t scells) { +MemoryDeviceInfoList *info, *info_list =qmp_memory_device_list (); +MemoryDeviceInfo *mi; +int ret =0 ; + +for (info = info_list; info !=NULL ; info = info->next) { + mi = info->value; +switch (mi->type) { +case MEMORY_DEVICE_INFO_KIND_DIMM : + { +PCDIMMDeviceInfo *di = mi->u.dimm.data; + + ret =fdt_add_memory_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add PCDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + } +default : +fprintf (stderr,"%s memory nodes are not yet supported\n" , +MemoryDeviceInfoKind_str (mi->type)); + ret = -ENOENT ; + goto out; + } + } +out: +qapi_free_MemoryDeviceInfoList (info_list); +return ret; +} +int arm_load_dtb (hwaddr addr,const struct arm_boot_info *binfo,hwaddr addr_limit,AddressSpace *as) {
hw/arm/boot.c int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, · line 663
Quoted 7 lines
hw/arm/boot.c +4
-0@@ -621,6 +657,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, } } + rc =fdt_add_hotpluggable_memory_nodes (fdt, acells, scells); +if (rc <0 ) { +fprintf (stderr,"couldn't add hotpluggable memory nodes\n" ); + goto fail;
wrong indent
Quoted 5 lines
hw/arm/boot.c +2
-0+ } + rc =fdt_path_offset (fdt,"/chosen" );if (rc <0 ) {qemu_fdt_add_subnode (fdt,"/chosen" );
PATCH v7 12/17 hw/arm/virt-acpi-build: Add PC-DIMM in SRAT
In reply to Eric AugerFrom: Shameer Kolothum <[email protected]>
Generate Memory Affinity Structures for PC-DIMM ranges.
---
v6 -> v7:
- add Igor's R-b
v5 -> v6:
- fix mingw compil issue
v4 -> v5:
- Align to x86 code and especially
"pc: acpi: revert back to 1 SRAT entry for hotpluggable area"
v3 -> v4:
- do not use vms->bootinfo.device_memory_start/device_memory_size anymore
v1 -> v2:
- build_srat_hotpluggable_memory movedc to aml-build
--- hw/arm/virt-acpi-build.c | 9 +++++++++ 1 file changed, 9 insertions(+)
hw/arm/virt-acpi-build.c +9 -0
@@ -516,6 +516,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)int i, srat_start;uint64_t mem_base;MachineClass *mc =MACHINE_GET_CLASS (vms); +MachineState *ms =MACHINE (vms);const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids (MACHINE (vms)); srat_start = table_data->len;
@@ -541,6 +542,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } } +if (ms->device_memory) { + numamem =acpi_data_push (table_data,sizeof *numamem); +build_srat_memory (numamem, ms->device_memory->base, +memory_region_size (&ms->device_memory->mr), + nb_numa_nodes -1 , +MEM_AFFINITY_HOTPLUGGABLE |MEM_AFFINITY_ENABLED ); + } +build_header (linker, table_data, (void *)(table_data->data + srat_start),"SRAT" , table_data->len - srat_start,3 ,NULL ,NULL ); }
Signature · 1 line
2.20.1
PATCH v7 13/17 hw/arm/virt: Allocate device_memory
In reply to Eric AugerThe device memory region is located after the initial RAM. its start/size are 1GB aligned.
---
v6 -> v7:
- check the device memory top does not wrap
- check the device memory can fit the slots
v4 -> v5:
- device memory set after the initial RAM
v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM
--- hw/arm/virt.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
hw/arm/virt.c +33 -0
@@ -62,6 +62,7 @@#include "target/arm/internals.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1263,6 +1264,34 @@ static void create_secure_ram(VirtMachineState *vms,g_free (nodename); } +static void create_device_memory (VirtMachineState *vms,MemoryRegion *sysmem) +{ +MachineState *ms =MACHINE (vms); + +if (!vms->device_memory_size) { +return ; + } + +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); +exit (EXIT_FAILURE ); + } + + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = vms->device_memory_base; + +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , vms->device_memory_size); +memory_region_add_subregion (sysmem, ms->device_memory->base, + &ms->device_memory->mr); +} +static void *machvirt_dtb (const struct arm_boot_info *binfo,int *fdt_size) { const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1610,6 +1639,10 @@ static void machvirt_init(MachineState *machine) machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (vms->extended_memmap) { +create_device_memory (vms, sysmem); + } +create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);create_gic (vms, pic);
Signature · 1 line
2.20.1
Re: [Qemu-arm] [PATCH v7 13/17] hw/arm/virt: Allocate device_memory
Quoted 34 lines · On Wed, 20 Feb 2019 23:39:59 +0100 Eric Auger <[email protected]> wrote:
The device memory region is located after the initial RAM. its start/size are 1GB aligned.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Kwangwoo Lee <[email protected]>---
v6 -> v7:
- check the device memory top does not wrap
- check the device memory can fit the slotsv4 -> v5:
- device memory set after the initial RAMv3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- renameVIRT_HOTPLUG_MEMintoVIRT_DEVICE_MEM--- hw/arm/virt.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)hw/arm/virt.c +1
-0@@ -62,6 +62,7 @@#include "target/arm/internals.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1271
Quoted 8 lines
hw/arm/virt.c +5
-0@@ -1263,6 +1264,34 @@ static void create_secure_ram(VirtMachineState *vms,g_free (nodename); } +static void create_device_memory (VirtMachineState *vms,MemoryRegion *sysmem) +{ +MachineState *ms =MACHINE (vms); + +if (!vms->device_memory_size) {
having vms->device_memory_size seems like duplicate, why not to reuse memory_region_size(MachineState::device_memory::mr) like we do elsewhere.
Also it would be better to all device_memory allocation/initialization compact and close to each other like it's done in pc/spapr.
Quoted 37 lines
hw/arm/virt.c +27
-0+return ; + } + +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); +exit (EXIT_FAILURE ); + } + + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = vms->device_memory_base; + +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , vms->device_memory_size); +memory_region_add_subregion (sysmem, ms->device_memory->base, + &ms->device_memory->mr); +} +static void *machvirt_dtb (const struct arm_boot_info *binfo,int *fdt_size) { const VirtMachineState *board = container_of(binfo, VirtMachineState,@@ -1610,6 +1639,10 @@ static void machvirt_init(MachineState *machine) machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (vms->extended_memmap) { +create_device_memory (vms, sysmem); + } +create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);create_gic (vms, pic);
Re: [Qemu-arm] [PATCH v7 13/17] hw/arm/virt: Allocate device_memory
Hi Igor,
Quoted 40 lines · Igor Mammedov
Quoted 37 lines · On Wed, 20 Feb 2019 23:39:59 +0100 Eric Auger <[email protected]> wrote:
The device memory region is located after the initial RAM. its start/size are 1GB aligned.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Kwangwoo Lee <[email protected]>---
v6 -> v7:
- check the device memory top does not wrap
- check the device memory can fit the slotsv4 -> v5:
- device memory set after the initial RAMv3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- renameVIRT_HOTPLUG_MEMintoVIRT_DEVICE_MEM--- hw/arm/virt.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)hw/arm/virt.c +1
-0@@ -62,6 +62,7 @@#include "target/arm/internals.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -1263,6 +1264,34 @@ static void create_secure_ram(VirtMachineState *vms,g_free (nodename); }
Quoted 7 lines
hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1271
Quoted 5 lines
hw/arm/virt.c +5
-0+static void create_device_memory (VirtMachineState *vms,MemoryRegion *sysmem) +{ +MachineState *ms =MACHINE (vms); + +if (!vms->device_memory_size) {having
vms->device_memory_sizeseems like duplicate, why not to reusememory_region_size(MachineState::device_memory::mr)like we do elsewhere.
OK so you mean allocating the ms->device_memory in virt_setmmap? In that case, I wonder if all that code shouldn't land in set_memmap directly?
Quoted 3 lines
Also it would be better to all
device_memoryallocation/initialization compact and close to each other like it's done in pc/spapr.
I am not sure I get your point here. alloc & init are done in this function. Or do you mean I should rather move that into set_memmap?
Thanks
Eric
Quoted 39 lines
Quoted 38 lines
hw/arm/virt.c +27
-0+return ; + } + +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); +exit (EXIT_FAILURE ); + } + + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = vms->device_memory_base; + +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , vms->device_memory_size); +memory_region_add_subregion (sysmem, ms->device_memory->base, + &ms->device_memory->mr); +} +static void *machvirt_dtb (const struct arm_boot_info *binfo,int *fdt_size) { const VirtMachineState *board = container_of(binfo, VirtMachineState,@@ -1610,6 +1639,10 @@ static void machvirt_init(MachineState *machine) machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (vms->extended_memmap) { +create_device_memory (vms, sysmem); + } +create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);create_gic (vms, pic);
Re: [Qemu-devel] [PATCH v7 13/17] hw/arm/virt: Allocate device_memory
Quoted 50 lines · On Fri, 22 Feb 2019 15:15:36 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 40 lines · Igor Mammedov
Quoted 37 lines · On Wed, 20 Feb 2019 23:39:59 +0100 Eric Auger <[email protected]> wrote:
The device memory region is located after the initial RAM. its start/size are 1GB aligned.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Kwangwoo Lee <[email protected]>---
v6 -> v7:
- check the device memory top does not wrap
- check the device memory can fit the slotsv4 -> v5:
- device memory set after the initial RAMv3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- renameVIRT_HOTPLUG_MEMintoVIRT_DEVICE_MEM--- hw/arm/virt.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)hw/arm/virt.c +1
-0@@ -62,6 +62,7 @@#include "target/arm/internals.h" #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" +#include "hw/acpi/acpi.h" #define DEFINE_VIRT_MACHINE_LATEST (major, minor, latest) \ static void virt_##major##_##minor##_class_init(ObjectClass *oc, \@@ -1263,6 +1264,34 @@ static void create_secure_ram(VirtMachineState *vms,g_free (nodename); }Quoted 7 lines
hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1271
Quoted 5 lines
hw/arm/virt.c +5
-0+static void create_device_memory (VirtMachineState *vms,MemoryRegion *sysmem) +{ +MachineState *ms =MACHINE (vms); + +if (!vms->device_memory_size) {having
vms->device_memory_sizeseems like duplicate, why not to reusememory_region_size(MachineState::device_memory::mr)like we do elsewhere.
Quoted 7 lines
OK so you mean allocating the
ms->device_memoryinvirt_setmmap? In that case, I wonder if all that code shouldn't land inset_memmapdirectly?Quoted 3 lines
Also it would be better to all
device_memoryallocation/initialization compact and close to each other like it's done in pc/spapr.I am not sure I get your point here. alloc & init are done in this function. Or do you mean I should rather move that into
set_memmap?
I don't have a preference where to put it, could be here or in set_memmap, what I do care is to keep all device_memory related parts together (like calculating device_memory base and size and allocating/initializing structure itself, it's much easier to read when one doesn't have to jump around code and could see all related code in one place)
Quoted 44 lines
Thanks
Eric
Quoted 39 lines
Quoted 38 lines
hw/arm/virt.c +27
-0+return ; + } + +if (ms->ram_slots >ACPI_MAX_RAM_SLOTS ) { +error_report ("unsupported number of memory slots: %" PRIu64, + ms->ram_slots); +exit (EXIT_FAILURE ); + } + +if (QEMU_ALIGN_UP (ms->maxram_size, GiB) != ms->maxram_size) { +error_report ("maximum memory size must be GiB aligned" ); +exit (EXIT_FAILURE ); + } + + ms->device_memory =g_malloc0 (sizeof (*ms->device_memory)); + ms->device_memory->base = vms->device_memory_base; + +memory_region_init (&ms->device_memory->mr,OBJECT (vms), +"device-memory" , vms->device_memory_size); +memory_region_add_subregion (sysmem, ms->device_memory->base, + &ms->device_memory->mr); +} +static void *machvirt_dtb (const struct arm_boot_info *binfo,int *fdt_size) { const VirtMachineState *board = container_of(binfo, VirtMachineState,@@ -1610,6 +1639,10 @@ static void machvirt_init(MachineState *machine) machine->ram_size);memory_region_add_subregion (sysmem, vms->memmap[VIRT_MEM ].base, ram); +if (vms->extended_memmap) { +create_device_memory (vms, sysmem); + } +create_flash (vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);create_gic (vms, pic);
PATCH v7 14/17 nvdimm: use configurable ACPI IO base and size
In reply to Eric AugerFrom: Kwangwoo Lee <[email protected]>
This patch uses configurable IO base and size to create NPIO AML for ACPI NFIT. Since a different architecture like AArch64 does not use port-mapped IO, a configurable IO base is required to create correct mapping of ACPI IO address and size.
---
v6 -> v7:
- Use NvdimmDsmIO constant
- use AcpiGenericAddress instead of AcpiNVDIMMIOEntry
v2 -> v3:
- s/size/len in pc_piix.c and pc_q35.c
--- hw/acpi/nvdimm.c | 31 ++++++++++++++++++++++--------- hw/i386/pc_piix.c | 6 +++++- hw/i386/pc_q35.c | 6 +++++- include/hw/mem/nvdimm.h | 4 ++++ 4 files changed, 36 insertions(+), 11 deletions(-)
hw/acpi/nvdimm.c +22 -9
@@ -33,6 +33,9 @@#include "hw/nvram/fw_cfg.h" #include "hw/mem/nvdimm.h" +const struct AcpiGenericAddress NvdimmDsmIO = { .space_id =AML_AS_SYSTEM_IO , + .bit_width =NVDIMM_ACPI_IO_LEN <<3 , .address =NVDIMM_ACPI_IO_BASE }; +static int nvdimm_device_list (Object *obj,void *opaque) {GSList **list = opaque;
@@ -929,8 +932,8 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io, FWCfgState *fw_cfg, Object *owner) {memory_region_init_io (&state->io_mr, owner, &nvdimm_dsm_ops, state, -"nvdimm-acpi-io" ,NVDIMM_ACPI_IO_LEN ); -memory_region_add_subregion (io,NVDIMM_ACPI_IO_BASE , &state->io_mr); +"nvdimm-acpi-io" , state->dsm_io.bit_width >>3 ); +memory_region_add_subregion (io, state->dsm_io.address, &state->io_mr); state->dsm_mem =g_array_new (false, true/* clear */ ,1 );acpi_data_push (state->dsm_mem,sizeof (NvdimmDsmIn));
@@ -959,12 +962,14 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,#define NVDIMM_QEMU_RSVD_UUID "648B9CF2-CDA1-4312-8AD9-49C4AF32BD62" -static void nvdimm_build_common_dsm (Aml *dev) +static void nvdimm_build_common_dsm (Aml *dev, +AcpiNVDIMMState *acpi_nvdimm_state) {Aml *method, *ifctx, *function, *handle, *uuid, *dsm_mem, *elsectx2;Aml *elsectx, *unsupport, *unpatched, *expected_uuid, *uuid_invalid;Aml *pckg, *pckg_index, *pckg_buf, *field, *dsm_out_buf, *dsm_out_buf_size;uint8_t byte_list[1 ]; +AmlRegionSpace rs; method =aml_method (NVDIMM_COMMON_DSM ,5 ,AML_SERIALIZED ); uuid =aml_arg (0 );
@@ -975,9 +980,16 @@ static void nvdimm_build_common_dsm(Aml *dev)aml_append (method,aml_store (aml_name (NVDIMM_ACPI_MEM_ADDR ), dsm_mem)); +if (acpi_nvdimm_state->dsm_io.space_id ==AML_AS_SYSTEM_IO ) { + rs =AML_SYSTEM_IO ; + }else { + rs =AML_SYSTEM_MEMORY ; + } +/* map DSM memory and IO into ACPI namespace. */ -aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT ,AML_SYSTEM_IO , -aml_int (NVDIMM_ACPI_IO_BASE ),NVDIMM_ACPI_IO_LEN )); +aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT , rs, +aml_int (acpi_nvdimm_state->dsm_io.address), + acpi_nvdimm_state->dsm_io.bit_width >>3 ));aml_append (method,aml_operation_region (NVDIMM_DSM_MEMORY ,AML_SYSTEM_MEMORY , dsm_mem,sizeof (NvdimmDsmIn)));
@@ -1260,7 +1272,8 @@ static void nvdimm_build_nvdimm_devices(Aml *root_dev, uint32_t ram_slots) }static void nvdimm_build_ssdt (GArray *table_offsets,GArray *table_data, -BIOSLinker *linker,GArray *dsm_dma_arrea, +BIOSLinker *linker, +AcpiNVDIMMState *acpi_nvdimm_state,uint32_t ram_slots) {Aml *ssdt, *sb_scope, *dev;
@@ -1288,7 +1301,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,*/ aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0012" ))); -nvdimm_build_common_dsm (dev); +nvdimm_build_common_dsm (dev, acpi_nvdimm_state);/* 0 is reserved for root device. */ nvdimm_build_device_dsm (dev,0 );
@@ -1307,7 +1320,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data, NVDIMM_ACPI_MEM_ADDR);bios_linker_loader_alloc (linker, -NVDIMM_DSM_MEM_FILE , dsm_dma_arrea, +NVDIMM_DSM_MEM_FILE , acpi_nvdimm_state->dsm_mem,sizeof (NvdimmDsmIn), false/* high memory */ ); bios_linker_loader_add_pointer(linker, ACPI_BUILD_TABLE_FILE, mem_addr_offset, sizeof(uint32_t),
@@ -1329,7 +1342,7 @@ void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data,return ; } -nvdimm_build_ssdt (table_offsets, table_data, linker, state->dsm_mem, +nvdimm_build_ssdt (table_offsets, table_data, linker, state, ram_slots); device_list =nvdimm_get_device_list ();
hw/i386/pc_piix.c +5 -1
@@ -298,7 +298,11 @@ static void pc_init1(MachineState *machine, }if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io = NvdimmDsmIO; + +nvdimm_init_acpi_state (acpi_nvdimm_state, system_io, pcms->fw_cfg,OBJECT (pcms)); } }
hw/i386/pc_q35.c +5 -1
@@ -330,7 +330,11 @@ static void pc_q35_init(MachineState *machine)pc_nic_init (pcmc, isa_bus, host_bus);if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io = NvdimmDsmIO; + +nvdimm_init_acpi_state (acpi_nvdimm_state, system_io, pcms->fw_cfg,OBJECT (pcms)); } }
include/hw/mem/nvdimm.h +4 -0
@@ -25,6 +25,7 @@#include "hw/mem/pc-dimm.h" #include "hw/acpi/bios-linker-loader.h" +#include "hw/acpi/aml-build.h" #define NVDIMM_DEBUG 0#define nvdimm_debug (fmt, ...) \
@@ -123,6 +124,8 @@ struct NvdimmFitBuffer { };typedef struct NvdimmFitBuffer NvdimmFitBuffer ; +extern const struct AcpiGenericAddress NvdimmDsmIO; +struct AcpiNVDIMMState {/* detect if NVDIMM support is enabled. */ bool is_enabled;
@@ -140,6 +143,7 @@ struct AcpiNVDIMMState {*/ int32_t persistence;char *persistence_string; +struct AcpiGenericAddress dsm_io; };typedef struct AcpiNVDIMMState AcpiNVDIMMState ;
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v7 14/17] nvdimm: use configurable ACPI IO base and size
Quoted 3 lines · On Wed, 20 Feb 2019 23:40:00 +0100 Eric Auger <[email protected]> wrote:
From: Kwangwoo Lee <[email protected]>
This patch uses configurable IO base and size to create NPIO AML for
s/uses ... /makes IO base and size configurable/
Quoted 26 lines
ACPI NFIT. Since a different architecture like AArch64 does not use port-mapped IO, a configurable IO base is required to create correct mapping of ACPI IO address and size.
Signed-off-by Kwangwoo Lee <[email protected]>Signed-off-by Eric Auger <[email protected]>---
v6 -> v7:
- Use NvdimmDsmIO constant
- use AcpiGenericAddress instead of AcpiNVDIMMIOEntryv2 -> v3:
- s/size/len inpc_piix.candpc_q35.c--- hw/acpi/nvdimm.c | 31 ++++++++++++++++++++++--------- hw/i386/pc_piix.c | 6 +++++- hw/i386/pc_q35.c | 6 +++++- include/hw/mem/nvdimm.h | 4 ++++ 4 files changed, 36 insertions(+), 11 deletions(-)hw/acpi/nvdimm.c +0
-0
hw/acpi/nvdimm.c line 37
Quoted 5 lines
hw/acpi/nvdimm.c +2
-0@@ -33,6 +33,9 @@#include "hw/nvram/fw_cfg.h" #include "hw/mem/nvdimm.h" +const struct AcpiGenericAddress NvdimmDsmIO = { .space_id =AML_AS_SYSTEM_IO , + .bit_width =NVDIMM_ACPI_IO_LEN <<3 , .address =NVDIMM_ACPI_IO_BASE };
it's target specific data, so I'd put this part in x86 specific place hw/i386/acpi-build.[ch] or pc.[ch]
and initialize it like you don in the next patch for virt-arm
maybe add pc_/x86_ prefix to variable name and also we don't do CamelCase with variable names if I recall correctly.
Quoted 92 lines
hw/acpi/nvdimm.c +20
-9+static int nvdimm_device_list (Object *obj,void *opaque) {GSList **list = opaque;@@ -929,8 +932,8 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io, FWCfgState *fw_cfg, Object *owner) {memory_region_init_io (&state->io_mr, owner, &nvdimm_dsm_ops, state, -"nvdimm-acpi-io" ,NVDIMM_ACPI_IO_LEN ); -memory_region_add_subregion (io,NVDIMM_ACPI_IO_BASE , &state->io_mr); +"nvdimm-acpi-io" , state->dsm_io.bit_width >>3 ); +memory_region_add_subregion (io, state->dsm_io.address, &state->io_mr); state->dsm_mem =g_array_new (false, true/* clear */ ,1 );acpi_data_push (state->dsm_mem,sizeof (NvdimmDsmIn));@@ -959,12 +962,14 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,#define NVDIMM_QEMU_RSVD_UUID "648B9CF2-CDA1-4312-8AD9-49C4AF32BD62" -static void nvdimm_build_common_dsm (Aml *dev) +static void nvdimm_build_common_dsm (Aml *dev, +AcpiNVDIMMState *acpi_nvdimm_state) {Aml *method, *ifctx, *function, *handle, *uuid, *dsm_mem, *elsectx2;Aml *elsectx, *unsupport, *unpatched, *expected_uuid, *uuid_invalid;Aml *pckg, *pckg_index, *pckg_buf, *field, *dsm_out_buf, *dsm_out_buf_size;uint8_t byte_list[1 ]; +AmlRegionSpace rs; method =aml_method (NVDIMM_COMMON_DSM ,5 ,AML_SERIALIZED ); uuid =aml_arg (0 );@@ -975,9 +980,16 @@ static void nvdimm_build_common_dsm(Aml *dev)aml_append (method,aml_store (aml_name (NVDIMM_ACPI_MEM_ADDR ), dsm_mem)); +if (acpi_nvdimm_state->dsm_io.space_id ==AML_AS_SYSTEM_IO ) { + rs =AML_SYSTEM_IO ; + }else { + rs =AML_SYSTEM_MEMORY ; + } +/* map DSM memory and IO into ACPI namespace. */ -aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT ,AML_SYSTEM_IO , -aml_int (NVDIMM_ACPI_IO_BASE ),NVDIMM_ACPI_IO_LEN )); +aml_append (method,aml_operation_region (NVDIMM_DSM_IOPORT , rs, +aml_int (acpi_nvdimm_state->dsm_io.address), + acpi_nvdimm_state->dsm_io.bit_width >>3 ));aml_append (method,aml_operation_region (NVDIMM_DSM_MEMORY ,AML_SYSTEM_MEMORY , dsm_mem,sizeof (NvdimmDsmIn)));@@ -1260,7 +1272,8 @@ static void nvdimm_build_nvdimm_devices(Aml *root_dev, uint32_t ram_slots) }static void nvdimm_build_ssdt (GArray *table_offsets,GArray *table_data, -BIOSLinker *linker,GArray *dsm_dma_arrea, +BIOSLinker *linker, +AcpiNVDIMMState *acpi_nvdimm_state,uint32_t ram_slots) {Aml *ssdt, *sb_scope, *dev;@@ -1288,7 +1301,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,*/ aml_append (dev,aml_name_decl ("_HID" ,aml_string ("ACPI0012" ))); -nvdimm_build_common_dsm (dev); +nvdimm_build_common_dsm (dev, acpi_nvdimm_state);/* 0 is reserved for root device. */ nvdimm_build_device_dsm (dev,0 );@@ -1307,7 +1320,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data, NVDIMM_ACPI_MEM_ADDR);bios_linker_loader_alloc (linker, -NVDIMM_DSM_MEM_FILE , dsm_dma_arrea, +NVDIMM_DSM_MEM_FILE , acpi_nvdimm_state->dsm_mem,sizeof (NvdimmDsmIn), false/* high memory */ ); bios_linker_loader_add_pointer(linker, ACPI_BUILD_TABLE_FILE, mem_addr_offset, sizeof(uint32_t),@@ -1329,7 +1342,7 @@ void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data,return ; } -nvdimm_build_ssdt (table_offsets, table_data, linker, state->dsm_mem, +nvdimm_build_ssdt (table_offsets, table_data, linker, state, ram_slots); device_list =nvdimm_get_device_list ();hw/i386/pc_piix.c +0
-0
hw/i386/pc_piix.c static void pc_init1(MachineState *machine, · line 306
Quoted 10 lines
hw/i386/pc_piix.c +5
-1@@ -298,7 +298,11 @@ static void pc_init1(MachineState *machine, }if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io = NvdimmDsmIO; + +nvdimm_init_acpi_state (acpi_nvdimm_state, system_io, pcms->fw_cfg,OBJECT (pcms));
I dislike split initialization, suggest to pass AcpiGenericAddress as an argument to nvdimm_init_acpi_state() and assign it to acpi_nvdimm_state inside, so we could see acpi_nvdimm_state initialization in one place (almost all).
Quoted 48 lines
hw/i386/pc_piix.c +0
-0} }hw/i386/pc_q35.c +5
-1@@ -330,7 +330,11 @@ static void pc_q35_init(MachineState *machine)pc_nic_init (pcmc, isa_bus, host_bus);if (pcms->acpi_nvdimm_state.is_enabled) { -nvdimm_init_acpi_state (&pcms->acpi_nvdimm_state, system_io, +AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io = NvdimmDsmIO; + +nvdimm_init_acpi_state (acpi_nvdimm_state, system_io, pcms->fw_cfg,OBJECT (pcms)); } }include/hw/mem/nvdimm.h +4
-0@@ -25,6 +25,7 @@#include "hw/mem/pc-dimm.h" #include "hw/acpi/bios-linker-loader.h" +#include "hw/acpi/aml-build.h" #define NVDIMM_DEBUG 0#define nvdimm_debug (fmt, ...) \@@ -123,6 +124,8 @@ struct NvdimmFitBuffer { };typedef struct NvdimmFitBuffer NvdimmFitBuffer ; +extern const struct AcpiGenericAddress NvdimmDsmIO; +struct AcpiNVDIMMState {/* detect if NVDIMM support is enabled. */ bool is_enabled;@@ -140,6 +143,7 @@ struct AcpiNVDIMMState {*/ int32_t persistence;char *persistence_string; +struct AcpiGenericAddress dsm_io; };typedef struct AcpiNVDIMMState AcpiNVDIMMState ;
PATCH v7 15/17 hw/arm/virt: Add nvdimm hot-plug infrastructure
In reply to Eric AugerFrom: Kwangwoo Lee <[email protected]>
Pre-plug and plug handlers are prepared for NVDIMM support.
--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt-acpi-build.c | 6 ++++++ hw/arm/virt.c | 22 ++++++++++++++++++++++ include/hw/arm/virt.h | 3 +++ 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
@@ -165,3 +165,5 @@ CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y CONFIG_MEM_DEVICE=y CONFIG_DIMM=y +CONFIG_NVDIMM=y +CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
@@ -784,6 +784,7 @@ staticvoid virt_acpi_build (VirtMachineState *vms,AcpiBuildTables *tables) {VirtMachineClass *vmc =VIRT_MACHINE_GET_CLASS (vms); +MachineState *ms =MACHINE (vms);GArray *table_offsets;unsigned dsdt, xsdt;GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables) } } +if (vms->acpi_nvdimm_state.is_enabled) { +nvdimm_build_acpi (table_offsets, tables_blob, tables->linker, + &vms->acpi_nvdimm_state, ms->ram_slots); + } +if (its_class_name () && !vmc->no_its) {acpi_add_table (table_offsets, tables_blob);build_iort (tables_blob, tables->linker, vms);
hw/arm/virt.c +22 -0
@@ -134,6 +134,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, [VIRT_SMMU] = { 0x09050000, 0x00020000 }, + [VIRT_ACPI_IO] = { 0x09070000, 0x00010000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 },
@@ -1675,6 +1676,18 @@ static void machvirt_init(MachineState *machine)create_platform_bus (vms, pic); +if (vms->acpi_nvdimm_state.is_enabled) { +AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io.space_id =AML_AS_SYSTEM_MEMORY ; + acpi_nvdimm_state->dsm_io.address = + vms->memmap[VIRT_ACPI_IO ].base +NVDIMM_ACPI_IO_BASE ; + acpi_nvdimm_state->dsm_io.bit_width =NVDIMM_ACPI_IO_LEN <<3 ; + +nvdimm_init_acpi_state (acpi_nvdimm_state, sysmem, + vms->fw_cfg,OBJECT (vms)); + } + vms->bootinfo.ram_size = machine->ram_size; vms->bootinfo.kernel_filename = machine->kernel_filename; vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1860,10 +1873,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) {VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM );Error *local_err =NULL ;pc_dimm_plug (PC_DIMM (dev),MACHINE (vms), &local_err); +if (local_err) { + goto out; + } +if (is_nvdimm) { +nvdimm_plug (&vms->acpi_nvdimm_state); + } + +out:error_propagate (errp, local_err); }
include/hw/arm/virt.h +3 -0
@@ -37,6 +37,7 @@#include "hw/arm/arm.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/mem/nvdimm.h" #define NUM_GICV2M_SPIS 64#define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , +VIRT_ACPI_IO , VIRT_LOWMEMMAP_LAST, };
@@ -137,6 +139,7 @@ typedef struct {hwaddr device_memory_base;hwaddr device_memory_size;bool extended_memmap; +AcpiNVDIMMState acpi_nvdimm_state; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 15/17] hw/arm/virt: Add nvdimm hot-plug infrastructure
Quoted 60 lines · On Wed, 20 Feb 2019 23:40:01 +0100 Eric Auger <[email protected]> wrote:
From: Kwangwoo Lee <[email protected]>
Pre-plug and plug handlers are prepared for NVDIMM support.
Signed-off-by Eric Auger <[email protected]>Signed-off-by Kwangwoo Lee <[email protected]>--- default-configs/arm-softmmu.mak | 2 ++ hw/arm/virt-acpi-build.c | 6 ++++++ hw/arm/virt.c | 22 ++++++++++++++++++++++ include/hw/arm/virt.h | 3 +++ 4 files changed, 33 insertions(+)default-configs/arm-softmmu.mak +2
-0@@ -165,3 +165,5 @@ CONFIG_HIGHBANK=y CONFIG_MUSICPAL=y CONFIG_MEM_DEVICE=y CONFIG_DIMM=y +CONFIG_NVDIMM=y +CONFIG_ACPI_NVDIMM=yhw/arm/virt-acpi-build.c +6
-0@@ -784,6 +784,7 @@ staticvoid virt_acpi_build (VirtMachineState *vms,AcpiBuildTables *tables) {VirtMachineClass *vmc =VIRT_MACHINE_GET_CLASS (vms); +MachineState *ms =MACHINE (vms);GArray *table_offsets;unsigned dsdt, xsdt;GArray *tables_blob = tables->table_data;@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables) } } +if (vms->acpi_nvdimm_state.is_enabled) { +nvdimm_build_acpi (table_offsets, tables_blob, tables->linker, + &vms->acpi_nvdimm_state, ms->ram_slots); + } +if (its_class_name () && !vmc->no_its) {acpi_add_table (table_offsets, tables_blob);build_iort (tables_blob, tables->linker, vms);hw/arm/virt.c +1
-0@@ -134,6 +134,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, [VIRT_SMMU] = { 0x09050000, 0x00020000 }, + [VIRT_ACPI_IO] = { 0x09070000, 0x00010000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 },
hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1685
Quoted 10 lines
hw/arm/virt.c +7
-0@@ -1675,6 +1676,18 @@ static void machvirt_init(MachineState *machine)create_platform_bus (vms, pic); +if (vms->acpi_nvdimm_state.is_enabled) { +AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state; + + acpi_nvdimm_state->dsm_io.space_id =AML_AS_SYSTEM_MEMORY ; + acpi_nvdimm_state->dsm_io.address = + vms->memmap[VIRT_ACPI_IO ].base +NVDIMM_ACPI_IO_BASE ; + acpi_nvdimm_state->dsm_io.bit_width =NVDIMM_ACPI_IO_LEN <<3 ;
I'd prefer following style (well at least that's direction I try to push towards to when dealing with ACPI)
const AcpiGenericAddress dsmio = {
.space_id = AML_AS_SYSTEM_MEMORY,
.address = vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE,
.bit_width = NVDIMM_ACPI_IO_LEN << 3
};
nvdimm_init_acpi_state(&vms->acpi_nvdimm_state, sysmem, &dsmio,
vms->fw_cfg, OBJECT(vms),);Quoted 56 lines
hw/arm/virt.c +14
-0+ +nvdimm_init_acpi_state (acpi_nvdimm_state, sysmem, + vms->fw_cfg,OBJECT (vms)); + } + vms->bootinfo.ram_size = machine->ram_size; vms->bootinfo.kernel_filename = machine->kernel_filename; vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;@@ -1860,10 +1873,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) {VirtMachineState *vms =VIRT_MACHINE (hotplug_dev); +bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM );Error *local_err =NULL ;pc_dimm_plug (PC_DIMM (dev),MACHINE (vms), &local_err); +if (local_err) { + goto out; + } +if (is_nvdimm) { +nvdimm_plug (&vms->acpi_nvdimm_state); + } + +out:error_propagate (errp, local_err); }include/hw/arm/virt.h +3
-0@@ -37,6 +37,7 @@#include "hw/arm/arm.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" +#include "hw/mem/nvdimm.h" #define NUM_GICV2M_SPIS 64#define NUM_VIRTIO_TRANSPORTS 32@@ -77,6 +78,7 @@ enum {VIRT_GPIO ,VIRT_SECURE_UART ,VIRT_SECURE_MEM , +VIRT_ACPI_IO , VIRT_LOWMEMMAP_LAST, };@@ -137,6 +139,7 @@ typedef struct {hwaddr device_memory_base;hwaddr device_memory_size;bool extended_memmap; +AcpiNVDIMMState acpi_nvdimm_state; } VirtMachineState;#define VIRT_ECAM_ID (high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
PATCH v7 16/17 hw/arm/boot: Expose the pmem nodes in the DT
In reply to Eric AugerIn case of NV-DIMM slots, let's add /pmem DT nodes.
---
v6 -> v7
- does the same rework as for fdt_add_memory_node
--- hw/arm/boot.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
hw/arm/boot.c +40 -0
@@ -450,6 +450,32 @@ out:return ret; } +static int fdt_add_pmem_node (void *fdt,uint32_t acells,hwaddr mem_base, +uint32_t scells,hwaddr mem_len, +int numa_node_id) +{ +char *nodename; +int ret; + + nodename =g_strdup_printf ("/pmem@%" PRIx64, mem_base); +qemu_fdt_add_subnode (fdt, nodename); +qemu_fdt_setprop_string (fdt, nodename,"compatible" ,"pmem-region" ); + ret =qemu_fdt_setprop_sized_cells (fdt, nodename,"reg" , acells, mem_base, + scells, mem_len); +if (ret <0 ) { + goto out; + } + +/* only set the NUMA ID if it is specified */ +if (numa_node_id >=0 ) { + ret =qemu_fdt_setprop_cell (fdt, nodename, +"numa-node-id" , numa_node_id); + } +out: +g_free (nodename); +return ret; +} +static void fdt_add_psci_node (void *fdt) {uint32_t cpu_suspend_fn;
@@ -546,6 +572,20 @@ static int fdt_add_hotpluggable_memory_nodes(void *fdt, }break ; } +case MEMORY_DEVICE_INFO_KIND_NVDIMM : + { +PCDIMMDeviceInfo *di = mi->u.nvdimm.data; + + ret =fdt_add_pmem_node (fdt, acells, di->addr, + scells, di->size, di->node); +if (ret) { +fprintf (stderr, +"couldn't add NVDIMM /memory@%" PRIx64" node\n" , + di->addr); + goto out; + } +break ; + }default :fprintf (stderr,"%s memory nodes are not yet supported\n" ,MemoryDeviceInfoKind_str (mi->type));
Signature · 1 line
2.20.1
PATCH v7 17/17 hw/arm/virt: Add nvdimm and nvdimm-persistence options
In reply to Eric AugerMachine option nvdimm allows to turn NVDIMM support on.
--- hw/arm/virt.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-)
hw/arm/virt.c +57 -2
@@ -1814,6 +1814,47 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp) } } +static bool virt_get_nvdimm (Object *obj,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); + +return vms->acpi_nvdimm_state.is_enabled; +} + +static void virt_set_nvdimm (Object *obj,bool value,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); + + vms->acpi_nvdimm_state.is_enabled = value; +} + +static char *virt_get_nvdimm_persistence (Object *obj,Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); + +return g_strdup (vms->acpi_nvdimm_state.persistence_string); +} + +static void virt_set_nvdimm_persistence (Object *obj,const char *value, +Error **errp) +{ +VirtMachineState *vms =VIRT_MACHINE (obj); +AcpiNVDIMMState *nvdimm_state = &vms->acpi_nvdimm_state; + +if (strcmp (value,"cpu" ) ==0 ) + nvdimm_state->persistence =3 ; +else if (strcmp (value,"mem-ctrl" ) ==0 ) + nvdimm_state->persistence =2 ; +else { +error_report ("-machine nvdimm-persistence=%s: unsupported option" , + value); +exit (EXIT_FAILURE ); + } + +g_free (nvdimm_state->persistence_string); + nvdimm_state->persistence_string =g_strdup (value); +} +static CpuInstanceProperties virt_cpu_index_to_props (MachineState *ms,unsigned cpu_index) {
@@ -1856,13 +1897,14 @@ static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) {const bool is_nvdimm =object_dynamic_cast (OBJECT (dev),TYPE_NVDIMM ); +VirtMachineState *vms =VIRT_MACHINE (hotplug_dev);if (dev->hotplugged) {error_setg (errp,"memory hotplug is not supported" ); } -if (is_nvdimm) { -error_setg (errp,"nvdimm is not yet supported" ); +if (is_nvdimm && !vms->acpi_nvdimm_state.is_enabled) { +error_setg (errp,"nvdimm is not enabled: missing 'nvdimm' in '-M'" );return ; }
@@ -2076,6 +2118,19 @@ static void virt_instance_init(Object *obj) vms->extended_memmap = true; } +object_property_add_bool (obj,"nvdimm" , + virt_get_nvdimm, virt_set_nvdimm,NULL ); +object_property_set_description (obj,"nvdimm" , +"Set on/off to enable/disable NVDIMM " +"instantiation" ,NULL ); + +object_property_add_str (obj,"nvdimm-persistence" , + virt_get_nvdimm_persistence, + virt_set_nvdimm_persistence,NULL ); +object_property_set_description (obj,"nvdimm-persistence" , +"Set NVDIMM persistence" +"Valid values are cpu and mem-ctrl" ,NULL ); + vms->irqmap = a15irqmap; }
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH v7 17/17] hw/arm/virt: Add nvdimm and nvdimm-persistence options
Quoted 1 line · On Wed, 20 Feb 2019 23:40:03 +0100 Eric Auger <[email protected]> wrote:
Machine option nvdimm allows to turn NVDIMM support on.
With virt-arm and spapr[1] trying to add nvdimm, I think it's time to generalize and move acpi_nvdimm_state to generic Machine instead of duplicating the same code in several machines.
1) [RFC PATCH 3/4] spapr: Add NVDIMM device support
Quoted 3 lines
Signed-off-by Eric Auger <[email protected]>---
[...]
Re: [Qemu-arm] [Qemu-devel] [PATCH v7 17/17] hw/arm/virt: Add nvdimm and nvdimm-persistence options
Hi Igor,
Quoted 8 lines · Igor Mammedov
Quoted 1 line · On Wed, 20 Feb 2019 23:40:03 +0100 Eric Auger <[email protected]> wrote:
Machine option nvdimm allows to turn NVDIMM support on.
With virt-arm and spapr[1] trying to add nvdimm, I think it's time to generalize and move
acpi_nvdimm_stateto generic Machine instead of duplicating the same code in several machines.
OK I will go this way
Thanks a lot for your time!
Best Regards
Eric
Quoted 9 lines
1) [RFC PATCH 3/4] spapr: Add NVDIMM device support
Quoted 3 lines
Signed-off-by Eric Auger <[email protected]>---
[...]
Hi Peter,
Quoted 36 lines · Eric Auger
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
Quoted 6 lines
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory map
I respinned the whole series including PCDIMM and NVDIMM parts as Igor did a first review pass on those latter. However the first objective is to get [1 - 9] upstreamed as we discussed realier. So please consider those patches independently on the others.
Thanks
Eric
Quoted 138 lines
2) Support of PC-DIMM [10 - 13]
3) Support of NV-DIMM [14 - 17]1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Quoted 36 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
Quoted 8 lines
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory map2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for ACPI_POWER_BUTTON_DEVICE to deliver event. So that it would be easier to share this with their virt-x86 machine eventually.
Quoted 1 line
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 136 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory map
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest /proc/meminfo or "lshw -class memory" reflects the amount of mem added with the DIMM slots. So it looks fine to me. Isn't E820 a pure x86 matter? What else would you expect in the dsdt? I understand hotplug would require extra modifications but I don't see anything else missing for coldplug.
Quoted 6 lines
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
Shameer sent an RFC series for supporting hotplug.
[RFC PATCH 0/4] ARM virt: ACPI memory hotplug support
https://patchwork.kernel.org/cover/10783589/
I tested PCDIMM hotplug (with ACPI) this afternoon and it seemed to be OK, even after system_reset.
Note the hotplug kernel support on ARM is very recent. I would prefer to dissociate both efforts if we want to get a chance making coldplug for 4.0. Also we have an issue for NVDIMM since on reboot the guest does not boot properly.
Quoted 6 lines
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for
ACPI_POWER_BUTTON_DEVICEto deliver event. So that it would be easier to share this with their virt-x86 machine eventually.
Sounds like a different approach than the one initiated by Shameer?
Thanks
Eric
Quoted 144 lines
Quoted 3 lines
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 136 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory map
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.
Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFI GetMemoryMap() as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFI GetMemoryMap() or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with _CRS describing memory ranges
Quoted 15 lines
I understand hotplug
would require extra modifications but I don't see anything else missing for coldplug.Quoted 6 lines
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
Shameer sent an RFC series for supporting hotplug.
[RFC PATCH 0/4] ARM virt: ACPI memory hotplug support
https://patchwork.kernel.org/cover/10783589/
Quoted 7 lines
I tested PCDIMM hotplug (with ACPI) this afternoon and it seemed to be OK, even after
system_reset.Note the hotplug kernel support on ARM is very recent. I would prefer to dissociate both efforts if we want to get a chance making coldplug for 4.0. Also we have an issue for NVDIMM since on reboot the guest does not boot properly.
I guess we can merge implemetation that works on some kernel configs [DT based I'd guess], and add ACPI part later. Though that will be a bit of a mess as we do not version firmware parts (ACPI tables).
Quoted 6 lines
Quoted 5 lines
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for
ACPI_POWER_BUTTON_DEVICEto deliver event. So that it would be easier to share this with their virt-x86 machine eventually.Sounds like a different approach than the one initiated by Shameer?
ARM boards were first to use ACPI hw-reduced profile so they picked up available back then GPIO based way to deliver hotplug event, later spec introduced Generic Event Device for that means to use with hw-reduced profile, which NEMU implemented[1], so I'd use that rather than ad-hoc GPIO mapping. I'd guess it will more compatible with various contemporary guests and we could reuse the same code for both x86/arm virt boards)
1) https://github.com/intel/nemu/blob/topic/virt-x86/hw/acpi/ged.c
Quoted 148 lines
Thanks
Eric
Quoted 144 lines
Quoted 3 lines
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 136 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Hi Igor,
Quoted 14 lines
-----Original Message-----
From: Igor Mammedov [mailto:[email protected]]
Sent: 25 February 2019 09:42
To: Auger Eric <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Qemu-devel] [PATCH v7 00/17] ARM virt: Initial RAM expansion and PCDIMM/NVDIMM supportOn Fri, 22 Feb 2019 18:35:26 +0100
Auger Eric <[email protected]> wrote:
[...]
Quoted 32 lines
Quoted 15 lines
I understand hotplug
would require extra modifications but I don't see anything else missing for coldplug.Quoted 6 lines
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
Shameer sent an RFC series for supporting hotplug.
[RFC PATCH 0/4] ARM virt: ACPI memory hotplug support
https://patchwork.kernel.org/cover/10783589/Quoted 7 lines
I tested PCDIMM hotplug (with ACPI) this afternoon and it seemed to be OK, even after
system_reset.Note the hotplug kernel support on ARM is very recent. I would prefer to dissociate both efforts if we want to get a chance making coldplug for 4.0. Also we have an issue for NVDIMM since on reboot the guest does not boot properly.
I guess we can merge implemetation that works on some kernel configs [DT based I'd guess], and add ACPI part later. Though that will be a bit of a mess as we do not version firmware parts (ACPI tables).
Quoted 6 lines
Quoted 5 lines
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for
ACPI_POWER_BUTTON_DEVICEto deliver event. So that it would be easier to share this with their virt-x86 machine eventually.Sounds like a different approach than the one initiated by Shameer?
Quoted 8 lines
ARM boards were first to use ACPI hw-reduced profile so they picked up available back then GPIO based way to deliver hotplug event, later spec introduced Generic Event Device for that means to use with hw-reduced profile, which NEMU implemented[1], so I'd use that rather than ad-hoc GPIO mapping. I'd guess it will more compatible with various contemporary guests and we could reuse the same code for both x86/arm virt boards)
1) https://github.com/intel/nemu/blob/topic/virt-x86/hw/acpi/ged.c
Thanks for this suggestion. Looks like that is definitely a better way of handling ACPI events. I will take a look and use GED for the next revision of hotplug series.
Thanks,
Shameer
Quoted 150 lines
Quoted 53 lines
Thanks
Eric
Quoted 50 lines
Quoted 3 lines
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 44 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number ofsupported
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases thereview
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines
of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
accel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364++++++++++++++++++++++++++++----
Quoted 15 lines
Quoted 15 lines
Quoted 13 lines
hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.
Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory ranges
OK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
Quoted 33 lines
Quoted 16 lines
I understand hotplug
would require extra modifications but I don't see anything else missing for coldplug.Quoted 6 lines
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
Shameer sent an RFC series for supporting hotplug.
[RFC PATCH 0/4] ARM virt: ACPI memory hotplug support
https://patchwork.kernel.org/cover/10783589/Quoted 7 lines
I tested PCDIMM hotplug (with ACPI) this afternoon and it seemed to be OK, even after
system_reset.Note the hotplug kernel support on ARM is very recent. I would prefer to dissociate both efforts if we want to get a chance making coldplug for 4.0. Also we have an issue for NVDIMM since on reboot the guest does not boot properly.
I guess we can merge implemetation that works on some kernel configs [DT based I'd guess], and add ACPI part later. Though that will be a bit of a mess as we do not version firmware parts (ACPI tables).
Quoted 6 lines
Quoted 5 lines
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for
ACPI_POWER_BUTTON_DEVICEto deliver event. So that it would be easier to share this with their virt-x86 machine eventually.Sounds like a different approach than the one initiated by Shameer?
Quoted 8 lines
ARM boards were first to use ACPI hw-reduced profile so they picked up available back then GPIO based way to deliver hotplug event, later spec introduced Generic Event Device for that means to use with hw-reduced profile, which NEMU implemented[1], so I'd use that rather than ad-hoc GPIO mapping. I'd guess it will more compatible with various contemporary guests and we could reuse the same code for both x86/arm virt boards)
1) https://github.com/intel/nemu/blob/topic/virt-x86/hw/acpi/ged.c
That's really helpful for the ARM hotplug works. Thanks!
Eric
Quoted 150 lines
Quoted 149 lines
Thanks
Eric
Quoted 144 lines
Quoted 3 lines
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 136 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Hi Igor,
Quoted 78 lines · Auger Eric
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory rangesOK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the build_memory_hotplug_aml() call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.
Thanks
Eric
Quoted 196 lines
Quoted 34 lines
Quoted 17 lines
I understand hotplug
would require extra modifications but I don't see anything else missing for coldplug.Quoted 6 lines
Even though I've tried make mem hotplug ACPI parts not x86 specific, I'm afraid it might be tightly coupled with hotplug support. So here are 2 options make DSDT part work without hotplug or implement hotplug here. I think the former is just a waste of time and we should just add hotplug. It should take relatively minor effort since you already implemented most of boiler plate here.
Shameer sent an RFC series for supporting hotplug.
[RFC PATCH 0/4] ARM virt: ACPI memory hotplug support
https://patchwork.kernel.org/cover/10783589/Quoted 7 lines
I tested PCDIMM hotplug (with ACPI) this afternoon and it seemed to be OK, even after
system_reset.Note the hotplug kernel support on ARM is very recent. I would prefer to dissociate both efforts if we want to get a chance making coldplug for 4.0. Also we have an issue for NVDIMM since on reboot the guest does not boot properly.
I guess we can merge implemetation that works on some kernel configs [DT based I'd guess], and add ACPI part later. Though that will be a bit of a mess as we do not version firmware parts (ACPI tables).
Quoted 6 lines
Quoted 5 lines
As for how to implement ACPI HW part, I suggest to borrow GED device that NEMU guys trying to use instead of GPIO route, like we do now for
ACPI_POWER_BUTTON_DEVICEto deliver event. So that it would be easier to share this with their virt-x86 machine eventually.Sounds like a different approach than the one initiated by Shameer?
Quoted 8 lines
ARM boards were first to use ACPI hw-reduced profile so they picked up available back then GPIO based way to deliver hotplug event, later spec introduced Generic Event Device for that means to use with hw-reduced profile, which NEMU implemented[1], so I'd use that rather than ad-hoc GPIO mapping. I'd guess it will more compatible with various contemporary guests and we could reuse the same code for both x86/arm virt boards)
1) https://github.com/intel/nemu/blob/topic/virt-x86/hw/acpi/ged.c
That's really helpful for the ARM hotplug works. Thanks!
Eric
Quoted 150 lines
Quoted 149 lines
Thanks
Eric
Quoted 144 lines
Quoted 3 lines
3) Support of NV-DIMM [14 - 17]
The same might be true for NUMA but I haven't dug this deep in to that part.
Quoted 136 lines
1) can be upstreamed before 2 and 2 can be upstreamed before 3.
Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.
Best Regards
Eric
References:
[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.htmlThis series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v7History:
v6 -> v7:
- Addressed Peter and Igor comments (exceptions sent my email)
- Fixed TCG case. Now device memory works also for TCG and vcpu pamax is checked
- See individual logs for more detailsv5 -> v6:
- mingw compilation issue fix
-kvm_arm_get_max_vm_phys_shiftalways returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization intomachvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their sizev4 -> v5:
- change in the memory map
- see individual logsv3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm:pre_plug"slot" and "addr" assignment"
- kvm-type option not used anymore. We directly usemaxram_sizeandram_sizemachine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
-device_memory_startanddevice_memory_sizenot stored anymore invms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring ofarm_load_dtband the one related to the generation of the dtb aftersystem_resetwhich would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machineram_size,maxram_sizeandram_slotsearlier"v2 -> v3:
- fixpc_q35andpc_piixcompilation error
- kwangwoo's email being not valid anymore, remove his addressv1 -> v2:
-kvm_get_max_vm_phys_shiftmoved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoringv1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"Best Regards
Eric
Eric Auger (12):
hw/arm/virt: Rename highmem IO regions
hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter tokvm_typecallback
kvm: addkvm_arm_get_max_vm_ipa_size
vl: Set machineram_size,maxram_sizeandram_slotsearlier hw/arm/virt: Dynamic memory map depending on RAM requirements hw/arm/virt: Implementkvm_typefunction for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocatedevice_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence optionsKwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructureShameer Kolothum (3):
hw/arm/boot: introducefdt_add_memory_nodehelper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRATaccel/kvm/kvm-all.c | 2 +- default-configs/arm-softmmu.mak | 4 + hw/acpi/nvdimm.c | 31 ++- hw/arm/boot.c | 136 ++++++++++-- hw/arm/virt-acpi-build.c | 23 +- hw/arm/virt.c | 364 ++++++++++++++++++++++++++++---- hw/i386/pc_piix.c | 6 +- hw/i386/pc_q35.c | 6 +- hw/ppc/mac_newworld.c | 3 +- hw/ppc/mac_oldworld.c | 2 +- hw/ppc/spapr.c | 2 +- include/hw/arm/virt.h | 24 ++- include/hw/boards.h | 5 +- include/hw/mem/nvdimm.h | 4 + target/arm/kvm.c | 10 + target/arm/kvm_arm.h | 13 ++ vl.c | 6 +- 17 files changed, 556 insertions(+), 85 deletions(-)
Quoted 90 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 78 lines · Auger Eric
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory rangesOK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
Quoted 5 lines
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
Hi Igor,
Quoted 93 lines · Igor Mammedov
Quoted 90 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 86 lines · Auger Eric
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory rangesOK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
Quoted 8 lines
Quoted 5 lines
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest /proc/meminfo anymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.
Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
Thanks
Eric
Quoted 4 lines
Quoted 102 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 99 lines · Igor Mammedov
Quoted 95 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 78 lines · Auger Eric
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory rangesOK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.
Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI.
Only we don't have a switch for it something like, -machine fdt=on (with default off)
Quoted 8 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.
Quoted 7 lines
Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Quoted 2 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Hi Igor, Shameer,
Quoted 126 lines · Igor Mammedov
Quoted 102 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 99 lines · Igor Mammedov
Quoted 95 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 78 lines · Auger Eric
Hi Igor,
Quoted 75 lines · Igor Mammedov
Quoted 49 lines · On Fri, 22 Feb 2019 18:35:26 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 46 lines · Igor Mammedov
Quoted 42 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.
In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on
ram_sizeandmaxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 8 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount of mem added with the DIMM slots.Question is how does it get there? Does it come from DT or from firmware via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap().
On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it as normal memory early at boot and later put that memory into zone normal and hence make it non-hot-un-pluggable. The same concerns apply to DT based means of discovery.
(That's guest issue but it's easy to workaround it not putting hotpluggable memory into UEFIGetMemoryMap()or DT and let DSDT describe it properly) That way memory doesn't get (ab)used by firmware or early boot kernel stages and doesn't get locked up.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSdescribing memory rangesOK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI.
Only we don't have a switch for it something like, -machine fdt=on (with default off)Quoted 13 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
Quoted 5 lines
Quoted 2 lines
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Shameer, would you agree if we merge PATCH 1 of your RFC hotplug series (without the reduced hw_reduced_acpi flag) in this series and isolate in a second PATCH the acpi_memory_hotplug_init() + build_memory_hotplug_aml called in virt code?
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Hi Eric,
Quoted 165 lines
-----Original Message-----
From: Auger Eric [mailto:[email protected]]
Sent: 27 February 2019 10:27
To: Igor Mammedov <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Qemu-devel] [PATCH v7 00/17] ARM virt: Initial RAM expansion and PCDIMM/NVDIMM supportHi Igor, Shameer,
Quoted 18 lines · Igor Mammedov
Quoted 15 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 12 lines · Igor Mammedov
Quoted 9 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Auger Eric
Hi Igor,
Quoted 3 lines · Igor Mammedov
On Fri, 22 Feb 2019 18:35:26 +0100
Auger Eric <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Igor,
Quoted 5 lines · Igor Mammedov
Quoted 2 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially
PCDIMM/NVDIMM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
additional GICv3 RDIST region, high PCIe ECAM region and high
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMIO region. The address map was 1TB large. This corresponded
to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB.
The
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
max GPA size depends on the host kernel configuration and physical
CPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this series we use this feature and allow the RAM to grow
without
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentiallyhotpluggable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved
after
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the RAM. Their offset is dynamically computed, depends on
ram_sizeQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
maxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory
map
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to
do
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount ofmem
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
added with the DIMM slots.
Question is how does it get there? Does it come from DT or from
firmware
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap(). On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it asnormal
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory early at boot and later put that memory into zone normal and
hence
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make it non-hot-un-pluggable. The same concerns apply to DT based
means
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
of discovery.
(That's guest issue but it's easy to workaround it not puttinghotpluggable
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory into UEFI
GetMemoryMap()or DT and let DSDT describe itproperly)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
That way memory doesn't get (ab)used by firmware or early boot
kernel stages
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and doesn't get locked up.
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSQuoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines
Quoted 1 line
describing memory ranges
OK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI. Only we don't have a switch for it something like, -machine fdt=on (with
default off)
Quoted 16 lines
Quoted 16 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
Quoted 8 lines
Quoted 3 lines
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Shameer, would you agree if we merge PATCH 1 of your RFC hotplug series (without the reduced
hw_reduced_acpiflag) in this series and isolate in a second PATCH theacpi_memory_hotplug_init()+build_memory_hotplug_amlcalled in virt code?
Sure, that’s fine with me. So what would you use for the event_handler_method in build_memory_hotplug_aml()? GPO0 device?
Thanks,
Shameer
Quoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Quoted 175 lines · On Wed, 27 Feb 2019 10:41:45 +0000 Shameerali Kolothum Thodi…
Hi Eric,
Quoted 173 lines
-----Original Message-----
From: Auger Eric [mailto:[email protected]]
Sent: 27 February 2019 10:27
To: Igor Mammedov <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Qemu-devel] [PATCH v7 00/17] ARM virt: Initial RAM expansion and PCDIMM/NVDIMM supportHi Igor, Shameer,
Quoted 18 lines · Igor Mammedov
Quoted 15 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 12 lines · Igor Mammedov
Quoted 9 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Auger Eric
Hi Igor,
Quoted 3 lines · Igor Mammedov
On Fri, 22 Feb 2019 18:35:26 +0100
Auger Eric <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Igor,
Quoted 5 lines · Igor Mammedov
Quoted 2 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially
PCDIMM/NVDIMM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
additional GICv3 RDIST region, high PCIe ECAM region and high
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMIO region. The address map was 1TB large. This corresponded
to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB.
The
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
max GPA size depends on the host kernel configuration and physical
CPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this series we use this feature and allow the RAM to grow
without
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentiallyhotpluggable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved
after
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the RAM. Their offset is dynamically computed, depends on
ram_sizeQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
maxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory
map
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to
do
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount ofmem
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
added with the DIMM slots.
Question is how does it get there? Does it come from DT or from
firmware
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap(). On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it asnormal
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory early at boot and later put that memory into zone normal and
hence
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make it non-hot-un-pluggable. The same concerns apply to DT based
means
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
of discovery.
(That's guest issue but it's easy to workaround it not puttinghotpluggable
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory into UEFI
GetMemoryMap()or DT and let DSDT describe itproperly)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
That way memory doesn't get (ab)used by firmware or early boot
kernel stages
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and doesn't get locked up.
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSQuoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines
Quoted 1 line
describing memory ranges
OK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI. Only we don't have a switch for it something like, -machine fdt=on (with
default off)
Quoted 14 lines
Quoted 14 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
Quoted 5 lines
Quoted 2 lines
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Shameer, would you agree if we merge PATCH 1 of your RFC hotplug series (without the reduced
hw_reduced_acpiflag) in this series and isolate in a second PATCH theacpi_memory_hotplug_init()+build_memory_hotplug_amlcalled in virt code?
probably we can do it as transitional step as we need working mmio interface in place for build_memory_hotplug_aml() to work, provided it won't create migration issues (do we need VMSTATE_MEMORY_HOTPLUG for cold-plug case?).
What about dummy initial GED (empty device), that manages mmio region only and then later it will be filled with remaining logic IRQ. In this case mmio region and vmstate won't change (maybe) so it won't cause ABI or migration issues.
Quoted 2 lines
Sure, that’s fine with me. So what would you use for the
event_handler_methodinbuild_memory_hotplug_aml()? GPO0 device?
a method name not defined in spec, so it won't be called might do.
Quoted 37 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Hi Igor, Shameer,
Quoted 189 lines · Igor Mammedov
Quoted 175 lines · On Wed, 27 Feb 2019 10:41:45 +0000 Shameerali Kolothum Thodi…
Hi Eric,
Quoted 173 lines
-----Original Message-----
From: Auger Eric [mailto:[email protected]]
Sent: 27 February 2019 10:27
To: Igor Mammedov <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Qemu-devel] [PATCH v7 00/17] ARM virt: Initial RAM expansion and PCDIMM/NVDIMM supportHi Igor, Shameer,
Quoted 18 lines · Igor Mammedov
Quoted 15 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 12 lines · Igor Mammedov
Quoted 9 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Auger Eric
Hi Igor,
Quoted 3 lines · Igor Mammedov
On Fri, 22 Feb 2019 18:35:26 +0100
Auger Eric <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Igor,
Quoted 5 lines · Igor Mammedov
Quoted 2 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially
PCDIMM/NVDIMM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
additional GICv3 RDIST region, high PCIe ECAM region and high
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMIO region. The address map was 1TB large. This corresponded
to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB.
The
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
max GPA size depends on the host kernel configuration and physical
CPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this series we use this feature and allow the RAM to grow
without
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_sizeand then comes the device memory (,maxmem) of sizemaxram_size-ram_size. The device memory is potentiallyhotpluggable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved
after
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the RAM. Their offset is dynamically computed, depends on
ram_sizeQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and
maxram_size. Size alignment is enforced.In case maxmem value is inferior to 255GB, the legacy memory
map
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still is used. The change of memory map becomes effective from 4.0 onwards.
As we keep the initial RAM at 1GB base address, we do not need to
do
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.
Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.
This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.
Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in the memory mapQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount ofmem
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
added with the DIMM slots.
Question is how does it get there? Does it come from DT or from
firmware
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap(). On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it asnormal
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory early at boot and later put that memory into zone normal and
hence
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make it non-hot-un-pluggable. The same concerns apply to DT based
means
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
of discovery.
(That's guest issue but it's easy to workaround it not puttinghotpluggable
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory into UEFI
GetMemoryMap()or DT and let DSDT describe itproperly)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
That way memory doesn't get (ab)used by firmware or early boot
kernel stages
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and doesn't get locked up.
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSQuoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines
Quoted 1 line
describing memory ranges
OK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI. Only we don't have a switch for it something like, -machine fdt=on (with
default off)
Quoted 14 lines
Quoted 14 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
Quoted 5 lines
Quoted 2 lines
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Shameer, would you agree if we merge PATCH 1 of your RFC hotplug series (without the reduced
hw_reduced_acpiflag) in this series and isolate in a second PATCH theacpi_memory_hotplug_init()+build_memory_hotplug_amlcalled in virt code?probably we can do it as transitional step as we need working mmio interface in place for
build_memory_hotplug_aml()to work, provided it won't create migration issues (do we needVMSTATE_MEMORY_HOTPLUGfor cold-plug case?).What about dummy initial GED (empty device), that manages mmio region only and then later it will be filled with remaining logic IRQ. In this case mmio region and vmstate won't change (maybe) so it won't cause ABI or migration issues.
Quoted 4 lines
Quoted 2 lines
Sure, that’s fine with me. So what would you use for the
event_handler_methodinbuild_memory_hotplug_aml()? GPO0 device?a method name not defined in spec, so it won't be called might do.
At this point the event_handler_method, ie. \_SB.GPO0._E02, is not supposed to be called, right? So effectivily we should be able to use any other method name (unlinked to any GPIO/GED). I guess at this stage only the PNP0C80 definition blocks + methods are used.
What still remains fuzzy for me is in case of cold plug the mmio hotplug control region part only is read (despite the slot selection of course) and returns 0 for addr/size and also flags meaning the slot is not enabled. So despite the slots are advertised as hotpluggable/enabled in the SRAT; I am not sure for the OS it actually makes any difference whether the DSDT definition blocks are described or not.
To be honest I am afraid this is too late to add those additional features for 4.0 now. This is going to jeopardize the first preliminary part which is the introduction of the new memory map, allowing the expansion of the initial RAM and paving the way for device memory introduction. So I think I am going to resend the first 10 patches in a standalone series. And we can iterate on the PCDIMM/NVDIMM parts independently.
Thanks
Eric
Quoted 41 lines
Quoted 39 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Quoted 195 lines · On Thu, 28 Feb 2019 08:48:18 +0100 Auger Eric <[email protected]> wrote:
Hi Igor, Shameer,
Quoted 192 lines · Igor Mammedov
Quoted 175 lines · On Wed, 27 Feb 2019 10:41:45 +0000 Shameerali Kolothum Thodi…
Hi Eric,
Quoted 173 lines
-----Original Message-----
From: Auger Eric [mailto:[email protected]]
Sent: 27 February 2019 10:27
To: Igor Mammedov <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [Qemu-devel] [PATCH v7 00/17] ARM virt: Initial RAM expansion and PCDIMM/NVDIMM supportHi Igor, Shameer,
Quoted 18 lines · Igor Mammedov
Quoted 15 lines · On Tue, 26 Feb 2019 18:53:24 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 12 lines · Igor Mammedov
Quoted 9 lines · On Tue, 26 Feb 2019 14:11:58 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
Quoted 6 lines · Auger Eric
Hi Igor,
Quoted 3 lines · Igor Mammedov
On Fri, 22 Feb 2019 18:35:26 +0100
Auger Eric <[email protected]> wrote:Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Hi Igor,
Quoted 5 lines · Igor Mammedov
Quoted 2 lines · On Wed, 20 Feb 2019 23:39:46 +0100 Eric Auger <[email protected]> wrote:
This series aims to bump the 255GB RAM limit in machvirt and to
support device memory in general, and especiallyPCDIMM/NVDIMM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
In machvirt versions < 4.0, the initial RAM starts at 1GB and can
grow up to 255GB. From 256GB onwards we find IO regions such asthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
additional GICv3 RDIST region, high PCIe ECAM region and high
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMIO region. The address map was 1TB large. This corresponded
to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the max IPA capacity KVM was able to manage.
Since 4.20, the host kernel is able to support a larger and dynamic
IPA range. So the guest physical address can go beyond the 1TB.The
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
max GPA size depends on the host kernel configuration and physical
CPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
In this series we use this feature and allow the RAM to grow
without
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
any other limit than the one put by the host kernel.
The RAM still starts at 1GB. First comes the initial ram (-m) of size
ram_size and then comes the device memory (,maxmem) of size
maxram_size - ram_size. The device memory is potentiallyhotpluggable
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
depending on the instantiated memory objects.
IO regions previously located between 256GB and 1TB are moved
after
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the RAM. Their offset is dynamically computed, depends on
ram_sizeQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
and maxram_size. Size alignment is enforced.
In case maxmem value is inferior to 255GB, the legacy memory
map
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
still is used. The change of memory map becomes effective from 4.0
onwards.As we keep the initial RAM at 1GB base address, we do not need to
do
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
invasive changes in the EDK2 FW. It seems nobody is eager to do
that job at the moment.Device memory being put just after the initial RAM, it is possible
to get access to this feature while keeping a 1TB address map.This series reuses/rebases patches initially submitted by Shameer
in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 9] and change in
the memory mapQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
2) Support of PC-DIMM [10 - 13]
Is this part complete ACPI wise (for coldplug)? I haven't noticed DSDT AML here no E820 changes, so ACPI wise pc-dimm shouldn't be visible to the guest. It might be that DT is masking problem but well, that won't work on ACPI only guests.
guest
/proc/meminfoor "lshw -class memory" reflects the amount ofmem
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
added with the DIMM slots.
Question is how does it get there? Does it come from DT or from
firmware
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
via UEFI interfaces?
Quoted 1 line
So it looks fine to me. Isn't E820 a pure x86 matter?
sorry for misleading, I've meant is UEFI
GetMemoryMap(). On x86, I'm wary of adding PC-DIMMs to E802 which then gets exposed via UEFIGetMemoryMap()as guest kernel might start using it asnormal
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory early at boot and later put that memory into zone normal and
hence
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make it non-hot-un-pluggable. The same concerns apply to DT based
means
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
of discovery.
(That's guest issue but it's easy to workaround it not puttinghotpluggable
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
memory into UEFI
GetMemoryMap()or DT and let DSDT describe itproperly)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
That way memory doesn't get (ab)used by firmware or early boot
kernel stages
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
and doesn't get locked up.
Quoted 1 line
What else would you expect in the dsdt?
Memory device descriptions, look for code that adds PNP0C80 with
_CRSQuoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 6 lines
Quoted 1 line
describing memory ranges
OK thank you for the explanations. I will work on PNP0C80 addition then. Does it mean that in ACPI mode we must not output DT hotplug memory nodes or assuming that PNP0C80 is properly described, it will "override" DT description?
After further investigations, I think the pieces you pointed out are added by Shameer's series, ie. through the
build_memory_hotplug_aml()call. So I suggest we separate the concerns: this series brings support for DIMM coldplug. hotplug, including all the relevant ACPI structures will be added later on by Shameer.Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Maybe we should not put pc-dimms in DT for this series until it gets clear if it doesn't conflict with ACPI in some way.
I guess you mean removing the DT hotpluggable memory nodes only in ACPI mode? Otherwise you simply remove the DIMM feature, right?
Something like this so DT won't get in conflict with ACPI. Only we don't have a switch for it something like, -machine fdt=on (with
default off)
Quoted 14 lines
Quoted 14 lines
I double checked and if you remove the hotpluggable memory DT nodes in ACPI mode:
- you do not see the PCDIMM slots in guest/proc/meminfoanymore. So I guess you're right, if the DT nodes are available, that memory is considered as not unpluggable by the guest.
- You can see the NVDIMM slots using ndctl list -u. You can mount a DAX system.Hotplug/unplug is clearly not supported by this series and any attempt results in "memory hotplug is not supported". Is it really an issue if the guest does not consider DIMM slots as not hot-unpluggable memory? I am not even sure the guest kernel would support to unplug that memory.
Quoted 5 lines
Quoted 2 lines
In case we want all ACPI tables to be ready for making this memory seen as hot-unpluggable we need some Shameer's patches on top of this series.
May be we should push for this way (into 4.0), it's just a several patches after all or even merge them in your series (I'd guess it would need to be rebased on top of your latest work)
Shameer, would you agree if we merge PATCH 1 of your RFC hotplug series (without the reduced
hw_reduced_acpiflag) in this series and isolate in a second PATCH theacpi_memory_hotplug_init()+build_memory_hotplug_amlcalled in virt code?probably we can do it as transitional step as we need working mmio interface in place for
build_memory_hotplug_aml()to work, provided it won't create migration issues (do we needVMSTATE_MEMORY_HOTPLUGfor cold-plug case?).What about dummy initial GED (empty device), that manages mmio region only and then later it will be filled with remaining logic IRQ. In this case mmio region and vmstate won't change (maybe) so it won't cause ABI or migration issues.
Quoted 2 lines
Sure, that’s fine with me. So what would you use for the
event_handler_methodinbuild_memory_hotplug_aml()? GPO0 device?
Quoted 6 lines
Quoted 1 line
a method name not defined in spec, so it won't be called might do.
At this point the
event_handler_method, ie. \_SB.GPO0._E02, is not supposed to be called, right? So effectivily we should be able to use any other method name (unlinked to any GPIO/GED). I guess at this stage only the PNP0C80 definition blocks + methods are used.
pretty much yes.
Quoted 4 lines
What still remains fuzzy for me is in case of cold plug the mmio hotplug control region part only is read (despite the slot selection of course) and returns 0 for addr/size and also flags meaning the slot is not enabled.
If you mean guest reads 0s than it looks broken, could you show trace log with mhp_* tracepoints enabled during a dimm hotplug.
Quoted 3 lines
So despite the slots are advertised as hotpluggable/enabled in the SRAT; I am not sure for the OS it actually makes any difference whether the DSDT definition blocks are described or not.
SRAT isn't used fro informing guests about amount of present RAM, it holds affinity information for present and possible RAM
Quoted 7 lines
To be honest I am afraid this is too late to add those additional features for 4.0 now. This is going to jeopardize the first preliminary part which is the introduction of the new memory map, allowing the expansion of the initial RAM and paving the way for device memory introduction. So I think I am going to resend the first 10 patches in a standalone series. And we can iterate on the PCDIMM/NVDIMM parts independently.
sounds good to me, I'll try to review 1-10 today
Quoted 44 lines
Thanks
Eric
Quoted 41 lines
Quoted 39 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Hi Igor,
[..]
Quoted 7 lines
Quoted 5 lines
What still remains fuzzy for me is in case of cold plug the mmio hotplug control region part only is read (despite the slot selection of course) and returns 0 for addr/size and also flags meaning the slot is not enabled.
If you mean guest reads 0s than it looks broken, could you show trace log with mhp_* tracepoints enabled during a dimm hotplug.
Please find the traces + cmd line on x86
/qemu-system-x86_64 -M
q35,usb=off,dump-guest-core=off,kernel_irqchip=split,nvdimm -cpu
Haswell,-hle,-rtm -smp 4,sockets=4,cores=1,threads=1 -m
16G,maxmem=32G,slots=4 -display none --enable-kvm -serial
tcp:localhost:4444,server -trace
events=/home/augere/UPSTREAM/qemu2/nvdimm.txt -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -realtime
mlock=off -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -machine kernel_irqchip=split
-object memory-backend-file,id=mem3,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-3,size=2G,align=128M -device nvdimm,memdev=mem3,id=dimm3,label-size=2M -object memory-backend-file,id=mem4,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-4,size=2G,align=128M -device nvdimm,memdev=mem4,id=dimm4,label-size=2M -device virtio-blk-pci,bus=pcie.0,scsi=off,drive=drv0,id=virtio-disk0,bootindex=1,werror=stop,rerror=stop -drive file=/home/augere/VM/IMAGES/x86_64-vm1-f28.raw,format=raw,if=none,cache=writethrough,id=drv0 -device virtio-net-pci,bus=pcie.0,netdev=nic0,mac=6a:f5:10:b1:3d:d2
-netdev tap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown,vhost=on -net none -d guest_errors
******************************************************************
ioctl(TUNSETIFF): Device or resource busy
qemu-system-x86_64: -serial tcp:localhost:4444,server: info: QEMU waiting for connection on: disconnected:tcp:::1:4444,server
qemu-system-x86_64: warning: global PIIX4_PM.disable_s3=1 not used
qemu-system-x86_64: warning: global PIIX4_PM.disable_s4=1 not used [email protected]:mhp_acpi_write_slot set active slot: 0x0 [email protected]:mhp_acpi_read_addr_hi slot[0x0] addr hi: 0x0 [email protected]:mhp_acpi_read_addr_lo slot[0x0] addr lo: 0x0 [email protected]:mhp_acpi_read_size_hi slot[0x0] size hi: 0x0 [email protected]:mhp_acpi_read_size_lo slot[0x0] size lo: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x0 [email protected]:mhp_acpi_read_flags slot[0x0] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x1 [email protected]:mhp_acpi_read_addr_hi slot[0x1] addr hi: 0x0 [email protected]:mhp_acpi_read_addr_lo slot[0x1] addr lo: 0x0 [email protected]:mhp_acpi_read_size_hi slot[0x1] size hi: 0x0 [email protected]:mhp_acpi_read_size_lo slot[0x1] size lo: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x1 [email protected]:mhp_acpi_read_flags slot[0x1] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x2 [email protected]:mhp_acpi_read_addr_hi slot[0x2] addr hi: 0x0 [email protected]:mhp_acpi_read_addr_lo slot[0x2] addr lo: 0x0 [email protected]:mhp_acpi_read_size_hi slot[0x2] size hi: 0x0 [email protected]:mhp_acpi_read_size_lo slot[0x2] size lo: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x2 [email protected]:mhp_acpi_read_flags slot[0x2] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x3 [email protected]:mhp_acpi_read_addr_hi slot[0x3] addr hi: 0x0 [email protected]:mhp_acpi_read_addr_lo slot[0x3] addr lo: 0x0 [email protected]:mhp_acpi_read_size_hi slot[0x3] size hi: 0x0 [email protected]:mhp_acpi_read_size_lo slot[0x3] size lo: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x3 [email protected]:mhp_acpi_read_flags slot[0x3] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x0 [email protected]:mhp_acpi_read_flags slot[0x0] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x1 [email protected]:mhp_acpi_read_flags slot[0x1] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x2 [email protected]:mhp_acpi_read_flags slot[0x2] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x3 [email protected]:mhp_acpi_read_flags slot[0x3] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x0 [email protected]:mhp_acpi_read_flags slot[0x0] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x1 [email protected]:mhp_acpi_read_flags slot[0x1] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x2 [email protected]:mhp_acpi_read_flags slot[0x2] flags: 0x0 [email protected]:mhp_acpi_write_slot set active slot: 0x3 [email protected]:mhp_acpi_read_flags slot[0x3] flags: 0x0
That's the only traces I get until I get the login prompt.
Thanks
Eric
Quoted 61 lines
Quoted 4 lines
So despite the slots are advertised as hotpluggable/enabled in the SRAT; I am not sure for the OS it actually makes any difference whether the DSDT definition blocks are described or not.
SRAT isn't used fro informing guests about amount of present RAM, it holds affinity information for present and possible RAM
Quoted 7 lines
To be honest I am afraid this is too late to add those additional features for 4.0 now. This is going to jeopardize the first preliminary part which is the introduction of the new memory map, allowing the expansion of the initial RAM and paving the way for device memory introduction. So I think I am going to resend the first 10 patches in a standalone series. And we can iterate on the PCDIMM/NVDIMM parts independently.
sounds good to me, I'll try to review 1-10 today
Quoted 44 lines
Thanks
Eric
Quoted 41 lines
Quoted 39 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Quoted 5 lines · On Fri, 1 Mar 2019 15:18:14 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
[..]
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
What still remains fuzzy for me is in case of cold plug the mmio hotplug control region part only is read (despite the slot selection of course) and returns 0 for addr/size and also flags meaning the slot is not enabled.
If you mean guest reads 0s than it looks broken, could you show trace log with mhp_* tracepoints enabled during a dimm hotplug.
Please find the traces + cmd line on x86
I've thought that you where talking about pc-dimms, so here it goes:
nvdimm is not part of memory hotplug interface, they have their own mmio region through which they let guest read completely rebuilt NFIT table and their own GPE._E04 event handler.
you see 0's in trace because guest enumerates all PNP0C80 in DSDT to check if for any present pc-dimm for which mem hotplug interface reports 0s since there is none.
PS:
In ACPI spec there is an example of NVDIMMs where they also have associated memory device (PNP0C80) and that it's somehow related to nvdimm hotplug, but it's not described in sufficient detail so I'm honestly do not know what to do with it. Hence QEMU doesn't have PNP0C80 counterpart for nvdimm. To me it looks more like a mistake in the spec, but that's a topic for another discussion.
Quoted 142 lines
/qemu-system-x86_64 -M q35,usb=off,dump-guest-core=off,kernel_irqchip=split,nvdimm -cpu Haswell,-hle,-rtm -smp 4,sockets=4,cores=1,threads=1 -m
16G,maxmem=32G,slots=4 -display none --enable-kvm -serial tcp:localhost:4444,server -trace
events=/home/augere/UPSTREAM/qemu2/nvdimm.txt -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -realtime
mlock=off -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -machine kernel_irqchip=split
-object memory-backend-file,id=mem3,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-3,size=2G,align=128M -device nvdimm,memdev=mem3,id=dimm3,label-size=2M -object memory-backend-file,id=mem4,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-4,size=2G,align=128M -device nvdimm,memdev=mem4,id=dimm4,label-size=2M -device virtio-blk-pci,bus=pcie.0,scsi=off,drive=drv0,id=virtio-disk0,bootindex=1,werror=stop,rerror=stop -drive file=/home/augere/VM/IMAGES/x86_64-vm1-f28.raw,format=raw,if=none,cache=writethrough,id=drv0(quoted diff) +0
-2-device virtio-net-pci,bus=pcie.0,netdev=nic0,mac=6a:f5:10:b1:3d:d2 -netdevtap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown,vhost=on -net none -d
guest_errors******************************************************************
ioctl(TUNSETIFF): Device or resource busy
qemu-system-x86_64: -serial tcp:localhost:4444,server: info: QEMU waiting for connection on: disconnected:tcp:::1:4444,server
qemu-system-x86_64: warning: global PIIX4_PM.disable_s3=1 not used
qemu-system-x86_64: warning: global PIIX4_PM.disable_s4=1 not used [email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_addr_hi slot[0x0] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x0] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x0] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x0] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_addr_hi slot[0x1] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x1] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x1] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x1] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_addr_hi slot[0x2] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x2] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x2] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x2] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_addr_hi slot[0x3] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x3] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x3] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x3] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0That's the only traces I get until I get the login prompt.
Thanks
Eric
Quoted 61 lines
Quoted 4 lines
So despite the slots are advertised as hotpluggable/enabled in the SRAT; I am not sure for the OS it actually makes any difference whether the DSDT definition blocks are described or not.
SRAT isn't used fro informing guests about amount of present RAM, it holds affinity information for present and possible RAM
Quoted 7 lines
To be honest I am afraid this is too late to add those additional features for 4.0 now. This is going to jeopardize the first preliminary part which is the introduction of the new memory map, allowing the expansion of the initial RAM and paving the way for device memory introduction. So I think I am going to resend the first 10 patches in a standalone series. And we can iterate on the PCDIMM/NVDIMM parts independently.
sounds good to me, I'll try to review 1-10 today
Quoted 44 lines
Thanks
Eric
Quoted 41 lines
Quoted 39 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines
Hi Igor,
Quoted 26 lines · Igor Mammedov
Quoted 5 lines · On Fri, 1 Mar 2019 15:18:14 +0100 Auger Eric <[email protected]> wrote:
Hi Igor,
[..]
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
What still remains fuzzy for me is in case of cold plug the mmio hotplug control region part only is read (despite the slot selection of course) and returns 0 for addr/size and also flags meaning the slot is not enabled.
If you mean guest reads 0s than it looks broken, could you show trace log with mhp_* tracepoints enabled during a dimm hotplug.
Please find the traces + cmd line on x86
I've thought that you where talking about pc-dimms, so here it goes:
nvdimm is not part of memory hotplug interface, they have their own mmio region through which they let guest read completely rebuilt NFIT table and their own GPE._E04 event handler.
you see 0's in trace because guest enumerates all PNP0C80 in DSDT to check if for any present pc-dimm for which mem hotplug interface reports 0s since there is none.
Quoted 7 lines
PS:
In ACPI spec there is an example of NVDIMMs where they also have associated memory device (PNP0C80) and that it's somehow related to nvdimm hotplug, but it's not described in sufficient detail so I'm honestly do not know what to do with it. Hence QEMU doesn't have PNP0C80 counterpart for nvdimm. To me it looks more like a mistake in the spec, but that's a topic for another discussion.
Oh my bad. Indeed with PCDIMM I can see the actual addresses being read.
Thank you for the explanation!
Have a nice WE.
Eric
Quoted 146 lines
Quoted 144 lines
/qemu-system-x86_64 -M q35,usb=off,dump-guest-core=off,kernel_irqchip=split,nvdimm -cpu Haswell,-hle,-rtm -smp 4,sockets=4,cores=1,threads=1 -m
16G,maxmem=32G,slots=4 -display none --enable-kvm -serial tcp:localhost:4444,server -trace
events=/home/augere/UPSTREAM/qemu2/nvdimm.txt -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -realtime
mlock=off -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -machine kernel_irqchip=split
-object memory-backend-file,id=mem3,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-3,size=2G,align=128M -device nvdimm,memdev=mem3,id=dimm3,label-size=2M -object memory-backend-file,id=mem4,share,mem-path=/home/augere/TEST/QEMU/nv-dimm-4,size=2G,align=128M -device nvdimm,memdev=mem4,id=dimm4,label-size=2M -device virtio-blk-pci,bus=pcie.0,scsi=off,drive=drv0,id=virtio-disk0,bootindex=1,werror=stop,rerror=stop -drive file=/home/augere/VM/IMAGES/x86_64-vm1-f28.raw,format=raw,if=none,cache=writethrough,id=drv0(quoted diff) +0
-2-device virtio-net-pci,bus=pcie.0,netdev=nic0,mac=6a:f5:10:b1:3d:d2 -netdevtap,id=nic0,script=/home/augere/TEST/SCRIPTS/qemu-ifup,downscript=/home/augere/TEST/SCRIPTS/qemu-ifdown,vhost=on -net none -d
guest_errors******************************************************************
ioctl(TUNSETIFF): Device or resource busy
qemu-system-x86_64: -serial tcp:localhost:4444,server: info: QEMU waiting for connection on: disconnected:tcp:::1:4444,server
qemu-system-x86_64: warning: global PIIX4_PM.disable_s3=1 not used
qemu-system-x86_64: warning: global PIIX4_PM.disable_s4=1 not used [email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_addr_hi slot[0x0] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x0] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x0] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x0] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_addr_hi slot[0x1] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x1] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x1] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x1] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_addr_hi slot[0x2] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x2] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x2] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x2] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_addr_hi slot[0x3] addr hi:0x0[email protected]:mhp_acpi_read_addr_lo slot[0x3] addr lo:0x0[email protected]:mhp_acpi_read_size_hi slot[0x3] size hi:0x0[email protected]:mhp_acpi_read_size_lo slot[0x3] size lo:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x0[email protected]:mhp_acpi_read_flags slot[0x0] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x1[email protected]:mhp_acpi_read_flags slot[0x1] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x2[email protected]:mhp_acpi_read_flags slot[0x2] flags:0x0[email protected]:mhp_acpi_write_slot set active slot:0x3[email protected]:mhp_acpi_read_flags slot[0x3] flags:0x0That's the only traces I get until I get the login prompt.
Thanks
Eric
Quoted 61 lines
Quoted 4 lines
So despite the slots are advertised as hotpluggable/enabled in the SRAT; I am not sure for the OS it actually makes any difference whether the DSDT definition blocks are described or not.
SRAT isn't used fro informing guests about amount of present RAM, it holds affinity information for present and possible RAM
Quoted 7 lines
To be honest I am afraid this is too late to add those additional features for 4.0 now. This is going to jeopardize the first preliminary part which is the introduction of the new memory map, allowing the expansion of the initial RAM and paving the way for device memory introduction. So I think I am going to resend the first 10 patches in a standalone series. And we can iterate on the PCDIMM/NVDIMM parts independently.
sounds good to me, I'll try to review 1-10 today
Quoted 44 lines
Thanks
Eric
Quoted 41 lines
Quoted 39 lines
Thanks,
ShameerQuoted 33 lines
Then would remain the GED/GPIO actual integration.
Thanks
Eric
Quoted 28 lines
Quoted 3 lines
Also don't DIMM slots already make sense in DT mode. Usually we accept to add one feature in DT and then in ACPI. For instance we can benefit
usually it doesn't conflict with each other (at least I'm not aware of it) but I see a problem with in this case.
Quoted 2 lines
from nvdimm in dt mode right? So, considering an incremental approach I would be in favour of keeping the DT nodes.
I'd guess it is the same as for DIMMs, ACPI support for NVDIMMs is much more versatile.
I consider target application of arm/virt as a board that's used to run in production generic ACPI capable guest in most use cases and various DT only guests as secondary ones. It's hard to make both usecases be happy with defaults (that's probably one of the reasons why 'sbsa' board is being added).
So I'd give priority to ACPI based arm/virt versus DT when defaults are considered.
Quoted 7 lines
Thanks
Eric
Quoted 4 lines