Skip to contentSkip to conversation

ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support

PATCH Superseded Eric Auger 6 participants 56 messages Feb 5, 2019 – Feb 21, 2019

v6 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v6 cover ARM virt: Initial RAM expansion and PCDIMM/NVDIMM support

This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.

In machvirt versions < 4.0, the initial RAM starts at 1GB and can grow up to 255GB. From 256GB onwards we find IO regions such as the additional GICv3 RDIST region, high PCIe ECAM region and high PCIe MMIO region. The address map was 1TB large. This corresponded to the max IPA capacity KVM was able to manage.

Since 4.20, the host kernel is able to support a larger and dynamic IPA range. So the guest physical address can go beyond the 1TB. The max GPA size depends on the host kernel configuration and physical CPUs.

In this series we use this feature and allow the RAM to grow without any other limit than the one put by the host.

The RAM still starts at 1GB. First comes the initial ram (-m) of size ram_size and then comes the device memory (,maxmem) of size maxram_size - ram_size. The device memory is potentially hotpluggable depending on the instantiated memory objects.

IO regions previously located between 256GB and 1TB are moved after the RAM. Their offset is dynamically computed, depends on ram_size and maxram_size. Size alignment is enforced.

In case maxmem value is inferior to 255GB, the legacy memory map still is used. The change of memory map becomes effective from 4.0 onwards.

As we keep the initial RAM at 1GB base address, we do not need to do invasive changes in the EDK2 FW. It seems nobody is eager to do that job at the moment.

Device memory being put just after the initial RAM, it is possible to get access to this feature while keeping a 1TB address map.

This series reuses/rebases patches initially submitted by Shameer in [1] and Kwangwoo in [2] for the PC-DIMM and NV-DIMM parts.

Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 10] and change in the memory map
2) Support of PC-DIMM [11 - 14]
3) Support of NV-DIMM [15 - 18]

1) can be upstreamed before 2 and 2 can be upstreamed before 3.

Work is ongoing to transform the whole memory as device memory. However this move is not trivial and to me, is independent on the improvements brought by this series:
- if we were to use DIMM for initial RAM, those DIMMs would use use slots. Although they would not be part of the ones provided using the ",slots" options, they are ACPI limited resources.
- DT and ACPI description needs to be reworked
- NUMA integration needs special care
- a special device memory object may be required to avoid consuming slots and easing the FW description.

So I preferred to separate the concerns. This new implementation based on device memory could be candidate for another virt version.

Best Regards

Eric

References:

[0] [RFC v2 0/6] hw/arm: Add support for non-contiguous iova regions
http://patchwork.ozlabs.org/cover/914694/

[1] [RFC PATCH 0/3] add nvdimm support on AArch64 virt platform
https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04599.html

This series can be found at:
https://github.com/eauger/qemu/tree/v3.1.0-dimm-v6

Tests:
- On HPE system, allocated a VM with 300GB initial RAM and 220 vcpus
- tested on kernel versions not supporting variable IPA range
- tested with different virt versions

History:

v5 -> v6:
- mingw compilation issue fix
- kvm_arm_get_max_vm_phys_shift always returns the number of supported IPA bits
- new patch "hw/arm/virt: Rename highmem IO regions" that eases the review of "hw/arm/virt: Split the memory map description"
- "hw/arm/virt: Move memory map initialization into machvirt_init" squashed into the previous patch
- change alignment of IO regions beyond the RAM so that it matches their size

v4 -> v5:
- change in the memory map
- see individual logs

v3 -> v4:
- rebase on David's "pc-dimm: next bunch of cleanups" and "pc-dimm: pre_plug "slot" and "addr" assignment"
- kvm-type option not used anymore. We directly use maxram_size and ram_size machine fields to compute the MAX IPA range. Migration is naturally handled as CLI option are kept between source and destination. This was suggested by David.
- device_memory_start and device_memory_size not stored anymore in vms->bootinfo
- I did not take into account 2 Igor's comments: the one related to the refactoring of arm_load_dtb and the one related to the generation of the dtb after system_reset which would contain nodes of hotplugged devices (we do not support hotplug at this stage)
- check the end-user does not attempt to hotplug a device
- addition of "vl: Set machine ram_size, maxram_size and ram_slots earlier"

v2 -> v3:
- fix pc_q35 and pc_piix compilation error
- kwangwoo's email being not valid anymore, remove his address

v1 -> v2:
- kvm_get_max_vm_phys_shift moved in arch specific file
- addition of NVDIMM part
- single series
- rebase on David's refactoring

v1:
- was "[RFC 0/6] KVM/ARM: Dynamic and larger GPA size"
- was "[RFC 0/5] ARM virt: Support PC-DIMM at 2TB"

Best Regards

Eric

Alexey Kardashevskiy (1):
update-linux-headers.sh: Copy new headers

Eric Auger (12):
linux-headers: Update to v5.0-rc2
hw/arm/virt: Rename highmem IO regions hw/arm/virt: Split the memory map description hw/boards: Add a MachineState parameter to kvm_type callback
kvm: add kvm_arm_get_max_vm_phys_shift
vl: Set machine ram_size, maxram_size and ram_slots earlier hw/arm/virt: Implement kvm_type function for 4.0 machine hw/arm/virt: Bump the 255GB initial RAM limit hw/arm/virt: Add memory hotplug framework hw/arm/virt: Allocate device_memory
hw/arm/boot: Expose the pmem nodes in the DT hw/arm/virt: Add nvdimm and nvdimm-persistence options

Kwangwoo Lee (2):
nvdimm: use configurable ACPI IO base and size
hw/arm/virt: Add nvdimm hot-plug infrastructure

Shameer Kolothum (3):
hw/arm/boot: introduce fdt_add_memory_node helper hw/arm/boot: Expose the PC-DIMM nodes in the DT hw/arm/virt-acpi-build: Add PC-DIMM in SRAT

 accel/kvm/kvm-all.c                           |    2 +-
 default-configs/arm-softmmu.mak               |    4 +
 hw/acpi/nvdimm.c                              |   28 +-
 hw/arm/boot.c                                 |  120 +-
 hw/arm/virt-acpi-build.c                      |   23 +-
 hw/arm/virt.c                                 |  311 ++++-
 hw/i386/pc_piix.c                             |    8 +-
 hw/i386/pc_q35.c                              |    8 +-
 hw/ppc/mac_newworld.c                         |    3 +-
 hw/ppc/mac_oldworld.c                         |    2 +-
 hw/ppc/spapr.c                                |    2 +-
 include/hw/arm/virt.h                         |   21 +-
 include/hw/boards.h                           |    2 +-
 include/hw/mem/nvdimm.h                       |   12 +
 include/standard-headers/drm/drm_fourcc.h     |   63 +
 include/standard-headers/linux/ethtool.h      |   19 +-
 .../linux/input-event-codes.h                 |   19 +
 include/standard-headers/linux/pci_regs.h     |    1 +
 .../standard-headers/linux/virtio_balloon.h   |    8 +
 include/standard-headers/linux/virtio_blk.h   |   54 +
 .../standard-headers/linux/virtio_config.h    |    3 +
 include/standard-headers/linux/virtio_gpu.h   |   18 +
 include/standard-headers/linux/virtio_ring.h  |   52 +
 .../standard-headers/rdma/vmw_pvrdma-abi.h    |    1 +
 linux-headers/asm-arm/unistd-common.h         |    1 +
 linux-headers/asm-arm64/unistd.h              |    1 +
 linux-headers/asm-generic/unistd.h            |   10 +-
 linux-headers/asm-mips/sgidefs.h              |    8 -
 linux-headers/asm-mips/unistd.h               | 1074 +----------------
 linux-headers/asm-mips/unistd_n32.h           |  338 ++++++
 linux-headers/asm-mips/unistd_n64.h           |  334 +++++
 linux-headers/asm-mips/unistd_o32.h           |  374 ++++++
 linux-headers/asm-powerpc/unistd.h            |  389 +-----
 linux-headers/asm-powerpc/unistd_32.h         |  381 ++++++
 linux-headers/asm-powerpc/unistd_64.h         |  372 ++++++
 linux-headers/linux/kvm.h                     |   29 +
 linux-headers/linux/vfio.h                    |   92 ++
 linux-headers/linux/vhost.h                   |  113 +-
 linux-headers/linux/vhost_types.h             |  128 ++
 scripts/update-linux-headers.sh               |   11 +-
 target/arm/kvm.c                              |   10 +
 target/arm/kvm_arm.h                          |   13 +
 vl.c                                          |    6 +-
 43 files changed, 2797 insertions(+), 1671 deletions(-)
 create mode 100644 linux-headers/asm-mips/unistd_n32.h
 create mode 100644 linux-headers/asm-mips/unistd_n64.h
 create mode 100644 linux-headers/asm-mips/unistd_o32.h
 create mode 100644 linux-headers/asm-powerpc/unistd_32.h
 create mode 100644 linux-headers/asm-powerpc/unistd_64.h
 create mode 100644 linux-headers/linux/vhost_types.h
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v6 1/18 update-linux-headers.sh: Copy new headers

From: Alexey Kardashevskiy <[email protected]>

Since Linux'es ab66dcc76d "powerpc: generate uapi header and system call table files" there are 2 new files: unistd_32.h and unistd_64.h. These files content is moved from unistd.h so now we have to copy new files as well, just like we already do for other architectures; this does it for MIPS as well.

Also, v5.0-rc2 moved vhost bits around in 4b86713236e4bd "vhost: split structs into a separate header file", add those too.

---
 scripts/update-linux-headers.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
scripts/update-linux-headers.sh +10 -1
index 0a964fe240..1cd8bd57be 100755
@@ -121,11 +121,20 @@ for arch in $ARCHLIST; do
         cp "$tmpdir/include/asm/unistd_64.h" "$output/linux-headers/asm-x86/"
         cp_portable "$tmpdir/include/asm/kvm_para.h" "$output/include/standard-headers/asm-$arch"
     fi
+    if [ $arch = powerpc ]; then
+        cp "$tmpdir/include/asm/unistd_32.h" "$output/linux-headers/asm-powerpc/"
+        cp "$tmpdir/include/asm/unistd_64.h" "$output/linux-headers/asm-powerpc/"
+    fi
+    if [ $arch = mips ]; then
+        cp "$tmpdir/include/asm/unistd_o32.h" "$output/linux-headers/asm-mips/"
+        cp "$tmpdir/include/asm/unistd_n32.h" "$output/linux-headers/asm-mips/"
+        cp "$tmpdir/include/asm/unistd_n64.h" "$output/linux-headers/asm-mips/"
+    fi
 done
 
 rm -rf "$output/linux-headers/linux"
 mkdir -p "$output/linux-headers/linux"
-for header in kvm.h vfio.h vfio_ccw.h vhost.h \
+for header in kvm.h vfio.h vfio_ccw.h vhost.h vhost_types.h \
               psci.h psp-sev.h userfaultfd.h; do
     cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux"
 done
Signature · 1 line
2.20.1
#2 lore raw

Re: [Qemu-devel] [PATCH v6 01/18] update-linux-headers.sh: Copy new headers

Quoted 9 lines · Eric Auger

From: Alexey Kardashevskiy <[email protected]>

Since Linux'es ab66dcc76d "powerpc: generate uapi header and system call table files" there are 2 new files: unistd_32.h and unistd_64.h. These files content is moved from unistd.h so now we have to copy new files as well, just like we already do for other architectures; this does it for MIPS as well.

Quoted 4 lines

Also, v5.0-rc2 moved vhost bits around in 4b86713236e4bd "vhost: split structs into a separate header file", add those too.

I think this fix is handled by commit a0a6ef91a4a4edde27 (now in master), yes ?

thanks
-- PMM

#3 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 01/18] update-linux-headers.sh: Copy new headers

Quoted 10 lines · Peter Maydell
Quoted 9 lines · Eric Auger

From: Alexey Kardashevskiy <[email protected]>

Since Linux'es ab66dcc76d "powerpc: generate uapi header and system call table files" there are 2 new files: unistd_32.h and unistd_64.h. These files content is moved from unistd.h so now we have to copy new files as well, just like we already do for other architectures; this does it for MIPS as well.

Quoted 7 lines
Quoted 4 lines

Also, v5.0-rc2 moved vhost bits around in 4b86713236e4bd "vhost: split structs into a separate header file", add those too.

I think this fix is handled by commit a0a6ef91a4a4edde27 (now in master), yes ?

uff, just noticed this mail. yes, it is done by a0a6ef91a4a4edde27. Thanks,

Signature · 1 line
Alexey
author
#4 lore raw

PATCH v6 2/18 linux-headers: Update to v5.0-rc2

In reply to Eric Auger

Here is the update script update and what it did to the v5.0-rc2 kernel.

This is based on sha1: d7393226d15add056285c8fc86723d54d7e0c77d

---
 include/standard-headers/drm/drm_fourcc.h     |   63 +
 include/standard-headers/linux/ethtool.h      |   19 +-
 .../linux/input-event-codes.h                 |   19 +
 include/standard-headers/linux/pci_regs.h     |    1 +
 .../standard-headers/linux/virtio_balloon.h   |    8 +
 include/standard-headers/linux/virtio_blk.h   |   54 +
 .../standard-headers/linux/virtio_config.h    |    3 +
 include/standard-headers/linux/virtio_gpu.h   |   18 +
 include/standard-headers/linux/virtio_ring.h  |   52 +
 .../standard-headers/rdma/vmw_pvrdma-abi.h    |    1 +
 linux-headers/asm-arm/unistd-common.h         |    1 +
 linux-headers/asm-arm64/unistd.h              |    1 +
 linux-headers/asm-generic/unistd.h            |   10 +-
 linux-headers/asm-mips/sgidefs.h              |    8 -
 linux-headers/asm-mips/unistd.h               | 1074 +----------------
 linux-headers/asm-mips/unistd_n32.h           |  338 ++++++
 linux-headers/asm-mips/unistd_n64.h           |  334 +++++
 linux-headers/asm-mips/unistd_o32.h           |  374 ++++++
 linux-headers/asm-powerpc/unistd.h            |  389 +-----
 linux-headers/asm-powerpc/unistd_32.h         |  381 ++++++
 linux-headers/asm-powerpc/unistd_64.h         |  372 ++++++
 linux-headers/linux/kvm.h                     |   29 +
 linux-headers/linux/vfio.h                    |   92 ++
 linux-headers/linux/vhost.h                   |  113 +-
 linux-headers/linux/vhost_types.h             |  128 ++
 25 files changed, 2293 insertions(+), 1589 deletions(-)
 create mode 100644 linux-headers/asm-mips/unistd_n32.h
 create mode 100644 linux-headers/asm-mips/unistd_n64.h
 create mode 100644 linux-headers/asm-mips/unistd_o32.h
 create mode 100644 linux-headers/asm-powerpc/unistd_32.h
 create mode 100644 linux-headers/asm-powerpc/unistd_64.h
 create mode 100644 linux-headers/linux/vhost_types.h
include/standard-headers/drm/drm_fourcc.h +63 -0
index b53f8d7c8c..44490607f9 100644
@@ -29,11 +29,50 @@
 extern "C" {
 #endif
 
+/**
+ * DOC: overview
+ *
+ * In the DRM subsystem, framebuffer pixel formats are described using the
+ * fourcc codes defined in `include/uapi/drm/drm_fourcc.h`. In addition to the
+ * fourcc code, a Format Modifier may optionally be provided, in order to
+ * further describe the buffer's format - for example tiling or compression.
+ *
+ * Format Modifiers
+ * ----------------
+ *
+ * Format modifiers are used in conjunction with a fourcc code, forming a
+ * unique fourcc:modifier pair. This format:modifier pair must fully define the
+ * format and data layout of the buffer, and should be the only way to describe
+ * that particular buffer.
+ *
+ * Having multiple fourcc:modifier pairs which describe the same layout should
+ * be avoided, as such aliases run the risk of different drivers exposing
+ * different names for the same data format, forcing userspace to understand
+ * that they are aliases.
+ *
+ * Format modifiers may change any property of the buffer, including the number
+ * of planes and/or the required allocation size. Format modifiers are
+ * vendor-namespaced, and as such the relationship between a fourcc code and a
+ * modifier is specific to the modifer being used. For example, some modifiers
+ * may preserve meaning - such as number of planes - from the fourcc code,
+ * whereas others may not.
+ *
+ * Vendors should document their modifier usage in as much detail as
+ * possible, to ensure maximum compatibility across devices, drivers and
+ * applications.
+ *
+ * The authoritative list of format modifier codes is found in
+ * `include/uapi/drm/drm_fourcc.h`
+ */
+
 #define fourcc_code(a, b, c, d) ((uint32_t)(a) | ((uint32_t)(b) << 8) | \
 				 ((uint32_t)(c) << 16) | ((uint32_t)(d) << 24))
 
 #define DRM_FORMAT_BIG_ENDIAN (1<<31) /* format is big endian instead of little endian */
 
+/* Reserve 0 for the invalid format specifier */
+#define DRM_FORMAT_INVALID	0
+
 /* color index */
 #define DRM_FORMAT_C8		fourcc_code('C', '8', ' ', ' ') /* [7:0] C */
 
@@ -111,6 +150,21 @@ extern "C" {
 #define DRM_FORMAT_VYUY		fourcc_code('V', 'Y', 'U', 'Y') /* [31:0] Y1:Cb0:Y0:Cr0 8:8:8:8 little endian */
 
 #define DRM_FORMAT_AYUV		fourcc_code('A', 'Y', 'U', 'V') /* [31:0] A:Y:Cb:Cr 8:8:8:8 little endian */
+#define DRM_FORMAT_XYUV8888		fourcc_code('X', 'Y', 'U', 'V') /* [31:0] X:Y:Cb:Cr 8:8:8:8 little endian */
+
+/*
+ * packed YCbCr420 2x2 tiled formats
+ * first 64 bits will contain Y,Cb,Cr components for a 2x2 tile
+ */
+/* [63:0]   A3:A2:Y3:0:Cr0:0:Y2:0:A1:A0:Y1:0:Cb0:0:Y0:0  1:1:8:2:8:2:8:2:1:1:8:2:8:2:8:2 little endian */
+#define DRM_FORMAT_Y0L0		fourcc_code('Y', '0', 'L', '0')
+/* [63:0]   X3:X2:Y3:0:Cr0:0:Y2:0:X1:X0:Y1:0:Cb0:0:Y0:0  1:1:8:2:8:2:8:2:1:1:8:2:8:2:8:2 little endian */
+#define DRM_FORMAT_X0L0		fourcc_code('X', '0', 'L', '0')
+
+/* [63:0]   A3:A2:Y3:Cr0:Y2:A1:A0:Y1:Cb0:Y0  1:1:10:10:10:1:1:10:10:10 little endian */
+#define DRM_FORMAT_Y0L2		fourcc_code('Y', '0', 'L', '2')
+/* [63:0]   X3:X2:Y3:Cr0:Y2:X1:X0:Y1:Cb0:Y0  1:1:10:10:10:1:1:10:10:10 little endian */
+#define DRM_FORMAT_X0L2		fourcc_code('X', '0', 'L', '2')
 
 /*
  * 2 plane RGB + A
@@ -298,6 +352,15 @@ extern "C" {
  */
 #define DRM_FORMAT_MOD_SAMSUNG_64_32_TILE	fourcc_mod_code(SAMSUNG, 1)
 
+/*
+ * Tiled, 16 (pixels) x 16 (lines) - sized macroblocks
+ *
+ * This is a simple tiled layout using tiles of 16x16 pixels in a row-major
+ * layout. For YCbCr formats Cb/Cr components are taken in such a way that
+ * they correspond to their 16x16 luma block.
+ */
+#define DRM_FORMAT_MOD_SAMSUNG_16_16_TILE	fourcc_mod_code(SAMSUNG, 2)
+
 /*
  * Qualcomm Compressed Format
  *
include/standard-headers/linux/ethtool.h +5 -14
index 57ffcb5341..063c814278 100644
@@ -91,10 +91,6 @@
  * %ETHTOOL_GSET to get the current values before making specific
  * changes and then applying them with %ETHTOOL_SSET.
  *
- * Drivers that implement set_settings() should validate all fields
- * other than @cmd that are not described as read-only or deprecated,
- * and must ignore all fields described as read-only.
- *
  * Deprecated fields should be ignored by both users and drivers.
  */
 struct ethtool_cmd {
@@ -886,7 +882,7 @@ struct ethtool_rx_flow_spec {
 	uint32_t		location;
 };
 
-/* How rings are layed out when accessing virtual functions or
+/* How rings are laid out when accessing virtual functions or
  * offloaded queues is device specific. To allow users to do flow
  * steering and specify these queues the ring cookie is partitioned
  * into a 32bit queue index with an 8 bit virtual function id.
@@ -895,7 +891,7 @@ struct ethtool_rx_flow_spec {
  * devices start supporting PCIe w/ARI. However at the moment I
  * do not know of any devices that support this so I do not reserve
  * space for this at this time. If a future patch consumes the next
- * byte it should be aware of this possiblity.
+ * byte it should be aware of this possibility.
  */
 #define ETHTOOL_RX_FLOW_SPEC_RING	0x00000000FFFFFFFFLL
 #define ETHTOOL_RX_FLOW_SPEC_RING_VF	0x000000FF00000000LL
@@ -1800,14 +1796,9 @@ enum ethtool_reset_flags {
  * rejected.
  *
  * Deprecated %ethtool_cmd fields transceiver, maxtxpkt and maxrxpkt
- * are not available in %ethtool_link_settings. Until all drivers are
- * converted to ignore them or to the new %ethtool_link_settings API,
- * for both queries and changes, users should always try
- * %ETHTOOL_GLINKSETTINGS first, and if it fails with -ENOTSUPP stick
- * only to %ETHTOOL_GSET and %ETHTOOL_SSET consistently. If it
- * succeeds, then users should stick to %ETHTOOL_GLINKSETTINGS and
- * %ETHTOOL_SLINKSETTINGS (which would support drivers implementing
- * either %ethtool_cmd or %ethtool_link_settings).
+ * are not available in %ethtool_link_settings. These fields will be
+ * always set to zero in %ETHTOOL_GSET reply and %ETHTOOL_SSET will
+ * fail if any of them is set to non-zero value.
  *
  * Users should assume that all fields not marked read-only are
  * writable and subject to validation by the driver.  They should use
include/standard-headers/linux/input-event-codes.h +19 -0
index 9e6a8ba4ce..ff2e1ebcc9 100644
@@ -708,6 +708,16 @@
 #define REL_DIAL		0x07
 #define REL_WHEEL		0x08
 #define REL_MISC		0x09
+/*
+ * 0x0a is reserved and should not be used in input drivers.
+ * It was used by HID as REL_MISC+1 and userspace needs to detect if
+ * the next REL_* event is correct or is just REL_MISC + n.
+ * We define here REL_RESERVED so userspace can rely on it and detect
+ * the situation described above.
+ */
+#define REL_RESERVED		0x0a
+#define REL_WHEEL_HI_RES	0x0b
+#define REL_HWHEEL_HI_RES	0x0c
 #define REL_MAX			0x0f
 #define REL_CNT			(REL_MAX+1)
 
@@ -744,6 +754,15 @@
 
 #define ABS_MISC		0x28
 
+/*
+ * 0x2e is reserved and should not be used in input drivers.
+ * It was used by HID as ABS_MISC+6 and userspace needs to detect if
+ * the next ABS_* event is correct or is just ABS_MISC + n.
+ * We define here ABS_RESERVED so userspace can rely on it and detect
+ * the situation described above.
+ */
+#define ABS_RESERVED		0x2e
+
 #define ABS_MT_SLOT		0x2f	/* MT slot being modified */
 #define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
 #define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis (omit if circular) */
include/standard-headers/linux/pci_regs.h +1 -0
index ee556ccc93..e1e9888c85 100644
@@ -52,6 +52,7 @@
 #define  PCI_COMMAND_INTX_DISABLE 0x400 /* INTx Emulation Disable */
 
 #define PCI_STATUS		0x06	/* 16 bits */
+#define  PCI_STATUS_IMM_READY	0x01	/* Immediate Readiness */
 #define  PCI_STATUS_INTERRUPT	0x08	/* Interrupt status */
 #define  PCI_STATUS_CAP_LIST	0x10	/* Support Capability List */
 #define  PCI_STATUS_66MHZ	0x20	/* Support 66 MHz PCI 2.1 bus */
include/standard-headers/linux/virtio_balloon.h +8 -0
index 4dbb7dc6c0..9375ca2a70 100644
@@ -34,15 +34,23 @@
 #define VIRTIO_BALLOON_F_MUST_TELL_HOST	0 /* Tell before reclaiming pages */
 #define VIRTIO_BALLOON_F_STATS_VQ	1 /* Memory Stats virtqueue */
 #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM	2 /* Deflate balloon on OOM */
+#define VIRTIO_BALLOON_F_FREE_PAGE_HINT	3 /* VQ to report free pages */
+#define VIRTIO_BALLOON_F_PAGE_POISON	4 /* Guest is using page poisoning */
 
 /* Size of a PFN in the balloon interface. */
 #define VIRTIO_BALLOON_PFN_SHIFT 12
 
+#define VIRTIO_BALLOON_CMD_ID_STOP	0
+#define VIRTIO_BALLOON_CMD_ID_DONE	1
 struct virtio_balloon_config {
 	/* Number of pages host wants Guest to give up. */
 	uint32_t num_pages;
 	/* Number of pages we've actually got in balloon. */
 	uint32_t actual;
+	/* Free page report command id, readonly by guest */
+	uint32_t free_page_report_cmd_id;
+	/* Stores PAGE_POISON if page poisoning is in use */
+	uint32_t poison_val;
 };
 
 #define VIRTIO_BALLOON_S_SWAP_IN  0   /* Amount of memory swapped in */
include/standard-headers/linux/virtio_blk.h +54 -0

78 lines not on this page · raw message

include/standard-headers/linux/virtio_config.h +3 -0
index 0b194365a0..24e30af5ec 100644
@@ -75,6 +75,9 @@
  */
 #define VIRTIO_F_IOMMU_PLATFORM		33
 
+/* This feature indicates support for the packed virtqueue layout. */
+#define VIRTIO_F_RING_PACKED		34
+
 /*
  * Does the device support Single Root I/O Virtualization?
  */
include/standard-headers/linux/virtio_gpu.h +18 -0
index 52a830dcf8..27bb5111f9 100644
@@ -41,6 +41,7 @@
 #include "standard-headers/linux/types.h"
 
 #define VIRTIO_GPU_F_VIRGL 0
+#define VIRTIO_GPU_F_EDID  1
 
 enum virtio_gpu_ctrl_type {
 	VIRTIO_GPU_UNDEFINED = 0,
@@ -56,6 +57,7 @@ enum virtio_gpu_ctrl_type {
 	VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING,
 	VIRTIO_GPU_CMD_GET_CAPSET_INFO,
 	VIRTIO_GPU_CMD_GET_CAPSET,
+	VIRTIO_GPU_CMD_GET_EDID,
 
 	/* 3d commands */
 	VIRTIO_GPU_CMD_CTX_CREATE = 0x0200,
@@ -76,6 +78,7 @@ enum virtio_gpu_ctrl_type {
 	VIRTIO_GPU_RESP_OK_DISPLAY_INFO,
 	VIRTIO_GPU_RESP_OK_CAPSET_INFO,
 	VIRTIO_GPU_RESP_OK_CAPSET,
+	VIRTIO_GPU_RESP_OK_EDID,
 
 	/* error responses */
 	VIRTIO_GPU_RESP_ERR_UNSPEC = 0x1200,
@@ -291,6 +294,21 @@ struct virtio_gpu_resp_capset {
 	uint8_t capset_data[];
 };
 
+/* VIRTIO_GPU_CMD_GET_EDID */
+struct virtio_gpu_cmd_get_edid {
+	struct virtio_gpu_ctrl_hdr hdr;
+	uint32_t scanout;
+	uint32_t padding;
+};
+
+/* VIRTIO_GPU_RESP_OK_EDID */
+struct virtio_gpu_resp_edid {
+	struct virtio_gpu_ctrl_hdr hdr;
+	uint32_t size;
+	uint32_t padding;
+	uint8_t edid[1024];
+};
+
 #define VIRTIO_GPU_EVENT_DISPLAY (1 << 0)
 
 struct virtio_gpu_config {
include/standard-headers/linux/virtio_ring.h +52 -0

68 lines not on this page · raw message

include/standard-headers/rdma/vmw_pvrdma-abi.h +1 -0
index 6c2bc46116..336a8d596f 100644
@@ -78,6 +78,7 @@ enum pvrdma_wr_opcode {
 	PVRDMA_WR_MASKED_ATOMIC_FETCH_AND_ADD,
 	PVRDMA_WR_BIND_MW,
 	PVRDMA_WR_REG_SIG_MR,
+	PVRDMA_WR_ERROR,
 };
 
 enum pvrdma_wc_status {
linux-headers/asm-arm/unistd-common.h +1 -0
index 60c2d931d0..8c84bcf10f 100644
@@ -355,5 +355,6 @@
 #define __NR_pkey_free (__NR_SYSCALL_BASE + 396)
 #define __NR_statx (__NR_SYSCALL_BASE + 397)
 #define __NR_rseq (__NR_SYSCALL_BASE + 398)
+#define __NR_io_pgetevents (__NR_SYSCALL_BASE + 399)
 
 #endif /* _ASM_ARM_UNISTD_COMMON_H */
linux-headers/asm-arm64/unistd.h +1 -0
index 5072cbd15c..dae1584cf0 100644
@@ -16,5 +16,6 @@
  */
 
 #define __ARCH_WANT_RENAMEAT
+#define __ARCH_WANT_NEW_STAT
 
 #include <asm-generic/unistd.h>
linux-headers/asm-generic/unistd.h +9 -1
index df4bedb9b0..d90127298f 100644
@@ -242,10 +242,12 @@ __SYSCALL(__NR_tee, sys_tee)
 /* fs/stat.c */
 #define __NR_readlinkat 78
 __SYSCALL(__NR_readlinkat, sys_readlinkat)
+#if defined(__ARCH_WANT_NEW_STAT) || defined(__ARCH_WANT_STAT64)
 #define __NR3264_fstatat 79
 __SC_3264(__NR3264_fstatat, sys_fstatat64, sys_newfstatat)
 #define __NR3264_fstat 80
 __SC_3264(__NR3264_fstat, sys_fstat64, sys_newfstat)
+#endif
 
 /* fs/sync.c */
 #define __NR_sync 81
@@ -736,9 +738,11 @@ __SYSCALL(__NR_statx,     sys_statx)
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
 #define __NR_rseq 293
 __SYSCALL(__NR_rseq, sys_rseq)
+#define __NR_kexec_file_load 294
+__SYSCALL(__NR_kexec_file_load,     sys_kexec_file_load)
 
 #undef __NR_syscalls
-#define __NR_syscalls 294
+#define __NR_syscalls 295
 
 /*
  * 32 bit systems traditionally used different
@@ -758,8 +762,10 @@ __SYSCALL(__NR_rseq, sys_rseq)
 #define __NR_ftruncate __NR3264_ftruncate
 #define __NR_lseek __NR3264_lseek
 #define __NR_sendfile __NR3264_sendfile
+#if defined(__ARCH_WANT_NEW_STAT) || defined(__ARCH_WANT_STAT64)
 #define __NR_newfstatat __NR3264_fstatat
 #define __NR_fstat __NR3264_fstat
+#endif
 #define __NR_mmap __NR3264_mmap
 #define __NR_fadvise64 __NR3264_fadvise64
 #ifdef __NR3264_stat
@@ -774,8 +780,10 @@ __SYSCALL(__NR_rseq, sys_rseq)
 #define __NR_ftruncate64 __NR3264_ftruncate
 #define __NR_llseek __NR3264_lseek
 #define __NR_sendfile64 __NR3264_sendfile
+#if defined(__ARCH_WANT_NEW_STAT) || defined(__ARCH_WANT_STAT64)
 #define __NR_fstatat64 __NR3264_fstatat
 #define __NR_fstat64 __NR3264_fstat
+#endif
 #define __NR_mmap2 __NR3264_mmap
 #define __NR_fadvise64_64 __NR3264_fadvise64
 #ifdef __NR3264_stat
linux-headers/asm-mips/sgidefs.h +0 -8
index 26143e3b7c..69c3de90c5 100644
@@ -11,14 +11,6 @@
 #ifndef __ASM_SGIDEFS_H
 #define __ASM_SGIDEFS_H
 
-/*
- * Using a Linux compiler for building Linux seems logic but not to
- * everybody.
- */
-#ifndef __linux__
-#error Use a Linux compiler or give up.
-#endif
-
 /*
  * Definitions for the ISA levels
  *
linux-headers/asm-mips/unistd.h +6 -1068

1,091 lines not on this page · raw message

linux-headers/asm-mips/unistd_n32.h new file +338 -0

338 lines not on this page · raw message

linux-headers/asm-mips/unistd_n64.h new file +334 -0

334 lines not on this page · raw message

linux-headers/asm-mips/unistd_o32.h new file +374 -0

374 lines not on this page · raw message

linux-headers/asm-powerpc/unistd.h +2 -387

397 lines not on this page · raw message

linux-headers/asm-powerpc/unistd_32.h new file +381 -0

2 lines not on this page · raw message

linux-headers/asm-powerpc/unistd_64.h new file +372 -0

372 lines not on this page · raw message

linux-headers/linux/kvm.h +29 -0
index f11a7eb49c..b53ee59748 100644

Lines of this file are not shown. View the raw message.

linux-headers/linux/vfio.h +92 -0

92 lines not on this page · raw message

linux-headers/linux/vhost.h +2 -111

113 lines not on this page · raw message

linux-headers/linux/vhost_types.h new file +128 -0

128 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

author
#5 lore raw

PATCH v6 3/18 hw/arm/boot: introduce fdt_add_memory_node helper

In reply to Eric Auger

From: Shameer Kolothum <[email protected]>

We introduce an helper to create a memory node.

---
 hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +34 -20
index 05762d0fc1..2ef367e15b 100644
@@ -423,6 +423,36 @@ static void set_kernel_args_old(const struct arm_boot_info *info,
     }
 }
 
+static int fdt_add_memory_node(void *fdt, uint32_t acells, hwaddr mem_base,
+                               uint32_t scells, hwaddr mem_len,
+                               int numa_node_id)
+{
+    char *nodename = NULL;
+    int ret;
+
+    nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
+    ret = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, mem_base,
+                                       scells, mem_len);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/reg\n", nodename);
+        goto out;
+    }
+    if (numa_node_id < 0) {
+        goto out;
+    }
+
+    ret = qemu_fdt_setprop_cell(fdt, nodename, "numa-node-id", numa_node_id);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/numa-node-id\n", nodename);
+    }
+
+out:
+    g_free(nodename);
+    return ret;
+}
+
 static void fdt_add_psci_node(void *fdt)
 {
     uint32_t cpu_suspend_fn;
@@ -502,7 +532,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
     void *fdt = NULL;
     int size, rc, n = 0;
     uint32_t acells, scells;
-    char *nodename;
     unsigned int i;
     hwaddr mem_base, mem_len;
     char **node_path;
@@ -576,35 +605,20 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         mem_base = binfo->loader_start;
         for (i = 0; i < nb_numa_nodes; i++) {
             mem_len = numa_info[i].node_mem;
-            nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
-            qemu_fdt_add_subnode(fdt, nodename);
-            qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
-            rc = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg",
-                                              acells, mem_base,
-                                              scells, mem_len);
+            rc = fdt_add_memory_node(fdt, acells, mem_base,
+                                     scells, mem_len, i);
             if (rc < 0) {
-                fprintf(stderr, "couldn't set %s/reg for node %d\n", nodename,
-                        i);
                 goto fail;
             }
 
-            qemu_fdt_setprop_cell(fdt, nodename, "numa-node-id", i);
             mem_base += mem_len;
-            g_free(nodename);
         }
     } else {
-        nodename = g_strdup_printf("/memory@%" PRIx64, binfo->loader_start);
-        qemu_fdt_add_subnode(fdt, nodename);
-        qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
-
-        rc = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg",
-                                          acells, binfo->loader_start,
-                                          scells, binfo->ram_size);
+        rc = fdt_add_memory_node(fdt, acells, binfo->loader_start,
+                                 scells, binfo->ram_size, -1);
         if (rc < 0) {
-            fprintf(stderr, "couldn't set %s reg\n", nodename);
             goto fail;
         }
-        g_free(nodename);
     }
 
     rc = fdt_path_offset(fdt, "/chosen");
Signature · 1 line
2.20.1
#6 lore raw

Re: [Qemu-arm] [PATCH v6 03/18] hw/arm/boot: introduce fdt_add_memory_node helper

Quoted 16 lines · Eric Auger

From: Shameer Kolothum <[email protected]>

We introduce an helper to create a memory node.

---
 hw/arm/boot.c | 54 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 34 insertions(+), 20 deletions(-)
hw/arm/boot.c +0 -0
index 05762d0fc1..2ef367e15b 100644

hw/arm/boot.c static void set_kernel_args_old(const struct arm_boot_info *info, · line 430

Quoted 8 lines
hw/arm/boot.c +5 -0
@@ -423,6 +423,36 @@ static void set_kernel_args_old(const struct arm_boot_info *info,
     }
 }
 
+static int fdt_add_memory_node(void *fdt, uint32_t acells, hwaddr mem_base,
+                               uint32_t scells, hwaddr mem_len,
+                               int numa_node_id)
+{
+    char *nodename = NULL;

You set nodename immediately below, so no need for the NULL initialization here.

hw/arm/boot.c static void set_kernel_args_old(const struct arm_boot_info *info, · line 441

Quoted 11 lines
hw/arm/boot.c +11 -0
+    int ret;
+
+    nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
+    ret = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, mem_base,
+                                       scells, mem_len);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/reg\n", nodename);
+        goto out;
+    }

I think error handling (ie whether we print messages or not) ought to be done by the calling function, rather than here.

hw/arm/boot.c static void set_kernel_args_old(const struct arm_boot_info *info, · line 442

Quoted 1 line
hw/arm/boot.c +1 -0
+    if (numa_node_id < 0) {

What is this for? My original theory was that this was an error case that should probably be an assert(), but we seem to use it in one of the callers below. A brief comment at the top of the function documenting its API would assist here. If this is "only set the NUMA ID if it is specified" then I think writing it as

  if (numa_node_id >= 0) {
     set the id;

}

is clearer than making it look like an error-exit check.

Quoted 65 lines
hw/arm/boot.c +17 -21
+        goto out;
+    }
+
+    ret = qemu_fdt_setprop_cell(fdt, nodename, "numa-node-id", numa_node_id);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/numa-node-id\n", nodename);
+    }
+
+out:
+    g_free(nodename);
+    return ret;
+}
+
 static void fdt_add_psci_node(void *fdt)
 {
     uint32_t cpu_suspend_fn;
@@ -502,7 +532,6 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
     void *fdt = NULL;
     int size, rc, n = 0;
     uint32_t acells, scells;
-    char *nodename;
     unsigned int i;
     hwaddr mem_base, mem_len;
     char **node_path;
@@ -576,35 +605,20 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         mem_base = binfo->loader_start;
         for (i = 0; i < nb_numa_nodes; i++) {
             mem_len = numa_info[i].node_mem;
-            nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
-            qemu_fdt_add_subnode(fdt, nodename);
-            qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
-            rc = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg",
-                                              acells, mem_base,
-                                              scells, mem_len);
+            rc = fdt_add_memory_node(fdt, acells, mem_base,
+                                     scells, mem_len, i);
             if (rc < 0) {
-                fprintf(stderr, "couldn't set %s/reg for node %d\n", nodename,
-                        i);
                 goto fail;
             }
 
-            qemu_fdt_setprop_cell(fdt, nodename, "numa-node-id", i);
             mem_base += mem_len;
-            g_free(nodename);
         }
     } else {
-        nodename = g_strdup_printf("/memory@%" PRIx64, binfo->loader_start);
-        qemu_fdt_add_subnode(fdt, nodename);
-        qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
-
-        rc = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg",
-                                          acells, binfo->loader_start,
-                                          scells, binfo->ram_size);
+        rc = fdt_add_memory_node(fdt, acells, binfo->loader_start,
+                                 scells, binfo->ram_size, -1);
         if (rc < 0) {
-            fprintf(stderr, "couldn't set %s reg\n", nodename);
             goto fail;
         }
-        g_free(nodename);
     }
 
     rc = fdt_path_offset(fdt, "/chosen");
--
Quoted 2 lines

2.20.1

thanks
-- PMM

author
#7 lore raw

PATCH v6 4/18 hw/arm/virt: Rename highmem IO regions

In reply to Eric Auger

In preparation for a split of the memory map into a static part and a dynamic part floating after the RAM, let's rename the regions located after the RAM

---

v6 : creation

---
 hw/arm/virt-acpi-build.c |  8 ++++----
 hw/arm/virt.c            | 21 +++++++++++----------
 include/hw/arm/virt.h    |  8 ++++----
 3 files changed, 19 insertions(+), 18 deletions(-)
hw/arm/virt-acpi-build.c +4 -4
index 04b62c714d..829d2f0035 100644
@@ -229,8 +229,8 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
                      size_pio));
 
     if (use_highmem) {
-        hwaddr base_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH].base;
-        hwaddr size_mmio_high = memmap[VIRT_PCIE_MMIO_HIGH].size;
+        hwaddr base_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO].base;
+        hwaddr size_mmio_high = memmap[VIRT_HIGH_PCIE_MMIO].size;
 
         aml_append(rbuf,
             aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -663,8 +663,8 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
             gicr = acpi_data_push(table_data, sizeof(*gicr));
             gicr->type = ACPI_APIC_GENERIC_REDISTRIBUTOR;
             gicr->length = sizeof(*gicr);
-            gicr->base_address = cpu_to_le64(memmap[VIRT_GIC_REDIST2].base);
-            gicr->range_length = cpu_to_le32(memmap[VIRT_GIC_REDIST2].size);
+            gicr->base_address = cpu_to_le64(memmap[VIRT_HIGH_GIC_REDIST2].base);
+            gicr->range_length = cpu_to_le32(memmap[VIRT_HIGH_GIC_REDIST2].size);
         }
 
         if (its_class_name() && !vmc->no_its) {
hw/arm/virt.c +11 -10
index 99c2b6e60d..a1955e7764 100644
@@ -150,10 +150,10 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
     [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
     /* Additional 64 MB redist region (can contain up to 512 redistributors) */
-    [VIRT_GIC_REDIST2] =        { 0x4000000000ULL, 0x4000000 },
-    [VIRT_PCIE_ECAM_HIGH] =     { 0x4010000000ULL, 0x10000000 },
+    [VIRT_HIGH_GIC_REDIST2] =   { 0x4000000000ULL, 0x4000000 },
+    [VIRT_HIGH_PCIE_ECAM] =     { 0x4010000000ULL, 0x10000000 },
     /* Second PCIe window, 512GB wide at the 512GB boundary */
-    [VIRT_PCIE_MMIO_HIGH] =   { 0x8000000000ULL, 0x8000000000ULL },
+    [VIRT_HIGH_PCIE_MMIO] =     { 0x8000000000ULL, 0x8000000000ULL },
 };
 
 static const int a15irqmap[] = {
@@ -435,8 +435,8 @@ static void fdt_add_gic_node(VirtMachineState *vms)
                                          2, vms->memmap[VIRT_GIC_DIST].size,
                                          2, vms->memmap[VIRT_GIC_REDIST].base,
                                          2, vms->memmap[VIRT_GIC_REDIST].size,
-                                         2, vms->memmap[VIRT_GIC_REDIST2].base,
-                                         2, vms->memmap[VIRT_GIC_REDIST2].size);
+                                         2, vms->memmap[VIRT_HIGH_GIC_REDIST2].base,
+                                         2, vms->memmap[VIRT_HIGH_GIC_REDIST2].size);
         }
 
         if (vms->virt) {
@@ -584,7 +584,7 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)
 
         if (nb_redist_regions == 2) {
             uint32_t redist1_capacity =
-                        vms->memmap[VIRT_GIC_REDIST2].size / GICV3_REDIST_SIZE;
+                    vms->memmap[VIRT_HIGH_GIC_REDIST2].size / GICV3_REDIST_SIZE;
 
             qdev_prop_set_uint32(gicdev, "redist-region-count[1]",
                 MIN(smp_cpus - redist0_count, redist1_capacity));
@@ -601,7 +601,8 @@ static void create_gic(VirtMachineState *vms, qemu_irq *pic)
     if (type == 3) {
         sysbus_mmio_map(gicbusdev, 1, vms->memmap[VIRT_GIC_REDIST].base);
         if (nb_redist_regions == 2) {
-            sysbus_mmio_map(gicbusdev, 2, vms->memmap[VIRT_GIC_REDIST2].base);
+            sysbus_mmio_map(gicbusdev, 2,
+                            vms->memmap[VIRT_HIGH_GIC_REDIST2].base);
         }
     } else {
         sysbus_mmio_map(gicbusdev, 1, vms->memmap[VIRT_GIC_CPU].base);
@@ -1088,8 +1089,8 @@ static void create_pcie(VirtMachineState *vms, qemu_irq *pic)
 {
     hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO].base;
     hwaddr size_mmio = vms->memmap[VIRT_PCIE_MMIO].size;
-    hwaddr base_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH].base;
-    hwaddr size_mmio_high = vms->memmap[VIRT_PCIE_MMIO_HIGH].size;
+    hwaddr base_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO].base;
+    hwaddr size_mmio_high = vms->memmap[VIRT_HIGH_PCIE_MMIO].size;
     hwaddr base_pio = vms->memmap[VIRT_PCIE_PIO].base;
     hwaddr size_pio = vms->memmap[VIRT_PCIE_PIO].size;
     hwaddr base_ecam, size_ecam;
@@ -1418,7 +1419,7 @@ static void machvirt_init(MachineState *machine)
      */
     if (vms->gic_version == 3) {
         virt_max_cpus = vms->memmap[VIRT_GIC_REDIST].size / GICV3_REDIST_SIZE;
-        virt_max_cpus += vms->memmap[VIRT_GIC_REDIST2].size / GICV3_REDIST_SIZE;
+        virt_max_cpus += vms->memmap[VIRT_HIGH_GIC_REDIST2].size / GICV3_REDIST_SIZE;
     } else {
         virt_max_cpus = GIC_NCPU;
     }
include/hw/arm/virt.h +4 -4
index 4cc57a7ef6..a27086d524 100644
@@ -64,7 +64,7 @@ enum {
     VIRT_GIC_VCPU,
     VIRT_GIC_ITS,
     VIRT_GIC_REDIST,
-    VIRT_GIC_REDIST2,
+    VIRT_HIGH_GIC_REDIST2,
     VIRT_SMMU,
     VIRT_UART,
     VIRT_MMIO,
@@ -74,9 +74,9 @@ enum {
     VIRT_PCIE_MMIO,
     VIRT_PCIE_PIO,
     VIRT_PCIE_ECAM,
-    VIRT_PCIE_ECAM_HIGH,
+    VIRT_HIGH_PCIE_ECAM,
     VIRT_PLATFORM_BUS,
-    VIRT_PCIE_MMIO_HIGH,
+    VIRT_HIGH_PCIE_MMIO,
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
@@ -128,7 +128,7 @@ typedef struct {
     int psci_conduit;
 } VirtMachineState;
 
-#define VIRT_ECAM_ID(high) (high ? VIRT_PCIE_ECAM_HIGH : VIRT_PCIE_ECAM)
+#define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
 
 #define TYPE_VIRT_MACHINE   MACHINE_TYPE_NAME("virt")
 #define VIRT_MACHINE(obj) \
Signature · 1 line
2.20.1
author
#9 lore raw

PATCH v6 5/18 hw/arm/virt: Split the memory map description

In reply to Eric Auger

In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:

- the former a15memmap contains regions below and including the RAM
- extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.

This new split will allow to grow the RAM size without changing the description of the high regions.

The patch also moves the memory map setup into machvirt_init(). The rationale is the memory map will be soon affected by the kvm_type() call that happens after virt_instance_init() and before machvirt_init().

At that point the memory map is not changed, ie. the initial RAM can grow up to 256GiB. Then come the high IO regions with same layout as before.

---
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments

---
 hw/arm/virt.c         | 45 ++++++++++++++++++++++++++++++++++++++-----
 include/hw/arm/virt.h | 14 ++++++++++----
 2 files changed, 50 insertions(+), 9 deletions(-)
hw/arm/virt.c +40 -5
index a1955e7764..2b15839d0b 100644
@@ -29,6 +29,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/units.h"
 #include "qapi/error.h"
 #include "hw/sysbus.h"
 #include "hw/arm/arm.h"
@@ -149,11 +150,20 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
     [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
+};
+
+/*
+ * Highmem IO Regions: This memory map is floating, located after the RAM.
+ * Each IO region offset will be dynamically computed, depending on the
+ * top of the RAM, so that its base get the same alignment as the size,
+ * ie. a 512GiB region will be aligned on a 512GiB boundary.
+ */
+static MemMapEntry extended_memmap[] = {
     /* Additional 64 MB redist region (can contain up to 512 redistributors) */
-    [VIRT_HIGH_GIC_REDIST2] =   { 0x4000000000ULL, 0x4000000 },
-    [VIRT_HIGH_PCIE_ECAM] =     { 0x4010000000ULL, 0x10000000 },
-    /* Second PCIe window, 512GB wide at the 512GB boundary */
-    [VIRT_HIGH_PCIE_MMIO] =     { 0x8000000000ULL, 0x8000000000ULL },
+    [VIRT_HIGH_GIC_REDIST2] =   { 0x0, 64 * MiB },
+    [VIRT_HIGH_PCIE_ECAM] =     { 0x0, 256 * MiB },
+    /* Second PCIe window */
+    [VIRT_HIGH_PCIE_MMIO] =     { 0x0, 512 * GiB },
 };
 
 static const int a15irqmap[] = {
@@ -1354,6 +1364,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
     return arm_cpu_mp_affinity(idx, clustersz);
 }
 
+static void virt_set_memmap(VirtMachineState *vms)
+{
+    hwaddr base;
+    int i;
+
+    vms->memmap = extended_memmap;
+
+    for (i = 0; i < ARRAY_SIZE(a15memmap); i++) {
+        vms->memmap[i] = a15memmap[i];
+    }
+
+    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    base = vms->high_io_base;
+
+    for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
+        hwaddr size = extended_memmap[i].size;
+
+        base = ROUND_UP(base, size);
+        vms->memmap[i].base = base;
+        vms->memmap[i].size = size;
+        base += size;
+    }
+}
+
 static void machvirt_init(MachineState *machine)
 {
     VirtMachineState *vms = VIRT_MACHINE(machine);
@@ -1368,6 +1402,8 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
+    virt_set_memmap(vms);
+
     /* We can probe only here because during property set
      * KVM is not available yet
      */
@@ -1843,7 +1879,6 @@ static void virt_instance_init(Object *obj)
                                     "Valid values are none and smmuv3",
                                     NULL);
 
-    vms->memmap = a15memmap;
     vms->irqmap = a15irqmap;
 }
 
include/hw/arm/virt.h +10 -4
index a27086d524..3dc7a6c5d5 100644
@@ -64,7 +64,6 @@ enum {
     VIRT_GIC_VCPU,
     VIRT_GIC_ITS,
     VIRT_GIC_REDIST,
-    VIRT_HIGH_GIC_REDIST2,
     VIRT_SMMU,
     VIRT_UART,
     VIRT_MMIO,
@@ -74,12 +73,18 @@ enum {
     VIRT_PCIE_MMIO,
     VIRT_PCIE_PIO,
     VIRT_PCIE_ECAM,
-    VIRT_HIGH_PCIE_ECAM,
     VIRT_PLATFORM_BUS,
-    VIRT_HIGH_PCIE_MMIO,
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_LOWMEMMAP_LAST,
+};
+
+/* indices of IO regions located after the RAM */
+enum {
+    VIRT_HIGH_GIC_REDIST2 =  VIRT_LOWMEMMAP_LAST,
+    VIRT_HIGH_PCIE_ECAM,
+    VIRT_HIGH_PCIE_MMIO,
 };
 
 typedef enum VirtIOMMUType {
@@ -116,7 +121,7 @@ typedef struct {
     int32_t gic_version;
     VirtIOMMUType iommu;
     struct arm_boot_info bootinfo;
-    const MemMapEntry *memmap;
+    MemMapEntry *memmap;
     const int *irqmap;
     int smp_cpus;
     void *fdt;
@@ -126,6 +131,7 @@ typedef struct {
     uint32_t msi_phandle;
     uint32_t iommu_phandle;
     int psci_conduit;
+    hwaddr high_io_base;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
Reviewed-by on v6 5/18
#10 lore raw

Re: [Qemu-arm] [PATCH v6 05/18] hw/arm/virt: Split the memory map description

Quoted 4 lines · Eric Auger

In the prospect to introduce an extended memory map supporting more RAM, let's split the memory map array into two parts:

Quoted 8 lines

- the former a15memmap contains regions below and including the RAM
- extended_memmap, only initialized with entries located after the RAM. Only the size of the region is initialized there since their base address will be dynamically computed, depending on the top of the RAM (initial RAM at the moment), with same alignment as their size.

This new split will allow to grow the RAM size without changing the description of the high regions.

This change makes it clear that "a15memmap" is badly misnamed. I think we should change it to "base_memmap" here.

Quoted 7 lines

The patch also moves the memory map setup into machvirt_init(). The rationale is the memory map will be soon affected by the kvm_type() call that happens after virt_instance_init() and before machvirt_init().

At that point the memory map is not changed, ie. the initial RAM can

"At this point" ?

Quoted 29 lines

grow up to 256GiB. Then come the high IO regions with same layout as before.

---
v5 -> v6
- removal of many macros in units.h
- introduce the virt_set_memmap helper
- new computation for offsets of high IO regions
- add comments

---
 hw/arm/virt.c         | 45 ++++++++++++++++++++++++++++++++++++++-----
 include/hw/arm/virt.h | 14 ++++++++++----
 2 files changed, 50 insertions(+), 9 deletions(-)
hw/arm/virt.c +1 -0
index a1955e7764..2b15839d0b 100644
@@ -29,6 +29,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/units.h"
 #include "qapi/error.h"
 #include "hw/sysbus.h"
 #include "hw/arm/arm.h"

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 159

Quoted 10 lines
hw/arm/virt.c +7 -0
@@ -149,11 +150,20 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
     [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
+};
+
+/*
+ * Highmem IO Regions: This memory map is floating, located after the RAM.
+ * Each IO region offset will be dynamically computed, depending on the
+ * top of the RAM, so that its base get the same alignment as the size,
+ * ie. a 512GiB region will be aligned on a 512GiB boundary.

I think you should say here that if there is less than 256GiB of RAM then the floating area starts at the 256GiB mark.

Quoted 15 lines
hw/arm/virt.c +6 -4
+ */
+static MemMapEntry extended_memmap[] = {
     /* Additional 64 MB redist region (can contain up to 512 redistributors) */
-    [VIRT_HIGH_GIC_REDIST2] =   { 0x4000000000ULL, 0x4000000 },
-    [VIRT_HIGH_PCIE_ECAM] =     { 0x4010000000ULL, 0x10000000 },
-    /* Second PCIe window, 512GB wide at the 512GB boundary */
-    [VIRT_HIGH_PCIE_MMIO] =     { 0x8000000000ULL, 0x8000000000ULL },
+    [VIRT_HIGH_GIC_REDIST2] =   { 0x0, 64 * MiB },
+    [VIRT_HIGH_PCIE_ECAM] =     { 0x0, 256 * MiB },
+    /* Second PCIe window */
+    [VIRT_HIGH_PCIE_MMIO] =     { 0x0, 512 * GiB },
 };
 
 static const int a15irqmap[] = {

hw/arm/virt.c static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) · line 1365

Quoted 2 lines
hw/arm/virt.c +0 -0
@@ -1354,6 +1364,30 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
     return arm_cpu_mp_affinity(idx, clustersz);
 }

Otherwise

thanks
-- PMM

author
#11 lore raw

PATCH v6 6/18 hw/boards: Add a MachineState parameter to kvm_type callback

In reply to Eric Auger

On ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.

---
 accel/kvm/kvm-all.c   | 2 +-
 hw/ppc/mac_newworld.c | 3 +--
 hw/ppc/mac_oldworld.c | 2 +-
 hw/ppc/spapr.c        | 2 +-
 include/hw/boards.h   | 2 +-
 5 files changed, 5 insertions(+), 6 deletions(-)
accel/kvm/kvm-all.c +1 -1
index 4e1de942ce..503900604c 100644
@@ -1590,7 +1590,7 @@ static int kvm_init(MachineState *ms)
 
     kvm_type = qemu_opt_get(qemu_get_machine_opts(), "kvm-type");
     if (mc->kvm_type) {
-        type = mc->kvm_type(kvm_type);
+        type = mc->kvm_type(ms, kvm_type);
     } else if (kvm_type) {
         ret = -EINVAL;
         fprintf(stderr, "Invalid argument kvm-type=%s\n", kvm_type);
hw/ppc/mac_newworld.c +1 -2
index f1c8400efd..3cce612ffa 100644
@@ -563,8 +563,7 @@ static char *core99_fw_dev_path(FWPathProvider *p, BusState *bus,
 
     return NULL;
 }
-
-static int core99_kvm_type(const char *arg)
+static int core99_kvm_type(MachineState *ms, const char *arg)
 {
     /* Always force PR KVM */
     return 2;
hw/ppc/mac_oldworld.c +1 -1
index 98d531d114..f3a594d7dc 100644
@@ -419,7 +419,7 @@ static char *heathrow_fw_dev_path(FWPathProvider *p, BusState *bus,
     return NULL;
 }
 
-static int heathrow_kvm_type(const char *arg)
+static int heathrow_kvm_type(MachineState *ms, const char *arg)
 {
     /* Always force PR KVM */
     return 2;
hw/ppc/spapr.c +1 -1
index 0fcdd35cbe..90cd0acb74 100644
@@ -2919,7 +2919,7 @@ static void spapr_machine_init(MachineState *machine)
     }
 }
 
-static int spapr_kvm_type(const char *vm_type)
+static int spapr_kvm_type(MachineState *ms, const char *vm_type)
 {
     if (!vm_type) {
         return 0;
include/hw/boards.h +1 -1
index 02f114085f..425d2c86a6 100644
@@ -171,7 +171,7 @@ struct MachineClass {
     void (*init)(MachineState *state);
     void (*reset)(void);
     void (*hot_add_cpu)(const int64_t id, Error **errp);
-    int (*kvm_type)(const char *arg);
+    int (*kvm_type)(MachineState *ms, const char *arg);
 
     BlockInterfaceType block_default_type;
     int units_per_default_bus;
Signature · 1 line
2.20.1
Reviewed-by on v6 6/18
#12 lore raw

Re: [Qemu-devel] [PATCH v6 06/18] hw/boards: Add a MachineState parameter to kvm_type callback

Quoted 24 lines · Eric Auger

On ARM, the kvm_type will be resolved by querying the KVMState. Let's add the MachineState handle to the callback so that we can retrieve the KVMState handle. in kvm_init, when the callback is called, the kvm_state variable is not yet set.

---
 accel/kvm/kvm-all.c   | 2 +-
 hw/ppc/mac_newworld.c | 3 +--
 hw/ppc/mac_oldworld.c | 2 +-
 hw/ppc/spapr.c        | 2 +-
 include/hw/boards.h   | 2 +-
 5 files changed, 5 insertions(+), 6 deletions(-)
include/hw/boards.h +0 -0
index 02f114085f..425d2c86a6 100644

include/hw/boards.h struct MachineClass {

Quoted 9 lines
include/hw/boards.h +1 -2
@@ -171,7 +171,7 @@ struct MachineClass {
     void (*init)(MachineState *state);
     void (*reset)(void);
     void (*hot_add_cpu)(const int64_t id, Error **errp);
-    int (*kvm_type)(const char *arg);
+    int (*kvm_type)(MachineState *ms, const char *arg);
 
     BlockInterfaceType block_default_type;
     int units_per_default_bus;
--

Can you add a line to the struct's documentation comment for the @kvm_type field, please ?

We're rather inconsistent about what we name the MachineState* parameter in methods here:
"state" x 1 (init)
"machine" x 3 (get_hotplug_handler, cpu_index_to_instance_props,
possible_cpu_arch_ids)
"ms" x 1 (get_default_cpu_node_id)

It would probably be better to follow the most common option rather than one of the rarer ones.

Otherwise

thanks
-- PMM

author
#13 lore raw

PATCH v6 7/18 kvm: add kvm_arm_get_max_vm_phys_shift

In reply to Eric Auger

Add the kvm_arm_get_max_vm_phys_shift() helper that returns the log of the maximum IPA size supported by KVM. This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.

---
v4 -> v5:
- return 40 if the host does not support the capability

v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension

v1 -> v2:
- put this in ARM specific code

---
 target/arm/kvm.c     | 10 ++++++++++
 target/arm/kvm_arm.h | 13 +++++++++++++
 2 files changed, 23 insertions(+)
target/arm/kvm.c +10 -0
index e00ccf9c98..fc1dd3ec6a 100644
@@ -18,6 +18,7 @@
 #include "qemu/error-report.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
 #include "kvm_arm.h"
 #include "cpu.h"
 #include "trace.h"
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     env->features = arm_host_cpu_features.features;
 }
 
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms)
+{
+    KVMState *s = KVM_STATE(ms->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_VM_IPA_SIZE);
+    return ret > 0 ? ret : 40;
+}
+
 int kvm_arch_init(MachineState *ms, KVMState *s)
 {
     /* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +13 -0
index 6393455b1d..0728bbfa6b 100644
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
  */
 void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
 
+/**
+ * kvm_arm_get_max_vm_phys_shift - Returns log2 of the max IPA size
+ * supported by KVM
+ *
+ * @ms: Machine state handle
+ */
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms);
+
 /**
  * kvm_arm_sync_mpstate_to_kvm
  * @cpu: ARMCPU
@@ -239,6 +247,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     cpu->host_cpu_probe_failed = true;
 }
 
+static inline int kvm_arm_get_max_vm_phys_shift(MachineState *ms)
+{
+    return -ENOENT;
+}
+
 static inline int kvm_arm_vgic_probe(void)
 {
     return 0;
Signature · 1 line
2.20.1
#14 lore raw

Re: [Qemu-devel] [PATCH v6 07/18] kvm: add kvm_arm_get_max_vm_phys_shift

Quoted 36 lines · Eric Auger

Add the kvm_arm_get_max_vm_phys_shift() helper that returns the log of the maximum IPA size supported by KVM. This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.

---
v4 -> v5:
- return 40 if the host does not support the capability

v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension

v1 -> v2:
- put this in ARM specific code

---
 target/arm/kvm.c     | 10 ++++++++++
 target/arm/kvm_arm.h | 13 +++++++++++++
 2 files changed, 23 insertions(+)
target/arm/kvm.c +1 -0
index e00ccf9c98..fc1dd3ec6a 100644
@@ -18,6 +18,7 @@
 #include "qemu/error-report.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
 #include "kvm_arm.h"
 #include "cpu.h"
 #include "trace.h"

target/arm/kvm.c void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) · line 171

Quoted 9 lines
target/arm/kvm.c +6 -0
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     env->features = arm_host_cpu_features.features;
 }
 
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms)
+{
+    KVMState *s = KVM_STATE(ms->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_VM_IPA_SIZE);

Why not name the function the same as the extension name?

Quoted 11 lines
target/arm/kvm.c +3 -0
+    return ret > 0 ? ret : 40;
+}
+
 int kvm_arch_init(MachineState *ms, KVMState *s)
 {
     /* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +0 -0
index 6393455b1d..0728bbfa6b 100644

target/arm/kvm_arm.h bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf); · line 212

Quoted 6 lines
target/arm/kvm_arm.h +3 -0
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
  */
 void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
 
+/**
+ * kvm_arm_get_max_vm_phys_shift - Returns log2 of the max IPA size
+ * supported by KVM

This is the number of bits in the IPA address space, right (ie 40 for a 40-bit IPA, and so on) ? If so, then I think "Return number of bits in the IPA address space" might be clearer.

target/arm/kvm_arm.h bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf); · line 216

Quoted 4 lines
target/arm/kvm_arm.h +4 -0
+ *
+ * @ms: Machine state handle
+ */
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms);

thanks
-- PMM

author
#15 lore raw

Re: [Qemu-arm] [PATCH v6 07/18] kvm: add kvm_arm_get_max_vm_phys_shift

Hi Peter,

Quoted 63 lines · Peter Maydell
Quoted 36 lines · Eric Auger

Add the kvm_arm_get_max_vm_phys_shift() helper that returns the log of the maximum IPA size supported by KVM. This capability needs to be known to create the VM with a specific IPA max size (kvm_type passed along KVM_CREATE_VM ioctl.

---
v4 -> v5:
- return 40 if the host does not support the capability

v3 -> v4:
- s/s/ms in kvm_arm_get_max_vm_phys_shift function comment
- check KVM_CAP_ARM_VM_IPA_SIZE extension

v1 -> v2:
- put this in ARM specific code

---
 target/arm/kvm.c     | 10 ++++++++++
 target/arm/kvm_arm.h | 13 +++++++++++++
 2 files changed, 23 insertions(+)
target/arm/kvm.c +1 -0
index e00ccf9c98..fc1dd3ec6a 100644
@@ -18,6 +18,7 @@
 #include "qemu/error-report.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
 #include "kvm_arm.h"
 #include "cpu.h"
 #include "trace.h"

target/arm/kvm.c void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu) · line 171

Quoted 9 lines
target/arm/kvm.c +6 -0
@@ -162,6 +163,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     env->features = arm_host_cpu_features.features;
 }
 
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms)
+{
+    KVMState *s = KVM_STATE(ms->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_VM_IPA_SIZE);

Why not name the function the same as the extension name?

Quoted 14 lines
target/arm/kvm.c +3 -0
+    return ret > 0 ? ret : 40;
+}
+
 int kvm_arch_init(MachineState *ms, KVMState *s)
 {
     /* For ARM interrupt delivery is always asynchronous,
target/arm/kvm_arm.h +0 -0
index 6393455b1d..0728bbfa6b 100644
@@ -207,6 +207,14 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
  */
 void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
 
Quoted 8 lines

target/arm/kvm_arm.h bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf); · line 212

Quoted 3 lines
target/arm/kvm_arm.h +3 -0
+/**
+ * kvm_arm_get_max_vm_phys_shift - Returns log2 of the max IPA size
+ * supported by KVM

This is the number of bits in the IPA address space, right (ie 40 for a 40-bit IPA, and so on) ? If so, then I think "Return number of bits in the IPA address space" might be clearer.

It actually returns the MAX number of bits in the IPA address space supported by the host kernel. What about naming this function "kvm_arm_get_max_vm_ipa_size".

Thanks

Eric

Quoted 9 lines

target/arm/kvm_arm.h bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf); · line 216

Quoted 5 lines
target/arm/kvm_arm.h +4 -0
+ *
+ * @ms: Machine state handle
+ */
+int kvm_arm_get_max_vm_phys_shift(MachineState *ms);

thanks
-- PMM

author
#16 lore raw

PATCH v6 8/18 vl: Set machine ram_size, maxram_size and ram_slots earlier

In reply to Eric Auger

The machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.

---

v4: new

---
 vl.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +3 -3
index 9cf0fbe0b8..28f6bbebe2 100644
@@ -4324,6 +4324,9 @@ int main(int argc, char **argv, char **envp)
     machine_opts = qemu_get_machine_opts();
     qemu_opt_foreach(machine_opts, machine_set_property, current_machine,
                      &error_fatal);
+    current_machine->ram_size = ram_size;
+    current_machine->maxram_size = maxram_size;
+    current_machine->ram_slots = ram_slots;
 
     configure_accelerator(current_machine, argv[0]);
 
@@ -4521,9 +4524,6 @@ int main(int argc, char **argv, char **envp)
     replay_checkpoint(CHECKPOINT_INIT);
     qdev_machine_init();
 
-    current_machine->ram_size = ram_size;
-    current_machine->maxram_size = maxram_size;
-    current_machine->ram_slots = ram_slots;
     current_machine->boot_order = boot_order;
 
     /* parse features once if machine provides default cpu_type */
Signature · 1 line
2.20.1
Reviewed-by on v6 8/18
#17 lore raw

Re: [Qemu-arm] [PATCH v6 08/18] vl: Set machine ram_size, maxram_size and ram_slots earlier

Quoted 21 lines · Eric Auger

The machine RAM attributes will need to be analyzed during the configure_accelerator() process. especially kvm_type() arm64 machine callback will use them to know how many IPA/GPA bits are needed to model the whole RAM range. So let's assign those machine state fields before calling configure_accelerator.

---

v4: new

---
 vl.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
vl.c +0 -0
index 9cf0fbe0b8..28f6bbebe2 100644

vl.c int main(int argc, char **argv, char **envp) · line 4331

Quoted 8 lines
vl.c +3 -0
@@ -4324,6 +4324,9 @@ int main(int argc, char **argv, char **envp)
     machine_opts = qemu_get_machine_opts();
     qemu_opt_foreach(machine_opts, machine_set_property, current_machine,
                      &error_fatal);
+    current_machine->ram_size = ram_size;
+    current_machine->maxram_size = maxram_size;
+    current_machine->ram_slots = ram_slots;
 
     configure_accelerator(current_machine, argv[0]);

This is still after the call to set_memory_options(), so it's OK.

thanks
-- PMM

author
#18 lore raw

PATCH v6 9/18 hw/arm/virt: Implement kvm_type function for 4.0 machine

In reply to Eric Auger

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +50 -2
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }
 
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -0
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
#19 lore raw

Re: [Qemu-devel] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

Quoted 42 lines · Eric Auger

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +1 -0
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }

hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1386

Quoted 10 lines
hw/arm/virt.c +8 -1
 
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);

I don't understand this expression...

Quoted 41 lines
hw/arm/virt.c +19 -1
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);

hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1822

Quoted 10 lines
hw/arm/virt.c +10 -0
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }

Presumably we should have some equivalent check for TCG, so that we don't let the user create a setup which wants more bits of physical address than the TCG CPU allows ?

Quoted 20 lines
hw/arm/virt.c +9 -0
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 6 lines
hw/arm/virt.c +3 -0
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;

When is there a difference between setting this to NULL, and setting it to virt_kvm_type but having the memory size be <= 256GiB ?

If there isn't any difference, why can't we just let the pre-4.0 versions behave like the new ones? No existing VM setup will have > 256GB of memory, so as long as there's no behaviour change for the <=256GB case we don't need to take special effort to ensure that the >256GB case continues to give an error message, do we ?

Quoted 17 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -1
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
--
Quoted 1 line

2.20.1

thanks
-- PMM

author
#20 lore raw

Re: [Qemu-arm] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

Hi Peter,

Quoted 44 lines · Peter Maydell
Quoted 43 lines · Eric Auger

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +1 -0
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }
 
Quoted 11 lines

hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1386

Quoted 9 lines
hw/arm/virt.c +8 -1
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);

I don't understand this expression...

My intent was to align the start of the device memory on a GiB boundary, just after the initial RAM (ram_size). And then align the floating IO region on a GiB boundary after the device memory (of size ms->maxram_size - ms->ram_size). What do I miss?

Quoted 48 lines
Quoted 48 lines
hw/arm/virt.c +25 -1
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
Quoted 8 lines

hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1822

Quoted 4 lines
hw/arm/virt.c +4 -0
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }

Presumably we should have some equivalent check for TCG, so that we don't let the user create a setup which wants more bits of physical address than the TCG CPU allows ?

kvm_type() sets the new memory map. For TCG we should stick to the 1TB GPA address space which should be consistent with the existing ID_AA64MMFR0_EL1 settings (arm/internals.h implements arm_pamax(ARMCPU *cpu) which decodes hardcoded cpu->id_aa64mmfr0).

Quoted 23 lines
Quoted 23 lines
hw/arm/virt.c +9 -0
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
Quoted 8 lines

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 4 lines
hw/arm/virt.c +3 -0
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;

When is there a difference between setting this to NULL, and setting it to virt_kvm_type but having the memory size be <= 256GiB ?

There shouldn't be any difference. When size <= 255GiB we stick to the 1TB PA address space.

Quoted 7 lines

If there isn't any difference, why can't we just let the pre-4.0 versions behave like the new ones? No existing VM setup will have > 256GB of memory, so as long as there's no behaviour change for the <=256GB case we don't need to take special effort to ensure that the >256GB case continues to give an error message, do we ?

But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?

Thanks

Eric

Quoted 23 lines
Quoted 18 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -1
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
--
Quoted 1 line

2.20.1

thanks
-- PMM

#21 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

Quoted 58 lines · On Mon, 18 Feb 2019 22:29:40 +0100 Auger Eric <[email protected]> wrote:

Hi Peter,

Quoted 44 lines · Peter Maydell
Quoted 43 lines · Eric Auger

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +1 -0
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }
 
Quoted 11 lines

hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1386

Quoted 9 lines
hw/arm/virt.c +8 -1
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);  

I don't understand this expression...

Quoted 4 lines

My intent was to align the start of the device memory on a GiB boundary, just after the initial RAM (ram_size). And then align the floating IO region on a GiB boundary after the device memory (of size ms->maxram_size - ms->ram_size). What do I miss?

It's not obvious what "GiB + ms->ram_size" means and where it comes from, maybe substitute GiB with properly named constant/macro that's also re-used in memmap definition so it would be obvious that's it's where initial RAM is mapped. Also I'd move both ROUND_UPs into separate expressions using reasonable named local vars and possible overflow checks on top of that, so one won't have to guess that it's initial RAM end + device RAM end.

Quoted 94 lines
Quoted 48 lines
Quoted 48 lines
hw/arm/virt.c +25 -1
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
Quoted 8 lines

hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1822

Quoted 4 lines
hw/arm/virt.c +4 -0
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }  

Presumably we should have some equivalent check for TCG, so that we don't let the user create a setup which wants more bits of physical address than the TCG CPU allows ?

kvm_type() sets the new memory map. For TCG we should stick to the 1TB GPA address space which should be consistent with the existing ID_AA64MMFR0_EL1 settings (arm/internals.h implements arm_pamax(ARMCPU *cpu) which decodes hardcoded cpu->id_aa64mmfr0).

Quoted 23 lines
Quoted 23 lines
hw/arm/virt.c +9 -0
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
Quoted 8 lines

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 4 lines
hw/arm/virt.c +3 -0
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;  

When is there a difference between setting this to NULL, and setting it to virt_kvm_type but having the memory size be <= 256GiB ?

There shouldn't be any difference. When size <= 255GiB we stick to the 1TB PA address space.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

If there isn't any difference, why can't we just let the pre-4.0 versions behave like the new ones? No existing VM setup will have > 256GB of memory, so as long as there's no behaviour change for the <=256GB case we don't need to take special effort to ensure that the >256GB case continues to give an error message, do we ?

But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?

Why would we if it doesn't break migration?

Quoted 27 lines

Thanks

Eric

Quoted 23 lines
Quoted 18 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -1
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
--
Quoted 1 line

2.20.1

thanks
-- PMM

author
#22 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

Hi Igor,

Quoted 61 lines · Igor Mammedov
Quoted 58 lines · On Mon, 18 Feb 2019 22:29:40 +0100 Auger Eric <[email protected]> wrote:

Hi Peter,

Quoted 55 lines · Peter Maydell
Quoted 42 lines · Eric Auger

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +1 -0
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }

hw/arm/virt.c static void virt_set_memmap(VirtMachineState *vms) · line 1386

Quoted 10 lines
hw/arm/virt.c +8 -1
 
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);  

I don't understand this expression...

Quoted 6 lines
Quoted 4 lines

My intent was to align the start of the device memory on a GiB boundary, just after the initial RAM (ram_size). And then align the floating IO region on a GiB boundary after the device memory (of size ms->maxram_size - ms->ram_size). What do I miss?

It's not obvious what "GiB + ms->ram_size" means and where it comes from,

I agree

Quoted 5 lines

maybe substitute GiB with properly named constant/macro that's also re-used in memmap definition so it would be obvious that's it's where initial RAM is mapped. Also I'd move both ROUND_UPs into separate expressions using reasonable named local vars and possible overflow checks on top of that, so one won't have to guess that it's initial RAM end + device RAM end.

Makes sense too.

Thanks

Eric

Quoted 135 lines
Quoted 95 lines
Quoted 49 lines
Quoted 49 lines
hw/arm/virt.c +25 -1
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
Quoted 8 lines

hw/arm/virt.c static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, · line 1822

Quoted 4 lines
hw/arm/virt.c +4 -0
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }  

Presumably we should have some equivalent check for TCG, so that we don't let the user create a setup which wants more bits of physical address than the TCG CPU allows ?

kvm_type() sets the new memory map. For TCG we should stick to the 1TB GPA address space which should be consistent with the existing ID_AA64MMFR0_EL1 settings (arm/internals.h implements arm_pamax(ARMCPU *cpu) which decodes hardcoded cpu->id_aa64mmfr0).

Quoted 23 lines
Quoted 23 lines
hw/arm/virt.c +9 -0
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
Quoted 8 lines

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 4 lines
hw/arm/virt.c +3 -0
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;  

When is there a difference between setting this to NULL, and setting it to virt_kvm_type but having the memory size be <= 256GiB ?

There shouldn't be any difference. When size <= 255GiB we stick to the 1TB PA address space.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

If there isn't any difference, why can't we just let the pre-4.0 versions behave like the new ones? No existing VM setup will have > 256GB of memory, so as long as there's no behaviour change for the <=256GB case we don't need to take special effort to ensure that the >256GB case continues to give an error message, do we ?

But don't we want to forbid any pre-4.0 machvirt to run with more than 255GiB RAM?

Why would we if it doesn't break migration?

Quoted 27 lines

Thanks

Eric

Quoted 23 lines
Quoted 18 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -1
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
--
Quoted 1 line

2.20.1

thanks
-- PMM

#23 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

In reply to Eric Auger
Quoted 122 lines · On Tue, 5 Feb 2019 18:32:57 +0100 Eric Auger <[email protected]> wrote:

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +47 -2
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }
 
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 6 lines
hw/arm/virt.c +3 -0
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;

it's quite confusing, you have vms->extended_memmap and mc->kvm_type and the later for some reason enables device memory.

to me it seems that both are not related, device memory should work just fine without kvm nor dynamic IPA (within TCG supported limits).

I'd make extended_memmap virt machine class member the will enable pc-dimm support and then it add checks for supported IPA range on top

Quoted 16 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -0
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
author
#24 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 09/18] hw/arm/virt: Implement kvm_type function for 4.0 machine

Hi Igor,

Quoted 134 lines · Igor Mammedov
Quoted 122 lines · On Tue, 5 Feb 2019 18:32:57 +0100 Eric Auger <[email protected]> wrote:

This patch implements the machine class kvm_type() callback. It returns the max IPA shift needed to implement the whole GPA range including the RAM and IO regions located beyond. The returned value in passed though the KVM_CREATE_VM ioctl and this allows KVM to set the stage2 tables dynamically.

At this stage the RAM limit still is limited to 255GB.

Setting all the existing highmem IO regions beyond the RAM allows to have a single contiguous RAM region (initial RAM and possible hotpluggable device memory). That way we do not need to do invasive changes in the EDK2 FW to support a dynamic RAM base.

---

v5 -> v6:
- add some comments
- high IO region cannot start before 256GiB

---
 hw/arm/virt.c         | 52 +++++++++++++++++++++++++++++++++++++++++--
 include/hw/arm/virt.h |  2 ++
 2 files changed, 52 insertions(+), 2 deletions(-)
hw/arm/virt.c +47 -2
index 2b15839d0b..b90ffc2e5d 100644
@@ -1366,6 +1366,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 
 static void virt_set_memmap(VirtMachineState *vms)
 {
+    MachineState *ms = MACHINE(vms);
     hwaddr base;
     int i;
 
@@ -1375,7 +1376,17 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i] = a15memmap[i];
     }
 
-    vms->high_io_base = 256 * GiB; /* Top of the legacy initial RAM region */
+    /*
+     * We now compute the base of the high IO region depending on the
+     * amount of initial and device memory. The device memory start/size
+     * is aligned on 1GiB. We never put the high IO region below 256GiB
+     * so that if maxram_size is < 255GiB we keep the legacy memory map
+     */
+    vms->high_io_base = ROUND_UP(GiB + ms->ram_size, GiB) +
+                        ROUND_UP(ms->maxram_size - ms->ram_size, GiB);
+    if (vms->high_io_base < 256 * GiB) {
+        vms->high_io_base = 256 * GiB;
+    }
     base = vms->high_io_base;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
@@ -1386,6 +1397,7 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
+    vms->highest_gpa = base - 1;
 }
 
 static void machvirt_init(MachineState *machine)
@@ -1402,7 +1414,9 @@ static void machvirt_init(MachineState *machine)
     bool firmware_loaded = bios_name || drive_get(IF_PFLASH, 0, 0);
     bool aarch64 = true;
 
-    virt_set_memmap(vms);
+    if (!vms->extended_memmap) {
+        virt_set_memmap(vms);
+    }
 
     /* We can probe only here because during property set
      * KVM is not available yet
@@ -1784,6 +1798,36 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
     return NULL;
 }
 
+/*
+ * for arm64 kvm_type [7-0] encodes the IPA size shift
+ */
+static int virt_kvm_type(MachineState *ms, const char *type_str)
+{
+    VirtMachineState *vms = VIRT_MACHINE(ms);
+    int max_vm_phys_shift = kvm_arm_get_max_vm_phys_shift(ms);
+    int max_pa_shift;
+
+    vms->extended_memmap = true;
+
+    virt_set_memmap(vms);
+
+    max_pa_shift = 64 - clz64(vms->highest_gpa);
+
+    if (max_pa_shift > max_vm_phys_shift) {
+        error_report("-m and ,maxmem option values "
+                     "require an IPA range (%d bits) larger than "
+                     "the one supported by the host (%d bits)",
+                     max_pa_shift, max_vm_phys_shift);
+       exit(1);
+    }
+    /*
+     * By default we return 0 which corresponds to an implicit legacy
+     * 40b IPA setting. Otherwise we return the actual requested IPA
+     * logsize
+     */
+    return max_pa_shift > 40 ? max_pa_shift : 0;
+}
+
 static void virt_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
@@ -1808,6 +1852,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
     mc->cpu_index_to_instance_props = virt_cpu_index_to_props;
     mc->default_cpu_type = ARM_CPU_TYPE_NAME("cortex-a15");
     mc->get_default_cpu_node_id = virt_get_default_cpu_node_id;
+    mc->kvm_type = virt_kvm_type;
     assert(!mc->get_hotplug_handler);
     mc->get_hotplug_handler = virt_machine_get_hotplug_handler;
     hc->plug = virt_machine_device_plug_cb;

hw/arm/virt.c static void virt_machine_3_1_options(MachineClass *mc) · line 1961

Quoted 6 lines
hw/arm/virt.c +3 -0
@@ -1911,6 +1956,9 @@ static void virt_machine_3_1_options(MachineClass *mc)
 {
     virt_machine_4_0_options(mc);
     compat_props_add(mc->compat_props, hw_compat_3_1, hw_compat_3_1_len);
+
+    /* extended memory map is enabled from 4.0 onwards */
+    mc->kvm_type = NULL;

it's quite confusing, you have vms->extended_memmap and mc->kvm_type and the later for some reason enables device memory.

Quoted 5 lines

to me it seems that both are not related, device memory should work just fine without kvm nor dynamic IPA (within TCG supported limits).

I'd make extended_memmap virt machine class member the will enable pc-dimm support and then it add checks for supported IPA range on top

I agree I did not take into account the TCG use case and this series does not enable device memory with TCG which is a pitty. I will decorrelate things.

Thanks!

Eric

Quoted 18 lines
Quoted 17 lines
hw/arm/virt.c +0 -0
 }
 DEFINE_VIRT_MACHINE(3, 1)
 
include/hw/arm/virt.h +2 -0
index 3dc7a6c5d5..c88f67a492 100644
@@ -132,6 +132,8 @@ typedef struct {
     uint32_t iommu_phandle;
     int psci_conduit;
     hwaddr high_io_base;
+    hwaddr highest_gpa;
+    bool extended_memmap;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
author
#25 lore raw

PATCH v6 10/18 hw/arm/virt: Bump the 255GB initial RAM limit

In reply to Eric Auger

Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel.

---
 hw/arm/virt.c | 26 +++++++-------------------
 1 file changed, 7 insertions(+), 19 deletions(-)
hw/arm/virt.c +7 -19
index b90ffc2e5d..f01886da22 100644
@@ -93,22 +93,9 @@
 
 #define PLATFORM_BUS_NUM_IRQS 64
 
-/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means
- * RAM can go up to the 256GB mark, leaving 256GB of the physical
- * address space unallocated and free for future use between 256G and 512G.
- * If we need to provide more RAM to VMs in the future then we need to:
- *  * allocate a second bank of RAM starting at 2TB and working up
- *  * fix the DT and ACPI table generation code in QEMU to correctly
- *    report two split lumps of RAM to the guest
- *  * fix KVM in the host kernel to allow guests with >40 bit address spaces
- * (We don't want to fill all the way up to 512GB with RAM because
- * we might want it for non-RAM purposes later. Conversely it seems
- * reasonable to assume that anybody configuring a VM with a quarter
- * of a terabyte of RAM will be doing it on a host with more than a
- * terabyte of physical address space.)
- */
-#define RAMLIMIT_GB 255
-#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024)
+/* Legacy RAM limit in GB (< version 4.0) */
+#define LEGACY_RAMLIMIT_GB 255
+#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
 
 /* Addresses and sizes of our components.
  * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
@@ -149,7 +136,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_MMIO] =          { 0x10000000, 0x2eff0000 },
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
-    [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
+    [VIRT_MEM] =                { 0x40000000, LEGACY_RAMLIMIT_BYTES },
 };
 
 /*
@@ -1483,8 +1470,9 @@ static void machvirt_init(MachineState *machine)
 
     vms->smp_cpus = smp_cpus;
 
-    if (machine->ram_size > vms->memmap[VIRT_MEM].size) {
-        error_report("mach-virt: cannot model more than %dGB RAM", RAMLIMIT_GB);
+    if (!vms->extended_memmap && machine->ram_size > LEGACY_RAMLIMIT_GB) {
+        error_report("mach-virt: cannot model more than %dGB RAM",
+                     LEGACY_RAMLIMIT_GB);
         exit(1);
     }
 
Signature · 1 line
2.20.1
#26 lore raw

Re: [Qemu-arm] [PATCH v6 10/18] hw/arm/virt: Bump the 255GB initial RAM limit

Hi Eric,

Quoted 62 lines

-----Original Message-----
From: Eric Auger [mailto:[email protected]]
Sent: 05 February 2019 17:33
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH v6 10/18] hw/arm/virt: Bump the 255GB initial RAM limit

Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel.

---
 hw/arm/virt.c | 26 +++++++-------------------
 1 file changed, 7 insertions(+), 19 deletions(-)
hw/arm/virt.c +3 -16
index b90ffc2e5d..f01886da22 100644
@@ -93,22 +93,9 @@
 
 #define PLATFORM_BUS_NUM_IRQS 64
 
-/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means
- * RAM can go up to the 256GB mark, leaving 256GB of the physical
- * address space unallocated and free for future use between 256G and 512G.
- * If we need to provide more RAM to VMs in the future then we need to:
- *  * allocate a second bank of RAM starting at 2TB and working up
- *  * fix the DT and ACPI table generation code in QEMU to correctly
- *    report two split lumps of RAM to the guest
- *  * fix KVM in the host kernel to allow guests with >40 bit address spaces
- * (We don't want to fill all the way up to 512GB with RAM because
- * we might want it for non-RAM purposes later. Conversely it seems
- * reasonable to assume that anybody configuring a VM with a quarter
- * of a terabyte of RAM will be doing it on a host with more than a
- * terabyte of physical address space.)
- */
-#define RAMLIMIT_GB 255
-#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024)
+/* Legacy RAM limit in GB (< version 4.0) */
+#define LEGACY_RAMLIMIT_GB 255
+#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
 
 /* Addresses and sizes of our components.
  * 0..128MB is space for a flash device so we can run bootrom code such as

UEFI.

hw/arm/virt.c +1 -1
@@ -149,7 +136,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_MMIO] =          { 0x10000000, 0x2eff0000 },
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
-    [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
+    [VIRT_MEM] =                { 0x40000000,
 LEGACY_RAMLIMIT_BYTES },
 };
 

/*

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1475

Quoted 8 lines
hw/arm/virt.c +1 -2
@@ -1483,8 +1470,9 @@ static void machvirt_init(MachineState *machine)
 
     vms->smp_cpus = smp_cpus;
 
-    if (machine->ram_size > vms->memmap[VIRT_MEM].size) {
-        error_report("mach-virt: cannot model more than %dGB RAM",
 RAMLIMIT_GB);
+    if (!vms->extended_memmap && machine->ram_size >
 LEGACY_RAMLIMIT_GB) {

Just hit this while testing, should this check be against LEGACY_RAMLIMIT_BYTES?

Thanks,
Shameer

Quoted 7 lines
hw/arm/virt.c +2 -0
+        error_report("mach-virt: cannot model more than %dGB RAM",
+                     LEGACY_RAMLIMIT_GB);
         exit(1);
     }

--
2.20.1

author
#27 lore raw

Re: [Qemu-arm] [PATCH v6 10/18] hw/arm/virt: Bump the 255GB initial RAM limit

Hi Shameer,

Quoted 67 lines · Shameerali Kolothum Thodi

Hi Eric,

Quoted 65 lines

-----Original Message-----
From: Eric Auger [mailto:[email protected]]
Sent: 05 February 2019 17:33
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH v6 10/18] hw/arm/virt: Bump the 255GB initial RAM limit

Now we have the extended memory map (high IO regions beyond the scalable RAM) and dynamic IPA range support at KVM/ARM level we can bump the legacy 255GB initial RAM limit. The actual maximum RAM size now depends on the physical CPU and host kernel.

---
 hw/arm/virt.c | 26 +++++++-------------------
 1 file changed, 7 insertions(+), 19 deletions(-)
hw/arm/virt.c +3 -16
index b90ffc2e5d..f01886da22 100644
@@ -93,22 +93,9 @@
 
 #define PLATFORM_BUS_NUM_IRQS 64
 
-/* RAM limit in GB. Since VIRT_MEM starts at the 1GB mark, this means
- * RAM can go up to the 256GB mark, leaving 256GB of the physical
- * address space unallocated and free for future use between 256G and 512G.
- * If we need to provide more RAM to VMs in the future then we need to:
- *  * allocate a second bank of RAM starting at 2TB and working up
- *  * fix the DT and ACPI table generation code in QEMU to correctly
- *    report two split lumps of RAM to the guest
- *  * fix KVM in the host kernel to allow guests with >40 bit address spaces
- * (We don't want to fill all the way up to 512GB with RAM because
- * we might want it for non-RAM purposes later. Conversely it seems
- * reasonable to assume that anybody configuring a VM with a quarter
- * of a terabyte of RAM will be doing it on a host with more than a
- * terabyte of physical address space.)
- */
-#define RAMLIMIT_GB 255
-#define RAMLIMIT_BYTES (RAMLIMIT_GB * 1024ULL * 1024 * 1024)
+/* Legacy RAM limit in GB (< version 4.0) */
+#define LEGACY_RAMLIMIT_GB 255
+#define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
 
 /* Addresses and sizes of our components.
  * 0..128MB is space for a flash device so we can run bootrom code such as

UEFI.

hw/arm/virt.c +1 -1
@@ -149,7 +136,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_PCIE_MMIO] =          { 0x10000000, 0x2eff0000 },
     [VIRT_PCIE_PIO] =           { 0x3eff0000, 0x00010000 },
     [VIRT_PCIE_ECAM] =          { 0x3f000000, 0x01000000 },
-    [VIRT_MEM] =                { 0x40000000, RAMLIMIT_BYTES },
+    [VIRT_MEM] =                { 0x40000000,
 LEGACY_RAMLIMIT_BYTES },
 };
 

/*

hw/arm/virt.c +0 -0
@@ -1483,8 +1470,9 @@ static void machvirt_init(MachineState *machine)
 
     vms->smp_cpus = smp_cpus;
 
Quoted 7 lines

hw/arm/virt.c static void machvirt_init(MachineState *machine) · line 1475

Quoted 5 lines
hw/arm/virt.c +1 -2
-    if (machine->ram_size > vms->memmap[VIRT_MEM].size) {
-        error_report("mach-virt: cannot model more than %dGB RAM",
 RAMLIMIT_GB);
+    if (!vms->extended_memmap && machine->ram_size >
 LEGACY_RAMLIMIT_GB) {

Just hit this while testing, should this check be against LEGACY_RAMLIMIT_BYTES?

Definitively, my mistake. Thank you for spotting that.

Thanks

Eric

Quoted 12 lines

Thanks,
Shameer

Quoted 7 lines
hw/arm/virt.c +2 -0
+        error_report("mach-virt: cannot model more than %dGB RAM",
+                     LEGACY_RAMLIMIT_GB);
         exit(1);
     }

--
2.20.1

author
#28 lore raw

PATCH v6 11/18 hw/arm/virt: Add memory hotplug framework

In reply to Eric Auger

This patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.

---
v4 -> v5:
- change in pc_dimm_pre_plug signature
- CONFIG_MEM_HOTPLUG replaced by CONFIG_MEM_DEVICE and CONFIG_DIMM

v3 -> v4:
- check the memory device is not hotplugged

v2 -> v3:
- change in pc_dimm_plug()'s signature
- add pc_dimm_pre_plug call

v1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- added pre_plug cb
- don't handle get_memory_region failure anymore

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt.c                   | 64 ++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)
default-configs/arm-softmmu.mak +2 -0
index be88870799..dc4624794f 100644
@@ -160,3 +160,5 @@ CONFIG_PCI_DESIGNWARE=y
 CONFIG_STRONGARM=y
 CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
+CONFIG_MEM_DEVICE=y
+CONFIG_DIMM=y
hw/arm/virt.c +63 -1

94 lines not on this page · raw message

Signature · 1 line
2.20.1
#29 lore raw

Re: [Qemu-arm] [PATCH v6 11/18] hw/arm/virt: Add memory hotplug framework

Quoted 80 lines · Eric Auger

This patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.

---
v4 -> v5:
- change in pc_dimm_pre_plug signature
- CONFIG_MEM_HOTPLUG replaced by CONFIG_MEM_DEVICE and CONFIG_DIMM

v3 -> v4:
- check the memory device is not hotplugged

v2 -> v3:
- change in pc_dimm_plug()'s signature
- add pc_dimm_pre_plug call

v1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- added pre_plug cb
- don't handle get_memory_region failure anymore

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt.c                   | 64 ++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)
default-configs/arm-softmmu.mak +2 -0
index be88870799..dc4624794f 100644
@@ -160,3 +160,5 @@ CONFIG_PCI_DESIGNWARE=y
 CONFIG_STRONGARM=y
 CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
+CONFIG_MEM_DEVICE=y
+CONFIG_DIMM=y
hw/arm/virt.c +25 -0
index f01886da22..783468ba77 100644
@@ -59,6 +59,8 @@
 #include "qapi/visitor.h"
 #include "standard-headers/linux/input.h"
 #include "hw/arm/smmuv3.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/mem/nvdimm.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1763,6 +1765,49 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
     return ms->possible_cpus;
 }
 
+static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                                 Error **errp)
+{
+    const bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
+
+    if (dev->hotplugged) {
+        error_setg(errp, "memory hotplug is not supported");
+    }
+
+    if (is_nvdimm) {
+        error_setg(errp, "nvdimm is not yet supported");
+        return;
+    }
+
+    pc_dimm_pre_plug(PC_DIMM(dev), MACHINE(hotplug_dev), NULL, errp);
+}
+
+static void virt_memory_plug(HotplugHandler *hotplug_dev,
+                             DeviceState *dev, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    Error *local_err = NULL;
+

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 1800

Quoted 10 lines
hw/arm/virt.c +10 -0
+    pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+
+    error_propagate(errp, local_err);
+}
+
+static void virt_memory_unplug(HotplugHandler *hotplug_dev,
+                               DeviceState *dev, Error **errp)
+{
+    pc_dimm_unplug(PC_DIMM(dev), MACHINE(hotplug_dev));
+    object_unparent(OBJECT(dev));

Please note that this will soon change with

[PATCH RFCv2 0/9] qdev: Hotplug handler chaining + virtio-pmem

What you'll have to do then is to replace the object_unparent by a

object_property_set_bool(OBJECT(dev), false, "realized", NULL);

Signature · 3 lines
Thanks,

David / dhildenb
author
#30 lore raw

Re: [Qemu-arm] [PATCH v6 11/18] hw/arm/virt: Add memory hotplug framework

Hi David,

Quoted 94 lines · David Hildenbrand
Quoted 80 lines · Eric Auger

This patch adds the the memory hot-plug/hot-unplug infrastructure in machvirt. It is still not enabled as no device memory is allocated.

---
v4 -> v5:
- change in pc_dimm_pre_plug signature
- CONFIG_MEM_HOTPLUG replaced by CONFIG_MEM_DEVICE and CONFIG_DIMM

v3 -> v4:
- check the memory device is not hotplugged

v2 -> v3:
- change in pc_dimm_plug()'s signature
- add pc_dimm_pre_plug call

v1 -> v2:
- s/virt_dimm_plug|unplug/virt_memory_plug|unplug
- s/pc_dimm_memory_plug/pc_dimm_plug
- reworded title and commit message
- added pre_plug cb
- don't handle get_memory_region failure anymore

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt.c                   | 64 ++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)
default-configs/arm-softmmu.mak +2 -0
index be88870799..dc4624794f 100644
@@ -160,3 +160,5 @@ CONFIG_PCI_DESIGNWARE=y
 CONFIG_STRONGARM=y
 CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
+CONFIG_MEM_DEVICE=y
+CONFIG_DIMM=y
hw/arm/virt.c +25 -0
index f01886da22..783468ba77 100644
@@ -59,6 +59,8 @@
 #include "qapi/visitor.h"
 #include "standard-headers/linux/input.h"
 #include "hw/arm/smmuv3.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/mem/nvdimm.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1763,6 +1765,49 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms)
     return ms->possible_cpus;
 }
 
+static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
+                                 Error **errp)
+{
+    const bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
+
+    if (dev->hotplugged) {
+        error_setg(errp, "memory hotplug is not supported");
+    }
+
+    if (is_nvdimm) {
+        error_setg(errp, "nvdimm is not yet supported");
+        return;
+    }
+
+    pc_dimm_pre_plug(PC_DIMM(dev), MACHINE(hotplug_dev), NULL, errp);
+}
+
+static void virt_memory_plug(HotplugHandler *hotplug_dev,
+                             DeviceState *dev, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    Error *local_err = NULL;
+

hw/arm/virt.c static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) · line 1800

Quoted 10 lines
hw/arm/virt.c +10 -0
+    pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+
+    error_propagate(errp, local_err);
+}
+
+static void virt_memory_unplug(HotplugHandler *hotplug_dev,
+                               DeviceState *dev, Error **errp)
+{
+    pc_dimm_unplug(PC_DIMM(dev), MACHINE(hotplug_dev));
+    object_unparent(OBJECT(dev));

Please note that this will soon change with

Quoted 5 lines

[PATCH RFCv2 0/9] qdev: Hotplug handler chaining + virtio-pmem

What you'll have to do then is to replace the object_unparent by a

object_property_set_bool(OBJECT(dev), false, "realized", NULL);

Noted.

Thanks for the heads up!

Eric

Quoted 2 lines
author
#31 lore raw

PATCH v6 12/18 hw/arm/boot: Expose the PC-DIMM nodes in the DT

In reply to Eric Auger

From: Shameer Kolothum <[email protected]>

This patch add memory nodes corresponding to PC-DIMM regions.

NV_DIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NV-DIMM at this stage.

---
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes

v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop

---
 hw/arm/boot.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/boot.c +35 -0
index 2ef367e15b..2a70e8aa82 100644
@@ -19,6 +19,7 @@
 #include "sysemu/numa.h"
 #include "hw/boards.h"
 #include "hw/loader.h"
+#include "hw/mem/memory-device.h"
 #include "elf.h"
 #include "sysemu/device_tree.h"
 #include "qemu/config-file.h"
@@ -526,6 +527,34 @@ static void fdt_add_psci_node(void *fdt)
     qemu_fdt_setprop_cell(fdt, "/psci", "migrate", migrate_fn);
 }
 
+static int fdt_add_hotpluggable_memory_nodes(void *fdt,
+                                             uint32_t acells, uint32_t scells) {
+    MemoryDeviceInfoList *info, *info_list = qmp_memory_device_list();
+    MemoryDeviceInfo *mi;
+    PCDIMMDeviceInfo *di;
+    bool is_nvdimm;
+    int ret = 0;
+
+    for (info = info_list; info != NULL; info = info->next) {
+        mi = info->value;
+        is_nvdimm = (mi->type == MEMORY_DEVICE_INFO_KIND_NVDIMM);
+        di = !is_nvdimm ? mi->u.dimm.data : mi->u.nvdimm.data;
+
+        if (is_nvdimm) {
+            ret = -ENOENT; /* NV-DIMM not yet supported */
+        } else {
+            ret = fdt_add_memory_node(fdt, acells, di->addr,
+                                      scells, di->size, di->node);
+        }
+        if (ret < 0) {
+            goto out;
+        }
+    }
+out:
+    qapi_free_MemoryDeviceInfoList(info_list);
+    return ret;
+}
+
 int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
                  hwaddr addr_limit, AddressSpace *as)
 {
@@ -621,6 +650,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         }
     }
 
+    rc = fdt_add_hotpluggable_memory_nodes(fdt, acells, scells);
+    if (rc < 0) {
+            fprintf(stderr, "couldn't add hotpluggable memory nodes\n");
+            goto fail;
+    }
+
     rc = fdt_path_offset(fdt, "/chosen");
     if (rc < 0) {
         qemu_fdt_add_subnode(fdt, "/chosen");
Signature · 1 line
2.20.1
#32 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT

Quoted 3 lines · On Tue, 5 Feb 2019 18:33:00 +0100 Eric Auger <[email protected]> wrote:

From: Shameer Kolothum <[email protected]>

This patch add memory nodes corresponding to PC-DIMM regions.

s/add/adds/ or s/This patch add/Add/

Quoted 65 lines

NV_DIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NV-DIMM at this stage.

---
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes

v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop

---
 hw/arm/boot.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/boot.c +29 -0
index 2ef367e15b..2a70e8aa82 100644
@@ -19,6 +19,7 @@
 #include "sysemu/numa.h"
 #include "hw/boards.h"
 #include "hw/loader.h"
+#include "hw/mem/memory-device.h"
 #include "elf.h"
 #include "sysemu/device_tree.h"
 #include "qemu/config-file.h"
@@ -526,6 +527,34 @@ static void fdt_add_psci_node(void *fdt)
     qemu_fdt_setprop_cell(fdt, "/psci", "migrate", migrate_fn);
 }
 
+static int fdt_add_hotpluggable_memory_nodes(void *fdt,
+                                             uint32_t acells, uint32_t scells) {
+    MemoryDeviceInfoList *info, *info_list = qmp_memory_device_list();
+    MemoryDeviceInfo *mi;
+    PCDIMMDeviceInfo *di;
+    bool is_nvdimm;
+    int ret = 0;
+
+    for (info = info_list; info != NULL; info = info->next) {
+        mi = info->value;
+        is_nvdimm = (mi->type == MEMORY_DEVICE_INFO_KIND_NVDIMM);
+        di = !is_nvdimm ? mi->u.dimm.data : mi->u.nvdimm.data;
+
+        if (is_nvdimm) {
+            ret = -ENOENT; /* NV-DIMM not yet supported */
+        } else {
+            ret = fdt_add_memory_node(fdt, acells, di->addr,
+                                      scells, di->size, di->node);
+        }
+        if (ret < 0) {
+            goto out;
+        }
+    }
+out:
+    qapi_free_MemoryDeviceInfoList(info_list);
+    return ret;
+}
+
 int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
                  hwaddr addr_limit, AddressSpace *as)
 {

hw/arm/boot.c int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, · line 655

Quoted 6 lines
hw/arm/boot.c +3 -0
@@ -621,6 +650,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         }
     }
 
+    rc = fdt_add_hotpluggable_memory_nodes(fdt, acells, scells);
+    if (rc < 0) {
+            fprintf(stderr, "couldn't add hotpluggable memory nodes\n");

error message is rather vague, user using nvdimms + pc-dimms on CLI won't have a clue that the former is not supported.
Suggest pass in error_fatal as argument and report more specific error from fdt_add_hotpluggable_memory_nodes()

does this run on reboot?

Quoted 6 lines
hw/arm/boot.c +3 -0
+            goto fail;
+    }
+
     rc = fdt_path_offset(fdt, "/chosen");
     if (rc < 0) {
         qemu_fdt_add_subnode(fdt, "/chosen");
author
#33 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT

Hi Igor,

Quoted 84 lines · Igor Mammedov
Quoted 3 lines · On Tue, 5 Feb 2019 18:33:00 +0100 Eric Auger <[email protected]> wrote:

From: Shameer Kolothum <[email protected]>

This patch add memory nodes corresponding to PC-DIMM regions.

s/add/adds/ or s/This patch add/Add/

Quoted 65 lines

NV_DIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NV-DIMM at this stage.

---
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes

v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop

---
 hw/arm/boot.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/boot.c +29 -0
index 2ef367e15b..2a70e8aa82 100644
@@ -19,6 +19,7 @@
 #include "sysemu/numa.h"
 #include "hw/boards.h"
 #include "hw/loader.h"
+#include "hw/mem/memory-device.h"
 #include "elf.h"
 #include "sysemu/device_tree.h"
 #include "qemu/config-file.h"
@@ -526,6 +527,34 @@ static void fdt_add_psci_node(void *fdt)
     qemu_fdt_setprop_cell(fdt, "/psci", "migrate", migrate_fn);
 }
 
+static int fdt_add_hotpluggable_memory_nodes(void *fdt,
+                                             uint32_t acells, uint32_t scells) {
+    MemoryDeviceInfoList *info, *info_list = qmp_memory_device_list();
+    MemoryDeviceInfo *mi;
+    PCDIMMDeviceInfo *di;
+    bool is_nvdimm;
+    int ret = 0;
+
+    for (info = info_list; info != NULL; info = info->next) {
+        mi = info->value;
+        is_nvdimm = (mi->type == MEMORY_DEVICE_INFO_KIND_NVDIMM);
+        di = !is_nvdimm ? mi->u.dimm.data : mi->u.nvdimm.data;
+
+        if (is_nvdimm) {
+            ret = -ENOENT; /* NV-DIMM not yet supported */
+        } else {
+            ret = fdt_add_memory_node(fdt, acells, di->addr,
+                                      scells, di->size, di->node);
+        }
+        if (ret < 0) {
+            goto out;
+        }
+    }
+out:
+    qapi_free_MemoryDeviceInfoList(info_list);
+    return ret;
+}
+
 int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
                  hwaddr addr_limit, AddressSpace *as)
 {

hw/arm/boot.c int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, · line 655

Quoted 6 lines
hw/arm/boot.c +3 -0
@@ -621,6 +650,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         }
     }
 
+    rc = fdt_add_hotpluggable_memory_nodes(fdt, acells, scells);
+    if (rc < 0) {
+            fprintf(stderr, "couldn't add hotpluggable memory nodes\n");

error message is rather vague, user using nvdimms + pc-dimms on CLI won't have a clue that the former is not supported.
Suggest pass in error_fatal as argument and report more specific error from fdt_add_hotpluggable_memory_nodes()

Quoted 1 line

does this run on reboot?

Yes after a QMP system_reset I can see the DIMMS on guest in /proc/meminfo

Thanks

Eric

Quoted 8 lines
Quoted 7 lines
hw/arm/boot.c +3 -0
+            goto fail;
+    }
+
     rc = fdt_path_offset(fdt, "/chosen");
     if (rc < 0) {
         qemu_fdt_add_subnode(fdt, "/chosen");
#34 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 12/18] hw/arm/boot: Expose the PC-DIMM nodes in the DT

Quoted 87 lines · On Wed, 20 Feb 2019 16:30:13 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 84 lines · Igor Mammedov
Quoted 3 lines · On Tue, 5 Feb 2019 18:33:00 +0100 Eric Auger <[email protected]> wrote:

From: Shameer Kolothum <[email protected]>

This patch add memory nodes corresponding to PC-DIMM regions.

s/add/adds/ or s/This patch add/Add/

Quoted 65 lines

NV_DIMM and ACPI_NVDIMM configs are not yet set for ARM so we don't need to care about NV-DIMM at this stage.

---
v3 -> v4:
- git rid of @base and @len in fdt_add_hotpluggable_memory_nodes

v1 -> v2:
- added qapi_free_MemoryDeviceInfoList and simplify the loop

---
 hw/arm/boot.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
hw/arm/boot.c +29 -0
index 2ef367e15b..2a70e8aa82 100644
@@ -19,6 +19,7 @@
 #include "sysemu/numa.h"
 #include "hw/boards.h"
 #include "hw/loader.h"
+#include "hw/mem/memory-device.h"
 #include "elf.h"
 #include "sysemu/device_tree.h"
 #include "qemu/config-file.h"
@@ -526,6 +527,34 @@ static void fdt_add_psci_node(void *fdt)
     qemu_fdt_setprop_cell(fdt, "/psci", "migrate", migrate_fn);
 }
 
+static int fdt_add_hotpluggable_memory_nodes(void *fdt,
+                                             uint32_t acells, uint32_t scells) {
+    MemoryDeviceInfoList *info, *info_list = qmp_memory_device_list();
+    MemoryDeviceInfo *mi;
+    PCDIMMDeviceInfo *di;
+    bool is_nvdimm;
+    int ret = 0;
+
+    for (info = info_list; info != NULL; info = info->next) {
+        mi = info->value;
+        is_nvdimm = (mi->type == MEMORY_DEVICE_INFO_KIND_NVDIMM);
+        di = !is_nvdimm ? mi->u.dimm.data : mi->u.nvdimm.data;
+
+        if (is_nvdimm) {
+            ret = -ENOENT; /* NV-DIMM not yet supported */
+        } else {
+            ret = fdt_add_memory_node(fdt, acells, di->addr,
+                                      scells, di->size, di->node);
+        }
+        if (ret < 0) {
+            goto out;
+        }
+    }
+out:
+    qapi_free_MemoryDeviceInfoList(info_list);
+    return ret;
+}
+
 int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
                  hwaddr addr_limit, AddressSpace *as)
 {

hw/arm/boot.c int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, · line 655

Quoted 6 lines
hw/arm/boot.c +3 -0
@@ -621,6 +650,12 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         }
     }
 
+    rc = fdt_add_hotpluggable_memory_nodes(fdt, acells, scells);
+    if (rc < 0) {
+            fprintf(stderr, "couldn't add hotpluggable memory nodes\n");  

error message is rather vague, user using nvdimms + pc-dimms on CLI won't have a clue that the former is not supported.
Suggest pass in error_fatal as argument and report more specific error from fdt_add_hotpluggable_memory_nodes()

Quoted 3 lines
Quoted 1 line

does this run on reboot?

Yes after a QMP system_reset I can see the DIMMS on guest in /proc/meminfo

it siims that dimms come from ACPI and not DTB.

My worry was that arm_load_dtb() always end up with exit(1) on failure but it seems that it's not called on reboot so it should be fine.

Quoted 12 lines

Thanks

Eric

Quoted 8 lines
Quoted 7 lines
hw/arm/boot.c +3 -0
+            goto fail;
+    }
+
     rc = fdt_path_offset(fdt, "/chosen");
     if (rc < 0) {
         qemu_fdt_add_subnode(fdt, "/chosen");  
author
#35 lore raw

PATCH v6 13/18 hw/arm/virt-acpi-build: Add PC-DIMM in SRAT

In reply to Eric Auger

From: Shameer Kolothum <[email protected]>

Generate Memory Affinity Structures for PC-DIMM ranges.

---
v5 -> v6:
- fix mingw compil issue

v4 -> v5:
- Align to x86 code and especially
"pc: acpi: revert back to 1 SRAT entry for hotpluggable area"

v3 -> v4:
- do not use vms->bootinfo.device_memory_start/device_memory_size anymore

v1 -> v2:
- build_srat_hotpluggable_memory movedc to aml-build

---
 hw/arm/virt-acpi-build.c | 9 +++++++++
 1 file changed, 9 insertions(+)
hw/arm/virt-acpi-build.c +9 -0
index 829d2f0035..781eafaf5e 100644
@@ -516,6 +516,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     int i, srat_start;
     uint64_t mem_base;
     MachineClass *mc = MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids(MACHINE(vms));
 
     srat_start = table_data->len;
@@ -541,6 +542,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
     }
 
+    if (ms->device_memory) {
+        numamem = acpi_data_push(table_data, sizeof *numamem);
+        build_srat_memory(numamem, ms->device_memory->base,
+                          memory_region_size(&ms->device_memory->mr),
+                          nb_numa_nodes - 1,
+                          MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);
+    }
+
     build_header(linker, table_data, (void *)(table_data->data + srat_start),
                  "SRAT", table_data->len - srat_start, 3, NULL, NULL);
 }
Signature · 1 line
2.20.1
Reviewed-by on v6 13/18
#36 lore raw

Re: [Qemu-arm] [PATCH v6 13/18] hw/arm/virt-acpi-build: Add PC-DIMM in SRAT

Quoted 6 lines · On Tue, 5 Feb 2019 18:33:01 +0100 Eric Auger <[email protected]> wrote:

From: Shameer Kolothum <[email protected]>

Generate Memory Affinity Structures for PC-DIMM ranges.

Quoted 45 lines

---
v5 -> v6:
- fix mingw compil issue

v4 -> v5:
- Align to x86 code and especially
"pc: acpi: revert back to 1 SRAT entry for hotpluggable area"

v3 -> v4:
- do not use vms->bootinfo.device_memory_start/device_memory_size anymore

v1 -> v2:
- build_srat_hotpluggable_memory movedc to aml-build

---
 hw/arm/virt-acpi-build.c | 9 +++++++++
 1 file changed, 9 insertions(+)
hw/arm/virt-acpi-build.c +9 -0
index 829d2f0035..781eafaf5e 100644
@@ -516,6 +516,7 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     int i, srat_start;
     uint64_t mem_base;
     MachineClass *mc = MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     const CPUArchIdList *cpu_list = mc->possible_cpu_arch_ids(MACHINE(vms));
 
     srat_start = table_data->len;
@@ -541,6 +542,14 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         }
     }
 
+    if (ms->device_memory) {
+        numamem = acpi_data_push(table_data, sizeof *numamem);
+        build_srat_memory(numamem, ms->device_memory->base,
+                          memory_region_size(&ms->device_memory->mr),
+                          nb_numa_nodes - 1,
+                          MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);
+    }
+
     build_header(linker, table_data, (void *)(table_data->data + srat_start),
                  "SRAT", table_data->len - srat_start, 3, NULL, NULL);
 }
author
#37 lore raw

PATCH v6 14/18 hw/arm/virt: Allocate device_memory

In reply to Eric Auger

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +36 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;
+    uint64_t align = GiB;
+
+    if (!device_memory_size) {
+        return;
+    }
+
+    if (ms->ram_slots > ACPI_MAX_RAM_SLOTS) {
+        error_report("unsupported number of memory slots: %"PRIu64,
+                     ms->ram_slots);
+        exit(EXIT_FAILURE);
+    }
+
+    if (QEMU_ALIGN_UP(ms->maxram_size, align) != ms->maxram_size) {
+        error_report("maximum memory size must be aligned to multiple of 0x%"
+                     PRIx64, align);
+        exit(EXIT_FAILURE);
+    }
+
+    ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
+    ms->device_memory->base = QEMU_ALIGN_UP(GiB + ms->ram_size, GiB);
+
+    memory_region_init(&ms->device_memory->mr, OBJECT(vms),
+                       "device-memory", device_memory_size);
+    memory_region_add_subregion(sysmem, ms->device_memory->base,
+                                &ms->device_memory->mr);
+}
+
 static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 {
     const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1569,6 +1601,10 @@ static void machvirt_init(MachineState *machine)
                                          machine->ram_size);
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base, ram);
 
+    if (vms->extended_memmap) {
+        create_device_memory(vms, sysmem);
+    }
+
     create_flash(vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
 
     create_gic(vms, pic);
Signature · 1 line
2.20.1
#38 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Quoted 30 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 7 lines
hw/arm/virt.c +4 -0
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

Quoted 10 lines
hw/arm/virt.c +10 -0
+    uint64_t align = GiB;
+
+    if (!device_memory_size) {
+        return;
+    }
+
+    if (ms->ram_slots > ACPI_MAX_RAM_SLOTS) {
+        error_report("unsupported number of memory slots: %"PRIu64,
+                     ms->ram_slots);
+        exit(EXIT_FAILURE);

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1287

Quoted 10 lines
hw/arm/virt.c +10 -0
+    }
+
+    if (QEMU_ALIGN_UP(ms->maxram_size, align) != ms->maxram_size) {
+        error_report("maximum memory size must be aligned to multiple of 0x%"
+                     PRIx64, align);
+        exit(EXIT_FAILURE);
+    }
+
+    ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
+    ms->device_memory->base = QEMU_ALIGN_UP(GiB + ms->ram_size, GiB);

^^^ where does this come from?

Quoted 21 lines
hw/arm/virt.c +11 -0
+
+    memory_region_init(&ms->device_memory->mr, OBJECT(vms),
+                       "device-memory", device_memory_size);
+    memory_region_add_subregion(sysmem, ms->device_memory->base,
+                                &ms->device_memory->mr);
+}
+
 static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 {
     const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1569,6 +1601,10 @@ static void machvirt_init(MachineState *machine)
                                          machine->ram_size);
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base, ram);
 
+    if (vms->extended_memmap) {
+        create_device_memory(vms, sysmem);
+    }
+
     create_flash(vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
 
     create_gic(vms, pic);
author
#39 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

Quoted 14 lines
Quoted 14 lines
hw/arm/virt.c +13 -0
+    uint64_t align = GiB;
+
+    if (!device_memory_size) {
+        return;
+    }
+
+    if (ms->ram_slots > ACPI_MAX_RAM_SLOTS) {
+        error_report("unsupported number of memory slots: %"PRIu64,
+                     ms->ram_slots);
+        exit(EXIT_FAILURE);
+    }
+
+    if (QEMU_ALIGN_UP(ms->maxram_size, align) != ms->maxram_size) {
Quoted 8 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1287

Quoted 7 lines
hw/arm/virt.c +7 -0
+        error_report("maximum memory size must be aligned to multiple of 0x%"
+                     PRIx64, align);
+        exit(EXIT_FAILURE);
+    }
+
+    ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
+    ms->device_memory->base = QEMU_ALIGN_UP(GiB + ms->ram_size, GiB);

^^^ where does this come from?

OK, introduced RAMBASE macro

Thanks

Eric

Quoted 24 lines
Quoted 23 lines
hw/arm/virt.c +11 -0
+
+    memory_region_init(&ms->device_memory->mr, OBJECT(vms),
+                       "device-memory", device_memory_size);
+    memory_region_add_subregion(sysmem, ms->device_memory->base,
+                                &ms->device_memory->mr);
+}
+
 static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 {
     const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1569,6 +1601,10 @@ static void machvirt_init(MachineState *machine)
                                          machine->ram_size);
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base, ram);
 
+    if (vms->extended_memmap) {
+        create_device_memory(vms, sysmem);
+    }
+
     create_flash(vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
 
     create_gic(vms, pic);
#40 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Quoted 45 lines · Auger Eric

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

Quoted 5 lines

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

This is to be able to potentially align each slot as far as I know, so the "memory device address space" cannot that easily be fragmented.

E.g. Linux requires a certain alignment to make full use of a DIMM.

Quoted 5 lines

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

I guess the alignment check is only done because for that target, anything having sub-page granularity cannot be used either way.

Signature · 3 lines
Thanks,

David / dhildenb
#41 lore raw

Re: [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

In reply to Auger Eric
Quoted 51 lines · On Tue, 19 Feb 2019 16:53:22 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;  

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

Quoted 4 lines

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

See commit 085f8e88b for explanation,
What we are basically are doing there is sizing hotpluggbale address space to allow max possible huge page aligned DIMM to be successfully plugged in even if address space if fragmented.

Quoted 53 lines
Quoted 15 lines
Quoted 15 lines
hw/arm/virt.c +13 -0
+    uint64_t align = GiB;
+
+    if (!device_memory_size) {
+        return;
+    }
+
+    if (ms->ram_slots > ACPI_MAX_RAM_SLOTS) {
+        error_report("unsupported number of memory slots: %"PRIu64,
+                     ms->ram_slots);
+        exit(EXIT_FAILURE);
+    }
+
+    if (QEMU_ALIGN_UP(ms->maxram_size, align) != ms->maxram_size) {
Quoted 8 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1287

Quoted 7 lines
hw/arm/virt.c +7 -0
+        error_report("maximum memory size must be aligned to multiple of 0x%"
+                     PRIx64, align);
+        exit(EXIT_FAILURE);
+    }
+
+    ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
+    ms->device_memory->base = QEMU_ALIGN_UP(GiB + ms->ram_size, GiB);  

^^^ where does this come from?

OK, introduced RAMBASE macro

Thanks

Eric

Quoted 24 lines
Quoted 23 lines
hw/arm/virt.c +11 -0
+
+    memory_region_init(&ms->device_memory->mr, OBJECT(vms),
+                       "device-memory", device_memory_size);
+    memory_region_add_subregion(sysmem, ms->device_memory->base,
+                                &ms->device_memory->mr);
+}
+
 static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 {
     const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1569,6 +1601,10 @@ static void machvirt_init(MachineState *machine)
                                          machine->ram_size);
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base, ram);
 
+    if (vms->extended_memmap) {
+        create_device_memory(vms, sysmem);
+    }
+
     create_flash(vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
 
     create_gic(vms, pic);  
author
#42 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Hi Igor,

Quoted 54 lines · Igor Mammedov
Quoted 51 lines · On Tue, 19 Feb 2019 16:53:22 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;  

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

Quoted 8 lines
Quoted 4 lines

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

See commit 085f8e88b for explanation,
What we are basically are doing there is sizing hotpluggbale address space to allow max possible huge page aligned DIMM to be successfully plugged in even if address space if fragmented.

In v7, I also added ram_slots * GiB to (maxram_size - ram_size).

Thanks

Eric

Quoted 56 lines
Quoted 54 lines
Quoted 16 lines
Quoted 16 lines
hw/arm/virt.c +13 -0
+    uint64_t align = GiB;
+
+    if (!device_memory_size) {
+        return;
+    }
+
+    if (ms->ram_slots > ACPI_MAX_RAM_SLOTS) {
+        error_report("unsupported number of memory slots: %"PRIu64,
+                     ms->ram_slots);
+        exit(EXIT_FAILURE);
+    }
+
+    if (QEMU_ALIGN_UP(ms->maxram_size, align) != ms->maxram_size) {
Quoted 8 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1287

Quoted 7 lines
hw/arm/virt.c +7 -0
+        error_report("maximum memory size must be aligned to multiple of 0x%"
+                     PRIx64, align);
+        exit(EXIT_FAILURE);
+    }
+
+    ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
+    ms->device_memory->base = QEMU_ALIGN_UP(GiB + ms->ram_size, GiB);  

^^^ where does this come from?

OK, introduced RAMBASE macro

Thanks

Eric

Quoted 24 lines
Quoted 23 lines
hw/arm/virt.c +11 -0
+
+    memory_region_init(&ms->device_memory->mr, OBJECT(vms),
+                       "device-memory", device_memory_size);
+    memory_region_add_subregion(sysmem, ms->device_memory->base,
+                                &ms->device_memory->mr);
+}
+
 static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 {
     const VirtMachineState *board = container_of(binfo, VirtMachineState,
@@ -1569,6 +1601,10 @@ static void machvirt_init(MachineState *machine)
                                          machine->ram_size);
     memory_region_add_subregion(sysmem, vms->memmap[VIRT_MEM].base, ram);
 
+    if (vms->extended_memmap) {
+        create_device_memory(vms, sysmem);
+    }
+
     create_flash(vms, sysmem, secure_sysmem ? secure_sysmem : sysmem);
 
     create_gic(vms, pic);  
#43 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Quoted 61 lines · Auger Eric

Hi Igor,

Quoted 58 lines · Igor Mammedov
Quoted 55 lines · On Tue, 19 Feb 2019 16:53:22 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;  

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

Quoted 6 lines
Quoted 4 lines

See commit 085f8e88b for explanation,
What we are basically are doing there is sizing hotpluggbale address space to allow max possible huge page aligned DIMM to be successfully plugged in even if address space if fragmented.

In v7, I also added ram_slots * GiB to (maxram_size - ram_size).

Depending on the way the system handles it, this might be confusing for the end user and has to be documented somewhere.

E.g. if there are certain memory limits (say 2TB) and the user specifies something like "maxmem=2TB,slots=20" it might be confusing if he gets an error like "more than 2TB are not supported".

Quoted 3 lines

Thanks

Eric

Signature · 3 lines
Thanks,

David / dhildenb
author
#44 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 14/18] hw/arm/virt: Allocate device_memory

Hi David,

Quoted 69 lines · David Hildenbrand
Quoted 61 lines · Auger Eric

Hi Igor,

Quoted 58 lines · Igor Mammedov
Quoted 55 lines · On Tue, 19 Feb 2019 16:53:22 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 36 lines · Igor Mammedov
Quoted 33 lines · On Tue, 5 Feb 2019 18:33:02 +0100 Eric Auger <[email protected]> wrote:

The device memory region is located after the initial RAM. its start/size are 1GB aligned.

---
v4 -> v5:
- device memory set after the initial RAM

v3 -> v4:
- remove bootinfo.device_memory_start/device_memory_size
- rename VIRT_HOTPLUG_MEM into VIRT_DEVICE_MEM

---
 hw/arm/virt.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
hw/arm/virt.c +1 -0
index 783468ba77..b683902991 100644
@@ -61,6 +61,7 @@
 #include "hw/arm/smmuv3.h"
 #include "hw/mem/pc-dimm.h"
 #include "hw/mem/nvdimm.h"
+#include "hw/acpi/acpi.h"
 
 #define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
     static void virt_##major##_##minor##_class_init(ObjectClass *oc, \
@@ -1260,6 +1261,37 @@ static void create_secure_ram(VirtMachineState *vms,
     g_free(nodename);
 }
 
Quoted 6 lines

hw/arm/virt.c static void create_secure_ram(VirtMachineState *vms, · line 1267

Quoted 4 lines
hw/arm/virt.c +4 -0
+static void create_device_memory(VirtMachineState *vms, MemoryRegion *sysmem)
+{
+    MachineState *ms = MACHINE(vms);
+    uint64_t device_memory_size = ms->maxram_size - ms->ram_size;  

should size it with 1Gb alignment per slot from the start (to avoid x86 mistakes), see enforce_aligned_dimm usage and associated commit for more details

I don't understand the computation done in pc machine. eventually we are likely to have more device memory than requested by the user. Why don't we check (machine->maxram_size - machine->ram_size) >= machine->ram_slots * GiB
instead of adding 1GiB/slot to the initial user requirements?

Also machine->maxram_size - machine->ram_size is checked to be aligned with TARGET_PAGE_SIZE. Is TARGET_PAGE_SIZE representative of the guest PAGE in accelerated mode? Is it valid ro require an alignment on 1GB boundary as I do in this patch?

Quoted 6 lines
Quoted 4 lines

See commit 085f8e88b for explanation,
What we are basically are doing there is sizing hotpluggbale address space to allow max possible huge page aligned DIMM to be successfully plugged in even if address space if fragmented.

In v7, I also added ram_slots * GiB to (maxram_size - ram_size).

Quoted 6 lines

Depending on the way the system handles it, this might be confusing for the end user and has to be documented somewhere.

E.g. if there are certain memory limits (say 2TB) and the user specifies something like "maxmem=2TB,slots=20" it might be confusing if he gets an error like "more than 2TB are not supported".

I Agree. On ARM we also intend to put high IO regions above the RAM so if we overshoot the host limit, we warn the user the memory map requires more PA bits than the host can support and the end user is invited to lower maxmem/slots.

Thanks

Eric

Quoted 4 lines
Quoted 4 lines

Thanks

Eric

author
#45 lore raw

PATCH v6 15/18 nvdimm: use configurable ACPI IO base and size

In reply to Eric Auger

From: Kwangwoo Lee <[email protected]>

This patch uses configurable IO base and size to create NPIO AML for ACPI NFIT. Since a different architecture like AArch64 does not use port-mapped IO, a configurable IO base is required to create correct mapping of ACPI IO address and size.

---

v2 -> v3:
- s/size/len in pc_piix.c and pc_q35.c

---
 hw/acpi/nvdimm.c        | 28 +++++++++++++++++++---------
 hw/i386/pc_piix.c       |  8 +++++++-
 hw/i386/pc_q35.c        |  8 +++++++-
 include/hw/mem/nvdimm.h | 12 ++++++++++++
 4 files changed, 45 insertions(+), 11 deletions(-)
hw/acpi/nvdimm.c +19 -9
index e53b2cb681..da68de5535 100644
@@ -929,8 +929,8 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,
                             FWCfgState *fw_cfg, Object *owner)
 {
     memory_region_init_io(&state->io_mr, owner, &nvdimm_dsm_ops, state,
-                          "nvdimm-acpi-io", NVDIMM_ACPI_IO_LEN);
-    memory_region_add_subregion(io, NVDIMM_ACPI_IO_BASE, &state->io_mr);
+                          "nvdimm-acpi-io", state->dsm_io.len);
+    memory_region_add_subregion(io, state->dsm_io.base, &state->io_mr);
 
     state->dsm_mem = g_array_new(false, true /* clear */, 1);
     acpi_data_push(state->dsm_mem, sizeof(NvdimmDsmIn));
@@ -959,12 +959,14 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,
 
 #define NVDIMM_QEMU_RSVD_UUID   "648B9CF2-CDA1-4312-8AD9-49C4AF32BD62"
 
-static void nvdimm_build_common_dsm(Aml *dev)
+static void nvdimm_build_common_dsm(Aml *dev,
+                                    AcpiNVDIMMState *acpi_nvdimm_state)
 {
     Aml *method, *ifctx, *function, *handle, *uuid, *dsm_mem, *elsectx2;
     Aml *elsectx, *unsupport, *unpatched, *expected_uuid, *uuid_invalid;
     Aml *pckg, *pckg_index, *pckg_buf, *field, *dsm_out_buf, *dsm_out_buf_size;
     uint8_t byte_list[1];
+    AmlRegionSpace rs;
 
     method = aml_method(NVDIMM_COMMON_DSM, 5, AML_SERIALIZED);
     uuid = aml_arg(0);
@@ -975,9 +977,16 @@ static void nvdimm_build_common_dsm(Aml *dev)
 
     aml_append(method, aml_store(aml_name(NVDIMM_ACPI_MEM_ADDR), dsm_mem));
 
+    if (acpi_nvdimm_state->dsm_io.type == NVDIMM_ACPI_IO_PORT) {
+        rs = AML_SYSTEM_IO;
+    } else {
+        rs = AML_SYSTEM_MEMORY;
+    }
+
     /* map DSM memory and IO into ACPI namespace. */
-    aml_append(method, aml_operation_region(NVDIMM_DSM_IOPORT, AML_SYSTEM_IO,
-               aml_int(NVDIMM_ACPI_IO_BASE), NVDIMM_ACPI_IO_LEN));
+    aml_append(method, aml_operation_region(NVDIMM_DSM_IOPORT, rs,
+               aml_int(acpi_nvdimm_state->dsm_io.base),
+               acpi_nvdimm_state->dsm_io.len));
     aml_append(method, aml_operation_region(NVDIMM_DSM_MEMORY,
                AML_SYSTEM_MEMORY, dsm_mem, sizeof(NvdimmDsmIn)));
 
@@ -1260,7 +1269,8 @@ static void nvdimm_build_nvdimm_devices(Aml *root_dev, uint32_t ram_slots)
 }
 
 static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
-                              BIOSLinker *linker, GArray *dsm_dma_arrea,
+                              BIOSLinker *linker,
+                              AcpiNVDIMMState *acpi_nvdimm_state,
                               uint32_t ram_slots)
 {
     Aml *ssdt, *sb_scope, *dev;
@@ -1288,7 +1298,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
      */
     aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0012")));
 
-    nvdimm_build_common_dsm(dev);
+    nvdimm_build_common_dsm(dev, acpi_nvdimm_state);
 
     /* 0 is reserved for root device. */
     nvdimm_build_device_dsm(dev, 0);
@@ -1307,7 +1317,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
                                                NVDIMM_ACPI_MEM_ADDR);
 
     bios_linker_loader_alloc(linker,
-                             NVDIMM_DSM_MEM_FILE, dsm_dma_arrea,
+                             NVDIMM_DSM_MEM_FILE, acpi_nvdimm_state->dsm_mem,
                              sizeof(NvdimmDsmIn), false /* high memory */);
     bios_linker_loader_add_pointer(linker,
         ACPI_BUILD_TABLE_FILE, mem_addr_offset, sizeof(uint32_t),
@@ -1329,7 +1339,7 @@ void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data,
         return;
     }
 
-    nvdimm_build_ssdt(table_offsets, table_data, linker, state->dsm_mem,
+    nvdimm_build_ssdt(table_offsets, table_data, linker, state,
                       ram_slots);
 
     device_list = nvdimm_get_device_list();
hw/i386/pc_piix.c +7 -1
index 63c84e3827..d9c81c9aa6 100644
@@ -298,7 +298,13 @@ static void pc_init1(MachineState *machine,
     }
 
     if (pcms->acpi_nvdimm_state.is_enabled) {
-        nvdimm_init_acpi_state(&pcms->acpi_nvdimm_state, system_io,
+        AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_PORT;
+        acpi_nvdimm_state->dsm_io.base = NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, system_io,
                                pcms->fw_cfg, OBJECT(pcms));
     }
 }
hw/i386/pc_q35.c +7 -1
index b7b7959934..1110a26e34 100644
@@ -330,7 +330,13 @@ static void pc_q35_init(MachineState *machine)
     pc_nic_init(pcmc, isa_bus, host_bus);
 
     if (pcms->acpi_nvdimm_state.is_enabled) {
-        nvdimm_init_acpi_state(&pcms->acpi_nvdimm_state, system_io,
+        AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_PORT;
+        acpi_nvdimm_state->dsm_io.base = NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, system_io,
                                pcms->fw_cfg, OBJECT(pcms));
     }
 }
include/hw/mem/nvdimm.h +12 -0
index c5c9b3c7f8..af8a5fd034 100644
@@ -123,6 +123,17 @@ struct NvdimmFitBuffer {
 };
 typedef struct NvdimmFitBuffer NvdimmFitBuffer;
 
+typedef enum {
+    NVDIMM_ACPI_IO_PORT,
+    NVDIMM_ACPI_IO_MEMORY,
+} AcpiNVDIMMIOType;
+
+typedef struct AcpiNVDIMMIOEntry {
+    AcpiNVDIMMIOType type;
+    hwaddr base;
+    hwaddr len;
+} AcpiNVDIMMIOEntry;
+
 struct AcpiNVDIMMState {
     /* detect if NVDIMM support is enabled. */
     bool is_enabled;
@@ -140,6 +151,7 @@ struct AcpiNVDIMMState {
      */
     int32_t persistence;
     char    *persistence_string;
+    AcpiNVDIMMIOEntry dsm_io;
 };
 typedef struct AcpiNVDIMMState AcpiNVDIMMState;
 
Signature · 1 line
2.20.1
#46 lore raw

Re: [Qemu-arm] [PATCH v6 15/18] nvdimm: use configurable ACPI IO base and size

Quoted 113 lines · On Tue, 5 Feb 2019 18:33:03 +0100 Eric Auger <[email protected]> wrote:

From: Kwangwoo Lee <[email protected]>

This patch uses configurable IO base and size to create NPIO AML for ACPI NFIT. Since a different architecture like AArch64 does not use port-mapped IO, a configurable IO base is required to create correct mapping of ACPI IO address and size.

---

v2 -> v3:
- s/size/len in pc_piix.c and pc_q35.c

---
 hw/acpi/nvdimm.c        | 28 +++++++++++++++++++---------
 hw/i386/pc_piix.c       |  8 +++++++-
 hw/i386/pc_q35.c        |  8 +++++++-
 include/hw/mem/nvdimm.h | 12 ++++++++++++
 4 files changed, 45 insertions(+), 11 deletions(-)
hw/acpi/nvdimm.c +19 -9
index e53b2cb681..da68de5535 100644
@@ -929,8 +929,8 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,
                             FWCfgState *fw_cfg, Object *owner)
 {
     memory_region_init_io(&state->io_mr, owner, &nvdimm_dsm_ops, state,
-                          "nvdimm-acpi-io", NVDIMM_ACPI_IO_LEN);
-    memory_region_add_subregion(io, NVDIMM_ACPI_IO_BASE, &state->io_mr);
+                          "nvdimm-acpi-io", state->dsm_io.len);
+    memory_region_add_subregion(io, state->dsm_io.base, &state->io_mr);
 
     state->dsm_mem = g_array_new(false, true /* clear */, 1);
     acpi_data_push(state->dsm_mem, sizeof(NvdimmDsmIn));
@@ -959,12 +959,14 @@ void nvdimm_init_acpi_state(AcpiNVDIMMState *state, MemoryRegion *io,
 
 #define NVDIMM_QEMU_RSVD_UUID   "648B9CF2-CDA1-4312-8AD9-49C4AF32BD62"
 
-static void nvdimm_build_common_dsm(Aml *dev)
+static void nvdimm_build_common_dsm(Aml *dev,
+                                    AcpiNVDIMMState *acpi_nvdimm_state)
 {
     Aml *method, *ifctx, *function, *handle, *uuid, *dsm_mem, *elsectx2;
     Aml *elsectx, *unsupport, *unpatched, *expected_uuid, *uuid_invalid;
     Aml *pckg, *pckg_index, *pckg_buf, *field, *dsm_out_buf, *dsm_out_buf_size;
     uint8_t byte_list[1];
+    AmlRegionSpace rs;
 
     method = aml_method(NVDIMM_COMMON_DSM, 5, AML_SERIALIZED);
     uuid = aml_arg(0);
@@ -975,9 +977,16 @@ static void nvdimm_build_common_dsm(Aml *dev)
 
     aml_append(method, aml_store(aml_name(NVDIMM_ACPI_MEM_ADDR), dsm_mem));
 
+    if (acpi_nvdimm_state->dsm_io.type == NVDIMM_ACPI_IO_PORT) {
+        rs = AML_SYSTEM_IO;
+    } else {
+        rs = AML_SYSTEM_MEMORY;
+    }
+
     /* map DSM memory and IO into ACPI namespace. */
-    aml_append(method, aml_operation_region(NVDIMM_DSM_IOPORT, AML_SYSTEM_IO,
-               aml_int(NVDIMM_ACPI_IO_BASE), NVDIMM_ACPI_IO_LEN));
+    aml_append(method, aml_operation_region(NVDIMM_DSM_IOPORT, rs,
+               aml_int(acpi_nvdimm_state->dsm_io.base),
+               acpi_nvdimm_state->dsm_io.len));
     aml_append(method, aml_operation_region(NVDIMM_DSM_MEMORY,
                AML_SYSTEM_MEMORY, dsm_mem, sizeof(NvdimmDsmIn)));
 
@@ -1260,7 +1269,8 @@ static void nvdimm_build_nvdimm_devices(Aml *root_dev, uint32_t ram_slots)
 }
 
 static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
-                              BIOSLinker *linker, GArray *dsm_dma_arrea,
+                              BIOSLinker *linker,
+                              AcpiNVDIMMState *acpi_nvdimm_state,
                               uint32_t ram_slots)
 {
     Aml *ssdt, *sb_scope, *dev;
@@ -1288,7 +1298,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
      */
     aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0012")));
 
-    nvdimm_build_common_dsm(dev);
+    nvdimm_build_common_dsm(dev, acpi_nvdimm_state);
 
     /* 0 is reserved for root device. */
     nvdimm_build_device_dsm(dev, 0);
@@ -1307,7 +1317,7 @@ static void nvdimm_build_ssdt(GArray *table_offsets, GArray *table_data,
                                                NVDIMM_ACPI_MEM_ADDR);
 
     bios_linker_loader_alloc(linker,
-                             NVDIMM_DSM_MEM_FILE, dsm_dma_arrea,
+                             NVDIMM_DSM_MEM_FILE, acpi_nvdimm_state->dsm_mem,
                              sizeof(NvdimmDsmIn), false /* high memory */);
     bios_linker_loader_add_pointer(linker,
         ACPI_BUILD_TABLE_FILE, mem_addr_offset, sizeof(uint32_t),
@@ -1329,7 +1339,7 @@ void nvdimm_build_acpi(GArray *table_offsets, GArray *table_data,
         return;
     }
 
-    nvdimm_build_ssdt(table_offsets, table_data, linker, state->dsm_mem,
+    nvdimm_build_ssdt(table_offsets, table_data, linker, state,
                       ram_slots);
 
     device_list = nvdimm_get_device_list();
hw/i386/pc_piix.c +0 -0
index 63c84e3827..d9c81c9aa6 100644

hw/i386/pc_piix.c static void pc_init1(MachineState *machine, · line 306

Quoted 10 lines
hw/i386/pc_piix.c +5 -1
@@ -298,7 +298,13 @@ static void pc_init1(MachineState *machine,
     }
 
     if (pcms->acpi_nvdimm_state.is_enabled) {
-        nvdimm_init_acpi_state(&pcms->acpi_nvdimm_state, system_io,
+        AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_PORT;
 
+        acpi_nvdimm_state->dsm_io.base = NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;

above constants probably should by moved to target a specific header

Quoted 33 lines
hw/i386/pc_piix.c +2 -0
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, system_io,
                                pcms->fw_cfg, OBJECT(pcms));
     }

}

hw/i386/pc_q35.c +7 -1
index b7b7959934..1110a26e34 100644
@@ -330,7 +330,13 @@ static void pc_q35_init(MachineState *machine)
     pc_nic_init(pcmc, isa_bus, host_bus);
 
     if (pcms->acpi_nvdimm_state.is_enabled) {
-        nvdimm_init_acpi_state(&pcms->acpi_nvdimm_state, system_io,
+        AcpiNVDIMMState *acpi_nvdimm_state = &pcms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_PORT;
+        acpi_nvdimm_state->dsm_io.base = NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, system_io,
                                pcms->fw_cfg, OBJECT(pcms));
     }
 }
include/hw/mem/nvdimm.h +0 -0
index c5c9b3c7f8..af8a5fd034 100644
@@ -123,6 +123,17 @@ struct NvdimmFitBuffer {
 };
 typedef struct NvdimmFitBuffer NvdimmFitBuffer;
 
 

include/hw/mem/nvdimm.h struct NvdimmFitBuffer { · line 136

Quoted 10 lines
include/hw/mem/nvdimm.h +10 -0
+typedef enum {
+    NVDIMM_ACPI_IO_PORT,
+    NVDIMM_ACPI_IO_MEMORY,
+} AcpiNVDIMMIOType;
+
+typedef struct AcpiNVDIMMIOEntry {
+    AcpiNVDIMMIOType type;
+    hwaddr base;
+    hwaddr len;
+} AcpiNVDIMMIOEntry;

This one very much resembles AcpiGenericAddress,
why not to reuse it?

Quoted 11 lines
include/hw/mem/nvdimm.h +1 -0
 struct AcpiNVDIMMState {
     /* detect if NVDIMM support is enabled. */
     bool is_enabled;
@@ -140,6 +151,7 @@ struct AcpiNVDIMMState {
      */
     int32_t persistence;
     char    *persistence_string;
+    AcpiNVDIMMIOEntry dsm_io;
 };
 typedef struct AcpiNVDIMMState AcpiNVDIMMState;
 
author
#47 lore raw

PATCH v6 16/18 hw/arm/virt: Add nvdimm hot-plug infrastructure

In reply to Eric Auger

From: Kwangwoo Lee <[email protected]>

Pre-plug and plug handlers are prepared for NVDIMM support.

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt-acpi-build.c        |  6 ++++++
 hw/arm/virt.c                   | 22 ++++++++++++++++++++++
 include/hw/arm/virt.h           |  3 +++
 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
index dc4624794f..ddbe87ed15 100644
@@ -162,3 +162,5 @@ CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
 CONFIG_MEM_DEVICE=y
 CONFIG_DIMM=y
+CONFIG_NVDIMM=y
+CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
index 781eafaf5e..f086adfa82 100644
@@ -784,6 +784,7 @@ static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     GArray *table_offsets;
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
+                          &vms->acpi_nvdimm_state, ms->ram_slots);
+    }
+
     if (its_class_name() && !vmc->no_its) {
         acpi_add_table(table_offsets, tables_blob);
         build_iort(tables_blob, tables->linker, vms);
hw/arm/virt.c +22 -0
index b683902991..0c8c2cc191 100644
@@ -132,6 +132,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_SECURE_UART] =        { 0x09040000, 0x00001000 },
     [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_ACPI_IO] =            { 0x09070000, 0x00010000 },
     [VIRT_MMIO] =               { 0x0a000000, 0x00000200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     [VIRT_PLATFORM_BUS] =       { 0x0c000000, 0x02000000 },
@@ -1637,6 +1638,18 @@ static void machvirt_init(MachineState *machine)
 
     create_platform_bus(vms, pic);
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_MEMORY;
+        acpi_nvdimm_state->dsm_io.base =
+                vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, sysmem,
+                               vms->fw_cfg, OBJECT(vms));
+    }
+
     vms->bootinfo.ram_size = machine->ram_size;
     vms->bootinfo.kernel_filename = machine->kernel_filename;
     vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1822,10 +1835,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                              DeviceState *dev, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
     Error *local_err = NULL;
 
     pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+    if (local_err) {
+        goto out;
+    }
 
+    if (is_nvdimm) {
+        nvdimm_plug(&vms->acpi_nvdimm_state);
+    }
+
+out:
     error_propagate(errp, local_err);
 }
 
include/hw/arm/virt.h +3 -0
index c88f67a492..56d73b0e86 100644
@@ -37,6 +37,7 @@
 #include "hw/arm/arm.h"
 #include "sysemu/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/mem/nvdimm.h"
 
 #define NUM_GICV2M_SPIS       64
 #define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_ACPI_IO,
     VIRT_LOWMEMMAP_LAST,
 };
 
@@ -134,6 +136,7 @@ typedef struct {
     hwaddr high_io_base;
     hwaddr highest_gpa;
     bool extended_memmap;
+    AcpiNVDIMMState acpi_nvdimm_state;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
Signature · 1 line
2.20.1
#48 lore raw

Re: [Qemu-arm] [PATCH v6 16/18] hw/arm/virt: Add nvdimm hot-plug infrastructure

Quoted 52 lines · On Tue, 5 Feb 2019 18:33:04 +0100 Eric Auger <[email protected]> wrote:

From: Kwangwoo Lee <[email protected]>

Pre-plug and plug handlers are prepared for NVDIMM support.

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt-acpi-build.c        |  6 ++++++
 hw/arm/virt.c                   | 22 ++++++++++++++++++++++
 include/hw/arm/virt.h           |  3 +++
 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
index dc4624794f..ddbe87ed15 100644
@@ -162,3 +162,5 @@ CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
 CONFIG_MEM_DEVICE=y
 CONFIG_DIMM=y
+CONFIG_NVDIMM=y
+CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
index 781eafaf5e..f086adfa82 100644
@@ -784,6 +784,7 @@ static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     GArray *table_offsets;
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
+                          &vms->acpi_nvdimm_state, ms->ram_slots);
+    }
+
     if (its_class_name() && !vmc->no_its) {
         acpi_add_table(table_offsets, tables_blob);
         build_iort(tables_blob, tables->linker, vms);
hw/arm/virt.c +0 -0
index b683902991..0c8c2cc191 100644

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 135

Quoted 4 lines
hw/arm/virt.c +1 -0
@@ -132,6 +132,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_SECURE_UART] =        { 0x09040000, 0x00001000 },
     [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_ACPI_IO] =            { 0x09070000, 0x00010000 },

where does this range come from and is its size sufficient?

Quoted 70 lines
hw/arm/virt.c +21 -0
     [VIRT_MMIO] =               { 0x0a000000, 0x00000200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     [VIRT_PLATFORM_BUS] =       { 0x0c000000, 0x02000000 },
@@ -1637,6 +1638,18 @@ static void machvirt_init(MachineState *machine)
 
     create_platform_bus(vms, pic);
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_MEMORY;
+        acpi_nvdimm_state->dsm_io.base =
+                vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, sysmem,
+                               vms->fw_cfg, OBJECT(vms));
+    }
+
     vms->bootinfo.ram_size = machine->ram_size;
     vms->bootinfo.kernel_filename = machine->kernel_filename;
     vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1822,10 +1835,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                              DeviceState *dev, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
     Error *local_err = NULL;
 
     pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+    if (local_err) {
+        goto out;
+    }
 
+    if (is_nvdimm) {
+        nvdimm_plug(&vms->acpi_nvdimm_state);
+    }
+
+out:
     error_propagate(errp, local_err);
 }
 
include/hw/arm/virt.h +3 -0
index c88f67a492..56d73b0e86 100644
@@ -37,6 +37,7 @@
 #include "hw/arm/arm.h"
 #include "sysemu/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/mem/nvdimm.h"
 
 #define NUM_GICV2M_SPIS       64
 #define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_ACPI_IO,
     VIRT_LOWMEMMAP_LAST,
 };
 
@@ -134,6 +136,7 @@ typedef struct {
     hwaddr high_io_base;
     hwaddr highest_gpa;
     bool extended_memmap;
+    AcpiNVDIMMState acpi_nvdimm_state;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
author
#49 lore raw

Re: [Qemu-devel] [PATCH v6 16/18] hw/arm/virt: Add nvdimm hot-plug infrastructure

Hi Igor,

Quoted 52 lines · Igor Mammedov
Quoted 49 lines · On Tue, 5 Feb 2019 18:33:04 +0100 Eric Auger <[email protected]> wrote:

From: Kwangwoo Lee <[email protected]>

Pre-plug and plug handlers are prepared for NVDIMM support.

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt-acpi-build.c        |  6 ++++++
 hw/arm/virt.c                   | 22 ++++++++++++++++++++++
 include/hw/arm/virt.h           |  3 +++
 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
index dc4624794f..ddbe87ed15 100644
@@ -162,3 +162,5 @@ CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
 CONFIG_MEM_DEVICE=y
 CONFIG_DIMM=y
+CONFIG_NVDIMM=y
+CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
index 781eafaf5e..f086adfa82 100644
@@ -784,6 +784,7 @@ static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     GArray *table_offsets;
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
+                          &vms->acpi_nvdimm_state, ms->ram_slots);
+    }
+
     if (its_class_name() && !vmc->no_its) {
         acpi_add_table(table_offsets, tables_blob);
         build_iort(tables_blob, tables->linker, vms);
hw/arm/virt.c +0 -0
index b683902991..0c8c2cc191 100644
Quoted 8 lines

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 135

Quoted 7 lines
hw/arm/virt.c +1 -0
@@ -132,6 +132,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_SECURE_UART] =        { 0x09040000, 0x00001000 },
     [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_ACPI_IO] =            { 0x09070000, 0x00010000 },

where does this range come from and is its size sufficient?

I understand it can be anywhere in low mem and must be large enough to contain [NVDIMM_ACPI_IO_BASE, NDIMM_ACPI_IO_BASE + NVDIMM_ACPI_IO_LEN]. So one 64kB page should do the job?

Thanks

Eric

Quoted 72 lines
Quoted 71 lines
hw/arm/virt.c +21 -0
     [VIRT_MMIO] =               { 0x0a000000, 0x00000200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     [VIRT_PLATFORM_BUS] =       { 0x0c000000, 0x02000000 },
@@ -1637,6 +1638,18 @@ static void machvirt_init(MachineState *machine)
 
     create_platform_bus(vms, pic);
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_MEMORY;
+        acpi_nvdimm_state->dsm_io.base =
+                vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, sysmem,
+                               vms->fw_cfg, OBJECT(vms));
+    }
+
     vms->bootinfo.ram_size = machine->ram_size;
     vms->bootinfo.kernel_filename = machine->kernel_filename;
     vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1822,10 +1835,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                              DeviceState *dev, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
     Error *local_err = NULL;
 
     pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+    if (local_err) {
+        goto out;
+    }
 
+    if (is_nvdimm) {
+        nvdimm_plug(&vms->acpi_nvdimm_state);
+    }
+
+out:
     error_propagate(errp, local_err);
 }
 
include/hw/arm/virt.h +3 -0
index c88f67a492..56d73b0e86 100644
@@ -37,6 +37,7 @@
 #include "hw/arm/arm.h"
 #include "sysemu/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/mem/nvdimm.h"
 
 #define NUM_GICV2M_SPIS       64
 #define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_ACPI_IO,
     VIRT_LOWMEMMAP_LAST,
 };
 
@@ -134,6 +136,7 @@ typedef struct {
     hwaddr high_io_base;
     hwaddr highest_gpa;
     bool extended_memmap;
+    AcpiNVDIMMState acpi_nvdimm_state;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
#50 lore raw

Re: [Qemu-arm] [PATCH v6 16/18] hw/arm/virt: Add nvdimm hot-plug infrastructure

Quoted 63 lines · On Wed, 20 Feb 2019 16:21:05 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 52 lines · Igor Mammedov
Quoted 49 lines · On Tue, 5 Feb 2019 18:33:04 +0100 Eric Auger <[email protected]> wrote:

From: Kwangwoo Lee <[email protected]>

Pre-plug and plug handlers are prepared for NVDIMM support.

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt-acpi-build.c        |  6 ++++++
 hw/arm/virt.c                   | 22 ++++++++++++++++++++++
 include/hw/arm/virt.h           |  3 +++
 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
index dc4624794f..ddbe87ed15 100644
@@ -162,3 +162,5 @@ CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
 CONFIG_MEM_DEVICE=y
 CONFIG_DIMM=y
+CONFIG_NVDIMM=y
+CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
index 781eafaf5e..f086adfa82 100644
@@ -784,6 +784,7 @@ static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     GArray *table_offsets;
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
+                          &vms->acpi_nvdimm_state, ms->ram_slots);
+    }
+
     if (its_class_name() && !vmc->no_its) {
         acpi_add_table(table_offsets, tables_blob);
         build_iort(tables_blob, tables->linker, vms);
hw/arm/virt.c +0 -0
index b683902991..0c8c2cc191 100644
Quoted 8 lines

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 135

Quoted 7 lines
hw/arm/virt.c +1 -0
@@ -132,6 +132,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_SECURE_UART] =        { 0x09040000, 0x00001000 },
     [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_ACPI_IO] =            { 0x09070000, 0x00010000 },  

where does this range come from and is its size sufficient?

Quoted 2 lines

I understand it can be anywhere in low mem and must be large enough to contain [NVDIMM_ACPI_IO_BASE, NDIMM_ACPI_IO_BASE + NVDIMM_ACPI_IO_LEN].

it looked to like generic ACPI arrea rather than NVDIMM
so I'd suggest to name it properly and probably use NVDIMM_ACPI_IO_BASE & co to add this entry so that reader won't have to wonder where this magic numbers come from.

Quoted 1 line

So one 64kB page should do the job?

Should it be located in lowmem?
(do we care about device_memory & AVMF & ACPI & not 64bit guests?)

Quoted 75 lines

Thanks

Eric

Quoted 72 lines
Quoted 71 lines
hw/arm/virt.c +21 -0
     [VIRT_MMIO] =               { 0x0a000000, 0x00000200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     [VIRT_PLATFORM_BUS] =       { 0x0c000000, 0x02000000 },
@@ -1637,6 +1638,18 @@ static void machvirt_init(MachineState *machine)
 
     create_platform_bus(vms, pic);
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_MEMORY;
+        acpi_nvdimm_state->dsm_io.base =
+                vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, sysmem,
+                               vms->fw_cfg, OBJECT(vms));
+    }
+
     vms->bootinfo.ram_size = machine->ram_size;
     vms->bootinfo.kernel_filename = machine->kernel_filename;
     vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1822,10 +1835,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                              DeviceState *dev, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
     Error *local_err = NULL;
 
     pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+    if (local_err) {
+        goto out;
+    }
 
+    if (is_nvdimm) {
+        nvdimm_plug(&vms->acpi_nvdimm_state);
+    }
+
+out:
     error_propagate(errp, local_err);
 }
 
include/hw/arm/virt.h +3 -0
index c88f67a492..56d73b0e86 100644
@@ -37,6 +37,7 @@
 #include "hw/arm/arm.h"
 #include "sysemu/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/mem/nvdimm.h"
 
 #define NUM_GICV2M_SPIS       64
 #define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_ACPI_IO,
     VIRT_LOWMEMMAP_LAST,
 };
 
@@ -134,6 +136,7 @@ typedef struct {
     hwaddr high_io_base;
     hwaddr highest_gpa;
     bool extended_memmap;
+    AcpiNVDIMMState acpi_nvdimm_state;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)  
author
#51 lore raw

Re: [Qemu-arm] [Qemu-devel] [PATCH v6 16/18] hw/arm/virt: Add nvdimm hot-plug infrastructure

Hi Igor,

Quoted 66 lines · Igor Mammedov
Quoted 63 lines · On Wed, 20 Feb 2019 16:21:05 +0100 Auger Eric <[email protected]> wrote:

Hi Igor,

Quoted 60 lines · Igor Mammedov
Quoted 52 lines · On Tue, 5 Feb 2019 18:33:04 +0100 Eric Auger <[email protected]> wrote:

From: Kwangwoo Lee <[email protected]>

Pre-plug and plug handlers are prepared for NVDIMM support.

---
 default-configs/arm-softmmu.mak |  2 ++
 hw/arm/virt-acpi-build.c        |  6 ++++++
 hw/arm/virt.c                   | 22 ++++++++++++++++++++++
 include/hw/arm/virt.h           |  3 +++
 4 files changed, 33 insertions(+)
default-configs/arm-softmmu.mak +2 -0
index dc4624794f..ddbe87ed15 100644
@@ -162,3 +162,5 @@ CONFIG_HIGHBANK=y
 CONFIG_MUSICPAL=y
 CONFIG_MEM_DEVICE=y
 CONFIG_DIMM=y
+CONFIG_NVDIMM=y
+CONFIG_ACPI_NVDIMM=y
hw/arm/virt-acpi-build.c +6 -0
index 781eafaf5e..f086adfa82 100644
@@ -784,6 +784,7 @@ static
 void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
+    MachineState *ms = MACHINE(vms);
     GArray *table_offsets;
     unsigned dsdt, xsdt;
     GArray *tables_blob = tables->table_data;
@@ -824,6 +825,11 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
         }
     }
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
+                          &vms->acpi_nvdimm_state, ms->ram_slots);
+    }
+
     if (its_class_name() && !vmc->no_its) {
         acpi_add_table(table_offsets, tables_blob);
         build_iort(tables_blob, tables->linker, vms);
hw/arm/virt.c +0 -0
index b683902991..0c8c2cc191 100644

hw/arm/virt.c static const MemMapEntry a15memmap[] = { · line 135

Quoted 4 lines
hw/arm/virt.c +1 -0
@@ -132,6 +132,7 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_SECURE_UART] =        { 0x09040000, 0x00001000 },
     [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_ACPI_IO] =            { 0x09070000, 0x00010000 },  

where does this range come from and is its size sufficient?

Quoted 6 lines
Quoted 2 lines

I understand it can be anywhere in low mem and must be large enough to contain [NVDIMM_ACPI_IO_BASE, NDIMM_ACPI_IO_BASE + NVDIMM_ACPI_IO_LEN].

it looked to like generic ACPI arrea rather than NVDIMM
so I'd suggest to name it properly and probably use NVDIMM_ACPI_IO_BASE & co to add this entry so that reader won't have to wonder where this magic numbers come from.

OK. I will rename the region in v8

Quoted 4 lines
Quoted 2 lines

So one 64kB page should do the job?

Should it be located in lowmem?
(do we care about device_memory & AVMF & ACPI & not 64bit guests?)

I guess no. We could easily fit this region into the existing low mem IO, that's what I meant actually.

Thanks

Eric

Quoted 79 lines
Quoted 77 lines

Thanks

Eric

Quoted 72 lines
Quoted 71 lines
hw/arm/virt.c +21 -0
     [VIRT_MMIO] =               { 0x0a000000, 0x00000200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     [VIRT_PLATFORM_BUS] =       { 0x0c000000, 0x02000000 },
@@ -1637,6 +1638,18 @@ static void machvirt_init(MachineState *machine)
 
     create_platform_bus(vms, pic);
 
+    if (vms->acpi_nvdimm_state.is_enabled) {
+        AcpiNVDIMMState *acpi_nvdimm_state = &vms->acpi_nvdimm_state;
+
+        acpi_nvdimm_state->dsm_io.type = NVDIMM_ACPI_IO_MEMORY;
+        acpi_nvdimm_state->dsm_io.base =
+                vms->memmap[VIRT_ACPI_IO].base + NVDIMM_ACPI_IO_BASE;
+        acpi_nvdimm_state->dsm_io.len = NVDIMM_ACPI_IO_LEN;
+
+        nvdimm_init_acpi_state(acpi_nvdimm_state, sysmem,
+                               vms->fw_cfg, OBJECT(vms));
+    }
+
     vms->bootinfo.ram_size = machine->ram_size;
     vms->bootinfo.kernel_filename = machine->kernel_filename;
     vms->bootinfo.kernel_cmdline = machine->kernel_cmdline;
@@ -1822,10 +1835,19 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                              DeviceState *dev, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
+    bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
     Error *local_err = NULL;
 
     pc_dimm_plug(PC_DIMM(dev), MACHINE(vms), &local_err);
+    if (local_err) {
+        goto out;
+    }
 
+    if (is_nvdimm) {
+        nvdimm_plug(&vms->acpi_nvdimm_state);
+    }
+
+out:
     error_propagate(errp, local_err);
 }
 
include/hw/arm/virt.h +3 -0
index c88f67a492..56d73b0e86 100644
@@ -37,6 +37,7 @@
 #include "hw/arm/arm.h"
 #include "sysemu/kvm.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/mem/nvdimm.h"
 
 #define NUM_GICV2M_SPIS       64
 #define NUM_VIRTIO_TRANSPORTS 32
@@ -77,6 +78,7 @@ enum {
     VIRT_GPIO,
     VIRT_SECURE_UART,
     VIRT_SECURE_MEM,
+    VIRT_ACPI_IO,
     VIRT_LOWMEMMAP_LAST,
 };
 
@@ -134,6 +136,7 @@ typedef struct {
     hwaddr high_io_base;
     hwaddr highest_gpa;
     bool extended_memmap;
+    AcpiNVDIMMState acpi_nvdimm_state;
 } VirtMachineState;
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)  
author
#52 lore raw

PATCH v6 17/18 hw/arm/boot: Expose the pmem nodes in the DT

In reply to Eric Auger

In case of NV-DIMM slots, let's add /pmem DT nodes.

---
 hw/arm/boot.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
hw/arm/boot.c +32 -1
index 2a70e8aa82..b1aa866f07 100644
@@ -454,6 +454,36 @@ out:
     return ret;
 }
 
+static int fdt_add_pmem_node(void *fdt, uint32_t acells, hwaddr mem_base,
+                             uint32_t scells, hwaddr mem_len,
+                             int numa_node_id)
+{
+    char *nodename = NULL;
+    int ret;
+
+    nodename = g_strdup_printf("/pmem@%" PRIx64, mem_base);
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop_string(fdt, nodename, "compatible", "pmem-region");
+    ret = qemu_fdt_setprop_sized_cells(fdt, nodename, "reg", acells, mem_base,
+                                       scells, mem_len);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/reg\n", nodename);
+        goto out;
+    }
+    if (numa_node_id < 0) {
+        goto out;
+    }
+
+    ret = qemu_fdt_setprop_cell(fdt, nodename, "numa-node-id", numa_node_id);
+    if (ret < 0) {
+        fprintf(stderr, "couldn't set %s/numa-node-id\n", nodename);
+    }
+
+out:
+    g_free(nodename);
+    return ret;
+}
+
 static void fdt_add_psci_node(void *fdt)
 {
     uint32_t cpu_suspend_fn;
@@ -541,7 +571,8 @@ static int fdt_add_hotpluggable_memory_nodes(void *fdt,
         di = !is_nvdimm ? mi->u.dimm.data : mi->u.nvdimm.data;
 
         if (is_nvdimm) {
-            ret = -ENOENT; /* NV-DIMM not yet supported */
+            ret = fdt_add_pmem_node(fdt, acells, di->addr,
+                                    scells, di->size, di->node);
         } else {
             ret = fdt_add_memory_node(fdt, acells, di->addr,
                                       scells, di->size, di->node);
Signature · 1 line
2.20.1
author
#53 lore raw

PATCH v6 18/18 hw/arm/virt: Add nvdimm and nvdimm-persistence options

In reply to Eric Auger

Machine option nvdimm allows to turn NVDIMM support on.

---
 hw/arm/virt.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 57 insertions(+), 2 deletions(-)
hw/arm/virt.c +57 -2
index 0c8c2cc191..85ce9becdb 100644
@@ -1776,6 +1776,47 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp)
     }
 }
 
+static bool virt_get_nvdimm(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return vms->acpi_nvdimm_state.is_enabled;
+}
+
+static void virt_set_nvdimm(Object *obj, bool value, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    vms->acpi_nvdimm_state.is_enabled = value;
+}
+
+static char *virt_get_nvdimm_persistence(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return g_strdup(vms->acpi_nvdimm_state.persistence_string);
+}
+
+static void virt_set_nvdimm_persistence(Object *obj, const char *value,
+                                        Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+    AcpiNVDIMMState *nvdimm_state = &vms->acpi_nvdimm_state;
+
+    if (strcmp(value, "cpu") == 0)
+        nvdimm_state->persistence = 3;
+    else if (strcmp(value, "mem-ctrl") == 0)
+        nvdimm_state->persistence = 2;
+    else {
+        error_report("-machine nvdimm-persistence=%s: unsupported option",
+                     value);
+        exit(EXIT_FAILURE);
+    }
+
+    g_free(nvdimm_state->persistence_string);
+    nvdimm_state->persistence_string = g_strdup(value);
+}
+
 static CpuInstanceProperties
 virt_cpu_index_to_props(MachineState *ms, unsigned cpu_index)
 {
@@ -1818,13 +1859,14 @@ static void virt_memory_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
                                  Error **errp)
 {
     const bool is_nvdimm = object_dynamic_cast(OBJECT(dev), TYPE_NVDIMM);
+    VirtMachineState *vms = VIRT_MACHINE(hotplug_dev);
 
     if (dev->hotplugged) {
         error_setg(errp, "memory hotplug is not supported");
     }
 
-    if (is_nvdimm) {
-        error_setg(errp, "nvdimm is not yet supported");
+    if (is_nvdimm && !vms->acpi_nvdimm_state.is_enabled) {
+        error_setg(errp, "nvdimm is not enabled: missing 'nvdimm' in '-M'");
         return;
     }
 
@@ -2032,6 +2074,19 @@ static void virt_instance_init(Object *obj)
                                     "Valid values are none and smmuv3",
                                     NULL);
 
+    object_property_add_bool(obj, "nvdimm",
+                             virt_get_nvdimm, virt_set_nvdimm, NULL);
+    object_property_set_description(obj, "nvdimm",
+                                         "Set on/off to enable/disable NVDIMM "
+                                         "instantiation", NULL);
+
+    object_property_add_str(obj, "nvdimm-persistence",
+                            virt_get_nvdimm_persistence,
+                            virt_set_nvdimm_persistence, NULL);
+    object_property_set_description(obj, "nvdimm-persistence",
+                                    "Set NVDIMM persistence"
+                                    "Valid values are cpu and mem-ctrl", NULL);
+
     vms->irqmap = a15irqmap;
 }
 
Signature · 1 line
2.20.1
#54 lore raw
In reply to Eric Auger
Quoted 10 lines · Eric Auger

This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.

Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 10] and change in the memory map
2) Support of PC-DIMM [11 - 14]
3) Support of NV-DIMM [15 - 18]

1) can be upstreamed before 2 and 2 can be upstreamed before 3.

Hi Eric; sorry I haven't reviewed this series earlier. I think that 1-10 are pretty near to ready to go in; maybe the easiest path is to do a respin of just those with the review issues fixed?

I'm a long way from being expert in the PC-DIMM/NV-DIMM stuff, so I'm going to be reliant on other people to review those parts.

I don't know if your series needs anything from linux-headers which isn't already in QEMU master after the update to match 5.0rc1 -- if not you could drop the header-sync patch.

thanks
-- PMM

author
#55 lore raw

Hi Peter,

Quoted 10 lines · Peter Maydell
Quoted 9 lines · Eric Auger

This series aims to bump the 255GB RAM limit in machvirt and to support device memory in general, and especially PCDIMM/NVDIMM.

Functionally, the series is split into 3 parts:
1) bump of the initial RAM limit [1 - 10] and change in the memory map
2) Support of PC-DIMM [11 - 14]
3) Support of NV-DIMM [15 - 18]

Quoted 5 lines
Quoted 1 line

1) can be upstreamed before 2 and 2 can be upstreamed before 3.

Hi Eric; sorry I haven't reviewed this series earlier. I think that 1-10 are pretty near to ready to go in; maybe the easiest path is to do a respin of just those with the review issues fixed?

No problem. Thank you for the review.

Yes I will quickly respin the patches you reviewed.

Quoted 7 lines

I'm a long way from being expert in the PC-DIMM/NV-DIMM stuff, so I'm going to be reliant on other people to review those parts.

I don't know if your series needs anything from linux-headers which isn't already in QEMU master after the update to match 5.0rc1 -- if not you could drop the header-sync patch.

5.0-rc1 should be OK so I think I can drop the header sync.

Thanks

Eric

Quoted 4 lines

thanks
-- PMM