Skip to contentSkip to conversation

hw/arm/virt enable support for virtio-mem

PATCH In review Jonathan Cameron 6 participants 17 messages Nov 5, 2020 – Dec 2, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 hw/arm/virt enable support for virtio-mem

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 default-configs/devices/aarch64-softmmu.mak |  1 +
 hw/arm/Kconfig                              |  1 +
 hw/arm/virt.c                               | 64 ++++++++++++++++++++-
 hw/virtio/Kconfig                           |  4 ++
 hw/virtio/virtio-mem.c                      |  2 +
 5 files changed, 71 insertions(+), 1 deletion(-)
default-configs/devices/aarch64-softmmu.mak +1 -0
index 958b1e08e4..31d6128a29 100644
@@ -6,3 +6,4 @@ include arm-softmmu.mak
 CONFIG_XLNX_ZYNQMP_ARM=y
 CONFIG_XLNX_VERSAL=y
 CONFIG_SBSA_REF=y
+CONFIG_ARCH_VIRTIO_MEM_SUPPORTED=y
hw/arm/Kconfig +1 -0
index fdf4464b94..eeae77eee9 100644
@@ -20,6 +20,7 @@ config ARM_VIRT
     select PLATFORM_BUS
     select SMBIOS
     select VIRTIO_MMIO
+    select VIRTIO_MEM_SUPPORTED if ARCH_VIRTIO_MEM_SUPPORTED
     select ACPI_PCI
     select MEM_DEVICE
     select DIMM
hw/arm/virt.c +63 -1

109 lines not on this page · raw message

hw/virtio/Kconfig +4 -0
index 0eda25c4e1..00dbf2939e 100644
@@ -48,6 +48,10 @@ config VIRTIO_PMEM
     depends on VIRTIO_PMEM_SUPPORTED
     select MEM_DEVICE
 
+config ARCH_VIRTIO_MEM_SUPPORTED
+    bool
+    default n
+
 config VIRTIO_MEM_SUPPORTED
     bool
 
hw/virtio/virtio-mem.c +2 -0
index 7c8ca9f28b..16f9de6ab6 100644
@@ -53,6 +53,8 @@
  */
 #if defined(TARGET_X86_64) || defined(TARGET_I386)
 #define VIRTIO_MEM_USABLE_EXTENT (2 * (128 * MiB))
+#elif defined(TARGET_AARCH64)
+#define VIRTIO_MEM_USABLE_EXTENT (2 * (1024 * MiB))
 #else
 #error VIRTIO_MEM_USABLE_EXTENT not defined
 #endif
Signature · 1 line
2.19.1
#1 lore raw

Cc'ing Igor.

Quoted 205 lines · Jonathan Cameron

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 default-configs/devices/aarch64-softmmu.mak |  1 +
 hw/arm/Kconfig                              |  1 +
 hw/arm/virt.c                               | 64 ++++++++++++++++++++-
 hw/virtio/Kconfig                           |  4 ++
 hw/virtio/virtio-mem.c                      |  2 +
 5 files changed, 71 insertions(+), 1 deletion(-)
default-configs/devices/aarch64-softmmu.mak +1 -0
index 958b1e08e4..31d6128a29 100644
@@ -6,3 +6,4 @@ include arm-softmmu.mak
 CONFIG_XLNX_ZYNQMP_ARM=y
 CONFIG_XLNX_VERSAL=y
 CONFIG_SBSA_REF=y
+CONFIG_ARCH_VIRTIO_MEM_SUPPORTED=y
hw/arm/Kconfig +1 -0
index fdf4464b94..eeae77eee9 100644
@@ -20,6 +20,7 @@ config ARM_VIRT
     select PLATFORM_BUS
     select SMBIOS
     select VIRTIO_MMIO
+    select VIRTIO_MEM_SUPPORTED if ARCH_VIRTIO_MEM_SUPPORTED
     select ACPI_PCI
     select MEM_DEVICE
     select DIMM
hw/arm/virt.c +63 -1
index 8abb385d4e..6c96d71106 100644
@@ -73,9 +73,11 @@
 #include "hw/acpi/acpi.h"
 #include "target/arm/internals.h"
 #include "hw/mem/pc-dimm.h"
+#include "hw/mem/memory-device.h"
 #include "hw/mem/nvdimm.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/virtio/virtio-iommu.h"
+#include "hw/virtio/virtio-mem-pci.h"
 #include "hw/char/pl011.h"
 #include "qemu/guest-random.h"
 
@@ -2286,6 +2288,34 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                          dev, &error_abort);
 }
 
+static void virt_virtio_md_pci_pre_plug(HotplugHandler *hotplug_dev,
+                                      DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    if (!hotplug_dev2 && dev->hotplugged) {
+        /*
+         * Without a bus hotplug handler, we cannot control the plug/unplug
+         * order. We should never reach this point when hotplugging,
+         * however, better add a safety net.
+         */
+        error_setg(errp, "hotplug of virtio-mem based memory devices not"
+                   " supported on this bus.");
+        return;
+    }
+    /*
+     * First, see if we can plug this memory device at all. If that
+     * succeeds, branch of to the actual hotplug handler.
+     */
+    memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev), NULL,
+                           &local_err);
+    if (!local_err && hotplug_dev2) {
+        hotplug_handler_pre_plug(hotplug_dev2, dev, &local_err);
+    }
+    error_propagate(errp, local_err);
+}
+
 static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                             DeviceState *dev, Error **errp)
 {
@@ -2293,6 +2323,8 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
 
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_pre_plug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_pre_plug(hotplug_dev, dev, errp);
     } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
         hwaddr db_start = 0, db_end = 0;
         char *resv_prop_str;
@@ -2322,6 +2354,27 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
     }
 }
 
+static void virt_virtio_md_pci_plug(HotplugHandler *hotplug_dev,
+                                  DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    /*
+     * Plug the memory device first and then branch off to the actual
+     * hotplug handler. If that one fails, we can easily undo the memory
+     * device bits.
+     */
+    memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+    if (hotplug_dev2) {
+        hotplug_handler_plug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+        }
+    }
+    error_propagate(errp, local_err);
+}
+
 static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
                                         DeviceState *dev, Error **errp)
 {
@@ -2336,6 +2389,9 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_plug(hotplug_dev, dev, errp);
     }
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_plug(hotplug_dev, dev, errp);
+    }
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
         PCIDevice *pdev = PCI_DEVICE(dev);
 
@@ -2363,6 +2419,11 @@ static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev,
         goto out;
     }
 
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        error_setg(&local_err,
+                   "virtio-mem based memory devices cannot be unplugged.");
+        goto out;
+    }
     hotplug_handler_unplug_request(HOTPLUG_HANDLER(vms->acpi_dev), dev,
                                    &local_err);
 out:
@@ -2413,7 +2474,8 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
                                                         DeviceState *dev)
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_SYS_BUS_DEVICE) ||
-       (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM))) {
+        object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) ||
+        object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
         return HOTPLUG_HANDLER(machine);
     }
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
hw/virtio/Kconfig +4 -0
index 0eda25c4e1..00dbf2939e 100644
@@ -48,6 +48,10 @@ config VIRTIO_PMEM
     depends on VIRTIO_PMEM_SUPPORTED
     select MEM_DEVICE
 
+config ARCH_VIRTIO_MEM_SUPPORTED
+    bool
+    default n
+
 config VIRTIO_MEM_SUPPORTED
     bool
 
hw/virtio/virtio-mem.c +2 -0
index 7c8ca9f28b..16f9de6ab6 100644
@@ -53,6 +53,8 @@
  */
 #if defined(TARGET_X86_64) || defined(TARGET_I386)
 #define VIRTIO_MEM_USABLE_EXTENT (2 * (128 * MiB))
+#elif defined(TARGET_AARCH64)
+#define VIRTIO_MEM_USABLE_EXTENT (2 * (1024 * MiB))
 #else
 #error VIRTIO_MEM_USABLE_EXTENT not defined
 #endif
#2 lore raw
In reply to Jonathan Cameron

Hi Jonathan,

Quoted 11 lines · Jonathan Cameron

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

I would add in the commit message that the hot-unplug of the device is not supported.

Quoted 7 lines

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

Did you try with 64kB page guest as well?

Quoted 5 lines

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

I guess you meant CONFIG_ARCH_VIRTIO_MEM_SUPPORTED introduction

Quoted 122 lines

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 default-configs/devices/aarch64-softmmu.mak |  1 +
 hw/arm/Kconfig                              |  1 +
 hw/arm/virt.c                               | 64 ++++++++++++++++++++-
 hw/virtio/Kconfig                           |  4 ++
 hw/virtio/virtio-mem.c                      |  2 +
 5 files changed, 71 insertions(+), 1 deletion(-)
default-configs/devices/aarch64-softmmu.mak +1 -0
index 958b1e08e4..31d6128a29 100644
@@ -6,3 +6,4 @@ include arm-softmmu.mak
 CONFIG_XLNX_ZYNQMP_ARM=y
 CONFIG_XLNX_VERSAL=y
 CONFIG_SBSA_REF=y
+CONFIG_ARCH_VIRTIO_MEM_SUPPORTED=y
hw/arm/Kconfig +1 -0
index fdf4464b94..eeae77eee9 100644
@@ -20,6 +20,7 @@ config ARM_VIRT
     select PLATFORM_BUS
     select SMBIOS
     select VIRTIO_MMIO
+    select VIRTIO_MEM_SUPPORTED if ARCH_VIRTIO_MEM_SUPPORTED
     select ACPI_PCI
     select MEM_DEVICE
     select DIMM
hw/arm/virt.c +53 -0
index 8abb385d4e..6c96d71106 100644
@@ -73,9 +73,11 @@
 #include "hw/acpi/acpi.h"
 #include "target/arm/internals.h"
 #include "hw/mem/pc-dimm.h"
+#include "hw/mem/memory-device.h"
 #include "hw/mem/nvdimm.h"
 #include "hw/acpi/generic_event_device.h"
 #include "hw/virtio/virtio-iommu.h"
+#include "hw/virtio/virtio-mem-pci.h"
 #include "hw/char/pl011.h"
 #include "qemu/guest-random.h"
 
@@ -2286,6 +2288,34 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev,
                          dev, &error_abort);
 }
 
+static void virt_virtio_md_pci_pre_plug(HotplugHandler *hotplug_dev,
+                                      DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    if (!hotplug_dev2 && dev->hotplugged) {
+        /*
+         * Without a bus hotplug handler, we cannot control the plug/unplug
+         * order. We should never reach this point when hotplugging,
+         * however, better add a safety net.
+         */
+        error_setg(errp, "hotplug of virtio-mem based memory devices not"
+                   " supported on this bus.");
+        return;
+    }
+    /*
+     * First, see if we can plug this memory device at all. If that
+     * succeeds, branch of to the actual hotplug handler.
+     */
+    memory_device_pre_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev), NULL,
+                           &local_err);
+    if (!local_err && hotplug_dev2) {
+        hotplug_handler_pre_plug(hotplug_dev2, dev, &local_err);
+    }
+    error_propagate(errp, local_err);
+}
+
 static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
                                             DeviceState *dev, Error **errp)
 {
@@ -2293,6 +2323,8 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
 
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_pre_plug(hotplug_dev, dev, errp);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_pre_plug(hotplug_dev, dev, errp);
     } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
         hwaddr db_start = 0, db_end = 0;
         char *resv_prop_str;
@@ -2322,6 +2354,27 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
     }
 }
 
+static void virt_virtio_md_pci_plug(HotplugHandler *hotplug_dev,
+                                  DeviceState *dev, Error **errp)
+{
+    HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev);
+    Error *local_err = NULL;
+
+    /*
+     * Plug the memory device first and then branch off to the actual
+     * hotplug handler. If that one fails, we can easily undo the memory
+     * device bits.
+     */
+    memory_device_plug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+    if (hotplug_dev2) {
+        hotplug_handler_plug(hotplug_dev2, dev, &local_err);
+        if (local_err) {
+            memory_device_unplug(MEMORY_DEVICE(dev), MACHINE(hotplug_dev));
+        }
+    }
+    error_propagate(errp, local_err);
+}
+
 static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
                                         DeviceState *dev, Error **errp)
 {

hw/arm/virt.c static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, · line 2391

Quoted 3 lines
hw/arm/virt.c +0 -0
@@ -2336,6 +2389,9 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
     if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
         virt_memory_plug(hotplug_dev, dev, errp);
     }

nit: while at it we can use "else if" here and below

hw/arm/virt.c static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev, · line 2420

Quoted 9 lines
hw/arm/virt.c +3 -0
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_plug(hotplug_dev, dev, errp);
+    }
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
         PCIDevice *pdev = PCI_DEVICE(dev);
 
@@ -2363,6 +2419,11 @@ static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev,
         goto out;
     }

virt_dimm_unplug_request() is called if the device type is TYPE_PC_DIMM while here the device type is TYPE_VIRTIO_MEM_PCI.

Shouldn't this error be returned in virt_machine_device_unplug_request_cb instead or do like it is done in pc.c, add a virt_virtio_md_pci_unplug_request() helper with the error_setg there.

Quoted 40 lines
hw/arm/virt.c +7 -1
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        error_setg(&local_err,
+                   "virtio-mem based memory devices cannot be unplugged.");
+        goto out;
+    }
     hotplug_handler_unplug_request(HOTPLUG_HANDLER(vms->acpi_dev), dev,
                                    &local_err);
 out:
@@ -2413,7 +2474,8 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine,
                                                         DeviceState *dev)
 {
     if (object_dynamic_cast(OBJECT(dev), TYPE_SYS_BUS_DEVICE) ||
-       (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM))) {
+        object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) ||
+        object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
         return HOTPLUG_HANDLER(machine);
     }
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
hw/virtio/Kconfig +4 -0
index 0eda25c4e1..00dbf2939e 100644
@@ -48,6 +48,10 @@ config VIRTIO_PMEM
     depends on VIRTIO_PMEM_SUPPORTED
     select MEM_DEVICE
 
+config ARCH_VIRTIO_MEM_SUPPORTED
+    bool
+    default n
+
 config VIRTIO_MEM_SUPPORTED
     bool
 
hw/virtio/virtio-mem.c +0 -0
index 7c8ca9f28b..16f9de6ab6 100644

hw/virtio/virtio-mem.c line 60

Quoted 8 lines
hw/virtio/virtio-mem.c +2 -0
@@ -53,6 +53,8 @@
  */
 #if defined(TARGET_X86_64) || defined(TARGET_I386)
 #define VIRTIO_MEM_USABLE_EXTENT (2 * (128 * MiB))
+#elif defined(TARGET_AARCH64)
+#define VIRTIO_MEM_USABLE_EXTENT (2 * (1024 * MiB))
 #else
 #error VIRTIO_MEM_USABLE_EXTENT not defined
 #endif

Otherwise looks good to me.

Thanks!

Eric

#3 lore raw
In reply to Jonathan Cameron
Quoted 15 lines · Jonathan Cameron

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

I had my prototype living at

[email protected]:davidhildenbrand/qemu.git / virtio-mem-arm64

which looks very similar to your patch. That is good :)

[...]

hw/arm/virt.c static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, · line 2394

Quoted 10 lines

static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
DeviceState *dev, Error **errp)
{

hw/arm/virt.c +3 -0
@@ -2336,6 +2389,9 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
      if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
          virt_memory_plug(hotplug_dev, dev, errp);
      }
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_plug(hotplug_dev, dev, errp);
+    }

These better all be "else if".

hw/arm/virt.c static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev, · line 2426

Quoted 12 lines
hw/arm/virt.c +5 -0
      if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
          PCIDevice *pdev = PCI_DEVICE(dev);
 
@@ -2363,6 +2419,11 @@ static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev,
          goto out;
      }
 
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        error_setg(&local_err,
+                   "virtio-mem based memory devices cannot be unplugged.");
+        goto out;
+    }

This should go into virt_machine_device_unplug_request_cb() instead, no? [...]

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

4K guest on 64K host seems fine and no such limit is needed - though there may be performance issues doing that.

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

Jonathan

Quoted 40 lines

I had my prototype living at

[email protected]:davidhildenbrand/qemu.git / virtio-mem-arm64

which looks very similar to your patch. That is good :)

[...]

hw/arm/virt.c static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, · line 2394

Quoted 10 lines

static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
DeviceState *dev, Error **errp)
{

hw/arm/virt.c +3 -0
@@ -2336,6 +2389,9 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
      if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) {
          virt_memory_plug(hotplug_dev, dev, errp);
      }
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        virt_virtio_md_pci_plug(hotplug_dev, dev, errp);
+    }  

These better all be "else if".

hw/arm/virt.c static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev, · line 2426

Quoted 12 lines
hw/arm/virt.c +5 -0
      if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
          PCIDevice *pdev = PCI_DEVICE(dev);
 
@@ -2363,6 +2419,11 @@ static void virt_dimm_unplug_request(HotplugHandler *hotplug_dev,
          goto out;
      }
 
+    if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) {
+        error_setg(&local_err,
+                   "virtio-mem based memory devices cannot be unplugged.");
+        goto out;
+    }  

This should go into virt_machine_device_unplug_request_cb() instead, no? [...]

#5 lore raw
Quoted 46 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

Quoted 7 lines

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Quoted 3 lines

4K guest on 64K host seems fine and no such limit is needed - though there may be performance issues doing that.

Yeah, if one is lucky to get one of these 512 MiB huge pages at all :)

Quoted 4 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
Quoted 74 lines · David Hildenbrand
Quoted 46 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

Quoted 7 lines

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

Quoted 4 lines

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Even with 64k pages you may be able to have 2MB huge pages by setting default_hugepagesz=2M on the kernel command line.

Thanks,
drew

Quoted 20 lines
Quoted 4 lines

4K guest on 64K host seems fine and no such limit is needed - though there may be performance issues doing that.

Yeah, if one is lucky to get one of these 512 MiB huge pages at all :)

Quoted 4 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw
Quoted 75 lines · Andrew Jones
Quoted 74 lines · David Hildenbrand
Quoted 46 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

Quoted 7 lines

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

Quoted 7 lines
Quoted 4 lines

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Even with 64k pages you may be able to have 2MB huge pages by setting default_hugepagesz=2M on the kernel command line.

Yes, but not for THP, right? Last time I checked that move was not performed yet - resulting in MAX_ORDER/pageblock_order in Linux corresponding to 512 MB.

Quoted 3 lines

Thanks,
drew

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · Andrew Jones
Quoted 78 lines · David Hildenbrand
Quoted 46 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

Quoted 7 lines

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Quoted 7 lines
Quoted 2 lines

Even with 64k pages you may be able to have 2MB huge pages by setting default_hugepagesz=2M on the kernel command line.

Yes, but not for THP, right? Last time I checked that move was not performed yet - resulting in MAX_ORDER/pageblock_order in Linux corresponding to 512 MB.

Yes, I believe you're correct. At least on the machine I've booted with default_hugepagesz=2M, I see

 $ cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 536870912

(I'm not running a latest mainline kernel though.)

Thanks,
drew

#9 lore raw
Quoted 90 lines · Andrew Jones
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · Andrew Jones
Quoted 78 lines · David Hildenbrand
Quoted 46 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

Quoted 7 lines

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Quoted 7 lines
Quoted 2 lines

Even with 64k pages you may be able to have 2MB huge pages by setting default_hugepagesz=2M on the kernel command line.

Yes, but not for THP, right? Last time I checked that move was not performed yet - resulting in MAX_ORDER/pageblock_order in Linux corresponding to 512 MB.

Quoted 7 lines

Yes, I believe you're correct. At least on the machine I've booted with default_hugepagesz=2M, I see

 $ cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 536870912

(I'm not running a latest mainline kernel though.)

I remember some upstream discussions where people raised that switching to 2 MB THP might be possible (implemented via cont bits in the pagetables - similar to 2MB huge pages you mentioned). 512 MB really sounds more like gigantic pages after all.

Signature · 3 lines
Thanks,

David / dhildenb
#10 lore raw
In reply to David Hildenbrand
Quoted 52 lines · On Tue, 24 Nov 2020 20:17:35 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 51 lines · Jonathan Cameron

On Mon, 9 Nov 2020 20:47:09 +0100
David Hildenbrand <[email protected]> wrote:

+CC Eric based on similar query in other branch of the thread.

Quoted 25 lines · Jonathan Cameron
Quoted 15 lines

Basically a cut and paste job from the x86 support with the exception of needing a larger block size as the Memory Block Size (MIN_SECTION_SIZE) on ARM64 in Linux is 1G.

Tested:
* In full emulation and with KVM on an arm64 server.
* cold and hotplug for the virtio-mem-pci device.
* Wide range of memory sizes, added at creation and later.
* Fairly basic memory usage of memory added. Seems to function as normal.
* NUMA setup with virtio-mem-pci devices on each node.
* Simple migration test.

Related kernel patch just enables the Kconfig item for ARM64 as an alternative to x86 in drivers/virtio/Kconfig

Quoted 8 lines

The original patches from David Hildenbrand stated that he thought it should work for ARM64 but it wasn't enabled in the kernel [1]
It appears he was correct and everything 'just works'.

The build system related stuff is intended to ensure virtio-mem support is not built for arm32 (build will fail due no defined block size). If there is a more elegant way to do this, please point me in the right direction.

Quoted 5 lines

You might be aware of https://virtio-mem.gitlab.io/developer-guide.html
and the "issue" with 64k base pages - 512MB granularity. Similar as the question from Auger, have you tried running arm64 with differing page sizes in host/guest?

Hi David,

Quoted 4 lines

With recent kernels, you can use "memhp_default_state=online_movable" on the kernel cmdline to make memory unplug more likely to succeed - especially with 64k base pages. You just have to be sure to not hotplug "too much memory" to a VM.

Thanks for the pointer - that definitely simplifies testing. Was getting a bit tedious with out that.

As ever other stuff got in the way, so I only just got back to looking at this.

I've not done a particularly comprehensive set of tests yet, but things seem to 'work' with mixed page sizes.

Quoted 8 lines
Quoted 2 lines

With 64K pages in general, you run into a problem with the device block_size being smaller than the subblock_size. I've just added a check for that into the

"device block size smaller than subblock size" - that's very common, e.g., on x86-64.

E.g., device_block_size is 2MiB, subblock size 4MiB - until we improve that in the future in Linux guests.

Ah. I'd missed that quirk around MAX_ORDER. It's also true of ARM64 with 4k pages. As you can probably guess I'd forgotten to recompile my 4k test kernel after adding that particular check. :(

Ah well. Given we are already in a situation where adding 2MiB doesn't actually do anything useful, I guess it's not really a problem to merrily let the host add less than the guest can make use of. So we allow adding any multiple of 2MiB but reality is the guest isn't going to use anything other than 512MiB chunks.

Quoted 14 lines

Or did you mean something else?

Quoted 10 lines

virtio-mem kernel driver and have it fail to probe if that happens. I don't think such a setup makes any sense anyway so no loss there. Should it make sense to drop that restriction in the future we can deal with that then without breaking backwards compatibility.

So the question is whether it makes sense to bother with virtio-mem support at all on ARM64 with 64k pages given currently the minimum workable block_size is 512MiB? I guess there is an argument of virtio-mem being a possibly more convenient interface than full memory HP. Curious to hear what people think on this?

Quoted 4 lines

IMHO we really want it. For example, RHEL is always 64k. This is a current guest limitation, to be improved in the future - either by moving away from 512MB huge pages with 64k or by improving alloc_contig_range().

Sure. Would be great if we do one day support 2MiB on 64k.

Quoted 6 lines
Quoted 4 lines

4K guest on 64K host seems fine and no such limit is needed - though there may be performance issues doing that.

Yeah, if one is lucky to get one of these 512 MiB huge pages at all :)

Not too hard on my 1TB test system that's running nothing much else, but agreed it won't be trivial more generally.

Quoted 8 lines
Quoted 5 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

Jonathan

#11 lore raw
Quoted 9 lines

Ah. I'd missed that quirk around MAX_ORDER. It's also true of ARM64 with 4k pages. As you can probably guess I'd forgotten to recompile my 4k test kernel after adding that particular check. :(

Ah well. Given we are already in a situation where adding 2MiB doesn't actually do anything useful, I guess it's not really a problem to merrily let the host add less than the guest can make use of. So we allow adding any multiple of 2MiB but reality is the guest isn't going to use anything other than 512MiB chunks.

Right, and the host can observe the change not happening when not aligned to 512 MB. There are plans for a virtio-mem extension for the guest to present a status - e.g., why the requested size cannot be achieved (requested size not alignment, usable region too small, ENOMEM/EBUSY when unplugging, ...).

[...]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

4K guest on 64K host seems fine and no such limit is needed - though there may be performance issues doing that.

Yeah, if one is lucky to get one of these 512 MiB huge pages at all :)

Not too hard on my 1TB test system that's running nothing much else, but agreed it won't be trivial more generally.

Hehe, right ! (... and here I am, testing with 64GB machines ... :) )

It's more of an issue in the guest to get 512 MB without ZONE_MOVABLE to unplug ... especially with smaller VMs.

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

Perfect, thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#12 lore raw
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

Perfect, thanks!

Oh, btw, I have no idea what the state of vfio-pci + QEMU on arm64 is. In case it's supposed to work, you could give

https://lkml.kernel.org/r/[email protected]

to see what we're missing.

I added a short virtio-pci guide to

https://virtio-mem.gitlab.io/user-guide/user-guide-qemu.html

Signature · 3 lines
Thanks,

David / dhildenb
#13 lore raw
Quoted 10 lines · On Wed, 25 Nov 2020 16:54:53 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

Quoted 8 lines
Quoted 1 line

Perfect, thanks!

Oh, btw, I have no idea what the state of vfio-pci + QEMU on arm64 is. In case it's supposed to work, you could give

https://lkml.kernel.org/r/[email protected]

to see what we're missing.

vfio-pci works in general (and we use it a lot), so sure I'll give this a test run.

Quoted 5 lines

I added a short virtio-pci guide to

https://virtio-mem.gitlab.io/user-guide/user-guide-qemu.html

Thanks,

Jonathan

#14 lore raw
Quoted 18 lines · Jonathan Cameron
Quoted 15 lines · On Wed, 25 Nov 2020 16:54:53 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

64k guest on 4k host with 512MiB block size seems fine.

If there are any places anyone thinks need particular poking I'd appreciate a hint :)

If things seem to work for now, that's great :) Thanks!

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

Perfect, thanks!

Oh, btw, I have no idea what the state of vfio-pci + QEMU on arm64 is. In case it's supposed to work, you could give

Quoted 6 lines
Quoted 3 lines

https://lkml.kernel.org/r/[email protected]

to see what we're missing.

vfio-pci works in general (and we use it a lot), so sure I'll give this a test run.

Cool.

In case you get it to run, please test with both "online_kernel" and "online_movable" in the guest, and small boot memory (e.g., 2 GiB).

For example, on x86-64 I got my vfio-pci provided GPUs to consume virtio-mem memory easily when starting with 2-4 GiB boot memory and using "online_kernel". (I verified that when not creating the mappings, IO errors can be observed and graphics are messed up).

Signature · 3 lines
Thanks,

David / dhildenb
#15 lore raw
In reply to Jonathan Cameron
Quoted 2 lines · Jonathan Cameron

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

IIUC there will be another version of this patch, right?

Signature · 1 line
MST
#16 lore raw
Quoted 6 lines · On Wed, 2 Dec 2020 05:02:57 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 2 lines · Jonathan Cameron

Cool. I'll run a few more comprehensive tests then send out the trivial patch to enable the kernel option + v2 of the qemu support.

IIUC there will be another version of this patch, right?

Yes. Busy period so might be a little while yet to complete testing before v2.

Jonathan