Skip to contentSkip to conversation

s390: stop abusing memory_region_allocate_system_memory()

PATCH Superseded Igor Mammedov 5 participants 13 messages Aug 2, 2019 – Aug 5, 2019

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover s390: stop abusing memory_region_allocate_system_memory()

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Igor Mammedov (2):
kvm: s390: split too big memory section on several memslots
s390: do not call memory_region_allocate_system_memory() multiple
times

 include/hw/s390x/s390-virtio-ccw.h | 10 ++++
 include/sysemu/kvm_int.h           |  1 +
 accel/kvm/kvm-all.c                | 79 ++++++++++++++++++------------
 hw/s390x/s390-virtio-ccw.c         | 30 ++----------
 target/s390x/kvm.c                 |  1 +
 5 files changed, 63 insertions(+), 58 deletions(-)
Signature · 1 line
2.18.1
#1 lore raw

PATCH v3 1/2 kvm: s390: split too big memory section on several memslots

Max memslot size supported by kvm on s390 is 8Tb, move logic of splitting RAM in chunks upto 8T to KVM code.

This way it will hide KVM specific restrictions in KVM code and won't affect baord level design decisions. Which would allow us to avoid misusing memory_region_allocate_system_memory() API and eventually use a single hostmem backend for guest RAM.

---
patch prepares only KVM side for switching to single RAM memory region another patch will take care of dropping manual RAM partitioning in s390 code.

---
 include/hw/s390x/s390-virtio-ccw.h | 10 ++++
 include/sysemu/kvm_int.h           |  1 +
 accel/kvm/kvm-all.c                | 79 ++++++++++++++++++------------
 hw/s390x/s390-virtio-ccw.c         |  9 ----
 target/s390x/kvm.c                 |  1 +
 5 files changed, 60 insertions(+), 40 deletions(-)
include/hw/s390x/s390-virtio-ccw.h +10 -0
index 8aa27199c9..00632f94b4 100644
@@ -21,6 +21,16 @@
 #define S390_MACHINE_CLASS(klass) \
     OBJECT_CLASS_CHECK(S390CcwMachineClass, (klass), TYPE_S390_CCW_MACHINE)
 
+/*
+ * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
+ * as the dirty bitmap must be managed by bitops that take an int as
+ * position indicator. If we have a guest beyond that we will split off
+ * new subregions. The split must happen on a segment boundary (1MB).
+ */
+#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1)
+#define SEG_MSK (~0xfffffULL)
+#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
+
 typedef struct S390CcwMachineState {
     /*< private >*/
     MachineState parent_obj;
include/sysemu/kvm_int.h +1 -0
index 31df465fdc..7f7520bce2 100644
@@ -41,4 +41,5 @@ typedef struct KVMMemoryListener {
 void kvm_memory_listener_register(KVMState *s, KVMMemoryListener *kml,
                                   AddressSpace *as, int as_id);
 
+void kvm_set_max_memslot_size(hwaddr max_slot_size);
 #endif
accel/kvm/kvm-all.c +48 -31

108 lines not on this page · raw message

hw/s390x/s390-virtio-ccw.c +0 -9
index 5b6a9a4e55..0c03ffb7c7 100644
@@ -151,15 +151,6 @@ static void virtio_ccw_register_hcalls(void)
                                    virtio_ccw_hcall_early_printk);
 }
 
-/*
- * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
- * as the dirty bitmap must be managed by bitops that take an int as
- * position indicator. If we have a guest beyond that we will split off
- * new subregions. The split must happen on a segment boundary (1MB).
- */
-#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1)
-#define SEG_MSK (~0xfffffULL)
-#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
target/s390x/kvm.c +1 -0
index 6e814c230b..d0267da8e1 100644
@@ -347,6 +347,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
      */
     /* kvm_vm_enable_cap(s, KVM_CAP_S390_AIS, 0); */
 
+    kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
     return 0;
 }
 
Signature · 1 line
2.18.1
#2 lore raw

Re: [Qemu-devel] [PATCH for-4.2 v3 1/2] kvm: s390: split too big memory section on several memslots

Quoted 8 lines · Igor Mammedov

Max memslot size supported by kvm on s390 is 8Tb, move logic of splitting RAM in chunks upto 8T to KVM code.

This way it will hide KVM specific restrictions in KVM code and won't affect baord level design decisions. Which would allow us to avoid misusing memory_region_allocate_system_memory() API and eventually use a single hostmem backend for guest RAM.

Quoted 6 lines

---
patch prepares only KVM side for switching to single RAM memory region another patch will take care of dropping manual RAM partitioning in s390 code.
---

/home/cborntra/REPOS/qemu/target/s390x/kvm.c: In function ‘kvm_arch_init’:
/home/cborntra/REPOS/qemu/target/s390x/kvm.c:350:5: error: implicit declaration of function ‘kvm_set_max_memslot_size’; did you mean ‘kvm_get_max_memslots’? [-Werror=implicit-function-declaration]

  350 |     kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~
      |     kvm_get_max_memslots

/home/cborntra/REPOS/qemu/target/s390x/kvm.c:350:5: error: nested extern declaration of ‘kvm_set_max_memslot_size’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [/home/cborntra/REPOS/qemu/rules.mak:69: target/s390x/kvm.o] Error 1
make: *** [Makefile:472: s390x-softmmu/all] Error 2

We probably need an include of sysemu/kvm_int.h in target/s390x/kvm.c I will test with that fixup....

Quoted 187 lines
 include/hw/s390x/s390-virtio-ccw.h | 10 ++++
 include/sysemu/kvm_int.h           |  1 +
 accel/kvm/kvm-all.c                | 79 ++++++++++++++++++------------
 hw/s390x/s390-virtio-ccw.c         |  9 ----
 target/s390x/kvm.c                 |  1 +
 5 files changed, 60 insertions(+), 40 deletions(-)
include/hw/s390x/s390-virtio-ccw.h +10 -0
index 8aa27199c9..00632f94b4 100644
@@ -21,6 +21,16 @@
 #define S390_MACHINE_CLASS(klass) \
     OBJECT_CLASS_CHECK(S390CcwMachineClass, (klass), TYPE_S390_CCW_MACHINE)
 
+/*
+ * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
+ * as the dirty bitmap must be managed by bitops that take an int as
+ * position indicator. If we have a guest beyond that we will split off
+ * new subregions. The split must happen on a segment boundary (1MB).
+ */
+#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1)
+#define SEG_MSK (~0xfffffULL)
+#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
+
 typedef struct S390CcwMachineState {
     /*< private >*/
     MachineState parent_obj;
include/sysemu/kvm_int.h +1 -0
index 31df465fdc..7f7520bce2 100644
@@ -41,4 +41,5 @@ typedef struct KVMMemoryListener {
 void kvm_memory_listener_register(KVMState *s, KVMMemoryListener *kml,
                                   AddressSpace *as, int as_id);
 
+void kvm_set_max_memslot_size(hwaddr max_slot_size);
 #endif
accel/kvm/kvm-all.c +48 -31
index f450f25295..7a6efb9612 100644
@@ -138,6 +138,7 @@ bool kvm_direct_msi_allowed;
 bool kvm_ioeventfd_any_length_allowed;
 bool kvm_msi_use_devid;
 static bool kvm_immediate_exit;
+static hwaddr kvm_max_slot_size = ~0;
 
 static const KVMCapabilityInfo kvm_required_capabilites[] = {
     KVM_CAP_INFO(USER_MEMORY),
@@ -951,6 +952,14 @@ kvm_check_extension_list(KVMState *s, const KVMCapabilityInfo *list)
     return NULL;
 }
 
+void kvm_set_max_memslot_size(hwaddr max_slot_size)
+{
+    g_assert(
+        ROUND_UP(max_slot_size, qemu_real_host_page_size) == max_slot_size
+    );
+    kvm_max_slot_size = max_slot_size;
+}
+
 static void kvm_set_phys_mem(KVMMemoryListener *kml,
                              MemoryRegionSection *section, bool add)
 {
@@ -958,7 +967,7 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
     int err;
     MemoryRegion *mr = section->mr;
     bool writeable = !mr->readonly && !mr->rom_device;
-    hwaddr start_addr, size;
+    hwaddr start_addr, size, slot_size;
     void *ram;
 
     if (!memory_region_is_ram(mr)) {
@@ -983,41 +992,49 @@ static void kvm_set_phys_mem(KVMMemoryListener *kml,
     kvm_slots_lock(kml);
 
     if (!add) {
-        mem = kvm_lookup_matching_slot(kml, start_addr, size);
-        if (!mem) {
-            goto out;
-        }
-        if (mem->flags & KVM_MEM_LOG_DIRTY_PAGES) {
-            kvm_physical_sync_dirty_bitmap(kml, section);
-        }
+        do {
+            slot_size = kvm_max_slot_size < size ? kvm_max_slot_size : size;
+            mem = kvm_lookup_matching_slot(kml, start_addr, slot_size);
+            if (!mem) {
+                goto out;
+            }
+            if (mem->flags & KVM_MEM_LOG_DIRTY_PAGES) {
+                kvm_physical_sync_dirty_bitmap(kml, section);
+            }
 
-        /* unregister the slot */
-        g_free(mem->dirty_bmap);
-        mem->dirty_bmap = NULL;
-        mem->memory_size = 0;
-        mem->flags = 0;
-        err = kvm_set_user_memory_region(kml, mem, false);
-        if (err) {
-            fprintf(stderr, "%s: error unregistering slot: %s\n",
-                    __func__, strerror(-err));
-            abort();
-        }
+            /* unregister the slot */
+            g_free(mem->dirty_bmap);
+            mem->dirty_bmap = NULL;
+            mem->memory_size = 0;
+            mem->flags = 0;
+            err = kvm_set_user_memory_region(kml, mem, false);
+            if (err) {
+                fprintf(stderr, "%s: error unregistering slot: %s\n",
+                        __func__, strerror(-err));
+                abort();
+            }
+            start_addr += slot_size;
+        } while ((size -= slot_size));
         goto out;
     }
 
     /* register the new slot */
-    mem = kvm_alloc_slot(kml);
-    mem->memory_size = size;
-    mem->start_addr = start_addr;
-    mem->ram = ram;
-    mem->flags = kvm_mem_flags(mr);
-
-    err = kvm_set_user_memory_region(kml, mem, true);
-    if (err) {
-        fprintf(stderr, "%s: error registering slot: %s\n", __func__,
-                strerror(-err));
-        abort();
-    }
+    do {
+        slot_size = kvm_max_slot_size < size ? kvm_max_slot_size : size;
+        mem = kvm_alloc_slot(kml);
+        mem->memory_size = slot_size;
+        mem->start_addr = start_addr;
+        mem->ram = ram;
+        mem->flags = kvm_mem_flags(mr);
+
+        err = kvm_set_user_memory_region(kml, mem, true);
+        if (err) {
+            fprintf(stderr, "%s: error registering slot: %s\n", __func__,
+                    strerror(-err));
+            abort();
+        }
+        start_addr += slot_size;
+    } while ((size -= slot_size));
 
 out:
     kvm_slots_unlock(kml);
hw/s390x/s390-virtio-ccw.c +0 -9
index 5b6a9a4e55..0c03ffb7c7 100644
@@ -151,15 +151,6 @@ static void virtio_ccw_register_hcalls(void)
                                    virtio_ccw_hcall_early_printk);
 }
 
-/*
- * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
- * as the dirty bitmap must be managed by bitops that take an int as
- * position indicator. If we have a guest beyond that we will split off
- * new subregions. The split must happen on a segment boundary (1MB).
- */
-#define KVM_MEM_MAX_NR_PAGES ((1ULL << 31) - 1)
-#define SEG_MSK (~0xfffffULL)
-#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
target/s390x/kvm.c +1 -0
index 6e814c230b..d0267da8e1 100644
@@ -347,6 +347,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
      */
     /* kvm_vm_enable_cap(s, KVM_CAP_S390_AIS, 0); */
 
+    kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
     return 0;
 }
 
#3 lore raw

PATCH v3 2/2 s390: do not call memory_region_allocate_system_memory() multiple times

In reply to Igor Mammedov

s390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once.

Beside an invalid use of API, the approach also introduced migration issue, since RAM chunks for each KVM_SLOT_MAX_BYTES are transferred in migration stream as separate RAMBlocks.

After discussion [1], it was agreed to break migration from older QEMU for guest with RAM >8Tb (as it was relatively new (since 2.12) and considered to be not actually used downstream). Migration should keep working for guests with less than 8TB and for more than 8TB with QEMU 4.2 and newer binary.
In case user tries to migrate more than 8TB guest, between incompatible QEMU versions, migration should fail gracefully due to non-exiting RAMBlock ID or RAMBlock size mismatch.

Taking in account above and that now KVM code is able to split too big MemorySection into several memslots, stop abusing
memory_region_allocate_system_memory()
and use only one memory region for RAM.

1) [PATCH RFC v2 4/4] s390: do not call memory_region_allocate_system_memory() multiple times

---
v3:
- drop migration compat code

PS:
I don't have access to a suitable system to test it.

---
 hw/s390x/s390-virtio-ccw.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -18
index 0c03ffb7c7..e30058df38 100644
@@ -154,27 +154,12 @@ static void virtio_ccw_register_hcalls(void)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
-    ram_addr_t chunk, offset = 0;
-    unsigned int number = 0;
+    MemoryRegion *ram = g_new(MemoryRegion, 1);
     Error *local_err = NULL;
-    gchar *name;
 
     /* allocate RAM for core */
-    name = g_strdup_printf("s390.ram");
-    while (mem_size) {
-        MemoryRegion *ram = g_new(MemoryRegion, 1);
-        uint64_t size = mem_size;
-
-        /* KVM does not allow memslots >= 8 TB */
-        chunk = MIN(size, KVM_SLOT_MAX_BYTES);
-        memory_region_allocate_system_memory(ram, NULL, name, chunk);
-        memory_region_add_subregion(sysmem, offset, ram);
-        mem_size -= chunk;
-        offset += chunk;
-        g_free(name);
-        name = g_strdup_printf("s390.ram.%u", ++number);
-    }
-    g_free(name);
+    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
+    memory_region_add_subregion(sysmem, 0, ram);
 
     /*
      * Configure the maximum page size. As no memory devices were created
Signature · 1 line
2.18.1
Reviewed-by on v3 2/2
#4 lore raw

Re: [Qemu-devel] [PATCH for-4.2 v3 2/2] s390: do not call memory_region_allocate_system_memory() multiple times

Quoted 61 lines · Igor Mammedov

s390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once.

Beside an invalid use of API, the approach also introduced migration issue, since RAM chunks for each KVM_SLOT_MAX_BYTES are transferred in migration stream as separate RAMBlocks.

After discussion [1], it was agreed to break migration from older QEMU for guest with RAM >8Tb (as it was relatively new (since 2.12) and considered to be not actually used downstream). Migration should keep working for guests with less than 8TB and for more than 8TB with QEMU 4.2 and newer binary.
In case user tries to migrate more than 8TB guest, between incompatible QEMU versions, migration should fail gracefully due to non-exiting RAMBlock ID or RAMBlock size mismatch.

Taking in account above and that now KVM code is able to split too big MemorySection into several memslots, stop abusing
memory_region_allocate_system_memory()
and use only one memory region for RAM.

1) [PATCH RFC v2 4/4] s390: do not call memory_region_allocate_system_memory() multiple times

---
v3:
- drop migration compat code

PS:
I don't have access to a suitable system to test it.

---
 hw/s390x/s390-virtio-ccw.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)
hw/s390x/s390-virtio-ccw.c +1 -13
index 0c03ffb7c7..e30058df38 100644
@@ -154,27 +154,12 @@ static void virtio_ccw_register_hcalls(void)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
-    ram_addr_t chunk, offset = 0;
-    unsigned int number = 0;
+    MemoryRegion *ram = g_new(MemoryRegion, 1);
     Error *local_err = NULL;
-    gchar *name;
 
     /* allocate RAM for core */
-    name = g_strdup_printf("s390.ram");
-    while (mem_size) {
-        MemoryRegion *ram = g_new(MemoryRegion, 1);
-        uint64_t size = mem_size;
-
-        /* KVM does not allow memslots >= 8 TB */
-        chunk = MIN(size, KVM_SLOT_MAX_BYTES);
-        memory_region_allocate_system_memory(ram, NULL, name, chunk);
-        memory_region_add_subregion(sysmem, offset, ram);
-        mem_size -= chunk;

hw/s390x/s390-virtio-ccw.c static void virtio_ccw_register_hcalls(void) · line 165

Quoted 10 lines
hw/s390x/s390-virtio-ccw.c +2 -5
-        offset += chunk;
-        g_free(name);
-        name = g_strdup_printf("s390.ram.%u", ++number);
-    }
-    g_free(name);
+    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
+    memory_region_add_subregion(sysmem, 0, ram);
 
     /*
      * Configure the maximum page size. As no memory devices were created

We have to remember putting it into the next release notes. (or do we even want to get this into v4.1 ?)

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw

Re: [Qemu-devel] [PATCH for-4.2 v3 2/2] s390: do not call memory_region_allocate_system_memory() multiple times

Quoted 69 lines · David Hildenbrand
Quoted 68 lines · Igor Mammedov

s390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once.

Beside an invalid use of API, the approach also introduced migration issue, since RAM chunks for each KVM_SLOT_MAX_BYTES are transferred in migration stream as separate RAMBlocks.

After discussion [1], it was agreed to break migration from older QEMU for guest with RAM >8Tb (as it was relatively new (since 2.12) and considered to be not actually used downstream). Migration should keep working for guests with less than 8TB and for more than 8TB with QEMU 4.2 and newer binary.
In case user tries to migrate more than 8TB guest, between incompatible QEMU versions, migration should fail gracefully due to non-exiting RAMBlock ID or RAMBlock size mismatch.

Taking in account above and that now KVM code is able to split too big MemorySection into several memslots, stop abusing
memory_region_allocate_system_memory()
and use only one memory region for RAM.

1) [PATCH RFC v2 4/4] s390: do not call memory_region_allocate_system_memory() multiple times

---
v3:
- drop migration compat code

PS:
I don't have access to a suitable system to test it.

---
 hw/s390x/s390-virtio-ccw.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -18
index 0c03ffb7c7..e30058df38 100644
@@ -154,27 +154,12 @@ static void virtio_ccw_register_hcalls(void)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
-    ram_addr_t chunk, offset = 0;
-    unsigned int number = 0;
+    MemoryRegion *ram = g_new(MemoryRegion, 1);
     Error *local_err = NULL;
-    gchar *name;
 
     /* allocate RAM for core */
-    name = g_strdup_printf("s390.ram");
-    while (mem_size) {
-        MemoryRegion *ram = g_new(MemoryRegion, 1);
-        uint64_t size = mem_size;
-
-        /* KVM does not allow memslots >= 8 TB */
-        chunk = MIN(size, KVM_SLOT_MAX_BYTES);
-        memory_region_allocate_system_memory(ram, NULL, name, chunk);
-        memory_region_add_subregion(sysmem, offset, ram);
-        mem_size -= chunk;
-        offset += chunk;
-        g_free(name);
-        name = g_strdup_printf("s390.ram.%u", ++number);
-    }
-    g_free(name);
+    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
+    memory_region_add_subregion(sysmem, 0, ram);
 
Quoted 8 lines

hw/s390x/s390-virtio-ccw.c static void virtio_ccw_register_hcalls(void) · line 165

Quoted 3 lines
hw/s390x/s390-virtio-ccw.c +0 -0
     /*
      * Configure the maximum page size. As no memory devices were created

We have to remember putting it into the next release notes. (or do we even want to get this into v4.1 ?)

This is too invasive for 4.1

#6 lore raw

Re: [Qemu-devel] [PATCH for-4.2 v3 2/2] s390: do not call memory_region_allocate_system_memory() multiple times

Quoted 78 lines · Christian Borntraeger
Quoted 75 lines · David Hildenbrand
Quoted 61 lines · Igor Mammedov

s390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once.

Beside an invalid use of API, the approach also introduced migration issue, since RAM chunks for each KVM_SLOT_MAX_BYTES are transferred in migration stream as separate RAMBlocks.

After discussion [1], it was agreed to break migration from older QEMU for guest with RAM >8Tb (as it was relatively new (since 2.12) and considered to be not actually used downstream). Migration should keep working for guests with less than 8TB and for more than 8TB with QEMU 4.2 and newer binary.
In case user tries to migrate more than 8TB guest, between incompatible QEMU versions, migration should fail gracefully due to non-exiting RAMBlock ID or RAMBlock size mismatch.

Taking in account above and that now KVM code is able to split too big MemorySection into several memslots, stop abusing
memory_region_allocate_system_memory()
and use only one memory region for RAM.

1) [PATCH RFC v2 4/4] s390: do not call memory_region_allocate_system_memory() multiple times

---
v3:
- drop migration compat code

PS:
I don't have access to a suitable system to test it.

---
 hw/s390x/s390-virtio-ccw.c | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)
hw/s390x/s390-virtio-ccw.c +1 -13
index 0c03ffb7c7..e30058df38 100644
@@ -154,27 +154,12 @@ static void virtio_ccw_register_hcalls(void)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
-    ram_addr_t chunk, offset = 0;
-    unsigned int number = 0;
+    MemoryRegion *ram = g_new(MemoryRegion, 1);
     Error *local_err = NULL;
-    gchar *name;
 
     /* allocate RAM for core */
-    name = g_strdup_printf("s390.ram");
-    while (mem_size) {
-        MemoryRegion *ram = g_new(MemoryRegion, 1);
-        uint64_t size = mem_size;
-
-        /* KVM does not allow memslots >= 8 TB */
-        chunk = MIN(size, KVM_SLOT_MAX_BYTES);
-        memory_region_allocate_system_memory(ram, NULL, name, chunk);
-        memory_region_add_subregion(sysmem, offset, ram);
-        mem_size -= chunk;

hw/s390x/s390-virtio-ccw.c static void virtio_ccw_register_hcalls(void) · line 165

Quoted 10 lines
hw/s390x/s390-virtio-ccw.c +2 -5
-        offset += chunk;
-        g_free(name);
-        name = g_strdup_printf("s390.ram.%u", ++number);
-    }
-    g_free(name);
+    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
+    memory_region_add_subregion(sysmem, 0, ram);
 
     /*
      * Configure the maximum page size. As no memory devices were created
Quoted 5 lines
Quoted 2 lines

We have to remember putting it into the next release notes. (or do we even want to get this into v4.1 ?)

This is too invasive for 4.1

Makes sense

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw
In reply to Igor Mammedov
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with # HEAD pointing to a commit that has the patches applied on top of "base"

# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up

# make check -j4 V=1
make install
=== TEST SCRIPT END ===

CC s390x-softmmu/target/s390x/sigp.o
CC s390x-softmmu/target/s390x/kvm.o
/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c: In function ‘kvm_arch_init’:
/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c:350:5: error: implicit declaration of function ‘kvm_set_max_memslot_size’; did you mean ‘kvm_get_max_memslots’? [-Werror=implicit-function-declaration]

  350 |     kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~
      |     kvm_get_max_memslots

/var/tmp/patchew-tester-tmp-bcfxbkeg/src/target/s390x/kvm.c:350:5: error: nested extern declaration of ‘kvm_set_max_memslot_size’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[1]: *** [/var/tmp/patchew-tester-tmp-bcfxbkeg/src/rules.mak:69: target/s390x/kvm.o] Error 1
make: *** [Makefile:472: s390x-softmmu/all] Error 2

The full log is available at
http://patchew.org/logs/[email protected]/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#8 lore raw
In reply to Igor Mammedov
Quoted 38 lines · Igor Mammedov

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Quoted 6 lines

With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system.

Quoted 13 lines

Igor Mammedov (2):
kvm: s390: split too big memory section on several memslots
s390: do not call memory_region_allocate_system_memory() multiple
times

 include/hw/s390x/s390-virtio-ccw.h | 10 ++++
 include/sysemu/kvm_int.h           |  1 +
 accel/kvm/kvm-all.c                | 79 ++++++++++++++++++------------
 hw/s390x/s390-virtio-ccw.c         | 30 ++----------
 target/s390x/kvm.c                 |  1 +
 5 files changed, 63 insertions(+), 58 deletions(-)
#9 lore raw
Quoted 46 lines · Christian Borntraeger
Quoted 38 lines · Igor Mammedov

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Quoted 6 lines
Quoted 3 lines

With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system.

I have to correct myself. The 9TB guest started up but it does not seem to do anything useful (it hangs).

I will try to investigate.

Quoted 13 lines
Quoted 13 lines

Igor Mammedov (2):
kvm: s390: split too big memory section on several memslots
s390: do not call memory_region_allocate_system_memory() multiple
times

 include/hw/s390x/s390-virtio-ccw.h | 10 ++++
 include/sysemu/kvm_int.h           |  1 +
 accel/kvm/kvm-all.c                | 79 ++++++++++++++++++------------
 hw/s390x/s390-virtio-ccw.c         | 30 ++----------
 target/s390x/kvm.c                 |  1 +
 5 files changed, 63 insertions(+), 58 deletions(-)
#10 lore raw
Quoted 49 lines · Christian Borntraeger
Quoted 46 lines · Christian Borntraeger
Quoted 44 lines · Igor Mammedov

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Quoted 6 lines
Quoted 3 lines

With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system.

I have to correct myself. The 9TB guest started up but it does not seem to do anything useful (it hangs).

Seems that the userspace addr is wrong (its the same).
[pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0 [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0

#11 lore raw
Quoted 54 lines · On Fri, 2 Aug 2019 17:04:21 +0200 Christian Borntraeger…
Quoted 53 lines · Christian Borntraeger
Quoted 49 lines · Christian Borntraeger
Quoted 38 lines · Igor Mammedov

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review
- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Quoted 6 lines

With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system.

Quoted 6 lines
Quoted 2 lines

I have to correct myself. The 9TB guest started up but it does not seem to do anything useful (it hangs).

Seems that the userspace addr is wrong (its the same).
[pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0 [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0

It's a bug in 1/2, I forgot to advance mem->ram along with mem->start_addr. Let me fix it and simulate it on small s390 host (/me sorry for messy patches) it won't test migration properly but should be sufficient for testing KVM code patch.

#12 lore raw
Quoted 6 lines · On Mon, 5 Aug 2019 10:54:40 +0200 Igor Mammedov <[email protected]> wrote:
Quoted 3 lines · On Fri, 2 Aug 2019 17:04:21 +0200 Christian Borntraeger…
Quoted 2 lines · Christian Borntraeger
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Christian Borntraeger
Quoted 5 lines · Igor Mammedov
Quoted 4 lines

Changelog:
since v2:
- break migration from old QEMU (since 2.12-4.1) for guest with >8TB RAM and drop migratable aliases patch as was agreed during v2 review

FWIW, that seems reasonable to me as well.

Quoted 48 lines
Quoted 48 lines
Quoted 47 lines
Quoted 41 lines
Quoted 34 lines

- drop 4.2 machines patch as it's not prerequisite anymore
since v1:
- include 4.2 machines patch for adding compat RAM layout on top
- 2/4 add missing in v1 patch for splitting too big MemorySection on several memslots
- 3/4 amend code path on alias destruction to ensure that RAMBlock is cleaned properly
- 4/4 add compat machine code to keep old layout (migration-wise) for 4.1 and older machines

While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).

s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.

Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES. During v2 review it was agreed to ignore migration breakage (documenting it in release notes) and leaving only KVM fix.

In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit, following was done:
* [1/2] split too big RAM chunk inside of KVM code on several memory slots if necessary
* [2/2] drop manual ram splitting in s390 code

Quoted 6 lines
Quoted 3 lines

With the fixup this patch set seems to work on s390. I can start 9TB guests and I can migrate smaller guests between 4.1+patch and 4.0 and 3.1. I currently can not test migration for the 9TB guest due to lack of a 2nd system.

I have to correct myself. The 9TB guest started up but it does not seem to do anything useful (it hangs).

Quoted 8 lines
Quoted 3 lines

Seems that the userspace addr is wrong (its the same).
[pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=0, guest_phys_addr=0, memory_size=8796091973632, userspace_addr=0x3fff7d00000}) = 0 [pid 258234] ioctl(10, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0x7fffff00000, memory_size=1099512676352, userspace_addr=0x3fff7d00000}) = 0

It's a bug in 1/2, I forgot to advance mem->ram along with mem->start_addr. Let me fix it and simulate it on small s390 host (/me sorry for messy patches) it won't test migration properly but should be sufficient for testing KVM code patch.

Ok, I'll wait for a v4 before I spend any time on this :)