s390: stop abusing memory_region_allocate_system_memory()
cover s390: stop abusing memory_region_allocate_system_memory()
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring memory_region_allocate_system_memory() API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).
s390 is one of such boards where KVM limitation on memslot size got propagated to board design and memory_region_allocate_system_memory() was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.
Unfortunately, memory_region_allocate_system_memory() usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more than KVM_SLOT_MAX_BYTES.
In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM on KVM_SLOT_MAX_BYTES chunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessary
I haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (cd5ff8333a) dropping '-numa node,mem' completely even for old machines.
PS:
Tested with ping pong cross version migration on s390 machine (with reduced KVM_SLOT_MAX_BYTES since I don't have access to large
enough host)
Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not call memory_region_allocate_system_memory() multiple
times
exec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
Signature · 1 line
2.18.1
RFC PATCH 1/2 memory: make MemoryRegion alias migratable
use qemu_ram_alloc_from_ptr() to create aliased RAMBlock to the part of original memory region.
--- exec.c | 7 ++++--- memory.c | 5 +++++ 2 files changed, 9 insertions(+), 3 deletions(-)
exec.c +4 -3
@@ -2313,7 +2313,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared) new_block->used_length, DIRTY_CLIENTS_ALL); -if (new_block->host) { +if (new_block->host && !new_block->mr->alias) {qemu_ram_setup_dump (new_block->host, new_block->max_length);qemu_madvise (new_block->host, new_block->max_length,QEMU_MADV_HUGEPAGE );/* MADV_DONTFORK is also needed by KVM in absence of synchronous MMU */
@@ -2671,7 +2671,8 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,rcu_read_lock (); block =atomic_rcu_read (&ram_list.mru_block); -if (block && block->host && host - block->host < block->max_length) { +if (block && !block->mr->alias && block->host && + host - block->host < block->max_length) { goto found; }
@@ -2680,7 +2681,7 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,if (block->host ==NULL ) {continue ; } -if (host - block->host < block->max_length) { +if (!block->mr->alias && host - block->host < block->max_length) { goto found; } }
memory.c +5 -0
@@ -1678,6 +1678,11 @@ void memory_region_init_alias(MemoryRegion *mr,memory_region_init (mr, owner, name, size); mr->alias = orig; mr->alias_offset = offset; +if (orig->ram_block && size) { + mr->ram_block =qemu_ram_alloc_from_ptr (size, + orig->ram_block->host + offset, + mr, &error_fatal); + } } void memory_region_init_rom_nomigrate(MemoryRegion *mr,
Signature · 1 line
2.18.1
Re: [Qemu-devel] [PATCH RFC 1/2] memory: make MemoryRegion alias migratable
Quoted 46 lines · * Igor Mammedov ([email protected]) wrote:
use
qemu_ram_alloc_from_ptr()to create aliased RAMBlock to the part of original memory region.Signed-off-by Igor Mammedov <[email protected]>--- exec.c | 7 ++++--- memory.c | 5 +++++ 2 files changed, 9 insertions(+), 3 deletions(-)exec.c +4
-3@@ -2313,7 +2313,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared) new_block->used_length, DIRTY_CLIENTS_ALL); -if (new_block->host) { +if (new_block->host && !new_block->mr->alias) {qemu_ram_setup_dump (new_block->host, new_block->max_length);qemu_madvise (new_block->host, new_block->max_length,QEMU_MADV_HUGEPAGE );/* MADV_DONTFORK is also needed by KVM in absence of synchronous MMU */ @@ -2671,7 +2671,8 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,rcu_read_lock (); block =atomic_rcu_read (&ram_list.mru_block); -if (block && block->host && host - block->host < block->max_length) { +if (block && !block->mr->alias && block->host && + host - block->host < block->max_length) { goto found; }@@ -2680,7 +2681,7 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,if (block->host ==NULL ) {continue ; } -if (host - block->host < block->max_length) { +if (!block->mr->alias && host - block->host < block->max_length) { goto found; } }memory.c +0
-0
memory.c void memory_region_init_alias(MemoryRegion *mr, · line 1686
Quoted 9 lines
memory.c +5
-0@@ -1678,6 +1678,11 @@ void memory_region_init_alias(MemoryRegion *mr,memory_region_init (mr, owner, name, size); mr->alias = orig; mr->alias_offset = offset; +if (orig->ram_block && size) { + mr->ram_block =qemu_ram_alloc_from_ptr (size, + orig->ram_block->host + offset, + mr, &error_fatal); + } }
Doesn't this cause new memory regions to be created in other existing machines, e.g. x86's mem-smram, or the various PCI vga hacks?
Dave
Quoted 5 lines
memory.c +0
-0void memory_region_init_rom_nomigrate(MemoryRegion *mr,Signature · 1 line
2.18.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH RFC 1/2] memory: make MemoryRegion alias migratable
Quoted 51 lines · On Mon, 29 Jul 2019 18:53:15 +0100 "Dr. David Alan Gilbert"…
Quoted 50 lines · * Igor Mammedov ([email protected]) wrote:
use
qemu_ram_alloc_from_ptr()to create aliased RAMBlock to the part of original memory region.Signed-off-by Igor Mammedov <[email protected]>--- exec.c | 7 ++++--- memory.c | 5 +++++ 2 files changed, 9 insertions(+), 3 deletions(-)exec.c +4
-3@@ -2313,7 +2313,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared) new_block->used_length, DIRTY_CLIENTS_ALL); -if (new_block->host) { +if (new_block->host && !new_block->mr->alias) {qemu_ram_setup_dump (new_block->host, new_block->max_length);qemu_madvise (new_block->host, new_block->max_length,QEMU_MADV_HUGEPAGE );/* MADV_DONTFORK is also needed by KVM in absence of synchronous MMU */ @@ -2671,7 +2671,8 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,rcu_read_lock (); block =atomic_rcu_read (&ram_list.mru_block); -if (block && block->host && host - block->host < block->max_length) { +if (block && !block->mr->alias && block->host && + host - block->host < block->max_length) { goto found; }@@ -2680,7 +2681,7 @@ RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset,if (block->host ==NULL ) {continue ; } -if (host - block->host < block->max_length) { +if (!block->mr->alias && host - block->host < block->max_length) { goto found; } }memory.c +1
-0@@ -1678,6 +1678,11 @@ void memory_region_init_alias(MemoryRegion *mr,memory_region_init (mr, owner, name, size); mr->alias = orig; mr->alias_offset = offset; +if (orig->ram_block && size) {
Quoted 8 lines
memory.c void memory_region_init_alias(MemoryRegion *mr, · line 1686
Quoted 5 lines
memory.c +4
-0+ mr->ram_block =qemu_ram_alloc_from_ptr (size, + orig->ram_block->host + offset, + mr, &error_fatal); + } }Doesn't this cause new memory regions to be created in other existing machines, e.g. x86's mem-smram, or the various PCI vga hacks?
I'd guess you've meant RAMBlocks instead of memory regions, if that's it then yes, every alias pointing to RAM backed memory region will have RAMBlock that's points to aliased part of aliased memory region.
(I didn't not intended to limit it to s390 only as it potentially
could be used for x86 initial memory refactoring as well)
If it's issue we probably can address that by adding additional API
memory_region_alias_make_migratable()
and call it on selected aliases, but the less APIs we have the better so I went for reusing exiting vmstate_register_ram_global().
(I admit faking RAMBlock is a hack (albeit working one),
but my attempt to teach migration code to accept aliases directly didn't work for me well and was very fragile, I don't remember specifics any more and probably code as well since it was utter failure)
PS:
this patch is by no means a complete one (I probably cut corners on cleanup path) but should be sufficient to demo idea and see if it's acceptable.
Quoted 8 lines
Dave
Quoted 5 lines
memory.c +0
-0void memory_region_init_rom_nomigrate(MemoryRegion *mr,Signature · 1 line
2.18.1--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Qemu-devel] [PATCH RFC 1/2] memory: make MemoryRegion alias migratable
Quoted 3 lines · Igor Mammedov
I'd guess you've meant RAMBlocks instead of memory regions, if that's it then yes, every alias pointing to RAM backed memory region will have RAMBlock that's points to aliased part of aliased memory region.
The question is just, does it break migration from old QEMU to new QEMU on x86 (which has plenty of RAM-backed aliases)? If not, explain that in the commit message or in a code comment.
Paolo
Re: [Qemu-devel] [PATCH RFC 1/2] memory: make MemoryRegion alias migratable
Quoted 8 lines · On Tue, 30 Jul 2019 15:34:54 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 3 lines · Igor Mammedov
I'd guess you've meant RAMBlocks instead of memory regions, if that's it then yes, every alias pointing to RAM backed memory region will have RAMBlock that's points to aliased part of aliased memory region.
The question is just, does it break migration from old QEMU to new QEMU on x86 (which has plenty of RAM-backed aliases)? If not, explain that in the commit message or in a code comment.
pc/q35 old<->new new<->old ping pong tests worked fine (qemu with default devices).
My understanding was that one needs to call vmstate_register_ram() to make memory region migratable, which marks specific RAMBlock as migratable. So just assigning new RAMBlock to mr->ram_block shouldn't affect migration, unless vmstate_register_ram() is called on alias.
I'll add it to commit message.
Quoted 3 lines
Paolo
Re: [Qemu-devel] [PATCH RFC 1/2] memory: make MemoryRegion alias migratable
Quoted 12 lines · * Igor Mammedov ([email protected]) wrote:
Quoted 8 lines · On Tue, 30 Jul 2019 15:34:54 +0200 Paolo Bonzini <[email protected]> wrote:
Quoted 3 lines · Igor Mammedov
I'd guess you've meant RAMBlocks instead of memory regions, if that's it then yes, every alias pointing to RAM backed memory region will have RAMBlock that's points to aliased part of aliased memory region.
The question is just, does it break migration from old QEMU to new QEMU on x86 (which has plenty of RAM-backed aliases)? If not, explain that in the commit message or in a code comment.
Quoted 6 lines
pc/q35 old<->new new<->old ping pong tests worked fine (qemu with default devices).
My understanding was that one needs to call
vmstate_register_ram()to make memory region migratable, which marks specific RAMBlock as migratable. So just assigning new RAMBlock tomr->ram_blockshouldn't affect migration, unlessvmstate_register_ram()is called on alias.
You're right; that's actually relatively new and I forget about it.
From Cedric's b895de50271 in 3.0, before that we migrated every
RAMBlock.
So yes, I think you should be OK.
Dave
Quoted 6 lines
I'll add it to commit message.
Quoted 3 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
RFC PATCH 2/2 s390: do not call memory_region_allocate_system_memory() multiple times
In reply to Igor Mammedovs390 was trying to solve limited KVM memslot size issue by abusing memory_region_allocate_system_memory(), which breaks API contract where the function might be called only once.
s390 should have used memory aliases to fragment inital memory into smaller chunks to satisfy KVM's memslot limitation. But its a bit late now, since allocated pieces are transfered in migration stream separately, so it's not possible to just replace broken layout with correct one. To workaround previous patch made MemoryRegion alases migratable and this patch switches to use them to split big initial RAM chunk into smaller pieces up to KVM_SLOT_MAX_BYTES each and registers aliases for migration. That should keep migration and guest ABI (RAM mapping) compatible with previous QEMU versions.
---
I don't have access to a suitable system to test it, so I've simulated it with smaller chunks on x84 host. Ping-pong migration between old and new QEMU worked fine. KVM part should be fine as memslots using mapped MemoryRegions (in this case it would be aliases) as far as I know but is someone could test it on big enough host it would be nice.
--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
hw/s390x/s390-virtio-ccw.c +15 -5
@@ -163,21 +163,31 @@ static void virtio_ccw_register_hcalls(void)static void s390_memory_init (ram_addr_t mem_size) {MemoryRegion *sysmem =get_system_memory (); +MemoryRegion *ram =g_new (MemoryRegion,1 );ram_addr_t chunk, offset =0 ;unsigned int number =0 ;Error *local_err =NULL ;gchar *name;/* allocate RAM for core */ +memory_region_allocate_system_memory (ram,NULL ,"s390.whole.ram" , mem_size); +/* +* memory_region_allocate_system_memory() registers allocated RAM for +* migration, however for compat reasons the RAM should be passed over +* as RAMBlocks of the size upto KVM_SLOT_MAX_BYTES. So unregister just +* allocated RAM so it won't be migrated directly. Aliases will take care +* of segmenting RAM into legacy chunks that migration and KVM compatible. +*/ +vmstate_unregister_ram (ram,NULL ); name =g_strdup_printf ("s390.ram" );while (mem_size) { -MemoryRegion *ram =g_new (MemoryRegion,1 ); -uint64_t size = mem_size; +MemoryRegion *alias =g_new (MemoryRegion,1 );/* KVM does not allow memslots >= 8 TB */ - chunk =MIN (size,KVM_SLOT_MAX_BYTES ); -memory_region_allocate_system_memory (ram,NULL , name, chunk); -memory_region_add_subregion (sysmem, offset, ram); + chunk =MIN (mem_size,KVM_SLOT_MAX_BYTES ); +memory_region_init_alias (alias,NULL , name, ram, offset, chunk); +vmstate_register_ram_global (alias); +memory_region_add_subregion (sysmem, offset, alias); mem_size -= chunk; offset += chunk;g_free (name);
Signature · 1 line
2.18.1
On Mon, 29 Jul 2019 10:52:27 -0400
Igor Mammedov <[email protected]> wrote:
Quoted 44 lines
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
I remember that you send a similar patch a while ago and something broke on s390x. Have you changed something from the old patchs set?
Quoted 44 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
Quoted 2 lines · On Tue, 30 Jul 2019 17:22:01 +0200 Christian Borntraeger…
I remember that you send a similar patch a while ago and something broke on s390x. Have you changed something from the old patchs set?
Thanks for reminder, I totally forgot about it.
it was "[PATCH v1 5/5] s390: do not call memory_region_allocate_system_memory() multiple times" now looking at history it all comes back, so this series is incomplete as is
due to memory memory region aliases being merged back to one big memory section after flatview is rendered. So KVM get too big chunk of RAM and it breaks.
So aliases solve only half of the problem (keeping migration side working) and to fix KVM side, I'd add splitting memory section on chunks into kvm_set_phys_mem() to keep KVM specifics to kvm code only.
Board would only have to set max size value and kvm code would use it for splitting, I'll try to find that patch.
Quoted 47 lines · Igor Mammedov
Quoted 44 lines
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
Quoted 7 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).
Quoted 7 lines
s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.
So if I understand it correctly, we only call memory_region_allocate_system_memory() in case the guest initial memory size exceeds KVM_SLOT_MAX_BYTES - ~8TB.
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
Quoted 30 lines
In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
Signature · 3 lines
Thanks, David / dhildenb
Quoted 20 lines · David Hildenbrand
Quoted 7 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).Quoted 7 lines
s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.
Quoted 8 lines
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
For reference:
https://access.redhat.com/articles/rhel-kvm-limits
RHEL 7/8 supports up to 2TB maximum memory in KVM guests.
https://www.suse.com/releasenotes/s390x/SUSE-SLES/15-SP1/
SLES 15 seems to support up to 4TB in a VM
If migration failing would mean "abort migration, continue on migration source", I could sleep good at night. (as long as nothing crashes)
Signature · 3 lines
Thanks, David / dhildenb
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.
Quoted 7 lines
Quoted 3 lines
Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.
(to clarify, I meant: call it more than once)
Signature · 3 lines
Thanks, David / dhildenb
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.
Quoted 7 lines
Quoted 3 lines
Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.
We always call it. We just call it twice for > 8TB
Quoted 9 lines
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
For the time being we can block migration for guests > 8TB if that helps (it should not fail in a guest killing fashion), but we should
1. continue to be able to migrate guests < 8TB
2. continue to be
On the other hand I find "and suddenly it fails if you go beyond this" really unpleasant. So it would be interesting to see the next round of patches to check how "hacky" those really are.
Quoted 33 lines
Quoted 31 lines
In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)
Quoted 19 lines · Christian Borntraeger
Quoted 16 lines · David Hildenbrand
Quoted 14 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.
Quoted 5 lines
Quoted 3 lines
So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.We always call it. We just call it twice for > 8TB
Yeah, that's what I meant.
Quoted 10 lines
Quoted 9 lines
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
Quoted 8 lines
For the time being we can block migration for guests > 8TB if that helps (it should not fail in a guest killing fashion), but we should
1. continue to be able to migrate guests < 8TB
2. continue to beOn the other hand I find "and suddenly it fails if you go beyond this" really unpleasant. So it would be interesting to see the next round of patches to check how "hacky" those really are.
I mean migration will work perfectly fine once we fixed it for new QEMU versions. It's only the older QEMU versions to/from the > fixed one.
Looking at the log I can see that this was introduced with v2.12.0.
I would document this in the next release notes: "Migration of unusual big VMs (>= 8TB) will not work from/to previous QEMU versions (up to v2.12, before that starting such guests didn't even work)."
Signature · 3 lines
Thanks, David / dhildenb
Quoted 43 lines · David Hildenbrand
Quoted 19 lines · Christian Borntraeger
Quoted 16 lines · David Hildenbrand
Quoted 14 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.Quoted 5 lines
Quoted 3 lines
So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.We always call it. We just call it twice for > 8TB
Yeah, that's what I meant.
Quoted 15 lines
Quoted 9 lines
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
For the time being we can block migration for guests > 8TB if that helps (it should not fail in a guest killing fashion), but we should
1. continue to be able to migrate guests < 8TB
2. continue to be
Quoted 6 lines
Quoted 3 lines
On the other hand I find "and suddenly it fails if you go beyond this" really unpleasant. So it would be interesting to see the next round of patches to check how "hacky" those really are.
I mean migration will work perfectly fine once we fixed it for new QEMU versions. It's only the older QEMU versions to/from the > fixed one.
I think that would be fine. We just have to make sure that migration really fails in a way that the system continues to run.
Quoted 6 lines
Looking at the log I can see that this was introduced with v2.12.0.
I would document this in the next release notes: "Migration of unusual big VMs (>= 8TB) will not work from/to previous QEMU versions (up to v2.12, before that starting such guests didn't even work)."
Yes, sounds good.
Quoted 32 lines · On Fri, 2 Aug 2019 10:29:43 +0200 Christian Borntraeger…
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Igor Mammedov
While looking into unifying guest RAM allocation to use hostmem backends for initial RAM (especially when -mempath is used) and retiring
memory_region_allocate_system_memory()API, leaving only single hostmem backend, I was inspecting how currently it is used by boards and it turns out several boards abuse it by calling the function several times (despite documented contract forbiding it).s390 is one of such boards where KVM limitation on memslot size got propagated to board design and
memory_region_allocate_system_memory()was abused to satisfy KVM requirement for max RAM chunk where memory region alias would suffice.Quoted 7 lines
Quoted 3 lines
Unfortunately,
memory_region_allocate_system_memory()usage created migration dependency where guest RAM is transferred in migration stream as several RAMBlocks if it's more thanKVM_SLOT_MAX_BYTES.So if I understand it correctly, we only call
memory_region_allocate_system_memory()in case the guest initial memory size exceedsKVM_SLOT_MAX_BYTES- ~8TB.We always call it. We just call it twice for > 8TB
Quoted 9 lines
Do we *really* care about keeping migration of systems running that most probably nobody (except Christian ;) ) really uses? (especially not in production).
I am fine keeping migration running if it's easy, but introducing hacks (reading below) for such obscure use cases - I don't know.
@Christian: Please prove me wrong. :)
Quoted 8 lines
For the time being we can block migration for guests > 8TB if that helps (it should not fail in a guest killing fashion), but we should
1. continue to be able to migrate guests < 8TB
2. continue to beOn the other hand I find "and suddenly it fails if you go beyond this" really unpleasant. So it would be interesting to see the next round of patches to check how "hacky" those really are.
I've looked into cleaning up "migratable aliases",
so far it works fine (well in configurations I was able to test, there were no regressions in x86 machines which use aliases quite a bit). As I've written before, aliases could be used for x86 later but that yet to be proved, so I'd prefer to delay this patch if possible.
However, I'd prefer to intentionally break migration with >8TB guests to simpler code without keepeing around compat mode that won't/isn't used in practice.
As for new round of patches (including missing KVM fixup), I'm going to post them now-ish for you to check it out.
Quoted 35 lines
Quoted 34 lines
Quoted 32 lines
In order to replace these several RAM chunks with a single memdev and keep it working with KVM memslot size limit and migration compatible, following was done:
* [2/2] use memory region aliases to partition hostmem backend RAM onKVM_SLOT_MAX_BYTESchunks, which should keep KVM side working
* [1/2] hacked memory region aliases (to ram memory regions only) to have its own RAMBlocks pointing to RAM chunks owned by aliased memory region. While it's admittedly a hack, but it's relatively simple and allows board code rashape migration stream as necessaryI haven't tried to use migratable aliases on x86 machines, but with it it could be possible to drop legacy RAM allocation and compat knob (
cd5ff8333a) dropping '-numa node,mem' completely even for old machines.PS:
Tested with ping pong cross version migration on s390 machine (with reducedKVM_SLOT_MAX_BYTESsince I don't have access to large
enough host)Igor Mammedov (2):
memory: make MemoryRegion alias migratable
s390: do not callmemory_region_allocate_system_memory()multiple
timesexec.c | 7 ++++--- hw/s390x/s390-virtio-ccw.c | 20 +++++++++++++++----- memory.c | 5 +++++ 3 files changed, 24 insertions(+), 8 deletions(-)