migration/ram: background snapshot fixes and optimiations
cover migration/ram: background snapshot fixes and optimiations
Playing with background snapshots in combination with hugetlb and virtio-mem, I found two issues and some reasonable optimizations (skip unprotecting when unregistering).
With virtio-mem (RamDiscardManager), we now won't be allocating unnecessary page tables for unplugged ranges when using uffd-wp with shmem/hugetlb.
David Hildenbrand (5):
migration/ram: Fix populate_read_range()
migration/ram: Fix error handling in ram_write_tracking_start() migration/ram: Don't explicitly unprotect when unregistering uffd-wp migration/ram: Rely on used_length for uffd_change_protection() migration/ram: Optimize ram_write_tracking_start() for
RamDiscardManager
migration/ram.c | 54 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 14 deletions(-)
Signature · 1 line
2.39.0
PATCH 1/5 migration/ram: Fix populate_read_range()
Unfortunately, commit f7b9dcfbcf44 broke populate_read_range(): the loop end condition is very wrong, resulting in that function not populating the full range. Lets' fix that.
--- migration/ram.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
migration/ram.c +3 -1
@@ -1774,13 +1774,15 @@ out:static inline void populate_read_range (RAMBlock *block,ram_addr_t offset,ram_addr_t size) { +const ram_addr_t end = offset + size; +/* * We read one byte of each page; this will preallocate page tables if * required and populate the shared zeropage on MAP_PRIVATE anonymous memory * where no page was populated yet. This might require adaption when * supporting other mappings, like shmem. */ -for (; offset < size; offset += block->page_size) { +for (; offset < end; offset += block->page_size) {char tmp = *((char *)block->host + offset);/* Don't optimize the read out */
Signature · 1 line
2.39.0
Re: [PATCH v1 1/5] migration/ram: Fix populate_read_range()
Quoted 7 lines · David Hildenbrand
Unfortunately, commit
f7b9dcfbcf44brokepopulate_read_range(): the loop end condition is very wrong, resulting in that function not populating the full range. Lets' fix that.Fixes f7b9dcfbcf44 ("migration/ram: Factor out populating pages readable in ram_block_populate_pages()")Signed-off-by David Hildenbrand <[email protected]>
PATCH 2/5 migration/ram: Fix error handling in ram_write_tracking_start()
In reply to David HildenbrandIf something goes wrong during uffd_change_protection(), we would miss to unregister uffd-wp and not release our reference. Fix it by performing the uffd_change_protection(true) last.
Note that a uffd_change_protection(false) on the recovery path without a prior uffd_change_protection(false) is fine.
--- migration/ram.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
migration/ram.c +3 -2
@@ -1896,13 +1896,14 @@ int ram_write_tracking_start(void) block->max_length, UFFDIO_REGISTER_MODE_WP, NULL)) { goto fail; } + block->flags |=RAM_UF_WRITEPROTECT ; +memory_region_ref (block->mr); +/* Apply UFFD write protection to the block memory range */ if (uffd_change_protection (rs->uffdio_fd, block->host, block->max_length, true, false)) { goto fail; } - block->flags |=RAM_UF_WRITEPROTECT ; -memory_region_ref (block->mr);trace_ram_write_tracking_ramblock_start (block->idstr, block->page_size, block->host, block->max_length);
Signature · 1 line
2.39.0
Re: [PATCH v1 2/5] migration/ram: Fix error handling in ram_write_tracking_start()
Quoted 10 lines · David Hildenbrand
If something goes wrong during
uffd_change_protection(), we would miss to unregister uffd-wp and not release our reference. Fix it by performing theuffd_change_protection(true)last.Note that a
uffd_change_protection(false)on the recovery path without a prioruffd_change_protection(false)is fine.Fixes 278e2f551a09 ("migration: support UFFD write fault processing in ram_save_iterate()")Signed-off-by David Hildenbrand <[email protected]>
PATCH 3/5 migration/ram: Don't explicitly unprotect when unregistering uffd-wp
In reply to David HildenbrandWhen unregistering uffd-wp, older kernels before commit f369b07c86143 ("mm/uffd:reset write protection when unregister with wp-mode") won't clear the uffd-wp PTE bit. When re-registering uffd-wp, the previous uffd-wp PTE bits would trigger again. With above commit, the kernel will clear the uffd-wp PTE bits when unregistering itself.
Consequently, we'll clear the uffd-wp PTE bits now twice -- whereby we don't care about clearing them at all: a new background snapshot will re-register uffd-wp and re-protect all memory either way.
So let's skip the manual clearing of uffd-wp. If ever relevant, we could clear conditionally in uffd_unregister_memory() -- we just need a way to figure out more recent kernels.
--- migration/ram.c | 9 --------- 1 file changed, 9 deletions(-)
migration/ram.c +0 -9
@@ -1918,12 +1918,6 @@ fail:if ((block->flags &RAM_UF_WRITEPROTECT ) ==0 ) {continue ; } -/* -* In case some memory block failed to be write-protected -* remove protection and unregister all succeeded RAM blocks -*/ -uffd_change_protection (rs->uffdio_fd, block->host, block->max_length, - false, false);uffd_unregister_memory (rs->uffdio_fd, block->host, block->max_length);/* Cleanup flags and remove reference */ block->flags &= ~RAM_UF_WRITEPROTECT ;
@@ -1949,9 +1943,6 @@ void ram_write_tracking_stop(void)if ((block->flags &RAM_UF_WRITEPROTECT ) ==0 ) {continue ; } -/* Remove protection and unregister all affected RAM blocks */ -uffd_change_protection (rs->uffdio_fd, block->host, block->max_length, - false, false);uffd_unregister_memory (rs->uffdio_fd, block->host, block->max_length); trace_ram_write_tracking_ramblock_stop(block->idstr, block->page_size,
Signature · 1 line
2.39.0
Re: [PATCH v1 3/5] migration/ram: Don't explicitly unprotect when unregistering uffd-wp
Quoted 10 lines · David Hildenbrand
When unregistering uffd-wp, older kernels before commit
f369b07c86143("mm/uffd:reset write protection when unregister with wp-mode") won't clear the uffd-wp PTE bit. When re-registering uffd-wp, the previous uffd-wp PTE bits would trigger again. With above commit, the kernel will clear the uffd-wp PTE bits when unregistering itself.Consequently, we'll clear the uffd-wp PTE bits now twice -- whereby we don't care about clearing them at all: a new background snapshot will re-register uffd-wp and re-protect all memory either way.
Quoted 5 lines
So let's skip the manual clearing of uffd-wp. If ever relevant, we could clear conditionally in
uffd_unregister_memory()-- we just need a way to figure out more recent kernels.Signed-off-by David Hildenbrand <[email protected]>
Fixing a bug by removing code O:-)
PATCH 4/5 migration/ram: Rely on used_length for uffd_change_protection()
In reply to David Hildenbrandram_mig_ram_block_resized() will abort migration (including background snapshots) when resizing a RAMBlock. ram_block_populate_read() will only populate RAM up to used_length, so at least for anonymous memory protecting everything between used_length and max_length won't actually be protected and is just a NOP.
So let's only protect everything up to used_length.
Note: it still makes sense to register uffd-wp for max_length, such that RAM_UF_WRITEPROTECT is independent of a changing used_length.
--- migration/ram.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
migration/ram.c +1 -1
@@ -1901,7 +1901,7 @@ int ram_write_tracking_start(void)/* Apply UFFD write protection to the block memory range */ if (uffd_change_protection (rs->uffdio_fd, block->host, - block->max_length, true, false)) { + block->used_length, true, false)) { goto fail; }
Signature · 1 line
2.39.0
Re: [PATCH v1 4/5] migration/ram: Rely on used_length for uffd_change_protection()
Quoted 6 lines · David Hildenbrand
ram_mig_ram_block_resized()will abort migration (including background snapshots) when resizing a RAMBlock.ram_block_populate_read()will only populate RAM up toused_length, so at least for anonymous memory protecting everything betweenused_lengthandmax_lengthwon't actually be protected and is just a NOP.
Quoted 6 lines
So let's only protect everything up to
used_length.Note: it still makes sense to register uffd-wp for
max_length, such thatRAM_UF_WRITEPROTECTis independent of a changingused_length.Signed-off-by David Hildenbrand <[email protected]>
PATCH 5/5 migration/ram: Optimize ram_write_tracking_start() for RamDiscardManager
In reply to David Hildenbrandram_block_populate_read() already optimizes for RamDiscardManager. However, ram_write_tracking_start() will still try protecting discarded memory ranges.
Let's optimize, because discarded ranges don't map any pages and
(1) For anonymous memory, trying to protect using uffd-wp without a mapped page is ignored by the kernel and consequently a NOP.
(2) For shared/file-backed memory, we will fill present page tables in the range with PTE markers. However, we will even allocate page tables just to fill them with unnecessary PTE markers and effectively waste memory.
So let's exclude these ranges, just like ram_block_populate_read() already does.
--- migration/ram.c | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
migration/ram.c +34 -2
@@ -1865,6 +1865,39 @@ void ram_write_tracking_prepare(void) } } +static inline int uffd_protect_section (MemoryRegionSection *section, +void *opaque) +{ +const hwaddr size =int128_get64 (section->size); +const hwaddr offset = section->offset_within_region; +RAMBlock *rb = section->mr->ram_block; +int uffd_fd = (uintptr_t )opaque; + +return uffd_change_protection (uffd_fd, rb->host + offset, size, true, + false); +} + +static int ram_block_uffd_protect (RAMBlock *rb,int uffd_fd) +{ +assert (rb->flags &RAM_UF_WRITEPROTECT ); + +/* See ram_block_populate_read() */ +if (rb->mr &&memory_region_has_ram_discard_manager (rb->mr)) { +RamDiscardManager *rdm =memory_region_get_ram_discard_manager (rb->mr); +MemoryRegionSection section = { + .mr = rb->mr, + .offset_within_region =0 , + .size = rb->mr->size, + }; + +return ram_discard_manager_replay_populated (rdm, §ion, + uffd_protect_section, + (void *)(uintptr_t )uffd_fd); + } +return uffd_change_protection (uffd_fd, rb->host, + rb->used_length, true, false); +} +/* * ram_write_tracking_start: start UFFD-WP memory tracking *
@@ -1900,8 +1933,7 @@ int ram_write_tracking_start(void)memory_region_ref (block->mr);/* Apply UFFD write protection to the block memory range */ -if (uffd_change_protection (rs->uffdio_fd, block->host, - block->used_length, true, false)) { +if (ram_block_uffd_protect (block, uffd_fd)) { goto fail; }
Signature · 1 line
2.39.0
Re: [PATCH v1 5/5] migration/ram: Optimize ram_write_tracking_start() for RamDiscardManager
Quoted 14 lines · David Hildenbrand
ram_block_populate_read()already optimizes for RamDiscardManager. However,ram_write_tracking_start()will still try protecting discarded memory ranges.Let's optimize, because discarded ranges don't map any pages and
(1) For anonymous memory, trying to protect using uffd-wp without a mapped page is ignored by the kernel and consequently a NOP.
(2) For shared/file-backed memory, we will fill present page tables in the range with PTE markers. However, we will even allocate page tables just to fill them with unnecessary PTE markers and effectively waste memory.
Quoted 4 lines
So let's exclude these ranges, just like
ram_block_populate_read()already does.Signed-off-by David Hildenbrand <[email protected]>
Quoted 12 lines · David Hildenbrand
Playing with background snapshots in combination with hugetlb and virtio-mem, I found two issues and some reasonable optimizations (skip unprotecting when unregistering).
With virtio-mem (RamDiscardManager), we now won't be allocating unnecessary page tables for unplugged ranges when using uffd-wp with shmem/hugetlb.
Cc Juan Quintela <[email protected]> (maintainer:Migration)Cc "Dr. David Alan Gilbert" <[email protected]> (maintainer:Migration)Cc Peter Xu <[email protected]>Cc Andrey Gruzdev <[email protected]>
Quoted 7 lines
David Hildenbrand (5):
migration/ram: Fixpopulate_read_range()
migration/ram: Fix error handling inram_write_tracking_start()migration/ram: Don't explicitly unprotect when unregistering uffd-wp migration/ram: Rely onused_lengthforuffd_change_protection()migration/ram: Optimizeram_write_tracking_start()for
RamDiscardManager
For the series:
Thanks,
Signature · 1 line
Peter Xu
Quoted 21 lines · Peter Xu
Quoted 12 lines · David Hildenbrand
Playing with background snapshots in combination with hugetlb and virtio-mem, I found two issues and some reasonable optimizations (skip unprotecting when unregistering).
With virtio-mem (RamDiscardManager), we now won't be allocating unnecessary page tables for unplugged ranges when using uffd-wp with shmem/hugetlb.
Cc Juan Quintela <[email protected]> (maintainer:Migration)Cc "Dr. David Alan Gilbert" <[email protected]> (maintainer:Migration)Cc Peter Xu <[email protected]>Cc Andrey Gruzdev <[email protected]>Quoted 7 lines
David Hildenbrand (5):
migration/ram: Fixpopulate_read_range()
migration/ram: Fix error handling inram_write_tracking_start()migration/ram: Don't explicitly unprotect when unregistering uffd-wp migration/ram: Rely onused_lengthforuffd_change_protection()migration/ram: Optimizeram_write_tracking_start()for
RamDiscardManager
Quoted 3 lines
For the series:
Acked-by Peter Xu <[email protected]>
Thanks!
Signature · 3 lines
Thanks, David / dhildenb
Quoted 11 lines · David Hildenbrand
Playing with background snapshots in combination with hugetlb and virtio-mem, I found two issues and some reasonable optimizations (skip unprotecting when unregistering).
With virtio-mem (RamDiscardManager), we now won't be allocating unnecessary page tables for unplugged ranges when using uffd-wp with shmem/hugetlb.
Cc Juan Quintela <[email protected]> (maintainer:Migration)Cc "Dr. David Alan Gilbert" <[email protected]> (maintainer:Migration)Cc Peter Xu <[email protected]>Cc Andrey Gruzdev <[email protected]>
@Juan, David: I can take this via my tree (especially the last patch only optimized virtio-mem for now). Just let me know.
Signature · 3 lines
Thanks, David / dhildenb
Juan Quintela
AppliedQuoted 12 lines · David Hildenbrand
Quoted 10 lines · David Hildenbrand
Playing with background snapshots in combination with hugetlb and virtio-mem, I found two issues and some reasonable optimizations (skip unprotecting when unregistering).
With virtio-mem (RamDiscardManager), we now won't be allocating unnecessary
page tables for unplugged ranges when using uffd-wp with shmem/hugetlb.
Cc: Juan Quintela <[email protected]> (maintainer:Migration)
Cc: "Dr. David Alan Gilbert" <[email protected]> (maintainer:Migration)
Cc: Peter Xu <[email protected]>
Cc: Andrey Gruzdev <[email protected]>
Quoted 2 lines
@Juan, David: I can take this via my tree (especially the last patch only optimized virtio-mem for now). Just let me know.
I reviewed everything now.
Queued on my tree.
Later, Juan.