Optimize zone->contiguous update
Subsystems (6)
v9 cover Optimize zone->contiguous update
This series contains 2 patches. The first one encapsulate the mhp_init_memmap_on_memory() and online_pages() into online_memory_block_pages(), the mhp_deinit_memmap_on_memory() and offline_pages() into offline_memory_block_pages(). Then move most of memory_block_online() to the new function mhp_block_online(struct memory_block *block) and correspondingly memory_block_offline() to mhp_block_offline(struct memory_block *block). The second one add a fast path to check the zone->contiguous.
Changes History
===============
v9 changes:
1. Separate the bug fix and optimization into two patches.
2. This patchset depends on the https://lore.kernel.org/linux-mm/[email protected]/.
3. Refactor the code to move most of memory_block_online() to
mhp_block_online() and correspondingly memory_block_offline() to
mhp_block_offline().
v8 changes:
1. Rebased to 6.19-rc6
2. Add online_memory_block_pages() and offline_memory_block_pages()
v7 changes:
1. Rebased to 6.19-rc1
2. Reorder the patches so that the fix will be the first in the series.
v6 changes:
1. Improve coding style.
2. Add comments.
v5 changes:
1. Improve coding style.
2. Fix a issue in which zone->contiguous was always false when adding new memory, leveraging the fast path optimization.
v4 changes:
1. Improve coding style.
2. Add fast path for zone contiguity check in memory unplugged cases, and update test results.
3. Refactor set_zone_contiguous: the new set_zone_contiguous updates zone contiguity based on the fast path results.
v3 changes:
Add zone contiguity check for empty zones.
v2 changes:
Add check_zone_contiguous_fast function to check zone contiguity for new memory PFN ranges.
Tianyou Li (2):
mm/memory hotplug/unplug: Add online_memory_block_pages() and
offline_memory_block_pages()
mm/memory hotplug/unplug: Optimize zone->contiguous update when
changes pfn range
drivers/base/memory.c | 115 +----------------- include/linux/memory_hotplug.h | 13 +-- include/linux/mm.h | 6 + mm/internal.h | 8 +- mm/memory_hotplug.c | 208 ++++++++++++++++++++++++++++++--- mm/mm_init.c | 15 ++- 6 files changed, 228 insertions(+), 137 deletions(-)
Signature · 1 line
2.47.1
PATCH v9 1/2 mm/memory hotplug/unplug: Add online_memory_block_pages() and offline_memory_block_pages()
Encapsulate the mhp_init_memmap_on_memory() and online_pages() into online_memory_block_pages(). Thus we can further optimize the set_zone_contiguous() to check the whole memory block range, instead of check the zone contiguous in separate range.
Correspondingly, encapsulate the mhp_deinit_memmap_on_memory() and offline_pages() into offline_memory_block_pages().
Furthermore, move most of memory_block_online() to the new function mhp_block_online(struct memory_block *block) and correspondingly memory_block_offline() to mhp_block_offline(struct memory_block *block).
--- drivers/base/memory.c | 115 +--------------------------- include/linux/memory_hotplug.h | 13 +--- include/linux/mm.h | 6 ++ mm/memory_hotplug.c | 132 ++++++++++++++++++++++++++++++++- 4 files changed, 141 insertions(+), 125 deletions(-)
drivers/base/memory.c +3 -112
@@ -209,115 +209,6 @@ int memory_notify(enum memory_block_state state, void *v)return blocking_notifier_call_chain (&memory_chain, state, v); } -#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_MEMORY_HOTPLUG ) -static unsigned long memblk_nr_poison (struct memory_block *mem); -#else -static inline unsigned long memblk_nr_poison (struct memory_block *mem) -{ -return 0 ; -} -#endif - -/* -* Must acquire mem_hotplug_lock in write mode. -*/ -static int memory_block_online (struct memory_block *mem) -{ -unsigned long start_pfn =section_nr_to_pfn (mem->start_section_nr); -unsigned long nr_pages =PAGES_PER_SECTION * sections_per_block; -unsigned long nr_vmemmap_pages =0 ; -struct zone *zone; -int ret; - -if (memblk_nr_poison (mem)) -return -EHWPOISON ; - - zone =zone_for_pfn_range (mem->online_type, mem->nid, mem->group, - start_pfn, nr_pages); - -/* -* Although vmemmap pages have a different lifecycle than the pages -* they describe (they remain until the memory is unplugged), doing -* their initialization and accounting at memory onlining/offlining -* stage helps to keep accounting easier to follow - e.g vmemmaps -* belong to the same zone as the memory they backed. -*/ -if (mem->altmap) - nr_vmemmap_pages = mem->altmap->free; - -mem_hotplug_begin (); -if (nr_vmemmap_pages) { - ret =mhp_init_memmap_on_memory (start_pfn, nr_vmemmap_pages, zone); -if (ret) - goto out; - } - - ret =online_pages (start_pfn + nr_vmemmap_pages, - nr_pages - nr_vmemmap_pages, zone, mem->group); -if (ret) { -if (nr_vmemmap_pages) -mhp_deinit_memmap_on_memory (start_pfn, nr_vmemmap_pages); - goto out; - } - -/* -* Account once onlining succeeded. If the zone was unpopulated, it is -* now already properly populated. -*/ -if (nr_vmemmap_pages) -adjust_present_page_count (pfn_to_page (start_pfn), mem->group, - nr_vmemmap_pages); - - mem->zone = zone; -out: -mem_hotplug_done (); -return ret; -} - -/* -* Must acquire mem_hotplug_lock in write mode. -*/ -static int memory_block_offline (struct memory_block *mem) -{ -unsigned long start_pfn =section_nr_to_pfn (mem->start_section_nr); -unsigned long nr_pages =PAGES_PER_SECTION * sections_per_block; -unsigned long nr_vmemmap_pages =0 ; -int ret; - -if (!mem->zone) -return -EINVAL ; - -/* -* Unaccount before offlining, such that unpopulated zone and kthreads -* can properly be torn down in offline_pages(). -*/ -if (mem->altmap) - nr_vmemmap_pages = mem->altmap->free; - -mem_hotplug_begin (); -if (nr_vmemmap_pages) -adjust_present_page_count (pfn_to_page (start_pfn), mem->group, - -nr_vmemmap_pages); - - ret =offline_pages (start_pfn + nr_vmemmap_pages, - nr_pages - nr_vmemmap_pages, mem->zone, mem->group); -if (ret) { -/* offline_pages() failed. Account back. */ -if (nr_vmemmap_pages) -adjust_present_page_count (pfn_to_page (start_pfn), - mem->group, nr_vmemmap_pages); - goto out; - } - -if (nr_vmemmap_pages) -mhp_deinit_memmap_on_memory (start_pfn, nr_vmemmap_pages); - - mem->zone =NULL ; -out: -mem_hotplug_done (); -return ret; -} -/* * MEMORY_HOTPLUG depends on SPARSEMEM in mm/Kconfig, so it is * OK to have direct references to sparsemem variables in here.
@@ -329,10 +220,10 @@ memory_block_action(struct memory_block *mem, unsigned long action)switch (action) {case MEM_ONLINE : - ret =memory_block_online (mem); + ret =mhp_block_online (mem);break ;case MEM_OFFLINE : - ret =memory_block_offline (mem); + ret =mhp_block_offline (mem);break ;default : WARN(1, KERN_WARNING "%s(%ld, %ld) unknown action: "
@@ -1243,7 +1134,7 @@ void memblk_nr_poison_sub(unsigned long pfn, long i)atomic_long_sub (i, &mem->nr_hwpoison); } -static unsigned long memblk_nr_poison (struct memory_block *mem) +unsigned long memblk_nr_poison (struct memory_block *mem) {return atomic_long_read (&mem->nr_hwpoison); }
include/linux/memory_hotplug.h +4 -9
@@ -12,6 +12,7 @@ struct zone;struct pglist_data ;struct mem_section ;struct memory_group ; +struct memory_block ;struct resource ;struct vmem_altmap ;struct dev_pagemap ;
@@ -106,11 +107,7 @@ extern void adjust_present_page_count(struct page *page,struct memory_group *group, long nr_pages);/* VM interface that may be used by firmware interface */ -extern int mhp_init_memmap_on_memory (unsigned long pfn,unsigned long nr_pages, -struct zone *zone); -extern void mhp_deinit_memmap_on_memory (unsigned long pfn,unsigned long nr_pages); -extern int online_pages (unsigned long pfn,unsigned long nr_pages, -struct zone *zone,struct memory_group *group); +extern int mhp_block_online (struct memory_block *block);extern unsigned long __offline_isolated_pages (unsigned long start_pfn,unsigned long end_pfn);
@@ -261,8 +258,7 @@ static inline void pgdat_resize_init(struct pglist_data *pgdat) {}#ifdef CONFIG_MEMORY_HOTREMOVE extern void try_offline_node (int nid); -extern int offline_pages (unsigned long start_pfn,unsigned long nr_pages, -struct zone *zone,struct memory_group *group); +extern int mhp_block_offline (struct memory_block *block);extern int remove_memory (u64 start,u64 size);extern void __remove_memory (u64 start,u64 size);extern int offline_and_remove_memory (u64 start,u64 size);
@@ -270,8 +266,7 @@ extern int offline_and_remove_memory(u64 start, u64 size);#else static inline void try_offline_node (int nid) {} -static inline int offline_pages (unsigned long start_pfn,unsigned long nr_pages, -struct zone *zone,struct memory_group *group) +static inline int mhp_block_offline (struct memory_block *block) {return -EINVAL ; }
include/linux/mm.h +6 -0
@@ -4377,6 +4377,7 @@ static inline void num_poisoned_pages_sub(unsigned long pfn, long i)#if defined(CONFIG_MEMORY_FAILURE ) && defined(CONFIG_MEMORY_HOTPLUG )extern void memblk_nr_poison_inc (unsigned long pfn);extern void memblk_nr_poison_sub (unsigned long pfn,long i); +extern unsigned long memblk_nr_poison (struct memory_block *mem);#else static inline void memblk_nr_poison_inc (unsigned long pfn) {
@@ -4385,6 +4386,11 @@ static inline void memblk_nr_poison_inc(unsigned long pfn)static inline void memblk_nr_poison_sub (unsigned long pfn,long i) { } + +static inline unsigned long memblk_nr_poison (struct memory_block *mem) +{ +return 0 ; +}#endif #ifndef arch_memory_failure
mm/memory_hotplug.c +128 -4
168 lines not on this page · raw message
Signature · 1 line
2.47.1
PATCH v9 2/2 mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
In reply to Tianyou LiWhen invoke move_pfn_range_to_zone or remove_pfn_range_from_zone, it will update the zone->contiguous by checking the new zone's pfn range from the beginning to the end, regardless the previous state of the old zone. When the zone's pfn range is large, the cost of traversing the pfn range to update the zone->contiguous could be significant.
Add fast paths to quickly detect cases where zone is definitely not contiguous without scanning the new zone. The cases are: when the new range did not overlap with previous range, the contiguous should be false; if the new range adjacent with the previous range, just need to check the new range; if the new added pages could not fill the hole of previous zone, the contiguous should be false.
The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3].
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 33s | 6s | 81% | +----------------+------+---------------+--------------+----------------+
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 34s | 6s | 82% | +----------------+------+---------------+--------------+----------------+
[1] Qemu commands to hotplug 256G/512G memory for a VM:
object_add memory-backend-ram,id=hotmem0,size=256G/512G,share=on device_add virtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory)
[2] Hardware : Intel Icelake server
Guest Kernel : v6.18-rc2
Qemu : v9.0.0
Launch VM :
qemu-system-x86_64 -accel kvm -cpu host \
-drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \
-drive file=./seed.img,format=raw,if=virtio \
-smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \
-m 2G,slots=10,maxmem=2052472M \
-device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \
-device pcie-root-port,id=port2,bus=pcie.0,slot=2 \
-nographic -machine q35 \
-nic user,hostfwd=tcp::3000-:22
Guest kernel auto-onlines newly added memory blocks:
echo online > /sys/devices/system/memory/auto_online_blocks[3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal /proc/meminfo' on Guest reflects that all hotplugged memory is recognized.
--- mm/internal.h | 8 ++++- mm/memory_hotplug.c | 80 ++++++++++++++++++++++++++++++++++++++------- mm/mm_init.c | 15 +++++++-- 3 files changed, 89 insertions(+), 14 deletions(-)
mm/internal.h +7 -1
@@ -730,7 +730,13 @@ static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn,return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone); +enum zone_contig_state { +ZONE_CONTIG_YES , +ZONE_CONTIG_NO , +ZONE_CONTIG_MAYBE , +}; + +void set_zone_contiguous (struct zone *zone,enum zone_contig_state state);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages);
mm/memory_hotplug.c +69 -11
144 lines not on this page · raw message
mm/mm_init.c +13 -2
@@ -2263,11 +2263,22 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) +void set_zone_contiguous (struct zone *zone,enum zone_contig_state state) {unsigned long block_start_pfn = zone->zone_start_pfn;unsigned long block_end_pfn; +/* We expect an earlier call to clear_zone_contiguous(). */ +VM_WARN_ON_ONCE (zone->contiguous); + +if (state ==ZONE_CONTIG_YES ) { + zone->contiguous = true; +return ; + } + +if (state ==ZONE_CONTIG_NO ) +return ; + block_end_pfn =pageblock_end_pfn (block_start_pfn); for (; block_start_pfn < zone_end_pfn(zone); block_start_pfn = block_end_pfn,
@@ -2348,7 +2359,7 @@ void __init page_alloc_init_late(void)shuffle_free_memory (NODE_DATA (nid));for_each_populated_zone (zone) -set_zone_contiguous (zone); +set_zone_contiguous (zone,ZONE_CONTIG_MAYBE );/* Initialize page ext after all struct pages are initialized. */ if (deferred_struct_pages)
Signature · 1 line
2.47.1
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 71 lines · Tianyou Li
When invoke
move_pfn_range_to_zoneorremove_pfn_range_from_zone, it will update thezone->contiguousby checking the new zone's pfn range from the beginning to the end, regardless the previous state of the old zone. When the zone's pfn range is large, the cost of traversing the pfn range to update thezone->contiguouscould be significant.Add fast paths to quickly detect cases where zone is definitely not contiguous without scanning the new zone. The cases are: when the new range did not overlap with previous range, the contiguous should be false; if the new range adjacent with the previous range, just need to check the new range; if the new added pages could not fill the hole of previous zone, the contiguous should be false.
The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3].
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 33s | 6s | 81% | +----------------+------+---------------+--------------+----------------++----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 34s | 6s | 82% | +----------------+------+---------------+--------------+----------------+[1] Qemu commands to hotplug 256G/512G memory for a VM:
object_addmemory-backend-ram,id=hotmem0,size=256G/512G,share=ondevice_addvirtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory)[2] Hardware : Intel Icelake server
Guest Kernel : v6.18-rc2 Qemu : v9.0.0 Launch VM : qemu-system-x86_64 -accel kvm -cpu host \ -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \ -drive file=./seed.img,format=raw,if=virtio \ -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \ -m 2G,slots=10,maxmem=2052472M \ -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \ -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \ -nographic -machine q35 \ -nic user,hostfwd=tcp::3000-:22 Guest kernel auto-onlines newly added memory blocks: echo online > /sys/devices/system/memory/auto_online_blocks[3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal
/proc/meminfo' on Guest reflects that all hotplugged memory is recognized.Reported-by Nanhai Zou <[email protected]>Reported-by Chen Zhang <[email protected]>Tested-by Yuan Liu <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Yu C Chen <[email protected]>Reviewed-by Pan Deng <[email protected]>Reviewed-by Nanhai Zou <[email protected]>Reviewed-by Yuan Liu <[email protected]>Signed-off-by Tianyou Li <[email protected]>
Quoted 1 line
---
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate zone->contiguous + the pageblock walking, and how we could just get rid of it.
You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have zone_spanned_pages == zone_present_pages and the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.
(b) in which case would we have zone_spanned_pages != zone_present_pages and the zone *being* contiguous? I assume in some cases where we have small holes within a pageblock?
Reading the doc of __pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.
I.e., on my notebook I have
$ cat /proc/zoneinfo | grep -E "Node|spanned|present"
Node 0, zone DMA
spanned 4095
present 3999
Node 0, zone DMA32
spanned 1044480
present 439600
Node 0, zone Normal
spanned 7798784
present 7798784
Node 0, zone Movable
spanned 0
present 0
Node 0, zone Device
spanned 0
present 0For the most important zone regarding compaction, ZONE_NORMAL, it would be good enough.
We certainly don't care about detecting contigous for the DMA zone. For DMA32, I would suspect that it is not detected as contigous either way, because the holes are just way too large?
So we could maybe do (completely untested):
From 69093e5811b532812fde52b55a42dcb24d6e09dd Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Sat, 7 Feb 2026 11:45:21 +0100
Subject: [PATCH] tmp
--- include/linux/mmzone.h | 25 +++++++++++++++++++++++-- mm/internal.h | 8 +------- mm/memory_hotplug.c | 11 +---------- mm/mm_init.c | 25 ------------------------- 4 files changed, 25 insertions(+), 44 deletions(-)
include/linux/mmzone.h +23 -2
@@ -1051,8 +1051,6 @@ struct zone {bool compact_blockskip_flush;#endif -bool contiguous; -CACHELINE_PADDING (_pad3_);/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];
@@ -1124,6 +1122,29 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn)return zone->zone_start_pfn <= pfn && pfn <zone_end_pfn (zone); } +/** +* zone_is_contiguous - test whether a zone is contiguous +* @zone: the zone to test. +* +* In a contigous zone, it is valid to call pfn_to_page() on any pfn in the +* spanned zone without requiting pfn_valid() or pfn_to_online_page() checks. +* +* Returns: true if contiguous, otherwise false. +*/ +static inline bool zone_is_contiguous (const struct zone *zone) +{ +/* +* TODO: do we care about weird races? We could protect using a +* seqcount or sth. like that (zone_span_seqbegin etc). +* +* Concurrent hotplug is not an issue. But likely the caller must +* protect against concurrent hotunplug already? We should definitely +* read these values through READ_ONCE and update them through +* WRITE_ONCE(). +*/ +return zone->spanned_pages == zone->present_pages; +} +static inline bool zone_is_initialized (const struct zone *zone) {return zone->initialized;
mm/internal.h +1 -7
@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,static inline struct page *pageblock_pfn_to_page (unsigned long start_pfn,unsigned long end_pfn,struct zone *zone) { -if (zone->contiguous) +if (zone_is_contiguous (zone))return pfn_to_page (start_pfn);return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} -extern int __isolate_free_page (struct page *page,unsigned int order);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt);
mm/memory_hotplug.c +1 -10
@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone,/* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So -* we will not try to shrink the zones - which is okay as -* set_zone_contiguous() cannot deal with ZONE_DEVICE either way. +* we will not try to shrink the zones. */ if (zone_is_zone_device (zone))return ; -clear_zone_contiguous (zone); -shrink_zone_span (zone, start_pfn, start_pfn + nr_pages);update_pgdat_span (pgdat); - -set_zone_contiguous (zone); }/**
@@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id; -clear_zone_contiguous (zone); -if (zone_is_empty (zone))init_currently_empty_zone (zone, start_pfn, nr_pages);resize_zone_range (zone, start_pfn, nr_pages);
@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype, isolate_pageblock); - -set_zone_contiguous (zone); }struct auto_movable_stats {
mm/mm_init.c +0 -25
@@ -2263,28 +2263,6 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) -{ -unsigned long block_start_pfn = zone->zone_start_pfn; -unsigned long block_end_pfn; - - block_end_pfn =pageblock_end_pfn (block_start_pfn); -for (; block_start_pfn <zone_end_pfn (zone); - block_start_pfn = block_end_pfn, - block_end_pfn += pageblock_nr_pages) { - - block_end_pfn =min (block_end_pfn,zone_end_pfn (zone)); - -if (!__pageblock_pfn_to_page (block_start_pfn, - block_end_pfn, zone)) -return ; -cond_resched (); - } - -/* We confirm that there is no hole */ - zone->contiguous = true; -} -/* * Check if a PFN range intersects multiple zones on one or more * NUMA nodes. Specify the @nid argument if it is known that this
@@ -2347,9 +2325,6 @@ void __init page_alloc_init_late(void)for_each_node_state (nid,N_MEMORY )shuffle_free_memory (NODE_DATA (nid)); -for_each_populated_zone (zone) -set_zone_contiguous (zone); -/* Initialize page ext after all struct pages are initialized. */ if (deferred_struct_pages)page_ext_init ();
Signature · 34 lines
2.43.0
If we would want to cover the cases with "holes in zone, but there is a struct page and it's
assigned to the zone", all we would have to do is manually track them (during boot only,
cannot happen during memory hotplug) in zone->absent pages. That value would never change.
Then we would have instead:
static inline bool zone_is_contiguous(const struct zone *zone)
{
return zone->spanned_pages == zone->present_pages + zone->absent_pages;
}
I don't think we could just use "absent" as calculated in calculate_node_totalpages,
because I assume it could include "too many" things, not just these holes in pageblocks.
At least reading zone_absent_pages_in_node(), likely the value could return
* Pages that will not have a struct page in case of larger holes
* mirrored_kernelcore oddities
We'd need a reliably "absent pages that have a struct page that belongs to this zone".
Maybe Mike knows how to easily obtain that there to just set zone->absent_pages.
If we really need that optimization for these cases.
--
Cheers,
DavidMike Rapoport
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 71 lines · Tianyou Li
When invoke
move_pfn_range_to_zoneorremove_pfn_range_from_zone, it will update thezone->contiguousby checking the new zone's pfn range from the beginning to the end, regardless the previous state of the old zone. When the zone's pfn range is large, the cost of traversing the pfn range to update thezone->contiguouscould be significant.Add fast paths to quickly detect cases where zone is definitely not contiguous without scanning the new zone. The cases are: when the new range did not overlap with previous range, the contiguous should be false; if the new range adjacent with the previous range, just need to check the new range; if the new added pages could not fill the hole of previous zone, the contiguous should be false.
The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3].
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 33s | 6s | 81% | +----------------+------+---------------+--------------+----------------++----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 34s | 6s | 82% | +----------------+------+---------------+--------------+----------------+[1] Qemu commands to hotplug 256G/512G memory for a VM:
object_addmemory-backend-ram,id=hotmem0,size=256G/512G,share=ondevice_addvirtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory)[2] Hardware : Intel Icelake server
Guest Kernel : v6.18-rc2 Qemu : v9.0.0 Launch VM : qemu-system-x86_64 -accel kvm -cpu host \ -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \ -drive file=./seed.img,format=raw,if=virtio \ -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \ -m 2G,slots=10,maxmem=2052472M \ -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \ -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \ -nographic -machine q35 \ -nic user,hostfwd=tcp::3000-:22 Guest kernel auto-onlines newly added memory blocks: echo online > /sys/devices/system/memory/auto_online_blocks[3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal
/proc/meminfo' on Guest reflects that all hotplugged memory is recognized.Reported-by Nanhai Zou <[email protected]>Reported-by Chen Zhang <[email protected]>Tested-by Yuan Liu <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Yu C Chen <[email protected]>Reviewed-by Pan Deng <[email protected]>Reviewed-by Nanhai Zou <[email protected]>Reviewed-by Yuan Liu <[email protected]>Signed-off-by Tianyou Li <[email protected]>Quoted 1 line
---
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.
Quoted 6 lines
(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small holes within a pageblock?Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.
It seems that "zone->contigous" is really bad name for what this thing represents.
tl;dr I don't think zone_spanned_pages == zone_present_pages is related to zone->contigous at all :)
If you look at pageblock_pfn_to_page() and __pageblock_pfn_to_page(), the check for zone->contigous should guarantee that the entire pageblock has a valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.
For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map will be populated and will have struct pages.
When zone->contigous is false, the slow path in __pageblock_pfn_to_page() essentially checks if the first page in a pageblock is online and if first and last pages are in the zone being compacted.
AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, so zone->contigous won't change after the hotplug is complete.
We might set it to false in the beginning of the hotplug to avoid scanning offline pages, although I'm not sure if it's possible.
But in the end of hotplug we can simply restore the old value and move on.
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
Quoted 9 lines
I.e., on my notebook I have
$ cat /proc/zoneinfo | grep -E "Node|spanned|present" Node 0, zone DMA spanned 4095 present 3999 Node 0, zone DMA32 spanned 1044480 present 439600
I suspect this one is contigous ;-)
Quoted 20 lines
Node 0, zone Normal
spanned 7798784 present 7798784Node 0, zone Movable
spanned 0
present 0
Node 0, zone Device
spanned 0
present 0For the most important zone regarding compaction,
ZONE_NORMAL, it would be good enough.We certainly don't care about detecting contigous for the DMA zone. For DMA32, I would suspect that it is not detected as contigous either way, because the holes are just way too large?
Signature · 3 lines
Cheers, David
Signature · 2 lines
Sincerely yours, Mike.
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 95 lines · Mike Rapoport
Quoted 94 lines · David Hildenbrand (Arm)
Quoted 71 lines · Tianyou Li
When invoke
move_pfn_range_to_zoneorremove_pfn_range_from_zone, it will update thezone->contiguousby checking the new zone's pfn range from the beginning to the end, regardless the previous state of the old zone. When the zone's pfn range is large, the cost of traversing the pfn range to update thezone->contiguouscould be significant.Add fast paths to quickly detect cases where zone is definitely not contiguous without scanning the new zone. The cases are: when the new range did not overlap with previous range, the contiguous should be false; if the new range adjacent with the previous range, just need to check the new range; if the new added pages could not fill the hole of previous zone, the contiguous should be false.
The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3].
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 33s | 6s | 81% | +----------------+------+---------------+--------------+----------------++----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 34s | 6s | 82% | +----------------+------+---------------+--------------+----------------+[1] Qemu commands to hotplug 256G/512G memory for a VM:
object_addmemory-backend-ram,id=hotmem0,size=256G/512G,share=ondevice_addvirtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory)[2] Hardware : Intel Icelake server
Guest Kernel : v6.18-rc2 Qemu : v9.0.0 Launch VM : qemu-system-x86_64 -accel kvm -cpu host \ -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \ -drive file=./seed.img,format=raw,if=virtio \ -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \ -m 2G,slots=10,maxmem=2052472M \ -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \ -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \ -nographic -machine q35 \ -nic user,hostfwd=tcp::3000-:22 Guest kernel auto-onlines newly added memory blocks: echo online > /sys/devices/system/memory/auto_online_blocks[3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal
/proc/meminfo' on Guest reflects that all hotplugged memory is recognized.Reported-by Nanhai Zou <[email protected]>Reported-by Chen Zhang <[email protected]>Tested-by Yuan Liu <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Yu C Chen <[email protected]>Reviewed-by Pan Deng <[email protected]>Reviewed-by Nanhai Zou <[email protected]>Reviewed-by Yuan Liu <[email protected]>Signed-off-by Tianyou Li <[email protected]>Quoted 1 line
---
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small holes within a pageblock?
Quoted 8 lines
Quoted 2 lines
Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.It seems that "
zone->contigous" is really bad name for what this thing represents.tl;dr I don't think
zone_spanned_pages==zone_present_pagesis related tozone->contigousat all :)
My point in (a) was that with "zone_spanned_pages == zone_present_pages" there are no holes so -> contiguous.
(b), and what I said further below, is exactly about memory holes where we have a memmap, but it's not present memory.
Quoted 5 lines
If you look at
pageblock_pfn_to_page()and__pageblock_pfn_to_page(), the check forzone->contigousshould guarantee that the entire pageblock has a valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.
Right. But that must hold for each and ever pageblock in the spanned zone range for it to be contiguous.
zone->contigous tells you "pfn_to_page()" is valid on the complete zone range"
That's why set_zone_contiguous() probes __pageblock_pfn_to_page() on ech and ever pageblock.
Quoted 4 lines
For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map will be populated and will have struct pages.
There is this sub-section thing, and holes larger than a section might not have a memmap (unless reserved I guess).
Quoted 7 lines
When
zone->contigousis false, the slow path in__pageblock_pfn_to_page()essentially checks if the first page in a pageblock is online and if first and last pages are in the zone being compacted.AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, so
zone->contigouswon't change after the hotplug is complete.
I think you are missing a point: hotp(un)plug might create holes in the zone span. Then, pfn_to_page() is no longer valid to be called on arbitrary pageblocks within the zone.
Quoted 5 lines
We might set it to false in the beginning of the hotplug to avoid scanning offline pages, although I'm not sure if it's possible.
But in the end of hotplug we can simply restore the old value and move on.
No, you might create holes.
Quoted 4 lines
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
To recap:
My idea is that "zone_spanned_pages == zone_present_pages" tells you that the zone is contiguous because there are no holes.
To handle "non-memory with a struct page", you'd have to check
"zone_spanned_pages == zone_present_pages +
zone_non_present_memmap_pages"Or shorter
"zone_spanned_pages == zone_pages_with_memmap"
Then, pfn_to_page() is valid within the complete zone.
The question is how to best calculate the "zone_pages_with_memmap" during boot.
During hot(un)plug we only add/remove zone_present_pages. The zone_non_present_memmap_pages will not change due to hot(un)plug later.
Signature · 3 lines
Cheers, David
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 87 lines · David Hildenbrand (Arm)
Quoted 22 lines · Mike Rapoport
Quoted 21 lines · David Hildenbrand (Arm)
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small holes within a pageblock?Quoted 7 lines
Quoted 2 lines
Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.It seems that "
zone->contigous" is really bad name for what this thing represents.tl;dr I don't think
zone_spanned_pages==zone_present_pagesis related tozone->contigousat all :)My point in (a) was that with "
zone_spanned_pages==zone_present_pages" there are no holes so -> contiguous.(b), and what I said further below, is exactly about memory holes where we have a memmap, but it's not present memory.
Quoted 5 lines
If you look at
pageblock_pfn_to_page()and__pageblock_pfn_to_page(), the check forzone->contigousshould guarantee that the entire pageblock has a valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.Right. But that must hold for each and ever pageblock in the spanned zone range for it to be contiguous.
zone->contigoustells you "pfn_to_page()" is valid on the complete zone range"That's why
set_zone_contiguous()probes__pageblock_pfn_to_page()on ech and ever pageblock.Quoted 4 lines
For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map will be populated and will have struct pages.
There is this sub-section thing, and holes larger than a section might not have a memmap (unless reserved I guess).
Quoted 6 lines
When
zone->contigousis false, the slow path in__pageblock_pfn_to_page()essentially checks if the first page in a pageblock is online and if first and last pages are in the zone being compacted.
AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, sozone->contigouswon't change after the hotplug is complete.I think you are missing a point: hotp(un)plug might create holes in the zone span. Then,
pfn_to_page()is no longer valid to be called on arbitrary pageblocks within the zone.Quoted 5 lines
We might set it to false in the beginning of the hotplug to avoid scanning offline pages, although I'm not sure if it's possible.
But in the end of hotplug we can simply restore the old value and move on.
No, you might create holes.
Quoted 4 lines
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
To recap:
My idea is that "
zone_spanned_pages==zone_present_pages" tells you that the zone is contiguous because there are no holes.To handle "non-memory with a struct page", you'd have to check
"zone_spanned_pages ==
zone_present_pages+
zone_non_present_memmap_pages"Or shorter
Quoted 8 lines
"zone_spanned_pages ==
zone_pages_with_memmap"Then,
pfn_to_page()is valid within the complete zone.The question is how to best calculate the "
zone_pages_with_memmap" during boot.During hot(un)plug we only add/remove
zone_present_pages. Thezone_non_present_memmap_pageswill not change due to hot(un)plug later.
The following hack does the trick. But
(a) I wish we could get rid of the pageblock walking in calc_online_pages().
(b) "online_pages" has weird semantics due to the pageblock handling. "online_pageblock_pages"? not sure.
(c) Calculating "online_pages" when we know there is a hole does not make sense, as we could just keep it 0 if there are holes and simply set it to zone->online_pageblock_pages->zone->spanned_pages in case all are online.
From d4cb825e91a6363afc68fb994c5d9b29c38c5f42 Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Mon, 9 Feb 2026 13:40:24 +0100
Subject: [PATCH] tmp
--- include/linux/mmzone.h | 25 +++++++++++++++++++++++-- mm/internal.h | 8 +------- mm/memory_hotplug.c | 20 ++++++-------------- mm/mm_init.c | 12 ++++++------ 4 files changed, 36 insertions(+), 29 deletions(-)
include/linux/mmzone.h +20 -2
@@ -943,6 +943,11 @@ struct zone {* cma pages is present pages that are assigned for CMA use * (MIGRATE_CMA). * +* online_pages is pages within the zone that have an online memmap. +* online_pages include present pages and memory holes that have a +* memmap. When spanned_pages == online_pages, pfn_to_page() can be +* performed without further checks on any pfn within the zone span. +* * So present_pages may be used by memory hotplug or memory power * management logic to figure out unmanaged pages by checking * (present_pages - managed_pages). And managed_pages should be used
@@ -967,6 +972,7 @@ struct zone {atomic_long_t managed_pages;unsigned long spanned_pages;unsigned long present_pages; +unsigned long online_pages;#if defined(CONFIG_MEMORY_HOTPLUG )unsigned long present_early_pages;#endif
@@ -1051,8 +1057,6 @@ struct zone {bool compact_blockskip_flush;#endif -bool contiguous; -CACHELINE_PADDING (_pad3_);/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];
@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn)return zone->zone_start_pfn <= pfn && pfn <zone_end_pfn (zone); } +/** +* zone_is_contiguous - test whether a zone is contiguous +* @zone: the zone to test. +* +* In a contiguous zone, it is valid to call pfn_to_page() on any pfn in the +* spanned zone without requiting pfn_valid() or pfn_to_online_page() checks. +* +* Returns: true if contiguous, otherwise false. +*/ +static inline bool zone_is_contiguous (const struct zone *zone) +{ +return READ_ONCE (zone->spanned_pages) ==READ_ONCE (zone->online_pages); +} +static inline bool zone_is_initialized (const struct zone *zone) {return zone->initialized;
mm/internal.h +1 -7
@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,static inline struct page *pageblock_pfn_to_page (unsigned long start_pfn,unsigned long end_pfn,struct zone *zone) { -if (zone->contiguous) +if (zone_is_contiguous (zone))return pfn_to_page (start_pfn);return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} -extern int __isolate_free_page (struct page *page,unsigned int order);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt);
mm/memory_hotplug.c +6 -14
@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_smallest_section_pfn (nid, zone, end_pfn,zone_end_pfn (zone));if (pfn) { - zone->spanned_pages =zone_end_pfn (zone) - pfn; +WRITE_ONCE (zone->spanned_pages,zone_end_pfn (zone) - pfn); zone->zone_start_pfn = pfn; }else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } }else if (zone_end_pfn (zone) == end_pfn) {/*
@@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_biggest_section_pfn (nid, zone, zone->zone_start_pfn, start_pfn);if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn +1 ; +WRITE_ONCE (zone->spanned_pages, pfn - zone->zone_start_pfn +1 );else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } } }
@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone,/* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So -* we will not try to shrink the zones - which is okay as -* set_zone_contiguous() cannot deal with ZONE_DEVICE either way. +* we will not try to shrink the zones. */ if (zone_is_zone_device (zone))return ; -clear_zone_contiguous (zone); -shrink_zone_span (zone, start_pfn, start_pfn + nr_pages);update_pgdat_span (pgdat); - -set_zone_contiguous (zone); }/**
@@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id; -clear_zone_contiguous (zone); -if (zone_is_empty (zone))init_currently_empty_zone (zone, start_pfn, nr_pages);resize_zone_range (zone, start_pfn, nr_pages);
@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype, isolate_pageblock); - -set_zone_contiguous (zone); }struct auto_movable_stats {
@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group,if (early_section (__pfn_to_section (page_to_pfn (page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; +WRITE_ONCE (zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages;if (group && movable)
mm/mm_init.c +6 -6
@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) {unsigned long block_start_pfn = zone->zone_start_pfn; +unsigned long online_pages =0 ;unsigned long block_end_pfn; block_end_pfn =pageblock_end_pfn (block_start_pfn);
@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone)if (!__pageblock_pfn_to_page (block_start_pfn, block_end_pfn, zone)) -return ; +continue ;cond_resched (); + online_pages += block_end_pfn - block_start_pfn; } - -/* We confirm that there is no hole */ - zone->contiguous = true; + zone->online_pages = online_pages; }/*
@@ -2348,7 +2348,7 @@ void __init page_alloc_init_late(void)shuffle_free_memory (NODE_DATA (nid));for_each_populated_zone (zone) -set_zone_contiguous (zone); +calc_online_pages (zone);/* Initialize page ext after all struct pages are initialized. */ if (deferred_struct_pages)
Signature · 7 lines
2.43.0 -- Cheers, David
Mike Rapoport
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 143 lines · David Hildenbrand (Arm)
Quoted 105 lines · David Hildenbrand (Arm)
Quoted 23 lines · Mike Rapoport
Quoted 22 lines · David Hildenbrand (Arm)
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small
holes within a pageblock?Quoted 8 lines
Quoted 2 lines
Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.It seems that "
zone->contigous" is really bad name for what this thing represents.tl;dr I don't think
zone_spanned_pages==zone_present_pagesis related tozone->contigousat all :)My point in (a) was that with "
zone_spanned_pages==zone_present_pages" there are no holes so -> contiguous.(b), and what I said further below, is exactly about memory holes where we have a memmap, but it's not present memory.
Quoted 6 lines
If you look at
pageblock_pfn_to_page()and__pageblock_pfn_to_page(), the check forzone->contigousshould guarantee that the entire pageblock has a
valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.Right. But that must hold for each and ever pageblock in the spanned zone range for it to be contiguous.
zone->contigoustells you "pfn_to_page()" is valid on the complete zone range"That's why
set_zone_contiguous()probes__pageblock_pfn_to_page()on ech and ever pageblock.Quoted 5 lines
For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map
will be populated and will have struct pages.There is this sub-section thing, and holes larger than a section might not have a memmap (unless reserved I guess).
Quoted 7 lines
When
zone->contigousis false, the slow path in__pageblock_pfn_to_page()essentially checks if the first page in a pageblock is online and if first
and last pages are in the zone being compacted.
AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, sozone->contigouswon't change after the hotplug is complete.I think you are missing a point: hotp(un)plug might create holes in the zone span. Then,
pfn_to_page()is no longer valid to be called on arbitrary pageblocks within the zone.Quoted 7 lines
We might set it to false in the beginning of the hotplug to avoid scanning
offline pages, although I'm not sure if it's possible.But in the end of hotplug we can simply restore the old value and move on.
No, you might create holes.
Quoted 4 lines
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
To recap:
My idea is that "
zone_spanned_pages==zone_present_pages" tells you that the zone is contiguous because there are no holes.To handle "non-memory with a struct page", you'd have to check
"zone_spanned_pages ==
zone_present_pages+
zone_non_present_memmap_pages"Or shorter
"zone_spanned_pages ==
zone_pages_with_memmap"Quoted 8 lines
Then,
pfn_to_page()is valid within the complete zone.The question is how to best calculate the "
zone_pages_with_memmap" during boot.During hot(un)plug we only add/remove
zone_present_pages. Thezone_non_present_memmap_pageswill not change due to hot(un)plug later.The following hack does the trick. But
(a) I wish we could get rid of the pageblock walking in
calc_online_pages().
(b) "online_pages" has weird semantics due to the pageblock handling. "online_pageblock_pages"? not sure.
(c) Calculating "online_pages" when we know there is a hole does not make sense, as we could just keep it 0 if there are holes and simply set it tozone->online_pageblock_pages->zone->spanned_pagesin case all are online.From
d4cb825e91a6363afc68fb994c5d9b29c38c5f42Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Mon, 9 Feb 2026 13:40:24 +0100
Subject: [PATCH] tmpSigned-off-by David Hildenbrand (Arm) <[email protected]>--- include/linux/mmzone.h | 25 +++++++++++++++++++++++-- mm/internal.h | 8 +------- mm/memory_hotplug.c | 20 ++++++-------------- mm/mm_init.c | 12 ++++++------ 4 files changed, 36 insertions(+), 29 deletions(-)include/linux/mmzone.h +0
-0
include/linux/mmzone.h struct zone { · line 949
Quoted 7 lines
include/linux/mmzone.h +4
-0@@ -943,6 +943,11 @@ struct zone {* cma pages is present pages that are assigned for CMA use * (MIGRATE_CMA). * +* online_pages is pages within the zone that have an online memmap. +* online_pages include present pages and memory holes that have a +* memmap. When spanned_pages == online_pages, pfn_to_page() can be +* performed without further checks on any pfn within the zone span.
Maybe pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.
Quoted 150 lines
include/linux/mmzone.h +16
-2+* * So present_pages may be used by memory hotplug or memory power * management logic to figure out unmanaged pages by checking * (present_pages - managed_pages). And managed_pages should be used @@ -967,6 +972,7 @@ struct zone {atomic_long_t managed_pages;unsigned long spanned_pages;unsigned long present_pages; +unsigned long online_pages;#if defined(CONFIG_MEMORY_HOTPLUG )unsigned long present_early_pages;#endif @@ -1051,8 +1057,6 @@ struct zone {bool compact_blockskip_flush;#endif -bool contiguous; -CACHELINE_PADDING (_pad3_);/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn)return zone->zone_start_pfn <= pfn && pfn <zone_end_pfn (zone); } +/** +* zone_is_contiguous - test whether a zone is contiguous +* @zone: the zone to test. +* +* In a contiguous zone, it is valid to call pfn_to_page() on any pfn in the +* spanned zone without requiting pfn_valid() or pfn_to_online_page() checks. +* +* Returns: true if contiguous, otherwise false. +*/ +static inline bool zone_is_contiguous (const struct zone *zone) +{ +return READ_ONCE (zone->spanned_pages) ==READ_ONCE (zone->online_pages); +} +static inline bool zone_is_initialized (const struct zone *zone) {return zone->initialized;mm/internal.h +1
-7@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,static inline struct page *pageblock_pfn_to_page (unsigned long start_pfn,unsigned long end_pfn,struct zone *zone) { -if (zone->contiguous) +if (zone_is_contiguous (zone))return pfn_to_page (start_pfn);return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} -extern int __isolate_free_page (struct page *page,unsigned int order);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt);mm/memory_hotplug.c +6
-14@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_smallest_section_pfn (nid, zone, end_pfn,zone_end_pfn (zone));if (pfn) { - zone->spanned_pages =zone_end_pfn (zone) - pfn; +WRITE_ONCE (zone->spanned_pages,zone_end_pfn (zone) - pfn); zone->zone_start_pfn = pfn; }else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } }else if (zone_end_pfn (zone) == end_pfn) {/* @@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_biggest_section_pfn (nid, zone, zone->zone_start_pfn, start_pfn);if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn +1 ; +WRITE_ONCE (zone->spanned_pages, pfn - zone->zone_start_pfn +1 );else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } } }@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone,/* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So -* we will not try to shrink the zones - which is okay as -* set_zone_contiguous() cannot deal with ZONE_DEVICE either way. +* we will not try to shrink the zones. */ if (zone_is_zone_device (zone))return ; -clear_zone_contiguous (zone); -shrink_zone_span (zone, start_pfn, start_pfn + nr_pages);update_pgdat_span (pgdat); - -set_zone_contiguous (zone); }/** @@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id; -clear_zone_contiguous (zone); -if (zone_is_empty (zone))init_currently_empty_zone (zone, start_pfn, nr_pages);resize_zone_range (zone, start_pfn, nr_pages);@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype, isolate_pageblock); - -set_zone_contiguous (zone); }struct auto_movable_stats {@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group,if (early_section (__pfn_to_section (page_to_pfn (page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; +WRITE_ONCE (zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages;if (group && movable)mm/mm_init.c +2
-1@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) {unsigned long block_start_pfn = zone->zone_start_pfn; +unsigned long online_pages =0 ;unsigned long block_end_pfn; block_end_pfn =pageblock_end_pfn (block_start_pfn);
mm/mm_init.c void set_zone_contiguous(struct zone *zone) · line 2282
Quoted 6 lines
mm/mm_init.c +2
-1@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone)if (!__pageblock_pfn_to_page (block_start_pfn, block_end_pfn, zone)) -return ; +continue ;cond_resched (); + online_pages += block_end_pfn - block_start_pfn;
I think we can completely get rid of this with something like this untested patch to calculate zone->online_pages for coldplug:
mm/mm_init.c +7 -4
@@ -845,9 +845,9 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn)* zone/node above the hole except for the trailing pages in the last * section that will be appended to the zone/node below. */ -static void __init init_unavailable_range (unsigned long spfn, -unsigned long epfn, -int zone,int node) +static u64 __init init_unavailable_range (unsigned long spfn, +unsigned long epfn, +int zone,int node) {unsigned long pfn;u64 pgcnt =0 ;
@@ -861,6 +861,8 @@ static void __init init_unavailable_range(unsigned long spfn,if (pgcnt)pr_info ("On node %d, zone %s: %lld pages in unavailable ranges\n" , node, zone_names[zone], pgcnt); + +return pgcnt; }/*
@@ -959,9 +961,10 @@ static void __init memmap_init_zone_range(struct zone *zone,memmap_init_range (end_pfn - start_pfn, nid, zone_id, start_pfn, zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE , false); + zone->online_pages += (end_pfn - start_pfn);if (*hole_pfn < start_pfn) -init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); + zone->online_pages +=init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); *hole_pfn = end_pfn; }
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 327 lines · Mike Rapoport
Quoted 129 lines · David Hildenbrand (Arm)
Quoted 92 lines · David Hildenbrand (Arm)
Quoted 22 lines · Mike Rapoport
Quoted 21 lines · David Hildenbrand (Arm)
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small
holes within a pageblock?Quoted 8 lines
Quoted 2 lines
Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.It seems that "
zone->contigous" is really bad name for what this thing represents.tl;dr I don't think
zone_spanned_pages==zone_present_pagesis related tozone->contigousat all :)My point in (a) was that with "
zone_spanned_pages==zone_present_pages" there are no holes so -> contiguous.(b), and what I said further below, is exactly about memory holes where we have a memmap, but it's not present memory.
Quoted 5 lines
If you look at
pageblock_pfn_to_page()and__pageblock_pfn_to_page(), the check forzone->contigousshould guarantee that the entire pageblock has a
valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.Right. But that must hold for each and ever pageblock in the spanned zone range for it to be contiguous.
zone->contigoustells you "pfn_to_page()" is valid on the complete zone range"That's why
set_zone_contiguous()probes__pageblock_pfn_to_page()on ech and ever pageblock.Quoted 4 lines
For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map
will be populated and will have struct pages.There is this sub-section thing, and holes larger than a section might not have a memmap (unless reserved I guess).
Quoted 6 lines
When
zone->contigousis false, the slow path in__pageblock_pfn_to_page()essentially checks if the first page in a pageblock is online and if first
and last pages are in the zone being compacted.
AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, sozone->contigouswon't change after the hotplug is complete.I think you are missing a point: hotp(un)plug might create holes in the zone span. Then,
pfn_to_page()is no longer valid to be called on arbitrary pageblocks within the zone.Quoted 6 lines
We might set it to false in the beginning of the hotplug to avoid scanning
offline pages, although I'm not sure if it's possible.But in the end of hotplug we can simply restore the old value and move on.
No, you might create holes.
Quoted 3 lines
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
To recap:
My idea is that "
zone_spanned_pages==zone_present_pages" tells you that the zone is contiguous because there are no holes.To handle "non-memory with a struct page", you'd have to check
"zone_spanned_pages ==
zone_present_pages+
zone_non_present_memmap_pages"Or shorter
"zone_spanned_pages ==
zone_pages_with_memmap"Quoted 8 lines
Then,
pfn_to_page()is valid within the complete zone.The question is how to best calculate the "
zone_pages_with_memmap" during boot.During hot(un)plug we only add/remove
zone_present_pages. Thezone_non_present_memmap_pageswill not change due to hot(un)plug later.The following hack does the trick. But
(a) I wish we could get rid of the pageblock walking in
calc_online_pages().
(b) "online_pages" has weird semantics due to the pageblock handling. "online_pageblock_pages"? not sure.
(c) Calculating "online_pages" when we know there is a hole does not make sense, as we could just keep it 0 if there are holes and simply set it tozone->online_pageblock_pages->zone->spanned_pagesin case all are online.From
d4cb825e91a6363afc68fb994c5d9b29c38c5f42Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Mon, 9 Feb 2026 13:40:24 +0100
Subject: [PATCH] tmpSigned-off-by David Hildenbrand (Arm) <[email protected]>--- include/linux/mmzone.h | 25 +++++++++++++++++++++++-- mm/internal.h | 8 +------- mm/memory_hotplug.c | 20 ++++++-------------- mm/mm_init.c | 12 ++++++------ 4 files changed, 36 insertions(+), 29 deletions(-)include/linux/mmzone.h +0
-0include/linux/mmzone.h struct zone { · line 949
Quoted 7 lines
include/linux/mmzone.h +4
-0@@ -943,6 +943,11 @@ struct zone {* cma pages is present pages that are assigned for CMA use * (MIGRATE_CMA). * +* online_pages is pages within the zone that have an online memmap. +* online_pages include present pages and memory holes that have a +* memmap. When spanned_pages == online_pages, pfn_to_page() can be +* performed without further checks on any pfn within the zone span. Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.Quoted 150 lines
include/linux/mmzone.h +16
-2+* * So present_pages may be used by memory hotplug or memory power * management logic to figure out unmanaged pages by checking * (present_pages - managed_pages). And managed_pages should be used @@ -967,6 +972,7 @@ struct zone {atomic_long_t managed_pages;unsigned long spanned_pages;unsigned long present_pages; +unsigned long online_pages;#if defined(CONFIG_MEMORY_HOTPLUG )unsigned long present_early_pages;#endif @@ -1051,8 +1057,6 @@ struct zone {bool compact_blockskip_flush;#endif -bool contiguous; -CACHELINE_PADDING (_pad3_);/* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS ];@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn)return zone->zone_start_pfn <= pfn && pfn <zone_end_pfn (zone); } +/** +* zone_is_contiguous - test whether a zone is contiguous +* @zone: the zone to test. +* +* In a contiguous zone, it is valid to call pfn_to_page() on any pfn in the +* spanned zone without requiting pfn_valid() or pfn_to_online_page() checks. +* +* Returns: true if contiguous, otherwise false. +*/ +static inline bool zone_is_contiguous (const struct zone *zone) +{ +return READ_ONCE (zone->spanned_pages) ==READ_ONCE (zone->online_pages); +} +static inline bool zone_is_initialized (const struct zone *zone) {return zone->initialized;mm/internal.h +1
-7@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,static inline struct page *pageblock_pfn_to_page (unsigned long start_pfn,unsigned long end_pfn,struct zone *zone) { -if (zone->contiguous) +if (zone_is_contiguous (zone))return pfn_to_page (start_pfn);return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} -extern int __isolate_free_page (struct page *page,unsigned int order);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt);mm/memory_hotplug.c +6
-14@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_smallest_section_pfn (nid, zone, end_pfn,zone_end_pfn (zone));if (pfn) { - zone->spanned_pages =zone_end_pfn (zone) - pfn; +WRITE_ONCE (zone->spanned_pages,zone_end_pfn (zone) - pfn); zone->zone_start_pfn = pfn; }else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } }else if (zone_end_pfn (zone) == end_pfn) {/* @@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_biggest_section_pfn (nid, zone, zone->zone_start_pfn, start_pfn);if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn +1 ; +WRITE_ONCE (zone->spanned_pages, pfn - zone->zone_start_pfn +1 );else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } } }@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone,/* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So -* we will not try to shrink the zones - which is okay as -* set_zone_contiguous() cannot deal with ZONE_DEVICE either way. +* we will not try to shrink the zones. */ if (zone_is_zone_device (zone))return ; -clear_zone_contiguous (zone); -shrink_zone_span (zone, start_pfn, start_pfn + nr_pages);update_pgdat_span (pgdat); - -set_zone_contiguous (zone); }/** @@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id; -clear_zone_contiguous (zone); -if (zone_is_empty (zone))init_currently_empty_zone (zone, start_pfn, nr_pages);resize_zone_range (zone, start_pfn, nr_pages);@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype, isolate_pageblock); - -set_zone_contiguous (zone); }struct auto_movable_stats {@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group,if (early_section (__pfn_to_section (page_to_pfn (page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; +WRITE_ONCE (zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages;if (group && movable)mm/mm_init.c +2
-1@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) {unsigned long block_start_pfn = zone->zone_start_pfn; +unsigned long online_pages =0 ;unsigned long block_end_pfn; block_end_pfn =pageblock_end_pfn (block_start_pfn);mm/mm_init.c void set_zone_contiguous(struct zone *zone) · line 2282
Quoted 6 lines
mm/mm_init.c +2
-1@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone)if (!__pageblock_pfn_to_page (block_start_pfn, block_end_pfn, zone)) -return ; +continue ;cond_resched (); + online_pages += block_end_pfn - block_start_pfn;I think we can completely get rid of this with something like this untested patch to calculate
zone->online_pagesfor coldplug:mm/mm_init.c +5
-3@@ -845,9 +845,9 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn)* zone/node above the hole except for the trailing pages in the last * section that will be appended to the zone/node below. */ -static void __init init_unavailable_range (unsigned long spfn, -unsigned long epfn, -int zone,int node) +static u64 __init init_unavailable_range (unsigned long spfn, +unsigned long epfn, +int zone,int node) {unsigned long pfn;u64 pgcnt =0 ;@@ -861,6 +861,8 @@ static void __init init_unavailable_range(unsigned long spfn,if (pgcnt)pr_info ("On node %d, zone %s: %lld pages in unavailable ranges\n" , node, zone_names[zone], pgcnt); + +return pgcnt; }/* @@ -959,9 +961,10 @@ static void __init memmap_init_zone_range(struct zone *zone, memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn,
mm/mm_init.c static void __init memmap_init_zone_range(struct zone *zone, · line 970
Quoted 10 lines
mm/mm_init.c +2
-1zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE , false); + zone->online_pages += (end_pfn - start_pfn);if (*hole_pfn < start_pfn) -init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); + zone->online_pages +=init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); *hole_pfn = end_pfn; }
Sorry for late response, I am trying to catch up with the discussion:) Per my understanding, zone->contiguous has 2 semantics in combination, one is the pages are full filled in the zone span, the other is those pages could be access as it has been onlined. The check zone_spanned_pages == zone_online_pages guaranteed the both. Either resize_zone_span() or shrink_zone_span() will change the zone_spanned_pages so they need to use WRITE_ONCE to guarantee the ordering; so does to the adjust_present_page_count() where zone_online_pages get updated.
Regards,
Tianyou
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
In reply to Mike RapoportQuoted 8 lines
Review
Quoted 5 lines
*
(quoted diff) +4
-0+ * online_pages is pages within the zone that have an online memmap. + * online_pages include present pages and memory holes that have a + * memmap. When spanned_pages == online_pages, pfn_to_page() can be + * performed without further checks on any pfn within the zone span.Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.
offline pages also have a memmap, but that should not be touched as it might contain garbage. So it's a bit more tricky :)
Quoted 190 lines
Quoted 151 lines
(quoted diff) +16
-2+ * * So present_pages may be used by memory hotplug or memory power * management logic to figure out unmanaged pages by checking * (present_pages - managed_pages). And managed_pages should be used@@ -967,6 +972,7 @@ struct zone { atomic_long_t managed_pages; unsigned long spanned_pages; unsigned long present_pages; + unsigned long online_pages; #if defined(CONFIG_MEMORY_HOTPLUG) unsigned long present_early_pages; #endif@@ -1051,8 +1057,6 @@ struct zone { bool compact_blockskip_flush; #endif - bool contiguous; - CACHELINE_PADDING(_pad3_); /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn) return zone->zone_start_pfn <= pfn && pfn < zone_end_pfn(zone); } +/** + * zone_is_contiguous - test whether a zone is contiguous + * @zone: the zone to test. + * + * In a contiguous zone, it is valid to call pfn_to_page() on any pfn in the + * spanned zone without requiting pfn_valid() or pfn_to_online_page() checks. + * + * Returns: true if contiguous, otherwise false. + */ +static inline bool zone_is_contiguous(const struct zone *zone) +{ + return READ_ONCE(zone->spanned_pages) == READ_ONCE(zone->online_pages); +} + static inline bool zone_is_initialized(const struct zone *zone) { return zone->initialized;mm/internal.h +1
-7@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,static inline struct page *pageblock_pfn_to_page (unsigned long start_pfn,unsigned long end_pfn,struct zone *zone) { -if (zone->contiguous) +if (zone_is_contiguous (zone))return pfn_to_page (start_pfn);return __pageblock_pfn_to_page (start_pfn, end_pfn, zone); } -void set_zone_contiguous (struct zone *zone);bool pfn_range_intersects_zones (int nid,unsigned long start_pfn,unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} -extern int __isolate_free_page (struct page *page,unsigned int order);extern void __putback_isolated_page (struct page *page,unsigned int order,int mt);mm/memory_hotplug.c +6
-14@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_smallest_section_pfn (nid, zone, end_pfn,zone_end_pfn (zone));if (pfn) { - zone->spanned_pages =zone_end_pfn (zone) - pfn; +WRITE_ONCE (zone->spanned_pages,zone_end_pfn (zone) - pfn); zone->zone_start_pfn = pfn; }else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } }else if (zone_end_pfn (zone) == end_pfn) {/* @@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn =find_biggest_section_pfn (nid, zone, zone->zone_start_pfn, start_pfn);if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn +1 ; +WRITE_ONCE (zone->spanned_pages, pfn - zone->zone_start_pfn +1 );else { zone->zone_start_pfn =0 ; - zone->spanned_pages =0 ; +WRITE_ONCE (zone->spanned_pages,0 ); } } }@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone,/* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So -* we will not try to shrink the zones - which is okay as -* set_zone_contiguous() cannot deal with ZONE_DEVICE either way. +* we will not try to shrink the zones. */ if (zone_is_zone_device (zone))return ; -clear_zone_contiguous (zone); -shrink_zone_span (zone, start_pfn, start_pfn + nr_pages);update_pgdat_span (pgdat); - -set_zone_contiguous (zone); }/** @@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id; -clear_zone_contiguous (zone); -if (zone_is_empty (zone))init_currently_empty_zone (zone, start_pfn, nr_pages);resize_zone_range (zone, start_pfn, nr_pages);@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype, isolate_pageblock); - -set_zone_contiguous (zone); }struct auto_movable_stats {@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group,if (early_section (__pfn_to_section (page_to_pfn (page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; +WRITE_ONCE (zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages;if (group && movable)mm/mm_init.c +2
-1@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) }#endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) {unsigned long block_start_pfn = zone->zone_start_pfn; +unsigned long online_pages =0 ;unsigned long block_end_pfn; block_end_pfn =pageblock_end_pfn (block_start_pfn);mm/mm_init.c void set_zone_contiguous(struct zone *zone) · line 2282
Quoted 6 lines
mm/mm_init.c +2
-1@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone)if (!__pageblock_pfn_to_page (block_start_pfn, block_end_pfn, zone)) -return ; +continue ;cond_resched (); + online_pages += block_end_pfn - block_start_pfn;I think we can completely get rid of this with something like this untested patch to calculate
zone->online_pagesfor coldplug:mm/mm_init.c +5
-3@@ -845,9 +845,9 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn)* zone/node above the hole except for the trailing pages in the last * section that will be appended to the zone/node below. */ -static void __init init_unavailable_range (unsigned long spfn, -unsigned long epfn, -int zone,int node) +static u64 __init init_unavailable_range (unsigned long spfn, +unsigned long epfn, +int zone,int node) {unsigned long pfn;u64 pgcnt =0 ;@@ -861,6 +861,8 @@ static void __init init_unavailable_range(unsigned long spfn,if (pgcnt)pr_info ("On node %d, zone %s: %lld pages in unavailable ranges\n" , node, zone_names[zone], pgcnt); + +return pgcnt; }/* @@ -959,9 +961,10 @@ static void __init memmap_init_zone_range(struct zone *zone, memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn,
mm/mm_init.c static void __init memmap_init_zone_range(struct zone *zone, · line 970
Quoted 10 lines
mm/mm_init.c +2
-1zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE , false); + zone->online_pages += (end_pfn - start_pfn);if (*hole_pfn < start_pfn) -init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); + zone->online_pages +=init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid); *hole_pfn = end_pfn; }
Looking at set_zone_contiguous(), __pageblock_pfn_to_page() takes care of a weird case where the end of a zone falls into the middle of a pageblock.
I am not even sure if that is possible, but we could handle that easily in pageblock_pfn_to_page() by checking the requested range against the zone spanned range.
Then the semantics "zone->online_pages" would be less weird and more closely resemble "pages with online memmap".
init_unavailable_range() might indeed do the trick!
@Tianyou, can you explore that direction? I know, your PTO is coming up.
Signature · 3 lines
Cheers, David
Mike Rapoport
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 8 lines
Review
Quoted 5 lines
*
(quoted diff) +4
-0+ * online_pages is pages within the zone that have an online memmap. + * online_pages include present pages and memory holes that have a + * memmap. When spanned_pages == online_pages, pfn_to_page() can be + * performed without further checks on any pfn within the zone span.Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.offline pages also have a memmap, but that should not be touched as it might contain garbage. So it's a bit more tricky :)
Naming is hard :)
But I still think mentioning memmap there is useful :)
Quoted 4 lines
Looking at
set_zone_contiguous(),__pageblock_pfn_to_page()takes care of a weird case where the end of a zone falls into the middle of a pageblock.I am not even sure if that is possible
It's possible if a pageblock crosses node boundary. We also might add VM_BUG_ON(pageblock_crosses_nodes(), "FIX YOUR FIRMWARE!") there ;-)
Quoted 2 lines
but we could handle that easily in
pageblock_pfn_to_page()by checking the requested range against the zone spanned range.
Agree.
Quoted 11 lines
Then the semantics "
zone->online_pages" would be less weird and more closely resemble "pages with online memmap".
init_unavailable_range()might indeed do the trick!@Tianyou, can you explore that direction? I know, your PTO is coming up.
Signature · 3 lines
Cheers, David
Signature · 2 lines
Sincerely yours, Mike.
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 10 lines · Mike Rapoport
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines
Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.offline pages also have a memmap, but that should not be touched as it might contain garbage. So it's a bit more tricky :)
Naming is hard :)
But I still think mentioning memmap there is useful :)
pages_with_online_memmap
Is a bit mouthful, but that shouldn't really be a problem here.
Signature · 3 lines
Cheers, David
Re: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
In reply to David Hildenbrand (Arm)Quoted 231 lines · David Hildenbrand (Arm)
Quoted 11 lines
Quoted 8 lines
*
+ *online_pagesis pages within the zone that have an online memmap.
+ *online_pagesinclude present pages and memory holes that have a + * memmap. Whenspanned_pages==online_pages,pfn_to_page()can be
+ * performed without further checks on any pfn within the zone span.Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.offline pages also have a memmap, but that should not be touched as it might contain garbage. So it's a bit more tricky :)
Quoted 204 lines
Quoted 164 lines
+ *
* Sopresent_pagesmay be used by memory hotplug or memory power
* management logic to figure out unmanaged pages by checking
* (present_pages-managed_pages). Andmanaged_pagesshould be used(quoted diff) +6
-2@@ -967,6 +972,7 @@ struct zone { atomic_long_t managed_pages; unsigned long spanned_pages; unsigned long present_pages; + unsigned long online_pages; #if defined(CONFIG_MEMORY_HOTPLUG) unsigned long present_early_pages; #endif@@ -1051,8 +1057,6 @@ struct zone { bool compact_blockskip_flush; #endif - bool contiguous; - CACHELINE_PADDING(_pad3_); /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn) return zone->zone_start_pfn <= pfn && pfn < zone_end_pfn(zone); } +/** + * zone_is_contiguous - test whether a zone is contiguous + * @zone: the zone to test. + * + * In a contiguous zone, it is valid to call pfn_to_page() on anypfn in the
+ * spanned zone without requitingpfn_valid()orpfn_to_online_page()checks.
+ *
+ * Returns: true if contiguous, otherwise false.
+ */(quoted diff) +3
-0+static inline bool zone_is_contiguous(const struct zone *zone) +{ + return READ_ONCE(zone->spanned_pages) ==
READ_ONCE(zone->online_pages);(quoted diff) +2
-0+} +static inline bool
zone_is_initialized(const struct zone *zone){
returnzone->initialized;mm/internal.h +1
-7@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn, static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn, unsigned long end_pfn, struct zone *zone) { - if (zone->contiguous) + if (zone_is_contiguous(zone)) return pfn_to_page(start_pfn); return __pageblock_pfn_to_page(start_pfn, end_pfn, zone); } -void set_zone_contiguous(struct zone *zone); bool pfn_range_intersects_zones(int nid, unsigned long start_pfn, unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} - extern int __isolate_free_page(struct page *page, unsigned int order); extern void __putback_isolated_page(struct page *page, unsigned int order, int mt);mm/memory_hotplug.c +2
-2@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn = find_smallest_section_pfn(nid, zone, end_pfn, zone_end_pfn(zone)); if (pfn) { - zone->spanned_pages = zone_end_pfn(zone) - pfn; + WRITE_ONCE(zone->spanned_pages, zone_end_pfn(zone) - pfn); zone->zone_start_pfn = pfn; } else { zone->zone_start_pfn = 0; - zone->spanned_pages = 0; + WRITE_ONCE(zone->spanned_pages, 0); } } else if (zone_end_pfn(zone) == end_pfn) {/*
mm/memory_hotplug.c +2
-2@@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn = find_biggest_section_pfn(nid, zone, zone->zone_start_pfn, start_pfn); if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn + 1; + WRITE_ONCE(zone->spanned_pages, pfn - zone->zone_start_pfn +1 ); else { zone->zone_start_pfn = 0; - zone->spanned_pages = 0; + WRITE_ONCE(zone->spanned_pages, 0); }}
}mm/memory_hotplug.c +2
-10@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone, /* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So - * we will not try to shrink the zones - which is okay as - * set_zone_contiguous() cannot deal with ZONE_DEVICE either way. + * we will not try to shrink the zones. */ if (zone_is_zone_device(zone)) return; - clear_zone_contiguous(zone); - shrink_zone_span(zone, start_pfn, start_pfn + nr_pages); update_pgdat_span(pgdat); - - set_zone_contiguous(zone); } /**@@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn, struct pglist_data *pgdat = zone->zone_pgdat; int nid = pgdat->node_id; - clear_zone_contiguous(zone); - if (zone_is_empty(zone)) init_currently_empty_zone(zone, start_pfn, nr_pages); resize_zone_range(zone, start_pfn, nr_pages);@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone,unsigned long start_pfn, memmap_init_range(nr_pages, nid, zone_idx(zone), start_pfn, 0, MEMINIT_HOTPLUG, altmap, migratetype, isolate_pageblock); - - set_zone_contiguous(zone); } struct auto_movable_stats {@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group, if (early_section(__pfn_to_section(page_to_pfn(page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; + WRITE_ONCE(zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages; if (group && movable)mm/mm_init.c +2
-1@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) } #endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) { unsigned long block_start_pfn = zone->zone_start_pfn; + unsigned long online_pages = 0; unsigned long block_end_pfn; block_end_pfn = pageblock_end_pfn(block_start_pfn);mm/mm_init.c void set_zone_contiguous(struct zone *zone) · line 2282
Quoted 6 lines
mm/mm_init.c +2
-1@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone) if (!__pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone)) - return; + continue; cond_resched(); + online_pages += block_end_pfn - block_start_pfn;I think we can completely get rid of this with something like this untested patch to calculate
zone->online_pagesfor coldplug:mm/mm_init.c +3
-3@@ -845,9 +845,9 @@ overlap_memmap_init(unsigned long zone, unsignedlong *pfn) * zone/node above the hole except for the trailing pages in the last * section that will be appended to the zone/node below. */ -static void __init init_unavailable_range (unsigned long spfn, - unsigned long epfn, - int zone, int node) +static u64 __init init_unavailable_range (unsigned long spfn, + unsigned long epfn, + int zone, int node) { unsigned long pfn;u64 pgcnt = 0;
mm/mm_init.c +2
-0@@ -861,6 +861,8 @@ static void __init init_unavailable_range(unsigned long spfn, if (pgcnt) pr_info("On node %d, zone %s: %lld pages in unavailable ranges\n", node, zone_names[zone], pgcnt); + + return pgcnt; } /*@@ -959,9 +961,10 @@ static void __init memmap_init_zone_range(struct zone *zone,mm/mm_init.c static void __init memmap_init_zone_range(struct · line 970
Quoted 10 lines
mm/mm_init.c +2
-1memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE, false); + zone->online_pages += (end_pfn - start_pfn); if (*hole_pfn < start_pfn) - init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); + zone->online_pages += init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); *hole_pfn = end_pfn; }Looking at
set_zone_contiguous(),__pageblock_pfn_to_page()takes care of a weird case where the end of a zone falls into the middle of a pageblock.
Quoted 8 lines
I am not even sure if that is possible, but we could handle that easily in
pageblock_pfn_to_page()by checking the requested range against the zone spanned range.Then the semantics "
zone->online_pages" would be less weird and more closely resemble "pages with online memmap".
init_unavailable_range()might indeed do the trick!@Tianyou, can you explore that direction? I know, your PTO is coming up.
Hi David,
Sorry for the late response, just come back from CNY holiday. I am willing to explore the new direction. Will sync with Yuan in a day or two to confirm the overall implementation and testing plan. Thanks in advance for your time to review the new patch set once ready.
Regards,
Tianyou
Re: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Hi David, Mike & Other maintainers,
Just a quick update: Yuan and I created a new patch based on David and Mike's code, updated the documentation as well, performed the performance test, double check all the references etc. We observed a slightly performance regression, not sure it was caused by the environment configuration or we triggered some unexpected behavior. We might need a bit more time to investigate. Will send out the patch soon once we have confirmed data for the performance regression. Thanks.
Regards,
Tianyou
Quoted 255 lines · Tianyou
Quoted 233 lines · David Hildenbrand (Arm)
Quoted 10 lines
Quoted 9 lines
*
+ *online_pagesis pages within the zone that have an online memmap.
+ *online_pagesinclude present pages and memory holes that have a
+ * memmap. Whenspanned_pages==online_pages,pfn_to_page()can be
+ * performed without further checks on any pfn within the zone span.Quoted 2 lines
Maybe
pages_with_memmap? It would stand off from managed, spanned and present, but it's clearer than online IMHO.offline pages also have a memmap, but that should not be touched as it might contain garbage. So it's a bit more tricky :)
Quoted 205 lines
Quoted 165 lines
+ *
* Sopresent_pagesmay be used by memory hotplug or memory power
* management logic to figure out unmanaged pages by checking
* (present_pages-managed_pages). Andmanaged_pagesshould be used(quoted diff) +6
-2@@ -967,6 +972,7 @@ struct zone { atomic_long_t managed_pages; unsigned long spanned_pages; unsigned long present_pages; + unsigned long online_pages; #if defined(CONFIG_MEMORY_HOTPLUG) unsigned long present_early_pages; #endif@@ -1051,8 +1057,6 @@ struct zone { bool compact_blockskip_flush; #endif - bool contiguous; - CACHELINE_PADDING(_pad3_); /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];@@ -1124,6 +1128,23 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn) return zone->zone_start_pfn <= pfn && pfn < zone_end_pfn(zone); } +/** + * zone_is_contiguous - test whether a zone is contiguous + * @zone: the zone to test. + * + * In a contiguous zone, it is valid to call pfn_to_page() on anypfn in the
+ * spanned zone without requitingpfn_valid()orpfn_to_online_page()checks.
+ *
+ * Returns: true if contiguous, otherwise false.
+ */(quoted diff) +3
-0+static inline bool zone_is_contiguous(const struct zone *zone) +{ + return READ_ONCE(zone->spanned_pages) ==
READ_ONCE(zone->online_pages);(quoted diff) +2
-0+} +static inline bool
zone_is_initialized(const struct zone *zone){
returnzone->initialized;mm/internal.h +1
-7@@ -716,21 +716,15 @@ extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn, static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn, unsigned long end_pfn, struct zone *zone) { - if (zone->contiguous) + if (zone_is_contiguous(zone)) return pfn_to_page(start_pfn); return __pageblock_pfn_to_page(start_pfn, end_pfn, zone); } -void set_zone_contiguous(struct zone *zone); bool pfn_range_intersects_zones(int nid, unsigned long start_pfn, unsigned long nr_pages); -static inline void clear_zone_contiguous (struct zone *zone) -{ - zone->contiguous = false; -} - extern int __isolate_free_page(struct page *page, unsigned int order); extern void __putback_isolated_page(struct page *page, unsigned int order, int mt);mm/memory_hotplug.c +2
-2@@ -492,11 +492,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn = find_smallest_section_pfn(nid, zone, end_pfn, zone_end_pfn(zone)); if (pfn) { - zone->spanned_pages = zone_end_pfn(zone) - pfn; + WRITE_ONCE(zone->spanned_pages, zone_end_pfn(zone) - pfn); zone->zone_start_pfn = pfn; } else { zone->zone_start_pfn = 0; - zone->spanned_pages = 0; + WRITE_ONCE(zone->spanned_pages, 0); }} else if (
zone_end_pfn(zone)==end_pfn) {
/*mm/memory_hotplug.c +2
-2@@ -508,10 +508,10 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, pfn = find_biggest_section_pfn(nid, zone, zone->zone_start_pfn, start_pfn); if (pfn) - zone->spanned_pages = pfn - zone->zone_start_pfn + 1; + WRITE_ONCE(zone->spanned_pages, pfn - zone->zone_start_pfn +1 ); else { zone->zone_start_pfn = 0; - zone->spanned_pages = 0; + WRITE_ONCE(zone->spanned_pages, 0); }}
}mm/memory_hotplug.c +2
-10@@ -565,18 +565,13 @@ void remove_pfn_range_from_zone(struct zone *zone, /* * Zone shrinking code cannot properly deal with ZONE_DEVICE. So - * we will not try to shrink the zones - which is okay as - * set_zone_contiguous() cannot deal with ZONE_DEVICE either way. + * we will not try to shrink the zones. */ if (zone_is_zone_device(zone)) return; - clear_zone_contiguous(zone); - shrink_zone_span(zone, start_pfn, start_pfn + nr_pages); update_pgdat_span(pgdat); - - set_zone_contiguous(zone); } /**@@ -753,8 +748,6 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn, struct pglist_data *pgdat = zone->zone_pgdat; int nid = pgdat->node_id; - clear_zone_contiguous(zone); - if (zone_is_empty(zone)) init_currently_empty_zone(zone, start_pfn, nr_pages); resize_zone_range(zone, start_pfn, nr_pages);@@ -782,8 +775,6 @@ void move_pfn_range_to_zone(struct zone *zone,unsigned long start_pfn, memmap_init_range(nr_pages, nid, zone_idx(zone), start_pfn, 0, MEMINIT_HOTPLUG, altmap, migratetype, isolate_pageblock); - - set_zone_contiguous(zone); } struct auto_movable_stats {@@ -1079,6 +1070,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group, if (early_section(__pfn_to_section(page_to_pfn(page)))) zone->present_early_pages += nr_pages; zone->present_pages += nr_pages; + WRITE_ONCE(zone->online_pages, zone->online_pages + nr_pages); zone->zone_pgdat->node_present_pages += nr_pages; if (group && movable)mm/mm_init.c +2
-1@@ -2263,9 +2263,10 @@ void __init init_cma_pageblock(struct page *page) } #endif -void set_zone_contiguous (struct zone *zone) +static void calc_online_pages (struct zone *zone) { unsigned long block_start_pfn = zone->zone_start_pfn; + unsigned long online_pages = 0; unsigned long block_end_pfn; block_end_pfn = pageblock_end_pfn(block_start_pfn);mm/mm_init.c void set_zone_contiguous(struct zone *zone) · line 2282
Quoted 6 lines
mm/mm_init.c +2
-1@@ -2277,12 +2278,11 @@ void set_zone_contiguous(struct zone *zone) if (!__pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone)) - return; + continue; cond_resched(); + online_pages += block_end_pfn - block_start_pfn;I think we can completely get rid of this with something like this untested patch to calculate
zone->online_pagesfor coldplug:mm/mm_init.c +3
-3@@ -845,9 +845,9 @@ overlap_memmap_init(unsigned long zone, unsignedlong *pfn) * zone/node above the hole except for the trailing pages in the last * section that will be appended to the zone/node below. */ -static void __init init_unavailable_range (unsigned long spfn, - unsigned long epfn, - int zone, int node) +static u64 __init init_unavailable_range (unsigned long spfn, + unsigned long epfn, + int zone, int node) { unsigned long pfn;u64 pgcnt = 0;
mm/mm_init.c +2
-0@@ -861,6 +861,8 @@ static void __init init_unavailable_range(unsigned long spfn, if (pgcnt) pr_info("On node %d, zone %s: %lld pages in unavailable ranges\n", node, zone_names[zone], pgcnt); + + return pgcnt; } /*@@ -959,9 +961,10 @@ static void __init memmap_init_zone_range(struct zone *zone,mm/mm_init.c static void __init · line 970
Quoted 10 lines
mm/mm_init.c +2
-1memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE, false); + zone->online_pages += (end_pfn - start_pfn); if (*hole_pfn < start_pfn) - init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); + zone->online_pages += init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); *hole_pfn = end_pfn; }Looking at
set_zone_contiguous(),__pageblock_pfn_to_page()takes care of a weird case where the end of a zone falls into the middle of a pageblock.Quoted 8 lines
I am not even sure if that is possible, but we could handle that easily in
pageblock_pfn_to_page()by checking the requested range against the zone spanned range.Then the semantics "
zone->online_pages" would be less weird and more closely resemble "pages with online memmap".
init_unavailable_range()might indeed do the trick!@Tianyou, can you explore that direction? I know, your PTO is coming up.
Hi David,
Sorry for the late response, just come back from CNY holiday. I am willing to explore the new direction. Will sync with Yuan in a day or two to confirm the overall implementation and testing plan. Thanks in advance for your time to review the new patch set once ready.
Regards,
Tianyou
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 1 line · Tianyou
Hi David, Mike & Other maintainers,
Hi!
Quoted 8 lines
Just a quick update: Yuan and I created a new patch based on David and Mike's code, updated the documentation as well, performed the performance test, double check all the references etc. We observed a slightly performance regression, not sure it was caused by the environment configuration or we triggered some unexpected behavior. We might need a bit more time to investigate. Will send out the patch soon once we have confirmed data for the performance regression. Thanks.
A performance regression would be rather unexpected, unless we somehow don't ever set zone->contiguous=true any more :)
In which test / code path did you observe a regression?
Signature · 3 lines
Cheers, David
Re: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 1 line · Tianyou
Hi David, Mike & Other maintainers,
Hi!
Quoted 7 lines
Just a quick update: Yuan and I created a new patch based on David and Mike's code, updated the documentation as well, performed the performance test, double check all the references etc. We observed a slightly performance regression, not sure it was caused by the environment configuration or we triggered some unexpected behavior. We might need a bit more time to investigate. Will send out the patch soon once we have confirmed data for the performance regression. Thanks.
Quoted 4 lines
A performance regression would be rather unexpected, unless we somehow don't ever set zone->contiguous=true any more :)
In which test / code path did you observe a regression?
Sorry for the late response, I missed the email by accident. The test is simply hotplug and unplug the 256GB memory, the time became 3s instead of 2s for some reason compared to previous patch. Interestingly, for 512GB memory hotplug/unplug, the time remains as 6s. What we planned to do are: 1. double/triple check with the same and other servers; 2. collect the perf profile to see what is the diff compared with previous patch, if the regression consistently persisted.
I will work with Yuan on this and hopefully we can have more information to be shared here soon. Thanks.
Regards,
Tianyou
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
Quoted 16 lines · Tianyou
Quoted 14 lines · David Hildenbrand (Arm)
Quoted 1 line · Tianyou
Hi David, Mike & Other maintainers,
Hi!
Quoted 7 lines
Just a quick update: Yuan and I created a new patch based on David and Mike's code, updated the documentation as well, performed the performance test, double check all the references etc. We observed a slightly performance regression, not sure it was caused by the environment configuration or we triggered some unexpected behavior. We might need a bit more time to investigate. Will send out the patch soon once we have confirmed data for the performance regression. Thanks.
A performance regression would be rather unexpected, unless we somehow don't ever set zone->contiguous=true any more :)
Quoted 6 lines
Quoted 1 line
In which test / code path did you observe a regression?
Sorry for the late response, I missed the email by accident. The test is simply hotplug and unplug the 256GB memory, the time became 3s instead of 2s for some reason compared to previous patch.
Weird, because we should be doing less work :)
Quoted 5 lines
Interestingly, for
512GB memory hotplug/unplug, the time remains as 6s. What we planned to do are: 1. double/triple check with the same and other servers; 2. collect the perf profile to see what is the diff compared with previous patch, if the regression consistently persisted.
Sounds good.
Quoted 3 lines
I will work with Yuan on this and hopefully we can have more information to be shared here soon. Thanks.
Greta, thanks. Feel free to send out a new version in the meantime.
Signature · 3 lines
Cheers, David
Re: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
In reply to Li, TianyouHi David,
Yuan has create another patch with test result. He will submit the patch soon. Thanks in advance for your kind review.
Regards,
Tianyou
Quoted 28 lines · Tianyou
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 1 line · Tianyou
Hi David, Mike & Other maintainers,
Hi!
Quoted 7 lines
Just a quick update: Yuan and I created a new patch based on David and Mike's code, updated the documentation as well, performed the performance test, double check all the references etc. We observed a slightly performance regression, not sure it was caused by the environment configuration or we triggered some unexpected behavior. We might need a bit more time to investigate. Will send out the patch soon once we have confirmed data for the performance regression. Thanks.
Quoted 4 lines
A performance regression would be rather unexpected, unless we somehow don't ever set zone->contiguous=true any more :)
In which test / code path did you observe a regression?
Sorry for the late response, I missed the email by accident. The test is simply hotplug and unplug the 256GB memory, the time became 3s instead of 2s for some reason compared to previous patch. Interestingly, for 512GB memory hotplug/unplug, the time remains as 6s. What we planned to do are: 1. double/triple check with the same and other servers; 2. collect the perf profile to see what is the diff compared with previous patch, if the regression consistently persisted.
I will work with Yuan on this and hopefully we can have more information to be shared here soon. Thanks.
Regards,
Tianyou
David Hildenbrand (Arm)
maintainerRe: [PATCH v9 2/2] mm/memory hotplug/unplug: Optimize zone->contiguous update when changes pfn range
In reply to Mike RapoportQuoted 139 lines · Mike Rapoport
Quoted 90 lines · David Hildenbrand (Arm)
Quoted 71 lines · Tianyou Li
When invoke
move_pfn_range_to_zoneorremove_pfn_range_from_zone, it will update thezone->contiguousby checking the new zone's pfn range from the beginning to the end, regardless the previous state of the old zone. When the zone's pfn range is large, the cost of traversing the pfn range to update thezone->contiguouscould be significant.Add fast paths to quickly detect cases where zone is definitely not contiguous without scanning the new zone. The cases are: when the new range did not overlap with previous range, the contiguous should be false; if the new range adjacent with the previous range, just need to check the new range; if the new added pages could not fill the hole of previous zone, the contiguous should be false.
The following test cases of memory hotplug for a VM [1], tested in the environment [2], show that this optimization can significantly reduce the memory hotplug time [3].
+----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Plug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 33s | 6s | 81% | +----------------+------+---------------+--------------+----------------++----------------+------+---------------+--------------+----------------+ | | Size | Time (before) | Time (after) | Time Reduction | | +------+---------------+--------------+----------------+ | Unplug Memory | 256G | 10s | 2s | 80% | | +------+---------------+--------------+----------------+ | | 512G | 34s | 6s | 82% | +----------------+------+---------------+--------------+----------------+[1] Qemu commands to hotplug 256G/512G memory for a VM:
object_addmemory-backend-ram,id=hotmem0,size=256G/512G,share=ondevice_addvirtio-mem-pci,id=vmem1,memdev=hotmem0,bus=port1 qom-set vmem1 requested-size 256G/512G (Plug Memory) qom-set vmem1 requested-size 0G (Unplug Memory)[2] Hardware : Intel Icelake server
Guest Kernel : v6.18-rc2 Qemu : v9.0.0 Launch VM : qemu-system-x86_64 -accel kvm -cpu host \ -drive file=./Centos10_cloud.qcow2,format=qcow2,if=virtio \ -drive file=./seed.img,format=raw,if=virtio \ -smp 3,cores=3,threads=1,sockets=1,maxcpus=3 \ -m 2G,slots=10,maxmem=2052472M \ -device pcie-root-port,id=port1,bus=pcie.0,slot=1,multifunction=on \ -device pcie-root-port,id=port2,bus=pcie.0,slot=2 \ -nographic -machine q35 \ -nic user,hostfwd=tcp::3000-:22 Guest kernel auto-onlines newly added memory blocks: echo online > /sys/devices/system/memory/auto_online_blocks[3] The time from typing the QEMU commands in [1] to when the output of 'grep MemTotal
/proc/meminfo' on Guest reflects that all hotplugged memory is recognized.Reported-by Nanhai Zou <[email protected]>Reported-by Chen Zhang <[email protected]>Tested-by Yuan Liu <[email protected]>Reviewed-by Tim Chen <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Yu C Chen <[email protected]>Reviewed-by Pan Deng <[email protected]>Reviewed-by Nanhai Zou <[email protected]>Reviewed-by Yuan Liu <[email protected]>Signed-off-by Tianyou Li <[email protected]>Quoted 1 line
---
Thanks for all your work on this and sorry for being slower with review the last month.
While I was in the shower I was thinking about how much I hate
zone->contiguous+ the pageblock walking, and how we could just get rid of it.You know, just what you do while having a relaxing shower.
And I was wondering:
(a) in which case would we have
zone_spanned_pages==zone_present_pagesand the zone *not* being contiguous? I assume this just cannot happen, otherwise BUG.Quoted 6 lines
(b) in which case would we have
zone_spanned_pages!=zone_present_pagesand the zone *being* contiguous? I assume in some cases where we have small holes within a pageblock?Reading the doc of
__pageblock_pfn_to_page(), there are some weird scenarios with holes in pageblocks.It seems that "
zone->contigous" is really bad name for what this thing represents.tl;dr I don't think
zone_spanned_pages==zone_present_pagesis related tozone->contigousat all :)If you look at
pageblock_pfn_to_page()and__pageblock_pfn_to_page(), the check forzone->contigousshould guarantee that the entire pageblock has a valid memory map and that the entire pageblock fits a zone and does not cross zone/node boundaries.For coldplug memory the memory map is valid for every section that has present memory, i.e. even it there is a hole in a section, it's memory map will be populated and will have struct pages.
When
zone->contigousis false, the slow path in__pageblock_pfn_to_page()essentially checks if the first page in a pageblock is online and if first and last pages are in the zone being compacted.AFAIU, in the hotplug case an entire pageblock is always onlined to the same zone, so
zone->contigouswon't change after the hotplug is complete.We might set it to false in the beginning of the hotplug to avoid scanning offline pages, although I'm not sure if it's possible.
But in the end of hotplug we can simply restore the old value and move on.
For the coldplug case I'm also not sure it's worth the hassle, we could just let compaction scan a few more pfns for those rare weird pageblocks and bail out on wrong page conditions.
Quoted 9 lines
I.e., on my notebook I have
$ cat /proc/zoneinfo | grep -E "Node|spanned|present" Node 0, zone DMA spanned 4095 present 3999 Node 0, zone DMA32 spanned 1044480 present 439600
Quoted 1 line
I suspect this one is contigous ;-)
Just checked. It's not. Probably because there are some holes that are entirely without a memmap. (PCI hole)
Signature · 3 lines
Cheers, David