mm: use zonelist_zone() to get zone
Subsystems (5)
PATCH 1/4 mm: use zonelist_zone() to get zone
Instead of accessing zoneref->zone directly, use zonelist_zone() like other places for consistency.
No functional change.
--- include/linux/mmzone.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mmzone.h +1 -1
@@ -1688,7 +1688,7 @@ static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist, zone = zonelist_zone(z))#define for_next_zone_zonelist_nodemask (zone, z, highidx, nodemask) \ - for (zone = z->zone; \ + for (zone = zonelist_zone(z); \ zone; \ z = next_zones_zonelist(++z, highidx, nodemask), \ zone = zonelist_zone(z))
Signature · 1 line
2.34.1
PATCH 2/4 mm: not __SetPageReserved on initializing hot-plugged memory
Initialize all pages reserved is an ancient behavior.
Since commit 92923ca3aace ("mm: meminit: only set page reserved in the memblock region"), SetPageReserved is removed from __init_single_page(). Only those reserved pages are marked PG_reserved.
But we still set PG_reserved on offline and check it on online.
Following two commits removed both of them:
* Commit 0ee5f4f31d36 ("mm/page_alloc.c: don't set pages PageReserved() when offlining") removed the set on offline.
* Commit 5ecae6359e3a ("mm/memory_hotplug: drop PageReserved() check in online_pages_range()") removed the check on online.
This means we set PG_reserved for hot-plugged memory at initialization is not helpful and a little different from bootmem initialization path. Now we can remove it.
Memory hot-add and hot-remove have been tested.
--- mm/mm_init.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
mm/mm_init.c +2 -5
@@ -839,9 +839,8 @@ static void __init init_unavailable_range(unsigned long spfn, }/* -* Initially all pages are reserved - free ones are freed -* up by memblock_free_all() once the early boot process is -* done. Non-atomic initialization, single-pass. +* Free ones are freed up by memblock_free_all() once the early boot process +* is done. Non-atomic initialization, single-pass. * * All aligned pageblocks are initialized to the specified migratetype * (usually MIGRATE_MOVABLE). Besides setting the migratetype, no related
@@ -892,8 +891,6 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone page =pfn_to_page (pfn);__init_single_page (page, pfn, zone, nid); -if (context ==MEMINIT_HOTPLUG ) -__SetPageReserved (page);/* * Usually, we want to mark the pageblock MIGRATE_MOVABLE,
Signature · 1 line
2.34.1
Re: [PATCH 2/4] mm: not __SetPageReserved on initializing hot-plugged memory
Quoted 10 lines · Wei Yang
Initialize all pages reserved is an ancient behavior.
Since commit
92923ca3aace("mm: meminit: only set page reserved in the memblock region"), SetPageReserved is removed from__init_single_page(). Only those reserved pages are markedPG_reserved.But we still set
PG_reservedon offline and check it on online.Following two commits removed both of them:
Quoted 8 lines
* Commit
0ee5f4f31d36("mm/page_alloc.c: don't set pagesPageReserved()when offlining") removed the set on offline.
* Commit5ecae6359e3a("mm/memory_hotplug: dropPageReserved()check inonline_pages_range()") removed the check on online.This means we set
PG_reservedfor hot-plugged memory at initialization is not helpful and a little different from bootmem initialization path. Now we can remove it.
It's not that easy for ZONE_DEVICE.
Also, see mm/mm-stable
commit 3dadec1babf9eee0c67c967df931d6f0cb124a04
Author: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:36 2024 +0200
mm: pass meminit_context to __free_pages_core()
Patch series "mm/memory_hotplug: use PageOffline() instead of PageReserved() for !ZONE_DEVICE".
commit b873faaa609ab44c223b2327f55d2b6a2ba4ca9c
Author: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:37 2024 +0200
mm/memory_hotplug: initialize memmap of !ZONE_DEVICE withPageOffline() instead of PageReserved()
If you want to work on removing it for ZONE_DEVICE, one idea is to replace all relevant PageReserved() checks by a more generic function that would check PageReserved() and the zone
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/4] mm: not __SetPageReserved on initializing hot-plugged memory
Quoted 34 lines · David Hildenbrand
Quoted 10 lines · Wei Yang
Initialize all pages reserved is an ancient behavior.
Since commit
92923ca3aace("mm: meminit: only set page reserved in the memblock region"), SetPageReserved is removed from__init_single_page(). Only those reserved pages are markedPG_reserved.But we still set
PG_reservedon offline and check it on online.Following two commits removed both of them:
Quoted 8 lines
* Commit
0ee5f4f31d36("mm/page_alloc.c: don't set pagesPageReserved()when offlining") removed the set on offline.
* Commit5ecae6359e3a("mm/memory_hotplug: dropPageReserved()check inonline_pages_range()") removed the check on online.This means we set
PG_reservedfor hot-plugged memory at initialization is not helpful and a little different from bootmem initialization path. Now we can remove it.It's not that easy for
ZONE_DEVICE.Also, see
mm/mm-stablecommit
3dadec1babf9eee0c67c967df931d6f0cb124a04Author: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:36 2024 +0200mm: pass
meminit_contextto__free_pages_core()Patch series "
mm/memory_hotplug: usePageOffline()instead ofPageReserved()for !ZONE_DEVICE".
Quoted 7 lines
commit
b873faaa609ab44c223b2327f55d2b6a2ba4ca9cAuthor: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:37 2024 +0200
mm/memory_hotplug: initialize memmap of !ZONE_DEVICE withPageOffline()
instead ofPageReserved()
Let me try to understand this.
You also tries to get rid of PG_reserved but you want PG_offline instead, because this benefit virtio-mem, right?
But I don't get why PG_offline is wrong for ZONE_DEVICE. I may miss some knowledge for it.
Quoted 10 lines
If you want to work on removing it for
ZONE_DEVICE, one idea is to replace all relevantPageReserved()checks by a more generic function that would checkPageReserved()and the zoneSignature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 2/4] mm: not __SetPageReserved on initializing hot-plugged memory
Quoted 39 lines · Wei Yang
Quoted 38 lines · David Hildenbrand
Quoted 10 lines · Wei Yang
Initialize all pages reserved is an ancient behavior.
Since commit
92923ca3aace("mm: meminit: only set page reserved in the memblock region"), SetPageReserved is removed from__init_single_page(). Only those reserved pages are markedPG_reserved.But we still set
PG_reservedon offline and check it on online.Following two commits removed both of them:
Quoted 8 lines
* Commit
0ee5f4f31d36("mm/page_alloc.c: don't set pagesPageReserved()when offlining") removed the set on offline.
* Commit5ecae6359e3a("mm/memory_hotplug: dropPageReserved()check inonline_pages_range()") removed the check on online.This means we set
PG_reservedfor hot-plugged memory at initialization is not helpful and a little different from bootmem initialization path. Now we can remove it.It's not that easy for
ZONE_DEVICE.Also, see
mm/mm-stablecommit
3dadec1babf9eee0c67c967df931d6f0cb124a04Author: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:36 2024 +0200mm: pass
meminit_contextto__free_pages_core()Patch series "
mm/memory_hotplug: usePageOffline()instead ofPageReserved()for !ZONE_DEVICE".commit
b873faaa609ab44c223b2327f55d2b6a2ba4ca9cAuthor: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:37 2024 +0200
Quoted 8 lines
Quoted 3 lines
mm/memory_hotplug: initialize memmap of !ZONE_DEVICE withPageOffline()
instead ofPageReserved()Let me try to understand this.
You also tries to get rid of
PG_reservedbut you wantPG_offlineinstead, because this benefit virtio-mem, right?
We now make proper use of PG_offline. All hotplugged pages start out PG_offline once we turn the section online. Only the ones that actually get exposed to the buddy -- actually get onlined -- get PG_offline cleared. A side effect of that is less hacks for virtio-mem, and more natural handling for the other ballooning drivers that hotplug memory.
In the future, I'm planning on moving more fake-offlining code from virtio-mem the core, making use of more PG_offline in memory.
For now, it's stops the PG_reserved use while maintaining the same semantics as before: the page content and "struct page" is not to be touched by anybody except the "owner".
Quoted 3 lines
But I don't get why
PG_offlineis wrong forZONE_DEVICE. I may miss some knowledge for it.
I suggest you take a look at the PG_offline documentation. ZONE_DEVICE are certainly not logically offline pages. They will never be considered online as part of online sections. But they will never be handed to the buddy.
Maybe we want a dedicate page type for them in the future, not sure. We can right now identify them reliably using the zone idx.
Using a page type right now is very likely not possible, because we might be using the page->_mapcount in rmap code when mapping some of them to user space.
If we want to get rid of the PG_reserved for them right now, we'll have to make sure all existing PageReserved checks won't be degraded. For example, drivers/vfio/vfio_iommu_type1.c might need some work (no sure).
The KVM one in kvm_pfn_to_refcounted_page() should already be fine, because they really want to refcount them.
A lot of other ones like can_gather_numa_stats(), already refuse is_zone_device_page() manually, and maybe we want to factor both checks out into a separate function like "is_special_reserved_page()" or sth like that.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/4] mm: not __SetPageReserved on initializing hot-plugged memory
Quoted 81 lines · David Hildenbrand
Quoted 39 lines · Wei Yang
Quoted 38 lines · David Hildenbrand
Quoted 10 lines · Wei Yang
Initialize all pages reserved is an ancient behavior.
Since commit
92923ca3aace("mm: meminit: only set page reserved in the memblock region"), SetPageReserved is removed from__init_single_page(). Only those reserved pages are markedPG_reserved.But we still set
PG_reservedon offline and check it on online.Following two commits removed both of them:
Quoted 8 lines
* Commit
0ee5f4f31d36("mm/page_alloc.c: don't set pagesPageReserved()when offlining") removed the set on offline.
* Commit5ecae6359e3a("mm/memory_hotplug: dropPageReserved()check inonline_pages_range()") removed the check on online.This means we set
PG_reservedfor hot-plugged memory at initialization is not helpful and a little different from bootmem initialization path. Now we can remove it.It's not that easy for
ZONE_DEVICE.Also, see
mm/mm-stablecommit
3dadec1babf9eee0c67c967df931d6f0cb124a04Author: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:36 2024 +0200mm: pass
meminit_contextto__free_pages_core()Patch series "
mm/memory_hotplug: usePageOffline()instead ofPageReserved()for !ZONE_DEVICE".commit
b873faaa609ab44c223b2327f55d2b6a2ba4ca9cAuthor: David Hildenbrand <[email protected]> Date: Fri Jun 7 11:09:37 2024 +0200Quoted 8 lines
Quoted 3 lines
mm/memory_hotplug: initialize memmap of !ZONE_DEVICE withPageOffline()
instead ofPageReserved()Let me try to understand this.
You also tries to get rid of
PG_reservedbut you wantPG_offlineinstead, because this benefit virtio-mem, right?We now make proper use of
PG_offline. All hotplugged pages start outPG_offlineonce we turn the section online. Only the ones that actually get exposed to the buddy -- actually get onlined -- getPG_offlinecleared. A side effect of that is less hacks for virtio-mem, and more natural handling for the other ballooning drivers that hotplug memory.In the future, I'm planning on moving more fake-offlining code from virtio-mem the core, making use of more
PG_offlinein memory.For now, it's stops the
PG_reserveduse while maintaining the same semantics as before: the page content and "struct page" is not to be touched by anybody except the "owner".Quoted 3 lines
But I don't get why
PG_offlineis wrong forZONE_DEVICE. I may miss some knowledge for it.I suggest you take a look at the
PG_offlinedocumentation.ZONE_DEVICEare certainly not logically offline pages. They will never be considered online as part of online sections. But they will never be handed to the buddy.Maybe we want a dedicate page type for them in the future, not sure. We can right now identify them reliably using the zone idx.
Using a page type right now is very likely not possible, because we might be using the
page->_mapcountin rmap code when mapping some of them to user space.If we want to get rid of the
PG_reservedfor them right now, we'll have to make sure all existing PageReserved checks won't be degraded. For example,drivers/vfio/vfio_iommu_type1.cmight need some work (no sure).
Quoted 7 lines
The KVM one in
kvm_pfn_to_refcounted_page()should already be fine, because they really want to refcount them.A lot of other ones like
can_gather_numa_stats(), already refuseis_zone_device_page()manually, and maybe we want to factor both checks out into a separate function like "is_special_reserved_page()" or sth like that.
Thanks for the information, I need sometime to digest it :-)
Quoted 4 lines
Signature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Wei Yang Help you, Help me
PATCH 3/4 mm/page_alloc: put __free_pages_core() in __meminit section
In reply to Wei YangFunction __free_pages_core() is only used in bootmem init and hot-add memory init path. Let's put it in __meminit section.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1218,7 +1218,7 @@ static void __free_pages_ok(struct page *page, unsigned int order,__count_vm_events (PGFREE ,1 << order); } -void __free_pages_core (struct page *page,unsigned int order) +void __meminit __free_pages_core (struct page *page,unsigned int order) {unsigned int nr_pages =1 << order;struct page *p = page;
Signature · 1 line
2.34.1
PATCH 4/4 mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
In reply to Wei YangFunction __free_pages_core() is only used in the following two cases to put page to buddy system:
* free bootmem
* free hot-add memory
After the above cleanup, there is no case to free page with PG_reserved set. Let's remove the clear operation.
The page initialization time shows 6.5% faster with a 6G qemu virtual machine.
--- mm/page_alloc.c | 2 -- 1 file changed, 2 deletions(-)
mm/page_alloc.c +0 -2
@@ -1232,10 +1232,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order)prefetchw (p);for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {prefetchw (p +1 ); -__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p);set_page_count (p,0 );atomic_long_add (nr_pages, &page_zone (page)->managed_pages);
Signature · 1 line
2.34.1
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order) · line 1237
Quoted 10 lines · Wei Yang
+++
b/mm/page_alloc.cmm/page_alloc.c +0
-2@@ -1232,10 +1232,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order)prefetchw (p);for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {prefetchw (p +1 ); -__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p);set_page_count (p,0 );
Is the prefetchw() still useful? Any remotely competent CPU should be able to figure out this loop ...
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
Quoted 14 lines · Matthew Wilcox
mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order) · line 1237
Quoted 10 lines · Wei Yang
+++
b/mm/page_alloc.cmm/page_alloc.c +0
-2@@ -1232,10 +1232,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order)prefetchw (p);for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {prefetchw (p +1 ); -__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p);set_page_count (p,0 );Is the
prefetchw()still useful? Any remotely competent CPU should be able to figure out this loop ...
Hi, Matthew
Thanks for your question. But to be honest, I am not fully understand it.
Per my understanding, prefetchw() is trying to load data to cache before we really accessing it. By doing so, we won't hit a cache miss when we really need it.
This looks useful for me. And how remote competent CPU is involved in this process?
Signature · 2 lines
Wei Yang Help you, Help me
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
Quoted 16 lines · Wei Yang
Quoted 14 lines · Matthew Wilcox
mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order) · line 1237
Quoted 10 lines · Wei Yang
+++
b/mm/page_alloc.cmm/page_alloc.c +0
-2@@ -1232,10 +1232,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order)prefetchw (p);for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {prefetchw (p +1 ); -__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p);set_page_count (p,0 );Is the
prefetchw()still useful? Any remotely competent CPU should be able to figure out this loop ...
Quoted 3 lines
Hi, Matthew
Thanks for your question. But to be honest, I am not fully understand it.
Let's try this question:
If you remove the prefetchw() line, does the performance change?
Quoted 3 lines
Per my understanding,
prefetchw()is trying to load data to cache before we really accessing it. By doing so, we won't hit a cache miss when we really need it.
Yes, but the CPU can also do this by itself without needing an explicit hint from software. It can notice that we have a loop that's accessing successive cachelines for write. This is approximately the easiest prefetcher to design.
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
Quoted 9 lines · Matthew Wilcox
Quoted 3 lines · Wei Yang
Per my understanding,
prefetchw()is trying to load data to cache before we really accessing it. By doing so, we won't hit a cache miss when we really need it.Yes, but the CPU can also do this by itself without needing an explicit hint from software. It can notice that we have a loop that's accessing successive cachelines for write. This is approximately the easiest prefetcher to design.
I tracked down prefetchw() being added:
commit 3b901ea58a56
Author: Josh Aas <[email protected]> Date: Mon Aug 23 21:26:54 2004 -0700
[PATCH] improve speed of freeing bootmem
Attached is a patch that greatly improves the speed of freeing boot memory.
On ia64 machines with 2GB or more memory (I didn't test with less, but I
can't imagine there being a problem), the speed improvement is about 75% for the function free_all_bootmem_core. This translates to savings on the order of 1 minute / TB of memory during boot time. That number comes from testing on a machine with 512GB, and extrapolating based on profiling of an unpatched 4TB machine. For 4 and 8 TB machines, the time spent in this function is about 1 minutes/TB, which is painful especially given that there is no indication of what is going on put to the console (this issue to possibly be addressed later).
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
(quoting the entire commit message because it's buried in linux-fullhistory, being a pre-git patch). For the thread he's referring to, see
https://lore.kernel.org/linux-ia64/[email protected]/
Itanium CPUs of this era had no prefetchers.
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
Quoted 10 lines · Matthew Wilcox
Quoted 9 lines · Matthew Wilcox
Quoted 3 lines · Wei Yang
Per my understanding,
prefetchw()is trying to load data to cache before we really accessing it. By doing so, we won't hit a cache miss when we really need it.Yes, but the CPU can also do this by itself without needing an explicit hint from software. It can notice that we have a loop that's accessing successive cachelines for write. This is approximately the easiest prefetcher to design.
This is my first intuition on reading the code, while I *believed* there is a reason for the prefetchw() to be put here.
Quoted 26 lines
I tracked down
prefetchw()being added:commit
3b901ea58a56Author: Josh Aas <[email protected]> Date: Mon Aug 23 21:26:54 2004 -0700[PATCH] improve speed of freeing bootmem
Attached is a patch that greatly improves the speed of freeing boot memory.
On ia64 machines with 2GB or more memory (I didn't test with less, but I
can't imagine there being a problem), the speed improvement is about 75% for the functionfree_all_bootmem_core. This translates to savings on the order of 1 minute / TB of memory during boot time. That number comes from testing on a machine with 512GB, and extrapolating based on profiling of an unpatched 4TB machine. For 4 and 8 TB machines, the time spent in this function is about 1 minutes/TB, which is painful especially given that there is no indication of what is going on put to the console (this issue to possibly be addressed later).The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 4 lines
(quoting the entire commit message because it's buried in linux-fullhistory, being a pre-git patch). For the thread he's referring to, see
https://lore.kernel.org/linux-ia64/[email protected]/
Thanks for digging the ancient history. And now I know the linux-fullhistory tree :-)
Quoted 1 line
Itanium CPUs of this era had no prefetchers.
Oops, so sad to hear it.
So it looks the most helpful change is:
* free higher order pages
* non-atomic equivalent
A quick test on my 6G qemu virtual machine shows the prefetchw() here seems not helpful. After removal, the meminit process even a little bit faster. From 135ms to 121ms as the sum of 3 times bootup test.
If you think it is ok, I would wrap up one patch on current mm-stable for more audience to review.
Signature · 2 lines
Wei Yang Help you, Help me
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 4/4] mm/page_alloc: no need to ClearPageReserved on giving page to buddy system
Quoted 38 lines · David Hildenbrand
Quoted 37 lines · David Hildenbrand
Quoted 24 lines · Wei Yang
Function
__free_pages_core()is only used in the following two cases to put page to buddy system:* free bootmem
* free hot-add memoryAfter the above cleanup, there is no case to free page with
PG_reservedset. Let's remove the clear operation.The page initialization time shows 6.5% faster with a 6G qemu virtual machine.
Signed-off-by Wei Yang <[email protected]>CC David Hildenbrand <[email protected]>CC Mike Rapoport (IBM) <[email protected]>--- mm/page_alloc.c | 2 -- 1 file changed, 2 deletions(-)mm/page_alloc.c +0
-0mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order) · line 1238
Quoted 9 lines
mm/page_alloc.c +0
-2@@ -1232,10 +1232,8 @@ void __meminit __free_pages_core(struct page *page, unsigned int order)prefetchw (p);for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {prefetchw (p +1 ); -__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p);set_page_count (p,0 );atomic_long_add (nr_pages, &page_zone (page)->managed_pages);Again see
mm/mm-stablewhere that code changed.
Quoted 3 lines
Quoted 3 lines
I think we can still get reserved pages here, for example via
kmsan_memblock_free_pages().
You mean ZONDE_DEVICE pages?
If yes, I think the normal memblock_free_pages() still could get reserved pages. Am I right?
Quoted 2 lines
Sorry, I meant
kmsan_memblock_discard().
Yep.
Quoted 4 lines
Signature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Wei Yang Help you, Help me