mm/page_alloc: remove prefetchw() on freeing page to buddy system
Subsystems (2)
PATCH 1/1 mm/page_alloc: remove prefetchw() on freeing page to buddy system
The prefetchw() is introduced from an ancient patch[1].
The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
---
The patch is based on mm-stable with David's change.
--- mm/page_alloc.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)
mm/page_alloc.c +2 -11
@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,*/ if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); } -VM_WARN_ON_ONCE (PageReserved (p)); -__ClearPageOffline (p); -set_page_count (p,0 );/* * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,debug_pagealloc_map_pages (page, nr_pages);adjust_managed_page_count (page, nr_pages); }else { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p); -set_page_count (p,0 );/* memblock adjusts totalram_pages() manually. */ atomic_long_add (nr_pages, &page_zone (page)->managed_pages);
Signature · 1 line
2.34.1
Andrew Morton
maintainerQuoted 11 lines · On Tue, 2 Jul 2024 02:09:31 +0000 Wei Yang <[email protected]> wrote:
The
prefetchw()is introduced from an ancient patch[1].The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 8 lines
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.
I don't think I've ever seen prefetch make a damn bit of difference.
Quoted 8 lines
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.
Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.
Quoted 30 lines · Andrew Morton
Quoted 11 lines · On Tue, 2 Jul 2024 02:09:31 +0000 Wei Yang <[email protected]> wrote:
The
prefetchw()is introduced from an ancient patch[1].The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 8 lines
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.I don't think I've ever seen prefetch make a damn bit of difference.
Quoted 6 lines
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>
Quoted 6 lines
Quoted 2 lines
---
The patch is based on mm-stable with David's change.Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.
The commit is 3dadec1babf9eee0c67c967df931d6f0cb124a04
mm: pass meminit_context to __free_pages_core()
The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/
BTW, how we track the mail link? Just search in lore.kernel.org?
Signature · 2 lines
Wei Yang Help you, Help me
Andrew Morton
maintainerQuoted 7 lines · On Wed, 3 Jul 2024 00:01:42 +0000 Wei Yang <[email protected]> wrote:
Quoted 7 lines
Quoted 6 lines
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.
Quoted 7 lines
Quoted 3 lines
Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.
The commit is
3dadec1babf9eee0c67c967df931d6f0cb124a04mm: pass
meminit_contextto__free_pages_core()
OK, that's in mm-stable.
Quoted 4 lines
The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/BTW, how we track the mail link? Just search in lore.kernel.org?
I'm not sure what you mean? I extract the message-id from the email and concat it with "https://lkml.kernel.org/r".
Quoted 19 lines · Andrew Morton
Quoted 7 lines · On Wed, 3 Jul 2024 00:01:42 +0000 Wei Yang <[email protected]> wrote:
Quoted 7 lines
Quoted 6 lines
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.Quoted 7 lines
Quoted 3 lines
Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.
The commit is
3dadec1babf9eee0c67c967df931d6f0cb124a04mm: pass
meminit_contextto__free_pages_core()OK, that's in mm-stable.
Quoted 7 lines
Quoted 4 lines
The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/BTW, how we track the mail link? Just search in lore.kernel.org?
I'm not sure what you mean? I extract the message-id from the email and concat it with "https://lkml.kernel.org/r".
Thanks, I got something new :-)
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 11 lines · Wei Yang
The
prefetchw()is introduced from an ancient patch[1].The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 8 lines
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.
I suspect this is noise, though.
Quoted 19 lines
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.--- mm/page_alloc.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)mm/page_alloc.c +0
-0@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,*/
mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243
Quoted 10 lines
mm/page_alloc.c +1
-3if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); }
Something like:
for (;;) {
...
if (++loop >= nr_pages)
break;
p++;
}Might generate slightly better code, because we know that we execute the loop body at least once. We use that in set_ptes(), for example.
Quoted 12 lines
mm/page_alloc.c +0
-5-VM_WARN_ON_ONCE (PageReserved (p)); -__ClearPageOffline (p); -set_page_count (p,0 );/* * Freeing the page with debug_pagealloc enabled will try to @@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,debug_pagealloc_map_pages (page, nr_pages);adjust_managed_page_count (page, nr_pages); }else { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) {
mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1259
Quoted 10 lines
mm/page_alloc.c +1
-3-prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p); -set_page_count (p,0 );/* memblock adjusts totalram_pages() manually. */ atomic_long_add (nr_pages, &page_zone (page)->managed_pages);
Much better
Signature · 3 lines
Cheers, David / dhildenb
Quoted 55 lines · David Hildenbrand
Quoted 11 lines · Wei Yang
The
prefetchw()is introduced from an ancient patch[1].The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 8 lines
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.I suspect this is noise, though.
Quoted 19 lines
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.--- mm/page_alloc.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)mm/page_alloc.c +0
-0@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,*/ mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243
Quoted 10 lines
mm/page_alloc.c +1
-3if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); }Something like:
Quoted 7 lines
for (;;) {
...
if (++loop >= nr_pages)
break;
p++;
}So you prefer to have another version with this format? Sth like this?
mm/page_alloc.c +9 -3
@@ -1224,7 +1224,7 @@ void __meminit __free_pages_core(struct page *page, unsigned int order, {unsigned int nr_pages =1 << order;struct page *p = page; -unsigned int loop; +unsigned int loop =0 ;/* * When initializing the memmap, __init_single_page() sets the refcount
@@ -1236,10 +1236,13 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,*/ if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -for (loop =0 ; loop < nr_pages; loop++, p++) { +for (;;) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); +if (++loop >= nr_pages) +break ; + p++; }/*
@@ -1250,9 +1253,12 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,debug_pagealloc_map_pages (page, nr_pages);adjust_managed_page_count (page, nr_pages); }else { -for (loop =0 ; loop < nr_pages; loop++, p++) { +for (;;) {__ClearPageReserved (p);set_page_count (p,0 ); +if (++loop >= nr_pages) +break ; + p++; }/* memblock adjusts totalram_pages() manually. */
Quoted 33 lines
Might generate slightly better code, because we know that we execute the loop body at least once. We use that in
set_ptes(), for example.Quoted 10 lines
mm/page_alloc.c +0
-4-VM_WARN_ON_ONCE (PageReserved (p)); -__ClearPageOffline (p); -set_page_count (p,0 );/* * Freeing the page with debug_pagealloc enabled will try to @@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,debug_pagealloc_map_pages (page, nr_pages);adjust_managed_page_count (page, nr_pages); }else { -prefetchw (p);mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1258
Quoted 10 lines
mm/page_alloc.c +1
-4-for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p); -set_page_count (p,0 );/* memblock adjusts totalram_pages() manually. */ atomic_long_add (nr_pages, &page_zone (page)->managed_pages);Much better
Reviewed-by David Hildenbrand <[email protected]>Signature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Wei Yang Help you, Help me
Quoted 64 lines · Wei Yang
Quoted 55 lines · David Hildenbrand
Quoted 11 lines · Wei Yang
The
prefetchw()is introduced from an ancient patch[1].The change log says:
The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").
Quoted 8 lines
So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.I suspect this is noise, though.
Quoted 19 lines
[1]: https://lore.kernel.org/linux-ia64/[email protected]/
Signed-off-by Wei Yang <[email protected]>Suggested-by Matthew Wilcox <[email protected]>CC David Hildenbrand <[email protected]>---
The patch is based on mm-stable with David's change.--- mm/page_alloc.c | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)mm/page_alloc.c +0
-0@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,*/ mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243
Quoted 10 lines
mm/page_alloc.c +1
-3if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); }Something like:
Quoted 7 lines
for (;;) { ... if (++loop >= nr_pages) break; p++; }
Quoted 1 line
So you prefer to have another version with this format? Sth like this?
Whatever you prefer, just pointing it out that we can do slightly better :)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 6 lines · David Hildenbrand
Quoted 4 lines
I did 10 round bootup tests before and after this change, the data doesn't prove
prefetchw()help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time afterprefetchw()removal even 5.2% faster than before.I suspect this is noise, though.
I think it's real, though small. Each prefetchw() is an instruction, and if we can avoid issuing an instruction, we should.
Quoted 10 lines
Something like:
for (;;) { ... if (++loop >= nr_pages) break; p++; }
Quoted 2 lines
Might generate slightly better code, because we know that we execute the loop body at least once. We use that in
set_ptes(), for example.
I don't think it's worth doing. Keep the loop simple and obvious. set_ptes() is different because we actually expect to execute the loop exactly once (ie most folios are small). So two compares per call to set_ptes() instead of one makes a difference. Here, we're expecting to execute this loop, what, a million times? Doing a million-and-one compares instead of a million makes no observable difference.
I would like to see v2 of this patch dropped, please Andrew.
Andrew Morton
maintainerQuoted 13 lines · On Thu, 4 Jul 2024 04:30:50 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 10 lines
Something like:
for (;;) { ... if (++loop >= nr_pages) break; p++; }Quoted 2 lines
Might generate slightly better code, because we know that we execute the loop body at least once. We use that in
set_ptes(), for example.
Quoted 8 lines
I don't think it's worth doing. Keep the loop simple and obvious.
set_ptes()is different because we actually expect to execute the loop exactly once (ie most folios are small). So two compares per call toset_ptes()instead of one makes a difference. Here, we're expecting to execute this loop, what, a million times? Doing a million-and-one compares instead of a million makes no observable difference.I would like to see v2 of this patch dropped, please Andrew.
thud.
Are you supportive of v1?
mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-remove-prefetchw-on-freeing-page-to-buddy-system +2 -11
@@ -1236,16 +1236,11 @@ void __free_pages_core(struct page *page*/ if (IS_ENABLED (CONFIG_MEMORY_HOTPLUG ) &&unlikely (context ==MEMINIT_HOTPLUG )) { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {VM_WARN_ON_ONCE (PageReserved (p));__ClearPageOffline (p);set_page_count (p,0 ); } -VM_WARN_ON_ONCE (PageReserved (p)); -__ClearPageOffline (p); -set_page_count (p,0 );/* * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __free_pages_core(struct page *pagedebug_pagealloc_map_pages (page, nr_pages);adjust_managed_page_count (page, nr_pages); }else { -prefetchw (p); -for (loop =0 ; loop < (nr_pages -1 ); loop++, p++) { -prefetchw (p +1 ); +for (loop =0 ; loop < nr_pages; loop++, p++) {__ClearPageReserved (p);set_page_count (p,0 ); } -__ClearPageReserved (p); -set_page_count (p,0 );/* memblock adjusts totalram_pages() manually. */ atomic_long_add (nr_pages, &page_zone (page)->managed_pages);
_
Quoted 1 line · Andrew Morton
Are you supportive of v1?
Oh yes. I think it has my suggested-by on it, so I didn't see the need to add an R-b.
Andrew Morton
maintainerQuoted 5 lines · On Thu, 4 Jul 2024 04:39:42 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 1 line · Andrew Morton
Are you supportive of v1?
Oh yes. I think it has my suggested-by on it, so I didn't see the need to add an R-b.
I'm a sucker for r-bys. Added, thanks.