Skip to contentSkip to conversation

mm/page_alloc: remove prefetchw() on freeing page to buddy system

PATCH Applied Wei Yang 4 participants 12 messages Jul 2, 2024 – Jul 4, 2024

Subsystems (2)

v1 of 2 Applied 1/1 Latest is v2 →

lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH 1/1 mm/page_alloc: remove prefetchw() on freeing page to buddy system

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

---
The patch is based on mm-stable with David's change.

---
 mm/page_alloc.c | 13 ++-----------
 1 file changed, 2 insertions(+), 11 deletions(-)
mm/page_alloc.c +2 -11
index 116ee33fd1ce..c46aedfc9a12 100644
@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 	 */
 	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
 	    unlikely(context == MEMINIT_HOTPLUG)) {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
 			VM_WARN_ON_ONCE(PageReserved(p));
 			__ClearPageOffline(p);
 			set_page_count(p, 0);
 		}
-		VM_WARN_ON_ONCE(PageReserved(p));
-		__ClearPageOffline(p);
-		set_page_count(p, 0);
 
 		/*
 		 * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 		debug_pagealloc_map_pages(page, nr_pages);
 		adjust_managed_page_count(page, nr_pages);
 	} else {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
 			__ClearPageReserved(p);
 			set_page_count(p, 0);
 		}
-		__ClearPageReserved(p);
-		set_page_count(p, 0);
 
 		/* memblock adjusts totalram_pages() manually. */
 		atomic_long_add(nr_pages, &page_zone(page)->managed_pages);
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 11 lines · On Tue, 2 Jul 2024 02:09:31 +0000 Wei Yang <[email protected]> wrote:

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

Quoted 8 lines

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I don't think I've ever seen prefetch make a damn bit of difference.

Quoted 8 lines

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

---
The patch is based on mm-stable with David's change.

Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.

author
#2 lore raw
Quoted 30 lines · Andrew Morton
Quoted 11 lines · On Tue, 2 Jul 2024 02:09:31 +0000 Wei Yang <[email protected]> wrote:

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

Quoted 8 lines

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I don't think I've ever seen prefetch make a damn bit of difference.

Quoted 6 lines

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

Quoted 6 lines
Quoted 2 lines

---
The patch is based on mm-stable with David's change.

Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.

The commit is 3dadec1babf9eee0c67c967df931d6f0cb124a04

mm: pass meminit_context to __free_pages_core()

The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/

BTW, how we track the mail link? Just search in lore.kernel.org?

Signature · 2 lines
Wei Yang
Help you, Help me
maintainer
#3 lore raw
Quoted 7 lines · On Wed, 3 Jul 2024 00:01:42 +0000 Wei Yang <[email protected]> wrote:
Quoted 7 lines
Quoted 6 lines

---
The patch is based on mm-stable with David's change.

Quoted 7 lines
Quoted 3 lines

Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.

The commit is 3dadec1babf9eee0c67c967df931d6f0cb124a04

mm: pass meminit_context to __free_pages_core()

OK, that's in mm-stable.

Quoted 4 lines

The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/

BTW, how we track the mail link? Just search in lore.kernel.org?

I'm not sure what you mean? I extract the message-id from the email and concat it with "https://lkml.kernel.org/r".

author
#4 lore raw
Quoted 19 lines · Andrew Morton
Quoted 7 lines · On Wed, 3 Jul 2024 00:01:42 +0000 Wei Yang <[email protected]> wrote:
Quoted 7 lines
Quoted 6 lines

---
The patch is based on mm-stable with David's change.

Quoted 7 lines
Quoted 3 lines

Oh help. David makes many changes. Please identify patches with much care. Fully quoting the email title works, as does a link.

The commit is 3dadec1babf9eee0c67c967df931d6f0cb124a04

mm: pass meminit_context to __free_pages_core()

OK, that's in mm-stable.

Quoted 7 lines
Quoted 4 lines

The link is, if I am correct.
https://lore.kernel.org/all/[email protected]/

BTW, how we track the mail link? Just search in lore.kernel.org?

I'm not sure what you mean? I extract the message-id from the email and concat it with "https://lkml.kernel.org/r".

Thanks, I got something new :-)

Signature · 2 lines
Wei Yang
Help you, Help me
Reviewed-by on v1 1/1
#5 lore raw
In reply to Wei Yang
Quoted 11 lines · Wei Yang

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

Quoted 8 lines

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I suspect this is noise, though.

Quoted 19 lines

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

---
The patch is based on mm-stable with David's change.

---
  mm/page_alloc.c | 13 ++-----------
  1 file changed, 2 insertions(+), 11 deletions(-)
mm/page_alloc.c +0 -0
index 116ee33fd1ce..c46aedfc9a12 100644
@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
  	 */

mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243

Quoted 10 lines
mm/page_alloc.c +1 -3
  	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
  	    unlikely(context == MEMINIT_HOTPLUG)) {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
  			VM_WARN_ON_ONCE(PageReserved(p));
  			__ClearPageOffline(p);
  			set_page_count(p, 0);
  		}

Something like:

for (;;) {
	...
	if (++loop >= nr_pages)
		break;
	p++;
}

Might generate slightly better code, because we know that we execute the loop body at least once. We use that in set_ptes(), for example.

Quoted 12 lines
mm/page_alloc.c +0 -5
-		VM_WARN_ON_ONCE(PageReserved(p));
-		__ClearPageOffline(p);
-		set_page_count(p, 0);
 
  		/*
  		 * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
  		debug_pagealloc_map_pages(page, nr_pages);
  		adjust_managed_page_count(page, nr_pages);
  	} else {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {

mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1259

Quoted 10 lines
mm/page_alloc.c +1 -3
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
  			__ClearPageReserved(p);
  			set_page_count(p, 0);
  		}
-		__ClearPageReserved(p);
-		set_page_count(p, 0);
 
  		/* memblock adjusts totalram_pages() manually. */
  		atomic_long_add(nr_pages, &page_zone(page)->managed_pages);

Much better

Signature · 3 lines
Cheers,

David / dhildenb
author
#6 lore raw
Quoted 55 lines · David Hildenbrand
Quoted 11 lines · Wei Yang

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

Quoted 8 lines

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I suspect this is noise, though.

Quoted 19 lines

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

---
The patch is based on mm-stable with David's change.

---
  mm/page_alloc.c | 13 ++-----------
  1 file changed, 2 insertions(+), 11 deletions(-)
mm/page_alloc.c +0 -0
index 116ee33fd1ce..c46aedfc9a12 100644
@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
  	 */

mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243

Quoted 10 lines
mm/page_alloc.c +1 -3
  	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
  	    unlikely(context == MEMINIT_HOTPLUG)) {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
  			VM_WARN_ON_ONCE(PageReserved(p));
  			__ClearPageOffline(p);
  			set_page_count(p, 0);
  		}

Something like:

Quoted 7 lines
for (;;) {
	...
	if (++loop >= nr_pages)
		break;
	p++;
}

So you prefer to have another version with this format? Sth like this?

mm/page_alloc.c +9 -3
index c46aedfc9a12..5235015eba3d 100644
@@ -1224,7 +1224,7 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 {
 	unsigned int nr_pages = 1 << order;
 	struct page *p = page;
-	unsigned int loop;
+	unsigned int loop = 0;
 
 	/*
 	 * When initializing the memmap, __init_single_page() sets the refcount
@@ -1236,10 +1236,13 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 	 */
 	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
 	    unlikely(context == MEMINIT_HOTPLUG)) {
-		for (loop = 0; loop < nr_pages; loop++, p++) {
+		for (;;) {
 			VM_WARN_ON_ONCE(PageReserved(p));
 			__ClearPageOffline(p);
 			set_page_count(p, 0);
+			if (++loop >= nr_pages)
+				break;
+			p++;
 		}
 
 		/*
@@ -1250,9 +1253,12 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
 		debug_pagealloc_map_pages(page, nr_pages);
 		adjust_managed_page_count(page, nr_pages);
 	} else {
-		for (loop = 0; loop < nr_pages; loop++, p++) {
+		for (;;) {
 			__ClearPageReserved(p);
 			set_page_count(p, 0);
+			if (++loop >= nr_pages)
+				break;
+			p++;
 		}
 
 		/* memblock adjusts totalram_pages() manually. */
Quoted 33 lines

Might generate slightly better code, because we know that we execute the loop body at least once. We use that in set_ptes(), for example.

Quoted 10 lines
mm/page_alloc.c +0 -4
-		VM_WARN_ON_ONCE(PageReserved(p));
-		__ClearPageOffline(p);
-		set_page_count(p, 0);
  		/*
  		 * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
  		debug_pagealloc_map_pages(page, nr_pages);
  		adjust_managed_page_count(page, nr_pages);
  	} else {
-		prefetchw(p);

mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1258

Quoted 10 lines
mm/page_alloc.c +1 -4
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
  			__ClearPageReserved(p);
  			set_page_count(p, 0);
  		}
-		__ClearPageReserved(p);
-		set_page_count(p, 0);
  		/* memblock adjusts totalram_pages() manually. */
  		atomic_long_add(nr_pages, &page_zone(page)->managed_pages);

Much better

Signature · 3 lines
Cheers,

David / dhildenb
Signature · 2 lines
Wei Yang
Help you, Help me
#7 lore raw
Quoted 64 lines · Wei Yang
Quoted 55 lines · David Hildenbrand
Quoted 11 lines · Wei Yang

The prefetchw() is introduced from an ancient patch[1].

The change log says:

The basic idea is to free higher order pages instead of going through every single one. Also, some unnecessary atomic operations are done away with and replaced with non-atomic equivalents, and prefetching is done where it helps the most. For a more in-depth discusion of this patch, please see the linux-ia64 archives (topic is "free bootmem feedback patch").

Quoted 8 lines

So there are several changes improve the bootmem freeing, in which the most basic idea is freeing higher order pages. And as Matthew says, "Itanium CPUs of this era had no prefetchers."

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I suspect this is noise, though.

Quoted 19 lines

[1]: https://lore.kernel.org/linux-ia64/[email protected]/

---
The patch is based on mm-stable with David's change.

---
   mm/page_alloc.c | 13 ++-----------
   1 file changed, 2 insertions(+), 11 deletions(-)
mm/page_alloc.c +0 -0
index 116ee33fd1ce..c46aedfc9a12 100644
@@ -1236,16 +1236,11 @@ void __meminit __free_pages_core(struct page *page, unsigned int order,
   	 */

mm/page_alloc.c void __meminit __free_pages_core(struct page *page, unsigned int order, · line 1243

Quoted 10 lines
mm/page_alloc.c +1 -3
   	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
   	    unlikely(context == MEMINIT_HOTPLUG)) {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
   			VM_WARN_ON_ONCE(PageReserved(p));
   			__ClearPageOffline(p);
   			set_page_count(p, 0);
   		}

Something like:

Quoted 7 lines
for (;;) {
	...
	if (++loop >= nr_pages)
		break;
	p++;
}
Quoted 1 line

So you prefer to have another version with this format? Sth like this?

Whatever you prefer, just pointing it out that we can do slightly better :)

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
In reply to David Hildenbrand
Quoted 6 lines · David Hildenbrand
Quoted 4 lines

I did 10 round bootup tests before and after this change, the data doesn't prove prefetchw() help speeding up bootmem freeing. The sum of the 10 round bootmem freeing time after prefetchw() removal even 5.2% faster than before.

I suspect this is noise, though.

I think it's real, though small. Each prefetchw() is an instruction, and if we can avoid issuing an instruction, we should.

Quoted 10 lines

Something like:

for (;;) {
	...
	if (++loop >= nr_pages)
		break;
	p++;
}
Quoted 2 lines

Might generate slightly better code, because we know that we execute the loop body at least once. We use that in set_ptes(), for example.

I don't think it's worth doing. Keep the loop simple and obvious. set_ptes() is different because we actually expect to execute the loop exactly once (ie most folios are small). So two compares per call to set_ptes() instead of one makes a difference. Here, we're expecting to execute this loop, what, a million times? Doing a million-and-one compares instead of a million makes no observable difference.

I would like to see v2 of this patch dropped, please Andrew.

maintainer
#9 lore raw
Quoted 13 lines · On Thu, 4 Jul 2024 04:30:50 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 10 lines

Something like:

for (;;) {
	...
	if (++loop >= nr_pages)
		break;
	p++;
}
Quoted 2 lines

Might generate slightly better code, because we know that we execute the loop body at least once. We use that in set_ptes(), for example.

Quoted 8 lines

I don't think it's worth doing. Keep the loop simple and obvious. set_ptes() is different because we actually expect to execute the loop exactly once (ie most folios are small). So two compares per call to set_ptes() instead of one makes a difference. Here, we're expecting to execute this loop, what, a million times? Doing a million-and-one compares instead of a million makes no observable difference.

I would like to see v2 of this patch dropped, please Andrew.

thud.

Are you supportive of v1?

mm/page_alloc.c renamed from mm/page_alloc.c~mm-page_alloc-remove-prefetchw-on-freeing-page-to-buddy-system +2 -11
@@ -1236,16 +1236,11 @@ void __free_pages_core(struct page *page
 	 */
 	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG) &&
 	    unlikely(context == MEMINIT_HOTPLUG)) {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
 			VM_WARN_ON_ONCE(PageReserved(p));
 			__ClearPageOffline(p);
 			set_page_count(p, 0);
 		}
-		VM_WARN_ON_ONCE(PageReserved(p));
-		__ClearPageOffline(p);
-		set_page_count(p, 0);
 
 		/*
 		 * Freeing the page with debug_pagealloc enabled will try to
@@ -1255,14 +1250,10 @@ void __free_pages_core(struct page *page
 		debug_pagealloc_map_pages(page, nr_pages);
 		adjust_managed_page_count(page, nr_pages);
 	} else {
-		prefetchw(p);
-		for (loop = 0; loop < (nr_pages - 1); loop++, p++) {
-			prefetchw(p + 1);
+		for (loop = 0; loop < nr_pages; loop++, p++) {
 			__ClearPageReserved(p);
 			set_page_count(p, 0);
 		}
-		__ClearPageReserved(p);
-		set_page_count(p, 0);
 
 		/* memblock adjusts totalram_pages() manually. */
 		atomic_long_add(nr_pages, &page_zone(page)->managed_pages);

_

#10 lore raw
Quoted 1 line · Andrew Morton

Are you supportive of v1?

Oh yes. I think it has my suggested-by on it, so I didn't see the need to add an R-b.

maintainer
#11 lore raw
Quoted 5 lines · On Thu, 4 Jul 2024 04:39:42 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 1 line · Andrew Morton

Are you supportive of v1?

Oh yes. I think it has my suggested-by on it, so I didn't see the need to add an R-b.

I'm a sucker for r-bys. Added, thanks.