Skip to contentSkip to conversation

mm/page_alloc: don't corrupt pcppage_migratetype

PATCH Applied Doug Berger 3 participants 5 messages Aug 11, 2021 – Aug 19, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/page_alloc: don't corrupt pcppage_migratetype

When placing pages on a pcp list, migratetype values over MIGRATE_PCPTYPES get added to the MIGRATE_MOVABLE pcp list.

However, the actual migratetype is preserved in the page and should not be changed to MIGRATE_MOVABLE or the page may end up on the wrong free_list.

Fixes df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock")
---
 mm/page_alloc.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
mm/page_alloc.c +12 -13
index 73704e836649..8addb4919f75 100644
@@ -3436,19 +3436,10 @@ void free_unref_page_list(struct list_head *list)
 		 * comment in free_unref_page.
 		 */
 		migratetype = get_pcppage_migratetype(page);
-		if (unlikely(migratetype >= MIGRATE_PCPTYPES)) {
-			if (unlikely(is_migrate_isolate(migratetype))) {
-				list_del(&page->lru);
-				free_one_page(page_zone(page), page, pfn, 0,
-							migratetype, FPI_NONE);
-				continue;
-			}
-
-			/*
-			 * Non-isolated types over MIGRATE_PCPTYPES get added
-			 * to the MIGRATE_MOVABLE pcp list.
-			 */
-			set_pcppage_migratetype(page, MIGRATE_MOVABLE);
+		if (unlikely(is_migrate_isolate(migratetype))) {
+			list_del(&page->lru);
+			free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE);
+			continue;
 		}
 
 		set_page_private(page, pfn);
@@ -3458,7 +3449,15 @@ void free_unref_page_list(struct list_head *list)
 	list_for_each_entry_safe(page, next, list, lru) {
 		pfn = page_private(page);
 		set_page_private(page, 0);
+
+		/*
+		 * Non-isolated types over MIGRATE_PCPTYPES get added
+		 * to the MIGRATE_MOVABLE pcp list.
+		 */
 		migratetype = get_pcppage_migratetype(page);
+		if (unlikely(migratetype >= MIGRATE_PCPTYPES))
+			migratetype = MIGRATE_MOVABLE;
+
 		trace_mm_page_free_batched(page);
 		free_unref_page_commit(page, pfn, migratetype, 0);
 
Signature · 1 line
2.25.1
Acked-by on v1 1/1
#1 lore raw
Quoted 6 lines · Doug Berger

When placing pages on a pcp list, migratetype values over MIGRATE_PCPTYPES get added to the MIGRATE_MOVABLE pcp list.

However, the actual migratetype is preserved in the page and should not be changed to MIGRATE_MOVABLE or the page may end up on the wrong free_list.

Nice, how did you find out? Were there any user-visible effects? (Hint: which? the changelog should say that so that the severity of the bug can be judged). Otherwise I agree the bug is there and patch is fixing it. Thanks.

Quoted 2 lines
Fixes df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock")
Quoted 49 lines
---
 mm/page_alloc.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
mm/page_alloc.c +12 -13
index 73704e836649..8addb4919f75 100644
@@ -3436,19 +3436,10 @@ void free_unref_page_list(struct list_head *list)
 		 * comment in free_unref_page.
 		 */
 		migratetype = get_pcppage_migratetype(page);
-		if (unlikely(migratetype >= MIGRATE_PCPTYPES)) {
-			if (unlikely(is_migrate_isolate(migratetype))) {
-				list_del(&page->lru);
-				free_one_page(page_zone(page), page, pfn, 0,
-							migratetype, FPI_NONE);
-				continue;
-			}
-
-			/*
-			 * Non-isolated types over MIGRATE_PCPTYPES get added
-			 * to the MIGRATE_MOVABLE pcp list.
-			 */
-			set_pcppage_migratetype(page, MIGRATE_MOVABLE);
+		if (unlikely(is_migrate_isolate(migratetype))) {
+			list_del(&page->lru);
+			free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE);
+			continue;
 		}
 
 		set_page_private(page, pfn);
@@ -3458,7 +3449,15 @@ void free_unref_page_list(struct list_head *list)
 	list_for_each_entry_safe(page, next, list, lru) {
 		pfn = page_private(page);
 		set_page_private(page, 0);
+
+		/*
+		 * Non-isolated types over MIGRATE_PCPTYPES get added
+		 * to the MIGRATE_MOVABLE pcp list.
+		 */
 		migratetype = get_pcppage_migratetype(page);
+		if (unlikely(migratetype >= MIGRATE_PCPTYPES))
+			migratetype = MIGRATE_MOVABLE;
+
 		trace_mm_page_free_batched(page);
 		free_unref_page_commit(page, pfn, migratetype, 0);
 
#2 lore raw
Quoted 11 lines · Vlastimil Babka
Quoted 6 lines · Doug Berger

When placing pages on a pcp list, migratetype values over MIGRATE_PCPTYPES get added to the MIGRATE_MOVABLE pcp list.

However, the actual migratetype is preserved in the page and should not be changed to MIGRATE_MOVABLE or the page may end up on the wrong free_list.

Nice, how did you find out? Were there any user-visible effects? (Hint: which? the changelog should say that so that the severity of the bug can be judged). Otherwise I agree the bug is there and patch is fixing it. Thanks.

I did not observe the bug "in the wild", but rather noticed it while reviewing the current CMA implementation. I would imagine in the worst case that CMA memory could leak onto MIGRATE_MOVABLE free_lists and get "stolen" through fallback for MIGRATE_UNMOVABLE allocations effectively breaking CMA.

For full disclosure I should take this opportunity to say that I am investigating ways to improve the CMA/Movable memory implementations somewhat in line with what Joonsoo Kim has been trying, which is why I observed this bug.

However, I largely agree with the proposal Mel Gorman made in his comments on:
https://lkml.org/lkml/2016/4/28/244

I can appreciate that zones may provide a convenient way of grouping memory with common attributes, but it is a bastardization of the original intention of zones. I would prefer a mechanism for designating blocks of memory as "sticky" movable such that they can only be used to satisfy movable memory requests and not be converted to other migratetypes. I've wrestled with the terminology and currently favor Designated Movable Blocks (as opposed to "sticky") because these are blocks of memory that have the movable attribute by designation rather than by default or accident.

While CMA blocks should be a subset of Designated Movable Blocks since they have the same migration properties, it may be desirable/necessary to maintain a distinction to accommodate competing allocation objectives. Specifically, Designated Movable Blocks used by CMA to satisfy the needs of drivers will likely want the page allocator to be less aggressive than users of Memory Hot-plug or kernelcore= Designated Movable Blocks. Perhaps a round-robin scheme with MIGRATE_MOVABLE can satisfy all users, but I'm not too optimistic.

I don't know if Mel has changed his opinions on this matter, but any suggestions and encouragement are appreciated.

Thanks,
Doug

Quoted 56 lines
Quoted 3 lines
Fixes df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock")
Quoted 49 lines
---
 mm/page_alloc.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
mm/page_alloc.c +12 -13
index 73704e836649..8addb4919f75 100644
@@ -3436,19 +3436,10 @@ void free_unref_page_list(struct list_head *list)
 		 * comment in free_unref_page.
 		 */
 		migratetype = get_pcppage_migratetype(page);
-		if (unlikely(migratetype >= MIGRATE_PCPTYPES)) {
-			if (unlikely(is_migrate_isolate(migratetype))) {
-				list_del(&page->lru);
-				free_one_page(page_zone(page), page, pfn, 0,
-							migratetype, FPI_NONE);
-				continue;
-			}
-
-			/*
-			 * Non-isolated types over MIGRATE_PCPTYPES get added
-			 * to the MIGRATE_MOVABLE pcp list.
-			 */
-			set_pcppage_migratetype(page, MIGRATE_MOVABLE);
+		if (unlikely(is_migrate_isolate(migratetype))) {
+			list_del(&page->lru);
+			free_one_page(page_zone(page), page, pfn, 0, migratetype, FPI_NONE);
+			continue;
 		}
 
 		set_page_private(page, pfn);
@@ -3458,7 +3449,15 @@ void free_unref_page_list(struct list_head *list)
 	list_for_each_entry_safe(page, next, list, lru) {
 		pfn = page_private(page);
 		set_page_private(page, 0);
+
+		/*
+		 * Non-isolated types over MIGRATE_PCPTYPES get added
+		 * to the MIGRATE_MOVABLE pcp list.
+		 */
 		migratetype = get_pcppage_migratetype(page);
+		if (unlikely(migratetype >= MIGRATE_PCPTYPES))
+			migratetype = MIGRATE_MOVABLE;
+
 		trace_mm_page_free_batched(page);
 		free_unref_page_commit(page, pfn, migratetype, 0);
 
#3 lore raw
Quoted 13 lines · Doug Berger
Quoted 11 lines · Vlastimil Babka
Quoted 6 lines · Doug Berger

When placing pages on a pcp list, migratetype values over MIGRATE_PCPTYPES get added to the MIGRATE_MOVABLE pcp list.

However, the actual migratetype is preserved in the page and should not be changed to MIGRATE_MOVABLE or the page may end up on the wrong free_list.

Nice, how did you find out? Were there any user-visible effects? (Hint: which? the changelog should say that so that the severity of the bug can be judged). Otherwise I agree the bug is there and patch is fixing it. Thanks.

Quoted 5 lines

I did not observe the bug "in the wild", but rather noticed it while reviewing the current CMA implementation. I would imagine in the worst case that CMA memory could leak onto MIGRATE_MOVABLE free_lists and get "stolen" through fallback for MIGRATE_UNMOVABLE allocations effectively breaking CMA.

Right, thanks.

Quoted 19 lines

For full disclosure I should take this opportunity to say that I am investigating ways to improve the CMA/Movable memory implementations somewhat in line with what Joonsoo Kim has been trying, which is why I observed this bug.

However, I largely agree with the proposal Mel Gorman made in his comments on:
https://lkml.org/lkml/2016/4/28/244

I can appreciate that zones may provide a convenient way of grouping memory with common attributes, but it is a bastardization of the original intention of zones. I would prefer a mechanism for designating blocks of memory as "sticky" movable such that they can only be used to satisfy movable memory requests and not be converted to other migratetypes. I've wrestled with the terminology and currently favor Designated Movable Blocks (as opposed to "sticky") because these are blocks of memory that have the movable attribute by designation rather than by default or accident.

Quoted 8 lines

While CMA blocks should be a subset of Designated Movable Blocks since they have the same migration properties, it may be desirable/necessary to maintain a distinction to accommodate competing allocation objectives. Specifically, Designated Movable Blocks used by CMA to satisfy the needs of drivers will likely want the page allocator to be less aggressive than users of Memory Hot-plug or kernelcore= Designated Movable Blocks. Perhaps a round-robin scheme with MIGRATE_MOVABLE can satisfy all users, but I'm not too optimistic.

I was, on the other hand, in favor Joonsoo's zone based solution, as there's always problems with these designated pageblocks and watermarks. It's also described in the thread you linked above. See __zone_watermark_unusable_free() how it has to subtract NR_FREE_CMA_PAGES for non-movable allocations. So you need to track NR_FREE_CMA_PAGES which is rather fragile, and then reclaim target depends on whether your goal is movable or unmovable allocation. Either it's movable and you might end up in free memory only in those designated-movable (today, MIGRATE_CMA) blocks, and unmovable allocation has to direct reclaim and hope the reclaimed LRU pages will be in the "normal" pageblocks, or you make sure that there's enough free pages in normal pageblocks, and end up with lower memory utilization. A separate zone solves this problem.

Quoted 5 lines

I don't know if Mel has changed his opinions on this matter, but any suggestions and encouragement are appreciated.

Thanks,
Doug

Acked-by on v1 1/1
#4 lore raw
In reply to Doug Berger
Quoted 9 lines · Doug Berger

When placing pages on a pcp list, migratetype values over MIGRATE_PCPTYPES get added to the MIGRATE_MOVABLE pcp list.

However, the actual migratetype is preserved in the page and should not be changed to MIGRATE_MOVABLE or the page may end up on the wrong free_list.

Fixes df1acc856923 ("mm/page_alloc: avoid conflating IRQs disabled with zone->lock")

Oops, yes. The impact is that HIGHATOMIC or CMA pages getting bulk freed from the PCP lists could potentially end up on the wrong buddy list. There are various consequences but minimally NR_FREE_CMA_PAGES accounting could get screwed up.

Thanks Doug

Signature · 2 lines
Mel Gorman
SUSE Labs