Skip to contentSkip to conversation

mm-disable-lru-pagevec-during-the-migration-temporarily.patch removed from -mm tree

merged [email protected] 1 participant 1 message May 8, 2021

lore
Share & tools
lore

[email protected]

author tree notice Applied
#0 lore raw

The patch titled
Subject: mm: disable LRU pagevec during the migration temporarily
has been removed from the -mm tree. Its filename was
mm-disable-lru-pagevec-during-the-migration-temporarily.patch

This patch was dropped because it was merged into mainline or a subsystem tree

------------------------------------------------------
From: Minchan Kim <[email protected]>
Subject: mm: disable LRU pagevec during the migration temporarily

LRU pagevec holds refcount of pages until the pagevec are drained. It could prevent migration since the refcount of the page is greater than the expection in migration logic. To mitigate the issue, callers of migrate_pages drains LRU pagevec via migrate_prep or lru_add_drain_all before migrate_pages call.

However, it's not enough because pages coming into pagevec after the draining call still could stay at the pagevec so it could keep preventing page migration. Since some callers of migrate_pages have retrial logic with LRU draining, the page would migrate at next trail but it is still fragile in that it doesn't close the fundamental race between upcoming LRU pages into pagvec and migration so the migration failure could cause contiguous memory allocation failure in the end.

To close the race, this patch disables lru caches(i.e, pagevec) during ongoing migration until migrate is done.

Since it's really hard to reproduce, I measured how many times migrate_pages retried with force mode(it is about a fallback to a sync migration) with below debug code.

int migrate_pages(struct list_head *from, new_page_t get_new_page,

			..
			..
if (rc && reason == MR_CONTIG_RANGE && pass > 2) {
       printk(KERN_ERR, "pfn 0x%lx reason %d
", page_to_pfn(page), rc);
       dump_page(page, "fail to migrate");
}

The test was repeating android apps launching with cma allocation in background every five seconds. Total cma allocation count was about 500 during the testing. With this patch, the dump_page count was reduced from 400 to 30.

The new interface is also useful for memory hotplug which currently drains lru pcp caches after each migration failure. This is rather suboptimal as it has to disrupt others running during the operation. With the new interface the operation happens only once. This is also in line with pcp allocator cache which are disabled for the offlining as well.

---

 include/linux/migrate.h |    2 +
 include/linux/swap.h    |   14 ++++++++
 mm/memory_hotplug.c     |    3 +
 mm/mempolicy.c          |    4 ++
 mm/migrate.c            |   11 ++++--
 mm/page_alloc.c         |    2 +
 mm/swap.c               |   64 ++++++++++++++++++++++++++++++++------
 7 files changed, 86 insertions(+), 14 deletions(-)
include/linux/migrate.h renamed from include/linux/migrate.h~mm-disable-lru-pagevec-during-the-migration-temporarily +2 -0
@@ -46,6 +46,7 @@ extern int isolate_movable_page(struct p
 extern void putback_movable_page(struct page *page);
 
 extern void migrate_prep(void);
+extern void migrate_finish(void);
 extern void migrate_prep_local(void);
 extern void migrate_page_states(struct page *newpage, struct page *page);
 extern void migrate_page_copy(struct page *newpage, struct page *page);
@@ -67,6 +68,7 @@ static inline int isolate_movable_page(s
 	{ return -EBUSY; }
 
 static inline int migrate_prep(void) { return -ENOSYS; }
+static inline int migrate_finish(void) { return -ENOSYS; }
 static inline int migrate_prep_local(void) { return -ENOSYS; }
 
 static inline void migrate_page_states(struct page *newpage, struct page *page)
include/linux/swap.h renamed from include/linux/swap.h~mm-disable-lru-pagevec-during-the-migration-temporarily +14 -0
@@ -340,6 +340,20 @@ extern void lru_note_cost(struct lruvec
 extern void lru_note_cost_page(struct page *);
 extern void lru_cache_add(struct page *);
 extern void mark_page_accessed(struct page *);
+
+extern atomic_t lru_disable_count;
+
+static inline bool lru_cache_disabled(void)
+{
+	return atomic_read(&lru_disable_count);
+}
+
+static inline void lru_cache_enable(void)
+{
+	atomic_dec(&lru_disable_count);
+}
+
+extern void lru_cache_disable(void);
 extern void lru_add_drain(void);
 extern void lru_add_drain_cpu(int cpu);
 extern void lru_add_drain_cpu_zone(struct zone *zone);
mm/memory_hotplug.c renamed from mm/memory_hotplug.c~mm-disable-lru-pagevec-during-the-migration-temporarily +2 -1
@@ -1611,6 +1611,7 @@ int __ref offline_pages(unsigned long st
 	 * in a way that pages from isolated pageblock are left on pcplists.
 	 */
 	zone_pcp_disable(zone);
+	lru_cache_disable();
 
 	/* set above range as isolated */
 	ret = start_isolate_page_range(start_pfn, end_pfn,
@@ -1642,7 +1643,6 @@ int __ref offline_pages(unsigned long st
 			}
 
 			cond_resched();
-			lru_add_drain_all();
 
 			ret = scan_movable_pages(pfn, end_pfn, &pfn);
 			if (!ret) {
@@ -1687,6 +1687,7 @@ int __ref offline_pages(unsigned long st
 	zone->nr_isolate_pageblock -= nr_pages / pageblock_nr_pages;
 	spin_unlock_irqrestore(&zone->lock, flags);
 
+	lru_cache_enable();
 	zone_pcp_enable(zone);
 
 	/* removal success */
mm/mempolicy.c renamed from mm/mempolicy.c~mm-disable-lru-pagevec-during-the-migration-temporarily +4 -0
@@ -1208,6 +1208,8 @@ int do_migrate_pages(struct mm_struct *m
 			break;
 	}
 	mmap_read_unlock(mm);
+
+	migrate_finish();
 	if (err < 0)
 		return err;
 	return busy;
@@ -1371,6 +1373,8 @@ up_out:
 	mmap_write_unlock(mm);
 mpol_out:
 	mpol_put(new);
+	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
+		migrate_finish();
 	return err;
 }
 
mm/migrate.c renamed from mm/migrate.c~mm-disable-lru-pagevec-during-the-migration-temporarily +7 -4
@@ -66,11 +66,13 @@ void migrate_prep(void)
 {
 	/*
 	 * Clear the LRU lists so pages can be isolated.
-	 * Note that pages may be moved off the LRU after we have
-	 * drained them. Those pages will fail to migrate like other
-	 * pages that may be busy.
 	 */
-	lru_add_drain_all();
+	lru_cache_disable();
+}
+
+void migrate_finish(void)
+{
+	lru_cache_enable();
 }
 
 /* Do the necessary work of migrate_prep but not if it involves other CPUs */
@@ -1838,6 +1840,7 @@ out_flush:
 	if (err >= 0)
 		err = err1;
 out:
+	migrate_finish();
 	return err;
 }
 
mm/page_alloc.c renamed from mm/page_alloc.c~mm-disable-lru-pagevec-during-the-migration-temporarily +2 -0
@@ -8715,6 +8715,8 @@ static int __alloc_contig_migrate_range(
 		if (ret == -ENOMEM)
 			break;
 	}
+
+	migrate_finish();
 	if (ret < 0) {
 		alloc_contig_dump_pages(&cc->migratepages);
 		putback_movable_pages(&cc->migratepages);
mm/swap.c renamed from mm/swap.c~mm-disable-lru-pagevec-during-the-migration-temporarily +55 -9

136 lines not on this page · raw message

_

Patches currently in -mm which might be from [email protected] are