mmotm - unevictable lru/mlocked pages fixes/cleanup

Here are a set of fixes and cleanup to the unevictable lru and mlocked pages patches against 27-rc1+mmotm-080730-0356. In addition to the patches, I have a few suggestions for fixing up a few patch descriptions, and other potential cleanups, below.

The following patch names are the file names I'm using in my tree. The names indicate where in the series the patch goes. I've also mentioned in the patch descriptions which patch each one fixes.

unevictable-lru-infrastructure-putback-fix1.patch

I noticed that putback_lru_page() would forget the active state of a page in the event of a "redo". [Note: I added a vm event to record any redos. I've never seen any redos in any of my stress testing since Kamezawa-san reworked putback_lru_pages().] This patch remembers the page's incoming PageActive state. Not sure this is required.

Can be folded with unevictable-lru-infrastructure.patch

unevictable-lru-infrastructure-putback-fix2.patch

Remove the unevictable page vm event counting from putback_lru_page(), as this breaks a bisect. The necessary vmstat events have not been defined until later in the series. Subsequent patches will add back in the event counting at appropriate places

Can be folded with unevictable-lru-infrastructure.patch

unevictable-lru-page-statistics-add-events.patch

Define the vmstat events for the unevictable lru infrastructure and count them here, separately from the mlocked pages stats.

Renames the NORECL/noreclaim vmstat events to UNEVICTABLE/unevictable ...

This patch may be folded with the
unevictable-lru-page-statistics.patch

shm_locked-pages-are-unevictable-add-vm-events.patch

Add unevictable lru event counting to scan of the unevictable lru list[s] for a mapping's pages. May be folded with shm_locked-pages-are-unevictable.patch

vmstat-mlocked-pages-statistics-add-vm-events.patch

Add mlocked pages related vm event definitions and counting. May be folded with vmstat-mlocked-pages-statistics.patch

mlock-count-attempts-to-free-mlocked-page.patch

REPLACE the existing patch of this name with this one. It resolves patch conflicts resulting from the renaming of the events and handles the renaming in unconflicted hunks.

vmstat-unevictable-and-mlocked-pages-vm-events.patch

REMOVE this patch from the series. It is superceeded by the preceeding patches that define the vm events with the appropriate statistics patch.

Other notes and patch description updates:

1) Kosaki-san's munlock rework patch [removal of page table walker] has been folded with:
mmap-handle-mlocked-pages-during-map-remap-unmap.patch
Technically, it should be part of:
mlock-mlocked-pages-are-unevictable.patch
where it would have just removed/replaced the patch chunks that added the page table walker.

2) Patch description of ramfs-and-ram-disk-pages-are-unevictable.patch is stale, since the brd driver replaced rd--ram disk pages are no longer on the lru. I'll send a suggested replacement description as part of this series.

3) Since the putback_lru_page() rework, the patch description of shm_locked-pages-are-unevictable.patch is stale. Suggest:

Remove the text from the last paragraph of the description, starting with:
"Note that scan_mapping_unevictable_page() must be able to sleep..."

4) Kosaki-san's munlock rework obsolete's note 5 in the description for mlock-mlocked-pages-are-unevictable.patch.

Could just make a note to this effect with the folded "introduce __get_user_pages()" patch description.

5) The putback_lru_page() and munlock() rework requires update of the unevictable-lru.txt in Documentation. I'll send an update patch [later].

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Fix to unevictable-lru-infrastructure.patch

Not sure this is absolutely required, but:

Preserve page's incoming active flag across retries in putback_lru_page().

 mm/vmscan.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmscan.c
===================================================================
@@ -483,12 +483,12 @@ int remove_mapping(struct address_space 
 void putback_lru_page(struct page *page)
 {
 	int lru;
+	int active = !!TestClearPageActive(page);
 	int was_unevictable = PageUnevictable(page);
 
 	VM_BUG_ON(PageLRU(page));
 
 redo:
-	lru = !!TestClearPageActive(page);
 	ClearPageUnevictable(page);
 
 	if (page_evictable(page, NULL)) {

@@ -498,7 +498,7 @@ redo:
 		 * unevictable page on [in]active list.
 		 * We know how to handle that.
 		 */
-		lru += page_is_file_cache(page);
+		lru = active + page_is_file_cache(page);
 		lru_cache_add_lru(page, lru);
 	} else {
 		/*

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Fix to unevictable-lru-infrastructure.patch

NORECL_* events are not defined this early in the series. Remove the event counting from this patch and add in with unevictable lru statistics [subsequent patch].

 mm/vmscan.c |    6 ------
 1 file changed, 6 deletions(-)

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmscan.c
===================================================================
@@ -484,7 +484,6 @@ void putback_lru_page(struct page *page)
 {
 	int lru;
 	int active = !!TestClearPageActive(page);
-	int was_unevictable = PageUnevictable(page);
 
 	VM_BUG_ON(PageLRU(page));
 

@@ -526,11 +525,6 @@ redo:
 		 */
 	}
 
-	if (was_unevictable && lru != LRU_UNEVICTABLE)
-		count_vm_event(NORECL_PGRESCUED);
-	else if (!was_unevictable && lru == LRU_UNEVICTABLE)
-		count_vm_event(NORECL_PGCULLED);
-
 	put_page(page);		/* drop ref from isolate */
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

2008/7/31 Lee Schermerhorn <[email protected]>:

Oops, sorry my bad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Fix to unevictable-lru-page-statistics.patch

Add unevictable lru infrastructure vm events to the statistics patch. Rename the "NORECL_" and "noreclaim_" symbols and text strings to "UNEVICTABLE_" and "unevictable_", respectively.

Currently, both the infrastructure and the mlocked pages event are added by a single patch later in the series. This makes it difficult to add or rework the incremental patches. The events actually "belong" with the stats, so pull them up to here.

Also, restore the event counting to putback_lru_page(). This was removed from previous patch in series where it was "misplaced". The actual events weren't defined that early.

 include/linux/vmstat.h |    5 +++++
 mm/vmscan.c            |    6 ++++++
 mm/vmstat.c            |    5 +++++
 3 files changed, 16 insertions(+)

Index: linux-2.6.27-rc1-mmotm-30jul/include/linux/vmstat.h
===================================================================
@@ -41,6 +41,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
 #ifdef CONFIG_HUGETLB_PAGE
 		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
 #endif
+#ifdef CONFIG_UNEVICTABLE_LRU
+		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
+		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
+		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
+#endif
 		NR_VM_EVENT_ITEMS
 };
 

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmscan.c
===================================================================
@@ -484,6 +484,7 @@ void putback_lru_page(struct page *page)
 {
 	int lru;
 	int active = !!TestClearPageActive(page);
+	int was_unevictable = PageUnevictable(page);
 
 	VM_BUG_ON(PageLRU(page));
 

@@ -525,6 +526,11 @@ redo:
 		 */
 	}
 
+	if (was_unevictable && lru != LRU_UNEVICTABLE)
+		count_vm_event(UNEVICTABLE_PGRESCUED);
+	else if (!was_unevictable && lru == LRU_UNEVICTABLE)
+		count_vm_event(UNEVICTABLE_PGCULLED);
+
 	put_page(page);		/* drop ref from isolate */
 }
 

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmstat.c
===================================================================
@@ -663,6 +663,11 @@ static const char * const vmstat_text[] 
 	"htlb_buddy_alloc_success",
 	"htlb_buddy_alloc_fail",
 #endif
+#ifdef CONFIG_UNEVICTABLE_LRU
+	"unevictable_pgs_culled",
+	"unevictable_pgs_scanned",
+	"unevictable_pgs_rescued",
+#endif
 #endif
 };
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

okey.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Fix to shm_locked-pages-are-unevictable.patch

More breaking up of the single unevictable lru/mlocked pages vm events patch.

Add the vm events counting to the unevictable lru list scanning for shm_unlock().

 mm/vmscan.c |    5 +++++
 1 file changed, 5 insertions(+)

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmscan.c
===================================================================
@@ -2407,6 +2407,7 @@ retry:
 		__dec_zone_state(zone, NR_UNEVICTABLE);
 		list_move(&page->lru, &zone->lru[l].list);
 		__inc_zone_state(zone, NR_INACTIVE_ANON + l);
+		__count_vm_event(UNEVICTABLE_PGRESCUED);
 	} else {
 		/*
 		 * rotate unevictable list

@@ -2440,6 +2441,7 @@ void scan_mapping_unevictable_pages(stru
 	while (next < end &&
 		pagevec_lookup(&pvec, mapping, next, PAGEVEC_SIZE)) {
 		int i;
+		int pg_scanned = 0;
 
 		zone = NULL;
 

@@ -2448,6 +2450,7 @@ void scan_mapping_unevictable_pages(stru
 			pgoff_t page_index = page->index;
 			struct zone *pagezone = page_zone(page);
 
+			pg_scanned++;
 			if (page_index > next)
 				next = page_index;
 			next++;

@@ -2465,6 +2468,8 @@ void scan_mapping_unevictable_pages(stru
 		if (zone)
 			spin_unlock_irq(&zone->lru_lock);
 		pagevec_release(&pvec);
+
+		count_vm_events(UNEVICTABLE_PGSCANNED, pg_scanned);
 	}
 
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Fix to vmstat-mlocked-pages-statistics.patch

Define mlocked pages vm events in the mlocked pages statistics patch. Makes for easier incremental patching.

 include/linux/vmstat.h |    4 ++++
 mm/internal.h          |    4 +++-
 mm/mlock.c             |   33 +++++++++++++++++++++++++--------
 mm/vmstat.c            |    4 ++++
 4 files changed, 36 insertions(+), 9 deletions(-)

Index: linux-2.6.27-rc1-mmotm-30jul/include/linux/vmstat.h
===================================================================
@@ -45,6 +45,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
 		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
 		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
 		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
+		UNEVICTABLE_PGMLOCKED,
+		UNEVICTABLE_PGMUNLOCKED,
+		UNEVICTABLE_PGCLEARED,	/* on COW, page truncate */
+		UNEVICTABLE_PGSTRANDED,	/* unable to isolate on unlock */
 #endif
 		NR_VM_EVENT_ITEMS
 };

Index: linux-2.6.27-rc1-mmotm-30jul/mm/mlock.c
===================================================================
@@ -61,6 +61,7 @@ void __clear_page_mlock(struct page *pag
 	}
 
 	dec_zone_page_state(page, NR_MLOCK);
+	count_vm_event(UNEVICTABLE_PGCLEARED);
 	if (!isolate_lru_page(page)) {
 		putback_lru_page(page);
 	} else {

@@ -70,6 +71,9 @@ void __clear_page_mlock(struct page *pag
 		lru_add_drain_all();
 		if (!isolate_lru_page(page))
 			putback_lru_page(page);
+		else if (PageUnevictable(page))
+			count_vm_event(UNEVICTABLE_PGSTRANDED);
+
 	}
 }
 

@@ -83,6 +87,7 @@ void mlock_vma_page(struct page *page)
 
 	if (!TestSetPageMlocked(page)) {
 		inc_zone_page_state(page, NR_MLOCK);
+		count_vm_event(UNEVICTABLE_PGMLOCKED);
 		if (!isolate_lru_page(page))
 			putback_lru_page(page);
 	}

@@ -113,16 +118,28 @@ static void munlock_vma_page(struct page
 	if (TestClearPageMlocked(page)) {
 		dec_zone_page_state(page, NR_MLOCK);
 		if (!isolate_lru_page(page)) {
-			try_to_munlock(page);	/* maybe relock the page */
+			int ret = try_to_munlock(page);
+			/*
+			 * did try_to_unlock() succeed or punt?
+			 */
+			if (ret == SWAP_SUCCESS || ret == SWAP_AGAIN)
+				count_vm_event(UNEVICTABLE_PGMUNLOCKED);
+
 			putback_lru_page(page);
+		} else {
+			/*
+			 * We lost the race.  let try_to_unmap() deal
+			 * with it.  At least we get the page state and
+			 * mlock stats right.  However, page is still on
+			 * the noreclaim list.  We'll fix that up when
+			 * the page is eventually freed or we scan the
+			 * noreclaim list.
+			 */
+			if (PageUnevictable(page))
+				count_vm_event(UNEVICTABLE_PGSTRANDED);
+			else
+				count_vm_event(UNEVICTABLE_PGMUNLOCKED);
 		}
-		/*
-		 * Else we lost the race.  let try_to_unmap() deal with it.
-		 * At least we get the page state and mlock stats right.
-		 * However, page is still on the noreclaim list.  We'll fix
-		 * that up when the page is eventually freed or we scan the
-		 * noreclaim list.
-		 */
 	}
 }
 

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmstat.c
===================================================================
@@ -668,6 +668,10 @@ static const char * const vmstat_text[] 
 	"unevictable_pgs_culled",
 	"unevictable_pgs_scanned",
 	"unevictable_pgs_rescued",
+	"unevictable_pgs_mlocked",
+	"unevictable_pgs_munlocked",
+	"unevictable_pgs_cleared",
+	"unevictable_pgs_stranded",
 #endif
 #endif
 };

Index: linux-2.6.27-rc1-mmotm-30jul/mm/internal.h
===================================================================
@@ -101,8 +101,10 @@ static inline int is_mlocked_vma(struct 
 	if (likely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) != VM_LOCKED))
 		return 0;
 
-	if (!TestSetPageMlocked(page))
+	if (!TestSetPageMlocked(page)) {
 		inc_zone_page_state(page, NR_MLOCK);
+		count_vm_event(UNEVICTABLE_PGMLOCKED);
+	}
 	return 1;
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yeah. I missed that one. Time for a global search, I guess.

Thanks,
Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Against: [27-rc1+]mmotm-080730-0356

Replaces: mlock-count-attempts-to-free-mlocked-page.patch in the mmotm series.

Reworked to resolve patch conflicts introduced by other patches, including rename of unevictable lru/mlocked pages events.

Allow free of mlock()ed pages. This shouldn't happen, but during developement, it occasionally did.

This patch allows us to survive that condition, while keeping the statistics and events correct for debug.

 include/linux/vmstat.h |    1 +
 mm/internal.h          |   17 +++++++++++++++++
 mm/page_alloc.c        |    1 +
 mm/vmstat.c            |    1 +
 4 files changed, 20 insertions(+)

Index: linux-2.6.27-rc1-mmotm-30jul/include/linux/vmstat.h
===================================================================
@@ -49,6 +49,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
 		UNEVICTABLE_PGMUNLOCKED,
 		UNEVICTABLE_PGCLEARED,	/* on COW, page truncate */
 		UNEVICTABLE_PGSTRANDED,	/* unable to isolate on unlock */
+		UNEVICTABLE_MLOCKFREED,
 #endif
 		NR_VM_EVENT_ITEMS
 };

Index: linux-2.6.27-rc1-mmotm-30jul/mm/internal.h
===================================================================
@@ -146,6 +146,22 @@ static inline void mlock_migrate_page(st
 	}
 }
 
+/*
+ * free_page_mlock() -- clean up attempts to free and mlocked() page.
+ * Page should not be on lru, so no need to fix that up.
+ * free_pages_check() will verify...
+ */
+static inline void free_page_mlock(struct page *page)
+{
+	if (unlikely(TestClearPageMlocked(page))) {
+		unsigned long flags;
+
+		local_irq_save(flags);
+		__dec_zone_page_state(page, NR_MLOCK);
+		__count_vm_event(UNEVICTABLE_MLOCKFREED);
+		local_irq_restore(flags);
+	}
+}
 
 #else /* CONFIG_UNEVICTABLE_LRU */
 static inline int is_mlocked_vma(struct vm_area_struct *v, struct page *p)

@@ -155,6 +171,7 @@ static inline int is_mlocked_vma(struct 
 static inline void clear_page_mlock(struct page *page) { }
 static inline void mlock_vma_page(struct page *page) { }
 static inline void mlock_migrate_page(struct page *new, struct page *old) { }
+static inline void free_page_mlock(struct page *page) { }
 
 #endif /* CONFIG_UNEVICTABLE_LRU */
 

Index: linux-2.6.27-rc1-mmotm-30jul/mm/page_alloc.c
===================================================================
@@ -451,6 +451,7 @@ static inline void __free_one_page(struc
 
 static inline int free_pages_check(struct page *page)
 {
+	free_page_mlock(page);
 	if (unlikely(page_mapcount(page) |
 		(page->mapping != NULL)  |
 		(page_get_page_cgroup(page) != NULL) |

Index: linux-2.6.27-rc1-mmotm-30jul/mm/vmstat.c
===================================================================
@@ -672,6 +672,7 @@ static const char * const vmstat_text[] 
 	"unevictable_pgs_munlocked",
 	"unevictable_pgs_cleared",
 	"unevictable_pgs_stranded",
+	"unevictable_pgs_mlockfreed",
 #endif
 #endif
 };

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I hope I was supposed to drop vmstat-unevictable-and-mlocked-pages-vm-events.patch - it was getting 100% rejects. After dropping it, everything applied. Dunno if it compiles yet.

I have a feeling that I merged all these patches too soon - the amount of rework has been tremendous. Are we done yet?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yeah. I did state that explicitly in the cover message. In upper case, even! I had applied all of these patches in my copy of the mmotm tree and it does compile. Of course, you've probably added a few patches that I don't have in the meantime.

By merging them, we did find problems on platforms and configurations that I can't easily test. Most of the rework has served to simplify the resulting code, IMO.

As for being done, I don't know. Depends on whether anyone else finds issues with other configs or platforms that haven't tested these series yet. And, I'm continuing to review the current patches.

I did run the mmotm-080729 series for over 20 hours last night under a fairly heavy stress load on one of my ia64 systems. No errors, no leaked pages, ... Looks pretty good. Until the next test comes along...

So, as I mentioned, I've rebuilt with today's fixes applied to mmotm-080730 and will test more tomorrow. [We just moved our lab across state lines and my test systems are just coming on-line.] I'll keep an eye on the mmotm site and try to stay current with that.

Thanks for applying these.
Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I can't speak for the unreclaimable part of the patch series, but the first half of the split LRU series should be in a good shape now that you merged the patch from Johannes Weiner.

I am not aware of a situation where the split LRU VM performs worse than what is currently upstream, but I will continue to try finding performance regressions to fix :)

All rights reversed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Not really a patch, per se:

Suggested patch description replacement for:
ramfs-and-ram-disk-pages-are-unevictable.patch
in 27-rc1-mmotm-080730...

---

Christoph Lameter pointed out that ram disk pages also clutter the LRU lists. When vmscan finds them dirty and tries to clean them, the ram disk writeback function just redirties the page so that it goes back onto the active list. Round and round she goes...

With the ram disk driver [rd.c] replaced by the newer 'brd.c', this is no longer the case, as ram disk pages are no longer maintained on the lru. [This makes them unmigratable for defrag or memory hot remove, but that can be addressed by a separate patch series.] However, the ramfs pages behave like ram disk pages used to, so:

Define new address_space flag [shares address_space flags member with mapping's gfp mask] to indicate that the address space contains all unevictable pages. This will provide for efficient testing of ramfs pages in page_evictable().

Also provide wrapper functions to set/test the unevictable state to minimize #ifdefs in ramfs driver and any other users of this facility.

Set the unevictable state on address_space structures for new ramfs inodes. Test the unevictable state in page_evictable() to cull unevictable pages.

These changes depend on [CONFIG_]UNEVICTABLE_LRU.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

looks good to me.
but I can't believe my english skill.....

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>