Skip to contentSkip to conversation

Move mlocked pages off the LRU and track them

RFC In review Christoph Lameter 4 participants 12 messages Feb 5, 2007 – Feb 7, 2007

Subsystems (10)
lore
Share & tools
lore
#0 lore raw

cover Move mlocked pages off the LRU and track them

[RFC] Remove mlocked pages from the LRU and track them

The patchset removes mlocked pages from the LRU and maintains a counter for the number of discovered mlocked pages.

This is a lazy scheme for accounting for mlocked pages. The pages may only be discovered to be mlocked during reclaim. However, we attempt to detect mlocked pages at various other opportune moments. So in general the mlock counter is not far off the number of actual mlocked pages in the system.

Patch against 2.6.20-rc6-mm3

Known problems to be resolved:
- Page state bit used to mark a page mlocked is not available on i386 with NUMA.
- Note tested on SMP, UP. Need to catch a plane in 2 hours.

Tested on:
IA64 NUMA 12p

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

RFC 1/7 Make try_to_unmap return a special exit code

[PATCH] Make try_to_unmap() return SWAP_MLOCK for mlocked pages

Modify try_to_unmap() so that we can distinguish failing to unmap due to a mlocked page from other causes.

include/linux/rmap.h +1 -0
Index: current/include/linux/rmap.h
===================================================================
@@ -134,5 +134,6 @@ static inline int page_mkclean(struct pa
 #define SWAP_SUCCESS	0
 #define SWAP_AGAIN	1
 #define SWAP_FAIL	2
+#define SWAP_MLOCK	3
 
 #endif	/* _LINUX_RMAP_H */
mm/rmap.c +15 -6
Index: current/mm/rmap.c
===================================================================
@@ -631,10 +631,16 @@ static int try_to_unmap_one(struct page 
 	 * If it's recently referenced (perhaps page_referenced
 	 * skipped over this mm) then we should reactivate it.
 	 */
-	if (!migration && ((vma->vm_flags & VM_LOCKED) ||
-			(ptep_clear_flush_young(vma, address, pte)))) {
-		ret = SWAP_FAIL;
-		goto out_unmap;
+	if (!migration) {
+		if (vma->vm_flags & VM_LOCKED) {
+			ret = SWAP_MLOCK;
+			goto out_unmap;
+		}
+
+		if (ptep_clear_flush_young(vma, address, pte)) {
+			ret = SWAP_FAIL;
+			goto out_unmap;
+		}
 	}
 
 	/* Nuke the page table entry. */
@@ -799,7 +805,8 @@ static int try_to_unmap_anon(struct page
 
 	list_for_each_entry(vma, &anon_vma->head, anon_vma_node) {
 		ret = try_to_unmap_one(page, vma, migration);
-		if (ret == SWAP_FAIL || !page_mapped(page))
+		if (ret == SWAP_FAIL || ret == SWAP_MLOCK ||
+				!page_mapped(page))
 			break;
 	}
 	spin_unlock(&anon_vma->lock);
@@ -830,7 +837,8 @@ static int try_to_unmap_file(struct page
 	spin_lock(&mapping->i_mmap_lock);
 	vma_prio_tree_foreach(vma, &iter, &mapping->i_mmap, pgoff, pgoff) {
 		ret = try_to_unmap_one(page, vma, migration);
-		if (ret == SWAP_FAIL || !page_mapped(page))
+		if (ret == SWAP_FAIL || ret == SWAP_MLOCK ||
+				!page_mapped(page))
 			goto out;
 	}
 
@@ -913,6 +921,7 @@ out:
  * SWAP_SUCCESS	- we succeeded in removing all mappings
  * SWAP_AGAIN	- we missed a mapping, try again later
  * SWAP_FAIL	- the page is unswappable
+ * SWAP_MLOCK	- the page is under mlock()
  */
 int try_to_unmap(struct page *page, int migration)
 {
mm/vmscan.c +1 -0
Index: current/mm/vmscan.c
===================================================================
@@ -516,6 +516,7 @@ static unsigned long shrink_page_list(st
 		if (page_mapped(page) && mapping) {
 			switch (try_to_unmap(page, 0)) {
 			case SWAP_FAIL:
+			case SWAP_MLOCK:
 				goto activate_locked;
 			case SWAP_AGAIN:
 				goto keep_locked;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

RFC 2/7 Add PageMlocked() page state bit and lru infrastructure

In reply to Christoph Lameter

Add PageMlocked() infrastructure

This adds a new PG_mlocked to mark pages that were taken off the LRU because they have a reference from a VM_LOCKED vma.

Also add pagevec handling for returning mlocked pages to the LRU.

include/linux/page-flags.h +11 -0
Index: current/include/linux/page-flags.h
===================================================================
@@ -93,6 +93,7 @@
 
 #define PG_readahead		20	/* Reminder to do read-ahead */
 
+#define PG_mlocked		21	/* Page is mlocked */
 
 #if (BITS_PER_LONG > 32)
 /*
@@ -235,6 +236,16 @@ static inline void SetPageUptodate(struc
 #define SetPageReadahead(page)	set_bit(PG_readahead, &(page)->flags)
 #define ClearPageReadahead(page) clear_bit(PG_readahead, &(page)->flags)
 
+/*
+ * PageMlocked set means that the page was taken off the LRU because
+ * a VM_LOCKED vma does exist. PageMlocked must be cleared before a
+ * page is put back onto the LRU. PageMlocked is only modified
+ * under the zone->lru_lock like PageLRU.
+ */
+#define PageMlocked(page)	test_bit(PG_mlocked, &(page)->flags)
+#define SetPageMlocked(page)	set_bit(PG_mlocked, &(page)->flags)
+#define ClearPageMlocked(page)	clear_bit(PG_mlocked, &(page)->flags)
+
 struct page;	/* forward declaration */
 
 extern void cancel_dirty_page(struct page *page, unsigned int account_size);
include/linux/pagevec.h +1 -0
Index: current/include/linux/pagevec.h
===================================================================
@@ -25,6 +25,7 @@ void __pagevec_release_nonlru(struct pag
 void __pagevec_free(struct pagevec *pvec);
 void __pagevec_lru_add(struct pagevec *pvec);
 void __pagevec_lru_add_active(struct pagevec *pvec);
+void __pagevec_lru_add_mlock(struct pagevec *pvec);
 void pagevec_strip(struct pagevec *pvec);
 unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
 		pgoff_t start, unsigned nr_pages);
include/linux/swap.h +1 -0
Index: current/include/linux/swap.h
===================================================================
@@ -181,6 +181,7 @@ extern unsigned int nr_free_pagecache_pa
 extern void FASTCALL(lru_cache_add(struct page *));
 extern void FASTCALL(lru_cache_add_active(struct page *));
 extern void FASTCALL(lru_cache_add_tail(struct page *));
+extern void FASTCALL(lru_cache_add_mlock(struct page *));
 extern void FASTCALL(activate_page(struct page *));
 extern void FASTCALL(mark_page_accessed(struct page *));
 extern void lru_add_drain(void);
mm/swap.c +46 -0

82 lines not on this page · raw message

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

RFC 3/7 Add NR_MLOCK ZVC

In reply to Christoph Lameter

Basic infrastructure to support NR_MLOCK

Add a new ZVC to support NR_MLOCK. NR_MLOCK counts the number of mlocked pages taken off the LRU. Get rid of wrong calculation of cache line size in mmzone.h.

drivers/base/node.c +2 -0
Index: current/drivers/base/node.c
===================================================================
@@ -60,6 +60,7 @@ static ssize_t node_read_meminfo(struct 
 		       "Node %d FilePages:    %8lu kB\n"
 		       "Node %d Mapped:       %8lu kB\n"
 		       "Node %d AnonPages:    %8lu kB\n"
+		       "Node %d Mlock:        %8lu KB\n"
 		       "Node %d PageTables:   %8lu kB\n"
 		       "Node %d NFS_Unstable: %8lu kB\n"
 		       "Node %d Bounce:       %8lu kB\n"
@@ -82,6 +83,7 @@ static ssize_t node_read_meminfo(struct 
 		       nid, K(node_page_state(nid, NR_FILE_PAGES)),
 		       nid, K(node_page_state(nid, NR_FILE_MAPPED)),
 		       nid, K(node_page_state(nid, NR_ANON_PAGES)),
+		       nid, K(node_page_state(nid, NR_MLOCK)),
 		       nid, K(node_page_state(nid, NR_PAGETABLE)),
 		       nid, K(node_page_state(nid, NR_UNSTABLE_NFS)),
 		       nid, K(node_page_state(nid, NR_BOUNCE)),
fs/proc/proc_misc.c +2 -0
Index: current/fs/proc/proc_misc.c
===================================================================
@@ -166,6 +166,7 @@ static int meminfo_read_proc(char *page,
 		"Writeback:    %8lu kB\n"
 		"AnonPages:    %8lu kB\n"
 		"Mapped:       %8lu kB\n"
+		"Mlock:        %8lu KB\n"
 		"Slab:         %8lu kB\n"
 		"SReclaimable: %8lu kB\n"
 		"SUnreclaim:   %8lu kB\n"
@@ -196,6 +197,7 @@ static int meminfo_read_proc(char *page,
 		K(global_page_state(NR_WRITEBACK)),
 		K(global_page_state(NR_ANON_PAGES)),
 		K(global_page_state(NR_FILE_MAPPED)),
+		K(global_page_state(NR_MLOCK)),
 		K(global_page_state(NR_SLAB_RECLAIMABLE) +
 				global_page_state(NR_SLAB_UNRECLAIMABLE)),
 		K(global_page_state(NR_SLAB_RECLAIMABLE)),
include/linux/mmzone.h +1 -2
Index: current/include/linux/mmzone.h
===================================================================
@@ -47,17 +47,16 @@ struct zone_padding {
 #endif
 
 enum zone_stat_item {
-	/* First 128 byte cacheline (assuming 64 bit words) */
 	NR_FREE_PAGES,
 	NR_INACTIVE,
 	NR_ACTIVE,
+	NR_MLOCK,	/* Mlocked pages */
 	NR_ANON_PAGES,	/* Mapped anonymous pages */
 	NR_FILE_MAPPED,	/* pagecache pages mapped into pagetables.
 			   only modified from process context */
 	NR_FILE_PAGES,
 	NR_FILE_DIRTY,
 	NR_WRITEBACK,
-	/* Second 128 byte cacheline */
 	NR_SLAB_RECLAIMABLE,
 	NR_SLAB_UNRECLAIMABLE,
 	NR_PAGETABLE,		/* used for pagetables */
mm/vmstat.c +1 -0
Index: current/mm/vmstat.c
===================================================================
@@ -434,6 +434,7 @@ static const char * const vmstat_text[] 
 	"nr_free_pages",
 	"nr_active",
 	"nr_inactive",
+	"nr_mlock",
 	"nr_anon_pages",
 	"nr_mapped",
 	"nr_file_pages",

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

RFC 4/7 Logic to move mlocked pages

In reply to Christoph Lameter

Add logic to lazily remove/add mlocked pages from LRU

This is the core of the patchset. It adds the necessary logic to remove mlocked pages from the LRU and put them back later. Basic idea by Andrew Morton and others.

During reclaim we attempt to unmap pages. In order to do so we have to scan all vmas that a page belongs to to check if VM_LOCKED is set.

If we find that this is the case for a page then we remove the page from the LRU and mark it with SetMlocked so that we know that we need to put the page back to the LRU later should the mlocked state be cleared.

We put the pages back in two places:

zap_pte_range: Pages are removed from a vma. If a page is mlocked then we
add it back to the LRU. If other vmas with VM_LOCKED set have mapped the page then we will discover that later during reclaim and move the page off the LRU again.

munlock/munlockall: We scan all pages in the vma and do the
same as in zap_pte_range.

We also have to modify the page migration logic to handle PageMlocked pages. We simply clear the PageMlocked bit and then we can treat the page as a regular page from the LRU.

Note that this is a lazy accounting for mlocked pages. NR_MLOCK may increase as the system discovers more mlocked pages. Some of the later patches opportunistically move pages off the LRU earlier avoiding some of the delayed accounting. However, the scheme is fundamentally lazy and one cannot count on NR_MLOCK to reflect the actual number of mlocked pages. It is the number of so far *discovered* mlocked pages which may be less than the actual number of mlocked pages.

mm/memory.c +2 -0
Index: current/mm/memory.c
===================================================================
@@ -682,6 +682,8 @@ static unsigned long zap_pte_range(struc
 				file_rss--;
 			}
 			page_remove_rmap(page, vma);
+			if (PageMlocked(page) && vma->vm_flags & VM_LOCKED)
+				lru_cache_add_mlock(page);
 			tlb_remove_page(tlb, page);
 			continue;
 		}
mm/migrate.c +5 -0
Index: current/mm/migrate.c
===================================================================
@@ -58,6 +58,11 @@ int isolate_lru_page(struct page *page, 
 			else
 				del_page_from_inactive_list(zone, page);
 			list_add_tail(&page->lru, pagelist);
+		} else
+		if (PageMlocked(page)) {
+			get_page(page);
+			ClearPageMlocked(page);
+			list_add_tail(&page->lru, pagelist);
 		}
 		spin_unlock_irq(&zone->lru_lock);
 	}
mm/mlock.c +19 -1
Index: current/mm/mlock.c
===================================================================
@@ -10,7 +10,7 @@
 #include <linux/mm.h>
 #include <linux/mempolicy.h>
 #include <linux/syscalls.h>
-
+#include <linux/swap.h>
 
 static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,
 	unsigned long start, unsigned long end, unsigned int newflags)
@@ -63,6 +63,24 @@ success:
 		pages = -pages;
 		if (!(newflags & VM_IO))
 			ret = make_pages_present(start, end);
+	} else {
+		unsigned long addr;
+
+		/*
+		 * We are clearing VM_LOCKED. Feed all pages back via
+		 * to the LRU via lru_cache_add_mlock()
+		 */
+		for (addr = start; addr < end; addr += PAGE_SIZE) {
+			/*
+			 * No need to get a page reference. mmap_sem
+			 * writelock is held.
+			 */
+			struct page *page = follow_page(vma, start, 0);
+
+			if (PageMlocked(page))
+				lru_cache_add_mlock(page);
+			cond_resched();
+		}
 	}
 
 	mm->locked_vm -= pages;
mm/vmscan.c +9 -1
Index: current/mm/vmscan.c
===================================================================
@@ -516,10 +516,11 @@ static unsigned long shrink_page_list(st
 		if (page_mapped(page) && mapping) {
 			switch (try_to_unmap(page, 0)) {
 			case SWAP_FAIL:
-			case SWAP_MLOCK:
 				goto activate_locked;
 			case SWAP_AGAIN:
 				goto keep_locked;
+			case SWAP_MLOCK:
+				goto mlocked;
 			case SWAP_SUCCESS:
 				; /* try to free the page below */
 			}
@@ -594,6 +595,13 @@ free_it:
 			__pagevec_release_nonlru(&freed_pvec);
 		continue;
 
+mlocked:
+		ClearPageActive(page);
+		unlock_page(page);
+		__inc_zone_page_state(page, NR_MLOCK);
+		SetPageMlocked(page);
+		continue;
+
 activate_locked:
 		SetPageActive(page);
 		pgactivate++;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

RFC 5/7 Consolidate new anonymous page code paths

In reply to Christoph Lameter

Consolidate code to add an anonymous page in memory.c

There are two location in which we add anonymous pages. Both implement the same logic. Create a new function add_anon_page() to have a common code path.

mm/memory.c +15 -8
Index: current/mm/memory.c
===================================================================
@@ -900,6 +900,17 @@ unsigned long zap_page_range(struct vm_a
 }
 
 /*
+ * Add a new anonymous page
+ */
+static void add_anon_page(struct vm_area_struct *vma, struct page *page,
+				unsigned long address)
+{
+	inc_mm_counter(vma->vm_mm, anon_rss);
+	lru_cache_add_active(page);
+	page_add_new_anon_rmap(page, vma, address);
+}
+
+/*
  * Do a quick page-table lookup for a single page.
  */
 struct page *follow_page(struct vm_area_struct *vma, unsigned long address,
@@ -2103,9 +2114,7 @@ static int do_anonymous_page(struct mm_s
 		page_table = pte_offset_map_lock(mm, pmd, address, &ptl);
 		if (!pte_none(*page_table))
 			goto release;
-		inc_mm_counter(mm, anon_rss);
-		lru_cache_add_active(page);
-		page_add_new_anon_rmap(page, vma, address);
+		add_anon_page(vma, page, address);
 	} else {
 		/* Map the ZERO_PAGE - vm_page_prot is readonly */
 		page = ZERO_PAGE(address);
@@ -2249,11 +2258,9 @@ retry:
 		if (write_access)
 			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		set_pte_at(mm, address, page_table, entry);
-		if (anon) {
-			inc_mm_counter(mm, anon_rss);
-			lru_cache_add_active(new_page);
-			page_add_new_anon_rmap(new_page, vma, address);
-		} else {
+		if (anon)
+			add_anon_page(vma, new_page, address);
+		else {
 			inc_mm_counter(mm, file_rss);
 			page_add_file_rmap(new_page);
 			if (write_access) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw

RFC 6/7 Avoid putting new mlocked anonymous pages on LRU

In reply to Christoph Lameter

Mark new anonymous pages mlocked if they are in a mlocked VMA.

Avoid putting pages onto the LRU that are allocated in a VMA with VM_LOCKED set. NR_MLOCK will be more accurate.

mm/memory.c +9 -1
Index: current/mm/memory.c
===================================================================
@@ -906,7 +906,15 @@ static void add_anon_page(struct vm_area
 				unsigned long address)
 {
 	inc_mm_counter(vma->vm_mm, anon_rss);
-	lru_cache_add_active(page);
+	if (vma->vm_flags & VM_LOCKED) {
+		/*
+		 * Page is new and therefore not on the LRU
+		 * so we can directly mark it as mlocked
+		 */
+		SetPageMlocked(page);
+		inc_zone_page_state(page, NR_MLOCK);
+	} else
+		lru_cache_add_active(page);
 	page_add_new_anon_rmap(page, vma, address);
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

RFC 7/7 Opportunistically move mlocked pages off the LRU

In reply to Christoph Lameter

Opportunistically move mlocked pages off the LRU

Add a new function try_to_mlock() that attempts to move a page off the LRU and marks it mlocked.

This function can then be used in various code paths to move pages off the LRU immediately. Early discovery will make NR_MLOCK track the actual number of mlocked pages in the system more closely.

mm/memory.c +28 -0
Index: current/mm/memory.c
===================================================================
@@ -919,6 +919,30 @@ static void add_anon_page(struct vm_area
 }
 
 /*
+ * Opportunistically move the page off the LRU
+ * if possible. If we do not succeed then the LRU
+ * scans will take the page off.
+ */
+void try_to_set_mlocked(struct page *page)
+{
+	struct zone *zone;
+	unsigned long flags;
+
+	if (!PageLRU(page) || PageMlocked(page))
+		return;
+
+	zone = page_zone(page);
+	if (spin_trylock_irqsave(&zone->lru_lock, flags)) {
+		if (PageLRU(page) && !PageMlocked(page)) {
+			ClearPageLRU(page);
+			list_del(&page->lru);
+			SetPageMlocked(page);
+			__inc_zone_page_state(page, NR_MLOCK);
+		}
+		spin_unlock_irqrestore(&zone->lru_lock, flags);
+	}
+}
+/*
  * Do a quick page-table lookup for a single page.
  */
 struct page *follow_page(struct vm_area_struct *vma, unsigned long address,
@@ -978,6 +1002,8 @@ struct page *follow_page(struct vm_area_
 			set_page_dirty(page);
 		mark_page_accessed(page);
 	}
+	if (vma->vm_flags & VM_LOCKED)
+		try_to_set_mlocked(page);
 unlock:
 	pte_unmap_unlock(ptep, ptl);
 out:
@@ -2271,6 +2297,8 @@ retry:
 		else {
 			inc_mm_counter(mm, file_rss);
 			page_add_file_rmap(new_page);
+			if (vma->vm_flags & VM_LOCKED)
+				try_to_set_mlocked(new_page);
 			if (write_access) {
 				dirty_page = new_page;
 				get_page(dirty_page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
In reply to Christoph Lameter
Quoted 13 lines · Christoph Lameter

[RFC] Remove mlocked pages from the LRU and track them

The patchset removes mlocked pages from the LRU and maintains a counter for the number of discovered mlocked pages.

This is a lazy scheme for accounting for mlocked pages. The pages may only be discovered to be mlocked during reclaim. However, we attempt to detect mlocked pages at various other opportune moments. So in general the mlock counter is not far off the number of actual mlocked pages in the system.

Patch against 2.6.20-rc6-mm3

Quoted 7 lines

Known problems to be resolved:
- Page state bit used to mark a page mlocked is not available on i386 with NUMA.
- Note tested on SMP, UP. Need to catch a plane in 2 hours.

Tested on:
IA64 NUMA 12p

Note that anon [and shmem] pages in excess of available swap are effectively mlocked(). In the field, we have seen non-NUMA x86_64 systems with 64-128GB [16-32million 4k pages] with little to no swap--big data base servers. The majority of the memory is dedicated to large data base shared memory areas. The remaining is divided between program anon and page cache [executable, libs] pages and any other page cache pages used by data base utilities, system daemons, ...

The system runs fine until someone runs a backup [or multiple, as there are multiple data base instances running]. This over commits memory and we end up with all cpus in reclaim, contending for the zone lru lock, and walking an active list of 10s of millions of pages looking for pages to reclaim. The reclaim logic spends a lot of time walking the lru lists, nominating shmem pages [the majority of pages on the list] for reclaim, only to find in shrink_pages() that it can't move the page to swap. So, it puts it back on the list to be retried by the other cpus once they obtain the zone lru lock. System appears to be hung for long periods of time.

There are a lot of behaviors in the reclaim code that exacerbate the problems when we get into this mode, but the long lists of unswappable anon/shmem pages is the major culprit. One of the guys at Red Hat has tried a "proof of concept" patch to move all anon/shmem pages in excess of swap space to "wired list" [currently global, per node/zone in progress] and it seems to alleviate the problem.

So, Christoph's patch addresses a real problem that we've seen. Unfortunately, not all data base applications lock their shmem areas into memory. Excluding pages from consideration for reclaim that can't possibly be swapped out due to lack of swap space seems a natural extension of this concept. I expect that many Christoph's customers run with swap space that is much smaller than system memory and would benefit from this extension.

Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 34 lines · Lee Schermerhorn
Quoted 13 lines · Christoph Lameter

[RFC] Remove mlocked pages from the LRU and track them

The patchset removes mlocked pages from the LRU and maintains a counter for the number of discovered mlocked pages.

This is a lazy scheme for accounting for mlocked pages. The pages may only be discovered to be mlocked during reclaim. However, we attempt to detect mlocked pages at various other opportune moments. So in general the mlock counter is not far off the number of actual mlocked pages in the system.

Patch against 2.6.20-rc6-mm3

Quoted 9 lines

Known problems to be resolved:
- Page state bit used to mark a page mlocked is not available on i386 with NUMA.
- Note tested on SMP, UP. Need to catch a plane in 2 hours.

Tested on:
IA64 NUMA 12p

Note that anon [and shmem] pages in excess of available swap are effectively mlocked(). In the field, we have seen non-NUMA x86_64 systems with 64-128GB [16-32million 4k pages] with little to no swap--big data base servers. The majority of the memory is dedicated to large data base shared memory areas. The remaining is divided between program anon and page cache [executable, libs] pages and any other page cache pages used by data base utilities, system daemons, ...

Quoted 11 lines

The system runs fine until someone runs a backup [or multiple, as there are multiple data base instances running]. This over commits memory and we end up with all cpus in reclaim, contending for the zone lru lock, and walking an active list of 10s of millions of pages looking for pages to reclaim. The reclaim logic spends a lot of time walking the lru lists, nominating shmem pages [the majority of pages on the list] for reclaim, only to find in shrink_pages() that it can't move the page to swap. So, it puts it back on the list to be retried by the other cpus once they obtain the zone lru lock. System appears to be hung for long periods of time.

Its actually more complicated that this. The pagecache pages from the backup are
placed on the head of inactive list when they are initialized via add_to_page_cache(). If they get referenced again, mark_page_accessed() moves then to the tail of the active list. At this point they wont be considered for deactivation and reclamation
until every other page in the system is. They probably should go on the head of the
inactive list if they are not mapped, for at least one more time. Once we start adding
clean, unmapped pagecache pages behind anonymous and mapped system V shared
memory pages on the active list we start swapping while there are better pages to select
for reclamation. This prevents kswapd from keeping the free pages above min and
then every caller of __alloc_pages() starts reclaiming memory on every zone in the
zone list. Once the system is in that state the whole distress algorithm in shrink_active_list()
doesnt work right any more because it uses prev_priority as the distress shift. This
means that if one caller of shrink_active_list() is deactivating mapped anonymous memoy
because the priority got that high(actually low) every caller is deactivating mapped
anonymous memoy. The system degrades to a point where both the active and inactive
lists contain a mixture of dirty and clean, mapped and unmapped, pagecache and
anonymous memory and every CPU is down in try_to_free_pages() mixing it even
further.

Quoted 16 lines

There are a lot of behaviors in the reclaim code that exacerbate the problems when we get into this mode, but the long lists of unswappable anon/shmem pages is the major culprit. One of the guys at Red Hat has tried a "proof of concept" patch to move all anon/shmem pages in excess of swap space to "wired list" [currently global, per node/zone in progress] and it seems to alleviate the problem.

So, Christoph's patch addresses a real problem that we've seen. Unfortunately, not all data base applications lock their shmem areas into memory. Excluding pages from consideration for reclaim that can't possibly be swapped out due to lack of swap space seems a natural extension of this concept. I expect that many Christoph's customers run with swap space that is much smaller than system memory and would benefit from this extension.

Quoted 4 lines

Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#10 lore raw
In reply to Lee Schermerhorn
Quoted 26 lines · On Tue, 06 Feb 2007 11:04:42 -0500 Lee Schermerhorn <[email protected]>…

Note that anon [and shmem] pages in excess of available swap are effectively mlocked(). In the field, we have seen non-NUMA x86_64 systems with 64-128GB [16-32million 4k pages] with little to no swap--big data base servers. The majority of the memory is dedicated to large data base shared memory areas. The remaining is divided between program anon and page cache [executable, libs] pages and any other page cache pages used by data base utilities, system daemons, ...

The system runs fine until someone runs a backup [or multiple, as there are multiple data base instances running]. This over commits memory and we end up with all cpus in reclaim, contending for the zone lru lock, and walking an active list of 10s of millions of pages looking for pages to reclaim. The reclaim logic spends a lot of time walking the lru lists, nominating shmem pages [the majority of pages on the list] for reclaim, only to find in shrink_pages() that it can't move the page to swap. So, it puts it back on the list to be retried by the other cpus once they obtain the zone lru lock. System appears to be hung for long periods of time.

There are a lot of behaviors in the reclaim code that exacerbate the problems when we get into this mode, but the long lists of unswappable anon/shmem pages is the major culprit. One of the guys at Red Hat has tried a "proof of concept" patch to move all anon/shmem pages in excess of swap space to "wired list" [currently global, per node/zone in progress] and it seems to alleviate the problem.

Quoted 7 lines

So, Christoph's patch addresses a real problem that we've seen. Unfortunately, not all data base applications lock their shmem areas into memory. Excluding pages from consideration for reclaim that can't possibly be swapped out due to lack of swap space seems a natural extension of this concept. I expect that many Christoph's customers run with swap space that is much smaller than system memory and would benefit from this extension.

Yeah.

The scanner at present tries to handle out-of-swap by moving these pages onto the active list (shrink_page_list) then keeping them there (shrink_active_list) so it _should_ be the case that the performance problems which you're observing are due to active list scanning. Is that correct?

If not, something's busted.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw
Quoted 43 lines · Andrew Morton
Quoted 26 lines · On Tue, 06 Feb 2007 11:04:42 -0500 Lee Schermerhorn <[email protected]>…

Note that anon [and shmem] pages in excess of available swap are effectively mlocked(). In the field, we have seen non-NUMA x86_64 systems with 64-128GB [16-32million 4k pages] with little to no swap--big data base servers. The majority of the memory is dedicated to large data base shared memory areas. The remaining is divided between program anon and page cache [executable, libs] pages and any other page cache pages used by data base utilities, system daemons, ...

The system runs fine until someone runs a backup [or multiple, as there are multiple data base instances running]. This over commits memory and we end up with all cpus in reclaim, contending for the zone lru lock, and walking an active list of 10s of millions of pages looking for pages to reclaim. The reclaim logic spends a lot of time walking the lru lists, nominating shmem pages [the majority of pages on the list] for reclaim, only to find in shrink_pages() that it can't move the page to swap. So, it puts it back on the list to be retried by the other cpus once they obtain the zone lru lock. System appears to be hung for long periods of time.

There are a lot of behaviors in the reclaim code that exacerbate the problems when we get into this mode, but the long lists of unswappable anon/shmem pages is the major culprit. One of the guys at Red Hat has tried a "proof of concept" patch to move all anon/shmem pages in excess of swap space to "wired list" [currently global, per node/zone in progress] and it seems to alleviate the problem.

Quoted 9 lines

So, Christoph's patch addresses a real problem that we've seen. Unfortunately, not all data base applications lock their shmem areas into memory. Excluding pages from consideration for reclaim that can't possibly be swapped out due to lack of swap space seems a natural extension of this concept. I expect that many Christoph's customers run with swap space that is much smaller than system memory and would benefit from this extension.

Yeah.

Quoted 8 lines

The scanner at present tries to handle out-of-swap by moving these pages onto the active list (shrink_page_list) then keeping them there (shrink_active_list) so it _should_ be the case that the performance problems which you're observing are due to active list scanning. Is that correct?

If not, something's busted.

This is true but when mark_page_accessed() activates referenced pagecache pages it mixes them with the non-swapable anonymous and system V shared memory pages
on the active list. This combined with lots of heavy filesystem writing prevent kswapd
from keeping up with the memory demmand so the free list(s) fall below zone->pages_min and every call to __alloc_pages() results in calling try_to_free_pages(). Once all CPUs are scanning and trying to reclaim the system chokes, especially on systems with lots of CPUs and lots of RAM.

Larry Woodman

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>