VM pageout scalability improvements (V7)
Subsystems (17)
cover VM pageout scalability improvements (V7)
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state.
Against 2.6.25-rc2-mm1
This patch series improves VM scalability by:
1) putting filesystem backed, swap backed and non-reclaimable pages onto their own LRUs, so the system only scans the pages that it can/should evict from memory
2) switching to SEQ replacement for the anonymous LRUs, so the number of pages that need to be scanned when the system starts swapping is bound to a reasonable number
3) keeping non-reclaimable pages off the LRU completely, so the VM does not waste CPU time scanning them. Currently only ramfs and SHM_LOCKED pages are kept on the noreclaim list, mlock()ed VMAs will be added later
More info on the overall design can be found at:
http://linux-mm.org/PageReplacementDesign
An all-in-one patch can be found at:
http://people.redhat.com/riel/splitvm/
Changelog:
- improve swap space freeing to deal with COW shared space
(Lee Schermerhorn, Daisuke Nishimura & Minchan Kim)
- clean up PG_swapbacked setting in swapin path (Minchan Kim)
- properly invoke shrink_active_list for background aging (Minchan Kim)
- add authorship info to all patches (Rik van Riel)
- clean up (or move below ---) the comments for the commit logs (Rik van Riel)
- after some tests, reduce default swappiness to 20 for now (Rik van Riel)
- several code cleanups (minchan Kim)
- noreclaim patch refactoring and improvements (Lee Schermerhorn)
- several PROT_NONE and vma merging fixes (KOSAKI Motohiro)
- SMP bugfixes and efficiency improvements (Rik van Riel, Lee Schermerhorn)
- fix NUMA node stats printing (Lee Schermerhorn)
- remove the mlocked-VMA-noreclaim code for now, it still has bugs on IA64 and is holding up the merge (Rik van Riel)
- make page_alloc.c compile without CONFIG_NORECLAIM_MLOCK (minchan Kim)
- BUG() does not take an argument (minchan Kim)
- clean up is_active_lru and is_file_lru (Andy Whitcroft)
- clean up shrink_active_list temp list names (KOSAKI Motohiro)
- add total active & inactive memory totals for vmstat -a (KOSAKI Motohiro)
- only try global anon page aging on global lru scans (KOSAKI Motohiro)
- make function descriptions follow the kernel-doc format (Rik van Riel)
- simplify mlock_vma_pages_range and munlock_vma_pages_range (Lee Schermerhorn)
- remove some more arguments, rename to mlock_vma_pages_all (Lee Schermerhorn)
- many code cleanups (Lee Schermerhorn)
- pass correct vma arg to mlock_vma_pages_range from do_brk (Rik van Riel)
- port to 2.6.25-rc3-mm1
- pull the memcontrol lru arrayification earlier into the patch series
- use a pagevec array similar to the lru array
- clean up the code in various places
- improved pageout balancing and reduced pageout cpu use
- fix compilation on PPC and without memcontrol
- make page_is_pagecache more readable
- replace get_scan_ratio with correct version
- merge memcontroller split LRU code into the main split LRU patch, since it is not functionally different (it was split up only to help people who had seen the last version of the patch series review it)
- drop the page_file_cache debugging patch, since it never triggered
- reintroduce code to not scan anon list if swap is full
- add code to scan anon list if page cache is very small already
- use lumpy reclaim more aggressively for smaller order > 1 allocations
Signature · 1 line
All Rights Reversed
PATCH 1/14 move isolate_lru_page() to vmscan.c
From: Nick Piggin <[email protected]>
isolate_lru_page logically belongs to be in vmscan.c than migrate.c.
It is tough, because we don't need that function without memory migration so there is a valid argument to have it in migrate.c. However a subsequent patch needs to make use of it in the core mm, so we can happily move it to vmscan.c.
Also, make the function a little more generic by not requiring that it adds an isolated page to a given list. Callers can do that.
Note that we now have '__isolate_lru_page()', that does something quite different, visible outside of vmscan.c for use with memory controller. Methinks we need to
rationalize these names/purposes. --lts
Signed-off-by Nick Piggin <[email protected]>
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by Lee Schermerhorn <[email protected]>
---
V1 -> V2 [lts]:
+ fix botched merge -- add back "get_page_unless_zero()"
From: Nick Piggin <[email protected]>
To: Linux Memory Management <[email protected]>
Subject: [patch 1/4] mm: move and rework isolate_lru_page
Date: Mon, 12 Mar 2007 07:38:44 +0100 (CET)
include/linux/migrate.h +0 -3
@@ -25,7 +25,6 @@ static inline int vma_migratable(structreturn 1 ; } -extern int isolate_lru_page (struct page *p,struct list_head *pagelist);extern int putback_lru_pages (struct list_head *l);extern int migrate_page (struct address_space *,struct page *,struct page *);
@@ -42,8 +41,6 @@ extern int migrate_vmas(struct mm_structstatic inline int vma_migratable (struct vm_area_struct *vma) {return 0 ; } -static inline int isolate_lru_page (struct page *p,struct list_head *list) - {return -ENOSYS ; }static inline int putback_lru_pages (struct list_head *l) {return 0 ; }static inline int migrate_pages (struct list_head *l,new_page_t x,unsigned long private) {return -ENOSYS ; }
mm/internal.h +2 -0
@@ -34,6 +34,8 @@ static inline void __put_page(struct pagatomic_dec (&page->_count); } +extern int isolate_lru_page (struct page *page); +extern void __free_pages_bootmem (struct page *page,unsigned int order);/*
mm/migrate.c +3 -31
@@ -37,36 +37,6 @@#define lru_to_page (_head) (list_entry((_head)->prev, struct page, lru))/* -* Isolate one page from the LRU lists. If successful put it onto -* the indicated list with elevated page count. -* -* Result: -* -EBUSY: page not on LRU list -* 0: page removed from LRU list and added to the specified list. -*/ -int isolate_lru_page (struct page *page,struct list_head *pagelist) -{ -int ret = -EBUSY ; - -if (PageLRU (page)) { -struct zone *zone =page_zone (page); - -spin_lock_irq (&zone->lru_lock); -if (PageLRU (page) &&get_page_unless_zero (page)) { - ret =0 ; -ClearPageLRU (page); -if (PageActive (page)) -del_page_from_active_list (zone, page); -else -del_page_from_inactive_list (zone, page); -list_add_tail (&page->lru, pagelist); - } -spin_unlock_irq (&zone->lru_lock); - } -return ret; -} - -/* * migrate_prep() needs to be called before we start compiling a list of pages * to be migrated using isolate_lru_page(). */
@@ -895,7 +865,9 @@ static int do_move_pages(struct mm_struc !migrate_all) goto put_and_set; - err =isolate_lru_page (page, &pagelist); + err =isolate_lru_page (page); +if (!err) +list_add_tail (&page->lru, &pagelist); put_and_set:/* * Either remove the duplicate refcount from
mm/vmscan.c +45 -0
51 lines not on this page · raw message
mm/mempolicy.c +7 -2
@@ -93,6 +93,8 @@#include <asm/tlbflush.h> #include <asm/uaccess.h> +#include "internal.h" +/* Internal flags */ #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0)/* Skip checks for continuous vmas */ #define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1)/* Invert check for nodemask */
@@ -758,8 +760,11 @@ static void migrate_page_add(struct page/* * Avoid migrating a page that is shared with others. */ -if ((flags &MPOL_MF_MOVE_ALL ) ||page_mapcount (page) ==1 ) -isolate_lru_page (page, pagelist); +if ((flags &MPOL_MF_MOVE_ALL ) ||page_mapcount (page) ==1 ) { +if (!isolate_lru_page (page)) { +list_add_tail (&page->lru, pagelist); + } + } }static struct page *new_node_page (struct page *page,unsigned long node,int **x)
Signature · 1 line
All Rights Reversed
PATCH 2/14 Use an indexed array for LRU variables
In reply to Rik van RielFrom: Christoph Lameter <[email protected]>
Currently we are defining explicit variables for the inactive and active list. An indexed array can be more generic and avoid repeating similar code in several places in the reclaim code.
We are saving a few bytes in terms of code size:
Before:
text data bss dec hex filename
4097753 573120 4092484 8763357 85b7dd vmlinux
After:
text data bss dec hex filename
4097729 573120 4092484 8763333 85b7c5 vmlinux
Having an easy way to add new lru lists may ease future work on the reclaim code.
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>
---
V3 [riel]: memcontrol LRU arrayification
V1 -> V2 [lts]:
+ Remove extraneous __dec_zone_state(zone, NR_ACTIVE) pointed out by Mel G.
include/linux/memcontrol.h | 17 +--- include/linux/mm_inline.h | 33 +++++--- include/linux/mmzone.h | 17 ++-- mm/memcontrol.c | 116 ++++++++++------------------ mm/page_alloc.c | 9 +- mm/swap.c | 2 mm/vmscan.c | 141 +++++++++++++++++------------------ mm/vmstat.c | 3 8 files changed, 158 insertions(+), 180 deletions(-)
include/linux/mmzone.h +11 -6
@@ -81,8 +81,8 @@ struct zone_padding {enum zone_stat_item {/* First 128 byte cacheline (assuming 64 bit words) */ NR_FREE_PAGES , -NR_INACTIVE , -NR_ACTIVE , +NR_INACTIVE ,/* must match order of LRU_[IN]ACTIVE */ +NR_ACTIVE ,/* " " " " " */ NR_ANON_PAGES ,/* Mapped anonymous pages */ NR_FILE_MAPPED ,/* pagecache pages mapped into pagetables. only modified from process context */
@@ -107,6 +107,13 @@ enum zone_stat_item {#endif NR_VM_ZONE_STAT_ITEMS }; +enum lru_list { +LRU_INACTIVE ,/* must match order of NR_[IN]ACTIVE */ +LRU_ACTIVE ,/* " " " " " */ +NR_LRU_LISTS }; + +#define for_each_lru (l) for (l = 0; l < NR_LRU_LISTS; l++) +struct per_cpu_pages {int count;/* number of pages in the list */ int high;/* high watermark, emptying needed */
@@ -251,10 +258,8 @@ struct zone {/* Fields commonly accessed by the page reclaim scanner */ spinlock_t lru_lock; -struct list_head active_list; -struct list_head inactive_list; -unsigned long nr_scan_active; -unsigned long nr_scan_inactive; +struct list_head list[NR_LRU_LISTS ]; +unsigned long nr_scan[NR_LRU_LISTS ];unsigned long pages_scanned;/* since last reclaim */ unsigned long flags;/* zone flags, see below */
include/linux/mm_inline.h +22 -11
@@ -1,40 +1,51 @@static inline void +add_page_to_lru_list (struct zone *zone,struct page *page,enum lru_list l) +{ +list_add (&page->lru, &zone->list[l]); +__inc_zone_state (zone,NR_INACTIVE + l); +} + +static inline void +del_page_from_lru_list (struct zone *zone,struct page *page,enum lru_list l) +{ +list_del (&page->lru); +__dec_zone_state (zone,NR_INACTIVE + l); +} + +static inline void add_page_to_active_list (struct zone *zone,struct page *page) { -list_add (&page->lru, &zone->active_list); -__inc_zone_state (zone,NR_ACTIVE ); +add_page_to_lru_list (zone, page,LRU_ACTIVE ); }static inline void add_page_to_inactive_list (struct zone *zone,struct page *page) { -list_add (&page->lru, &zone->inactive_list); -__inc_zone_state (zone,NR_INACTIVE ); +add_page_to_lru_list (zone, page,LRU_INACTIVE ); }static inline void del_page_from_active_list (struct zone *zone,struct page *page) { -list_del (&page->lru); -__dec_zone_state (zone,NR_ACTIVE ); +del_page_from_lru_list (zone, page,LRU_ACTIVE ); }static inline void del_page_from_inactive_list (struct zone *zone,struct page *page) { -list_del (&page->lru); -__dec_zone_state (zone,NR_INACTIVE ); +del_page_from_lru_list (zone, page,LRU_INACTIVE ); }static inline void del_page_from_lru (struct zone *zone,struct page *page) { +enum lru_list l =LRU_INACTIVE ; +list_del (&page->lru);if (PageActive (page)) {__ClearPageActive (page); -__dec_zone_state (zone,NR_ACTIVE ); - }else { -__dec_zone_state (zone,NR_INACTIVE ); + l =LRU_ACTIVE ; } +__dec_zone_state (zone,NR_INACTIVE + l); }
mm/page_alloc.c +5 -4
@@ -3444,6 +3444,7 @@ static void __paginginit free_area_init_for (j =0 ; j <MAX_NR_ZONES ; j++) {struct zone *zone = pgdat->node_zones + j;unsigned long size, realsize, memmap_pages; +enum lru_list l; size =zone_spanned_pages_in_node (nid, j, zones_size); realsize = size - zone_absent_pages_in_node(nid, j,
@@ -3494,10 +3495,10 @@ static void __paginginit free_area_init_ zone->prev_priority =DEF_PRIORITY ;zone_pcp_init (zone); -INIT_LIST_HEAD (&zone->active_list); -INIT_LIST_HEAD (&zone->inactive_list); - zone->nr_scan_active =0 ; - zone->nr_scan_inactive =0 ; +for_each_lru (l) { +INIT_LIST_HEAD (&zone->list[l]); + zone->nr_scan[l] =0 ; + }zap_zone_vm_stats (zone); zone->flags =0 ;if (!size)
mm/swap.c +1 -1
@@ -117,7 +117,7 @@ static void pagevec_move_tail(struct pagspin_lock (&zone->lru_lock); }if (PageLRU (page) && !PageActive (page)) { -list_move_tail (&page->lru, &zone->inactive_list); +list_move_tail (&page->lru, &zone->list[LRU_INACTIVE ]); pgmoved++; } }
mm/vmscan.c +57 -63
198 lines not on this page · raw message
mm/vmstat.c +2 -1
@@ -772,7 +772,8 @@ static void zoneinfo_show_print(struct s zone->pages_low, zone->pages_high, zone->pages_scanned, - zone->nr_scan_active, zone->nr_scan_inactive, + zone->nr_scan[LRU_ACTIVE ], + zone->nr_scan[LRU_INACTIVE ], zone->spanned_pages, zone->present_pages);
include/linux/memcontrol.h +5 -12
@@ -67,10 +67,8 @@ extern void mem_cgroup_note_reclaim_prioextern void mem_cgroup_record_reclaim_priority (struct mem_cgroup *mem,int priority); -extern long mem_cgroup_calc_reclaim_active (struct mem_cgroup *mem, -struct zone *zone,int priority); -extern long mem_cgroup_calc_reclaim_inactive (struct mem_cgroup *mem, -struct zone *zone,int priority); +extern long mem_cgroup_calc_reclaim (struct mem_cgroup *mem,struct zone *zone, +int priority,enum lru_list lru);#else /* CONFIG_CGROUP_MEM_RES_CTLR */ static inline void page_reset_bad_cgroup (struct page *page)
@@ -152,14 +150,9 @@ static inline void mem_cgroup_record_rec { } -static inline long mem_cgroup_calc_reclaim_active (struct mem_cgroup *mem, -struct zone *zone,int priority) -{ -return 0 ; -} - -static inline long mem_cgroup_calc_reclaim_inactive (struct mem_cgroup *mem, -struct zone *zone,int priority) +static inline long mem_cgroup_calc_reclaim (struct mem_cgroup *mem, +struct zone *zone,int priority, +enum lru_list lru) {return 0 ; }
mm/memcontrol.c +43 -72
228 lines not on this page · raw message
Signature · 1 line
All Rights Reversed
PATCH 3/14 use an array for the LRU pagevecs
In reply to Rik van RielFrom: KOSAKI Motohiro <[email protected]>
Turn the pagevecs into an array just like the LRUs. This significantly cleans up the source code and reduces the size of the kernel by about 13kB after all the LRU lists have been created further down in the split VM patch series.
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by KOSAKI Motohiro <[email protected]>
--- include/linux/mmzone.h | 15 +++++- include/linux/pagevec.h | 13 ++++- include/linux/swap.h | 18 ++++++- mm/migrate.c | 11 ---- mm/swap.c | 87 +++++++++++++++----------------------- 5 files changed, 76 insertions(+), 68 deletions(-)
include/linux/mmzone.h +11 -2
@@ -107,13 +107,22 @@ enum zone_stat_item {#endif NR_VM_ZONE_STAT_ITEMS }; +#define LRU_BASE 0 +enum lru_list { -LRU_INACTIVE ,/* must match order of NR_[IN]ACTIVE */ -LRU_ACTIVE ,/* " " " " " */ +LRU_INACTIVE =LRU_BASE ,/* must match order of NR_[IN]ACTIVE */ +LRU_ACTIVE ,/* " " " " " */ NR_LRU_LISTS };#define for_each_lru (l) for (l = 0; l < NR_LRU_LISTS; l++) +static inline int is_active_lru (enum lru_list l) +{ +return (l ==LRU_ACTIVE ); +} + +enum lru_list page_lru (struct page *page); +struct per_cpu_pages {int count;/* number of pages in the list */ int high;/* high watermark, emptying needed */
mm/swap.c +44 -52
149 lines not on this page · raw message
include/linux/pagevec.h +11 -2
@@ -23,8 +23,7 @@ struct pagevec {void __pagevec_release (struct pagevec *pvec);void __pagevec_release_nonlru (struct pagevec *pvec);void __pagevec_free (struct pagevec *pvec); -void __pagevec_lru_add (struct pagevec *pvec); -void __pagevec_lru_add_active (struct pagevec *pvec); +void ____pagevec_lru_add (struct pagevec *pvec,enum lru_list lru);void pagevec_strip (struct pagevec *pvec);unsigned pagevec_lookup (struct pagevec *pvec,struct address_space *mapping,pgoff_t start,unsigned nr_pages);
@@ -81,6 +80,16 @@ static inline void pagevec_free(struct p__pagevec_free (pvec); } +static inline void __pagevec_lru_add (struct pagevec *pvec) +{ +____pagevec_lru_add (pvec,LRU_INACTIVE ); +} + +static inline void __pagevec_lru_add_active (struct pagevec *pvec) +{ +____pagevec_lru_add (pvec,LRU_ACTIVE ); +} +static inline void pagevec_lru_add (struct pagevec *pvec) {if (pagevec_count (pvec))
include/linux/swap.h +16 -2
@@ -171,8 +171,8 @@ extern unsigned int nr_free_pagecache_pa/* linux/mm/swap.c */ -extern void lru_cache_add (struct page *); -extern void lru_cache_add_active (struct page *); +extern void __lru_cache_add (struct page *,enum lru_list lru); +extern void lru_cache_add_lru (struct page *,enum lru_list lru);extern void activate_page (struct page *);extern void mark_page_accessed (struct page *);extern void lru_add_drain (void );
@@ -180,6 +180,20 @@ extern int lru_add_drain_all(void);extern void rotate_reclaimable_page (struct page *page);extern void swap_setup (void ); +/** +* lru_cache_add: add a page to the page lists +* @page: the page to add +*/ +static inline void lru_cache_add (struct page *page) +{ +__lru_cache_add (page,LRU_INACTIVE ); +} + +static inline void lru_cache_add_active (struct page *page) +{ +__lru_cache_add (page,LRU_ACTIVE ); +} +/* linux/mm/vmscan.c */ extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order,gfp_t gfp_mask);
mm/migrate.c +1 -10
@@ -55,16 +55,7 @@ int migrate_prep(void)static inline void move_to_lru (struct page *page) { -if (PageActive (page)) { -/* -* lru_cache_add_active checks that -* the PG_active bit is off. -*/ -ClearPageActive (page); -lru_cache_add_active (page); - }else { -lru_cache_add (page); - } +lru_cache_add_lru (page,page_lru (page));put_page (page); }
Signature · 1 line
All Rights Reversed
PATCH 4/14 free swap space on swap-in/activation
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Free swap cache entries when swapping in pages if vm_swap_full() [swap space > 1/2 used]. Uses new pagevec to reduce pressure on locks.
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by Lee Schermerhorn <[email protected]>
mm/vmscan.c +7 -0
@@ -619,6 +619,9 @@ free_it:continue ; activate_locked: +/* Not a candidate for swapping, so reclaim swap space. */ +if (PageSwapCache (page) &&vm_swap_full ()) +remove_exclusive_swap_page_ref (page);SetPageActive (page); pgactivate++; keep_locked:
@@ -1203,6 +1206,8 @@ static void shrink_active_list(unsigned__mod_zone_page_state (zone,NR_ACTIVE , pgmoved); pgmoved =0 ;spin_unlock_irq (&zone->lru_lock); +if (vm_swap_full ()) +pagevec_swap_free (&pvec);__pagevec_release (&pvec);spin_lock_irq (&zone->lru_lock); }
@@ -1212,6 +1217,8 @@ static void shrink_active_list(unsigned__count_zone_vm_events (PGREFILL , zone, pgscanned);__count_vm_events (PGDEACTIVATE , pgdeactivate);spin_unlock_irq (&zone->lru_lock); +if (vm_swap_full ()) +pagevec_swap_free (&pvec);pagevec_release (&pvec); }
mm/swap.c +18 -0
@@ -443,6 +443,24 @@ void pagevec_strip(struct pagevec *pvec) } } +/* +* Try to free swap space from the pages in a pagevec +*/ +void pagevec_swap_free (struct pagevec *pvec) +{ +int i; + +for (i =0 ; i <pagevec_count (pvec); i++) { +struct page *page = pvec->pages[i]; + +if (PageSwapCache (page) && !TestSetPageLocked (page)) { +if (PageSwapCache (page)) +remove_exclusive_swap_page_ref (page); +unlock_page (page); + } + } +} +/** * pagevec_lookup - gang pagecache lookup * @pvec: Where the resulting pages are placed
include/linux/pagevec.h +1 -0
@@ -25,6 +25,7 @@ void __pagevec_release_nonlru(struct pagvoid __pagevec_free (struct pagevec *pvec);void ____pagevec_lru_add (struct pagevec *pvec,enum lru_list lru);void pagevec_strip (struct pagevec *pvec); +void pagevec_swap_free (struct pagevec *pvec);unsigned pagevec_lookup (struct pagevec *pvec,struct address_space *mapping,pgoff_t start,unsigned nr_pages); unsigned pagevec_lookup_tag(struct pagevec *pvec,
include/linux/swap.h +1 -0
@@ -265,6 +265,7 @@ extern sector_t swapdev_block(int, pgoffextern struct swap_info_struct *get_swap_info_struct (unsigned );extern int can_share_swap_page (struct page *);extern int remove_exclusive_swap_page (struct page *); +extern int remove_exclusive_swap_page_ref (struct page *);struct backing_dev_info ;/* linux/mm/thrash.c */
mm/swapfile.c +22 -3
@@ -343,7 +343,7 @@ int can_share_swap_page(struct page *pag* Work out if there are any other processes sharing this * swap cache page. Free it if you can. Return success. */ -int remove_exclusive_swap_page (struct page *page) +static int remove_exclusive_swap_page_count (struct page *page,int count) {int retval;struct swap_info_struct * p;
@@ -356,7 +356,7 @@ int remove_exclusive_swap_page(struct pareturn 0 ;if (PageWriteback (page))return 0 ; -if (page_count (page) !=2 )/* 2: us + cache */ +if (page_count (page) != count)/* us + cache + ptes */ return 0 ; entry.val =page_private (page);
@@ -369,7 +369,7 @@ int remove_exclusive_swap_page(struct paif (p->swap_map[swp_offset (entry)] ==1 ) {/* Recheck the page count with the swapcache lock held.. */ write_lock_irq (&swapper_space.tree_lock); -if ((page_count (page) ==2 ) && !PageWriteback (page)) { +if ((page_count (page) == count) && !PageWriteback (page)) {__delete_from_swap_cache (page);SetPageDirty (page); retval =1 ;
@@ -387,6 +387,25 @@ int remove_exclusive_swap_page(struct pa }/* +* Most of the time the page should have two references: one for the +* process and one for the swap cache. +*/ +int remove_exclusive_swap_page (struct page *page) +{ +return remove_exclusive_swap_page_count (page,2 ); +} + +/* +* The pageout code holds an extra reference to the page. That raises +* the reference count to test for to 2 for a page that is only in the +* swap cache plus 1 for each process that maps the page. +*/ +int remove_exclusive_swap_page_ref (struct page *page) +{ +return remove_exclusive_swap_page_count (page,2 +page_mapcount (page)); +} + +/* * Free the swap entry like above, but also try to * free the page cache entry if it is the last user. */
Signature · 1 line
All Rights Reversed
Re: [patch 04/14] free swap space on swap-in/activation
Hi, Rik.
I don't know where to send this patch. you or mm ? I guess VM pageout patches didn't send mm-commits yet.
It made compile error in my system that didn't have swap device.
----
Subject: [PATCH] Fix compile error about remove_exclusive_swap_page_ref
It happened compiled error when I removed CONFIG_SWAP in config file It caused by new API that is 'remove_exclusive_swap_page_ref'.
Signed-off-by MinChan Kim <[email protected]>
--- include/linux/swap.h | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
include/linux/swap.h +5 -0
@@ -390,6 +390,11 @@ static inline int remove_exclusive_swap_page(struct page *p)return 0 ; } +static inline int remove_exclusive_swap_page_ref (struct page *page) +{ +return 0 ; +} +static inline swp_entry_t get_swap_page (void ) {swp_entry_t entry;
Signature · 1 line
1.5.4.3
Re: [patch 04/14] free swap space on swap-in/activation
Quoted 2 lines · On Mon, 19 May 2008 21:35:38 +0900 MinChan Kim <[email protected]> wrote:
I don't know where to send this patch. you or mm ? I guess VM pageout patches didn't send mm-commits yet.
If Andrew merges the patches, this could go in as an additional fix. If Andrew does not merge the patches, I will queue it up in my tree for the next submission :)
Quoted 1 line
It made compile error in my system that didn't have swap device.
Good catch. Thank you for this compile fix with CONFIG_SWAP=n
Signature · 1 line
All rights reversed.
PATCH 5/14 define page_file_cache() function
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Define page_file_cache() function to answer the question:
is page backed by a file?
Originally part of Rik van Riel's split-lru patch. Extracted to make available for other, independent reclaim patches.
Moved inline function to linux/mm_inline.h where it will be needed by subsequent "split LRU" and "noreclaim" patches.
Unfortunately this needs to use a page flag, since the PG_swapbacked state needs to be preserved all the way to the point where the page is last removed from the LRU. Trying to derive the status from other info in the page resulted in wrong VM statistics in earlier split VM patchsets.
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by MinChan Kim <[email protected]>
include/linux/mm_inline.h +24 -0
@@ -1,3 +1,26 @@ +#ifndef LINUX_MM_INLINE_H +#define LINUX_MM_INLINE_H + +/** +* page_file_cache - should the page be on a file LRU or anon LRU? +* @page: the page to test +* +* Returns !0 if @page is page cache page backed by a regular filesystem, +* or 0 if @page is anonymous, tmpfs or otherwise ram or swap backed. +* +* We would like to get this info without a page flag, but the state +* needs to survive until the page is last deleted from the LRU, which +* could be as far down as __page_cache_release. +*/ +static inline int page_file_cache (struct page *page) +{ +if (PageSwapBacked (page)) +return 0 ; + +/* The page is page cache backed by a normal filesystem. */ +return 2 ; +} +static inline void add_page_to_lru_list (struct zone *zone,struct page *page,enum lru_list l) {
@@ -49,3 +72,4 @@ del_page_from_lru(struct zone *zone, str __dec_zone_state(zone, NR_INACTIVE + l); } +#endif
mm/shmem.c +1 -0
@@ -1378,6 +1378,7 @@ repeat: goto failed; } +SetPageSwapBacked (filepage);spin_lock (&info->lock); entry =shmem_swp_alloc (info, idx, sgp);if (IS_ERR (entry))
include/linux/page-flags.h +2 -0
@@ -93,6 +93,7 @@ enum pageflags { PG_mappedtodisk,/* Has blocks allocated on-disk */ PG_reclaim,/* To be reclaimed asap */ PG_buddy,/* Page is free, on buddy lists */ + PG_swapbacked,/* Page is backed by RAM/swap */ #ifdef CONFIG_IA64_UNCACHED_ALLOCATOR PG_uncached, /* Page has been mapped as uncached */ #endif
@@ -160,6 +161,7 @@ PAGEFLAG(Pinned, owner_priv_1) TESTSCFLA PAGEFLAG(Reserved, reserved) __CLEARPAGEFLAG(Reserved, reserved) PAGEFLAG(Private, private) __CLEARPAGEFLAG(Private, private) __SETPAGEFLAG(Private, private) +PAGEFLAG(SwapBacked, swapbacked) __CLEARPAGEFLAG(SwapBacked, swapbacked)/* * Only test-and-set exist for PG_writeback. The unconditional operators are
mm/memory.c +3 -0
@@ -1765,6 +1765,7 @@ gotten:ptep_clear_flush (vma, address, page_table);set_pte_at (mm, address, page_table, entry);update_mmu_cache (vma, address, entry); +SetPageSwapBacked (new_page);lru_cache_add_active (new_page);page_add_new_anon_rmap (new_page, vma, address);
@@ -2233,6 +2234,7 @@ static int do_anonymous_page(struct mm_sif (!pte_none (*page_table)) goto release;inc_mm_counter (mm, anon_rss); +SetPageSwapBacked (page);lru_cache_add_active (page);page_add_new_anon_rmap (page, vma, address);set_pte_at (mm, address, page_table, entry);
@@ -2374,6 +2376,7 @@ static int __do_fault(struct mm_struct *set_pte_at (mm, address, page_table, entry);if (anon) {inc_mm_counter (mm, anon_rss); +SetPageSwapBacked (page);lru_cache_add_active (page);page_add_new_anon_rmap (page, vma, address); }else {
mm/swap_state.c +3 -0
@@ -74,6 +74,7 @@ int add_to_swap_cache(struct page *page,BUG_ON (!PageLocked (page));BUG_ON (PageSwapCache (page));BUG_ON (PagePrivate (page)); +BUG_ON (!PageSwapBacked (page)); error =radix_tree_preload (gfp_mask);if (!error) {write_lock_irq (&swapper_space.tree_lock);
@@ -295,6 +296,7 @@ struct page *read_swap_cache_async(swp_e* May fail (-ENOMEM) if radix-tree node allocation failed. */ SetPageLocked (new_page); +SetPageSwapBacked (new_page); err =add_to_swap_cache (new_page, entry, gfp_mask &GFP_KERNEL );if (!err) {/*
@@ -304,6 +306,7 @@ struct page *read_swap_cache_async(swp_eswap_readpage (NULL , new_page);return new_page; } +ClearPageSwapBacked (new_page);ClearPageLocked (new_page);swap_free (entry); }while (err != -ENOMEM );
mm/migrate.c +2 -0
@@ -558,6 +558,8 @@ static int move_to_new_page(struct page/* Prepare mapping for the new page.*/ newpage->index = page->index; newpage->mapping = page->mapping; +if (PageSwapBacked (page)) +SetPageSwapBacked (newpage); mapping =page_mapping (page);if (!mapping)
mm/page_alloc.c +4 -0
@@ -261,6 +261,7 @@ static void bad_page(struct page *page)1 << PG_slab |1 << PG_swapcache |1 << PG_writeback | +1 << PG_swapbacked | 1 << PG_buddy );set_page_count (page,0 );reset_page_mapcount (page);
@@ -494,6 +495,8 @@ static inline int free_pages_check(strucbad_page (page);if (PageDirty (page))__ClearPageDirty (page); +if (PageSwapBacked (page)) +__ClearPageSwapBacked (page);/* * For now, we report if PG_reserved was found set, but do not * clear it, and do not free the page. But we shall soon need
@@ -644,6 +647,7 @@ static int prep_new_page(struct page *pa 1 << PG_swapcache | 1 << PG_writeback | 1 << PG_reserved | + 1 << PG_swapbacked | 1 << PG_buddy ))))bad_page (page);
Signature · 1 line
All Rights Reversed
PATCH 6/14 split LRU lists into anon & file sets
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Split the LRU lists in two, one set for pages that are backed by real file systems ("file") and one for pages that are backed by memory and swap ("anon"). The latter includes tmpfs.
Eventually mlocked pages will be taken off the LRUs alltogether. A patch for that already exists and just needs to be integrated into this series.
This patch mostly has the infrastructure and a basic policy to balance how much we scan the anon lists and how much we scan the file lists. The big policy changes are in separate patches.
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by Lee Schermerhorn <[email protected]>
fs/proc/proc_misc.c +49 -32
@@ -132,6 +132,10 @@ static int meminfo_read_proc(char *page,unsigned long allowed;struct vmalloc_info vmi;long cached; +unsigned long inactive_anon; +unsigned long active_anon; +unsigned long inactive_file; +unsigned long active_file;/* * display in kilobytes.
@@ -150,48 +154,61 @@ static int meminfo_read_proc(char *page,get_vmalloc_info (&vmi); + inactive_anon =global_page_state (NR_INACTIVE_ANON ); + active_anon =global_page_state (NR_ACTIVE_ANON ); + inactive_file =global_page_state (NR_INACTIVE_FILE ); + active_file =global_page_state (NR_ACTIVE_FILE ); +/* * Tagged format, for easy grepping and expansion. */ len = sprintf(page, - "MemTotal: %8lu kB\n" - "MemFree: %8lu kB\n" - "Buffers: %8lu kB\n" - "Cached: %8lu kB\n" - "SwapCached: %8lu kB\n" - "Active: %8lu kB\n" - "Inactive: %8lu kB\n" + "MemTotal: %8lu kB\n" + "MemFree: %8lu kB\n" + "Buffers: %8lu kB\n" + "Cached: %8lu kB\n" + "SwapCached: %8lu kB\n" + "Active: %8lu kB\n" + "Inactive: %8lu kB\n" + "Active(anon): %8lu kB\n" + "Inactive(anon): %8lu kB\n" + "Active(file): %8lu kB\n" + "Inactive(file): %8lu kB\n" #ifdef CONFIG_HIGHMEM - "HighTotal: %8lu kB\n" - "HighFree: %8lu kB\n" - "LowTotal: %8lu kB\n" - "LowFree: %8lu kB\n" -#endif - "SwapTotal: %8lu kB\n" - "SwapFree: %8lu kB\n" - "Dirty: %8lu kB\n" - "Writeback: %8lu kB\n" - "AnonPages: %8lu kB\n" - "Mapped: %8lu kB\n" - "Slab: %8lu kB\n" - "SReclaimable: %8lu kB\n" - "SUnreclaim: %8lu kB\n" - "PageTables: %8lu kB\n" - "NFS_Unstable: %8lu kB\n" - "Bounce: %8lu kB\n" - "WritebackTmp: %8lu kB\n" - "CommitLimit: %8lu kB\n" - "Committed_AS: %8lu kB\n" - "VmallocTotal: %8lu kB\n" - "VmallocUsed: %8lu kB\n" - "VmallocChunk: %8lu kB\n", + "HighTotal: %8lu kB\n" + "HighFree: %8lu kB\n" + "LowTotal: %8lu kB\n" + "LowFree: %8lu kB\n" +#endif + "SwapTotal: %8lu kB\n" + "SwapFree: %8lu kB\n" + "Dirty: %8lu kB\n" + "Writeback: %8lu kB\n" + "AnonPages: %8lu kB\n" + "Mapped: %8lu kB\n" + "Slab: %8lu kB\n" + "SReclaimable: %8lu kB\n" + "SUnreclaim: %8lu kB\n" + "PageTables: %8lu kB\n" + "NFS_Unstable: %8lu kB\n" + "Bounce: %8lu kB\n" + "WritebackTmp: %8lu kB\n" + "CommitLimit: %8lu kB\n" + "Committed_AS: %8lu kB\n" + "VmallocTotal: %8lu kB\n" + "VmallocUsed: %8lu kB\n" + "VmallocChunk: %8lu kB\n", K(i.totalram), K(i.freeram), K(i.bufferram), K(cached), K(total_swapcache_pages), - K(global_page_state(NR_ACTIVE)), - K(global_page_state(NR_INACTIVE)), + K(active_anon + active_file), + K(inactive_anon + inactive_file), + K(active_anon), + K(inactive_anon), + K(active_file), + K(inactive_file), #ifdef CONFIG_HIGHMEM K(i.totalhigh), K(i.freehigh),
fs/cifs/file.c +2 -2
@@ -1778,7 +1778,7 @@ static void cifs_copy_cache_pages(structSetPageUptodate (page);unlock_page (page);if (!pagevec_add (plru_pvec, page)) -__pagevec_lru_add (plru_pvec); +__pagevec_lru_add_file (plru_pvec); data +=PAGE_CACHE_SIZE ; }return ;
@@ -1912,7 +1912,7 @@ static int cifs_readpages(struct file *f bytes_read =0 ; } -pagevec_lru_add (&lru_pvec); +pagevec_lru_add_file (&lru_pvec);/* need to free smb_read_data buf before exit */ if (smb_read_data) {
fs/ntfs/file.c +2 -2
@@ -439,7 +439,7 @@ static inline int __ntfs_grab_cache_page pages[nr] = *cached_page;page_cache_get (*cached_page);if (unlikely (!pagevec_add (lru_pvec, *cached_page))) -__pagevec_lru_add (lru_pvec); +__pagevec_lru_add_file (lru_pvec); *cached_page =NULL ; } index++;
@@ -2084,7 +2084,7 @@ err_out: OSYNC_METADATA|OSYNC_DATA); } } -pagevec_lru_add (&lru_pvec); +pagevec_lru_add_file (&lru_pvec);ntfs_debug ("Done. Returning %s (written 0x%lx, status %li)." , written ?"written" :"status" , (unsigned long )written, (long )status);
fs/nfs/dir.c +1 -1
@@ -1524,7 +1524,7 @@ static int nfs_symlink(struct inode *dirif (!add_to_page_cache (page, dentry->d_inode->i_mapping,0 ,GFP_KERNEL )) {pagevec_add (&lru_pvec, page); -pagevec_lru_add (&lru_pvec); +pagevec_lru_add_file (&lru_pvec);SetPageUptodate (page);unlock_page (page); }else
fs/ramfs/file-nommu.c +2 -2
@@ -111,12 +111,12 @@ static int ramfs_nommu_expand_for_mappin goto add_error;if (!pagevec_add (&lru_pvec, page)) -__pagevec_lru_add (&lru_pvec); +__pagevec_lru_add_file (&lru_pvec);unlock_page (page); } -pagevec_lru_add (&lru_pvec); +pagevec_lru_add_file (&lru_pvec);return 0 ; fsize_exceeded:
drivers/base/node.c +33 -23
67 lines not on this page · raw message
mm/memory.c +3 -3
@@ -1766,7 +1766,7 @@ gotten:set_pte_at (mm, address, page_table, entry);update_mmu_cache (vma, address, entry);SetPageSwapBacked (new_page); -lru_cache_add_active (new_page); +lru_cache_add_active_anon (new_page);page_add_new_anon_rmap (new_page, vma, address);/* Free the old page.. */
@@ -2235,7 +2235,7 @@ static int do_anonymous_page(struct mm_s goto release;inc_mm_counter (mm, anon_rss);SetPageSwapBacked (page); -lru_cache_add_active (page); +lru_cache_add_active_anon (page);page_add_new_anon_rmap (page, vma, address);set_pte_at (mm, address, page_table, entry);
@@ -2377,7 +2377,7 @@ static int __do_fault(struct mm_struct *if (anon) {inc_mm_counter (mm, anon_rss);SetPageSwapBacked (page); -lru_cache_add_active (page); +lru_cache_add_active_anon (page);page_add_new_anon_rmap (page, vma, address); }else {inc_mm_counter (mm, file_rss);
mm/page_alloc.c +17 -7
@@ -1908,10 +1908,13 @@ void show_free_areas(void) } } - printk("Active:%lu inactive:%lu dirty:%lu writeback:%lu unstable:%lu\n" + printk("Active_anon:%lu active_file:%lu inactive_anon%lu\n" + " inactive_file:%lu dirty:%lu writeback:%lu unstable:%lu\n" " free:%lu slab:%lu mapped:%lu pagetables:%lu bounce:%lu\n", - global_page_state(NR_ACTIVE), - global_page_state(NR_INACTIVE), + global_page_state(NR_ACTIVE_ANON), + global_page_state(NR_ACTIVE_FILE), + global_page_state(NR_INACTIVE_ANON), + global_page_state(NR_INACTIVE_FILE), global_page_state(NR_FILE_DIRTY), global_page_state(NR_WRITEBACK), global_page_state(NR_UNSTABLE_NFS),
@@ -1934,8 +1937,10 @@ void show_free_areas(void)" min:%lukB" " low:%lukB" " high:%lukB" -" active:%lukB" -" inactive:%lukB" +" active_anon:%lukB" +" inactive_anon:%lukB" +" active_file:%lukB" +" inactive_file:%lukB" " present:%lukB" " pages_scanned:%lu" " all_unreclaimable? %s"
@@ -1945,8 +1950,10 @@ void show_free_areas(void)K (zone->pages_min),K (zone->pages_low),K (zone->pages_high), -K (zone_page_state (zone,NR_ACTIVE )), -K (zone_page_state (zone,NR_INACTIVE )), +K (zone_page_state (zone,NR_ACTIVE_ANON )), +K (zone_page_state (zone,NR_INACTIVE_ANON )), +K (zone_page_state (zone,NR_ACTIVE_FILE )), +K (zone_page_state (zone,NR_INACTIVE_FILE )),K (zone->present_pages), zone->pages_scanned, (zone_is_all_unreclaimable (zone) ?"yes" :"no" )
@@ -3503,6 +3510,9 @@ static void __paginginit free_area_init_INIT_LIST_HEAD (&zone->list[l]); zone->nr_scan[l] =0 ; } + zone->recent_rotated_anon =0 ; + zone->recent_rotated_file =0 ; +//TODO recent_scanned_* ??? zap_zone_vm_stats (zone); zone->flags =0 ;if (!size)
mm/swap.c +9 -3
@@ -108,6 +108,7 @@ enum lru_list page_lru(struct page *pageif (PageActive (page)) lru +=LRU_ACTIVE ; + lru +=page_file_cache (page);return lru; }
@@ -133,7 +134,8 @@ static void pagevec_move_tail(struct pagspin_lock (&zone->lru_lock); }if (PageLRU (page) && !PageActive (page)) { -list_move_tail (&page->lru, &zone->list[LRU_INACTIVE ]); +int lru =page_file_cache (page); +list_move_tail (&page->lru, &zone->list[lru]); pgmoved++; } }
@@ -174,9 +176,13 @@ void activate_page(struct page *page)spin_lock_irq (&zone->lru_lock);if (PageLRU (page) && !PageActive (page)) { -del_page_from_inactive_list (zone, page); +int lru =LRU_BASE ; + lru +=page_file_cache (page); +del_page_from_lru_list (zone, page, lru); +SetPageActive (page); -add_page_to_active_list (zone, page); + lru +=LRU_ACTIVE ; +add_page_to_lru_list (zone, page, lru);__count_vm_event (PGACTIVATE );mem_cgroup_move_lists (page, true); }
mm/readahead.c +1 -1
@@ -229,7 +229,7 @@ int do_page_cache_readahead(struct addre*/ unsigned long max_sane_readahead (unsigned long nr) { -return min (nr, (node_page_state (numa_node_id (),NR_INACTIVE ) +return min (nr, (node_page_state (numa_node_id (),NR_INACTIVE_FILE ) +node_page_state (numa_node_id (),NR_FREE_PAGES )) /2 ); }
mm/filemap.c +7 -2
@@ -33,6 +33,7 @@#include <linux/cpuset.h> #include <linux/hardirq.h> /* for BUG_ON(!in_atomic()) only */ #include <linux/memcontrol.h> +#include <linux/mm_inline.h> /* for page_file_cache() */ #include "internal.h" /*
@@ -489,8 +490,12 @@ int add_to_page_cache_lru(struct page *p pgoff_t offset, gfp_t gfp_mask) { int ret = add_to_page_cache(page, mapping, offset, gfp_mask); -if (ret ==0 ) -lru_cache_add (page); +if (ret ==0 ) { +if (page_file_cache (page)) +lru_cache_add_file (page); +else +lru_cache_add_active_anon (page); + }return ret; }
mm/vmstat.c +9 -5
@@ -695,8 +695,10 @@ const struct seq_operations pagetypeinfostatic const char *const vmstat_text[] = {/* Zoned VM counters */ "nr_free_pages" , -"nr_inactive" , -"nr_active" , +"nr_inactive_anon" , +"nr_active_anon" , +"nr_inactive_file" , +"nr_active_file" ,"nr_anon_pages" ,"nr_mapped" ,"nr_file_pages" ,
@@ -764,7 +766,7 @@ static void zoneinfo_show_print(struct s"\n min %lu" "\n low %lu" "\n high %lu" -"\n scanned %lu (a: %lu i: %lu)" +"\n scanned %lu (aa: %lu ia: %lu af: %lu if: %lu)" "\n spanned %lu" "\n present %lu" , zone_page_state(zone, NR_FREE_PAGES),
@@ -772,8 +774,10 @@ static void zoneinfo_show_print(struct s zone->pages_low, zone->pages_high, zone->pages_scanned, - zone->nr_scan[LRU_ACTIVE ], - zone->nr_scan[LRU_INACTIVE ], + zone->nr_scan[LRU_ACTIVE_ANON ], + zone->nr_scan[LRU_INACTIVE_ANON ], + zone->nr_scan[LRU_ACTIVE_FILE ], + zone->nr_scan[LRU_INACTIVE_FILE ], zone->spanned_pages, zone->present_pages);
mm/vmscan.c +199 -229
730 lines not on this page · raw message
mm/swap_state.c +1 -1
@@ -301,7 +301,7 @@ struct page *read_swap_cache_async(swp_e/* * Initiate read into locked page and return. */ -lru_cache_add_active (new_page); +lru_cache_add_active_anon (new_page);swap_readpage (NULL , new_page);return new_page; }
include/linux/mmzone.h +27 -6
@@ -81,21 +81,23 @@ struct zone_padding {enum zone_stat_item {/* First 128 byte cacheline (assuming 64 bit words) */ NR_FREE_PAGES , -NR_INACTIVE ,/* must match order of LRU_[IN]ACTIVE */ -NR_ACTIVE ,/* " " " " " */ +NR_INACTIVE_ANON ,/* must match order of LRU_[IN]ACTIVE_* */ +NR_ACTIVE_ANON ,/* " " " " " */ +NR_INACTIVE_FILE ,/* " " " " " */ +NR_ACTIVE_FILE ,/* " " " " " */ NR_ANON_PAGES ,/* Mapped anonymous pages */ NR_FILE_MAPPED ,/* pagecache pages mapped into pagetables. only modified from process context */ NR_FILE_PAGES ,NR_FILE_DIRTY ,NR_WRITEBACK , -/* Second 128 byte cacheline */ NR_SLAB_RECLAIMABLE ,NR_SLAB_UNRECLAIMABLE ,NR_PAGETABLE ,/* used for pagetables */ NR_UNSTABLE_NFS ,/* NFS unstable pages */ NR_BOUNCE ,NR_VMSCAN_WRITE , +/* Second 128 byte cacheline */ NR_WRITEBACK_TEMP ,/* Writeback using temporary buffers */ #ifdef CONFIG_NUMA NUMA_HIT ,/* allocated in intended node */
@@ -107,18 +109,33 @@ enum zone_stat_item {#endif NR_VM_ZONE_STAT_ITEMS }; +/* +* We do arithmetic on the LRU lists in various places in the code, +* so it is important to keep the active lists LRU_ACTIVE higher in +* the array than the corresponding inactive lists, and to keep +* the *_FILE lists LRU_FILE higher than the corresponding _ANON lists. +*/ #define LRU_BASE 0 +#define LRU_ACTIVE 1 +#define LRU_FILE 2enum lru_list { -LRU_INACTIVE =LRU_BASE ,/* must match order of NR_[IN]ACTIVE */ -LRU_ACTIVE ,/* " " " " " */ +LRU_INACTIVE_ANON =LRU_BASE , +LRU_ACTIVE_ANON =LRU_BASE +LRU_ACTIVE , +LRU_INACTIVE_FILE =LRU_BASE +LRU_FILE , +LRU_ACTIVE_FILE =LRU_BASE +LRU_FILE +LRU_ACTIVE ,NR_LRU_LISTS };#define for_each_lru (l) for (l = 0; l < NR_LRU_LISTS; l++) +static inline int is_file_lru (enum lru_list l) +{ +return (l ==LRU_INACTIVE_FILE || l ==LRU_ACTIVE_FILE ); +} +static inline int is_active_lru (enum lru_list l) { -return (l ==LRU_ACTIVE ); +return (l ==LRU_ACTIVE_ANON || l ==LRU_ACTIVE_FILE ); }enum lru_list page_lru (struct page *page);
@@ -269,6 +286,10 @@ struct zone {spinlock_t lru_lock;struct list_head list[NR_LRU_LISTS ];unsigned long nr_scan[NR_LRU_LISTS ]; + +unsigned long recent_rotated_anon; +unsigned long recent_rotated_file; +unsigned long pages_scanned;/* since last reclaim */ unsigned long flags;/* zone flags, see below */
include/linux/mm_inline.h +40 -15
105 lines not on this page · raw message
include/linux/pagevec.h +23 -6
@@ -81,20 +81,37 @@ static inline void pagevec_free(struct p__pagevec_free (pvec); } -static inline void __pagevec_lru_add (struct pagevec *pvec) +static inline void __pagevec_lru_add_anon (struct pagevec *pvec) { -____pagevec_lru_add (pvec,LRU_INACTIVE ); +____pagevec_lru_add (pvec,LRU_INACTIVE_ANON ); } -static inline void __pagevec_lru_add_active (struct pagevec *pvec) +static inline void __pagevec_lru_add_active_anon (struct pagevec *pvec) { -____pagevec_lru_add (pvec,LRU_ACTIVE ); +____pagevec_lru_add (pvec,LRU_ACTIVE_ANON ); } -static inline void pagevec_lru_add (struct pagevec *pvec) +static inline void __pagevec_lru_add_file (struct pagevec *pvec) +{ +____pagevec_lru_add (pvec,LRU_INACTIVE_FILE ); +} + +static inline void __pagevec_lru_add_active_file (struct pagevec *pvec) +{ +____pagevec_lru_add (pvec,LRU_ACTIVE_FILE ); +} + + +static inline void pagevec_lru_add_file (struct pagevec *pvec) +{ +if (pagevec_count (pvec)) +__pagevec_lru_add_file (pvec); +} + +static inline void pagevec_lru_add_anon (struct pagevec *pvec) {if (pagevec_count (pvec)) -__pagevec_lru_add (pvec); +__pagevec_lru_add_anon (pvec); }#endif /* _LINUX_PAGEVEC_H */
include/linux/vmstat.h +10 -0
@@ -159,6 +159,16 @@ static inline unsigned long zone_page_streturn x; } +extern unsigned long global_lru_pages (void ); + +static inline unsigned long zone_lru_pages (struct zone *zone) +{ +return (zone_page_state (zone,NR_ACTIVE_ANON ) + +zone_page_state (zone,NR_ACTIVE_FILE ) + +zone_page_state (zone,NR_INACTIVE_ANON ) + +zone_page_state (zone,NR_INACTIVE_FILE )); +} +#ifdef CONFIG_NUMA /* * Determine the per node value of a stat item. This function
mm/page-writeback.c +2 -6
@@ -331,9 +331,7 @@ static unsigned long highmem_dirtyable_mstruct zone *z = &NODE_DATA (node)->node_zones[ZONE_HIGHMEM ]; - x +=zone_page_state (z,NR_FREE_PAGES ) - +zone_page_state (z,NR_INACTIVE ) - +zone_page_state (z,NR_ACTIVE ); + x +=zone_page_state (z,NR_FREE_PAGES ) +zone_lru_pages (z); }/* * Make sure that the number of highmem pages is never larger
@@ -351,9 +349,7 @@ static unsigned long determine_dirtyable {unsigned long x; - x =global_page_state (NR_FREE_PAGES ) - +global_page_state (NR_INACTIVE ) - +global_page_state (NR_ACTIVE ); + x =global_page_state (NR_FREE_PAGES ) +global_lru_pages ();if (!vm_highmem_is_dirtyable) x -=highmem_dirtyable_memory (x);
include/linux/swap.h +15 -5
@@ -184,14 +184,24 @@ extern void swap_setup(void);* lru_cache_add: add a page to the page lists * @page: the page to add */ -static inline void lru_cache_add (struct page *page) +static inline void lru_cache_add_anon (struct page *page) { -__lru_cache_add (page,LRU_INACTIVE ); +__lru_cache_add (page,LRU_INACTIVE_ANON ); } -static inline void lru_cache_add_active (struct page *page) +static inline void lru_cache_add_active_anon (struct page *page) { -__lru_cache_add (page,LRU_ACTIVE ); +__lru_cache_add (page,LRU_ACTIVE_ANON ); +} + +static inline void lru_cache_add_file (struct page *page) +{ +__lru_cache_add (page,LRU_INACTIVE_FILE ); +} + +static inline void lru_cache_add_active_file (struct page *page) +{ +__lru_cache_add (page,LRU_ACTIVE_FILE ); }/* linux/mm/vmscan.c */
@@ -199,7 +209,7 @@ extern unsigned long try_to_free_pages(s gfp_t gfp_mask);extern unsigned long try_to_free_mem_cgroup_pages (struct mem_cgroup *mem,gfp_t gfp_mask); -extern int __isolate_lru_page (struct page *page,int mode); +extern int __isolate_lru_page (struct page *page,int mode,int file);extern unsigned long shrink_all_memory (unsigned long nr_pages);extern int vm_swappiness;extern int remove_mapping (struct address_space *mapping,struct page *page);
include/linux/memcontrol.h +1 -1
@@ -41,7 +41,7 @@ extern unsigned long mem_cgroup_isolate_ unsigned long *scanned, int order, int mode, struct zone *z, struct mem_cgroup *mem_cont, - int active); + int active, int file);extern void mem_cgroup_out_of_memory (struct mem_cgroup *mem,gfp_t gfp_mask);int task_in_mem_cgroup (struct task_struct *task,const struct mem_cgroup *mem);
mm/memcontrol.c +38 -35
150 lines not on this page · raw message
Signature · 1 line
All Rights Reversed
PATCH 7/14 second chance replacement for anonymous pages
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
We avoid evicting and scanning anonymous pages for the most part, but under some workloads we can end up with most of memory filled with anonymous pages. At that point, we suddenly need to clear the referenced bits on all of memory, which can take ages on very large memory systems.
We can reduce the maximum number of pages that need to be scanned by not taking the referenced state into account when deactivating an anonymous page. After all, every anonymous page starts out referenced, so why check?
If an anonymous page gets referenced again before it reaches the end of the inactive list, we move it back to the active list.
To keep the maximum amount of necessary work reasonable, we scale the active to inactive ratio with the size of memory, using the formula active:inactive ratio = sqrt(memory in GB * 10).
Kswapd CPU use now seems to scale by the amount of pageout bandwidth, instead of by the amount of memory present in the system.
Signed-off-by Rik van Riel <[email protected]>
Reviewed-by KOSAKI Motohiro <[email protected]>
include/linux/mm_inline.h +12 -0
@@ -97,4 +97,16 @@ del_page_from_lru(struct zone *zone, str __dec_zone_state(zone, NR_INACTIVE_ANON + l); } +static inline int inactive_anon_low (struct zone *zone) +{ +unsigned long active, inactive; + + active =zone_page_state (zone,NR_ACTIVE_ANON ); + inactive =zone_page_state (zone,NR_INACTIVE_ANON ); + +if (inactive * zone->inactive_ratio < active) +return 1 ; + +return 0 ; +}#endif
include/linux/mmzone.h +5 -0
@@ -311,6 +311,11 @@ struct zone {*/ int prev_priority; +/* +* The ratio of active to inactive pages. +*/ +unsigned int inactive_ratio; +ZONE_PADDING (_pad2_)/* Rarely used or read-mostly fields */
mm/page_alloc.c +40 -0
@@ -4269,6 +4269,45 @@ void setup_per_zone_pages_min(void)calculate_totalreserve_pages (); } +/** +* setup_per_zone_inactive_ratio - called when min_free_kbytes changes. +* +* The inactive anon list should be small enough that the VM never has to +* do too much work, but large enough that each inactive page has a chance +* to be referenced again before it is swapped out. +* +* The inactive_anon ratio is the ratio of active to inactive anonymous +* pages. Ie. a ratio of 3 means 3:1 or 25% of the anonymous pages are +* on the inactive list. +* +* total return max +* memory value inactive anon +* ------------------------------------- +* 10MB 1 5MB +* 100MB 1 50MB +* 1GB 3 250MB +* 10GB 10 0.9GB +* 100GB 31 3GB +* 1TB 101 10GB +* 10TB 320 32GB +*/ +void setup_per_zone_inactive_ratio (void ) +{ +struct zone *zone; + +for_each_zone (zone) { +unsigned int gb, ratio; + +/* Zone size in gigabytes */ + gb = zone->present_pages >> (30 -PAGE_SHIFT ); + ratio =int_sqrt (10 * gb); +if (!ratio) + ratio =1 ; + + zone->inactive_ratio = ratio; + } +} +/* * Initialise min_free_kbytes. *
@@ -4306,6 +4345,7 @@ static int __init init_per_zone_pages_mi min_free_kbytes =65536 ;setup_per_zone_pages_min ();setup_per_zone_lowmem_reserve (); +setup_per_zone_inactive_ratio ();return 0 ; }module_init (init_per_zone_pages_min)
mm/vmscan.c +31 -7
@@ -114,7 +114,7 @@ struct scan_control {/* * From 0 .. 100. Higher means more swappy. */ -int vm_swappiness =60 ; +int vm_swappiness =20 ;long vm_total_pages;/* The total number of pages which the VM controls */ static LIST_HEAD (shrinker_list);
@@ -1008,7 +1008,7 @@ static inline int zone_is_near_oom(strucstatic void shrink_active_list (unsigned long nr_pages,struct zone *zone,struct scan_control *sc,int priority,int file) { -unsigned long pgmoved; +unsigned long pgmoved =0 ;int pgdeactivate =0 ;unsigned long pgscanned;LIST_HEAD (l_hold);/* The pages which were snipped off */
@@ -1040,13 +1040,25 @@ static void shrink_active_list(unsignedcond_resched (); page =lru_to_page (&l_hold);list_del (&page->lru); -if (page_referenced (page,0 , sc->mem_cgroup)) +if (page_referenced (page,0 , sc->mem_cgroup) && file) { +/* Referenced file pages stay active. */ list_add (&page->lru, &l_active); -else + }else {list_add (&page->lru, &l_inactive); +if (!file) +/* Anonymous pages always get deactivated. */ + pgmoved++; + } }/* +* Count the referenced anon pages as rotated, to balance pageout +* scan pressure between file and anonymous pages in get_sacn_ratio. +*/ +if (!file) + zone->recent_rotated_anon += pgmoved; + +/* * Now put the pages back on the appropriate [file or anon] inactive * and active lists. */
@@ -1129,7 +1141,11 @@ static unsigned long shrink_list(enum lr {int file =is_file_lru (lru); -if (lru ==LRU_ACTIVE_ANON || lru ==LRU_ACTIVE_FILE ) { +if (lru ==LRU_ACTIVE_FILE ) { +shrink_active_list (nr_to_scan, zone, sc, priority, file); +return 0 ; + } +if (lru ==LRU_ACTIVE_ANON &&inactive_anon_low (zone)) {shrink_active_list (nr_to_scan, zone, sc, priority, file);return 0 ; }
@@ -1239,8 +1255,8 @@ static unsigned long shrink_zone(int pri } } -while (nr[LRU_ACTIVE_ANON ] || nr[LRU_INACTIVE_ANON ] || - nr[LRU_ACTIVE_FILE ] || nr[LRU_INACTIVE_FILE ]) { +while (nr[LRU_INACTIVE_ANON ] || nr[LRU_ACTIVE_FILE ] || + nr[LRU_INACTIVE_FILE ]) {for_each_lru (l) {if (nr[l]) { nr_to_scan = min(nr[l],
@@ -1542,6 +1558,14 @@ loop_again: priority != DEF_PRIORITY)continue ; +/* +* Do some background aging of the anon list, to give +* pages a chance to be referenced before reclaiming. +*/ +if (inactive_anon_low (zone)) +shrink_active_list (SWAP_CLUSTER_MAX , zone, + &sc, priority,0 ); +if (!zone_watermark_ok (zone, order, zone->pages_high,0 ,0 )) { end_zone = i;
mm/vmstat.c +4 -2
@@ -814,10 +814,12 @@ static void zoneinfo_show_print(struct sseq_printf (m,"\n all_unreclaimable: %u" "\n prev_priority: %i" -"\n start_pfn: %lu" , +"\n start_pfn: %lu" +"\n inactive_ratio: %u" ,zone_is_all_unreclaimable (zone), zone->prev_priority, - zone->zone_start_pfn); + zone->zone_start_pfn, + zone->inactive_ratio);seq_putc (m,'\n' ); }
Signature · 1 line
All Rights Reversed
PATCH 8/14 add some sanity checks to get_scan_ratio
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
The access ratio based scan rate determination in get_scan_ratio works ok in most situations, but needs to be corrected in some corner cases:
- if we run out of swap space, do not bother scanning the anon LRUs
- if we have already freed all of the page cache, we need to scan the anon LRUs
- restore the *actual* access ratio based scan rate algorithm, the previous versions of this patch series had the wrong version
- scale the number of pages added to zone->nr_scan[l]
Signed-off-by Rik van Riel <[email protected]>
Signed-off-by MinChan Kim <[email protected]>
mm/vmscan.c +70 -34
@@ -893,8 +893,13 @@ static unsigned long shrink_inactive_lis__mod_zone_page_state (zone,NR_INACTIVE_ANON , -count[LRU_INACTIVE_ANON ]); -if (scan_global_lru (sc)) +if (scan_global_lru (sc)) { zone->pages_scanned += nr_scan; + zone->recent_scanned_anon += count[LRU_ACTIVE_ANON ] + + count[LRU_INACTIVE_ANON ]; + zone->recent_scanned_file += count[LRU_ACTIVE_FILE ] + + count[LRU_INACTIVE_FILE ]; + }spin_unlock_irq (&zone->lru_lock); nr_scanned += nr_scan;
@@ -944,11 +949,13 @@ static unsigned long shrink_inactive_lisVM_BUG_ON (PageLRU (page));SetPageLRU (page);list_del (&page->lru); -if (page_file_cache (page)) { +if (page_file_cache (page)) lru +=LRU_FILE ; - zone->recent_rotated_file++; - }else { - zone->recent_rotated_anon++; +if (scan_global_lru (sc)) { +if (page_file_cache (page)) + zone->recent_rotated_file++; +else + zone->recent_rotated_anon++; }if (PageActive (page)) lru +=LRU_ACTIVE ;
@@ -1027,8 +1034,13 @@ static void shrink_active_list(unsigned* zone->pages_scanned is used for detect zone's oom * mem_cgroup remembers nr_scan by itself. */ -if (scan_global_lru (sc)) +if (scan_global_lru (sc)) { zone->pages_scanned += pgscanned; +if (file) + zone->recent_scanned_file += pgscanned; +else + zone->recent_scanned_anon += pgscanned; + }if (file)__mod_zone_page_state (zone,NR_ACTIVE_FILE , -pgmoved);
@@ -1165,9 +1177,8 @@ static unsigned long shrink_list(enum lrstatic void get_scan_ratio (struct zone *zone,struct scan_control * sc,unsigned long *percent) { -unsigned long anon, file; +unsigned long anon, file, free;unsigned long anon_prio, file_prio; -unsigned long rotate_sum;unsigned long ap, fp; anon = zone_page_state(zone, NR_ACTIVE_ANON) +
@@ -1175,15 +1186,19 @@ static void get_scan_ratio(struct zone * file =zone_page_state (zone,NR_ACTIVE_FILE ) +zone_page_state (zone,NR_INACTIVE_FILE ); - rotate_sum = zone->recent_rotated_file + zone->recent_rotated_anon; -/* Keep a floating average of RECENT references. */ -if (unlikely (rotate_sum >min (anon, file))) { +if (unlikely (zone->recent_scanned_anon > anon / zone->inactive_ratio)) {spin_lock_irq (&zone->lru_lock); - zone->recent_rotated_file /=2 ; + zone->recent_scanned_anon /=2 ; zone->recent_rotated_anon /=2 ;spin_unlock_irq (&zone->lru_lock); - rotate_sum /=2 ; + } + +if (unlikely (zone->recent_scanned_file > file /4 )) { +spin_lock_irq (&zone->lru_lock); + zone->recent_scanned_file /=2 ; + zone->recent_rotated_file /=2 ; +spin_unlock_irq (&zone->lru_lock); }/*
@@ -1196,23 +1211,33 @@ static void get_scan_ratio(struct zone */* * anon recent_rotated_anon * %anon = 100 * ----------- / ------------------- * IO cost -* anon + file rotate_sum +* anon + file recent_scanned_anon */ - ap = (anon_prio * anon) / (anon + file +1 ); - ap *= rotate_sum / (zone->recent_rotated_anon +1 ); -if (ap ==0 ) - ap =1 ; -else if (ap >100 ) - ap =100 ; - percent[0 ] = ap; - - fp = (file_prio * file) / (anon + file +1 ); - fp *= rotate_sum / (zone->recent_rotated_file +1 ); -if (fp ==0 ) - fp =1 ; -else if (fp >100 ) - fp =100 ; - percent[1 ] = fp; + ap = (anon_prio +1 ) * (zone->recent_scanned_anon +1 ); + ap /= zone->recent_rotated_anon +1 ; + + fp = (file_prio +1 ) * (zone->recent_scanned_file +1 ); + fp /= zone->recent_rotated_file +1 ; + +/* Normalize to percentages */ + percent[0 ] =100 * ap / (ap + fp +1 ); + percent[1 ] =100 - percent[0 ]; + + free =zone_page_state (zone,NR_FREE_PAGES ); + +/* +* If we have no swap space, do not bother scanning anon pages. +*/ +if (nr_swap_pages <=0 ) { + percent[0 ] =0 ; + percent[1 ] =100 ; + } +/* +* If we already freed most file pages, scan the anon pages +* regardless of the page access ratios or swappiness setting. +*/ +else if (file + free <= zone->pages_high) + percent[0 ] =100 ; }
@@ -1233,13 +1258,17 @@ static unsigned long shrink_zone(int prifor_each_lru (l) {if (scan_global_lru (sc)) {int file =is_file_lru (l); +int scan;/* * Add one to nr_to_scan just to make sure that the -* kernel will slowly sift through the active list. +* kernel will slowly sift through each list. */ - zone->nr_scan[l] += (zone_page_state (zone, -NR_INACTIVE_ANON + l) >> priority) +1 ; - nr[l] = zone->nr_scan[l] * percent[file] /100 ; + scan =zone_page_state (zone,NR_INACTIVE_ANON + l); + scan >>= priority; + scan = (scan * percent[file]) /100 ; + + zone->nr_scan[l] += scan +1 ; + nr[l] = zone->nr_scan[l];if (nr[l] >= sc->swap_cluster_max) zone->nr_scan[l] =0 ;else
@@ -1256,7 +1285,7 @@ static unsigned long shrink_zone(int pri }while (nr[LRU_INACTIVE_ANON ] || nr[LRU_ACTIVE_FILE ] || - nr[LRU_INACTIVE_FILE ]) { + nr[LRU_INACTIVE_FILE ]) {for_each_lru (l) {if (nr[l]) { nr_to_scan = min(nr[l],
@@ -1269,6 +1298,13 @@ static unsigned long shrink_zone(int pri } } +/* +* Even if we did not try to evict anon pages at all, we want to +* rebalance the anon lru active/inactive ratio. +*/ +if (scan_global_lru (sc) &&inactive_anon_low (zone)) +shrink_active_list (SWAP_CLUSTER_MAX , zone, sc, priority,0 ); +throttle_vm_writeout (sc->gfp_mask);return nr_reclaimed; }
include/linux/mmzone.h +2 -0
@@ -289,6 +289,8 @@ struct zone {unsigned long recent_rotated_anon;unsigned long recent_rotated_file; +unsigned long recent_scanned_anon; +unsigned long recent_scanned_file;unsigned long pages_scanned;/* since last reclaim */ unsigned long flags;/* zone flags, see below */
mm/page_alloc.c +2 -1
@@ -3512,7 +3512,8 @@ static void __paginginit free_area_init_ } zone->recent_rotated_anon =0 ; zone->recent_rotated_file =0 ; -//TODO recent_scanned_* ??? + zone->recent_scanned_anon =0 ; + zone->recent_scanned_file =0 ;zap_zone_vm_stats (zone); zone->flags =0 ;if (!size)
mm/swap.c +11 -2
@@ -176,8 +176,8 @@ void activate_page(struct page *page)spin_lock_irq (&zone->lru_lock);if (PageLRU (page) && !PageActive (page)) { -int lru =LRU_BASE ; - lru +=page_file_cache (page); +int file =page_file_cache (page); +int lru =LRU_BASE + file;del_page_from_lru_list (zone, page, lru);SetPageActive (page);
@@ -185,6 +185,15 @@ void activate_page(struct page *page)add_page_to_lru_list (zone, page, lru);__count_vm_event (PGACTIVATE );mem_cgroup_move_lists (page, true); + +if (file) { + zone->recent_scanned_file++; + zone->recent_rotated_file++; + }else { +/* Can this happen? Maybe through tmpfs... */ + zone->recent_scanned_anon++; + zone->recent_rotated_anon++; + } }spin_unlock_irq (&zone->lru_lock); }
Signature · 1 line
All Rights Reversed
PATCH 9/14 add newly swapped in pages to the inactive list
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Swapin_readahead can read in a lot of data that the processes in memory never need. Adding swap cache pages to the inactive list prevents them from putting too much pressure on the working set.
This has the potential to help the programs that are already in memory, but it could also be a disadvantage to processes that are trying to get swapped in.
In short, this patch needs testing.
Signed-off-by Rik van Riel <[email protected]>
mm/swap_state.c +1 -1
@@ -301,7 +301,7 @@ struct page *read_swap_cache_async(swp_e/* * Initiate read into locked page and return. */ -lru_cache_add_active_anon (new_page); +lru_cache_add_anon (new_page);swap_readpage (NULL , new_page);return new_page; }
Signature · 1 line
All Rights Reversed
PATCH 10/14 more aggressively use lumpy reclaim
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
During an AIM7 run on a 16GB system, fork started failing around 32000 threads, despite the system having plenty of free swap and 15GB of pageable memory.
If normal pageout does not result in contiguous free pages for kernel stacks, fall back to lumpy reclaim instead of failing fork or doing excessive pageout IO.
I do not know whether this change is needed due to the extreme stress test or because the inactive list is a smaller fraction of system memory on huge systems.
Signed-off-by Rik van Riel <[email protected]>
mm/vmscan.c +16 -4
@@ -857,7 +857,8 @@ int isolate_lru_page(struct page *page)* of reclaimed pages */ static unsigned long shrink_inactive_list (unsigned long max_scan, -struct zone *zone,struct scan_control *sc,int file) +struct zone *zone,struct scan_control *sc, +int priority,int file) {LIST_HEAD (page_list);struct pagevec pvec;
@@ -875,8 +876,19 @@ static unsigned long shrink_inactive_lisunsigned long nr_freed;unsigned long nr_active;unsigned int count[NR_LRU_LISTS ] = {0 , }; -int mode = (sc->order >PAGE_ALLOC_COSTLY_ORDER ) ? -ISOLATE_BOTH :ISOLATE_INACTIVE ; +int mode =ISOLATE_INACTIVE ; + +/* +* If we need a large contiguous chunk of memory, or have +* trouble getting a small set of contiguous pages, we +* will reclaim both active and inactive pages. +* +* We use the same threshold as pageout congestion_wait below. +*/ +if (sc->order >PAGE_ALLOC_COSTLY_ORDER ) + mode =ISOLATE_BOTH ; +else if (sc->order && priority <DEF_PRIORITY -2 ) + mode =ISOLATE_BOTH ; nr_taken = sc->isolate_pages(sc->swap_cluster_max, &page_list, &nr_scan, sc->order, mode,
@@ -1161,7 +1173,7 @@ static unsigned long shrink_list(enum lrshrink_active_list (nr_to_scan, zone, sc, priority, file);return 0 ; } -return shrink_inactive_list (nr_to_scan, zone, sc, file); +return shrink_inactive_list (nr_to_scan, zone, sc, priority, file); }/*
Signature · 1 line
All Rights Reversed
PATCH 11/14 No Reclaim LRU Infrastructure
In reply to Rik van RielFrom: Lee Schermerhorn <[email protected]>
Infrastructure to manage pages excluded from reclaim--i.e., hidden from vmscan. Based on a patch by Larry Woodman of Red Hat. Reworked to maintain "nonreclaimable" pages on a separate per-zone LRU list, to "hide" them from vmscan. A separate noreclaim pagevec is provided for shrink_active_list() to move nonreclaimable pages to the noreclaim list without over burdening the zone lru_lock.
Pages on the noreclaim list have both PG_noreclaim and PG_lru set. Thus, PG_noreclaim is analogous to and mutually exclusive with PG_active--it specifies which LRU list the page is on.
The noreclaim infrastructure is enabled by a new mm Kconfig option [CONFIG_]NORECLAIM_LRU.
A new function 'page_reclaimable(page, vma)' in vmscan.c tests whether or not a page is reclaimable. Subsequent patches will add the various !reclaimable tests. We'll want to keep these tests light-weight for use in shrink_active_list() and, possibly, the fault path.
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by Rik van Riel <[email protected]>
---
V3 -> V6:
+ remove lru_cache_add_active_or_noreclaim(). Only used by optional patch to cull nonreclaimable pages in fault path. Will add back to that patch.
+ misc cleanup pointed out by review of V5
V1 -> V3:
+ rebase to 23-mm1 atop RvR's split LRU series
+ define NR_NORECLAIM and LRU_NORECLAIM to avoid errors when not configured.
V1 -> V2:
+ handle review comments -- various typos and errors.
+ extract "putback_all_noreclaim_pages()" into a separate patch and rework as "scan_all_zones_noreclaim_pages().
include/linux/mm_inline.h | 33 +++++++++++- include/linux/mmzone.h | 24 +++++++++ include/linux/page-flags.h | 17 ++++++ include/linux/pagevec.h | 6 ++ include/linux/swap.h | 22 ++++++++ mm/Kconfig | 10 +++ mm/mempolicy.c | 2 mm/migrate.c | 6 +- mm/page_alloc.c | 9 +++ mm/swap.c | 30 ++++++++--- mm/vmscan.c | 115 ++++++++++++++++++++++++++++++++++++++------- 11 files changed, 242 insertions(+), 32 deletions(-)
mm/Kconfig +10 -0
@@ -205,3 +205,13 @@ config NR_QUICK config VIRT_TO_BUS def_bool y depends on !ARCH_NO_VIRT_TO_BUS + +config NORECLAIM_LRU + bool "Add LRU list to track non-reclaimable pages (EXPERIMENTAL, 64BIT only)" + depends on EXPERIMENTAL && 64BIT + help + Supports tracking of non-reclaimable pages off the [in]active lists + to avoid excessive reclaim overhead on large memory systems. Pages + may be non-reclaimable because: they are locked into memory, they + are anonymous pages for which no swap space exists, or they are anon + pages that are expensive to unmap [long anon_vma "related vma" list.]
include/linux/page-flags.h +15 -0
@@ -94,6 +94,9 @@ enum pageflags { PG_reclaim,/* To be reclaimed asap */ PG_buddy,/* Page is free, on buddy lists */ PG_swapbacked,/* Page is backed by RAM/swap */ +#ifdef CONFIG_NORECLAIM_LRU + PG_noreclaim,/* Page is "non-reclaimable" */ +#endif #ifdef CONFIG_IA64_UNCACHED_ALLOCATOR PG_uncached, /* Page has been mapped as uncached */ #endif
@@ -155,6 +158,7 @@ PAGEFLAG(Referenced, referenced) TESTCLE PAGEFLAG(Dirty, dirty) TESTSCFLAG(Dirty, dirty) __CLEARPAGEFLAG(Dirty, dirty) PAGEFLAG(LRU, lru) __CLEARPAGEFLAG(LRU, lru) PAGEFLAG(Active, active) __CLEARPAGEFLAG(Active, active) + TESTCLEARFLAG(Active, active) __PAGEFLAG(Slab, slab) PAGEFLAG(Checked, owner_priv_1) /* Used by some filesystems */ PAGEFLAG(Pinned, owner_priv_1) TESTSCFLAG(Pinned, owner_priv_1) /* Xen */
@@ -191,6 +195,17 @@ PAGEFLAG(SwapCache, swapcache) PAGEFLAG_FALSE(SwapCache)#endif +#ifdef CONFIG_NORECLAIM_LRU +PAGEFLAG(Noreclaim, noreclaim) __CLEARPAGEFLAG(Noreclaim, noreclaim) +TESTCLEARFLAG(Noreclaim, noreclaim) +#else +PAGEFLAG_FALSE(Noreclaim) +#define SetPageNoreclaim (page) +#define ClearPageNoreclaim (page) +#define __ClearPageNoreclaim (page) +#define TestClearPageNoreclaim (page) 0 +#endif +#ifdef CONFIG_IA64_UNCACHED_ALLOCATOR PAGEFLAG (Uncached, uncached) #else
include/linux/mmzone.h +23 -1
@@ -85,6 +85,11 @@ enum zone_stat_item {NR_ACTIVE_ANON ,/* " " " " " */ NR_INACTIVE_FILE ,/* " " " " " */ NR_ACTIVE_FILE ,/* " " " " " */ +#ifdef CONFIG_NORECLAIM_LRU +NR_NORECLAIM ,/* " " " " " */ +#else +NR_NORECLAIM =NR_ACTIVE_FILE ,/* avoid compiler errors in dead code */ +#endifNR_ANON_PAGES ,/* Mapped anonymous pages */ NR_FILE_MAPPED ,/* pagecache pages mapped into pagetables. only modified from process context */
@@ -124,10 +129,18 @@ enum lru_list {LRU_ACTIVE_ANON =LRU_BASE +LRU_ACTIVE ,LRU_INACTIVE_FILE =LRU_BASE +LRU_FILE ,LRU_ACTIVE_FILE =LRU_BASE +LRU_FILE +LRU_ACTIVE , - NR_LRU_LISTS }; +#ifdef CONFIG_NORECLAIM_LRU +LRU_NORECLAIM , +#else +LRU_NORECLAIM =LRU_ACTIVE_FILE ,/* avoid compiler errors in dead code */ +#endif + NR_LRU_LISTS +};#define for_each_lru (l) for (l = 0; l < NR_LRU_LISTS; l++) +#define for_each_reclaimable_lru (l) for (l = 0; l <= LRU_ACTIVE_FILE; l++) +static inline int is_file_lru (enum lru_list l) {return (l ==LRU_INACTIVE_FILE || l ==LRU_ACTIVE_FILE );
@@ -138,6 +151,15 @@ static inline int is_active_lru(enum lrureturn (l ==LRU_ACTIVE_ANON || l ==LRU_ACTIVE_FILE ); } +static inline int is_noreclaim_lru (enum lru_list l) +{ +#ifdef CONFIG_NORECLAIM_LRU +return (l ==LRU_NORECLAIM ); +#else +return 0 ; +#endif +} +enum lru_list page_lru (struct page *page);struct per_cpu_pages {
mm/page_alloc.c +9 -0
@@ -256,6 +256,9 @@ static void bad_page(struct page *page) 1 << PG_private | 1 << PG_locked | 1 << PG_active | +#ifdef CONFIG_NORECLAIM_LRU + 1 << PG_noreclaim | +#endif 1 << PG_dirty | 1 << PG_reclaim | 1 << PG_slab |
@@ -491,6 +494,9 @@ static inline int free_pages_check(struc 1 << PG_swapcache | 1 << PG_writeback | 1 << PG_reserved | +#ifdef CONFIG_NORECLAIM_LRU + 1 << PG_noreclaim | +#endif 1 << PG_buddy ))))bad_page (page);if (PageDirty (page))
@@ -642,6 +648,9 @@ static int prep_new_page(struct page *pa 1 << PG_private | 1 << PG_locked | 1 << PG_active | +#ifdef CONFIG_NORECLAIM_LRU + 1 << PG_noreclaim | +#endif 1 << PG_dirty | 1 << PG_slab | 1 << PG_swapcache |
include/linux/mm_inline.h +29 -4
@@ -83,17 +83,42 @@ del_page_from_active_file_list(struct zodel_page_from_lru_list (zone, page,LRU_INACTIVE_FILE ); } +#ifdef CONFIG_NORECLAIM_LRU +static inline void +add_page_to_noreclaim_list (struct zone *zone,struct page *page) +{ +add_page_to_lru_list (zone, page,LRU_NORECLAIM ); +} + +static inline void +del_page_from_noreclaim_list (struct zone *zone,struct page *page) +{ +del_page_from_lru_list (zone, page,LRU_NORECLAIM ); +} +#else +static inline void +add_page_to_noreclaim_list (struct zone *zone,struct page *page) { } + +static inline void +del_page_from_noreclaim_list (struct zone *zone,struct page *page) { } +#endif +static inline void del_page_from_lru (struct zone *zone,struct page *page) {enum lru_list l =LRU_INACTIVE_ANON ;list_del (&page->lru); -if (PageActive (page)) { -__ClearPageActive (page); - l +=LRU_ACTIVE ; +if (PageNoreclaim (page)) { +__ClearPageNoreclaim (page); + l =LRU_NORECLAIM ; + }else { +if (PageActive (page)) { +__ClearPageActive (page); + l +=LRU_ACTIVE ; + } + l +=page_file_cache (page); } - l +=page_file_cache (page);__dec_zone_state (zone,NR_INACTIVE_ANON + l); }
include/linux/swap.h +22 -0
@@ -204,6 +204,18 @@ static inline void lru_cache_add_active___lru_cache_add (page,LRU_ACTIVE_FILE ); } +#ifdef CONFIG_NORECLAIM_LRU +static inline void lru_cache_add_noreclaim (struct page *page) +{ +__lru_cache_add (page,LRU_NORECLAIM ); +} +#else +static inline void lru_cache_add_noreclaim (struct page *page) +{ +BUG (); +} +#endif +/* linux/mm/vmscan.c */ extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order,gfp_t gfp_mask);
@@ -228,6 +240,16 @@ static inline int zone_reclaim(struct zo }#endif +#ifdef CONFIG_NORECLAIM_LRU +extern int page_reclaimable (struct page *page,struct vm_area_struct *vma); +#else +static inline int page_reclaimable (struct page *page, +struct vm_area_struct *vma) +{ +return 1 ; +} +#endif +extern int kswapd_run (int nid);#ifdef CONFIG_MMU
include/linux/pagevec.h +6 -0
@@ -101,6 +101,12 @@ static inline void __pagevec_lru_add_act____pagevec_lru_add (pvec,LRU_ACTIVE_FILE ); } +#ifdef CONFIG_NORECLAIM_LRU +static inline void __pagevec_lru_add_noreclaim (struct pagevec *pvec) +{ +____pagevec_lru_add (pvec,LRU_NORECLAIM ); +} +#endif static inline void pagevec_lru_add_file (struct pagevec *pvec) {
mm/swap.c +22 -8
@@ -106,9 +106,13 @@ enum lru_list page_lru(struct page *page {enum lru_list lru =LRU_BASE ; -if (PageActive (page)) - lru +=LRU_ACTIVE ; - lru +=page_file_cache (page); +if (PageNoreclaim (page)) + lru =LRU_NORECLAIM ; +else { +if (PageActive (page)) + lru +=LRU_ACTIVE ; + lru +=page_file_cache (page); + }return lru; }
@@ -133,7 +137,8 @@ static void pagevec_move_tail(struct pag zone = pagezone;spin_lock (&zone->lru_lock); } -if (PageLRU (page) && !PageActive (page)) { +if (PageLRU (page) && !PageActive (page) && + !PageNoreclaim (page)) {int lru =page_file_cache (page);list_move_tail (&page->lru, &zone->list[lru]); pgmoved++;
@@ -154,7 +159,7 @@ static void pagevec_move_tail(struct pagvoid rotate_reclaimable_page (struct page *page) {if (!PageLocked (page) && !PageDirty (page) && !PageActive (page) && -PageLRU (page)) { + !PageNoreclaim (page) &&PageLRU (page)) {struct pagevec *pvec;unsigned long flags;
@@ -175,7 +180,7 @@ void activate_page(struct page *page)struct zone *zone =page_zone (page);spin_lock_irq (&zone->lru_lock); -if (PageLRU (page) && !PageActive (page)) { +if (PageLRU (page) && !PageActive (page) && !PageNoreclaim (page)) {int file =page_file_cache (page);int lru =LRU_BASE + file;del_page_from_lru_list (zone, page, lru);
@@ -207,7 +212,8 @@ void activate_page(struct page *page)*/ void mark_page_accessed (struct page *page) { -if (!PageActive (page) &&PageReferenced (page) &&PageLRU (page)) { +if (!PageActive (page) && !PageNoreclaim (page) && +PageReferenced (page) &&PageLRU (page)) {activate_page (page);ClearPageReferenced (page); }else if (!PageReferenced (page)) {
@@ -235,10 +241,14 @@ void __lru_cache_add(struct page *page,void lru_cache_add_lru (struct page *page,enum lru_list lru) {if (PageActive (page)) { +VM_BUG_ON (PageNoreclaim (page));ClearPageActive (page); + }else if (PageNoreclaim (page)) { +VM_BUG_ON (PageActive (page)); +ClearPageNoreclaim (page); } -VM_BUG_ON (PageLRU (page) ||PageActive (page)); +VM_BUG_ON (PageLRU (page) ||PageActive (page) ||PageNoreclaim (page));__lru_cache_add (page, lru); }
@@ -339,6 +349,7 @@ void release_pages(struct page **pages,if (PageLRU (page)) {struct zone *pagezone =page_zone (page); +if (pagezone != zone) {if (zone) spin_unlock_irqrestore(&zone->lru_lock,
@@ -426,10 +437,13 @@ void ____pagevec_lru_add(struct pagevec zone = pagezone;spin_lock_irq (&zone->lru_lock); } +VM_BUG_ON (PageActive (page) ||PageNoreclaim (page));VM_BUG_ON (PageLRU (page));SetPageLRU (page);if (is_active_lru (lru))SetPageActive (page); +else if (is_noreclaim_lru (lru)) +SetPageNoreclaim (page);add_page_to_lru_list (zone, page, lru); }if (zone)
mm/migrate.c +4 -2
@@ -340,8 +340,11 @@ static void migrate_page_copy(struct pagSetPageReferenced (newpage);if (PageUptodate (page))SetPageUptodate (newpage); -if (PageActive (page)) +if (TestClearPageActive (page)) { +VM_BUG_ON (PageNoreclaim (page));SetPageActive (newpage); + }else if (TestClearPageNoreclaim (page)) +SetPageNoreclaim (newpage);if (PageChecked (page))SetPageChecked (newpage);if (PageMappedToDisk (page))
@@ -362,7 +365,6 @@ static void migrate_page_copy(struct pag#ifdef CONFIG_SWAP ClearPageSwapCache (page);#endif -ClearPageActive (page);ClearPagePrivate (page);set_page_private (page,0 ); page->mapping =NULL ;
mm/vmscan.c +99 -16
205 lines not on this page · raw message
mm/mempolicy.c +1 -1
@@ -2199,7 +2199,7 @@ static void gather_stats(struct page *paif (PageSwapCache (page)) md->swapcache++; -if (PageActive (page)) +if (PageActive (page) ||PageNoreclaim (page)) md->active++;if (PageWriteback (page))
Signature · 1 line
All Rights Reversed
PATCH 12/14 Non-reclaimable page statistics
In reply to Rik van RielFrom: Lee Schermerhorn <[email protected]>
Report non-reclaimable pages per zone and system wide.
Note: may want to track/report some specific reasons for nonreclaimability for deciding when to splice the noreclaim lists back to the normal lru. That will be tricky, especially in shrink_active_list(), where we'd need someplace to save the per page reason for non-reclaimability until the pages are dumped back onto the noreclaim list from the pagevec.
Note: my tests indicate that NR_NORECLAIM and probably the other LRU stats aren't being maintained properly--especially with large amounts of mlocked memory and the mlock patch in this series installed. Can't be sure of this, as I don't know why the pages are on the noreclaim list. Needs further investigation.
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by Rik van Riel <[email protected]>
---
V2 -> V3:
+ rebase to 23-mm1 atop RvR's split LRU series
V1 -> V2:
no changes
drivers/base/node.c | 6 ++++++ fs/proc/proc_misc.c | 6 ++++++ mm/page_alloc.c | 16 +++++++++++++++- mm/vmstat.c | 3 +++ 4 files changed, 30 insertions(+), 1 deletion(-)
mm/page_alloc.c +15 -1
@@ -1918,12 +1918,20 @@ void show_free_areas(void) } printk("Active_anon:%lu active_file:%lu inactive_anon%lu\n" - " inactive_file:%lu dirty:%lu writeback:%lu unstable:%lu\n" + " inactive_file:%lu" +//TODO: check/adjust line lengths +#ifdef CONFIG_NORECLAIM_LRU +" noreclaim:%lu" +#endif +" dirty:%lu writeback:%lu unstable:%lu\n" " free:%lu slab:%lu mapped:%lu pagetables:%lu bounce:%lu\n" ,global_page_state (NR_ACTIVE_ANON ),global_page_state (NR_ACTIVE_FILE ),global_page_state (NR_INACTIVE_ANON ),global_page_state (NR_INACTIVE_FILE ), +#ifdef CONFIG_NORECLAIM_LRU +global_page_state (NR_NORECLAIM ), +#endifglobal_page_state (NR_FILE_DIRTY ),global_page_state (NR_WRITEBACK ), global_page_state(NR_UNSTABLE_NFS),
@@ -1950,6 +1958,9 @@ void show_free_areas(void) " inactive_anon:%lukB" " active_file:%lukB" " inactive_file:%lukB" +#ifdef CONFIG_NORECLAIM_LRU + " noreclaim:%lukB" +#endif" present:%lukB" " pages_scanned:%lu" " all_unreclaimable? %s"
@@ -1963,6 +1974,9 @@ void show_free_areas(void)K (zone_page_state (zone,NR_INACTIVE_ANON )),K (zone_page_state (zone,NR_ACTIVE_FILE )),K (zone_page_state (zone,NR_INACTIVE_FILE )), +#ifdef CONFIG_NORECLAIM_LRU +K (zone_page_state (zone,NR_NORECLAIM )), +#endifK (zone->present_pages), zone->pages_scanned, (zone_is_all_unreclaimable (zone) ?"yes" :"no" )
mm/vmstat.c +3 -0
@@ -699,6 +699,9 @@ static const char * const vmstat_text[]"nr_active_anon" ,"nr_inactive_file" ,"nr_active_file" , +#ifdef CONFIG_NORECLAIM_LRU +"nr_noreclaim" , +#endif"nr_anon_pages" ,"nr_mapped" , "nr_file_pages",
drivers/base/node.c +6 -0
@@ -67,6 +67,9 @@ static ssize_t node_read_meminfo(struct "Node %d Inactive(anon): %8lu kB\n" "Node %d Active(file): %8lu kB\n" "Node %d Inactive(file): %8lu kB\n" +#ifdef CONFIG_NORECLAIM_LRU + "Node %d Noreclaim: %8lu kB\n" +#endif#ifdef CONFIG_HIGHMEM "Node %d HighTotal: %8lu kB\n" "Node %d HighFree: %8lu kB\n"
@@ -96,6 +99,9 @@ static ssize_t node_read_meminfo(struct nid,node_page_state (nid,NR_INACTIVE_ANON ), nid,node_page_state (nid,NR_ACTIVE_FILE ), nid,node_page_state (nid,NR_INACTIVE_FILE ), +#ifdef CONFIG_NORECLAIM_LRU + nid,node_page_state (nid,NR_NORECLAIM ), +#endif #ifdef CONFIG_HIGHMEM nid,K (i.totalhigh), nid, K(i.freehigh),
fs/proc/proc_misc.c +6 -0
@@ -174,6 +174,9 @@ static int meminfo_read_proc(char *page, "Inactive(anon): %8lu kB\n" "Active(file): %8lu kB\n" "Inactive(file): %8lu kB\n" +#ifdef CONFIG_NORECLAIM_LRU + "Noreclaim: %8lu kB\n" +#endif#ifdef CONFIG_HIGHMEM "HighTotal: %8lu kB\n" "HighFree: %8lu kB\n"
@@ -209,6 +212,9 @@ static int meminfo_read_proc(char *page,K (inactive_anon),K (active_file),K (inactive_file), +#ifdef CONFIG_NORECLAIM_LRU +K (global_page_state (NR_NORECLAIM )), +#endif #ifdef CONFIG_HIGHMEMK (i.totalhigh), K(i.freehigh),
Signature · 1 line
All Rights Reversed
PATCH 13/14 ramfs pages are non-reclaimable
In reply to Rik van RielFrom: Lee Schermerhorn <[email protected]>
Christoph Lameter pointed out that ram disk pages also clutter the LRU lists. When vmscan finds them dirty and tries to clean them, the ram disk writeback function just redirties the page so that it goes back onto the active list. Round and round she goes...
Define new address_space flag [shares address_space flags member with mapping's gfp mask] to indicate that the address space contains all non-reclaimable pages. This will provide for efficient testing of ramdisk pages in page_reclaimable().
Also provide wrapper functions to set/test the noreclaim state to minimize #ifdefs in ramdisk driver and any other users of this facility.
Set the noreclaim state on address_space structures for new ramdisk inodes. Test the noreclaim state in page_reclaimable() to cull non-reclaimable pages.
Similarly, ramfs pages are non-reclaimable. Set the 'noreclaim' address_space flag for new ramfs inodes.
These changes depend on [CONFIG_]NORECLAIM_LRU.
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by Rik van Riel <[email protected]>
---
V3 -> V4:
+ drivers/block/rd.c was replaced by brd.c in 24-rc4-mm1. Update patch to add brd_open() to mark mapping as nonreclaimable
V2 -> V3:
+ rebase to 23-mm1 atop RvR's split LRU series [no changes]
V1 -> V2:
+ add ramfs pages to this class of non-reclaimable pages by marking ramfs address_space [mapping] as non-reclaimble.
drivers/block/brd.c | 13 +++++++++++++ fs/ramfs/inode.c | 1 + include/linux/pagemap.h | 22 ++++++++++++++++++++++ mm/vmscan.c | 5 +++++ 4 files changed, 41 insertions(+)
include/linux/pagemap.h +22 -0
@@ -30,6 +30,28 @@ static inline void mapping_set_error(str } } +#ifdef CONFIG_NORECLAIM_LRU +#define AS_NORECLAIM (__GFP_BITS_SHIFT + 2)/* e.g., ramdisk, SHM_LOCK */ + +static inline void mapping_set_noreclaim (struct address_space *mapping) +{ +set_bit (AS_NORECLAIM , &mapping->flags); +} + +static inline int mapping_non_reclaimable (struct address_space *mapping) +{ +if (mapping && (mapping->flags &AS_NORECLAIM )) +return 1 ; +return 0 ; +} +#else +static inline void mapping_set_noreclaim (struct address_space *mapping) { } +static inline int mapping_non_reclaimable (struct address_space *mapping) +{ +return 0 ; +} +#endif +static inline gfp_t mapping_gfp_mask (struct address_space * mapping) {return (__force gfp_t )mapping->flags & __GFP_BITS_MASK;
mm/vmscan.c +5 -0
@@ -2258,6 +2258,8 @@ int zone_reclaim(struct zone *zone, gfp_* lists vs noreclaim list. * * Reasons page might not be reclaimable: +* (1) page's mapping marked non-reclaimable +* * TODO - later patches */ int page_reclaimable (struct page *page,struct vm_area_struct *vma)
@@ -2265,6 +2267,9 @@ int page_reclaimable(struct page *page,VM_BUG_ON (PageNoreclaim (page)); +if (mapping_non_reclaimable (page_mapping (page))) +return 0 ; +/* TODO: test page [!]reclaimable conditions */ return 1 ;
fs/ramfs/inode.c +1 -0
@@ -61,6 +61,7 @@ struct inode *ramfs_get_inode(struct sup inode->i_mapping->a_ops = &ramfs_aops; inode->i_mapping->backing_dev_info = &ramfs_backing_dev_info;mapping_set_gfp_mask (inode->i_mapping,GFP_HIGHUSER ); +mapping_set_noreclaim (inode->i_mapping); inode->i_atime = inode->i_mtime = inode->i_ctime =CURRENT_TIME ;switch (mode &S_IFMT ) {default :
drivers/block/brd.c +13 -0
@@ -374,8 +374,21 @@ static int brd_ioctl(struct inode *inodereturn error; } +/* +* brd_open(): +* Just mark the mapping as containing non-reclaimable pages +*/ +static int brd_open (struct inode *inode,struct file *filp) +{ +struct address_space *mapping = inode->i_mapping; + +mapping_set_noreclaim (mapping); +return 0 ; +} +static struct block_device_operations brd_fops = { .owner =THIS_MODULE , + .open = brd_open, .ioctl = brd_ioctl, #ifdef CONFIG_BLK_DEV_XIP .direct_access = brd_direct_access,
Signature · 1 line
All Rights Reversed
PATCH 14/14 SHM_LOCKED pages are nonreclaimable
In reply to Rik van RielFrom: Lee Schermerhorn <[email protected]>
While working with Nick Piggin's mlock patches, I noticed that shmem segments locked via shmctl(SHM_LOCKED) were not being handled. SHM_LOCKed pages work like ramdisk pages--the writeback function just redirties the page so that it can't be reclaimed. Deal with these using the same approach as for ram disk pages.
Use the AS_NORECLAIM flag to mark address_space of SHM_LOCKed shared memory regions as non-reclaimable. Then these pages will be culled off the normal LRU lists during vmscan.
Add new wrapper function to clear the mapping's noreclaim state when/if shared memory segment is munlocked.
Add 'scan_mapping_noreclaim_page()' to mm/vmscan.c to scan all pages in the shmem segment's mapping [struct address_space] for reclaimability now that they're no longer locked. If so, move them to the appropriate zone lru list.
Changes depend on [CONFIG_]NORECLAIM_LRU.
Signed-off-by Lee Schermerhorn <[email protected]>
Signed-off-by Rik van Riel <[email protected]>
---
V2 -> V3:
+ rebase to 23-mm1 atop RvR's split LRU series.
+ Use scan_mapping_noreclaim_page() on unlock. See below.
V1 -> V2:
+ modify to use reworked 'scan_all_zones_noreclaim_pages()' See 'TODO' below - still pending.
include/linux/pagemap.h | 10 ++++- include/linux/swap.h | 4 ++ mm/shmem.c | 3 + mm/vmscan.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 100 insertions(+), 2 deletions(-)
mm/shmem.c +3 -0
@@ -1469,10 +1469,13 @@ int shmem_lock(struct file *file, int loif (!user_shm_lock (inode->i_size, user)) goto out_nomem; info->flags |=VM_LOCKED ; +mapping_set_noreclaim (file->f_mapping); }if (!lock && (info->flags &VM_LOCKED ) && user) {user_shm_unlock (inode->i_size, user); info->flags &= ~VM_LOCKED ; +mapping_clear_noreclaim (file->f_mapping); +scan_mapping_noreclaim_pages (file->f_mapping); } retval =0 ; out_nomem:
include/linux/pagemap.h +8 -2
@@ -38,14 +38,20 @@ static inline void mapping_set_noreclaim set_bit(AS_NORECLAIM, &mapping->flags); } +static inline void mapping_clear_noreclaim (struct address_space *mapping) +{ +clear_bit (AS_NORECLAIM , &mapping->flags); +} +static inline int mapping_non_reclaimable (struct address_space *mapping) { -if (mapping && (mapping->flags &AS_NORECLAIM )) -return 1 ; +if (mapping) +return test_bit (AS_NORECLAIM , &mapping->flags);return 0 ; }#else static inline void mapping_set_noreclaim (struct address_space *mapping) { } +static inline void mapping_clear_noreclaim (struct address_space *mapping) { }static inline int mapping_non_reclaimable (struct address_space *mapping) {return 0 ;
mm/vmscan.c +85 -0
89 lines not on this page · raw message
include/linux/swap.h +4 -0
@@ -242,12 +242,16 @@ static inline int zone_reclaim(struct zo#ifdef CONFIG_NORECLAIM_LRU extern int page_reclaimable (struct page *page,struct vm_area_struct *vma); +extern void scan_mapping_noreclaim_pages (struct address_space *);#else static inline int page_reclaimable (struct page *page,struct vm_area_struct *vma) {return 1 ; } +static inline void scan_mapping_noreclaim_pages (struct address_space *mapping) +{ +}#endif extern int kswapd_run (int nid);
Signature · 1 line
All Rights Reversed
Quoted 7 lines · Rik van Riel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state.
Against 2.6.25-rc2-mm1
I think it's typo error.
This patch is based on 2.6.26-rc2-mm1. :-)
Quoted 1 line
This patch series improves VM scalability by:
Quoted 10 lines · On Sat, 17 May 2008 14:51:02 +0900 MinChan Kim <[email protected]> wrote:
Quoted 6 lines · Rik van Riel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state.
Against 2.6.25-rc2-mm1
I think it's typo error.
This patch is based on 2.6.26-rc2-mm1. :-)
Yeah, that was a typo. I just inserted -rc2 after forward
porting the stuff from 2.6.25-mm1 :)
Andrew, the code is against your latest tree and has been performance tested in a few scenarios. I believe it should be ready to be merged now.
Signature · 1 line
All rights reversed.