free reclaimed pages by paging out instantly
Subsystems (6)
cover free reclaimed pages by paging out instantly
Normally, I/O completed pages for reclaim would be rotated into inactive LRU tail. IMHO, the why we did is we can't remove the page from page cache and (swap cache, swap slot) by locking problem.
So for reclaiming the I/O completed pages, we need one more iteration of reclaim and it could make unnecessary CPU overhead(ex, acitve->inactive deactivation, isolation, shrink_page_list).
Another concern is related to per process reclaim, which smart platform can reclaim some of the process's pages forcely without OOM kill before VM reaches a latency trouble. Assuming that people does per process reclaim on some processes before even kswapd runs(ie, free pages > high watermark).
And they believe nr_free_pages in vmstat should be increased but it's not true because reclaimed pages caused by paging out(ex, swap page, dirty pages) will be not freed until kswapd runs in the future so that the platform confused and discard more workingset unnecessary or reclaims more processes until the nr_free_pages is increased by our goal.
This patch makes swap cache free logic being aware of irq context so we can free reclaimed pages asap without rotating them back into LRU's tail so that it can reduce unnecessary CPU overhead and LRU churning and makes VM more intuitive.
Big problem of this patch is how to handle for memcg.
I hope please memcg guys look at description [3/4] and get feedback.
Minchan Kim (4):
[1] mm: Don't hide spin_lock in swap_info_get
[2] mm: introduce __swapcache_free
[3] mm: support remove_mapping in irqcontext
[4] mm: free reclaimed pages instantly without depending next reclaim
fs/splice.c | 2 +- include/linux/swap.h | 12 ++++++++++- mm/filemap.c | 6 +++--- mm/swap.c | 14 ++++++++++++- mm/swapfile.c | 22 +++++++++++++++------ mm/truncate.c | 2 +- mm/vmscan.c | 56 +++++++++++++++++++++++++++++++++++++++++++--------- 7 files changed, 92 insertions(+), 22 deletions(-)
Signature · 1 line
1.8.2.1
RFC 1/4 mm: Don't hide spin_lock in swap_info_get
Now, swap_info_get hides lock holding by doing it internally but releasing the lock is caller's duty. It's not serious bad pattern but not good for readability, either.
More concern that if we uses swap_info_get in irq context, the lock should be held with irq disabled.
So it would be better for caller to hold it because he can judge the function will be used in irqcontext or not.
This patch will be used next patchset in this series.
--- mm/swapfile.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
mm/swapfile.c +5 -1
@@ -523,7 +523,6 @@ static struct swap_info_struct *swap_info_get(swp_entry_t entry) goto bad_offset;if (!p->swap_map[offset]) goto bad_free; -spin_lock (&p->lock);return p; bad_free:
@@ -629,6 +628,7 @@ void swap_free(swp_entry_t entry) p =swap_info_get (entry);if (p) { +spin_lock (&p->lock);swap_entry_free (p, entry,1 );spin_unlock (&p->lock); }
@@ -644,6 +644,7 @@ void swapcache_free(swp_entry_t entry, struct page *page) p =swap_info_get (entry);if (p) { +spin_lock (&p->lock); count =swap_entry_free (p, entry,SWAP_HAS_CACHE );if (page)mem_cgroup_uncharge_swapcache (page, entry, count !=0 );
@@ -665,6 +666,7 @@ int page_swapcount(struct page *page) entry.val =page_private (page); p =swap_info_get (entry);if (p) { +spin_lock (&p->lock); count =swap_count (p->swap_map[swp_offset (entry)]);spin_unlock (&p->lock); }
@@ -747,6 +749,7 @@ int free_swap_and_cache(swp_entry_t entry) p =swap_info_get (entry);if (p) { +spin_lock (&p->lock);if (swap_entry_free (p, entry,1 ) ==SWAP_HAS_CACHE ) { page =find_get_page (swap_address_space (entry), entry.val);
@@ -2373,6 +2376,7 @@ int add_swap_count_continuation(swp_entry_t entry, gfp_t gfp_mask) goto outer; } +spin_lock (&si->lock); offset =swp_offset (entry); count = si->swap_map[offset] & ~SWAP_HAS_CACHE ;
Signature · 1 line
1.8.2.1
RFC 2/4 mm: introduce __swapcache_free
In reply to Minchan KimThe __swapcache_free is almost same with swapcache_free
but only difference is that caller should pass stable swap_info_struct.
This function will be used by next patchsets.
--- mm/swapfile.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
mm/swapfile.c +10 -4
@@ -634,20 +634,26 @@ void swap_free(swp_entry_t entry) } } + +void __swapcache_free (struct swap_info_struct *p, +swp_entry_t entry,struct page *page) +{ +unsigned char count; + count =swap_entry_free (p, entry,SWAP_HAS_CACHE ); +mem_cgroup_uncharge_swapcache (page, entry, count !=0 ); +} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */ void swapcache_free (swp_entry_t entry,struct page *page) {struct swap_info_struct *p; -unsigned char count; p =swap_info_get (entry);if (p) {spin_lock (&p->lock); - count =swap_entry_free (p, entry,SWAP_HAS_CACHE ); -if (page) -mem_cgroup_uncharge_swapcache (page, entry, count !=0 ); +__swapcache_free (p, entry, page);spin_unlock (&p->lock); } }
Signature · 1 line
1.8.2.1
RFC 3/4 mm: support remove_mapping in irqcontext
In reply to Minchan KimThis patch makes to use remove_mapping in irq context. For it, this patch adds irqcontext argument in some functions (ex, remove_maping and __remove_mapping) but these functions are not hot path so i believe it's not a problem.
And it exports swap_info_get and check that we can get a swap_info_struct->s lock in advance in irq context. Because __swapcache_free must be succesful.
Otherwise, If __swapcache_free can be failed, we should rollback things done by __delete_from_swap_cache and it could need memory allocation for radix tree node in irq context.
More concern is handling for mem_cgroup_uncharge_swapcache in irqcontext, which isn't aware of irqcontext at the moment and it should be successful like above reason.
After I review that code, I think it's not a big challenge if I missed somethings.
My rough plan is following as,
1. Make mctz->lock beging aware of irq by changing spin_lock with spin_lock_irqsave.
2. Introuduce new argument "locked" in __mem_cgroup_uncharge_common so that __mem_cgroup_uncharge_common can avoid lock_page_cgroup in irqcontext to avoid deadlock but caller in irqcontext should be held it in advance by next patch.
3. Introduce try_lock_page_cgroup, which will be used __swapcache_free.
4. __remove_mapping can held a page_cgroup lock in advance before calling __swapcache_free
I'd like to listen memcg people's opinions before diving into coding.
--- fs/splice.c | 2 +- include/linux/swap.h | 12 ++++++++++- mm/swapfile.c | 2 +- mm/truncate.c | 2 +- mm/vmscan.c | 56 +++++++++++++++++++++++++++++++++++++++++++--------- 5 files changed, 61 insertions(+), 13 deletions(-)
fs/splice.c +1 -1
@@ -70,7 +70,7 @@ static int page_cache_pipe_buf_steal(struct pipe_inode_info *pipe,* If we succeeded in removing the mapping, set LRU flag * and return good. */ -if (remove_mapping (mapping, page)) { +if (remove_mapping (mapping, page, false)) { buf->flags |=PIPE_BUF_FLAG_LRU ;return 0 ; }
include/linux/swap.h +11 -1
@@ -274,7 +274,8 @@ extern unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *mem, unsigned long *nr_scanned);extern unsigned long shrink_all_memory (unsigned long nr_pages);extern int vm_swappiness; -extern int remove_mapping (struct address_space *mapping,struct page *page); +extern int remove_mapping (struct address_space *mapping,struct page *page, +bool irqcontext);extern unsigned long vm_total_pages;#ifdef CONFIG_NUMA
@@ -407,6 +408,9 @@ mem_cgroup_uncharge_swapcache(struct page *page, swp_entry_t ent, bool swapout) }#endif + +extern struct swap_info_struct *swap_info_get (swp_entry_t entry); +#else /* CONFIG_SWAP */ #define get_nr_swap_pages () 0L
@@ -430,6 +434,12 @@ static inline void show_swap_cache_info(void)#define free_swap_and_cache (swp) is_migration_entry(swp)#define swapcache_prepare (swp) is_migration_entry(swp) + +struct swap_info_struct *swap_info_get (swp_entry_t entry) +{ +return NULL ; +} +static inline int add_swap_count_continuation (swp_entry_t swp,gfp_t gfp_mask) {return 0 ;
mm/swapfile.c +1 -1
@@ -505,7 +505,7 @@ swp_entry_t get_swap_page_of_type(int type)return (swp_entry_t ) {0 }; } -static struct swap_info_struct *swap_info_get (swp_entry_t entry) +struct swap_info_struct *swap_info_get (swp_entry_t entry) {struct swap_info_struct *p;unsigned long offset, type;
mm/truncate.c +1 -1
@@ -131,7 +131,7 @@ invalidate_complete_page(struct address_space *mapping, struct page *page)if (page_has_private (page) && !try_to_release_page (page,0 ))return 0 ; - ret =remove_mapping (mapping, page); + ret =remove_mapping (mapping, page, false);return ret; }
mm/vmscan.c +47 -9
99 lines not on this page · raw message
Signature · 7 lines
1.8.2.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 3/4] mm: support remove_mapping in irqcontext
On Mon 13-05-13 11:10:47, Minchan Kim wrote:
[...]
Quoted 4 lines
My rough plan is following as,
1. Make
mctz->lockbeging aware of irq by changingspin_lockwithspin_lock_irqsave.
I wouldn't be worried about this one as it is on its way out with the soft limit rework (the core uncontroversial part ;))
Quoted 9 lines
2. Introuduce new argument "locked" in
__mem_cgroup_uncharge_commonso that__mem_cgroup_uncharge_commoncan avoidlock_page_cgroupin irqcontext to avoid deadlock but caller in irqcontext should be held it in advance by next patch.
3. Introducetry_lock_page_cgroup, which will be used__swapcache_free.
4.__remove_mappingcan held apage_cgrouplock in advance before calling__swapcache_freeI'd like to listen memcg people's opinions before diving into coding.
It should work. It will require some code moving, though.
Quoted 196 lines
Signed-off-by Minchan Kim <[email protected]>--- fs/splice.c | 2 +- include/linux/swap.h | 12 ++++++++++- mm/swapfile.c | 2 +- mm/truncate.c | 2 +- mm/vmscan.c | 56 +++++++++++++++++++++++++++++++++++++++++++--------- 5 files changed, 61 insertions(+), 13 deletions(-)fs/splice.c +1
-1@@ -70,7 +70,7 @@ static int page_cache_pipe_buf_steal(struct pipe_inode_info *pipe,* If we succeeded in removing the mapping, set LRU flag * and return good. */ -if (remove_mapping (mapping, page)) { +if (remove_mapping (mapping, page, false)) { buf->flags |=PIPE_BUF_FLAG_LRU ;return 0 ; }include/linux/swap.h +11
-1@@ -274,7 +274,8 @@ extern unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *mem, unsigned long *nr_scanned);extern unsigned long shrink_all_memory (unsigned long nr_pages);extern int vm_swappiness; -extern int remove_mapping (struct address_space *mapping,struct page *page); +extern int remove_mapping (struct address_space *mapping,struct page *page, +bool irqcontext);extern unsigned long vm_total_pages;#ifdef CONFIG_NUMA @@ -407,6 +408,9 @@ mem_cgroup_uncharge_swapcache(struct page *page, swp_entry_t ent, bool swapout) }#endif + +extern struct swap_info_struct *swap_info_get (swp_entry_t entry); +#else /* CONFIG_SWAP */ #define get_nr_swap_pages () 0L@@ -430,6 +434,12 @@ static inline void show_swap_cache_info(void)#define free_swap_and_cache (swp) is_migration_entry(swp)#define swapcache_prepare (swp) is_migration_entry(swp) + +struct swap_info_struct *swap_info_get (swp_entry_t entry) +{ +return NULL ; +} +static inline int add_swap_count_continuation (swp_entry_t swp,gfp_t gfp_mask) {return 0 ;mm/swapfile.c +1
-1@@ -505,7 +505,7 @@ swp_entry_t get_swap_page_of_type(int type)return (swp_entry_t ) {0 }; } -static struct swap_info_struct *swap_info_get (swp_entry_t entry) +struct swap_info_struct *swap_info_get (swp_entry_t entry) {struct swap_info_struct *p;unsigned long offset, type;mm/truncate.c +1
-1@@ -131,7 +131,7 @@ invalidate_complete_page(struct address_space *mapping, struct page *page)if (page_has_private (page) && !try_to_release_page (page,0 ))return 0 ; - ret =remove_mapping (mapping, page); + ret =remove_mapping (mapping, page, false);return ret; }mm/vmscan.c +47
-9@@ -450,12 +450,18 @@ static pageout_t pageout(struct page *page, struct address_space *mapping,* Same as remove_mapping, but if the page is removed from the mapping, it * gets returned with a refcount of 0. */ -static int __remove_mapping (struct address_space *mapping,struct page *page) +static int __remove_mapping (struct address_space *mapping,struct page *page, +bool irqcontext) { +unsigned long flags; +BUG_ON (!PageLocked (page));BUG_ON (mapping !=page_mapping (page)); -spin_lock_irq (&mapping->tree_lock); +if (irqcontext) +spin_lock_irqsave (&mapping->tree_lock, flags); +else +spin_lock_irq (&mapping->tree_lock);/* * The non racy check for a busy page. * @@ -490,17 +496,45 @@ static int __remove_mapping(struct address_space *mapping, struct page *page) }if (PageSwapCache (page)) { +struct swap_info_struct *p;swp_entry_t swap = { .val =page_private (page) }; + p =swap_info_get (swap); +/* +* If we are irq context, check that we can get a +* swap_info_strcut->lock before removing the page from +* swap cache. Because __swapcache_free must be successful. +* If __swapcache_free can be failed, we should rollback +* things done by __delete_from_swap_cache and it needs +* memory allocation for radix tree node in irqcontext +* That's thing we really want to avoid. +* TODO : memcg mem_cgroup_uncharge_swapcache handling +* in irqcontext +*/ +if (irqcontext && p && !spin_trylock (&p->lock)) { +page_unfreeze_refs (page,2 ); + goto cannot_free; + } +__delete_from_swap_cache (page); -spin_unlock_irq (&mapping->tree_lock); -swapcache_free (swap, page); +if (irqcontext) { +spin_unlock_irqrestore (&mapping->tree_lock, flags); +if (p) +__swapcache_free (p, swap, page); +spin_unlock (&p->lock); + }else { +spin_unlock_irq (&mapping->tree_lock); +swapcache_free (swap, page); + } }else {void (*freepage)(struct page *); freepage = mapping->a_ops->freepage;__delete_from_page_cache (page); -spin_unlock_irq (&mapping->tree_lock); +if (irqcontext) +spin_unlock_irqrestore (&mapping->tree_lock, flags); +else +spin_unlock_irq (&mapping->tree_lock);mem_cgroup_uncharge_cache_page (page);if (freepage !=NULL )@@ -510,7 +544,10 @@ static int __remove_mapping(struct address_space *mapping, struct page *page)return 1 ; cannot_free: -spin_unlock_irq (&mapping->tree_lock); +if (irqcontext) +spin_unlock_irqrestore (&mapping->tree_lock, flags); +else +spin_unlock_irq (&mapping->tree_lock);return 0 ; }@@ -520,9 +557,10 @@ cannot_free:* successfully detached, return 1. Assumes the caller has a single ref on * this page. */ -int remove_mapping (struct address_space *mapping,struct page *page) +int remove_mapping (struct address_space *mapping,struct page *page, +bool irqcontext) { -if (__remove_mapping (mapping, page)) { +if (__remove_mapping (mapping, page, irqcontext)) {/* * Unfreezing the refcount with 1 rather than 2 effectively * drops the pagecache ref for us without requiring another @@ -904,7 +942,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, } } -if (!mapping || !__remove_mapping (mapping, page)) +if (!mapping || !__remove_mapping (mapping, page, false)) goto keep_locked;/* Signature · 1 line
1.8.2.1
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 3/4] mm: support remove_mapping in irqcontext
Hey Michal,
Quoted 9 lines · Michal Hocko
On Mon 13-05-13 11:10:47, Minchan Kim wrote:
[...]Quoted 4 lines
My rough plan is following as,
1. Make
mctz->lockbeging aware of irq by changingspin_lockwithspin_lock_irqsave.I wouldn't be worried about this one as it is on its way out with the soft limit rework (the core uncontroversial part ;))
Good to hear!
Quoted 9 lines
Quoted 9 lines
2. Introuduce new argument "locked" in
__mem_cgroup_uncharge_commonso that__mem_cgroup_uncharge_commoncan avoidlock_page_cgroupin irqcontext to avoid deadlock but caller in irqcontext should be held it in advance by next patch.
3. Introducetry_lock_page_cgroup, which will be used__swapcache_free.
4.__remove_mappingcan held apage_cgrouplock in advance before calling__swapcache_free
Quoted 3 lines
Quoted 1 line
I'd like to listen memcg people's opinions before diving into coding.
It should work. It will require some code moving, though.
Yeb. I will give it a shot!
Thanks for the review!
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
RFC 4/4 mm: free reclaimed pages instantly without depending next reclaim
In reply to Minchan KimNormally, file I/O for reclaiming is asynchronous so that when page writeback is completed, reclaimed page will be rotated into LRU tail for fast reclaiming in next turn. But it makes unnecessary CPU overhead and more iteration with higher priority of reclaim could reclaim too many pages than needed pages.
This patch frees reclaimed pages by paging out instantly without rotating back them into LRU's tail when the I/O is completed so that we can get out of reclaim loop as soon as poosbile and avoid unnecessary CPU overhead for moving them.
--- mm/filemap.c | 6 +++--- mm/swap.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 4 deletions(-)
mm/filemap.c +3 -3
@@ -618,12 +618,12 @@ EXPORT_SYMBOL(unlock_page);*/ void end_page_writeback (struct page *page) { -if (TestClearPageReclaim (page)) -rotate_reclaimable_page (page); -if (!test_clear_page_writeback (page))BUG (); +if (TestClearPageReclaim (page)) +rotate_reclaimable_page (page); +smp_mb__after_clear_bit ();wake_up_page (page, PG_writeback); }
mm/swap.c +13 -1
@@ -324,7 +324,19 @@ static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec,int *pgmoved = arg;if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) { -enum lru_list lru =page_lru_base_type (page); +enum lru_list lru; + +if (!trylock_page (page)) + goto move_tail; + +if (!remove_mapping (page_mapping (page), page, true)) { +unlock_page (page); + goto move_tail; + } +unlock_page (page); +return ; +move_tail: + lru =page_lru_base_type (page);list_move_tail (&page->lru, &lruvec->lists[lru]); (*pgmoved)++; }
Signature · 7 lines
1.8.2.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 4/4] mm: free reclaimed pages instantly without depending next reclaim
Quoted 7 lines · Minchan Kim
Normally, file I/O for reclaiming is asynchronous so that when page writeback is completed, reclaimed page will be rotated into LRU tail for fast reclaiming in next turn. But it makes unnecessary CPU overhead and more iteration with higher priority of reclaim could reclaim too many pages than needed pages.
Quoted 6 lines
This patch frees reclaimed pages by paging out instantly without rotating back them into LRU's tail when the I/O is completed so that we can get out of reclaim loop as soon as poosbile and avoid unnecessary CPU overhead for moving them.
Signed-off-by Minchan Kim <[email protected]>
I like this approach and am looking forward to your v2 series, with the reworked patch 3/4.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [RFC 4/4] mm: free reclaimed pages instantly without depending next reclaim
Hey Rik,
Quoted 13 lines · Rik van Riel
Quoted 12 lines · Minchan Kim
Normally, file I/O for reclaiming is asynchronous so that when page writeback is completed, reclaimed page will be rotated into LRU tail for fast reclaiming in next turn. But it makes unnecessary CPU overhead and more iteration with higher priority of reclaim could reclaim too many pages than needed pages.
This patch frees reclaimed pages by paging out instantly without rotating back them into LRU's tail when the I/O is completed so that we can get out of reclaim loop as soon as poosbile and avoid unnecessary CPU overhead for moving them.
Quoted 4 lines
Quoted 1 line
Signed-off-by Minchan Kim <[email protected]>I like this approach and am looking forward to your v2 series, with the reworked patch 3/4.
I will do it after I finish more urgent works. :)
I am looking forward to seeing your review, then.
Thanks for the interest.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>