per memcg lru lock
Subsystems (16)
v20 cover per memcg lru lock
This version just a rebase on v5.10-rc1. and moves the lru_lock position down after lists[] in lruvec, which resolves a fio.read regression that revealed by Rong Chen -- Intel LKP.
Many thanks for line by line review by Hugh Dickins and Alexander Duyck.
So now this patchset includes 3 parts:
1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition. 3, replace per node lru_lock with per memcg per node lru_lock.
Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain.
Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock.
The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it.
The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!).
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box on v18, which has no much different with this v20.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox, Alexander Duyck etc.
Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks!
Alex Shi (17):
mm/memcg: warning on !memcg after readahead page charged mm/memcg: bail early from swap accounting if memcg disabled mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: use head for head page in lru_add_page_tail mm/thp: Simplify lru_add_page_tail()
mm/thp: narrow lru locking
mm/vmscan: remove unnecessary lruvec adding mm/memcg: add debug checking in lock_page_memcg mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lock into lru_note_cost
mm/vmscan: remove lruvec reget in move_pages_to_lru mm/mlock: remove lru_lock on TestClearPageMlocked mm/mlock: remove __munlock_isolate_lru_page mm/lru: introduce TestClearPageLRU
mm/compaction: do page isolation first in compaction mm/swap.c: serialize memcg changes in pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock
Alexander Duyck (1):
mm/lru: introduce the relock_page_lruvec function
Hugh Dickins (2):
mm: page_idle_get_page() does not need lru_lock
mm/lru: revise the comments of lru_lock
Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +-- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/memcontrol.h | 110 +++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmdebug.h | 13 ++ include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- mm/compaction.c | 94 +++++++--- mm/filemap.c | 4 +- mm/huge_memory.c | 45 +++-- mm/memcontrol.c | 85 ++++++++- mm/mlock.c | 63 ++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 4 - mm/rmap.c | 4 +- mm/swap.c | 199 ++++++++------------ mm/vmscan.c | 203 +++++++++++---------- mm/workingset.c | 2 - 22 files changed, 523 insertions(+), 378 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v20 1/20 mm/memcg: warning on !memcg after readahead page charged
Add VM_WARN_ON_ONCE_PAGE() macro.
Since readahead page is charged on memcg too, in theory we don't have to check this exception now. Before safely remove them all, add a warning for the unexpected !memcg.
--- include/linux/mmdebug.h | 13 +++++++++++++ mm/memcontrol.c | 11 ++++------- 2 files changed, 17 insertions(+), 7 deletions(-)
include/linux/mmdebug.h +13 -0
@@ -37,6 +37,18 @@BUG (); \ } \ } while (0) +#define VM_WARN_ON_ONCE_PAGE (cond, page) ({ \ + static bool __section(".data.once") __warned; \ + int __ret_warn_once = !!(cond); \ + \ + if (unlikely(__ret_warn_once && !__warned)) { \ + dump_page(page, "VM_WARN_ON_ONCE_PAGE(" __stringify(cond)")");\ + __warned = true; \ + WARN_ON(1); \ + } \ + unlikely(__ret_warn_once); \ +}) +#define VM_WARN_ON (cond) (void)WARN_ON(cond)#define VM_WARN_ON_ONCE (cond) (void)WARN_ON_ONCE(cond) #define VM_WARN_ONCE(cond, format...) (void)WARN_ONCE(cond, format)
@@ -48,6 +60,7 @@#define VM_BUG_ON_MM (cond, mm) VM_BUG_ON(cond)#define VM_WARN_ON (cond) BUILD_BUG_ON_INVALID(cond)#define VM_WARN_ON_ONCE (cond) BUILD_BUG_ON_INVALID(cond) +#define VM_WARN_ON_ONCE_PAGE (cond, page) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond) #define VM_WARN(cond, format...) BUILD_BUG_ON_INVALID(cond)#endif
mm/memcontrol.c +4 -7
@@ -1350,10 +1350,7 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd } memcg = page->mem_cgroup; -/* -* Swapcache readahead pages are added to the LRU - and -* possibly migrated - before they are charged. -*/ +VM_WARN_ON_ONCE_PAGE (!memcg, page);if (!memcg) memcg = root_mem_cgroup;
@@ -6979,8 +6976,8 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)if (newpage->mem_cgroup)return ; -/* Swapcache readahead pages can get replaced before being charged */ memcg = oldpage->mem_cgroup; +VM_WARN_ON_ONCE_PAGE (!memcg, oldpage);if (!memcg)return ;
@@ -7177,7 +7174,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) memcg = page->mem_cgroup; -/* Readahead page, never charged */ +VM_WARN_ON_ONCE_PAGE (!memcg, page);if (!memcg)return ;
@@ -7241,7 +7238,7 @@ int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry) memcg = page->mem_cgroup; -/* Readahead page, never charged */ +VM_WARN_ON_ONCE_PAGE (!memcg, page);if (!memcg)return 0 ;
Signature · 1 line
1.8.3.1
Re: [PATCH v20 01/20] mm/memcg: warning on !memcg after readahead page charged
Quoted 6 lines · Alex Shi
Add
VM_WARN_ON_ONCE_PAGE()macro.Since readahead page is charged on memcg too, in theory we don't have to check this exception now. Before safely remove them all, add a warning for the unexpected !memcg.
Quoted 10 lines
Signed-off-by Alex Shi <[email protected]>Acked-by Michal Hocko <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>
PATCH v20 2/20 mm/memcg: bail early from swap accounting if memcg disabled
In reply to Alex ShiIf we disabled memcg by cgroup_disable=memory, page->memcg will be NULL and so the charge is skipped and that will trigger a warning like below. Let's return from the funcs earlier.
anon flags:0x5005b48008000d(locked|uptodate|dirty|swapbacked)
raw: 005005b48008000d dead000000000100 dead000000000122 ffff8897c7c76ad1 raw: 0000000000000022 0000000000000000 0000000200000000 0000000000000000 page dumped because: VM_WARN_ON_ONCE_PAGE(!memcg) ... RIP: 0010:vprintk_emit+0x1f7/0x260 Code: 00 84 d2 74 72 0f b6 15 27 58 64 01 48 c7 c0 00 d4 72 82 84 d2 74 09 f3 90 0f b6 10 84 d2 75 f7 e8 de 0d 00 00 4c 89 e7 57 9d <0f> 1f 44 00 00 e9 62 ff ff ff 80 3d 88 c9 3a 01 00 0f 85 54 fe ff RSP: 0018:ffffc9000faab358 EFLAGS: 00000202 RAX: ffffffff8272d400 RBX: 000000000000005e RCX: ffff88afd80d0040 RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000202 RBP: ffffc9000faab3a8 R08: ffffffff8272d440 R09: 0000000000022480 R10: 00120c77be68bfac R11: 0000000000cd7568 R12: 0000000000000202 R13: 0057ffffc0080005 R14: ffffffff820a0130 R15: ffffc9000faab3e8 ? vprintk_emit+0x140/0x260 vprintk_default+0x1a/0x20 vprintk_func+0x4f/0xc4 ? vprintk_func+0x4f/0xc4 printk+0x53/0x6a ? xas_load+0xc/0x80 __dump_page.cold.6+0xff/0x4ee ? xas_init_marks+0x23/0x50 ? xas_store+0x30/0x40 ? free_swap_slot+0x43/0xd0 ? put_swap_page+0x119/0x320 ? update_load_avg+0x82/0x580 dump_page+0x9/0xb mem_cgroup_try_charge_swap+0x16e/0x1d0 get_swap_page+0x130/0x210 add_to_swap+0x41/0xc0 shrink_page_list+0x99e/0xdf0 shrink_inactive_list+0x199/0x360 shrink_lruvec+0x40d/0x650 ? _cond_resched+0x14/0x30 ? _cond_resched+0x14/0x30 shrink_node+0x226/0x6e0 do_try_to_free_pages+0xd0/0x400 try_to_free_pages+0xef/0x130 __alloc_pages_slowpath.constprop.127+0x38d/0xbd0 ? ___slab_alloc+0x31d/0x6f0 __alloc_pages_nodemask+0x27f/0x2c0 alloc_pages_vma+0x75/0x220 shmem_alloc_page+0x46/0x90 ? release_pages+0x1ae/0x410 shmem_alloc_and_acct_page+0x77/0x1c0 shmem_getpage_gfp+0x162/0x910 shmem_fault+0x74/0x210 ? filemap_map_pages+0x29c/0x410 __do_fault+0x37/0x190 handle_mm_fault+0x120a/0x1770 exc_page_fault+0x251/0x450 ? asm_exc_page_fault+0x8/0x30 asm_exc_page_fault+0x1e/0x30
--- mm/memcontrol.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memcontrol.c +6 -0
@@ -7169,6 +7169,9 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)VM_BUG_ON_PAGE (PageLRU (page), page);VM_BUG_ON_PAGE (page_count (page), page); +if (mem_cgroup_disabled ()) +return ; +if (cgroup_subsys_on_dfl (memory_cgrp_subsys))return ;
@@ -7233,6 +7236,9 @@ int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry)struct mem_cgroup *memcg;unsigned short oldid; +if (mem_cgroup_disabled ()) +return 0 ; +if (!cgroup_subsys_on_dfl (memory_cgrp_subsys))return 0 ;
Signature · 1 line
1.8.3.1
Re: [PATCH v20 02/20] mm/memcg: bail early from swap accounting if memcg disabled
Quoted 57 lines · Alex Shi
If we disabled memcg by cgroup_disable=memory,
page->memcgwill be NULL and so the charge is skipped and that will trigger a warning like below. Let's return from the funcs earlier.anon flags:0x5005b48008000d(locked|uptodate|dirty|swapbacked)
raw: 005005b48008000d dead000000000100 dead000000000122 ffff8897c7c76ad1 raw: 0000000000000022 0000000000000000 0000000200000000 0000000000000000 page dumped because: VM_WARN_ON_ONCE_PAGE(!memcg) ... RIP: 0010:vprintk_emit+0x1f7/0x260 Code: 00 84 d2 74 72 0f b6 15 27 58 64 01 48 c7 c0 00 d4 72 82 84 d2 74 09 f3 90 0f b6 10 84 d2 75 f7 e8 de 0d 00 00 4c 89 e7 57 9d <0f> 1f 44 00 00 e9 62 ff ff ff 80 3d 88 c9 3a 01 00 0f 85 54 fe ff RSP: 0018:ffffc9000faab358 EFLAGS: 00000202 RAX: ffffffff8272d400 RBX: 000000000000005e RCX: ffff88afd80d0040 RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000202 RBP: ffffc9000faab3a8 R08: ffffffff8272d440 R09: 0000000000022480 R10: 00120c77be68bfac R11: 0000000000cd7568 R12: 0000000000000202 R13: 0057ffffc0080005 R14: ffffffff820a0130 R15: ffffc9000faab3e8 ? vprintk_emit+0x140/0x260 vprintk_default+0x1a/0x20 vprintk_func+0x4f/0xc4 ? vprintk_func+0x4f/0xc4 printk+0x53/0x6a ? xas_load+0xc/0x80 __dump_page.cold.6+0xff/0x4ee ? xas_init_marks+0x23/0x50 ? xas_store+0x30/0x40 ? free_swap_slot+0x43/0xd0 ? put_swap_page+0x119/0x320 ? update_load_avg+0x82/0x580 dump_page+0x9/0xb mem_cgroup_try_charge_swap+0x16e/0x1d0 get_swap_page+0x130/0x210 add_to_swap+0x41/0xc0 shrink_page_list+0x99e/0xdf0 shrink_inactive_list+0x199/0x360 shrink_lruvec+0x40d/0x650 ? _cond_resched+0x14/0x30 ? _cond_resched+0x14/0x30 shrink_node+0x226/0x6e0 do_try_to_free_pages+0xd0/0x400 try_to_free_pages+0xef/0x130 __alloc_pages_slowpath.constprop.127+0x38d/0xbd0 ? ___slab_alloc+0x31d/0x6f0 __alloc_pages_nodemask+0x27f/0x2c0 alloc_pages_vma+0x75/0x220 shmem_alloc_page+0x46/0x90 ? release_pages+0x1ae/0x410 shmem_alloc_and_acct_page+0x77/0x1c0 shmem_getpage_gfp+0x162/0x910 shmem_fault+0x74/0x210 ? filemap_map_pages+0x29c/0x410 __do_fault+0x37/0x190 handle_mm_fault+0x120a/0x1770 exc_page_fault+0x251/0x450 ? asm_exc_page_fault+0x8/0x30 asm_exc_page_fault+0x1e/0x30
Quoted 11 lines
Signed-off-by Alex Shi <[email protected]>Reviewed-by Roman Gushchin <[email protected]>Acked-by Michal Hocko <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>
This should go in before the previous patch that adds the WARN for it.
Re: [PATCH v20 02/20] mm/memcg: bail early from swap accounting if memcg disabled
ÔÚ 2020/10/29 ÏÂÎç9:46, Johannes Weiner дµÀ:
Quoted 22 lines
Quoted 11 lines
? release_pages+0x1ae/0x410 shmem_alloc_and_acct_page+0x77/0x1c0 shmem_getpage_gfp+0x162/0x910 shmem_fault+0x74/0x210 ? filemap_map_pages+0x29c/0x410 __do_fault+0x37/0x190 handle_mm_fault+0x120a/0x1770 exc_page_fault+0x251/0x450 ? asm_exc_page_fault+0x8/0x30 asm_exc_page_fault+0x1e/0x30Quoted 11 lines
Signed-off-by Alex Shi <[email protected]>Reviewed-by Roman Gushchin <[email protected]>Acked-by Michal Hocko <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>
Quoted 3 lines
Acked-by Johannes Weiner <[email protected]>This should go in before the previous patch that adds the WARN for it.
Right, but than the long ops may not weird. Should I remove the ops and resend the whole patchset?
Which way is convenient for you?
Thanks
Alex
Johannes Weiner
maintainerRe: [PATCH v20 02/20] mm/memcg: bail early from swap accounting if memcg disabled
Quoted 25 lines · Alex Shi
Quoted 22 lines · Johannes Weiner
Quoted 22 lines
? release_pages+0x1ae/0x410 shmem_alloc_and_acct_page+0x77/0x1c0 shmem_getpage_gfp+0x162/0x910 shmem_fault+0x74/0x210 ? filemap_map_pages+0x29c/0x410 __do_fault+0x37/0x190 handle_mm_fault+0x120a/0x1770 exc_page_fault+0x251/0x450 ? asm_exc_page_fault+0x8/0x30 asm_exc_page_fault+0x1e/0x30Reviewed-by Roman Gushchin <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
Quoted 5 lines
Quoted 3 lines
Acked-by Johannes Weiner <[email protected]>This should go in before the previous patch that adds the WARN for it.
Right, but than the long ops may not weird. Should I remove the ops and resend the whole patchset?
You mean the warning in the changelog? I think that's alright. You can just say that you're about to remove the !page->memcg check in the next patch because the original reasons for having it are gone, and memcg being disabled is the only remaining exception, so this patch makes that check explicit in preparation for the next.
Sorry, it's all a bit of a hassle, I just wouldn't want to introduce a known warning into the kernel between those two patches (could confuse bisection runs, complicates partial reverts etc.)
Re: [PATCH v20 02/20] mm/memcg: bail early from swap accounting if memcg disabled
Quoted 10 lines · Johannes Weiner
Quoted 4 lines
Quoted 3 lines
Acked-by Johannes Weiner <[email protected]>This should go in before the previous patch that adds the WARN for it.
Right, but than the long ops may not weird. Should I remove the ops and resend the whole patchset?
You mean the warning in the changelog? I think that's alright. You can just say that you're about to remove the !page->memcg check in the next patch because the original reasons for having it are gone, and memcg being disabled is the only remaining exception, so this patch makes that check explicit in preparation for the next.
Quoted 3 lines
Sorry, it's all a bit of a hassle, I just wouldn't want to introduce a known warning into the kernel between those two patches (could confuse bisection runs, complicates partial reverts etc.)
H Johannes,
I see, I will exchange the 1st and 2nd patch place with above comments in commit log. I guess you could give more comments on other patches, so I am going to wait you for more comments and send v21 as a whole. :)
Many thanks!
Alex
PATCH v20 3/20 mm/thp: move lru_add_page_tail func to huge_memory.c
In reply to Alex ShiThe func is only used in huge_memory.c, defining it in other file with a CONFIG_TRANSPARENT_HUGEPAGE macro restrict just looks weird.
Let's move it THP. And make it static as Hugh Dickin suggested.
--- include/linux/swap.h | 2 -- mm/huge_memory.c | 30 ++++++++++++++++++++++++++++++ mm/swap.c | 33 --------------------------------- 3 files changed, 30 insertions(+), 35 deletions(-)
include/linux/swap.h +0 -2
@@ -338,8 +338,6 @@ extern void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages);extern void lru_note_cost_page (struct page *);extern void lru_cache_add (struct page *); -extern void lru_add_page_tail (struct page *page,struct page *page_tail, -struct lruvec *lruvec,struct list_head *head);extern void mark_page_accessed (struct page *);extern void lru_add_drain (void );extern void lru_add_drain_cpu (int cpu);
mm/huge_memory.c +30 -0
@@ -2346,6 +2346,36 @@ static void remap_page(struct page *page, unsigned int nr) } } +static void lru_add_page_tail (struct page *page,struct page *page_tail, +struct lruvec *lruvec,struct list_head *list) +{ +VM_BUG_ON_PAGE (!PageHead (page), page); +VM_BUG_ON_PAGE (PageCompound (page_tail), page); +VM_BUG_ON_PAGE (PageLRU (page_tail), page); +lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); + +if (!list) +SetPageLRU (page_tail); + +if (likely (PageLRU (page))) +list_add_tail (&page_tail->lru, &page->lru); +else if (list) { +/* page reclaim is reclaiming a huge page */ +get_page (page_tail); +list_add_tail (&page_tail->lru, list); + }else { +/* +* Head page has not yet been counted, as an hpage, +* so we must account for each subpage individually. +* +* Put page_tail on the list at the correct position +* so they all end up in order. +*/ +add_page_to_lru_list_tail (page_tail, lruvec, +page_lru (page_tail)); + } +} +static void __split_huge_page_tail (struct page *head,int tail,struct lruvec *lruvec,struct list_head *list) {
mm/swap.c +0 -33
@@ -974,39 +974,6 @@ void __pagevec_release(struct pagevec *pvec) }EXPORT_SYMBOL (__pagevec_release); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -/* used by __split_huge_page_refcount() */ -void lru_add_page_tail (struct page *page,struct page *page_tail, -struct lruvec *lruvec,struct list_head *list) -{ -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); - -if (!list) -SetPageLRU (page_tail); - -if (likely (PageLRU (page))) -list_add_tail (&page_tail->lru, &page->lru); -else if (list) { -/* page reclaim is reclaiming a huge page */ -get_page (page_tail); -list_add_tail (&page_tail->lru, list); - }else { -/* -* Head page has not yet been counted, as an hpage, -* so we must account for each subpage individually. -* -* Put page_tail on the list at the correct position -* so they all end up in order. -*/ -add_page_to_lru_list_tail (page_tail, lruvec, -page_lru (page_tail)); - } -} -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec,void *arg) {
Signature · 1 line
1.8.3.1
Re: [PATCH v20 03/20] mm/thp: move lru_add_page_tail func to huge_memory.c
Quoted 5 lines · Alex Shi
The func is only used in
huge_memory.c, defining it in other file with aCONFIG_TRANSPARENT_HUGEPAGEmacro restrict just looks weird.Let's move it THP. And make it static as Hugh Dickin suggested.
Quoted 9 lines
Signed-off-by Alex Shi <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
PATCH v20 4/20 mm/thp: use head for head page in lru_add_page_tail
In reply to Alex ShiSince the first parameter is only used by head page, it's better to make it explicit.
--- mm/huge_memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
mm/huge_memory.c +6 -6
@@ -2346,19 +2346,19 @@ static void remap_page(struct page *page, unsigned int nr) } } -static void lru_add_page_tail (struct page *page,struct page *page_tail, +static void lru_add_page_tail (struct page *head,struct page *page_tail,struct lruvec *lruvec,struct list_head *list) { -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON_PAGE (!PageHead (head), head); +VM_BUG_ON_PAGE (PageCompound (page_tail), head); +VM_BUG_ON_PAGE (PageLRU (page_tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock);if (!list)SetPageLRU (page_tail); -if (likely (PageLRU (page))) -list_add_tail (&page_tail->lru, &page->lru); +if (likely (PageLRU (head))) +list_add_tail (&page_tail->lru, &head->lru);else if (list) {/* page reclaim is reclaiming a huge page */ get_page (page_tail);
Signature · 1 line
1.8.3.1
Johannes Weiner
maintainerRe: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
Quoted 21 lines · Alex Shi
Since the first parameter is only used by head page, it's better to make it explicit.
Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>--- mm/huge_memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)mm/huge_memory.c +0
-0
mm/huge_memory.c static void remap_page(struct page *page, unsigned int nr) · line 2349
Quoted 5 lines
mm/huge_memory.c +1
-1@@ -2346,19 +2346,19 @@ static void remap_page(struct page *page, unsigned int nr) } } -static void lru_add_page_tail(struct page *page, struct page *page_tail, +static void lru_add_page_tail(struct page *head, struct page *page_tail,
It may be better to pick either
head and tail
or
page_head and page_tail
?
Re: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
ÔÚ 2020/10/29 ÏÂÎç9:50, Johannes Weiner дµÀ:
Quoted 25 lines · Alex Shi
Quoted 24 lines
Since the first parameter is only used by head page, it's better to make it explicit.
Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>--- mm/huge_memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)mm/huge_memory.c +0
-0@@ -2346,19 +2346,19 @@ static void remap_page(struct page *page, unsigned int nr) } }
Quoted 5 lines
mm/huge_memory.c static void remap_page(struct page *page, unsigned int nr) · line 2349
Quoted 2 lines
mm/huge_memory.c +1
-1-static void lru_add_page_tail(struct page *page, struct page *page_tail, +static void lru_add_page_tail(struct page *head, struct page *page_tail,It may be better to pick either
head and tail
Hi Johannes,
Thanks for comments!
Right, Consider functions in this file are using head/tail more as parameters I will change to use head/tail too. And then, the 04th, 05th, and 18th patch will be changed accordingly.
Thanks
Alex
Quoted 5 lines
or
page_headandpage_tail?
From a9ee63a213f40eb4d5a69b52fbb348ff9cd7cf6c Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 26 May 2020 16:49:22 +0800
Subject: [PATCH v21 04/20] mm/thp: use head for head page in lru_add_page_tail
Since the first parameter is only used by head page, it's better to make it explicit.
--- mm/huge_memory.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
mm/huge_memory.c +11 -12
@@ -2346,33 +2346,32 @@ static void remap_page(struct page *page, unsigned int nr) } } -static void lru_add_page_tail (struct page *page,struct page *page_tail, +static void lru_add_page_tail (struct page *head,struct page *tail,struct lruvec *lruvec,struct list_head *list) { -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON_PAGE (!PageHead (head), head); +VM_BUG_ON_PAGE (PageCompound (tail), head); +VM_BUG_ON_PAGE (PageLRU (tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock);if (!list) -SetPageLRU (page_tail); +SetPageLRU (tail); -if (likely (PageLRU (page))) -list_add_tail (&page_tail->lru, &page->lru); +if (likely (PageLRU (head))) +list_add_tail (&tail->lru, &head->lru);else if (list) {/* page reclaim is reclaiming a huge page */ -get_page (page_tail); -list_add_tail (&page_tail->lru, list); +get_page (tail); +list_add_tail (&tail->lru, list); }else {/* * Head page has not yet been counted, as an hpage, * so we must account for each subpage individually. * -* Put page_tail on the list at the correct position +* Put tail on the list at the correct position * so they all end up in order. */ -add_page_to_lru_list_tail (page_tail, lruvec, -page_lru (page_tail)); +add_page_to_lru_list_tail (tail, lruvec,page_lru (tail)); } }
Signature · 1 line
1.8.3.1
Re: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
Quoted 11 lines · Alex Shi
Quoted 2 lines · Johannes Weiner
It may be better to pick either
head and tailHi Johannes,
Thanks for comments!
Right, Consider functions in this file are using head/tail more as parameters I will change to use head/tail too. And then, the 04th, 05th, and 18th patch will be changed accordingly.
That's great, thank you!
Quoted 8 lines
From
a9ee63a213f40eb4d5a69b52fbb348ff9cd7cf6cMon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 26 May 2020 16:49:22 +0800
Subject: [PATCH v21 04/20] mm/thp: use head for head page inlru_add_page_tailSince the first parameter is only used by head page, it's better to make it explicit.
Quoted 9 lines
Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
Re: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
Quoted 18 lines · Johannes Weiner
Quoted 9 lines
From
a9ee63a213f40eb4d5a69b52fbb348ff9cd7cf6cMon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Tue, 26 May 2020 16:49:22 +0800
Subject: [PATCH v21 04/20] mm/thp: use head for head page inlru_add_page_tailSince the first parameter is only used by head page, it's better to make it explicit.
Quoted 9 lines
Signed-off-by Alex Shi <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
Quoted 1 line
Acked-by Johannes Weiner <[email protected]>
Thanks a lot!
Alex
Re: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
In reply to Alex Shimm/huge_memory.c static void remap_page(struct page *page, unsigned int nr) · line 2354
Quoted 10 lines · Alex Shi
mm/huge_memory.c +4
-4-static void lru_add_page_tail (struct page *page,struct page *page_tail, +static void lru_add_page_tail (struct page *head,struct page *tail,struct lruvec *lruvec,struct list_head *list) { -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON_PAGE (!PageHead (head), head); +VM_BUG_ON_PAGE (PageCompound (tail), head); +VM_BUG_ON_PAGE (PageLRU (tail), head);
These last two should surely have been
VM_BUG_ON_PAGE(PageCompound(tail), tail); VM_BUG_ON_PAGE(PageLRU(tail), tail);
Also, what do people think about converting these to VM_BUG_ON_PGFLAGS?
Either way:
Re: [PATCH v20 04/20] mm/thp: use head for head page in lru_add_page_tail
ÔÚ 2020/11/3 ÉÏÎç12:03, Matthew Wilcox дµÀ:
Quoted 13 lines · Alex Shi
mm/huge_memory.c static void remap_page(struct page *page, unsigned int nr) · line 2354
Quoted 10 lines
mm/huge_memory.c +4
-4-static void lru_add_page_tail (struct page *page,struct page *page_tail, +static void lru_add_page_tail (struct page *head,struct page *tail,struct lruvec *lruvec,struct list_head *list) { -VM_BUG_ON_PAGE (!PageHead (page), page); -VM_BUG_ON_PAGE (PageCompound (page_tail), page); -VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON_PAGE (!PageHead (head), head); +VM_BUG_ON_PAGE (PageCompound (tail), head); +VM_BUG_ON_PAGE (PageLRU (tail), head);These last two should surely have been
Quoted 4 lines
VM_BUG_ON_PAGE(PageCompound(tail), tail); VM_BUG_ON_PAGE(PageLRU(tail), tail);Also, what do people think about converting these to
VM_BUG_ON_PGFLAGS?
Hi Matthew,
Thanks for reminder! Looks these changes worth for another patch.
Quoted 5 lines
Either way:
Reviewed-by Matthew Wilcox (Oracle) <[email protected]>
I will take this option this time. :)
Thanks!
Alex
PATCH v20 5/20 mm/thp: Simplify lru_add_page_tail()
In reply to Alex ShiSimplify lru_add_page_tail(), there are actually only two cases possible:split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; or split_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than its page_ref_freeze().
So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.
--- mm/huge_memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
mm/huge_memory.c +6 -15
@@ -2354,25 +2354,16 @@ static void lru_add_page_tail(struct page *head, struct page *page_tail,VM_BUG_ON_PAGE (PageLRU (page_tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); -if (!list) -SetPageLRU (page_tail); - -if (likely (PageLRU (head))) -list_add_tail (&page_tail->lru, &head->lru); -else if (list) { +if (list) {/* page reclaim is reclaiming a huge page */ +VM_WARN_ON (PageLRU (head));get_page (page_tail);list_add_tail (&page_tail->lru, list); }else { -/* -* Head page has not yet been counted, as an hpage, -* so we must account for each subpage individually. -* -* Put page_tail on the list at the correct position -* so they all end up in order. -*/ -add_page_to_lru_list_tail (page_tail, lruvec, -page_lru (page_tail)); +/* head is still on lru (and we have it frozen) */ +VM_WARN_ON (!PageLRU (head)); +SetPageLRU (page_tail); +list_add_tail (&page_tail->lru, &head->lru); } }
Signature · 1 line
1.8.3.1
Re: [PATCH v20 05/20] mm/thp: Simplify lru_add_page_tail()
Quoted 9 lines · Alex Shi
Simplify
lru_add_page_tail(), there are actually only two cases possible:split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; orsplit_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than itspage_ref_freeze().So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.
Quoted 12 lines
[Hugh Dickins: rewrite commit log]Signed-off-by Alex Shi <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Kirill A. Shutemov <[email protected]>Cc Andrew Morton <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>Cc Mika Penttilä <[email protected]>
Re: [PATCH v20 05/20] mm/thp: Simplify lru_add_page_tail()
In reply to Alex Shipatch changed since varible rename in 4th patch:
From 5014c78418284f70be232a37fa3a4660a54e83c0 Mon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Fri, 10 Jul 2020 12:53:22 +0800
Subject: [PATCH v21 05/20] mm/thp: Simplify lru_add_page_tail()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplify lru_add_page_tail(), there are actually only two cases possible:split_huge_page_to_list(), with list supplied and head isolated from lru by its caller; or split_huge_page(), with NULL list and head on lru - because when head is racily isolated from lru, the isolator's reference will stop the split from getting any further than its page_ref_freeze().
So decide between the two cases by "list", but add VM_WARN_ON()s to verify that they match our lru expectations.
--- mm/huge_memory.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-)
mm/huge_memory.c +6 -14
@@ -2354,24 +2354,16 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (PageLRU (tail), head);lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); -if (!list) -SetPageLRU (tail); - -if (likely (PageLRU (head))) -list_add_tail (&tail->lru, &head->lru); -else if (list) { +if (list) {/* page reclaim is reclaiming a huge page */ +VM_WARN_ON (PageLRU (head));get_page (tail);list_add_tail (&tail->lru, list); }else { -/* -* Head page has not yet been counted, as an hpage, -* so we must account for each subpage individually. -* -* Put tail on the list at the correct position -* so they all end up in order. -*/ -add_page_to_lru_list_tail (tail, lruvec,page_lru (tail)); +/* head is still on lru (and we have it frozen) */ +VM_WARN_ON (!PageLRU (head)); +SetPageLRU (tail); +list_add_tail (&tail->lru, &head->lru); } }
Signature · 1 line
1.8.3.1
PATCH v20 6/20 mm/thp: narrow lru locking
In reply to Alex Shilru_lock and page cache xa_lock have no obvious reason to be taken one way round or the other: until now, lru_lock has been taken before page cache xa_lock, when splitting a THP; but nothing else takes them together. Reverse that ordering: let's narrow the lru locking - but leave local_irq_disable to block interrupts throughout, like before.
Hugh Dickins point: split_huge_page_to_list() was already silly, to be using the _irqsave variant: it's just been taking sleeping locks, so would already be broken if entered with interrupts enabled. So we can save passing flags argument down to __split_huge_page().
Why change the lock ordering here? That was hard to decide. One reason:
when this series reaches per-memcg lru locking, it relies on the THP's memcg to be stable when taking the lru_lock: that is now done after the THP's refcount has been frozen, which ensures page memcg cannot change.
Another reason: previously, lock_page_memcg()'s move_lock was presumed to nest inside lru_lock; but now lru_lock must nest inside (page cache lock inside) move_lock, so it becomes possible to use lock_page_memcg() to stabilize page memcg before taking its lru_lock. That is not the mechanism used in this series, but it is an option we want to keep open.
--- mm/huge_memory.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
mm/huge_memory.c +13 -12
@@ -2433,7 +2433,7 @@ static void __split_huge_page_tail(struct page *head, int tail, }static void __split_huge_page (struct page *page,struct list_head *list, -pgoff_t end,unsigned long flags) +pgoff_t end) {struct page *head =compound_head (page);pg_data_t *pgdat =page_pgdat (head);
@@ -2443,8 +2443,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,unsigned int nr =thp_nr_pages (head);int i; - lruvec =mem_cgroup_page_lruvec (head, pgdat); -/* complete memcg works before add pages to LRU */ mem_cgroup_split_huge_fixup (head);
@@ -2456,6 +2454,11 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } +/* prevent PageLRU to go away from under us, and freeze lru stats */ +spin_lock (&pgdat->lru_lock); + + lruvec =mem_cgroup_page_lruvec (head, pgdat); +for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);/* Some pages can be beyond i_size: drop them from page cache */
@@ -2475,6 +2478,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); +spin_unlock (&pgdat->lru_lock); +/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
@@ -2492,8 +2497,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,page_ref_add (head,2 );xa_unlock (&head->mapping->i_pages); } - -spin_unlock_irqrestore (&pgdat->lru_lock, flags); +local_irq_enable ();remap_page (head, nr);
@@ -2639,12 +2643,10 @@ bool can_split_huge_page(struct page *page, int *pextra_pins)int split_huge_page_to_list (struct page *page,struct list_head *list) {struct page *head =compound_head (page); -struct pglist_data *pgdata =NODE_DATA (page_to_nid (head));struct deferred_split *ds_queue =get_deferred_split_queue (head);struct anon_vma *anon_vma =NULL ;struct address_space *mapping =NULL ;int count, mapcount, extra_pins, ret; -unsigned long flags;pgoff_t end;VM_BUG_ON_PAGE (is_huge_zero_page (head), head);
@@ -2705,9 +2707,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)unmap_page (head);VM_BUG_ON_PAGE (compound_mapcount (head), head); -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock_irqsave (&pgdata->lru_lock, flags); - +/* block interrupt reentry in xa_lock and spinlock */ +local_irq_disable ();if (mapping) {XA_STATE (xas, &mapping->i_pages,page_index (head));
@@ -2737,7 +2738,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)__dec_node_page_state (head,NR_FILE_THPS ); } -__split_huge_page (page, list, end, flags); +__split_huge_page (page, list, end); ret =0 ; }else {if (IS_ENABLED (CONFIG_DEBUG_VM ) && mapcount) {
@@ -2751,7 +2752,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)spin_unlock (&ds_queue->split_queue_lock); fail:if (mapping)xa_unlock (&mapping->i_pages); -spin_unlock_irqrestore (&pgdata->lru_lock, flags); +local_irq_enable ();remap_page (head,thp_nr_pages (head)); ret = -EBUSY ; }
Signature · 1 line
1.8.3.1
PATCH v20 7/20 mm/vmscan: remove unnecessary lruvec adding
In reply to Alex ShiWe don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.
The SetPageLRU needs to be kept before put_page_testzero for list integrity, otherwise:
#0 move_pages_to_lru #1 release_pages
if !put_page_testzero
if (put_page_testzero())
!PageLRU //skip lru_lock
SetPageLRU()
list_add(&page->lru,)
list_add(&page->lru,)--- mm/vmscan.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)
mm/vmscan.c +25 -13
@@ -1852,26 +1852,30 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,while (!list_empty (list)) { page =lru_to_page (list);VM_BUG_ON_PAGE (PageLRU (page), page); +list_del (&page->lru);if (unlikely (!page_evictable (page))) { -list_del (&page->lru);spin_unlock_irq (&pgdat->lru_lock);putback_lru_page (page);spin_lock_irq (&pgdat->lru_lock);continue ; } - lruvec =mem_cgroup_page_lruvec (page, pgdat); +/* +* The SetPageLRU needs to be kept here for list integrity. +* Otherwise: +* #0 move_pages_to_lru #1 release_pages +* if !put_page_testzero +* if (put_page_testzero()) +* !PageLRU //skip lru_lock +* SetPageLRU() +* list_add(&page->lru,) +* list_add(&page->lru,) +*/ SetPageLRU (page); - lru =page_lru (page); - nr_pages =thp_nr_pages (page); -update_lru_size (lruvec, lru,page_zonenum (page), nr_pages); -list_move (&page->lru, &lruvec->lists[lru]); - -if (put_page_testzero (page)) { +if (unlikely (put_page_testzero (page))) {__ClearPageLRU (page);__ClearPageActive (page); -del_page_from_lru_list (page, lruvec, lru);if (unlikely (PageCompound (page))) {spin_unlock_irq (&pgdat->lru_lock);
@@ -1879,11 +1883,19 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,spin_lock_irq (&pgdat->lru_lock); }else list_add (&page->lru, &pages_to_free); - }else { - nr_moved += nr_pages; -if (PageActive (page)) -workingset_age_nonresident (lruvec, nr_pages); + +continue ; } + + lruvec =mem_cgroup_page_lruvec (page, pgdat); + lru =page_lru (page); + nr_pages =thp_nr_pages (page); + +update_lru_size (lruvec, lru,page_zonenum (page), nr_pages); +list_add (&page->lru, &lruvec->lists[lru]); + nr_moved += nr_pages; +if (PageActive (page)) +workingset_age_nonresident (lruvec, nr_pages); }/*
Signature · 1 line
1.8.3.1
Re: [PATCH v20 07/20] mm/vmscan: remove unnecessary lruvec adding
Quoted 14 lines · Alex Shi
We don't have to add a freeable page into lru and then remove from it. This change saves a couple of actions and makes the moving more clear.
The SetPageLRU needs to be kept before
put_page_testzerofor list integrity, otherwise:#0
move_pages_to_lru#1release_pages
if !put_page_testzeroif (put_page_testzero()) !PageLRU //skip lru_lock SetPageLRU() list_add(&page->lru,) list_add(&page->lru,)
Quoted 10 lines
[[email protected]: coding style fixes]Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <[email protected]>Cc Johannes Weiner <[email protected]>Cc Tejun Heo <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
PATCH v20 8/20 mm: page_idle_get_page() does not need lru_lock
In reply to Alex ShiFrom: Hugh Dickins <[email protected]>
It is necessary for page_idle_get_page() to recheck PageLRU() after get_page_unless_zero(), but holding lru_lock around that serves no useful purpose, and adds to lru_lock contention: delete it.
See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to lru_lock there; but __page_set_anon_rmap() now uses WRITE_ONCE(), and I see no other risk in page_idle_clear_pte_refs() using rmap_walk() (beyond the risk of racing PageAnon->PageKsm, mostly but not entirely prevented by page_count() check in ksm.c's write_protect_page(): that risk being shared with page_referenced() and not helped by lru_lock).
--- mm/page_idle.c | 4 ---- 1 file changed, 4 deletions(-)
mm/page_idle.c +0 -4
@@ -32,19 +32,15 @@static struct page *page_idle_get_page (unsigned long pfn) {struct page *page =pfn_to_online_page (pfn); -pg_data_t *pgdat;if (!page || !PageLRU (page) || !get_page_unless_zero (page))return NULL ; - pgdat =page_pgdat (page); -spin_lock_irq (&pgdat->lru_lock);if (unlikely (!PageLRU (page))) {put_page (page); page =NULL ; } -spin_unlock_irq (&pgdat->lru_lock);return page; }
Signature · 1 line
1.8.3.1
Johannes Weiner
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 9 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),
That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).
Isn't it possible, as per Minchan's description, for page->mapping to point to a struct anon_vma without PAGE_MAPPING_ANON set, and rmap thinking it's looking at a struct address_space?
Matthew Wilcox
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 20 lines · Johannes Weiner
Quoted 9 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).
Quoted 3 lines
Isn't it possible, as per Minchan's description, for
page->mappingto point to a structanon_vmawithoutPAGE_MAPPING_ANONset, and rmap thinking it's looking at a structaddress_space?
I don't think it can point to an anon_vma without the ANON bit set. Minchan's concern in that email was that it might still be NULL.
Johannes Weiner
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · Johannes Weiner
Quoted 9 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).
Quoted 6 lines
Quoted 3 lines
Isn't it possible, as per Minchan's description, for
page->mappingto point to a structanon_vmawithoutPAGE_MAPPING_ANONset, and rmap thinking it's looking at a structaddress_space?I don't think it can point to an
anon_vmawithout the ANON bit set. Minchan's concern in that email was that it might still be NULL.
Hm, no, the thread is a lengthy discussion about whether the store could be split such that page->mapping is actually pointing to something invalid (anon_vma without the PageAnon bit).
From his email:
CPU 0 CPU 1
do_anonymous_page
__page_set_anon_rmap
/* out of order happened so SetPageLRU is done ahead */
SetPageLRU(page)
/* Compilr changed store operation like below */
page->mapping = (struct address_space *) anon_vma;
/* Big stall happens */
/* idletacking judged it as LRU page so pass the page
in page_reference */
page_refernced
page_rmapping return true because page->mapping has some vaule but not complete so it calls rmap_walk_file.
it's okay to pass non-completed anon page in rmap_walk_file?
Re: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
ÔÚ 2020/11/3 ÉÏÎç4:20, Johannes Weiner дµÀ:
Quoted 33 lines · Matthew Wilcox
Quoted 21 lines · Johannes Weiner
Quoted 20 lines · Alex Shi
Quoted 9 lines
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).Quoted 6 lines
Quoted 3 lines
Isn't it possible, as per Minchan's description, for
page->mappingto point to a structanon_vmawithoutPAGE_MAPPING_ANONset, and rmap thinking it's looking at a structaddress_space?I don't think it can point to an
anon_vmawithout the ANON bit set. Minchan's concern in that email was that it might still be NULL.Hm, no, the thread is a lengthy discussion about whether the store could be split such that
page->mappingis actually pointing to something invalid (anon_vmawithout the PageAnon bit).
Quoted 8 lines
From his email:
CPU 0 CPU 1
do_anonymous_page
__page_set_anon_rmap
/* out of order happened so SetPageLRU is done ahead */ SetPageLRU(page)
This SetPageLRU done in __pagevec_lru_add_fn() which under the lru_lock protection, so the original memory barrier or lock concern isn't correct. that means, the SetPageLRU isn't possible to be here. And then no warry on right side 'CPU 1' problem.
Quoted 6 lines
/* Compilr changed store operation like below */
page->mapping= (structaddress_space*)anon_vma;
/* Big stall happens */
/* idletacking judged it as LRU page so pass the page
inpage_reference*/
page_refernced
page_referenced should be page_idle_clear_pte_refs_one here?
Quoted 5 lines
page_rmappingreturn true becausepage->mappinghas some vaule but not complete so it callsrmap_walk_file.
it's okay to pass non-completed anon page inrmap_walk_file?
For this patch, According to comments of page_idle_get_page() PageLRU just used to judge if the page is in user using. for this purpose, a unguard PageLRU check is good enough. So this patch should be fine.
Thanks
Alex
Johannes Weiner
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 43 lines · Alex Shi
Quoted 33 lines · Johannes Weiner
Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · Johannes Weiner
Quoted 9 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).Quoted 6 lines
Quoted 3 lines
Isn't it possible, as per Minchan's description, for
page->mappingto point to a structanon_vmawithoutPAGE_MAPPING_ANONset, and rmap thinking it's looking at a structaddress_space?I don't think it can point to an
anon_vmawithout the ANON bit set. Minchan's concern in that email was that it might still be NULL.Hm, no, the thread is a lengthy discussion about whether the store could be split such that
page->mappingis actually pointing to something invalid (anon_vmawithout the PageAnon bit).Quoted 8 lines
From his email:
CPU 0 CPU 1
do_anonymous_page
__page_set_anon_rmap
/* out of order happened so SetPageLRU is done ahead */ SetPageLRU(page)
Quoted 4 lines
This SetPageLRU done in
__pagevec_lru_add_fn()which under thelru_lockprotection, so the original memory barrier or lock concern isn't correct. that means, the SetPageLRU isn't possible to be here. And then no warry on right side 'CPU 1' problem.
The SetPageLRU is done under lru_lock, but the store to page->mapping is not, so what ensures ordering between them? And what prevents the compiler from tearing the store to page->mapping?
The writer does this:
CPU 0 page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock)
The concern is what CPU 1 will observe at page->mapping after observing PageLRU set on the page.
1. anon_vma + PAGE_MAPPING_ANON
That's the in-order scenario and is fine.
2. NULL
That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.
3. anon_vma without the PAGE_MAPPING_ANON bit
That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.
That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment that's referenced in the changelog of this patch but I cannot find in any tree or patch.
Re: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 44 lines · Johannes Weiner
Quoted 43 lines · Alex Shi
Quoted 41 lines · Johannes Weiner
Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · Johannes Weiner
Quoted 9 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Quoted 5 lines
and I see no other risk in
page_idle_clear_pte_refs()usingrmap_walk()(beyond the risk of racingPageAnon->PageKsm, mostly but not entirely prevented bypage_count()check in ksm.c'swrite_protect_page(): that risk being shared withpage_referenced()and not helped bylru_lock).Quoted 6 lines
Quoted 3 lines
Isn't it possible, as per Minchan's description, for
page->mappingto point to a structanon_vmawithoutPAGE_MAPPING_ANONset, and rmap thinking it's looking at a structaddress_space?I don't think it can point to an
anon_vmawithout the ANON bit set. Minchan's concern in that email was that it might still be NULL.Hm, no, the thread is a lengthy discussion about whether the store could be split such that
page->mappingis actually pointing to something invalid (anon_vmawithout the PageAnon bit).From his email:
CPU 0 CPU 1
do_anonymous_page
__page_set_anon_rmap
/* out of order happened so SetPageLRU is done ahead */ SetPageLRU(page)
Quoted 9 lines
Quoted 4 lines
This SetPageLRU done in
__pagevec_lru_add_fn()which under thelru_lockprotection, so the original memory barrier or lock concern isn't correct. that means, the SetPageLRU isn't possible to be here. And then no warry on right side 'CPU 1' problem.The SetPageLRU is done under
lru_lock, but the store topage->mappingis not, so what ensures ordering between them? And what prevents the compiler from tearing the store topage->mapping?
Right, I misunderstand the spin_lock in memory barrier. Thanks a lot for point out this.
So, is this patch fine to fix the problem?
From 8427121da195a6a386d1ce71abcb41b31211e21f Mon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Thu, 5 Nov 2020 11:38:24 +0800
Subject: [PATCH] mm/rmap: stop store reordering issue on page->mapping
Hugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:
CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)
Johannes give detailed examples how the store reordering could cause a trouble:
The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.
1. anon_vma + PAGE_MAPPING_ANON
That's the in-order scenario and is fine.
2. NULL
That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.
3. anon_vma without the PAGE_MAPPING_ANON bit
That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.
That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment.
--- mm/rmap.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
mm/rmap.c +20 -1
@@ -1054,8 +1054,27 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* w/o the WRITE_ONCE here the following scenario may happens due to +* store reordering. +* +* CPU 0 CPU 1 +* +* do_anonymous_page page_idle_clear_pte_refs +* __page_set_anon_rmap +* page->mapping = anon_vma + PAGE_MAPPING_ANON +* lru_cache_add_inactive_or_unevictable() +* SetPageLRU(page) +* rmap_walk +* if PageAnon(page) +* +* The 'SetPageLRU' may reordered before page->mapping setting, and +* page->mapping may set with anon_vma, w/o anon bit, then rmap_walk +* may goes to rmap_walk_file() for a anon page. +*/ + anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
Signature · 1 line
1.8.3.1
Quoted 35 lines
The writer does this:
CPU 0 page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock)The concern is what CPU 1 will observe at
page->mappingafter observing PageLRU set on the page.1.
anon_vma+PAGE_MAPPING_ANONThat's the in-order scenario and is fine.
2. NULL
That's possible if the
page->mappingstore gets reordered to occur after SetPageLRU. That's fine too because we check for it.3.
anon_vmawithout thePAGE_MAPPING_ANONbitThat would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to
page->mappingand I don't see anything that would prevent it.That said, I also don't see how the reader testing PageLRU under the
lru_lockwould prevent that in the first place. AFAICT we need thatWRITE_ONCE()around thepage->mappingassignment that's referenced in the changelog of this patch but I cannot find in any tree or patch.
Matthew Wilcox
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 18 lines · Alex Shi
mm/rmap.c +14
-0@@ -1054,8 +1054,27 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* w/o the WRITE_ONCE here the following scenario may happens due to +* store reordering. +* +* CPU 0 CPU 1 +* +* do_anonymous_page page_idle_clear_pte_refs +* __page_set_anon_rmap +* page->mapping = anon_vma + PAGE_MAPPING_ANON +* lru_cache_add_inactive_or_unevictable() +* SetPageLRU(page) +* rmap_walk +* if PageAnon(page) +*
mm/rmap.c static void __page_set_anon_rmap(struct page *page, · line 1079
Quoted 10 lines
mm/rmap.c +6
-1+* The 'SetPageLRU' may reordered before page->mapping setting, and +* page->mapping may set with anon_vma, w/o anon bit, then rmap_walk +* may goes to rmap_walk_file() for a anon page. +*/ + anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
I don't like these verbose comments with detailed descriptions in the source code. They're fine in changelogs, but they clutter the code, and they get outdated really quickly. My preference is for something more brief:
/*
* Prevent page->mapping from pointing to an anon_vma without
* the PAGE_MAPPING_ANON bit set. This could happen if the
* compiler stores anon_vma and then adds PAGE_MAPPING_ANON to it. */
Re: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
ÔÚ 2020/11/5 ÏÂÎç12:57, Matthew Wilcox дµÀ:
Quoted 35 lines · Alex Shi
Quoted 18 lines
(quoted diff) +14
-0@@ -1054,8 +1054,27 @@ static void __page_set_anon_rmap(struct page *page, if (!exclusive) anon_vma = anon_vma->root; + /* + * w/o the WRITE_ONCE here the following scenario may happens due to + * store reordering. + * + * CPU 0 CPU 1 + * + * do_anonymous_page page_idle_clear_pte_refs + * __page_set_anon_rmap + * page->mapping = anon_vma + PAGE_MAPPING_ANON + * lru_cache_add_inactive_or_unevictable() + * SetPageLRU(page) + * rmap_walk + * if PageAnon(page) + *Review static void __page_set_anon_rmap(struct page *page, · line 1079
Quoted 10 lines
(quoted diff) +6
-1+ * The 'SetPageLRU' may reordered before page->mapping setting, and + * page->mapping may set with anon_vma, w/o anon bit, then rmap_walk + * may goes to rmap_walk_file() for a anon page. + */ + anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON; - page->mapping = (struct address_space *) anon_vma; + WRITE_ONCE(page->mapping, (struct address_space *) anon_vma); page->index = linear_page_index(vma, address); }I don't like these verbose comments with detailed descriptions in the source code. They're fine in changelogs, but they clutter the code, and they get outdated really quickly. My preference is for something more brief:
Quoted 6 lines
/*
* Preventpage->mappingfrom pointing to ananon_vmawithout
* thePAGE_MAPPING_ANONbit set. This could happen if the
* compiler storesanon_vmaand then addsPAGE_MAPPING_ANONto it. */
Yes, it's reansonble. So is the following fine?
From f166f0d5df350c5eae1218456b9e6e1bd43434e7 Mon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Thu, 5 Nov 2020 11:38:24 +0800
Subject: [PATCH] mm/rmap: stop store reordering issue on page->mapping
Hugh Dickins and Minchan Kim observed a long time issue which discussed here, but actully the mentioned fix missed.
https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop/
The store reordering may cause problem in the scenario:
CPU 0 CPU1 do_anonymous_page page_add_new_anon_rmap() page->mapping = anon_vma + PAGE_MAPPING_ANON lru_cache_add_inactive_or_unevictable() spin_lock(lruvec->lock) SetPageLRU() spin_unlock(lruvec->lock) /* idletacking judged it as LRU * page so pass the page in * page_idle_clear_pte_refs */ page_idle_clear_pte_refs rmap_walk if PageAnon(page)
Johannes give detailed examples how the store reordering could cause a trouble:
The concern is the SetPageLRU may get reorder before 'page->mapping' setting, That would make CPU 1 will observe at page->mapping after observing PageLRU set on the page.
1. anon_vma + PAGE_MAPPING_ANON
That's the in-order scenario and is fine.
2. NULL
That's possible if the page->mapping store gets reordered to occur after SetPageLRU. That's fine too because we check for it.
3. anon_vma without the PAGE_MAPPING_ANON bit
That would be a problem and could lead to all kinds of undesirable behavior including crashes and data corruption.
Is it possible? AFAICT the compiler is allowed to tear the store to page->mapping and I don't see anything that would prevent it.
That said, I also don't see how the reader testing PageLRU under the lru_lock would prevent that in the first place. AFAICT we need that WRITE_ONCE() around the page->mapping assignment.
--- mm/rmap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/rmap.c +6 -1
@@ -1054,8 +1054,13 @@ static void __page_set_anon_rmap(struct page *page,if (!exclusive) anon_vma = anon_vma->root; +/* +* Prevent page->mapping from pointing to an anon_vma without +* the PAGE_MAPPING_ANON bit set. This could happen if the +* compiler stores anon_vma and then adds PAGE_MAPPING_ANON to it. +*/ anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
Signature · 1 line
1.8.3.1
Johannes Weiner
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 38 lines · Alex Shi
Quoted 35 lines · Matthew Wilcox
Quoted 18 lines · Alex Shi
(quoted diff) +14
-0@@ -1054,8 +1054,27 @@ static void __page_set_anon_rmap(struct page *page, if (!exclusive) anon_vma = anon_vma->root; + /* + * w/o the WRITE_ONCE here the following scenario may happens due to + * store reordering. + * + * CPU 0 CPU 1 + * + * do_anonymous_page page_idle_clear_pte_refs + * __page_set_anon_rmap + * page->mapping = anon_vma + PAGE_MAPPING_ANON + * lru_cache_add_inactive_or_unevictable() + * SetPageLRU(page) + * rmap_walk + * if PageAnon(page) + *Review static void __page_set_anon_rmap(struct page *page, · line 1079
Quoted 10 lines
(quoted diff) +6
-1+ * The 'SetPageLRU' may reordered before page->mapping setting, and + * page->mapping may set with anon_vma, w/o anon bit, then rmap_walk + * may goes to rmap_walk_file() for a anon page. + */ + anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON; - page->mapping = (struct address_space *) anon_vma; + WRITE_ONCE(page->mapping, (struct address_space *) anon_vma); page->index = linear_page_index(vma, address); }I don't like these verbose comments with detailed descriptions in the source code. They're fine in changelogs, but they clutter the code, and they get outdated really quickly. My preference is for something more brief:
Quoted 6 lines
Quoted 6 lines
/*
* Preventpage->mappingfrom pointing to ananon_vmawithout
* thePAGE_MAPPING_ANONbit set. This could happen if the
* compiler storesanon_vmaand then addsPAGE_MAPPING_ANONto it. */
Yeah, I don't think this scenario warrants the full race diagram in the code itself.
But the code is highly specific - synchronizing one struct page member for one particular use case. Let's keep at least a reference to what we are synchronizing against. There is a non-zero chance that if the comment goes out of date, so does the code. How about this?
/*
* page_idle does a lockless/optimistic rmap scan on page->mapping.
* Make sure the compiler doesn't split the stores of anon_vma and
* the PAGE_MAPPING_ANON type identifier, otherwise the rmap code
* could mistake the mapping for a struct address_space and crash. */
Matthew Wilcox
maintainerRe: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
Quoted 11 lines · Johannes Weiner
But the code is highly specific - synchronizing one struct page member for one particular use case. Let's keep at least a reference to what we are synchronizing against. There is a non-zero chance that if the comment goes out of date, so does the code. How about this?
/*
*page_idledoes a lockless/optimistic rmap scan onpage->mapping.
* Make sure the compiler doesn't split the stores ofanon_vmaand
* thePAGE_MAPPING_ANONtype identifier, otherwise the rmap code
* could mistake the mapping for a structaddress_spaceand crash. */
Fine by me! There may be other cases where seeing a split store would be bad, so I didn't want to call out page_idle explicitly. But if you want to, I'm happy with this comment.
Re: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
In reply to Johannes WeinerQuoted 10 lines · Johannes Weiner
Quoted 2 lines
Quoted 2 lines
*/Yeah, I don't think this scenario warrants the full race diagram in the code itself.
But the code is highly specific - synchronizing one struct page member for one particular use case. Let's keep at least a reference to what we are synchronizing against. There is a non-zero chance that if the comment goes out of date, so does the code. How about this?
Quoted 6 lines
/*
*page_idledoes a lockless/optimistic rmap scan onpage->mapping.
* Make sure the compiler doesn't split the stores ofanon_vmaand
* thePAGE_MAPPING_ANONtype identifier, otherwise the rmap code
* could mistake the mapping for a structaddress_spaceand crash. */
Thanks a lot to you all. I will update to v21 patch
Alex
Re: [PATCH v20 08/20] mm: page_idle_get_page() does not need lru_lock
In reply to Johannes WeinerQuoted 7 lines · Johannes Weiner
Quoted 6 lines · Alex Shi
From: Hugh Dickins <[email protected]>
It is necessary for
page_idle_get_page()to recheckPageLRU()afterget_page_unless_zero(), but holdinglru_lockaround that serves no useful purpose, and adds tolru_lockcontention: delete it.
Quoted 6 lines
Quoted 3 lines
See https://lore.kernel.org/lkml/20150504031722.GA2768@blaptop for the discussion that led to
lru_lockthere; but__page_set_anon_rmap()now usesWRITE_ONCE(),That doesn't seem to be the case in Linus's or Andrew's tree. Am I missing a dependent patch series?
Sorry, I was out of action, then slower than ever, for a while.
Many thanks for calling out my falsehood there, Johannes.
What led me to write that? It has baffled me, but at last I see:
this patch to page_idle_get_page() was 0002 in my lru_lock patchset against v5.3 last year, and 0001 was the patch which made it true. Then when I checked against mainline, I must have got confused by the similar WRITE_ONCE in page_move_anon_rmap().
Appended below, but not rediffed, and let's not hold up Alex's set for the rest of it: it is all theoretical until the kernel gets to be built with a suitably malicious compiler; but I'll follow up with a fresh version of the below after his set is safely in.
From a1abcbc2aac70c6ba47b8991992bb85b86b4a160 Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Thu, 22 Aug 2019 15:49:44 -0700
Subject: [PATCH 1/9] mm: more WRITE_ONCE and READ_ONCE on page->mapping
v4.2 commit 414e2fb8ce5a ("rmap: fix theoretical race between do_wp_page and shrink_active_list") added a WRITE_ONCE() where page_move_anon_rmap() composes page->mapping from anon_vma pointer and PAGE_MAPPING_ANON.
Now do the same where __page_set_anon_rmap() does the same, and where compaction.c applies PAGE_MAPPING_MOVABLE, and ksm.c PAGE_MAPPING_KSM.
rmap.c already uses READ_ONCE(page->mapping), but util.c should too:
add READ_ONCE() in page_rmapping(), page_anon_vma() and page_mapping(). Delete the then unused helper __page_rmapping().
I doubt that this commit fixes anything, but it's harmless and unintrusive, and makes reasoning about page mapping flags easier.
What if a compiler implements "page->mapping = mapping" in other places by, say, first assigning the odd bits of mapping, then adding in the even bits? Then we shall not build the kernel with such a compiler.
--- mm/compaction.c | 7 ++++--- mm/ksm.c | 2 +- mm/rmap.c | 7 ++++++- mm/util.c | 24 ++++++++++-------------- 4 files changed, 21 insertions(+), 19 deletions(-)
mm/compaction.c +4 -3
@@ -113,7 +113,8 @@ void __SetPageMovable(struct page *page, struct address_space *mapping) {VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE ((unsigned long )mapping &PAGE_MAPPING_MOVABLE , page); - page->mapping = (void *)((unsigned long )mapping |PAGE_MAPPING_MOVABLE ); +WRITE_ONCE (page->mapping, + (unsigned long )mapping |PAGE_MAPPING_MOVABLE ); }EXPORT_SYMBOL (__SetPageMovable);
@@ -126,8 +127,8 @@ void __ClearPageMovable(struct page *page)* flag so that VM can catch up released page by driver after isolation. * With it, VM migration doesn't try to put it back. */ - page->mapping = (void *)((unsigned long )page->mapping & -PAGE_MAPPING_MOVABLE ); +WRITE_ONCE (page->mapping, + (unsigned long )page->mapping &PAGE_MAPPING_MOVABLE ); }EXPORT_SYMBOL (__ClearPageMovable);
mm/ksm.c +1 -1
@@ -865,7 +865,7 @@ static inline struct stable_node *page_stable_node(struct page *page)static inline void set_page_stable_node (struct page *page,struct stable_node *stable_node) { - page->mapping = (void *)((unsigned long )stable_node |PAGE_MAPPING_KSM ); +WRITE_ONCE (page->mapping, (unsigned long )stable_node |PAGE_MAPPING_KSM ); }#ifdef CONFIG_SYSFS
mm/rmap.c +6 -1
@@ -1044,7 +1044,12 @@ static void __page_set_anon_rmap(struct page *page, anon_vma = anon_vma->root; anon_vma = (void *) anon_vma +PAGE_MAPPING_ANON ; - page->mapping = (struct address_space *) anon_vma; +/* +* Ensure that anon_vma and the PAGE_MAPPING_ANON bit are written +* simultaneously, so a concurrent reader (eg page_referenced()'s +* PageAnon()) will not see one without the other. +*/ +WRITE_ONCE (page->mapping, (struct address_space *) anon_vma); page->index =linear_page_index (vma, address); }
mm/util.c +10 -14
@@ -489,21 +489,14 @@ void kvfree(const void *addr) }EXPORT_SYMBOL (kvfree); -static inline void *__page_rmapping (struct page *page) -{ -unsigned long mapping; - - mapping = (unsigned long )page->mapping; - mapping &= ~PAGE_MAPPING_FLAGS ; - -return (void *)mapping; -} -/* Neutral page->mapping pointer to address_space or anon_vma or other */ void *page_rmapping (struct page *page) { +unsigned long mapping; + page =compound_head (page); -return __page_rmapping (page); + mapping = (unsigned long )READ_ONCE (page->mapping); +return (void *)(mapping & ~PAGE_MAPPING_FLAGS ); }/*
@@ -534,10 +527,11 @@ struct anon_vma *page_anon_vma(struct page *page)unsigned long mapping; page =compound_head (page); - mapping = (unsigned long )page->mapping; + mapping = (unsigned long )READ_ONCE (page->mapping); +/* Return NULL if file or PageMovable or PageKsm */ if ((mapping &PAGE_MAPPING_FLAGS ) !=PAGE_MAPPING_ANON )return NULL ; -return __page_rmapping (page); +return (struct anon_vma *)(mapping & ~PAGE_MAPPING_FLAGS ); }struct address_space *page_mapping (struct page *page)
@@ -557,10 +551,12 @@ struct address_space *page_mapping(struct page *page)return swap_address_space (entry); } - mapping = page->mapping; + mapping =READ_ONCE (page->mapping); +/* Return NULL if PageAnon (including PageKsm) */ if ((unsigned long )mapping &PAGE_MAPPING_ANON )return NULL ; +/* Return struct address_space pointer if file or PageMovable */ return (void *)((unsigned long )mapping & ~PAGE_MAPPING_FLAGS ); }EXPORT_SYMBOL (page_mapping);
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
PATCH v20 9/20 mm/memcg: add debug checking in lock_page_memcg
In reply to Alex ShiAdd a debug checking in lock_page_memcg, then we could get alarm if anything wrong here.
--- mm/memcontrol.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memcontrol.c +6 -0
@@ -2142,6 +2142,12 @@ struct mem_cgroup *lock_page_memcg(struct page *page)if (unlikely (!memcg))return NULL ; +#ifdef CONFIG_PROVE_LOCKING +local_irq_save (flags); +might_lock (&memcg->move_lock); +local_irq_restore (flags); +#endif +if (atomic_read (&memcg->moving_account) <=0 )return memcg;
Signature · 1 line
1.8.3.1
Re: [PATCH v20 09/20] mm/memcg: add debug checking in lock_page_memcg
Quoted 13 lines · Alex Shi
Add a debug checking in
lock_page_memcg, then we could get alarm if anything wrong here.Suggested-by Johannes Weiner <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
PATCH v20 10/20 mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn
In reply to Alex ShiFold the PGROTATED event collection into pagevec_move_tail_fn call back func like other funcs does in pagevec_lru_move_fn. Thus we could save func call pagevec_move_tail().
Now all usage of pagevec_lru_move_fn are same and no needs of its 3rd parameter.
It's just simply the calling. No functional change.
--- mm/swap.c | 65 ++++++++++++++++++++++----------------------------------------- 1 file changed, 23 insertions(+), 42 deletions(-)
mm/swap.c +23 -42
@@ -204,8 +204,7 @@ int get_kernel_page(unsigned long start, int write, struct page **pages)EXPORT_SYMBOL_GPL (get_kernel_page);static void pagevec_lru_move_fn (struct pagevec *pvec, -void (*move_fn)(struct page *page,struct lruvec *lruvec,void *arg), -void *arg) +void (*move_fn)(struct page *page,struct lruvec *lruvec)) {int i;struct pglist_data *pgdat =NULL ;
@@ -224,7 +223,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, } lruvec =mem_cgroup_page_lruvec (page, pgdat); - (*move_fn)(page, lruvec, arg); + (*move_fn)(page, lruvec); }if (pgdat)spin_unlock_irqrestore (&pgdat->lru_lock, flags);
@@ -232,35 +231,22 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,pagevec_reinit (pvec); } -static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec) { -int *pgmoved = arg; -if (PageLRU (page) && !PageUnevictable (page)) {del_page_from_lru_list (page, lruvec,page_lru (page));ClearPageActive (page);add_page_to_lru_list_tail (page, lruvec,page_lru (page)); - (*pgmoved) +=thp_nr_pages (page); +__count_vm_events (PGROTATED ,thp_nr_pages (page)); } }/* -* pagevec_move_tail() must be called with IRQ disabled. -* Otherwise this may cause nasty races. -*/ -static void pagevec_move_tail (struct pagevec *pvec) -{ -int pgmoved =0 ; - -pagevec_lru_move_fn (pvec, pagevec_move_tail_fn, &pgmoved); -__count_vm_events (PGROTATED , pgmoved); -} - -/* * Writeback is about to end against a page which has been marked for immediate * reclaim. If it still appears to be reclaimable, move it to the tail of the * inactive list. +* +* rotate_reclaimable_page() must disable IRQs, to prevent nasty races. */ void rotate_reclaimable_page (struct page *page) {
@@ -273,7 +259,7 @@ void rotate_reclaimable_page(struct page *page)local_lock_irqsave (&lru_rotate.lock, flags); pvec =this_cpu_ptr (&lru_rotate.pvec);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_move_tail (pvec); +pagevec_lru_move_fn (pvec, pagevec_move_tail_fn);local_unlock_irqrestore (&lru_rotate.lock, flags); } }
@@ -315,8 +301,7 @@ void lru_note_cost_page(struct page *page) page_is_file_lru(page), thp_nr_pages(page)); } -static void __activate_page (struct page *page,struct lruvec *lruvec, -void *arg) +static void __activate_page (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);
@@ -340,7 +325,7 @@ static void activate_page_drain(int cpu)struct pagevec *pvec = &per_cpu (lru_pvecs.activate_page, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, __activate_page,NULL ); +pagevec_lru_move_fn (pvec, __activate_page); }static bool need_activate_page_drain (int cpu)
@@ -358,7 +343,7 @@ static void activate_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.activate_page);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, __activate_page,NULL ); +pagevec_lru_move_fn (pvec, __activate_page);local_unlock (&lru_pvecs.lock); } }
@@ -374,7 +359,7 @@ static void activate_page(struct page *page) page =compound_head (page);spin_lock_irq (&pgdat->lru_lock); -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat),NULL ); +__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); spin_unlock_irq(&pgdat->lru_lock); }#endif
@@ -525,8 +510,7 @@ void lru_cache_add_inactive_or_unevictable(struct page *page,* be write it out by flusher threads as this is much more effective * than the single-page writeout from reclaim. */ -static void lru_deactivate_file_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_deactivate_file_fn (struct page *page,struct lruvec *lruvec) {int lru;bool active;
@@ -573,8 +557,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec, } } -static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);
@@ -591,8 +574,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec, } } -static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec) {if (PageLRU (page) &&PageAnon (page) &&PageSwapBacked (page) && !PageSwapCache (page) && !PageUnevictable (page)) {
@@ -636,21 +618,21 @@ void lru_add_drain_cpu(int cpu)/* No harm done if a racing interrupt already did this */ local_lock_irqsave (&lru_rotate.lock, flags); -pagevec_move_tail (pvec); +pagevec_lru_move_fn (pvec, pagevec_move_tail_fn);local_unlock_irqrestore (&lru_rotate.lock, flags); } pvec = &per_cpu (lru_pvecs.lru_deactivate_file, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_deactivate_file_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_file_fn); pvec = &per_cpu (lru_pvecs.lru_deactivate, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_deactivate_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_fn); pvec = &per_cpu (lru_pvecs.lru_lazyfree, cpu);if (pagevec_count (pvec)) -pagevec_lru_move_fn (pvec, lru_lazyfree_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); }
@@ -679,7 +661,7 @@ void deactivate_file_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_deactivate_file);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_deactivate_file_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_file_fn);local_unlock (&lru_pvecs.lock); } }
@@ -701,7 +683,7 @@ void deactivate_page(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_deactivate);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_deactivate_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_deactivate_fn);local_unlock (&lru_pvecs.lock); } }
@@ -723,7 +705,7 @@ void mark_page_lazyfree(struct page *page) pvec =this_cpu_ptr (&lru_pvecs.lru_lazyfree);get_page (page);if (!pagevec_add (pvec, page) ||PageCompound (page)) -pagevec_lru_move_fn (pvec, lru_lazyfree_fn,NULL ); +pagevec_lru_move_fn (pvec, lru_lazyfree_fn);local_unlock (&lru_pvecs.lock); } }
@@ -974,8 +956,7 @@ void __pagevec_release(struct pagevec *pvec) }EXPORT_SYMBOL (__pagevec_release); -static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec, -void *arg) +static void __pagevec_lru_add_fn (struct page *page,struct lruvec *lruvec) {enum lru_list lru;int was_unevictable =TestClearPageUnevictable (page);
@@ -1034,7 +1015,7 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,*/ void __pagevec_lru_add (struct pagevec *pvec) { -pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn,NULL ); +pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn); }/**
Signature · 1 line
1.8.3.1
Re: [PATCH v20 10/20] mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn
Quoted 6 lines · Alex Shi
Fold the PGROTATED event collection into
pagevec_move_tail_fncall back func like other funcs does inpagevec_lru_move_fn. Thus we could save func callpagevec_move_tail().
Now all usage ofpagevec_lru_move_fnare same and no needs of its 3rd parameter.
Quoted 8 lines
It's just simply the calling. No functional change.
[[email protected]: found a build issue in the original patch, thanks]Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <[email protected]>
Nice.
PATCH v20 11/20 mm/lru: move lock into lru_note_cost
In reply to Alex ShiWe have to move lru_lock into lru_note_cost, since it cycle up on memcg tree, for future per lruvec lru_lock replace. It's a bit ugly and may cost a bit more locking, but benefit from multiple memcg locking could cover the lost.
--- mm/swap.c | 3 +++ mm/vmscan.c | 4 +--- mm/workingset.c | 2 -- 3 files changed, 4 insertions(+), 5 deletions(-)
mm/swap.c +3 -0
@@ -268,7 +268,9 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) { do { unsigned long lrusize; + struct pglist_data *pgdat = lruvec_pgdat(lruvec); + spin_lock_irq(&pgdat->lru_lock); /* Record cost event */ if (file) lruvec->file_cost += nr_pages;
@@ -292,6 +294,7 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) lruvec->file_cost /=2 ; lruvec->anon_cost /=2 ; } +spin_unlock_irq (&pgdat->lru_lock); }while ((lruvec =parent_lruvec (lruvec))); }
mm/vmscan.c +1 -3
@@ -1973,19 +1973,17 @@ static int current_may_throttle(void) &stat, false);spin_lock_irq (&pgdat->lru_lock); -move_pages_to_lru (lruvec, &page_list);__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken); -lru_note_cost (lruvec, file, stat.nr_pageout); item =current_is_kswapd () ?PGSTEAL_KSWAPD :PGSTEAL_DIRECT ;if (!cgroup_reclaim (sc))__count_vm_events (item, nr_reclaimed);__count_memcg_events (lruvec_memcg (lruvec), item, nr_reclaimed);__count_vm_events (PGSTEAL_ANON + file, nr_reclaimed); -spin_unlock_irq (&pgdat->lru_lock); +lru_note_cost (lruvec, file, stat.nr_pageout);mem_cgroup_uncharge_list (&page_list);free_unref_page_list (&page_list);
mm/workingset.c +0 -2
@@ -381,9 +381,7 @@ void workingset_refault(struct page *page, void *shadow)if (workingset) {SetPageWorkingset (page);/* XXX: Move to lru_cache_add() when it supports new vs putback */ -spin_lock_irq (&page_pgdat (page)->lru_lock);lru_note_cost_page (page); -spin_unlock_irq (&page_pgdat (page)->lru_lock);inc_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + file); } out:
Signature · 1 line
1.8.3.1
Re: [PATCH v20 11/20] mm/lru: move lock into lru_note_cost
Quoted 11 lines · Alex Shi
We have to move
lru_lockintolru_note_cost, since it cycle up on memcg tree, for future per lruveclru_lockreplace. It's a bit ugly and may cost a bit more locking, but benefit from multiple memcg locking could cover the lost.Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
PATCH v20 12/20 mm/vmscan: remove lruvec reget in move_pages_to_lru
In reply to Alex ShiA isolated page shouldn't be recharged by memcg since the memcg migration isn't possible at the time.
So remove unnecessary regetting.
Thanks to Alexander Duyck for pointing this out.
--- mm/vmscan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/vmscan.c +2 -1
@@ -1887,7 +1887,8 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,continue ; } - lruvec =mem_cgroup_page_lruvec (page, pgdat); +VM_BUG_ON_PAGE (mem_cgroup_page_lruvec (page,page_pgdat (page)) + != lruvec, page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
Signature · 1 line
1.8.3.1
Re: [PATCH v20 12/20] mm/vmscan: remove lruvec reget in move_pages_to_lru
Quoted 6 lines · Alex Shi
A isolated page shouldn't be recharged by memcg since the memcg migration isn't possible at the time.
So remove unnecessary regetting.Thanks to Alexander Duyck for pointing this out.
Quoted 11 lines
Acked-by Hugh Dickins <[email protected]>Cc Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>
A brief comment in the code could be good: all pages were isolated from the same lruvec (and isolation inhibits memcg migration).
Re: [PATCH v20 12/20] mm/vmscan: remove lruvec reget in move_pages_to_lru
ÔÚ 2020/11/2 ÏÂÎç10:52, Johannes Weiner дµÀ:
Quoted 4 lines
Acked-by Johannes Weiner <[email protected]>A brief comment in the code could be good: all pages were isolated from the same lruvec (and isolation inhibits memcg migration).
Yes, I will add the words both in code and commit log.
Thanks
PATCH v20 13/20 mm/mlock: remove lru_lock on TestClearPageMlocked
In reply to Alex ShiIn the func munlock_vma_page, comments mentained lru_lock needed for serialization with split_huge_pages. But the page must be PageLocked as well as pages in split_huge_page series funcs. Thus the PageLocked is enough to serialize both funcs.
Further more, Hugh Dickins pointed: before splitting in split_huge_page_to_list, the page was unmap_page() to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard __split_huge_page_tail for mlock clean, just keep the lru_lock there for isolation purpose.
LKP found a preempt issue on __mod_zone_page_state which need change to mod_zone_page_state. Thanks!
--- mm/mlock.c | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-)
mm/mlock.c +5 -21
@@ -187,40 +187,24 @@ static void __munlock_isolation_failed(struct page *page)unsigned int munlock_vma_page (struct page *page) {int nr_pages; -pg_data_t *pgdat =page_pgdat (page);/* For try_to_munlock() and to serialize with page migration */ BUG_ON (!PageLocked (page)); -VM_BUG_ON_PAGE (PageTail (page), page); -/* -* Serialize with any parallel __split_huge_page_refcount() which -* might otherwise copy PageMlocked to part of the tail pages before -* we clear it in the head page. It also stabilizes thp_nr_pages(). -*/ -spin_lock_irq (&pgdat->lru_lock); -if (!TestClearPageMlocked (page)) {/* Potentially, PTE-mapped THP: do not skip the rest PTEs */ - nr_pages =1 ; - goto unlock_out; +return 0 ; } nr_pages =thp_nr_pages (page); -__mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); +mod_zone_page_state (page_zone (page),NR_MLOCK , -nr_pages); -if (__munlock_isolate_lru_page (page, true)) { -spin_unlock_irq (&pgdat->lru_lock); +if (!isolate_lru_page (page))__munlock_isolated_page (page); - goto out; - } -__munlock_isolation_failed (page); - -unlock_out: -spin_unlock_irq (&pgdat->lru_lock); +else +__munlock_isolation_failed (page); -out:return nr_pages -1 ; }
Signature · 1 line
1.8.3.1
Re: [PATCH v20 13/20] mm/mlock: remove lru_lock on TestClearPageMlocked
Quoted 14 lines · Alex Shi
In the func
munlock_vma_page, comments mentainedlru_lockneeded for serialization withsplit_huge_pages. But the page must be PageLocked as well as pages insplit_huge_pageseries funcs. Thus the PageLocked is enough to serialize both funcs.Further more, Hugh Dickins pointed: before splitting in
split_huge_page_to_list, the page wasunmap_page()to remove pmd/ptes which protect the page from munlock. Thus, no needs to guard__split_huge_page_tailfor mlock clean, just keep thelru_lockthere for isolation purpose.LKP found a preempt issue on
__mod_zone_page_statewhich need change tomod_zone_page_state. Thanks!
Quoted 7 lines
Acked-by Hugh Dickins <[email protected]>Cc Kirill A. Shutemov <kirill.shutemov-VuQAYsv1563Yd54FQh9/[email protected]>Cc Vlastimil Babka <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
PATCH v20 14/20 mm/mlock: remove __munlock_isolate_lru_page
In reply to Alex ShiThe func only has one caller, remove it to clean up code and simplify code.
--- mm/mlock.c | 31 +++++++++---------------------- 1 file changed, 9 insertions(+), 22 deletions(-)
mm/mlock.c +9 -22
@@ -106,26 +106,6 @@ void mlock_vma_page(struct page *page) }/* -* Isolate a page from LRU with optional get_page() pin. -* Assumes lru_lock already held and page already pinned. -*/ -static bool __munlock_isolate_lru_page (struct page *page,bool getpage) -{ -if (PageLRU (page)) { -struct lruvec *lruvec; - - lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (getpage) -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec,page_lru (page)); -return true; - } - -return false; -} - -/* * Finish munlock after successful page isolation * * Page must be locked. This is a wrapper for try_to_munlock()
@@ -296,9 +276,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* We already have pin from follow_page_mask() * so we can spare the get_page() here. */ -if (__munlock_isolate_lru_page (page, false)) +if (PageLRU (page)) { +struct lruvec *lruvec; + +ClearPageLRU (page); + lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +del_page_from_lru_list (page, lruvec, +page_lru (page));continue ; -else + }else __munlock_isolation_failed (page); }else { delta_munlocked++;
Signature · 1 line
1.8.3.1
Re: [PATCH v20 14/20] mm/mlock: remove __munlock_isolate_lru_page
Quoted 11 lines · Alex Shi
The func only has one caller, remove it to clean up code and simplify code.
Acked-by Hugh Dickins <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kirill A. Shutemov <kirill.shutemov-VuQAYsv1563Yd54FQh9/[email protected]>Cc Vlastimil Babka <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
PATCH v20 15/20 mm/lru: introduce TestClearPageLRU
In reply to Alex ShiCurrently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.
So now a standard steps of page isolation is following:
1, get_page(); #pin the page avoid to be free 2, TestClearPageLRU(); #block other isolation like memcg change 3, spin_lock on lru_lock; #serialize lru list access 4, delete page from lru list;
The step 2 could be optimzed/replaced in scenarios which page is unlikely be accessed or be moved between memcgs.
This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove BUG() checking accordingly.
There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.
As Andrew Morton mentioned this change would dirty cacheline for page isn't on LRU. But the lost would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/
--- include/linux/page-flags.h | 1 + mm/mlock.c | 3 +-- mm/vmscan.c | 39 +++++++++++++++++++-------------------- 3 files changed, 21 insertions(+), 22 deletions(-)
include/linux/page-flags.h +1 -0
@@ -335,6 +335,7 @@ static inline void page_init_poison(struct page *page, size_t size) PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD) __CLEARPAGEFLAG(Dirty, dirty, PF_HEAD) PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD) + TESTCLEARFLAG(LRU, lru, PF_HEAD) PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD) TESTCLEARFLAG(Active, active, PF_HEAD)PAGEFLAG (Workingset, workingset,PF_HEAD )
mm/mlock.c +1 -2
@@ -276,10 +276,9 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* We already have pin from follow_page_mask() * so we can spare the get_page() here. */ -if (PageLRU (page)) { +if (TestClearPageLRU (page)) {struct lruvec *lruvec; -ClearPageLRU (page); lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); del_page_from_lru_list(page, lruvec,
mm/vmscan.c +19 -20
@@ -1542,7 +1542,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,*/ int __isolate_lru_page (struct page *page,isolate_mode_t mode) { -int ret = -EINVAL ; +int ret = -EBUSY ;/* Only take pages on the LRU. */ if (!PageLRU (page))
@@ -1552,8 +1552,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)if (PageUnevictable (page) && !(mode &ISOLATE_UNEVICTABLE ))return ret; - ret = -EBUSY ; -/* * To minimise LRU disruption, the caller can indicate that it only * wants to isolate pages it will be able to operate on without
@@ -1600,8 +1598,10 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)* sure the page is not being freed elsewhere -- the * page release code relies on it. */ -ClearPageLRU (page); - ret =0 ; +if (TestClearPageLRU (page)) + ret =0 ; +else +put_page (page); }return ret;
@@ -1667,8 +1667,6 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, page =lru_to_page (src);prefetchw_prev_lru_page (page, src, flags); -VM_BUG_ON_PAGE (!PageLRU (page), page); - nr_pages =compound_nr (page); total_scan += nr_pages;
@@ -1765,21 +1763,18 @@ int isolate_lru_page(struct page *page)VM_BUG_ON_PAGE (!page_count (page), page);WARN_RATELIMIT (PageTail (page),"trying to isolate tail page" ); -if (PageLRU (page)) { +if (TestClearPageLRU (page)) {pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec; -spin_lock_irq (&pgdat->lru_lock); +get_page (page); lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (PageLRU (page)) { -int lru =page_lru (page); -get_page (page); -ClearPageLRU (page); -del_page_from_lru_list (page, lruvec, lru); - ret =0 ; - } +spin_lock_irq (&pgdat->lru_lock); +del_page_from_lru_list (page, lruvec,page_lru (page));spin_unlock_irq (&pgdat->lru_lock); + ret =0 ; } +return ret; }
@@ -4289,6 +4284,10 @@ void check_move_unevictable_pages(struct pagevec *pvec) nr_pages =thp_nr_pages (page); pgscanned += nr_pages; +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; +if (pagepgdat != pgdat) {if (pgdat)spin_unlock_irq (&pgdat->lru_lock);
@@ -4297,10 +4296,7 @@ void check_move_unevictable_pages(struct pagevec *pvec) } lruvec =mem_cgroup_page_lruvec (page, pgdat); -if (!PageLRU (page) || !PageUnevictable (page)) -continue ; - -if (page_evictable (page)) { +if (page_evictable (page) &&PageUnevictable (page)) {enum lru_list lru =page_lru_base_type (page);VM_BUG_ON_PAGE (PageActive (page), page);
@@ -4309,12 +4305,15 @@ void check_move_unevictable_pages(struct pagevec *pvec)add_page_to_lru_list (page, lruvec, lru); pgrescued += nr_pages; } +SetPageLRU (page); }if (pgdat) {__count_vm_events (UNEVICTABLE_PGRESCUED , pgrescued);__count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned);spin_unlock_irq (&pgdat->lru_lock); + }else if (pgscanned) { +count_vm_events (UNEVICTABLE_PGSCANNED , pgscanned); } }EXPORT_SYMBOL_GPL (check_move_unevictable_pages);
Signature · 1 line
1.8.3.1
Re: [PATCH v20 15/20] mm/lru: introduce TestClearPageLRU
Quoted 6 lines · Alex Shi
Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.
Small nit, but the use of "could" in this sentence sounds like you're describing one possible solution that isn't being taken, when in fact you are describing the chosen locking mechanism.
Replacing "could" with "will" would make things a bit clearer IMO.
Quoted 7 lines
So now a standard steps of page isolation is following:
1, get_page(); #pin the page avoid to be free 2, TestClearPageLRU(); #block other isolation like memcg change 3, spin_lock on lru_lock; #serialize lru list access 4, delete page from lru list;The step 2 could be optimzed/replaced in scenarios which page is unlikely be accessed or be moved between memcgs.
This is a bit ominous. I'd either elaborate / provide an example / clarify why some sites can deal with races - or just remove that sentence altogether from this part of the changelog.
Quoted 18 lines
This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove
BUG()checking accordingly.There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.As Andrew Morton mentioned this change would dirty cacheline for page isn't on LRU. But the lost would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/
Quoted 11 lines
Suggested-by Johannes Weiner <[email protected]>Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>
Re: [PATCH v20 15/20] mm/lru: introduce TestClearPageLRU
ÔÚ 2020/11/2 ÏÂÎç11:10, Johannes Weiner дµÀ:
Quoted 8 lines · Alex Shi
Quoted 6 lines
Currently
lru_lockstill guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could clear the lru bit out of locking and use it as pin down action to block the page isolation in memcg changing.
Quoted 6 lines
Small nit, but the use of "could" in this sentence sounds like you're describing one possible solution that isn't being taken, when in fact you are describing the chosen locking mechanism.
Replacing "could" with "will" would make things a bit clearer IMO.
Yes, 'will' is better here. Thanks!
Quoted 8 lines
Quoted 7 lines
So now a standard steps of page isolation is following:
1, get_page(); #pin the page avoid to be free 2, TestClearPageLRU(); #block other isolation like memcg change 3, spin_lock on lru_lock; #serialize lru list access 4, delete page from lru list;The step 2 could be optimzed/replaced in scenarios which page is unlikely be accessed or be moved between memcgs.
Quoted 4 lines
This is a bit ominous. I'd either elaborate / provide an example / clarify why some sites can deal with races - or just remove that sentence altogether from this part of the changelog.
A few scenarios here, so examples looks verbose or cann't describe whole. Maybe removing above 2 lines "The step 2 could be optimzed/replaced in scenarios which page is unlikely be accessed or be moved between memcgs." is better.
Thanks!
Quoted 30 lines
Quoted 18 lines
This patch start with the first part: TestClearPageLRU, which combines PageLRU check and ClearPageLRU into a macro func TestClearPageLRU. This function will be used as page isolation precondition to prevent other isolations some where else. Then there are may !PageLRU page on lru list, need to remove
BUG()checking accordingly.There 2 rules for lru bit now:
1, the lru bit still indicate if a page on lru list, just in some
temporary moment(isolating), the page may have no lru bit when it's on lru list. but the page still must be on lru list when the lru bit set.
2, have to remove lru bit before delete it from lru list.As Andrew Morton mentioned this change would dirty cacheline for page isn't on LRU. But the lost would be acceptable in Rong Chen <[email protected]> report:
https://lore.kernel.org/lkml/20200304090301.GB5972@shao2-debian/Quoted 11 lines
Suggested-by Johannes Weiner <[email protected]>Signed-off-by Alex Shi <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Andrew Morton <[email protected]>
Quoted 2 lines
Acked-by Johannes Weiner <[email protected]>
Thanks!
Alex
PATCH v20 16/20 mm/compaction: do page isolation first in compaction
In reply to Alex ShiCurrently, compaction would get the lru_lock and then do page isolation which works fine with pgdat->lru_lock, since any page isoltion would compete for the lru_lock. If we want to change to memcg lru_lock, we have to isolate the page before getting lru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcg lru_lock later.
The new page isolation use previous introduced TestClearPageLRU() + pgdat lru locking which will be changed to memcg lru lock later.
Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:
Fix lots of crashes under compaction load: isolate_migratepages_block() must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and a put_page() after get_page_unless_zero() cannot safely be done while holding locked_lruvec - it may turn out to be the final put_page(), which will take an lruvec lock when PageLRU. And move __isolate_lru_page_prepare back after get_page_unless_zero to make trylock_page() safe:trylock_page() is not safe to use at this time: its setting PG_locked can race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic __SetPageFlag().
--- include/linux/swap.h | 2 +- mm/compaction.c | 42 +++++++++++++++++++++++++++++++++--------- mm/vmscan.c | 43 ++++++++++++++++++++++--------------------- 3 files changed, 56 insertions(+), 31 deletions(-)
include/linux/swap.h +1 -1
@@ -356,7 +356,7 @@ extern void lru_cache_add_inactive_or_unevictable(struct page *page,extern unsigned long zone_reclaimable_pages (struct zone *zone);extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order,gfp_t gfp_mask,nodemask_t *mask); -extern int __isolate_lru_page (struct page *page,isolate_mode_t mode); +extern int __isolate_lru_page_prepare (struct page *page,isolate_mode_t mode); extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, unsigned long nr_pages, gfp_t gfp_mask,
mm/compaction.c +33 -9
86 lines not on this page · raw message
mm/vmscan.c +22 -21
71 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v20 16/20] mm/compaction: do page isolation first in compaction
Quoted 25 lines · Alex Shi
Currently, compaction would get the
lru_lockand then do page isolation which works fine withpgdat->lru_lock, since any page isoltion would compete for thelru_lock. If we want to change to memcglru_lock, we have to isolate the page before gettinglru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcglru_locklater.The new page isolation use previous introduced
TestClearPageLRU()+ pgdat lru locking which will be changed to memcg lru lock later.Hugh Dickins <[email protected]> fixed following bugs in this patch's early version:
Fix lots of crashes under compaction load:
isolate_migratepages_block()must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and aput_page()afterget_page_unless_zero()cannot safely be done while holdinglocked_lruvec- it may turn out to be the finalput_page(), which will take an lruvec lock when PageLRU. And move__isolate_lru_page_prepareback afterget_page_unless_zeroto maketrylock_page()safe:trylock_page()is not safe to use at this time: its settingPG_lockedcan race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic__SetPageFlag().
Quoted 7 lines
Suggested-by Johannes Weiner <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Matthew Wilcox <[email protected]>
PATCH v20 17/20 mm/swap.c: serialize memcg changes in pagevec_lru_move_fn
In reply to Alex ShiHugh Dickins' found a memcg change bug on original version:
If we want to change the pgdat->lru_lock to memcg's lruvec lock, we have to serialize mem_cgroup_move_account during pagevec_lru_move_fn. The possible bad scenario would like:
cpu 0 cpu 1 lruvec = mem_cgroup_page_lruvec() if (!isolate_lru_page()) mem_cgroup_move_account
spin_lock_irqsave(&lruvec->lru_lock <== wrong lock.
So we need TestClearPageLRU to block isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check in move_fn callee as the consequence.
__pagevec_lru_add_fn() is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so __pagevec_lru_add() goes its own way.
--- mm/swap.c | 44 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 9 deletions(-)
mm/swap.c +35 -9
@@ -222,8 +222,14 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,spin_lock_irqsave (&pgdat->lru_lock, flags); } +/* block memcg migration during page moving between lru */ +if (!TestClearPageLRU (page)) +continue ; + lruvec =mem_cgroup_page_lruvec (page, pgdat); (*move_fn)(page, lruvec); + +SetPageLRU (page); }if (pgdat)spin_unlock_irqrestore (&pgdat->lru_lock, flags);
@@ -233,7 +239,7 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,static void pagevec_move_tail_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageUnevictable (page)) { +if (!PageUnevictable (page)) {del_page_from_lru_list (page, lruvec,page_lru (page));ClearPageActive (page);add_page_to_lru_list_tail (page, lruvec,page_lru (page));
@@ -306,7 +312,7 @@ void lru_note_cost_page(struct page *page)static void __activate_page (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) && !PageActive (page) && !PageUnevictable (page)) { +if (!PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);
@@ -362,7 +368,8 @@ static void activate_page(struct page *page) page =compound_head (page);spin_lock_irq (&pgdat->lru_lock); -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); +if (PageLRU (page)) +__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); spin_unlock_irq(&pgdat->lru_lock); }#endif
@@ -519,9 +526,6 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)bool active;int nr_pages =thp_nr_pages (page); -if (!PageLRU (page)) -return ; -if (PageUnevictable (page))return ;
@@ -562,7 +566,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec)static void lru_deactivate_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageActive (page) && !PageUnevictable (page)) { +if (PageActive (page) && !PageUnevictable (page)) {int lru =page_lru_base_type (page);int nr_pages =thp_nr_pages (page);
@@ -579,7 +583,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec)static void lru_lazyfree_fn (struct page *page,struct lruvec *lruvec) { -if (PageLRU (page) &&PageAnon (page) &&PageSwapBacked (page) && +if (PageAnon (page) &&PageSwapBacked (page) && !PageSwapCache (page) && !PageUnevictable (page)) {bool active =PageActive (page);int nr_pages =thp_nr_pages (page);
@@ -1018,7 +1022,29 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)*/ void __pagevec_lru_add (struct pagevec *pvec) { -pagevec_lru_move_fn (pvec, __pagevec_lru_add_fn); +int i; +struct pglist_data *pgdat =NULL ; +struct lruvec *lruvec; +unsigned long flags =0 ; + +for (i =0 ; i <pagevec_count (pvec); i++) { +struct page *page = pvec->pages[i]; +struct pglist_data *pagepgdat =page_pgdat (page); + +if (pagepgdat != pgdat) { +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); + pgdat = pagepgdat; +spin_lock_irqsave (&pgdat->lru_lock, flags); + } + + lruvec =mem_cgroup_page_lruvec (page, pgdat); +__pagevec_lru_add_fn (page, lruvec); + } +if (pgdat) +spin_unlock_irqrestore (&pgdat->lru_lock, flags); +release_pages (pvec->pages, pvec->nr); +pagevec_reinit (pvec); }/**
Signature · 1 line
1.8.3.1
Re: [PATCH v20 17/20] mm/swap.c: serialize memcg changes in pagevec_lru_move_fn
Quoted 21 lines · Alex Shi
Hugh Dickins' found a memcg change bug on original version:
If we want to change thepgdat->lru_lockto memcg's lruvec lock, we have to serializemem_cgroup_move_accountduringpagevec_lru_move_fn. The possible bad scenario would like:cpu 0 cpu 1 lruvec = mem_cgroup_page_lruvec() if (!isolate_lru_page()) mem_cgroup_move_accountspin_lock_irqsave(&lruvec->lru_lock <== wrong lock.
So we need TestClearPageLRU to block
isolate_lru_page(), that serializes the memcg change. and then removing the PageLRU check inmove_fncallee as the consequence.
__pagevec_lru_add_fn()is different from the others, because the pages it deals with are, by definition, not yet on the lru. TestClearPageLRU is not needed and would not work, so__pagevec_lru_add()goes its own way.
Quoted 6 lines
Reported-by Hugh Dickins <[email protected]>Acked-by Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>
PATCH v20 18/20 mm/lru: replace pgdat lru_lock with lruvec lock
In reply to Alex ShiThis patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.
After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func isolate_migratepages_block, compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/[email protected]/
On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to lru_lock holding time increase and a bit regression.
Hugh Dickins helped on the patch polish, thanks!
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 +++++++++++++++--------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 62 ++++++++++++++++++++++++-- mm/mlock.c | 22 +++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 105 +++++++++++++++++++++------------------------ mm/vmscan.c | 55 +++++++++++------------- 10 files changed, 249 insertions(+), 125 deletions(-)
include/linux/memcontrol.h +58 -0
@@ -497,6 +497,19 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct mem_cgroup *get_mem_cgroup_from_page (struct page *page); +struct lruvec *lock_page_lruvec (struct page *page); +struct lruvec *lock_page_lruvec_irq (struct page *page); +struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flags); + +#ifdef CONFIG_DEBUG_VM +void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page); +#else +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +} +#endif +static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1038,6 +1051,31 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } +static inline struct lruvec *lock_page_lruvec (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irq (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flagsp) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp); +return &pgdat->__lruvec; +} +static inline struct mem_cgroup * mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup *prev,
@@ -1285,6 +1323,10 @@ static inline void count_memcg_page_event(struct page *page,void count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +}#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1414,6 +1456,22 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void unlock_page_lruvec (struct lruvec *lruvec) +{ +spin_unlock (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +{ +spin_unlock_irq (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +unsigned long flags) +{ +spin_unlock_irqrestore (&lruvec->lru_lock, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
include/linux/mmzone.h +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {struct lruvec {struct list_head lists[NR_LRU_LISTS ]; +/* per lruvec lru_lock for memcg */ +spinlock_t lru_lock;/* * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU
@@ -796,7 +798,6 @@ struct deferred_split {/* Write-intensive fields used by page reclaim */ ZONE_PADDING (_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT/*
mm/compaction.c +36 -20
105 lines not on this page · raw message
mm/huge_memory.c +4 -7
@@ -2352,7 +2352,7 @@ static void lru_add_page_tail(struct page *head, struct page *page_tail,VM_BUG_ON_PAGE (!PageHead (head), head);VM_BUG_ON_PAGE (PageCompound (page_tail), head);VM_BUG_ON_PAGE (PageLRU (page_tail), head); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); +lockdep_assert_held (&lruvec->lru_lock);if (list) {/* page reclaim is reclaiming a huge page */
@@ -2436,7 +2436,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) {struct page *head =compound_head (page); -pg_data_t *pgdat =page_pgdat (head);struct lruvec *lruvec;struct address_space *swap_cache =NULL ;unsigned long offset =0 ;
@@ -2454,10 +2453,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock (&pgdat->lru_lock); - - lruvec =mem_cgroup_page_lruvec (head, pgdat); +/* lock lru list/PageCompound, ref freezed by page_ref_freeze */ + lruvec =lock_page_lruvec (head);for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);
@@ -2478,7 +2475,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); -spin_unlock (&pgdat->lru_lock); +unlock_page_lruvec (lruvec);/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
mm/memcontrol.c +59 -3
81 lines not on this page · raw message
mm/mlock.c +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)int nr =pagevec_count (pvec);int delta_munlocked = -nr;struct pagevec pvec_putback; +struct lruvec *lruvec =NULL ;int pgrescued =0 ;pagevec_init (&pvec_putback);/* Phase 1: page isolation */ -spin_lock_irq (&zone->zone_pgdat->lru_lock);for (i =0 ; i < nr; i++) {struct page *page = pvec->pages[i];
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page); + } - lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page));del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)pagevec_add (&pvec_putback, pvec->pages[i]); pvec->pages[i] =NULL ; } -__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); -spin_unlock_irq (&zone->zone_pgdat->lru_lock); +if (lruvec) { +__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); +unlock_page_lruvec_irq (lruvec); + }else if (delta_munlocked) { +mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); + }/* Now we can release pins of pages that we are not munlocking */ pagevec_release (&pvec_putback);
mm/mmzone.c +1 -0
@@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec)enum lru_list lru;memset (lruvec,0 ,sizeof (struct lruvec )); +spin_lock_init (&lruvec->lru_lock);for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]);
mm/page_alloc.c +0 -1
@@ -6791,7 +6791,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->pfmemalloc_wait);pgdat_page_ext_init (pgdat); -spin_lock_init (&pgdat->lru_lock);lruvec_init (&pgdat->__lruvec); }
mm/swap.c +48 -57
215 lines not on this page · raw message
mm/vmscan.c +25 -30
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v20 18/20] mm/lru: replace pgdat lru_lock with lruvec lock
patch changed since variable renaming in 04th patch:
From e892e74a35c27e69bebb73d2e4cff54e438f8d7d Mon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Tue, 18 Aug 2020 16:44:21 +0800
Subject: [PATCH v21 18/20] mm/lru: replace pgdat lru_lock with lruvec lock
This patch moves per node lru_lock into lruvec, thus bring a lru_lock for each of memcg per node. So on a large machine, each of memcg don't have to suffer from per node pgdat->lru_lock competition. They could go fast with their self lru_lock.
After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func isolate_migratepages_block, compact_unlock_should_abort and lock_page_lruvec_irqsave are open coded to work with compact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.
Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/[email protected]/
On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to lru_lock holding time increase and a bit regression.
Hugh Dickins helped on the patch polish, thanks!
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 +++++++++++++++--------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 62 ++++++++++++++++++++++++-- mm/mlock.c | 22 +++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 105 +++++++++++++++++++++------------------------ mm/vmscan.c | 55 +++++++++++------------- 10 files changed, 249 insertions(+), 125 deletions(-)
include/linux/memcontrol.h +58 -0
@@ -497,6 +497,19 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct mem_cgroup *get_mem_cgroup_from_page (struct page *page); +struct lruvec *lock_page_lruvec (struct page *page); +struct lruvec *lock_page_lruvec_irq (struct page *page); +struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flags); + +#ifdef CONFIG_DEBUG_VM +void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page); +#else +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +} +#endif +static inline struct mem_cgroup *mem_cgroup_from_css (struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL;
@@ -1038,6 +1051,31 @@ static inline void mem_cgroup_put(struct mem_cgroup *memcg) { } +static inline struct lruvec *lock_page_lruvec (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irq (&pgdat->__lruvec.lru_lock); +return &pgdat->__lruvec; +} + +static inline struct lruvec *lock_page_lruvec_irqsave (struct page *page, +unsigned long *flagsp) +{ +struct pglist_data *pgdat =page_pgdat (page); + +spin_lock_irqsave (&pgdat->__lruvec.lru_lock, *flagsp); +return &pgdat->__lruvec; +} +static inline struct mem_cgroup * mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup *prev,
@@ -1285,6 +1323,10 @@ static inline void count_memcg_page_event(struct page *page,void count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void lruvec_memcg_debug (struct lruvec *lruvec,struct page *page) +{ +}#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
@@ -1414,6 +1456,22 @@ static inline struct lruvec *parent_lruvec(struct lruvec *lruvec)return mem_cgroup_lruvec (memcg,lruvec_pgdat (lruvec)); } +static inline void unlock_page_lruvec (struct lruvec *lruvec) +{ +spin_unlock (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irq (struct lruvec *lruvec) +{ +spin_unlock_irq (&lruvec->lru_lock); +} + +static inline void unlock_page_lruvec_irqrestore (struct lruvec *lruvec, +unsigned long flags) +{ +spin_unlock_irqrestore (&lruvec->lru_lock, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
include/linux/mmzone.h +2 -1
@@ -276,6 +276,8 @@ enum lruvec_flags {struct lruvec {struct list_head lists[NR_LRU_LISTS ]; +/* per lruvec lru_lock for memcg */ +spinlock_t lru_lock;/* * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU
@@ -796,7 +798,6 @@ struct deferred_split {/* Write-intensive fields used by page reclaim */ ZONE_PADDING (_pad1_) - spinlock_t lru_lock; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT/*
mm/compaction.c +36 -20
105 lines not on this page · raw message
mm/huge_memory.c +4 -7
@@ -2352,7 +2352,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail,VM_BUG_ON_PAGE (!PageHead (head), head);VM_BUG_ON_PAGE (PageCompound (tail), head);VM_BUG_ON_PAGE (PageLRU (tail), head); -lockdep_assert_held (&lruvec_pgdat (lruvec)->lru_lock); +lockdep_assert_held (&lruvec->lru_lock);if (list) {/* page reclaim is reclaiming a huge page */
@@ -2436,7 +2436,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) {struct page *head =compound_head (page); -pg_data_t *pgdat =page_pgdat (head);struct lruvec *lruvec;struct address_space *swap_cache =NULL ;unsigned long offset =0 ;
@@ -2454,10 +2453,8 @@ static void __split_huge_page(struct page *page, struct list_head *list,xa_lock (&swap_cache->i_pages); } -/* prevent PageLRU to go away from under us, and freeze lru stats */ -spin_lock (&pgdat->lru_lock); - - lruvec =mem_cgroup_page_lruvec (head, pgdat); +/* lock lru list/PageCompound, ref freezed by page_ref_freeze */ + lruvec =lock_page_lruvec (head);for (i = nr -1 ; i >=1 ; i--) {__split_huge_page_tail (head, i, lruvec, list);
@@ -2478,7 +2475,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, }ClearPageCompound (head); -spin_unlock (&pgdat->lru_lock); +unlock_page_lruvec (lruvec);/* Caller disabled irqs, so they are still disabled here */ split_page_owner (head, nr);
mm/memcontrol.c +59 -3
81 lines not on this page · raw message
mm/mlock.c +16 -6
@@ -262,12 +262,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)int nr =pagevec_count (pvec);int delta_munlocked = -nr;struct pagevec pvec_putback; +struct lruvec *lruvec =NULL ;int pgrescued =0 ;pagevec_init (&pvec_putback);/* Phase 1: page isolation */ -spin_lock_irq (&zone->zone_pgdat->lru_lock);for (i =0 ; i < nr; i++) {struct page *page = pvec->pages[i];
@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page); + } - lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page));del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
@@ -299,8 +305,12 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)pagevec_add (&pvec_putback, pvec->pages[i]); pvec->pages[i] =NULL ; } -__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); -spin_unlock_irq (&zone->zone_pgdat->lru_lock); +if (lruvec) { +__mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); +unlock_page_lruvec_irq (lruvec); + }else if (delta_munlocked) { +mod_zone_page_state (zone,NR_MLOCK , delta_munlocked); + }/* Now we can release pins of pages that we are not munlocking */ pagevec_release (&pvec_putback);
mm/mmzone.c +1 -0
@@ -91,6 +91,7 @@ void lruvec_init(struct lruvec *lruvec)enum lru_list lru;memset (lruvec,0 ,sizeof (struct lruvec )); +spin_lock_init (&lruvec->lru_lock);for_each_lru (lru)INIT_LIST_HEAD (&lruvec->lists[lru]);
mm/page_alloc.c +0 -1
@@ -6791,7 +6791,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)init_waitqueue_head (&pgdat->pfmemalloc_wait);pgdat_page_ext_init (pgdat); -spin_lock_init (&pgdat->lru_lock);lruvec_init (&pgdat->__lruvec); }
mm/swap.c +48 -57
215 lines not on this page · raw message
mm/vmscan.c +25 -30
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Johannes Weiner
maintainerRe: [PATCH v20 18/20] mm/lru: replace pgdat lru_lock with lruvec lock
Quoted 48 lines · Alex Shi
patch changed since variable renaming in 04th patch:
From
e892e74a35c27e69bebb73d2e4cff54e438f8d7dMon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Tue, 18 Aug 2020 16:44:21 +0800
Subject: [PATCH v21 18/20] mm/lru: replace pgdatlru_lockwith lruvec lockThis patch moves per node
lru_lockinto lruvec, thus bring alru_lockfor each of memcg per node. So on a large machine, each of memcg don't have to suffer from per nodepgdat->lru_lockcompetition. They could go fast with their selflru_lock.After move memcg charge before lru inserting, page isolation could serialize page's memcg, then per memcg lruvec lock is stable and could replace per node lru lock.
In func
isolate_migratepages_block,compact_unlock_should_abortandlock_page_lruvec_irqsaveare open coded to work withcompact_control. Also add a debug func in locking which may give some clues if there are sth out of hands.Daniel Jordan's testing show 62% improvement on modified readtwice case on his 2P * 10 core * 2 HT broadwell box.
https://lore.kernel.org/lkml/20200915165807.kpp7uhiw7l3loofu-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/On a large machine with memcg enabled but not used, the page's lruvec seeking pass a few pointers, that may lead to
lru_lockholding time increase and a bit regression.Hugh Dickins helped on the patch polish, thanks!
Acked-by Hugh Dickins <[email protected]>Cc Rong Chen <[email protected]>Cc Hugh Dickins <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Vladimir Davydov <[email protected]>Cc Yang Shi <yang.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>Cc Matthew Wilcox <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Tejun Heo <[email protected]>
Quoted 12 lines
--- include/linux/memcontrol.h | 58 +++++++++++++++++++++++++ include/linux/mmzone.h | 3 +- mm/compaction.c | 56 +++++++++++++++--------- mm/huge_memory.c | 11 ++--- mm/memcontrol.c | 62 ++++++++++++++++++++++++-- mm/mlock.c | 22 +++++++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/swap.c | 105 +++++++++++++++++++++------------------------ mm/vmscan.c | 55 +++++++++++------------- 10 files changed, 249 insertions(+), 125 deletions(-)
This came out really well. Thanks for persisting!
A few inline comments:
Quoted 38 lines
mm/memcontrol.c +34
-0@@ -1367,6 +1380,51 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgdreturn lruvec; } +struct lruvec *lock_page_lruvec (struct page *page) +{ +struct lruvec *lruvec; +struct pglist_data *pgdat =page_pgdat (page); + +rcu_read_lock (); + lruvec =mem_cgroup_page_lruvec (page, pgdat); +spin_lock (&lruvec->lru_lock); +rcu_read_unlock (); + +lruvec_memcg_debug (lruvec, page); + +return lruvec; +} + +struct lruvec *lock_page_lruvec_irq (struct page *page) +{ +struct lruvec *lruvec; +struct pglist_data *pgdat =page_pgdat (page); + +rcu_read_lock (); + lruvec =mem_cgroup_page_lruvec (page, pgdat); +spin_lock_irq (&lruvec->lru_lock); +rcu_read_unlock (); + +lruvec_memcg_debug (lruvec, page); + +return lruvec; +} + +struct lruvec *lock_page_lruvec_irqsave (struct page *page,unsigned long *flags) +{ +struct lruvec *lruvec; +struct pglist_data *pgdat =page_pgdat (page);
mm/memcontrol.c struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1426
Quoted 10 lines
mm/memcontrol.c +10
-0+ +rcu_read_lock (); + lruvec =mem_cgroup_page_lruvec (page, pgdat); +spin_lock_irqsave (&lruvec->lru_lock, *flags); +rcu_read_unlock (); + +lruvec_memcg_debug (lruvec, page); + +return lruvec; +}
As these are used quite widely throughout the VM code now, it would be good to give them kerneldoc comments that explain the interface.
In particular, I think it's necessary to explain the contexts from which this is safe to use (in particular wrt pages moving between memcgs - see the comment in commit_charge()).
I'm going to go through the callsites that follow and try to identify what makes them safe. It's mostly an exercise to double check our thinking here.
Most of them are straight-forward, and I don't think they warrant individual comments. But some do, IMO. And it appears at least one actually isn't safe yet:
mm/mlock.c static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone) · line 287
Quoted 13 lines
mm/mlock.c +8
-1@@ -277,10 +277,16 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *lruvec; +struct lruvec *new_lruvec; + + new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page);
This is safe because PageLRU has been cleared.
Quoted 6 lines
mm/swap.c +0
-1@@ -79,16 +79,14 @@ static DEFINE_PER_CPU(struct lru_pvecs, lru_pvecs) = {static void __page_cache_release (struct page *page) {if (PageLRU (page)) { -pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec;
mm/swap.c static DEFINE_PER_CPU(struct lru_pvecs, lru_pvecs) = { · line 89
Quoted 10 lines
mm/swap.c +2
-3unsigned long flags; -spin_lock_irqsave (&pgdat->lru_lock, flags); - lruvec =mem_cgroup_page_lruvec (page, pgdat); + lruvec =lock_page_lruvec_irqsave (page, &flags);VM_BUG_ON_PAGE (!PageLRU (page), page);__ClearPageLRU (page);del_page_from_lru_list (page, lruvec,page_off_lru (page)); -spin_unlock_irqrestore (&pgdat->lru_lock, flags); +unlock_page_lruvec_irqrestore (lruvec, flags);
This is safe because the page refcount is 0.
Quoted 21 lines
mm/swap.c +2
-10@@ -207,32 +205,30 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, void (*move_fn)(struct page *page, struct lruvec *lruvec)) {int i; -struct pglist_data *pgdat =NULL ; -struct lruvec *lruvec; +struct lruvec *lruvec =NULL ;unsigned long flags =0 ;for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct pglist_data *pagepgdat =page_pgdat (page); - -if (pagepgdat != pgdat) { -if (pgdat) -spin_unlock_irqrestore (&pgdat->lru_lock, flags); - pgdat = pagepgdat; -spin_lock_irqsave (&pgdat->lru_lock, flags); - } +struct lruvec *new_lruvec;
mm/swap.c static void pagevec_lru_move_fn(struct pagevec *pvec, · line 223
Quoted 10 lines
mm/swap.c +5
-1/* block memcg migration during page moving between lru */ if (!TestClearPageLRU (page))continue ; - lruvec =mem_cgroup_page_lruvec (page, pgdat); + new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); +if (lruvec != new_lruvec) { +if (lruvec) +unlock_page_lruvec_irqrestore (lruvec, flags); + lruvec =lock_page_lruvec_irqsave (page, &flags);
This is safe because PageLRU has been cleared.
Quoted 12 lines
mm/swap.c +1
-2@@ -274,9 +270,8 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) { do { unsigned long lrusize; - struct pglist_data *pgdat = lruvec_pgdat(lruvec); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&lruvec->lru_lock); /* Record cost event */ if (file) lruvec->file_cost += nr_pages;
mm/swap.c void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) · line 300
Quoted 7 lines
mm/swap.c +1
-1@@ -300,7 +295,7 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) lruvec->file_cost /=2 ; lruvec->anon_cost /=2 ; } -spin_unlock_irq (&pgdat->lru_lock); +spin_unlock_irq (&lruvec->lru_lock); }while ((lruvec =parent_lruvec (lruvec))); }
This is safe because it either comes from
1) the pinned lruvec in reclaim, or
2) from a pre-LRU page during refault (which also holds the rcu lock, so would be safe even if the page was on the LRU and could move simultaneously to a new lruvec).
The second one seems a bit tricky. It could be good to add a comment to lru_note_cost_page() that explains why its mem_cgroup_page_lruvec() is safe.
mm/swap.c static inline void activate_page_drain(int cpu) · line 365
Quoted 10 lines
mm/swap.c +2
-2@@ -364,13 +359,13 @@ static inline void activate_page_drain(int cpu)static void activate_page (struct page *page) { -pg_data_t *pgdat =page_pgdat (page); +struct lruvec *lruvec; page =compound_head (page); -spin_lock_irq (&pgdat->lru_lock); + lruvec =lock_page_lruvec_irq (page);
I don't see what makes this safe. There is nothing that appears to lock out a concurrent page move between memcgs/lruvecs, which means the following could manipulate an unlocked lru list:
mm/swap.c static inline void activate_page_drain(int cpu) · line 370
Quoted 7 lines
mm/swap.c +2
-2if (PageLRU (page)) -__activate_page (page,mem_cgroup_page_lruvec (page, pgdat)); - spin_unlock_irq(&pgdat->lru_lock); +__activate_page (page, lruvec); + unlock_page_lruvec_irq(lruvec); }#endif
Shouldn't this be something like this?
if (TestClearPageLRU()) {
lruvec = lock_page_lruvec_irq(page);
__activate_page(page, lruvec);
unlock_page_lruvec_irq(lruvec);
SetPageLRU(page);
}Quoted 22 lines
mm/swap.c +4
-7@@ -904,27 +897,27 @@ void release_pages(struct page **pages, int nr)continue ;if (PageCompound (page)) { -if (locked_pgdat) { -spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags); - locked_pgdat =NULL ; +if (lruvec) { +unlock_page_lruvec_irqrestore (lruvec, flags); + lruvec =NULL ; }__put_compound_page (page);continue ; }if (PageLRU (page)) { -struct pglist_data *pgdat =page_pgdat (page); +struct lruvec *new_lruvec; -if (pgdat != locked_pgdat) { -if (locked_pgdat) - spin_unlock_irqrestore(&locked_pgdat->lru_lock,
mm/swap.c void release_pages(struct page **pages, int nr) · line 918
Quoted 10 lines
mm/swap.c +6
-2+ new_lruvec =mem_cgroup_page_lruvec (page, +page_pgdat (page)); +if (new_lruvec != lruvec) { +if (lruvec) +unlock_page_lruvec_irqrestore (lruvec, flags); lock_batch =0 ; - locked_pgdat = pgdat; -spin_lock_irqsave (&locked_pgdat->lru_lock, flags); + lruvec =lock_page_lruvec_irqsave (page, &flags);
Safe because page refcount=0.
Quoted 14 lines
mm/swap.c +2
-3@@ -1023,26 +1016,24 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec)void __pagevec_lru_add (struct pagevec *pvec) {int i; -struct pglist_data *pgdat =NULL ; -struct lruvec *lruvec; +struct lruvec *lruvec =NULL ;unsigned long flags =0 ;for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct pglist_data *pagepgdat =page_pgdat (page); +struct lruvec *new_lruvec;
mm/swap.c static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec) · line 1030
Quoted 10 lines
mm/swap.c +5
-5-if (pagepgdat != pgdat) { -if (pgdat) -spin_unlock_irqrestore (&pgdat->lru_lock, flags); - pgdat = pagepgdat; -spin_lock_irqsave (&pgdat->lru_lock, flags); + new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); +if (lruvec != new_lruvec) { +if (lruvec) +unlock_page_lruvec_irqrestore (lruvec, flags); + lruvec =lock_page_lruvec_irqsave (page, &flags);
Safe because PageLRU hasn't been set yet.
mm/vmscan.c int isolate_lru_page(struct page *page) · line 1771
Quoted 11 lines
mm/vmscan.c +1
-3@@ -1765,14 +1765,12 @@ int isolate_lru_page(struct page *page)WARN_RATELIMIT (PageTail (page),"trying to isolate tail page" );if (TestClearPageLRU (page)) { -pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec;get_page (page); - lruvec =mem_cgroup_page_lruvec (page, pgdat); -spin_lock_irq (&pgdat->lru_lock); + lruvec =lock_page_lruvec_irq (page);
Safe because PageLRU is cleared.
Quoted 9 lines
mm/vmscan.c +0
-1@@ -1839,7 +1837,6 @@ static int too_many_isolated(struct pglist_data *pgdat, int file,static unsigned noinline_for_stack move_pages_to_lru (struct lruvec *lruvec,struct list_head *list) { -struct pglist_data *pgdat =lruvec_pgdat (lruvec);int nr_pages, nr_moved =0 ;LIST_HEAD (pages_to_free);struct page *page;
mm/vmscan.c static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, · line 1850
Quoted 5 lines
mm/vmscan.c +1
-1@@ -1850,9 +1847,9 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,VM_BUG_ON_PAGE (PageLRU (page), page);list_del (&page->lru);if (unlikely (!page_evictable (page))) { -spin_unlock_irq (&pgdat->lru_lock); +spin_unlock_irq (&lruvec->lru_lock);
[snipped all the reclaim lock sites as they start with lruvec]
Quoted 4 lines
mm/vmscan.c +0
-0@@ -4289,13 +4285,12 @@ void check_move_unevictable_pages(struct pagevec *pvec)if (!TestClearPageLRU (page))continue ;
mm/vmscan.c void check_move_unevictable_pages(struct pagevec *pvec) · line 4292
Quoted 10 lines
mm/vmscan.c +5
-5-if (pagepgdat != pgdat) { -if (pgdat) -spin_unlock_irq (&pgdat->lru_lock); - pgdat = pagepgdat; -spin_lock_irq (&pgdat->lru_lock); + new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); +if (lruvec != new_lruvec) { +if (lruvec) +unlock_page_lruvec_irq (lruvec); + lruvec =lock_page_lruvec_irq (page);
Safe because PageLRU is clear.
Re: [PATCH v20 18/20] mm/lru: replace pgdat lru_lock with lruvec lock
ÔÚ 2020/11/3 ÉÏÎç4:41, Johannes Weiner дµÀ:
Quoted 59 lines · Alex Shi
Quoted 40 lines
(quoted diff) +34
-0@@ -1367,6 +1380,51 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd return lruvec; } +struct lruvec *lock_page_lruvec(struct page *page) +{ + struct lruvec *lruvec; + struct pglist_data *pgdat = page_pgdat(page); + + rcu_read_lock(); + lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock(&lruvec->lru_lock); + rcu_read_unlock(); + + lruvec_memcg_debug(lruvec, page); + + return lruvec; +} + +struct lruvec *lock_page_lruvec_irq(struct page *page) +{ + struct lruvec *lruvec; + struct pglist_data *pgdat = page_pgdat(page); + + rcu_read_lock(); + lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irq(&lruvec->lru_lock); + rcu_read_unlock(); + + lruvec_memcg_debug(lruvec, page); + + return lruvec; +} + +struct lruvec *lock_page_lruvec_irqsave(struct page *page, unsigned long *flags) +{ + struct lruvec *lruvec; + struct pglist_data *pgdat = page_pgdat(page);Review struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1426
Quoted 10 lines
(quoted diff) +10
-0+ + rcu_read_lock(); + lruvec = mem_cgroup_page_lruvec(page, pgdat); + spin_lock_irqsave(&lruvec->lru_lock, *flags); + rcu_read_unlock(); + + lruvec_memcg_debug(lruvec, page); + + return lruvec; +}As these are used quite widely throughout the VM code now, it would be good to give them kerneldoc comments that explain the interface.
In particular, I think it's necessary to explain the contexts from which this is safe to use (in particular wrt pages moving between memcgs - see the comment in
commit_charge()).
Quoted 7 lines
I'm going to go through the callsites that follow and try to identify what makes them safe. It's mostly an exercise to double check our thinking here.
Most of them are straight-forward, and I don't think they warrant individual comments. But some do, IMO. And it appears at least one actually isn't safe yet:
Thanks a lot reminder. is the following comments fine?
/**
* lock_page_lruvec - return lruvec for the locked page.
* @page: the page
*
* This series functions should be used in either conditions:
* PageLRU is cleared or unset
* or, page->_refcount is zero
*/
struct lruvec *lock_page_lruvec(struct page *page) {
....
Quoted 24 lines
Quoted 12 lines
(quoted diff) +1
-2@@ -274,9 +270,8 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) { do { unsigned long lrusize; - struct pglist_data *pgdat = lruvec_pgdat(lruvec); - spin_lock_irq(&pgdat->lru_lock); + spin_lock_irq(&lruvec->lru_lock); /* Record cost event */ if (file) lruvec->file_cost += nr_pages;Review void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) · line 300
Quoted 7 lines
(quoted diff) +1
-1@@ -300,7 +295,7 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) lruvec->file_cost /= 2; lruvec->anon_cost /= 2; } - spin_unlock_irq(&pgdat->lru_lock); + spin_unlock_irq(&lruvec->lru_lock); } while ((lruvec = parent_lruvec(lruvec))); }This is safe because it either comes from
1) the pinned lruvec in reclaim, or
Quoted 7 lines
2) from a pre-LRU page during refault (which also holds the rcu lock, so would be safe even if the page was on the LRU and could move simultaneously to a new lruvec).
The second one seems a bit tricky. It could be good to add a comment to
lru_note_cost_page()that explains why itsmem_cgroup_page_lruvec()is safe.
Thanks for pointed, is the following comments fine?
mm/swap.c +7 -0
@@ -264,6 +264,13 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) do { unsigned long lrusize; + /* + * Hold lruvec->lru_lock is safe here, since + * 1) The pinned lruvec in reclaim, or + * 2) From a pre-LRU page during refault (which also holds the + * rcu lock, so would be safe even if the page was on the LRU + * and could move simultaneously to a new lruvec). + */ spin_lock_irq(&lruvec->lru_lock); /* Record cost event */
Quoted 8 lines
Quoted 8 lines
(quoted diff) +1
-1@@ -364,13 +359,13 @@ static inline void activate_page_drain(int cpu) static void activate_page(struct page *page) { - pg_data_t *pgdat = page_pgdat(page); + struct lruvec *lruvec;
Quoted 8 lines
Review static inline void activate_page_drain(int cpu) · line 365
Quoted 3 lines
(quoted diff) +1
-1page = compound_head(page); - spin_lock_irq(&pgdat->lru_lock); + lruvec = lock_page_lruvec_irq(page);I don't see what makes this safe. There is nothing that appears to lock out a concurrent page move between memcgs/lruvecs, which means the following could manipulate an unlocked lru list:
This funtion is for !CONFIG_SMP, could the cpu be preempt with RT kernel?
Quoted 10 lines
Review static inline void activate_page_drain(int cpu) · line 370
Quoted 7 lines
(quoted diff) +2
-2if (PageLRU(page)) - __activate_page(page, mem_cgroup_page_lruvec(page, pgdat)); - spin_unlock_irq(&pgdat->lru_lock); + __activate_page(page, lruvec); + unlock_page_lruvec_irq(lruvec); } #endifShouldn't this be something like this?
Quoted 6 lines
if (TestClearPageLRU()) {
lruvec = lock_page_lruvec_irq(page);
__activate_page(page, lruvec);
unlock_page_lruvec_irq(lruvec);
SetPageLRU(page);
}But anyway, your new changes are more beautiful and logcially. I will change to this.
Thanks a lot!
Alex
PATCH v20 19/20 mm/lru: introduce the relock_page_lruvec function
In reply to Alex ShiFrom: Alexander Duyck <[email protected]>
Use this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
In addition we can avoid the checks entirely if lruvec is currently NULL.
--- include/linux/memcontrol.h | 52 ++++++++++++++++++++++++++++++++++++++++++++++ mm/mlock.c | 11 +--------- mm/swap.c | 33 +++++++---------------------- mm/vmscan.c | 12 ++--------- 4 files changed, 62 insertions(+), 46 deletions(-)
include/linux/memcontrol.h +52 -0
@@ -491,6 +491,22 @@ static inline struct lruvec *mem_cgroup_lruvec(struct mem_cgroup *memcg,struct lruvec *mem_cgroup_page_lruvec (struct page *,struct pglist_data *); +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); +const struct mem_cgroup *memcg; +struct mem_cgroup_per_node *mz; + +if (mem_cgroup_disabled ()) +return lruvec == &pgdat->__lruvec; + + mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg = page->mem_cgroup ? : root_mem_cgroup; + +return lruvec->pgdat == pgdat && mz->memcg == memcg; +} +struct mem_cgroup *mem_cgroup_from_task (struct task_struct *p);struct mem_cgroup *get_mem_cgroup_from_mm (struct mm_struct *mm);
@@ -1026,6 +1042,14 @@ static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,return &pgdat->__lruvec; } +static inline bool lruvec_holds_page_lru_lock (struct page *page, +struct lruvec *lruvec) +{ +pg_data_t *pgdat =page_pgdat (page); + +return lruvec == &pgdat->__lruvec; +} +static inline struct mem_cgroup *parent_mem_cgroup (struct mem_cgroup *memcg) {return NULL ;
@@ -1472,6 +1496,34 @@ static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,spin_unlock_irqrestore (&lruvec->lru_lock, flags); } +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irq (struct page *page, +struct lruvec *locked_lruvec) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irq (locked_lruvec); + } + +return lock_page_lruvec_irq (page); +} + +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irqsave (struct page *page, +struct lruvec *locked_lruvec,unsigned long *flags) +{ +if (locked_lruvec) { +if (lruvec_holds_page_lru_lock (page, locked_lruvec)) +return locked_lruvec; + +unlock_page_lruvec_irqrestore (locked_lruvec, *flags); + } + +return lock_page_lruvec_irqsave (page, flags); +} +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain *mem_cgroup_wb_domain (struct bdi_writeback *wb);
mm/mlock.c +1 -10
@@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pagevec *pvec, struct zone *zone)* so we can spare the get_page() here. */ if (TestClearPageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);del_page_from_lru_list (page, lruvec,page_lru (page));continue ;
mm/swap.c +7 -26
@@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct pagevec *pvec,for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec;/* block memcg migration during page moving between lru */ if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } - + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags); (*move_fn)(page, lruvec);SetPageLRU (page);
@@ -906,17 +899,12 @@ void release_pages(struct page **pages, int nr) }if (PageLRU (page)) { -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page, -page_pgdat (page)); -if (new_lruvec != lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, - flags); +struct lruvec *prev_lruvec = lruvec; + + lruvec =relock_page_lruvec_irqsave (page, lruvec, + &flags); +if (prev_lruvec != lruvec) lock_batch =0 ; - lruvec =lock_page_lruvec_irqsave (page, &flags); - }VM_BUG_ON_PAGE (!PageLRU (page), page);__ClearPageLRU (page);
@@ -1021,15 +1009,8 @@ void __pagevec_lru_add(struct pagevec *pvec)for (i =0 ; i <pagevec_count (pvec); i++) {struct page *page = pvec->pages[i]; -struct lruvec *new_lruvec; - - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irqrestore (lruvec, flags); - lruvec =lock_page_lruvec_irqsave (page, &flags); - } + lruvec =relock_page_lruvec_irqsave (page, lruvec, &flags);__pagevec_lru_add_fn (page, lruvec); }if (lruvec)
mm/vmscan.c +2 -10
@@ -1880,8 +1880,7 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,continue ; } -VM_BUG_ON_PAGE (mem_cgroup_page_lruvec (page,page_pgdat (page)) - != lruvec, page); +VM_BUG_ON_PAGE (!lruvec_holds_page_lru_lock (page, lruvec), page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
@@ -4273,7 +4272,6 @@ void check_move_unevictable_pages(struct pagevec *pvec)for (i =0 ; i < pvec->nr; i++) {struct page *page = pvec->pages[i];int nr_pages; -struct lruvec *new_lruvec;if (PageTransTail (page))continue ;
@@ -4285,13 +4283,7 @@ void check_move_unevictable_pages(struct pagevec *pvec)if (!TestClearPageLRU (page))continue ; - new_lruvec =mem_cgroup_page_lruvec (page,page_pgdat (page)); -if (lruvec != new_lruvec) { -if (lruvec) -unlock_page_lruvec_irq (lruvec); - lruvec =lock_page_lruvec_irq (page); - } - + lruvec =relock_page_lruvec_irq (page, lruvec);if (page_evictable (page) &&PageUnevictable (page)) {enum lru_list lru =page_lru_base_type (page);
Signature · 1 line
1.8.3.1
Re: [PATCH v20 19/20] mm/lru: introduce the relock_page_lruvec function
Quoted 11 lines · Alex Shi
From: Alexander Duyck <alexander.h.duyck-VuQAYsv1563Yd54FQh9/[email protected]>
Use this new function to replace repeated same code, no func change.
When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup.
In addition we can avoid the checks entirely if lruvec is currently NULL.
Quoted 15 lines
Acked-by Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Mel Gorman <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Hugh Dickins <[email protected]>Cc Tejun Heo <[email protected]>
PATCH v20 20/20 mm/lru: revise the comments of lru_lock
In reply to Alex ShiFrom: Hugh Dickins <[email protected]>
Since we changed the pgdat->lru_lock to lruvec->lru_lock, it's time to fix the incorrect comments in code. Also fixed some zone->lru_lock comment error from ancient time. etc.
I struggled to understand the comment above move_pages_to_lru() (surely it never calls page_referenced()), and eventually realized that most of it had got separated from shrink_active_list(): move that comment back.
--- Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 ++------ Documentation/admin-guide/cgroup-v1/memory.rst | 21 +++++------ Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +++++------- include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 3 +- mm/filemap.c | 4 +-- mm/rmap.c | 4 +-- mm/vmscan.c | 41 ++++++++++++---------- 9 files changed, 50 insertions(+), 64 deletions(-)
Documentation/admin-guide/cgroup-v1/memcg_test.rst +3 -12
@@ -133,18 +133,9 @@ Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y. 8. LRU ====== - Each memcg has its own private LRU. Now, its handling is under global - VM's control (means that it's handled under global pgdat->lru_lock). - Almost all routines around memcg's LRU is called by global LRU's - list management functions under pgdat->lru_lock. - - A special function is mem_cgroup_isolate_pages(). This scans - memcg's private LRU and call __isolate_lru_page() to extract a page - from LRU. - - (By __isolate_lru_page(), the page is removed from both of global and - private LRU.) - + Each memcg has its own vector of LRUs (inactive anon, active anon, + inactive file, active file, unevictable) of pages from each node, + each LRU handled under a single lru_lock for that memcg and node. 9. Typical Tests. =================
Documentation/admin-guide/cgroup-v1/memory.rst +9 -12
@@ -285,20 +285,17 @@ When oom event notifier is registered, event will be delivered. 2.6 Locking ----------- - lock_page_cgroup()/unlock_page_cgroup() should not be called under - the i_pages lock. +Lock order is as follows: - Other lock order is following: + Page lock (PG_locked bit of page->flags) + mm->page_table_lock or split pte_lock + lock_page_memcg (memcg->move_lock) + mapping->i_pages lock + lruvec->lru_lock. - PG_locked. - mm->page_table_lock - pgdat->lru_lock - lock_page_cgroup. - - In many cases, just lock_page_cgroup() is called. - - per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by - pgdat->lru_lock, it has no lock of its own. +Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by +lruvec->lru_lock; PG_lru bit of page->flags is cleared before +isolating a page from its LRU under lruvec->lru_lock. 2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM) -----------------------------------------------
Documentation/trace/events-kmem.rst +1 -1
@@ -69,7 +69,7 @@ When pages are freed in batch, the also mm_page_free_batched is triggered. Broadly speaking, pages are taken off the LRU lock in bulk and freed in batch with a page list. Significant amounts of activity here could indicate that the system is under memory pressure and can also indicate -contention on the zone->lru_lock. +contention on the lruvec->lru_lock. 4. Per-CPU Allocator Activity =============================
Documentation/vm/unevictable-lru.rst +8 -14
@@ -33,7 +33,7 @@ reclaim in Linux. The problems have been observed at customer sites on large memory x86_64 systems. To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of -main memory will have over 32 million 4k pages in a single zone. When a large +main memory will have over 32 million 4k pages in a single node. When a large fraction of these pages are not evictable for any reason [see below], vmscan will spend a lot of time scanning the LRU lists looking for the small fraction of pages that are evictable. This can result in a situation where all CPUs are
@@ -55,7 +55,7 @@ unevictable, either by definition or by circumstance, in the future. The Unevictable Page List ------------------------- -The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list +The Unevictable LRU infrastructure consists of an additional, per-node, LRU list called the "unevictable" list and an associated page flag, PG_unevictable, to indicate that the page is being managed on the unevictable list.
@@ -84,15 +84,9 @@ The unevictable list does not differentiate between file-backed and anonymous, swap-backed pages. This differentiation is only important while the pages are, in fact, evictable. -The unevictable list benefits from the "arrayification" of the per-zone LRU +The unevictable list benefits from the "arrayification" of the per-node LRU lists and statistics originally proposed and posted by Christoph Lameter. -The unevictable list does not use the LRU pagevec mechanism. Rather, -unevictable pages are placed directly on the page's zone's unevictable list -under the zone lru_lock. This allows us to prevent the stranding of pages on -the unevictable list when one task has the page isolated from the LRU and other -tasks are changing the "evictability" state of the page. - Memory Control Group Interaction --------------------------------
@@ -101,8 +95,8 @@ The unevictable LRU facility interacts with the memory control group [aka memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by extending the lru_list enum. -The memory controller data structure automatically gets a per-zone unevictable -list as a result of the "arrayification" of the per-zone LRU lists (one per +The memory controller data structure automatically gets a per-node unevictable +list as a result of the "arrayification" of the per-node LRU lists (one per lru_list enum element). The memory controller tracks the movement of pages to and from the unevictable list.
@@ -196,7 +190,7 @@ for the sake of expediency, to leave a unevictable page on one of the regular active/inactive LRU lists for vmscan to deal with. vmscan checks for such pages in all of the shrink_{active|inactive|page}_list() functions and will "cull" such pages that it encounters: that is, it diverts those pages to the -unevictable list for the zone being scanned. +unevictable list for the node being scanned. There may be situations where a page is mapped into a VM_LOCKED VMA, but the page is not marked as PG_mlocked. Such pages will make it all the way to
@@ -328,7 +322,7 @@ If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the page from the LRU, as it is likely on the appropriate active or inactive list at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put back the page - by calling putback_lru_page() - which will notice that the page -is now mlocked and divert the page to the zone's unevictable list. If +is now mlocked and divert the page to the node's unevictable list. If mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle it later if and when it attempts to reclaim the page.
@@ -603,7 +597,7 @@ Some examples of these unevictable pages on the LRU lists are: unevictable list in mlock_vma_page(). shrink_inactive_list() also diverts any unevictable pages that it finds on the -inactive lists to the appropriate zone's unevictable list. +inactive lists to the appropriate node's unevictable list. shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd after shrink_active_list() had moved them to the inactive list, or pages mapped
include/linux/mm_types.h +1 -1
@@ -78,7 +78,7 @@ struct page {struct {/* Page cache and anonymous pages */ /** * @lru: Pageout list, eg. active_list protected by -* pgdat->lru_lock. Sometimes used as a generic list +* lruvec->lru_lock. Sometimes used as a generic list * by the page owner. */ struct list_head lru;
include/linux/mmzone.h +1 -2
@@ -113,8 +113,7 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)struct pglist_data ;/* -* zone->lock and the zone lru_lock are two of the hottest locks in the kernel. -* So add a wild amount of padding here to ensure that they fall into separate +* Add a wild amount of padding here to ensure datas fall into separate * cachelines. There are very few zone structures in the machine, so space * consumption is not a concern here. */
mm/filemap.c +2 -2
@@ -102,8 +102,8 @@* ->swap_lock (try_to_unmap_one) * ->private_lock (try_to_unmap_one) * ->i_pages lock (try_to_unmap_one) -* ->pgdat->lru_lock (follow_page->mark_page_accessed) -* ->pgdat->lru_lock (check_pte_range->isolate_lru_page) +* ->lruvec->lru_lock (follow_page->mark_page_accessed) +* ->lruvec->lru_lock (check_pte_range->isolate_lru_page) * ->private_lock (page_remove_rmap->set_page_dirty) * ->i_pages lock (page_remove_rmap->set_page_dirty) * bdi.wb->list_lock (page_remove_rmap->set_page_dirty)
mm/rmap.c +2 -2
@@ -28,12 +28,12 @@* hugetlb_fault_mutex (hugetlbfs specific page fault mutex) * anon_vma->rwsem * mm->page_table_lock or pte_lock -* pgdat->lru_lock (in mark_page_accessed, isolate_lru_page) * swap_lock (in swap_duplicate, swap_info_get) * mmlist_lock (in mmput, drain_mmlist and others) * mapping->private_lock (in __set_page_dirty_buffers) -* mem_cgroup_{begin,end}_page_stat (memcg->move_lock) +* lock_page_memcg move_lock (in __set_page_dirty_buffers) * i_pages lock (widely used) +* lruvec->lru_lock (in lock_page_lruvec_irq) * inode->i_lock (in set_page_dirty's __mark_inode_dirty) * bdi.wb->list_lock (in set_page_dirty's __mark_inode_dirty) * sb_lock (within inode_lock in fs/fs-writeback.c)
mm/vmscan.c +23 -18
68 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v20 20/20] mm/lru: revise the comments of lru_lock
Quoted 10 lines · Alex Shi
From: Hugh Dickins <[email protected]>
Since we changed the
pgdat->lru_locktolruvec->lru_lock, it's time to fix the incorrect comments in code. Also fixed somezone->lru_lockcomment error from ancient time. etc.I struggled to understand the comment above
move_pages_to_lru()(surely it never callspage_referenced()), and eventually realized that most of it had got separated fromshrink_active_list(): move that comment back.
Quoted 13 lines
Signed-off-by Hugh Dickins <[email protected]>Signed-off-by Alex Shi <[email protected]>Cc Andrew Morton <[email protected]>Cc Tejun Heo <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Jann Horn <[email protected]>Cc Mel Gorman <[email protected]>Cc Johannes Weiner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Hugh Dickins <[email protected]>
Hi Johannes & all,
Thanks for all comments and suggestions, here is a patch base on v20, as a summary for all you suggested:
Is this ok?
Many thanks!
Alex
mm/memcontrol.c +11 -0
@@ -20,6 +20,9 @@* Lockless page tracking & accounting * Unified hierarchy configuration model * Copyright (C) 2015 Red Hat, Inc., Johannes Weiner +* +* Per memcg lru locking +* Copyright (C) 2020 Alibaba, Inc, Alex Shi */ #include <linux/page_counter.h>
@@ -1380,6 +1383,14 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgdreturn lruvec; } +/** +* lock_page_lruvec - return lruvec for the locked page. +* @page: the page +* +* This series functions should be used in either conditions: +* PageLRU is cleared or unset +* or, page->_refcount is zero +*/ struct lruvec *lock_page_lruvec (struct page *page) {struct lruvec *lruvec;
mm/swap.c +12 -3
@@ -264,6 +264,13 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) do { unsigned long lrusize; + /* + * Hold lruvec->lru_lock is safe here, since + * 1) The pinned lruvec in reclaim, or + * 2) From a pre-LRU page during refault (which also holds the + * rcu lock, so would be safe even if the page was on the LRU + * and could move simultaneously to a new lruvec). + */ spin_lock_irq(&lruvec->lru_lock); /* Record cost event */ if (file)
@@ -355,10 +362,12 @@ static void activate_page(struct page *page)struct lruvec *lruvec; page =compound_head (page); - lruvec =lock_page_lruvec_irq (page); -if (PageLRU (page)) +if (TestClearPageLRU (page)) { + lruvec =lock_page_lruvec_irq (page);__activate_page (page, lruvec); - unlock_page_lruvec_irq(lruvec); +unlock_page_lruvec_irq (lruvec); +SetPageLRU (page); + } }#endif
mm/vmscan.c +4 -0
@@ -1868,6 +1868,10 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,continue ; } +/* +* All pages were isolated from the same lruvec (and isolation +* inhibits memcg migration). +*/ VM_BUG_ON_PAGE (!lruvec_holds_page_lru_lock (page, lruvec), page); lru =page_lru (page); nr_pages =thp_nr_pages (page);
Johannes Weiner
maintainermm/memcontrol.c struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1387
Quoted 6 lines · Alex Shi
mm/memcontrol.c +2
-0@@ -1380,6 +1383,14 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgdreturn lruvec; } +/** +* lock_page_lruvec - return lruvec for the locked page.
I would say "lock and return the lruvec for a given page"
mm/memcontrol.c struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1392
Quoted 5 lines
mm/memcontrol.c +5
-0+* @page: the page +* +* This series functions should be used in either conditions: +* PageLRU is cleared or unset +* or, page->_refcount is zero
or page is locked
The other changes look good to me, thanks!
ÔÚ 2020/11/5 ÉÏÎç12:59, Johannes Weiner дµÀ:
Quoted 16 lines · Alex Shi
mm/memcontrol.c struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1387
Quoted 6 lines
mm/memcontrol.c +2
-0@@ -1380,6 +1383,14 @@ struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgdreturn lruvec; } +/** +* lock_page_lruvec - return lruvec for the locked page. I would say "lock and return the lruvec for a given page"
mm/memcontrol.c struct lruvec *mem_cgroup_page_lruvec(struct page *page, struct pglist_data *pgd · line 1392
Quoted 5 lines
mm/memcontrol.c +5
-0+* @page: the page +* +* This series functions should be used in either conditions: +* PageLRU is cleared or unset +* or, page->_refcount is zero
Quoted 4 lines
or page is locked
The other changes look good to me, thanks!
Thanks a lot for both comments!
I will pick them and sent out in v21.
Thanks!
Alex