mm: memcontrol: charge swapin pages on instantiation
Subsystems (14)
Johannes Weiner
maintainercover mm: memcontrol: charge swapin pages on instantiation
This patch series reworks memcg to charge swapin pages directly at swapin time, rather than at fault time, which may be much later, or not happen at all.
The delayed charging scheme we have right now causes problems:
- Alex's per-cgroup lru_lock patches rely on pages that have been isolated from the LRU to have a stable page->mem_cgroup; otherwise the lock may change underneath him. Swapcache pages are charged only after they are added to the LRU, and charging doesn't follow the LRU isolation protocol.
- Joonsoo's anon workingset patches need a suitable LRU at the time the page enters the swap cache and displaces the non-resident info. But the correct LRU is only available after charging.
- It's a containment hole / DoS vector. Users can trigger arbitrarily large swap readahead using MADV_WILLNEED. The memory is never charged unless somebody actually touches it.
- It complicates the page->mem_cgroup stabilization rules
In order to charge pages directly at swapin time, the memcg code base needs to be prepared, and several overdue cleanups become a necessity:
To charge pages at swapin time, we need to always have cgroup ownership tracking of swap records. We also cannot rely on page->mapping to tell apart page types at charge time, because that's only set up during a page fault.
To eliminate the page->mapping dependency, memcg needs to ditch its private page type counters (MEMCG_CACHE, MEMCG_RSS, NR_SHMEM) in favor of the generic vmstat counters and accounting sites, such as NR_FILE_PAGES, NR_ANON_MAPPED etc.
To switch to generic vmstat counters, the charge sequence must be adjusted such that page->mem_cgroup is set up by the time these counters are modified.
The series is structured as follows:
1. Bug fixes
2. Decoupling charging from rmap
3. Swap controller integration into memcg
4. Direct swapin charging
The patches survive a simple swapout->swapin test inside a virtual machine. Because this is blocking two major patch sets, I'm sending these out early and will continue testing in parallel to the review.
include/linux/memcontrol.h | 53 +---- include/linux/mm.h | 4 +- include/linux/swap.h | 6 +- init/Kconfig | 17 +- kernel/events/uprobes.c | 10 +- mm/filemap.c | 43 ++--- mm/huge_memory.c | 45 ++--- mm/khugepaged.c | 25 +-- mm/memcontrol.c | 448 ++++++++++++++----------------------------- mm/memory.c | 51 ++--- mm/migrate.c | 20 +- mm/rmap.c | 53 +++-- mm/shmem.c | 117 +++++------ mm/swap_cgroup.c | 6 - mm/swap_state.c | 89 +++++---- mm/swapfile.c | 25 +-- mm/userfaultfd.c | 5 +- 17 files changed, 367 insertions(+), 650 deletions(-)
Johannes Weiner
maintainerPATCH 1/18 mm: fix NUMA node file count error in replace_page_cache()
When replacing one page with another one in the cache, we have to decrease the file count of the old page's NUMA node and increase the one of the new NUMA node, otherwise the old node leaks the count and the new node eventually underflows its counter.
--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/filemap.c +2 -2
@@ -808,11 +808,11 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask) old->mapping =NULL ;/* hugetlb pages do not participate in page cache accounting. */ if (!PageHuge (old)) -__dec_node_page_state (new,NR_FILE_PAGES ); +__dec_node_page_state (old,NR_FILE_PAGES );if (!PageHuge (new))__inc_node_page_state (new,NR_FILE_PAGES );if (PageSwapBacked (old)) -__dec_node_page_state (new,NR_SHMEM ); +__dec_node_page_state (old,NR_SHMEM );if (PageSwapBacked (new))__inc_node_page_state (new,NR_SHMEM );xas_unlock_irqrestore (&xas, flags);
Signature · 1 line
2.26.0
Re: [PATCH 01/18] mm: fix NUMA node file count error in replace_page_cache()
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 7 lines
When replacing one page with another one in the cache, we have to decrease the file count of the old page's NUMA node and increase the one of the new NUMA node, otherwise the old node leaks the count and the new node eventually underflows its counter.
Fixes 74d609585d8b ("page cache: Add and replace pages using the XArray")Signed-off-by Johannes Weiner <[email protected]>
Quoted 23 lines
--- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/filemap.c +2
-2@@ -808,11 +808,11 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask) old->mapping =NULL ;/* hugetlb pages do not participate in page cache accounting. */ if (!PageHuge (old)) -__dec_node_page_state (new,NR_FILE_PAGES ); +__dec_node_page_state (old,NR_FILE_PAGES );if (!PageHuge (new))__inc_node_page_state (new,NR_FILE_PAGES );if (PageSwapBacked (old)) -__dec_node_page_state (new,NR_SHMEM ); +__dec_node_page_state (old,NR_SHMEM );if (PageSwapBacked (new))__inc_node_page_state (new,NR_SHMEM );xas_unlock_irqrestore (&xas, flags);
Re: [PATCH 01/18] mm: fix NUMA node file count error in replace_page_cache()
In reply to Johannes WeinerQuoted 8 lines · Johannes Weiner
When replacing one page with another one in the cache, we have to decrease the file count of the old page's NUMA node and increase the one of the new NUMA node, otherwise the old node leaks the count and the new node eventually underflows its counter.
Fixes 74d609585d8b ("page cache: Add and replace pages using the XArray")Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 01/18] mm: fix NUMA node file count error in replace_page_cache()
In reply to Johannes WeinerQuoted 7 lines · Johannes Weiner
When replacing one page with another one in the cache, we have to decrease the file count of the old page's NUMA node and increase the one of the new NUMA node, otherwise the old node leaks the count and the new node eventually underflows its counter.
Fixes 74d609585d8b ("page cache: Add and replace pages using the XArray")Signed-off-by Johannes Weiner <[email protected]>
Thanks.
Johannes Weiner
maintainerPATCH 2/18 mm: memcontrol: fix theoretical race in charge moving
In reply to Johannes WeinerThe move_lock is a per-memcg lock, but the VM accounting code that needs to acquire it comes from the page and follows page->mem_cgroup under RCU protection. That means that the page becomes unlocked not when we drop the move_lock, but when we update page->mem_cgroup. And that assignment doesn't imply any memory ordering. If that pointer write gets reordered against the reads of the page state - page_mapped, PageDirty etc. the state may change while we rely on it being stable and we can end up corrupting the counters.
Place an SMP memory barrier to make sure we're done with all page state by the time the new page->mem_cgroup becomes visible.
Also replace the open-coded move_lock with a lock_page_memcg() to make it more obvious what we're serializing against.
--- mm/memcontrol.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-)
mm/memcontrol.c +14 -12
@@ -5372,7 +5372,6 @@ static int mem_cgroup_move_account(struct page *page, {struct lruvec *from_vec, *to_vec;struct pglist_data *pgdat; -unsigned long flags;unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ;int ret;bool anon;
@@ -5399,18 +5398,13 @@ static int mem_cgroup_move_account(struct page *page, from_vec =mem_cgroup_lruvec (from, pgdat); to_vec =mem_cgroup_lruvec (to, pgdat); -spin_lock_irqsave (&from->move_lock, flags); +lock_page_memcg (page);if (!anon &&page_mapped (page)) {__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages); } -/* -* move_lock grabbed above and caller set from->moving_account, so -* mod_memcg_page_state will serialize updates to PageDirty. -* So mapping should be stable for dirty pages. -*/ if (!anon &&PageDirty (page)) {struct address_space *mapping =page_mapping (page);
@@ -5426,15 +5420,23 @@ static int mem_cgroup_move_account(struct page *page, }/* +* All state has been migrated, let's switch to the new memcg. +* * It is safe to change page->mem_cgroup here because the page -* is referenced, charged, and isolated - we can't race with -* uncharging, charging, migration, or LRU putback. +* is referenced, charged, isolated, and locked: we can't race +* with (un)charging, migration, LRU putback, or anything else +* that would rely on a stable page->mem_cgroup. +* +* Note that lock_page_memcg is a memcg lock, not a page lock, +* to save space. As soon as we switch page->mem_cgroup to a +* new memcg that isn't locked, the above state can change +* concurrently again. Make sure we're truly done with it. */ +smp_mb (); -/* caller should have done css_get */ - page->mem_cgroup = to; + page->mem_cgroup = to;/* caller should have done css_get */ -spin_unlock_irqrestore (&from->move_lock, flags); +__unlock_page_memcg (from); ret =0 ;
Signature · 1 line
2.26.0
Re: [PATCH 02/18] mm: memcontrol: fix theoretical race in charge moving
Quoted 9 lines · Johannes Weiner
The
move_lockis a per-memcg lock, but the VM accounting code that needs to acquire it comes from the page and followspage->mem_cgroupunder RCU protection. That means that the page becomes unlocked not when we drop themove_lock, but when we updatepage->mem_cgroup. And that assignment doesn't imply any memory ordering. If that pointer write gets reordered against the reads of the page state -page_mapped, PageDirty etc. the state may change while we rely on it being stable and we can end up corrupting the counters.
Quoted 7 lines
Place an SMP memory barrier to make sure we're done with all page state by the time the new
page->mem_cgroupbecomes visible.Also replace the open-coded
move_lockwith alock_page_memcg()to make it more obvious what we're serializing against.Signed-off-by Johannes Weiner <[email protected]>
Thanks.
Re: [PATCH 02/18] mm: memcontrol: fix theoretical race in charge moving
In reply to Johannes WeinerQuoted 62 lines · Johannes Weiner
The
move_lockis a per-memcg lock, but the VM accounting code that needs to acquire it comes from the page and followspage->mem_cgroupunder RCU protection. That means that the page becomes unlocked not when we drop themove_lock, but when we updatepage->mem_cgroup. And that assignment doesn't imply any memory ordering. If that pointer write gets reordered against the reads of the page state -page_mapped, PageDirty etc. the state may change while we rely on it being stable and we can end up corrupting the counters.Place an SMP memory barrier to make sure we're done with all page state by the time the new
page->mem_cgroupbecomes visible.Also replace the open-coded
move_lockwith alock_page_memcg()to make it more obvious what we're serializing against.Signed-off-by Johannes Weiner <[email protected]>--- mm/memcontrol.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-)mm/memcontrol.c +3
-9@@ -5372,7 +5372,6 @@ static int mem_cgroup_move_account(struct page *page, {struct lruvec *from_vec, *to_vec;struct pglist_data *pgdat; -unsigned long flags;unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ;int ret;bool anon;@@ -5399,18 +5398,13 @@ static int mem_cgroup_move_account(struct page *page, from_vec =mem_cgroup_lruvec (from, pgdat); to_vec =mem_cgroup_lruvec (to, pgdat); -spin_lock_irqsave (&from->move_lock, flags); +lock_page_memcg (page);if (!anon &&page_mapped (page)) {__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages); } -/* -* move_lock grabbed above and caller set from->moving_account, so -* mod_memcg_page_state will serialize updates to PageDirty. -* So mapping should be stable for dirty pages. -*/ if (!anon &&PageDirty (page)) {struct address_space *mapping =page_mapping (page);@@ -5426,15 +5420,23 @@ static int mem_cgroup_move_account(struct page *page, }/* +* All state has been migrated, let's switch to the new memcg. +* * It is safe to change page->mem_cgroup here because the page -* is referenced, charged, and isolated - we can't race with -* uncharging, charging, migration, or LRU putback.
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5435
Quoted 10 lines
mm/memcontrol.c +9
-0+* is referenced, charged, isolated, and locked: we can't race +* with (un)charging, migration, LRU putback, or anything else +* that would rely on a stable page->mem_cgroup. +* +* Note that lock_page_memcg is a memcg lock, not a page lock, +* to save space. As soon as we switch page->mem_cgroup to a +* new memcg that isn't locked, the above state can change +* concurrently again. Make sure we're truly done with it. */ +smp_mb ();
You said theoretical race in the subject but the above comment convinced me that smp_mb() is required. So, why is the race still theoretical?
Quoted 13 lines
mm/memcontrol.c +2
-4-/* caller should have done css_get */ - page->mem_cgroup = to; + page->mem_cgroup = to;/* caller should have done css_get */ -spin_unlock_irqrestore (&from->move_lock, flags); +__unlock_page_memcg (from); ret =0 ; --2.26.0
Johannes Weiner
maintainerRe: [PATCH 02/18] mm: memcontrol: fix theoretical race in charge moving
Quoted 16 lines · Shakeel Butt
Quoted 15 lines · Johannes Weiner
mm/memcontrol.c +8
-2@@ -5426,15 +5420,23 @@ static int mem_cgroup_move_account(struct page *page, }/* +* All state has been migrated, let's switch to the new memcg. +* * It is safe to change page->mem_cgroup here because the page -* is referenced, charged, and isolated - we can't race with -* uncharging, charging, migration, or LRU putback. +* is referenced, charged, isolated, and locked: we can't race +* with (un)charging, migration, LRU putback, or anything else +* that would rely on a stable page->mem_cgroup. +* +* Note that lock_page_memcg is a memcg lock, not a page lock, +* to save space. As soon as we switch page->mem_cgroup to a
Quoted 8 lines
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5435
Quoted 4 lines
mm/memcontrol.c +3
-0+* new memcg that isn't locked, the above state can change +* concurrently again. Make sure we're truly done with it. */ +smp_mb ();You said theoretical race in the subject but the above comment convinced me that
smp_mb()is required. So, why is the race still theoretical?
Sorry about the confusion.
I said theoretical because I spotted it while thinking about the code. I'm not aware of any real users that suffered the consequences of this race condition. But they could exist in theory :-)
I think it's a real bug that needs fixing.
Re: [PATCH 02/18] mm: memcontrol: fix theoretical race in charge moving
Quoted 27 lines · Johannes Weiner
Quoted 16 lines · Shakeel Butt
Quoted 15 lines · Johannes Weiner
mm/memcontrol.c +8
-2@@ -5426,15 +5420,23 @@ static int mem_cgroup_move_account(struct page *page, }/* +* All state has been migrated, let's switch to the new memcg. +* * It is safe to change page->mem_cgroup here because the page -* is referenced, charged, and isolated - we can't race with -* uncharging, charging, migration, or LRU putback. +* is referenced, charged, isolated, and locked: we can't race +* with (un)charging, migration, LRU putback, or anything else +* that would rely on a stable page->mem_cgroup. +* +* Note that lock_page_memcg is a memcg lock, not a page lock, +* to save space. As soon as we switch page->mem_cgroup to a Quoted 8 lines
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5435
Quoted 4 lines
mm/memcontrol.c +3
-0+* new memcg that isn't locked, the above state can change +* concurrently again. Make sure we're truly done with it. */ +smp_mb ();You said theoretical race in the subject but the above comment convinced me that
smp_mb()is required. So, why is the race still theoretical?
Quoted 7 lines
Sorry about the confusion.
I said theoretical because I spotted it while thinking about the code. I'm not aware of any real users that suffered the consequences of this race condition. But they could exist in theory :-)
I think it's a real bug that needs fixing.
Thanks for the clarification. I would suggest removing "theoretical" from the subject as it undermines that a real bug is fixed by the patch.
Re: [PATCH 02/18] mm: memcontrol: fix theoretical race in charge moving
In reply to Johannes WeinerQuoted 10 lines · Johannes Weiner
The
move_lockis a per-memcg lock, but the VM accounting code that needs to acquire it comes from the page and followspage->mem_cgroupunder RCU protection. That means that the page becomes unlocked not when we drop themove_lock, but when we updatepage->mem_cgroup. And that assignment doesn't imply any memory ordering. If that pointer write gets reordered against the reads of the page state -page_mapped, PageDirty etc. the state may change while we rely on it being stable and we can end up corrupting the counters.
Quoted 7 lines
Place an SMP memory barrier to make sure we're done with all page state by the time the new
page->mem_cgroupbecomes visible.Also replace the open-coded
move_lockwith alock_page_memcg()to make it more obvious what we're serializing against.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 3/18 mm: memcontrol: drop @compound parameter from memcg charging API
In reply to Johannes WeinerThe memcg charging API carries a boolean @compound parameter that tells whether the page we're dealing with is a hugepage. mem_cgroup_commit_charge() has another boolean @lrucare that indicates whether the page needs LRU locking or not while charging. The majority of callsites know those parameters at compile time, which results in a lot of naked "false, false" argument lists. This makes for cryptic code and is a breeding ground for subtle mistakes.
Thankfully, the huge page state can be inferred from the page itself and doesn't need to be passed along. This is safe because charging completes before the page is published and somebody may split it.
Simplify the callsites by removing @compound, and let memcg infer the state by using hpage_nr_pages() unconditionally. That function does PageTransHuge() to identify huge pages, which also helpfully asserts that nobody passes in tail pages by accident.
The following patches will introduce a new charging API, best not to carry over unnecessary weight.
--- include/linux/memcontrol.h | 22 ++++++++-------------- kernel/events/uprobes.c | 6 +++--- mm/filemap.c | 6 +++--- mm/huge_memory.c | 23 +++++++++++------------ mm/khugepaged.c | 20 ++++++++++---------- mm/memcontrol.c | 38 +++++++++++++++----------------------- mm/memory.c | 32 +++++++++++++++----------------- mm/migrate.c | 6 +++--- mm/shmem.c | 22 +++++++++------------- mm/swapfile.c | 9 ++++----- mm/userfaultfd.c | 6 +++--- 11 files changed, 84 insertions(+), 106 deletions(-)
include/linux/memcontrol.h +8 -14
@@ -361,15 +361,12 @@ enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root, struct mem_cgroup *memcg);int mem_cgroup_try_charge (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp, -bool compound); +gfp_t gfp_mask,struct mem_cgroup **memcgp);int mem_cgroup_try_charge_delay (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp, -bool compound); +gfp_t gfp_mask,struct mem_cgroup **memcgp);void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg, -bool lrucare,bool compound); -void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg, -bool compound); +bool lrucare); +void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg);void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);
@@ -849,8 +846,7 @@ static inline enum mem_cgroup_protection mem_cgroup_protected(static inline int mem_cgroup_try_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, -struct mem_cgroup **memcgp, -bool compound) +struct mem_cgroup **memcgp) { *memcgp =NULL ;return 0 ;
@@ -859,8 +855,7 @@ static inline int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm,static inline int mem_cgroup_try_charge_delay (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, -struct mem_cgroup **memcgp, -bool compound) +struct mem_cgroup **memcgp) { *memcgp =NULL ;return 0 ;
@@ -868,13 +863,12 @@ static inline int mem_cgroup_try_charge_delay(struct page *page,static inline void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg, -bool lrucare,bool compound) +bool lrucare) { }static inline void mem_cgroup_cancel_charge (struct page *page, -struct mem_cgroup *memcg, -bool compound) +struct mem_cgroup *memcg) { }
kernel/events/uprobes.c +3 -3
@@ -169,7 +169,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,if (new_page) { err =mem_cgroup_try_charge (new_page, vma->vm_mm,GFP_KERNEL , - &memcg, false); + &memcg);if (err)return err; }
@@ -181,7 +181,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, err = -EAGAIN ;if (!page_vma_mapped_walk (&pvmw)) {if (new_page) -mem_cgroup_cancel_charge (new_page, memcg, false); +mem_cgroup_cancel_charge (new_page, memcg); goto unlock; }VM_BUG_ON_PAGE (addr != pvmw.address, old_page);
@@ -189,7 +189,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,if (new_page) {get_page (new_page);page_add_new_anon_rmap (new_page, vma, addr, false); -mem_cgroup_commit_charge (new_page, memcg, false, false); +mem_cgroup_commit_charge (new_page, memcg, false);lru_cache_add_active_or_unevictable (new_page, vma); }else /* no new page, just dec_mm_counter for old_page */
mm/filemap.c +3 -3
@@ -842,7 +842,7 @@ static int __add_to_page_cache_locked(struct page *page,if (!huge) { error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg, false); + gfp_mask, &memcg);if (error)return error; }
@@ -878,14 +878,14 @@ static int __add_to_page_cache_locked(struct page *page, goto error;if (!huge) -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ if (!huge) -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg);put_page (page);return xas_error (&xas); }
mm/huge_memory.c +11 -12
@@ -594,7 +594,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,VM_BUG_ON_PAGE (!PageCompound (page), page); -if (mem_cgroup_try_charge_delay (page, vma->vm_mm, gfp, &memcg, true)) { +if (mem_cgroup_try_charge_delay (page, vma->vm_mm, gfp, &memcg)) {put_page (page);count_vm_event (THP_FAULT_FALLBACK );count_vm_event (THP_FAULT_FALLBACK_CHARGE );
@@ -630,7 +630,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,vm_fault_t ret2;spin_unlock (vmf->ptl); -mem_cgroup_cancel_charge (page, memcg, true); +mem_cgroup_cancel_charge (page, memcg);put_page (page);pte_free (vma->vm_mm, pgtable); ret2 =handle_userfault (vmf,VM_UFFD_MISSING );
@@ -641,7 +641,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry =mk_huge_pmd (page, vma->vm_page_prot); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);page_add_new_anon_rmap (page, vma, haddr, true); -mem_cgroup_commit_charge (page, memcg, false, true); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma);pgtable_trans_huge_deposit (vma->vm_mm, vmf->pmd, pgtable);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry);
@@ -658,7 +658,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, release:if (pgtable)pte_free (vma->vm_mm, pgtable); -mem_cgroup_cancel_charge (page, memcg, true); +mem_cgroup_cancel_charge (page, memcg);put_page (page);return ret;
@@ -1280,14 +1280,13 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, vmf->address, page_to_nid(page));if (unlikely (!pages[i] ||mem_cgroup_try_charge_delay (pages[i], vma->vm_mm, -GFP_KERNEL , &memcg, false))) { +GFP_KERNEL , &memcg))) {if (pages[i])put_page (pages[i]);while (--i >=0 ) { memcg = (void *)page_private (pages[i]);set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, - false); +mem_cgroup_cancel_charge (pages[i], memcg);put_page (pages[i]); }kfree (pages);
@@ -1333,7 +1332,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, memcg = (void *)page_private (pages[i]);set_page_private (pages[i],0 );page_add_new_anon_rmap (pages[i], vmf->vma, haddr, false); -mem_cgroup_commit_charge (pages[i], memcg, false, false); +mem_cgroup_commit_charge (pages[i], memcg, false);lru_cache_add_active_or_unevictable (pages[i], vma); vmf->pte =pte_offset_map (&_pmd, haddr);VM_BUG_ON (!pte_none (*vmf->pte));
@@ -1365,7 +1364,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf,for (i =0 ; i <HPAGE_PMD_NR ; i++) { memcg = (void *)page_private (pages[i]);set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, false); +mem_cgroup_cancel_charge (pages[i], memcg);put_page (pages[i]); }kfree (pages);
@@ -1448,7 +1447,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) }if (unlikely (mem_cgroup_try_charge_delay (new_page, vma->vm_mm, - huge_gfp, &memcg, true))) { + huge_gfp, &memcg))) {put_page (new_page);split_huge_pmd (vma, vmf->pmd, vmf->address);if (page)
@@ -1478,7 +1477,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)put_page (page);if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) {spin_unlock (vmf->ptl); -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg);put_page (new_page); goto out_mn; }else {
@@ -1487,7 +1486,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma);pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd);page_add_new_anon_rmap (new_page, vma, haddr, true); -mem_cgroup_commit_charge (new_page, memcg, false, true); +mem_cgroup_commit_charge (new_page, memcg, false);lru_cache_add_active_or_unevictable (new_page, vma);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);
mm/khugepaged.c +10 -10
@@ -974,7 +974,7 @@ static void collapse_huge_page(struct mm_struct *mm, goto out_nolock; } -if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg, true))) { +if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out_nolock; }
@@ -982,7 +982,7 @@ static void collapse_huge_page(struct mm_struct *mm,down_read (&mm->mmap_sem); result =hugepage_vma_revalidate (mm, address, &vma);if (result) { -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -990,7 +990,7 @@ static void collapse_huge_page(struct mm_struct *mm, pmd =mm_find_pmd (mm, address);if (!pmd) { result =SCAN_PMD_NULL ; -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -1001,7 +1001,7 @@ static void collapse_huge_page(struct mm_struct *mm,* Continuing to collapse causes inconsistency. */ if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -1087,7 +1087,7 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));page_add_new_anon_rmap (new_page, vma, address, true); -mem_cgroup_commit_charge (new_page, memcg, false, true); +mem_cgroup_commit_charge (new_page, memcg, false);count_memcg_events (memcg,THP_COLLAPSE_ALLOC ,1 );lru_cache_add_active_or_unevictable (new_page, vma);pgtable_trans_huge_deposit (mm, pmd, pgtable);
@@ -1105,7 +1105,7 @@ static void collapse_huge_page(struct mm_struct *mm,trace_mm_collapse_huge_page (mm, isolated, result);return ; out: -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg); goto out_up_write; }
@@ -1534,7 +1534,7 @@ static void collapse_file(struct mm_struct *mm, goto out; } -if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg, true))) { +if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out; }
@@ -1547,7 +1547,7 @@ static void collapse_file(struct mm_struct *mm,break ;xas_unlock_irq (&xas);if (!xas_nomem (&xas,GFP_KERNEL )) { -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg); result =SCAN_FAIL ; goto out; }
@@ -1783,7 +1783,7 @@ static void collapse_file(struct mm_struct *mm,SetPageUptodate (new_page);page_ref_add (new_page,HPAGE_PMD_NR -1 ); -mem_cgroup_commit_charge (new_page, memcg, false, true); +mem_cgroup_commit_charge (new_page, memcg, false);if (is_shmem) {set_page_dirty (new_page);
@@ -1838,7 +1838,7 @@ static void collapse_file(struct mm_struct *mm,VM_BUG_ON (nr_none);xas_unlock_irq (&xas); -mem_cgroup_cancel_charge (new_page, memcg, true); +mem_cgroup_cancel_charge (new_page, memcg); new_page->mapping =NULL ; }
mm/memcontrol.c +15 -23
@@ -834,7 +834,7 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)static void mem_cgroup_charge_statistics (struct mem_cgroup *memcg,struct page *page, -bool compound,int nr_pages) +int nr_pages) {/* * Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is
@@ -848,7 +848,7 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,__mod_memcg_state (memcg,NR_SHMEM , nr_pages); } -if (compound) { +if (abs (nr_pages) >1 ) {VM_BUG_ON_PAGE (!PageTransHuge (page), page);__mod_memcg_state (memcg,MEMCG_RSS_HUGE , nr_pages); }
@@ -5441,9 +5441,9 @@ static int mem_cgroup_move_account(struct page *page, ret =0 ;local_irq_disable (); -mem_cgroup_charge_statistics (to, page, compound, nr_pages); +mem_cgroup_charge_statistics (to, page, nr_pages);memcg_check_events (to, page); -mem_cgroup_charge_statistics (from, page, compound, -nr_pages); +mem_cgroup_charge_statistics (from, page, -nr_pages);memcg_check_events (from, page);local_irq_enable (); out_unlock:
@@ -6434,7 +6434,6 @@ enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root,* @mm: mm context of the victim * @gfp_mask: reclaim mode * @memcgp: charged memcg return -* @compound: charge the page as compound or small page * * Try to charge @page to the memcg that @mm belongs to, reclaiming * pages according to @gfp_mask if necessary.
@@ -6447,11 +6446,10 @@ enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root,* with mem_cgroup_cancel_charge() in case page instantiation fails. */ int mem_cgroup_try_charge (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp, -bool compound) +gfp_t gfp_mask,struct mem_cgroup **memcgp) { +unsigned int nr_pages =hpage_nr_pages (page);struct mem_cgroup *memcg =NULL ; -unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ;int ret =0 ;if (mem_cgroup_disabled ())
@@ -6493,13 +6491,12 @@ int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm, }int mem_cgroup_try_charge_delay (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp, -bool compound) +gfp_t gfp_mask,struct mem_cgroup **memcgp) {struct mem_cgroup *memcg;int ret; - ret =mem_cgroup_try_charge (page, mm, gfp_mask, memcgp, compound); + ret =mem_cgroup_try_charge (page, mm, gfp_mask, memcgp); memcg = *memcgp;mem_cgroup_throttle_swaprate (memcg,page_to_nid (page), gfp_mask);return ret;
@@ -6510,7 +6507,6 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,* @page: page to charge * @memcg: memcg to charge the page to * @lrucare: page might be on LRU already -* @compound: charge the page as compound or small page * * Finalize a charge transaction started by mem_cgroup_try_charge(), * after page->mapping has been set up. This must happen atomically
@@ -6523,9 +6519,9 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,* Use mem_cgroup_cancel_charge() to cancel the transaction instead. */ void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg, -bool lrucare,bool compound) +bool lrucare) { -unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ; +unsigned int nr_pages =hpage_nr_pages (page);VM_BUG_ON_PAGE (!page->mapping, page);VM_BUG_ON_PAGE (PageLRU (page) && !lrucare, page);
@@ -6543,7 +6539,7 @@ void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg,commit_charge (page, memcg, lrucare);local_irq_disable (); -mem_cgroup_charge_statistics (memcg, page, compound, nr_pages); +mem_cgroup_charge_statistics (memcg, page, nr_pages);memcg_check_events (memcg, page);local_irq_enable ();
@@ -6562,14 +6558,12 @@ void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg,* mem_cgroup_cancel_charge - cancel a page charge * @page: page to charge * @memcg: memcg to charge the page to -* @compound: charge the page as compound or small page * * Cancel a charge transaction started by mem_cgroup_try_charge(). */ -void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg, -bool compound) +void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg) { -unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ; +unsigned int nr_pages =hpage_nr_pages (page);if (mem_cgroup_disabled ())return ;
@@ -6784,8 +6778,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)commit_charge (newpage, memcg, false);local_irq_save (flags); -mem_cgroup_charge_statistics (memcg, newpage,PageTransHuge (newpage), - nr_pages); +mem_cgroup_charge_statistics (memcg, newpage, nr_pages);memcg_check_events (memcg, newpage);local_irq_restore (flags); }
@@ -7015,8 +7008,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)* only synchronisation we have for updating the per-CPU variables. */ VM_BUG_ON (!irqs_disabled ()); -mem_cgroup_charge_statistics (memcg, page,PageTransHuge (page), - -nr_entries); +mem_cgroup_charge_statistics (memcg, page, -nr_entries);memcg_check_events (memcg, page);if (!mem_cgroup_is_root (memcg))
mm/memory.c +15 -17
@@ -2678,7 +2678,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } } -if (mem_cgroup_try_charge_delay (new_page, mm,GFP_KERNEL , &memcg, false)) +if (mem_cgroup_try_charge_delay (new_page, mm,GFP_KERNEL , &memcg)) goto oom_free_new;__SetPageUptodate (new_page);
@@ -2713,7 +2713,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)*/ ptep_clear_flush_notify (vma, vmf->address, vmf->pte);page_add_new_anon_rmap (new_page, vma, vmf->address, false); -mem_cgroup_commit_charge (new_page, memcg, false, false); +mem_cgroup_commit_charge (new_page, memcg, false);lru_cache_add_active_or_unevictable (new_page, vma);/* * We call the notify macro here because, when using secondary
@@ -2752,7 +2752,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) new_page = old_page; page_copied =1 ; }else { -mem_cgroup_cancel_charge (new_page, memcg, false); +mem_cgroup_cancel_charge (new_page, memcg); }if (new_page)
@@ -3195,8 +3195,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_page; } -if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , - &memcg, false)) { +if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , &memcg)) { ret =VM_FAULT_OOM ; goto out_page; }
@@ -3247,11 +3246,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* ksm created a completely new copy */ if (unlikely (page != swapcache && swapcache)) {page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }else {do_page_add_anon_rmap (page, vma, vmf->address, exclusive); -mem_cgroup_commit_charge (page, memcg, true, false); +mem_cgroup_commit_charge (page, memcg, true);activate_page (page); }
@@ -3287,7 +3286,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out:return ret; out_nomap: -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg);pte_unmap_unlock (vmf->pte, vmf->ptl); out_page:unlock_page (page);
@@ -3361,8 +3360,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (!page) goto oom; -if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , &memcg, - false)) +if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , &memcg)) goto oom_free_page;/*
@@ -3388,14 +3386,14 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing (vma)) {pte_unmap_unlock (vmf->pte, vmf->ptl); -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg);put_page (page);return handle_userfault (vmf,VM_UFFD_MISSING ); }inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);
@@ -3406,7 +3404,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return ret; release: -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg);put_page (page); goto unlock; oom_free_page:
@@ -3657,7 +3655,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }else {inc_mm_counter_fast (vma->vm_mm,mm_counter_file (page));
@@ -3866,8 +3864,8 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf)if (!vmf->cow_page)return VM_FAULT_OOM ; -if (mem_cgroup_try_charge_delay (vmf->cow_page, vma->vm_mm,GFP_KERNEL , - &vmf->memcg, false)) { +if (mem_cgroup_try_charge_delay (vmf->cow_page, vma->vm_mm, +GFP_KERNEL , &vmf->memcg)) {put_page (vmf->cow_page);return VM_FAULT_OOM ; }
@@ -3888,7 +3886,7 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) goto uncharge_out;return ret; uncharge_out: -mem_cgroup_cancel_charge (vmf->cow_page, vmf->memcg, false); +mem_cgroup_cancel_charge (vmf->cow_page, vmf->memcg);put_page (vmf->cow_page);return ret; }
mm/migrate.c +3 -3
@@ -2786,7 +2786,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,if (unlikely (anon_vma_prepare (vma))) goto abort; -if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , &memcg, false)) +if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , &memcg)) goto abort;/*
@@ -2832,7 +2832,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,inc_mm_counter (mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);if (!is_zone_device_page (page))lru_cache_add_active_or_unevictable (page, vma);get_page (page);
@@ -2854,7 +2854,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, unlock_abort:pte_unmap_unlock (ptep, ptl); -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg); abort: *src &= ~MIGRATE_PFN_MIGRATE ; }
mm/shmem.c +9 -13
@@ -1664,8 +1664,7 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg, - false); + error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg);if (!error) { error =shmem_add_to_page_cache (page, mapping, index,swp_to_radix_entry (swap), gfp);
@@ -1680,14 +1679,14 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,* the rest. */ if (error) { -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg);delete_from_swap_cache (page); } }if (error) goto failed; -mem_cgroup_commit_charge (page, memcg, true, false); +mem_cgroup_commit_charge (page, memcg, true);spin_lock_irq (&info->lock); info->swapped--;
@@ -1859,8 +1858,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,if (sgp ==SGP_WRITE )__SetPageReferenced (page); - error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg, -PageTransHuge (page)); + error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg);if (error) {if (PageTransHuge (page)) {count_vm_event (THP_FILE_FALLBACK );
@@ -1871,12 +1869,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, error =shmem_add_to_page_cache (page, mapping, hindex,NULL , gfp &GFP_RECLAIM_MASK );if (error) { -mem_cgroup_cancel_charge (page, memcg, -PageTransHuge (page)); +mem_cgroup_cancel_charge (page, memcg); goto unacct; } -mem_cgroup_commit_charge (page, memcg, false, -PageTransHuge (page)); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_anon (page);spin_lock_irq (&info->lock);
@@ -2361,7 +2357,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm,if (unlikely (offset >= max_off)) goto out_release; - ret =mem_cgroup_try_charge_delay (page, dst_mm, gfp, &memcg, false); + ret =mem_cgroup_try_charge_delay (page, dst_mm, gfp, &memcg);if (ret) goto out_release;
@@ -2370,7 +2366,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm,if (ret) goto out_release_uncharge; -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false); _dst_pte =mk_pte (page, dst_vma->vm_page_prot);if (dst_vma->vm_flags &VM_WRITE )
@@ -2421,7 +2417,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm,ClearPageDirty (page);delete_from_page_cache (page); out_release_uncharge: -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg); out_release:unlock_page (page);put_page (page);
mm/swapfile.c +4 -5
@@ -1864,15 +1864,14 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,if (unlikely (!page))return -ENOMEM ; -if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , - &memcg, false)) { +if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , &memcg)) { ret = -ENOMEM ; goto out_nolock; } pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);if (unlikely (!pte_same_as_swp (*pte,swp_entry_to_pte (entry)))) { -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg); ret =0 ; goto out; }
@@ -1884,10 +1883,10 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, pte_mkold(mk_pte(page, vma->vm_page_prot)));if (page == swapcache) {page_add_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, true, false); +mem_cgroup_commit_charge (page, memcg, true); }else {/* ksm created a completely new copy */ page_add_new_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }swap_free (entry);
mm/userfaultfd.c +3 -3
@@ -97,7 +97,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,__SetPageUptodate (page); ret = -ENOMEM ; -if (mem_cgroup_try_charge (page, dst_mm,GFP_KERNEL , &memcg, false)) +if (mem_cgroup_try_charge (page, dst_mm,GFP_KERNEL , &memcg)) goto out_release; _dst_pte =pte_mkdirty (mk_pte (page, dst_vma->vm_page_prot));
@@ -124,7 +124,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,inc_mm_counter (dst_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, dst_vma, dst_addr, false); -mem_cgroup_commit_charge (page, memcg, false, false); +mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, dst_vma);set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte);
@@ -138,7 +138,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,return ret; out_release_uncharge_unlock:pte_unmap_unlock (dst_pte, ptl); -mem_cgroup_cancel_charge (page, memcg, false); +mem_cgroup_cancel_charge (page, memcg); out_release:put_page (page); goto out;
Signature · 1 line
2.26.0
Re: [PATCH 03/18] mm: memcontrol: drop @compound parameter from memcg charging API
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 17 lines
The memcg charging API carries a boolean @compound parameter that tells whether the page we're dealing with is a hugepage.
mem_cgroup_commit_charge()has another boolean @lrucare that indicates whether the page needs LRU locking or not while charging. The majority of callsites know those parameters at compile time, which results in a lot of naked "false, false" argument lists. This makes for cryptic code and is a breeding ground for subtle mistakes.Thankfully, the huge page state can be inferred from the page itself and doesn't need to be passed along. This is safe because charging completes before the page is published and somebody may split it.
Simplify the callsites by removing @compound, and let memcg infer the state by using
hpage_nr_pages()unconditionally. That function doesPageTransHuge()to identify huge pages, which also helpfully asserts that nobody passes in tail pages by accident.
Quoted 4 lines
The following patches will introduce a new charging API, best not to carry over unnecessary weight.
Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 03/18] mm: memcontrol: drop @compound parameter from memcg charging API
In reply to Johannes WeinerQuoted 17 lines · Johannes Weiner
The memcg charging API carries a boolean @compound parameter that tells whether the page we're dealing with is a hugepage.
mem_cgroup_commit_charge()has another boolean @lrucare that indicates whether the page needs LRU locking or not while charging. The majority of callsites know those parameters at compile time, which results in a lot of naked "false, false" argument lists. This makes for cryptic code and is a breeding ground for subtle mistakes.Thankfully, the huge page state can be inferred from the page itself and doesn't need to be passed along. This is safe because charging completes before the page is published and somebody may split it.
Simplify the callsites by removing @compound, and let memcg infer the state by using
hpage_nr_pages()unconditionally. That function doesPageTransHuge()to identify huge pages, which also helpfully asserts that nobody passes in tail pages by accident.
Quoted 4 lines
The following patches will introduce a new charging API, best not to carry over unnecessary weight.
Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 03/18] mm: memcontrol: drop @compound parameter from memcg charging API
In reply to Johannes WeinerQuoted 18 lines · Johannes Weiner
The memcg charging API carries a boolean @compound parameter that tells whether the page we're dealing with is a hugepage.
mem_cgroup_commit_charge()has another boolean @lrucare that indicates whether the page needs LRU locking or not while charging. The majority of callsites know those parameters at compile time, which results in a lot of naked "false, false" argument lists. This makes for cryptic code and is a breeding ground for subtle mistakes.Thankfully, the huge page state can be inferred from the page itself and doesn't need to be passed along. This is safe because charging completes before the page is published and somebody may split it.
Simplify the callsites by removing @compound, and let memcg infer the state by using
hpage_nr_pages()unconditionally. That function doesPageTransHuge()to identify huge pages, which also helpfully asserts that nobody passes in tail pages by accident.
Quoted 4 lines
The following patches will introduce a new charging API, best not to carry over unnecessary weight.
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 4/18 mm: memcontrol: move out cgroup swaprate throttling
In reply to Johannes WeinerThe cgroup swaprate throttling is about matching new anon allocations to the rate of available IO when that is being throttled. It's the io controller hooking into the VM, rather than a memory controller thing.
Rename mem_cgroup_throttle_swaprate() to cgroup_throttle_swaprate(), and drop the @memcg argument which is only used to check whether the preceding page charge has succeeded and the fault is proceeding.
We could decouple the call from mem_cgroup_try_charge() here as well, but that would cause unnecessary churn: the following patches convert all callsites to a new charge API and we'll decouple as we go along.
--- include/linux/swap.h | 6 ++---- mm/memcontrol.c | 5 ++--- mm/swapfile.c | 14 +++++++------- 3 files changed, 11 insertions(+), 14 deletions(-)
include/linux/swap.h +2 -4
@@ -645,11 +645,9 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *mem)#endif #if defined(CONFIG_SWAP ) && defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP ) -extern void mem_cgroup_throttle_swaprate (struct mem_cgroup *memcg,int node, -gfp_t gfp_mask); +extern void cgroup_throttle_swaprate (struct page *page,gfp_t gfp_mask);#else -static inline void mem_cgroup_throttle_swaprate (struct mem_cgroup *memcg, -int node,gfp_t gfp_mask) +static inline void cgroup_throttle_swaprate (struct page *page,gfp_t gfp_mask) { }#endif
mm/memcontrol.c +2 -3
@@ -6493,12 +6493,11 @@ int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm,int mem_cgroup_try_charge_delay (struct page *page,struct mm_struct *mm,gfp_t gfp_mask,struct mem_cgroup **memcgp) { -struct mem_cgroup *memcg;int ret; ret =mem_cgroup_try_charge (page, mm, gfp_mask, memcgp); - memcg = *memcgp; -mem_cgroup_throttle_swaprate (memcg,page_to_nid (page), gfp_mask); +if (*memcgp) +cgroup_throttle_swaprate (page, gfp_mask);return ret; }
mm/swapfile.c +7 -7
@@ -3744,11 +3744,12 @@ static void free_swap_count_continuations(struct swap_info_struct *si) }#if defined(CONFIG_MEMCG ) && defined(CONFIG_BLK_CGROUP ) -void mem_cgroup_throttle_swaprate (struct mem_cgroup *memcg,int node, -gfp_t gfp_mask) +void cgroup_throttle_swaprate (struct page *page,gfp_t gfp_mask) {struct swap_info_struct *si, *next; -if (!(gfp_mask & __GFP_IO) || !memcg) +int nid =page_to_nid (page); + +if (!(gfp_mask & __GFP_IO))return ;if (!blk_cgroup_congested ())
@@ -3762,11 +3763,10 @@ void mem_cgroup_throttle_swaprate(struct mem_cgroup *memcg, int node,return ;spin_lock (&swap_avail_lock); -plist_for_each_entry_safe (si, next, &swap_avail_heads[node], - avail_lists[node]) { +plist_for_each_entry_safe (si, next, &swap_avail_heads[nid], + avail_lists[nid]) {if (si->bdev) { -blkcg_schedule_throttle (bdev_get_queue (si->bdev), - true); +blkcg_schedule_throttle (bdev_get_queue (si->bdev), true);break ; } }
Signature · 1 line
2.26.0
Re: [PATCH 04/18] mm: memcontrol: move out cgroup swaprate throttling
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 8 lines
The cgroup swaprate throttling is about matching new anon allocations to the rate of available IO when that is being throttled. It's the io controller hooking into the VM, rather than a memory controller thing.
Rename
mem_cgroup_throttle_swaprate()tocgroup_throttle_swaprate(), and drop the @memcg argument which is only used to check whether the preceding page charge has succeeded and the fault is proceeding.
Quoted 5 lines
We could decouple the call from
mem_cgroup_try_charge()here as well, but that would cause unnecessary churn: the following patches convert all callsites to a new charge API and we'll decouple as we go along.Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 04/18] mm: memcontrol: move out cgroup swaprate throttling
In reply to Johannes WeinerQuoted 8 lines · Johannes Weiner
The cgroup swaprate throttling is about matching new anon allocations to the rate of available IO when that is being throttled. It's the io controller hooking into the VM, rather than a memory controller thing.
Rename
mem_cgroup_throttle_swaprate()tocgroup_throttle_swaprate(), and drop the @memcg argument which is only used to check whether the preceding page charge has succeeded and the fault is proceeding.
Quoted 5 lines
We could decouple the call from
mem_cgroup_try_charge()here as well, but that would cause unnecessary churn: the following patches convert all callsites to a new charge API and we'll decouple as we go along.Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 04/18] mm: memcontrol: move out cgroup swaprate throttling
In reply to Johannes WeinerQuoted 9 lines · Johannes Weiner
The cgroup swaprate throttling is about matching new anon allocations to the rate of available IO when that is being throttled. It's the io controller hooking into the VM, rather than a memory controller thing.
Rename
mem_cgroup_throttle_swaprate()tocgroup_throttle_swaprate(), and drop the @memcg argument which is only used to check whether the preceding page charge has succeeded and the fault is proceeding.
Quoted 5 lines
We could decouple the call from
mem_cgroup_try_charge()here as well, but that would cause unnecessary churn: the following patches convert all callsites to a new charge API and we'll decouple as we go along.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 5/18 mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
In reply to Johannes WeinerThe try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a mem_cgroup_charge() function that simply charges a newly allocated page to a cgroup and sets up page->mem_cgroup in one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.
Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on page->mapping being already set up at the time of charging, so that it can maintain its own MEMCG_CACHE and MEMCG_RSS counters. For anon, page->mapping is set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.
The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the page->mapping dependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.
--- include/linux/memcontrol.h | 10 ++++ mm/filemap.c | 24 ++++------ mm/memcontrol.c | 27 +++++++++++ mm/shmem.c | 97 +++++++++++++++++--------------------- 4 files changed, 89 insertions(+), 69 deletions(-)
include/linux/memcontrol.h +10 -0
@@ -367,6 +367,10 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg,bool lrucare);void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); + +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare); +void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);
@@ -872,6 +876,12 @@ static inline void mem_cgroup_cancel_charge(struct page *page, { } +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask,bool lrucare) +{ +return 0 ; +} +static inline void mem_cgroup_uncharge (struct page *page) { }
mm/filemap.c +10 -14
@@ -832,7 +832,6 @@ static int __add_to_page_cache_locked(struct page *page, {XA_STATE (xas, &mapping->i_pages, offset);int huge =PageHuge (page); -struct mem_cgroup *memcg;int error;void *old;
@@ -840,17 +839,16 @@ static int __add_to_page_cache_locked(struct page *page,VM_BUG_ON_PAGE (PageSwapBacked (page), page);mapping_set_update (&xas, mapping); -if (!huge) { - error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg); -if (error) -return error; - } -get_page (page); page->mapping = mapping; page->index = offset; +if (!huge) { + error =mem_cgroup_charge (page, current->mm, gfp_mask, false); +if (error) + goto error; + } + do { xas_lock_irq(&xas); old = xas_load(&xas);
@@ -874,20 +872,18 @@ static int __add_to_page_cache_locked(struct page *page,xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK )); -if (xas_error (&xas)) +if (xas_error (&xas)) { + error =xas_error (&xas); goto error; + } -if (!huge) -mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ -if (!huge) -mem_cgroup_cancel_charge (page, memcg);put_page (page); -return xas_error (&xas); +return error; }ALLOW_ERROR_INJECTION (__add_to_page_cache_locked,ERRNO );
mm/memcontrol.c +27 -0
@@ -6577,6 +6577,33 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg)cancel_charge (memcg, nr_pages); } +/** +* mem_cgroup_charge - charge a newly allocated page to a cgroup +* @page: page to charge +* @mm: mm context of the victim +* @gfp_mask: reclaim mode +* @lrucare: page might be on the LRU already +* +* Try to charge @page to the memcg that @mm belongs to, reclaiming +* pages according to @gfp_mask if necessary. +* +* Returns 0 on success. Otherwise, an error code is returned. +*/ +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare) +{ +struct mem_cgroup *memcg; +int ret; + +VM_BUG_ON_PAGE (!page->mapping, page); + + ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg); +if (ret) +return ret; +mem_cgroup_commit_charge (page, memcg, lrucare); +return 0 ; +} +struct uncharge_gather {struct mem_cgroup *memcg;unsigned long pgpgout;
mm/shmem.c +42 -55
181 lines not on this page · raw message
Signature · 1 line
2.26.0
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 21 lines
The try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a
mem_cgroup_charge()function that simply charges a newly allocated page to a cgroup and sets uppage->mem_cgroupin one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on
page->mappingbeing already set up at the time of charging, so that it can maintain its ownMEMCG_CACHEandMEMCG_RSScounters. For anon,page->mappingis set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.
Quoted 7 lines
The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the
page->mappingdependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.Signed-off-by Johannes Weiner <[email protected]>---
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
In reply to Johannes WeinerQuoted 253 lines · Johannes Weiner
The try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a
mem_cgroup_charge()function that simply charges a newly allocated page to a cgroup and sets uppage->mem_cgroupin one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on
page->mappingbeing already set up at the time of charging, so that it can maintain its ownMEMCG_CACHEandMEMCG_RSScounters. For anon,page->mappingis set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the
page->mappingdependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.Signed-off-by Johannes Weiner <[email protected]>--- include/linux/memcontrol.h | 10 ++++ mm/filemap.c | 24 ++++------ mm/memcontrol.c | 27 +++++++++++ mm/shmem.c | 97 +++++++++++++++++--------------------- 4 files changed, 89 insertions(+), 69 deletions(-)include/linux/memcontrol.h +10
-0@@ -367,6 +367,10 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg,bool lrucare);void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); + +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare); +void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);@@ -872,6 +876,12 @@ static inline void mem_cgroup_cancel_charge(struct page *page, { } +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask,bool lrucare) +{ +return 0 ; +} +static inline void mem_cgroup_uncharge (struct page *page) { }mm/filemap.c +10
-14@@ -832,7 +832,6 @@ static int __add_to_page_cache_locked(struct page *page, {XA_STATE (xas, &mapping->i_pages, offset);int huge =PageHuge (page); -struct mem_cgroup *memcg;int error;void *old;@@ -840,17 +839,16 @@ static int __add_to_page_cache_locked(struct page *page,VM_BUG_ON_PAGE (PageSwapBacked (page), page);mapping_set_update (&xas, mapping); -if (!huge) { - error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg); -if (error) -return error; - } -get_page (page); page->mapping = mapping; page->index = offset; +if (!huge) { + error =mem_cgroup_charge (page, current->mm, gfp_mask, false); +if (error) + goto error; + } + do { xas_lock_irq(&xas); old = xas_load(&xas);@@ -874,20 +872,18 @@ static int __add_to_page_cache_locked(struct page *page,xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK )); -if (xas_error (&xas)) +if (xas_error (&xas)) { + error =xas_error (&xas); goto error; + } -if (!huge) -mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ -if (!huge) -mem_cgroup_cancel_charge (page, memcg);put_page (page); -return xas_error (&xas); +return error; }ALLOW_ERROR_INJECTION (__add_to_page_cache_locked,ERRNO );mm/memcontrol.c +27
-0@@ -6577,6 +6577,33 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg)cancel_charge (memcg, nr_pages); } +/** +* mem_cgroup_charge - charge a newly allocated page to a cgroup +* @page: page to charge +* @mm: mm context of the victim +* @gfp_mask: reclaim mode +* @lrucare: page might be on the LRU already +* +* Try to charge @page to the memcg that @mm belongs to, reclaiming +* pages according to @gfp_mask if necessary. +* +* Returns 0 on success. Otherwise, an error code is returned. +*/ +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare) +{ +struct mem_cgroup *memcg; +int ret; + +VM_BUG_ON_PAGE (!page->mapping, page); + + ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg); +if (ret) +return ret; +mem_cgroup_commit_charge (page, memcg, lrucare); +return 0 ; +} +struct uncharge_gather {struct mem_cgroup *memcg;unsigned long pgpgout;mm/shmem.c +24
-25@@ -605,11 +605,13 @@ static inline bool is_huge_enabled(struct shmem_sb_info *sbinfo)*/ static int shmem_add_to_page_cache (struct page *page,struct address_space *mapping, -pgoff_t index,void *expected,gfp_t gfp) +pgoff_t index,void *expected,gfp_t gfp, +struct mm_struct *charge_mm) {XA_STATE_ORDER (xas, &mapping->i_pages, index,compound_order (page));unsigned long i =0 ;unsigned long nr =compound_nr (page); +int error;VM_BUG_ON_PAGE (PageTail (page), page);VM_BUG_ON_PAGE (index !=round_down (index, nr), page);@@ -621,6 +623,16 @@ static int shmem_add_to_page_cache(struct page *page, page->mapping = mapping; page->index = index; + error =mem_cgroup_charge (page, charge_mm, gfp,PageSwapCache (page)); +if (error) { +if (!PageSwapCache (page) &&PageTransHuge (page)) { +count_vm_event (THP_FILE_FALLBACK ); +count_vm_event (THP_FILE_FALLBACK_CHARGE ); + } + goto error; + } +cgroup_throttle_swaprate (page, gfp); + do { void *entry; xas_lock_irq(&xas);@@ -648,12 +660,15 @@ static int shmem_add_to_page_cache(struct page *page, }while (xas_nomem (&xas, gfp));if (xas_error (&xas)) { - page->mapping =NULL ; -page_ref_sub (page, nr); -return xas_error (&xas); + error =xas_error (&xas); + goto error; }return 0 ; +error: + page->mapping =NULL ; +page_ref_sub (page, nr); +return error; }/* @@ -1619,7 +1634,6 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,struct address_space *mapping = inode->i_mapping;struct shmem_inode_info *info =SHMEM_I (inode);struct mm_struct *charge_mm = vma ? vma->vm_mm : current->mm; -struct mem_cgroup *memcg;struct page *page;swp_entry_t swap;int error;@@ -1664,29 +1678,22 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg); -if (!error) { - error =shmem_add_to_page_cache (page, mapping, index, -swp_to_radix_entry (swap), gfp); -/* -* We already confirmed swap under page lock, and make -* no memory allocation here, so usually no possibility -* of error; but free_swap_and_cache() only trylocks a -* page, so it is just possible that the entry has been -* truncated or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. -*/ -if (error) { -mem_cgroup_cancel_charge (page, memcg); -delete_from_swap_cache (page); - } - } -if (error) + error =shmem_add_to_page_cache (page, mapping, index, +swp_to_radix_entry (swap), gfp, + charge_mm); +/* +* We already confirmed swap under page lock, and make no
mm/shmem.c static int shmem_swapin_page(struct inode *inode, pgoff_t index, · line 1695
Quoted 10 lines
mm/shmem.c +9
-0+* memory allocation here, so usually no possibility of error; +* but free_swap_and_cache() only trylocks a page, so it is +* just possible that the entry has been truncated or +* holepunched since swap was confirmed. shmem_undo_range() +* will have done some of the unaccounting, now +* delete_from_swap_cache() will do the rest. +*/ +if (error) { +delete_from_swap_cache (page); goto failed;
-EEXIST (from swap cache) and -ENOMEM (from memcg) should be handled differently. delete_from_swap_cache() is for -EEXIST case.
Thanks.
Johannes Weiner
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 259 lines · Joonsoo Kim
Quoted 258 lines · Johannes Weiner
The try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a
mem_cgroup_charge()function that simply charges a newly allocated page to a cgroup and sets uppage->mem_cgroupin one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on
page->mappingbeing already set up at the time of charging, so that it can maintain its ownMEMCG_CACHEandMEMCG_RSScounters. For anon,page->mappingis set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the
page->mappingdependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.Signed-off-by Johannes Weiner <[email protected]>--- include/linux/memcontrol.h | 10 ++++ mm/filemap.c | 24 ++++------ mm/memcontrol.c | 27 +++++++++++ mm/shmem.c | 97 +++++++++++++++++--------------------- 4 files changed, 89 insertions(+), 69 deletions(-)include/linux/memcontrol.h +10
-0@@ -367,6 +367,10 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg,bool lrucare);void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); + +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare); +void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);@@ -872,6 +876,12 @@ static inline void mem_cgroup_cancel_charge(struct page *page, { } +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask,bool lrucare) +{ +return 0 ; +} +static inline void mem_cgroup_uncharge (struct page *page) { }mm/filemap.c +10
-14@@ -832,7 +832,6 @@ static int __add_to_page_cache_locked(struct page *page, {XA_STATE (xas, &mapping->i_pages, offset);int huge =PageHuge (page); -struct mem_cgroup *memcg;int error;void *old;@@ -840,17 +839,16 @@ static int __add_to_page_cache_locked(struct page *page,VM_BUG_ON_PAGE (PageSwapBacked (page), page);mapping_set_update (&xas, mapping); -if (!huge) { - error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg); -if (error) -return error; - } -get_page (page); page->mapping = mapping; page->index = offset; +if (!huge) { + error =mem_cgroup_charge (page, current->mm, gfp_mask, false); +if (error) + goto error; + } + do { xas_lock_irq(&xas); old = xas_load(&xas);@@ -874,20 +872,18 @@ static int __add_to_page_cache_locked(struct page *page,xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK )); -if (xas_error (&xas)) +if (xas_error (&xas)) { + error =xas_error (&xas); goto error; + } -if (!huge) -mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ -if (!huge) -mem_cgroup_cancel_charge (page, memcg);put_page (page); -return xas_error (&xas); +return error; }ALLOW_ERROR_INJECTION (__add_to_page_cache_locked,ERRNO );mm/memcontrol.c +27
-0@@ -6577,6 +6577,33 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg)cancel_charge (memcg, nr_pages); } +/** +* mem_cgroup_charge - charge a newly allocated page to a cgroup +* @page: page to charge +* @mm: mm context of the victim +* @gfp_mask: reclaim mode +* @lrucare: page might be on the LRU already +* +* Try to charge @page to the memcg that @mm belongs to, reclaiming +* pages according to @gfp_mask if necessary. +* +* Returns 0 on success. Otherwise, an error code is returned. +*/ +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare) +{ +struct mem_cgroup *memcg; +int ret; + +VM_BUG_ON_PAGE (!page->mapping, page); + + ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg); +if (ret) +return ret; +mem_cgroup_commit_charge (page, memcg, lrucare); +return 0 ; +} +struct uncharge_gather {struct mem_cgroup *memcg;unsigned long pgpgout;mm/shmem.c +29
-25@@ -605,11 +605,13 @@ static inline bool is_huge_enabled(struct shmem_sb_info *sbinfo)*/ static int shmem_add_to_page_cache (struct page *page,struct address_space *mapping, -pgoff_t index,void *expected,gfp_t gfp) +pgoff_t index,void *expected,gfp_t gfp, +struct mm_struct *charge_mm) {XA_STATE_ORDER (xas, &mapping->i_pages, index,compound_order (page));unsigned long i =0 ;unsigned long nr =compound_nr (page); +int error;VM_BUG_ON_PAGE (PageTail (page), page);VM_BUG_ON_PAGE (index !=round_down (index, nr), page);@@ -621,6 +623,16 @@ static int shmem_add_to_page_cache(struct page *page, page->mapping = mapping; page->index = index; + error =mem_cgroup_charge (page, charge_mm, gfp,PageSwapCache (page)); +if (error) { +if (!PageSwapCache (page) &&PageTransHuge (page)) { +count_vm_event (THP_FILE_FALLBACK ); +count_vm_event (THP_FILE_FALLBACK_CHARGE ); + } + goto error; + } +cgroup_throttle_swaprate (page, gfp); + do { void *entry; xas_lock_irq(&xas);@@ -648,12 +660,15 @@ static int shmem_add_to_page_cache(struct page *page, }while (xas_nomem (&xas, gfp));if (xas_error (&xas)) { - page->mapping =NULL ; -page_ref_sub (page, nr); -return xas_error (&xas); + error =xas_error (&xas); + goto error; }return 0 ; +error: + page->mapping =NULL ; +page_ref_sub (page, nr); +return error; }/* @@ -1619,7 +1634,6 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,struct address_space *mapping = inode->i_mapping;struct shmem_inode_info *info =SHMEM_I (inode);struct mm_struct *charge_mm = vma ? vma->vm_mm : current->mm; -struct mem_cgroup *memcg;struct page *page;swp_entry_t swap;int error;@@ -1664,29 +1678,22 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg); -if (!error) { - error =shmem_add_to_page_cache (page, mapping, index, -swp_to_radix_entry (swap), gfp); -/* -* We already confirmed swap under page lock, and make -* no memory allocation here, so usually no possibility -* of error; but free_swap_and_cache() only trylocks a -* page, so it is just possible that the entry has been -* truncated or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. -*/ -if (error) { -mem_cgroup_cancel_charge (page, memcg); -delete_from_swap_cache (page); - } - } -if (error) + error =shmem_add_to_page_cache (page, mapping, index, +swp_to_radix_entry (swap), gfp, + charge_mm); +/* +* We already confirmed swap under page lock, and make no +* memory allocation here, so usually no possibility of error; +* but free_swap_and_cache() only trylocks a page, so it is +* just possible that the entry has been truncated or +* holepunched since swap was confirmed. shmem_undo_range() +* will have done some of the unaccounting, now
Quoted 8 lines
mm/shmem.c static int shmem_swapin_page(struct inode *inode, pgoff_t index, · line 1695
Quoted 5 lines
mm/shmem.c +4
-0+* delete_from_swap_cache() will do the rest. +*/ +if (error) { +delete_from_swap_cache (page); goto failed;
-EEXIST(from swap cache) and-ENOMEM(from memcg) should be handled differently.delete_from_swap_cache()is for-EEXISTcase.
Good catch, I accidentally changed things here.
I was just going to change it back, but now I'm trying to understand how it actually works.
Who is removing the page from swap cache if shmem_undo_range() races but we fail to charge the page?
Here is how this race is supposed to be handled: The page is in the swapcache, we have it locked and confirmed that the entry in i_pages is indeed a swap entry. We charge the page, then we try to replace the swap entry in i_pages with the actual page. If we determine, under tree lock now, that shmem_undo_range has raced with us, unaccounted the swap space, but must have failed to get the page lock, we remove the page from swap cache on our side, to free up swap slot and page.
But what if shmem_undo_range() raced with us, deleted the swap entry from i_pages while we had the page locked, but then we simply failed to charge? We unlock the page and return -EEXIST (shmem_confirm_swap at the exit). The page with its userdata is now in swapcache, but no corresponding swap entry in i_pages. shmem_getpage_gfp() sees the -EEXIST, retries, finds nothing in i_pages and allocates a new, empty page.
Aren't we leaking the swap slot and the page?
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 293 lines · Johannes Weiner
Quoted 259 lines · Joonsoo Kim
Quoted 258 lines · Johannes Weiner
The try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a
mem_cgroup_charge()function that simply charges a newly allocated page to a cgroup and sets uppage->mem_cgroupin one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on
page->mappingbeing already set up at the time of charging, so that it can maintain its ownMEMCG_CACHEandMEMCG_RSScounters. For anon,page->mappingis set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the
page->mappingdependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.Signed-off-by Johannes Weiner <[email protected]>--- include/linux/memcontrol.h | 10 ++++ mm/filemap.c | 24 ++++------ mm/memcontrol.c | 27 +++++++++++ mm/shmem.c | 97 +++++++++++++++++--------------------- 4 files changed, 89 insertions(+), 69 deletions(-)include/linux/memcontrol.h +10
-0@@ -367,6 +367,10 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg,bool lrucare);void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); + +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare); +void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);@@ -872,6 +876,12 @@ static inline void mem_cgroup_cancel_charge(struct page *page, { } +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask,bool lrucare) +{ +return 0 ; +} +static inline void mem_cgroup_uncharge (struct page *page) { }mm/filemap.c +10
-14@@ -832,7 +832,6 @@ static int __add_to_page_cache_locked(struct page *page, {XA_STATE (xas, &mapping->i_pages, offset);int huge =PageHuge (page); -struct mem_cgroup *memcg;int error;void *old;@@ -840,17 +839,16 @@ static int __add_to_page_cache_locked(struct page *page,VM_BUG_ON_PAGE (PageSwapBacked (page), page);mapping_set_update (&xas, mapping); -if (!huge) { - error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg); -if (error) -return error; - } -get_page (page); page->mapping = mapping; page->index = offset; +if (!huge) { + error =mem_cgroup_charge (page, current->mm, gfp_mask, false); +if (error) + goto error; + } + do { xas_lock_irq(&xas); old = xas_load(&xas);@@ -874,20 +872,18 @@ static int __add_to_page_cache_locked(struct page *page,xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK )); -if (xas_error (&xas)) +if (xas_error (&xas)) { + error =xas_error (&xas); goto error; + } -if (!huge) -mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ -if (!huge) -mem_cgroup_cancel_charge (page, memcg);put_page (page); -return xas_error (&xas); +return error; }ALLOW_ERROR_INJECTION (__add_to_page_cache_locked,ERRNO );mm/memcontrol.c +27
-0@@ -6577,6 +6577,33 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg)cancel_charge (memcg, nr_pages); } +/** +* mem_cgroup_charge - charge a newly allocated page to a cgroup +* @page: page to charge +* @mm: mm context of the victim +* @gfp_mask: reclaim mode +* @lrucare: page might be on the LRU already +* +* Try to charge @page to the memcg that @mm belongs to, reclaiming +* pages according to @gfp_mask if necessary. +* +* Returns 0 on success. Otherwise, an error code is returned. +*/ +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare) +{ +struct mem_cgroup *memcg; +int ret; + +VM_BUG_ON_PAGE (!page->mapping, page); + + ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg); +if (ret) +return ret; +mem_cgroup_commit_charge (page, memcg, lrucare); +return 0 ; +} +struct uncharge_gather {struct mem_cgroup *memcg;unsigned long pgpgout;mm/shmem.c +29
-25@@ -605,11 +605,13 @@ static inline bool is_huge_enabled(struct shmem_sb_info *sbinfo)*/ static int shmem_add_to_page_cache (struct page *page,struct address_space *mapping, -pgoff_t index,void *expected,gfp_t gfp) +pgoff_t index,void *expected,gfp_t gfp, +struct mm_struct *charge_mm) {XA_STATE_ORDER (xas, &mapping->i_pages, index,compound_order (page));unsigned long i =0 ;unsigned long nr =compound_nr (page); +int error;VM_BUG_ON_PAGE (PageTail (page), page);VM_BUG_ON_PAGE (index !=round_down (index, nr), page);@@ -621,6 +623,16 @@ static int shmem_add_to_page_cache(struct page *page, page->mapping = mapping; page->index = index; + error =mem_cgroup_charge (page, charge_mm, gfp,PageSwapCache (page)); +if (error) { +if (!PageSwapCache (page) &&PageTransHuge (page)) { +count_vm_event (THP_FILE_FALLBACK ); +count_vm_event (THP_FILE_FALLBACK_CHARGE ); + } + goto error; + } +cgroup_throttle_swaprate (page, gfp); + do { void *entry; xas_lock_irq(&xas);@@ -648,12 +660,15 @@ static int shmem_add_to_page_cache(struct page *page, }while (xas_nomem (&xas, gfp));if (xas_error (&xas)) { - page->mapping =NULL ; -page_ref_sub (page, nr); -return xas_error (&xas); + error =xas_error (&xas); + goto error; }return 0 ; +error: + page->mapping =NULL ; +page_ref_sub (page, nr); +return error; }/* @@ -1619,7 +1634,6 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,struct address_space *mapping = inode->i_mapping;struct shmem_inode_info *info =SHMEM_I (inode);struct mm_struct *charge_mm = vma ? vma->vm_mm : current->mm; -struct mem_cgroup *memcg;struct page *page;swp_entry_t swap;int error;@@ -1664,29 +1678,22 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg); -if (!error) { - error =shmem_add_to_page_cache (page, mapping, index, -swp_to_radix_entry (swap), gfp); -/* -* We already confirmed swap under page lock, and make -* no memory allocation here, so usually no possibility -* of error; but free_swap_and_cache() only trylocks a -* page, so it is just possible that the entry has been -* truncated or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. -*/ -if (error) { -mem_cgroup_cancel_charge (page, memcg); -delete_from_swap_cache (page); - } - } -if (error) + error =shmem_add_to_page_cache (page, mapping, index, +swp_to_radix_entry (swap), gfp, + charge_mm); +/* +* We already confirmed swap under page lock, and make no +* memory allocation here, so usually no possibility of error; +* but free_swap_and_cache() only trylocks a page, so it is +* just possible that the entry has been truncated or +* holepunched since swap was confirmed. shmem_undo_range() +* will have done some of the unaccounting, now Quoted 8 lines
mm/shmem.c static int shmem_swapin_page(struct inode *inode, pgoff_t index, · line 1695
Quoted 5 lines
mm/shmem.c +4
-0+* delete_from_swap_cache() will do the rest. +*/ +if (error) { +delete_from_swap_cache (page); goto failed;
-EEXIST(from swap cache) and-ENOMEM(from memcg) should be handled differently.delete_from_swap_cache()is for-EEXISTcase.Good catch, I accidentally changed things here.
I was just going to change it back, but now I'm trying to understand how it actually works.
Who is removing the page from swap cache if
shmem_undo_range()races but we fail to charge the page?Here is how this race is supposed to be handled: The page is in the swapcache, we have it locked and confirmed that the entry in
i_pagesis indeed a swap entry. We charge the page, then we try to replace the swap entry ini_pageswith the actual page. If we determine, under tree lock now, thatshmem_undo_rangehas raced with us, unaccounted the swap space, but must have failed to get the page lock, we remove the page from swap cache on our side, to free up swap slot and page.But what if
shmem_undo_range()raced with us, deleted the swap entry fromi_pageswhile we had the page locked, but then we simply failed to charge? We unlock the page and return-EEXIST(shmem_confirm_swapat the exit). The page with its userdata is now in swapcache, but no corresponding swap entry ini_pages.shmem_getpage_gfp()sees the-EEXIST, retries, finds nothing ini_pagesand allocates a new, empty page.
Quoted 1 line
Aren't we leaking the swap slot and the page?
Yes, you're right! It seems that it's possible to leak the swap slot and the page. Race could happen for all the places after lock_page() and shmem_confirm_swap() are done. And, I think that it's not possible to fix the problem in shmem_swapin_page() side since we can't know the timing that trylock_page() is called. Maybe, solution would be, instead of using free_swap_and_cache() in shmem_undo_range() that calls trylock_page(), to use another function that calls lock_page().
Thanks.
Johannes Weiner
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 72 lines · Joonsoo Kim
Quoted 69 lines · Johannes Weiner
Quoted 35 lines · Joonsoo Kim
Quoted 34 lines · Johannes Weiner
(quoted diff) +10
-20@@ -1664,29 +1678,22 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error = mem_cgroup_try_charge_delay(page, charge_mm, gfp, &memcg); - if (!error) { - error = shmem_add_to_page_cache(page, mapping, index, - swp_to_radix_entry(swap), gfp); - /* - * We already confirmed swap under page lock, and make - * no memory allocation here, so usually no possibility - * of error; but free_swap_and_cache() only trylocks a - * page, so it is just possible that the entry has been - * truncated or holepunched since swap was confirmed. - * shmem_undo_range() will have done some of the - * unaccounting, now delete_from_swap_cache() will do - * the rest. - */ - if (error) { - mem_cgroup_cancel_charge(page, memcg); - delete_from_swap_cache(page); - } - } - if (error) + error = shmem_add_to_page_cache(page, mapping, index, + swp_to_radix_entry(swap), gfp, + charge_mm); + /* + * We already confirmed swap under page lock, and make no + * memory allocation here, so usually no possibility of error; + * but free_swap_and_cache() only trylocks a page, so it is + * just possible that the entry has been truncated or + * holepunched since swap was confirmed. shmem_undo_range() + * will have done some of the unaccounting, nowQuoted 8 lines
Review static int shmem_swapin_page(struct inode *inode, pgoff_t index, · line 1695
Quoted 5 lines
(quoted diff) +4
-0+ * delete_from_swap_cache() will do the rest. + */ + if (error) { + delete_from_swap_cache(page); goto failed;
-EEXIST(from swap cache) and-ENOMEM(from memcg) should be handled differently.delete_from_swap_cache()is for-EEXISTcase.Good catch, I accidentally changed things here.
I was just going to change it back, but now I'm trying to understand how it actually works.
Who is removing the page from swap cache if
shmem_undo_range()races but we fail to charge the page?Here is how this race is supposed to be handled: The page is in the swapcache, we have it locked and confirmed that the entry in
i_pagesis indeed a swap entry. We charge the page, then we try to replace the swap entry ini_pageswith the actual page. If we determine, under tree lock now, thatshmem_undo_rangehas raced with us, unaccounted the swap space, but must have failed to get the page lock, we remove the page from swap cache on our side, to free up swap slot and page.But what if
shmem_undo_range()raced with us, deleted the swap entry fromi_pageswhile we had the page locked, but then we simply failed to charge? We unlock the page and return-EEXIST(shmem_confirm_swapat the exit). The page with its userdata is now in swapcache, but no corresponding swap entry ini_pages.shmem_getpage_gfp()sees the-EEXIST, retries, finds nothing ini_pagesand allocates a new, empty page.Quoted 1 line
Aren't we leaking the swap slot and the page?
Quoted 7 lines
Yes, you're right! It seems that it's possible to leak the swap slot and the page. Race could happen for all the places after
lock_page()andshmem_confirm_swap()are done. And, I think that it's not possible to fix the problem inshmem_swapin_page()side since we can't know the timing thattrylock_page()is called. Maybe, solution would be, instead of usingfree_swap_and_cache()inshmem_undo_range()that callstrylock_page(), to use another function that callslock_page().
I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the zap_pte_range() code calls free_swap_and_cache() and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.
The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
That being said, my patch introduces potentially undesirable behavior (although AFAICS no correctness problem): We should only delete the page from swapcache when we actually raced with undo_range - which we see from the swap entry having been purged from the page cache tree. If we delete the page from swapcache just because we failed to charge it, the next fault has to read the still-valid page again from the swap device.
I'm going to include this:
mm/shmem.c +12 -11
@@ -640,7 +640,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, -EEXIST ); +xas_set_err (&xas, expected ? -ENOENT : -EEXIST );xas_create_range (&xas);if (xas_error (&xas)) goto unlock;
@@ -1683,17 +1683,18 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, error =shmem_add_to_page_cache (page, mapping, index,swp_to_radix_entry (swap), gfp, charge_mm); -/* -* We already confirmed swap under page lock, and make no -* memory allocation here, so usually no possibility of error; -* but free_swap_and_cache() only trylocks a page, so it is -* just possible that the entry has been truncated or -* holepunched since swap was confirmed. shmem_undo_range() -* will have done some of the unaccounting, now -* delete_from_swap_cache() will do the rest. -*/ if (error) { -delete_from_swap_cache (page); +/* +* We already confirmed swap under page lock, but +* free_swap_and_cache() only trylocks a page, so it +* is just possible that the entry has been truncated +* or holepunched since swap was confirmed. +* shmem_undo_range() will have done some of the +* unaccounting, now delete_from_swap_cache() will do +* the rest. +*/ +if (error == -ENOENT ) +delete_from_swap_cache (page); goto failed; }
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 97 lines · Johannes Weiner
Quoted 72 lines · Joonsoo Kim
Quoted 69 lines · Johannes Weiner
Quoted 35 lines · Joonsoo Kim
Quoted 34 lines · Johannes Weiner
(quoted diff) +10
-20@@ -1664,29 +1678,22 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, goto failed; } - error = mem_cgroup_try_charge_delay(page, charge_mm, gfp, &memcg); - if (!error) { - error = shmem_add_to_page_cache(page, mapping, index, - swp_to_radix_entry(swap), gfp); - /* - * We already confirmed swap under page lock, and make - * no memory allocation here, so usually no possibility - * of error; but free_swap_and_cache() only trylocks a - * page, so it is just possible that the entry has been - * truncated or holepunched since swap was confirmed. - * shmem_undo_range() will have done some of the - * unaccounting, now delete_from_swap_cache() will do - * the rest. - */ - if (error) { - mem_cgroup_cancel_charge(page, memcg); - delete_from_swap_cache(page); - } - } - if (error) + error = shmem_add_to_page_cache(page, mapping, index, + swp_to_radix_entry(swap), gfp, + charge_mm); + /* + * We already confirmed swap under page lock, and make no + * memory allocation here, so usually no possibility of error; + * but free_swap_and_cache() only trylocks a page, so it is + * just possible that the entry has been truncated or + * holepunched since swap was confirmed. shmem_undo_range() + * will have done some of the unaccounting, nowQuoted 8 lines
mm/shmem.c static int shmem_swapin_page(struct inode *inode, pgoff_t index, · line 1698
Quoted 5 lines
mm/shmem.c +4
-0+* delete_from_swap_cache() will do the rest. +*/ +if (error) { +delete_from_swap_cache (page); goto failed;
-EEXIST(from swap cache) and-ENOMEM(from memcg) should be handled differently.delete_from_swap_cache()is for-EEXISTcase.Good catch, I accidentally changed things here.
I was just going to change it back, but now I'm trying to understand how it actually works.
Who is removing the page from swap cache if
shmem_undo_range()races but we fail to charge the page?Here is how this race is supposed to be handled: The page is in the swapcache, we have it locked and confirmed that the entry in
i_pagesis indeed a swap entry. We charge the page, then we try to replace the swap entry ini_pageswith the actual page. If we determine, under tree lock now, thatshmem_undo_rangehas raced with us, unaccounted the swap space, but must have failed to get the page lock, we remove the page from swap cache on our side, to free up swap slot and page.But what if
shmem_undo_range()raced with us, deleted the swap entry fromi_pageswhile we had the page locked, but then we simply failed to charge? We unlock the page and return-EEXIST(shmem_confirm_swapat the exit). The page with its userdata is now in swapcache, but no corresponding swap entry ini_pages.shmem_getpage_gfp()sees the-EEXIST, retries, finds nothing ini_pagesand allocates a new, empty page.Quoted 1 line
Aren't we leaking the swap slot and the page?
Quoted 7 lines
Yes, you're right! It seems that it's possible to leak the swap slot and the page. Race could happen for all the places after
lock_page()andshmem_confirm_swap()are done. And, I think that it's not possible to fix the problem inshmem_swapin_page()side since we can't know the timing thattrylock_page()is called. Maybe, solution would be, instead of usingfree_swap_and_cache()inshmem_undo_range()that callstrylock_page(), to use another function that callslock_page().I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the
zap_pte_range()code callsfree_swap_and_cache()and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
Quoted 7 lines
That being said, my patch introduces potentially undesirable behavior (although AFAICS no correctness problem): We should only delete the page from swapcache when we actually raced with
undo_range- which we see from the swap entry having been purged from the page cache tree. If we delete the page from swapcache just because we failed to charge it, the next fault has to read the still-valid page again from the swap device.
I got it! Thanks for explanation.
Thanks.
Hugh Dickins
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
In reply to Johannes WeinerQuoted 12 lines · Johannes Weiner
I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the
zap_pte_range()code callsfree_swap_and_cache()and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.
Quoted 4 lines
The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
I think you've got all that exactly right, thanks for working it out. It originates from v3.7's 215c02bc33bb ("tmpfs: fix shmem_getpage_gfp() VM_BUG_ON") - in which I also had to thank you.
I think I chose to do the delete_from_swap_cache() right there, partly because of following shmem_unuse_inode() code which already did that, partly on the basis that while we have to observe the case then it's better to clean it up, and partly out of guilt that our page lock here is what had prevented shmem_undo_range() from completing its job; but I believe you're right that unused swapcache reclaim would sort it out eventually.
Quoted 8 lines
That being said, my patch introduces potentially undesirable behavior (although AFAICS no correctness problem): We should only delete the page from swapcache when we actually raced with
undo_range- which we see from the swap entry having been purged from the page cache tree. If we delete the page from swapcache just because we failed to charge it, the next fault has to read the still-valid page again from the swap device.
Yes.
Quoted 2 lines
I'm going to include this:
I haven't pulled down your V2 series into a tree yet (expecting perhaps a respin from Alex on top, when I hope to switch over to trying them both), so haven't looked into the context and may be wrong...
mm/shmem.c static int shmem_add_to_page_cache(struct page *page, · line 643
Quoted 11 lines
mm/shmem.c +1
-1@@ -640,7 +640,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, -EEXIST ); +xas_set_err (&xas, expected ? -ENOENT : -EEXIST );
Two things on this.
Minor matter of taste, I'd prefer that as
xas_set_err(&xas, entry ? -EEXIST : -ENOENT);
which would be more general and more understandable - but what you have written should be fine for the actual callers.
Except... I think returning -ENOENT there will not work correctly, in the case of a punched hole. Because (unless you've reworked it and I just haven't looked) shmem_getpage_gfp() knows to retry in the case of -EEXIST, but -ENOENT will percolate up to shmem_fault() and result in a SIGBUS, or a read/write error, when the hole should just get refilled instead.
Not something that needs fixing in a hurry (it took trinity to generate this racy case in the first place), I'll take another look once I've pulled it into a tree (or collected next mmotm) - unless you've already have changed it around by then.
Hugh
Quoted 33 lines
mm/shmem.c +11
-10xas_create_range (&xas);if (xas_error (&xas)) goto unlock;@@ -1683,17 +1683,18 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, error =shmem_add_to_page_cache (page, mapping, index,swp_to_radix_entry (swap), gfp, charge_mm); -/* -* We already confirmed swap under page lock, and make no -* memory allocation here, so usually no possibility of error; -* but free_swap_and_cache() only trylocks a page, so it is -* just possible that the entry has been truncated or -* holepunched since swap was confirmed. shmem_undo_range() -* will have done some of the unaccounting, now -* delete_from_swap_cache() will do the rest. -*/ if (error) { -delete_from_swap_cache (page); +/* +* We already confirmed swap under page lock, but +* free_swap_and_cache() only trylocks a page, so it +* is just possible that the entry has been truncated +* or holepunched since swap was confirmed. +* shmem_undo_range() will have done some of the +* unaccounting, now delete_from_swap_cache() will do +* the rest. +*/ +if (error == -ENOENT ) +delete_from_swap_cache (page); goto failed; }
Johannes Weiner
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 13 lines · Hugh Dickins
Quoted 12 lines · Johannes Weiner
I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the
zap_pte_range()code callsfree_swap_and_cache()and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.
Quoted 8 lines
Quoted 4 lines
The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
I think you've got all that exactly right, thanks for working it out. It originates from v3.7's
215c02bc33bb("tmpfs: fixshmem_getpage_gfp()VM_BUG_ON") - in which I also had to thank you.
I should have looked where it actually came from - I had forgotten about that patch!
Quoted 7 lines
I think I chose to do the
delete_from_swap_cache()right there, partly because of followingshmem_unuse_inode()code which already did that, partly on the basis that while we have to observe the case then it's better to clean it up, and partly out of guilt that our page lock here is what had preventedshmem_undo_range()from completing its job; but I believe you're right that unused swapcache reclaim would sort it out eventually.
That makes sense to me.
Quoted 13 lines
mm/shmem.c static int shmem_add_to_page_cache(struct page *page, · line 643
Quoted 10 lines
mm/shmem.c +1
-1@@ -640,7 +640,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, -EEXIST ); +xas_set_err (&xas, expected ? -ENOENT : -EEXIST );Two things on this.
Quoted 4 lines
Minor matter of taste, I'd prefer that as
xas_set_err(&xas, entry ? -EEXIST : -ENOENT);which would be more general and more understandable - but what you have written should be fine for the actual callers.
Yes, checking expected' was to differentiate the behavior depending on the callsite. But testing entry' is more obvious in that location.
Quoted 6 lines
Except... I think returning
-ENOENTthere will not work correctly, in the case of a punched hole. Because (unless you've reworked it and I just haven't looked)shmem_getpage_gfp()knows to retry in the case of-EEXIST, but-ENOENTwill percolate up toshmem_fault()and result in a SIGBUS, or a read/write error, when the hole should just get refilled instead.
Good catch, I had indeed missed that. I'm going to make it retry on -ENOENT as well.
We could have it go directly to allocating a new page, but it seems unnecessarily complicated: we've already been retrying in this situation until now, so I would stick to "there was a race, retry."
Quoted 4 lines
Not something that needs fixing in a hurry (it took trinity to generate this racy case in the first place), I'll take another look once I've pulled it into a tree (or collected next mmotm) - unless you've already have changed it around by then.
Attaching a delta fix based on your observations.
Andrew, barring any objections to this, could you please fold it into the version you have in your tree already?
---
From 33d03ceebce0a6261d472ddc9c5a07940f44714c Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Mon, 11 May 2020 10:45:14 -0400
Subject: [PATCH] mm: memcontrol: convert page cache to a new
mem_cgroup_charge() API fix
Incorporate Hugh's feedback:
- shmem_getpage_gfp() needs to handle the new -ENOENT that was previously implied in the -EEXIST when a swap entry changed under us in any way. Otherwise hole punching could cause a racing fault to SIGBUS instead of allocating a new page.
- It is indeed page reclaim that picks up any swapcache we leave stranded when free_swap_and_cache() runs on a page locked by somebody else. Document that our delete_from_swap_cache() is an optimization, not something we rely on for correctness.
- Style cleanup: testing expected' to decide on -EEXIST vs -ENOENT differentiates the callsites, but is a bit awkward to read. Test entry' instead.
--- mm/shmem.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
mm/shmem.c +9 -6
@@ -638,7 +638,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, expected ? -ENOENT : -EEXIST ); +xas_set_err (&xas, entry ? -EEXIST : -ENOENT );xas_create_range (&xas);if (xas_error (&xas)) goto unlock;
@@ -1686,10 +1686,13 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,* We already confirmed swap under page lock, but * free_swap_and_cache() only trylocks a page, so it * is just possible that the entry has been truncated -* or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. +* or holepunched since swap was confirmed. This could +* occur at any time while the page is locked, and +* usually page reclaim will take care of the stranded +* swapcache page. But when we catch it, we may as +* well clean up after ourselves: shmem_undo_range() +* will have done some of the unaccounting, now +* delete_from_swap_cache() will do the rest. */ if (error == -ENOENT )delete_from_swap_cache (page);
@@ -1765,7 +1768,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,if (xa_is_value (page)) { error =shmem_swapin_page (inode, index, &page, sgp, gfp, vma, fault_type); -if (error == -EEXIST ) +if (error == -EEXIST || error == -ENOENT ) goto repeat; *pagep = page;
Signature · 1 line
2.26.2
Hugh Dickins
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 76 lines · Johannes Weiner
Quoted 13 lines · Hugh Dickins
Quoted 12 lines · Johannes Weiner
I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the
zap_pte_range()code callsfree_swap_and_cache()and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.Quoted 8 lines
Quoted 4 lines
The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
I think you've got all that exactly right, thanks for working it out. It originates from v3.7's
215c02bc33bb("tmpfs: fixshmem_getpage_gfp()VM_BUG_ON") - in which I also had to thank you.I should have looked where it actually came from - I had forgotten about that patch!
Quoted 7 lines
I think I chose to do the
delete_from_swap_cache()right there, partly because of followingshmem_unuse_inode()code which already did that, partly on the basis that while we have to observe the case then it's better to clean it up, and partly out of guilt that our page lock here is what had preventedshmem_undo_range()from completing its job; but I believe you're right that unused swapcache reclaim would sort it out eventually.That makes sense to me.
Quoted 13 lines
mm/shmem.c static int shmem_add_to_page_cache(struct page *page, · line 643
Quoted 10 lines
mm/shmem.c +1
-1@@ -640,7 +640,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, -EEXIST ); +xas_set_err (&xas, expected ? -ENOENT : -EEXIST );Two things on this.
Quoted 4 lines
Minor matter of taste, I'd prefer that as
xas_set_err(&xas, entry ? -EEXIST : -ENOENT);which would be more general and more understandable - but what you have written should be fine for the actual callers.
Yes, checking
expected' was to differentiate the behavior depending on the callsite. But testingentry' is more obvious in that location.Quoted 6 lines
Except... I think returning
-ENOENTthere will not work correctly, in the case of a punched hole. Because (unless you've reworked it and I just haven't looked)shmem_getpage_gfp()knows to retry in the case of-EEXIST, but-ENOENTwill percolate up toshmem_fault()and result in a SIGBUS, or a read/write error, when the hole should just get refilled instead.Good catch, I had indeed missed that. I'm going to make it retry on
-ENOENTas well.We could have it go directly to allocating a new page, but it seems unnecessarily complicated: we've already been retrying in this situation until now, so I would stick to "there was a race, retry."
Quoted 4 lines
Not something that needs fixing in a hurry (it took trinity to generate this racy case in the first place), I'll take another look once I've pulled it into a tree (or collected next mmotm) - unless you've already have changed it around by then.
Quoted 4 lines
Attaching a delta fix based on your observations.
Andrew, barring any objections to this, could you please fold it into the version you have in your tree already?
Not so strong as an objection, and I won't get to see whether your retry on -ENOENT is good (can -ENOENT arrive at that point from any other case, that might endlessly retry?) until I've got the full context; but I had arrived at the opposite conclusion overnight.
Given that this case only appeared with a fuzzer, and stale swapcache reclaim is anyway relied upon to clean up after plenty of other such races, I think we should agree that I over-complicated the VM_BUG_ON removal originally, and it's best to kill that delete_from_swap_cache(), and the comment having to explain it, and your EEXIST/ENOENT distinction.
(I haven't checked, but I suspect that the shmem_unuse_inode() case that I copied from, actually really needed to delete_from_swap_cache(), in order to swapoff the page without full retry of the big swapoff loop.)
Hugh
Quoted 73 lines
---
From
33d03ceebce0a6261d472ddc9c5a07940f44714cMon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Mon, 11 May 2020 10:45:14 -0400
Subject: [PATCH] mm: memcontrol: convert page cache to a new
mem_cgroup_charge()API fixIncorporate Hugh's feedback:
-
shmem_getpage_gfp()needs to handle the new-ENOENTthat was previously implied in the-EEXISTwhen a swap entry changed under us in any way. Otherwise hole punching could cause a racing fault to SIGBUS instead of allocating a new page.- It is indeed page reclaim that picks up any swapcache we leave stranded when
free_swap_and_cache()runs on a page locked by somebody else. Document that ourdelete_from_swap_cache()is an optimization, not something we rely on for correctness.- Style cleanup: testing
expected' to decide on -EEXIST vs -ENOENT differentiates the callsites, but is a bit awkward to read. Testentry' instead.Signed-off-by Johannes Weiner <[email protected]>--- mm/shmem.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)mm/shmem.c +9
-6@@ -638,7 +638,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, expected ? -ENOENT : -EEXIST ); +xas_set_err (&xas, entry ? -EEXIST : -ENOENT );xas_create_range (&xas);if (xas_error (&xas)) goto unlock;@@ -1686,10 +1686,13 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index,* We already confirmed swap under page lock, but * free_swap_and_cache() only trylocks a page, so it * is just possible that the entry has been truncated -* or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. +* or holepunched since swap was confirmed. This could +* occur at any time while the page is locked, and +* usually page reclaim will take care of the stranded +* swapcache page. But when we catch it, we may as +* well clean up after ourselves: shmem_undo_range() +* will have done some of the unaccounting, now +* delete_from_swap_cache() will do the rest. */ if (error == -ENOENT )delete_from_swap_cache (page);@@ -1765,7 +1768,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,if (xa_is_value (page)) { error =shmem_swapin_page (inode, index, &page, sgp, gfp, vma, fault_type); -if (error == -EEXIST ) +if (error == -EEXIST || error == -ENOENT ) goto repeat; *pagep = page;Signature · 1 line
2.26.2
Johannes Weiner
maintainerRe: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 93 lines · Hugh Dickins
Quoted 76 lines · Johannes Weiner
Quoted 13 lines · Hugh Dickins
Quoted 12 lines · Johannes Weiner
I looked at this some more, as well as compared it to non-shmem swapping. My conclusion is - and Hugh may correct me on this - that the deletion looks mandatory but is actually an optimization. Page reclaim will ultimately pick these pages up.
When non-shmem pages are swapped in by readahead (locked until IO completes) and their page tables are simultaneously unmapped, the
zap_pte_range()code callsfree_swap_and_cache()and the locked pages are stranded in the swap cache with no page table references. We rely on page reclaim to pick them up later on.Quoted 8 lines
Quoted 4 lines
The same appears to be true for shmem. If the references to the swap page are zapped while we're trying to swap in, we can strand the page in the swap cache. But it's not up to swapin to detect this reliably, it just frees the page more quickly than having to wait for reclaim.
I think you've got all that exactly right, thanks for working it out. It originates from v3.7's
215c02bc33bb("tmpfs: fixshmem_getpage_gfp()VM_BUG_ON") - in which I also had to thank you.I should have looked where it actually came from - I had forgotten about that patch!
Quoted 7 lines
I think I chose to do the
delete_from_swap_cache()right there, partly because of followingshmem_unuse_inode()code which already did that, partly on the basis that while we have to observe the case then it's better to clean it up, and partly out of guilt that our page lock here is what had preventedshmem_undo_range()from completing its job; but I believe you're right that unused swapcache reclaim would sort it out eventually.That makes sense to me.
Quoted 13 lines
mm/shmem.c static int shmem_add_to_page_cache(struct page *page, · line 643
Quoted 10 lines
mm/shmem.c +1
-1@@ -640,7 +640,7 @@ static int shmem_add_to_page_cache(struct page *page,xas_lock_irq (&xas); entry =xas_find_conflict (&xas);if (entry != expected) -xas_set_err (&xas, -EEXIST ); +xas_set_err (&xas, expected ? -ENOENT : -EEXIST );Two things on this.
Quoted 4 lines
Minor matter of taste, I'd prefer that as
xas_set_err(&xas, entry ? -EEXIST : -ENOENT);which would be more general and more understandable - but what you have written should be fine for the actual callers.
Yes, checking
expected' was to differentiate the behavior depending on the callsite. But testingentry' is more obvious in that location.Quoted 6 lines
Except... I think returning
-ENOENTthere will not work correctly, in the case of a punched hole. Because (unless you've reworked it and I just haven't looked)shmem_getpage_gfp()knows to retry in the case of-EEXIST, but-ENOENTwill percolate up toshmem_fault()and result in a SIGBUS, or a read/write error, when the hole should just get refilled instead.Good catch, I had indeed missed that. I'm going to make it retry on
-ENOENTas well.We could have it go directly to allocating a new page, but it seems unnecessarily complicated: we've already been retrying in this situation until now, so I would stick to "there was a race, retry."
Quoted 4 lines
Not something that needs fixing in a hurry (it took trinity to generate this racy case in the first place), I'll take another look once I've pulled it into a tree (or collected next mmotm) - unless you've already have changed it around by then.
Quoted 4 lines
Attaching a delta fix based on your observations.
Andrew, barring any objections to this, could you please fold it into the version you have in your tree already?
Not so strong as an objection, and I won't get to see whether your retry on
-ENOENTis good (can-ENOENTarrive at that point from any other case, that might endlessly retry?) until I've got the full context; but I had arrived at the opposite conclusion overnight.Given that this case only appeared with a fuzzer, and stale swapcache reclaim is anyway relied upon to clean up after plenty of other such races, I think we should agree that I over-complicated the
VM_BUG_ONremoval originally, and it's best to kill thatdelete_from_swap_cache(), and the comment having to explain it, and your EEXIST/ENOENT distinction.
Quoted 3 lines
(I haven't checked, but I suspect that the
shmem_unuse_inode()case that I copied from, actually really needed todelete_from_swap_cache(), in order to swapoff the page without full retry of the big swapoff loop.)
Since commit b56a2d8af914 ("mm: rid swapoff of quadratic complexity"), shmem_unuse_inode() doesn't have its own copy anymore - it uses shmem_swapin_page().
However, that commit appears to have made shmem's private call to delete_from_swap_cache() obsolete as well. Whereas before this change we fully relied on shmem_unuse() to find and clear a shmem swap entry and its swapcache page, we now only need it to clean out shmem's private state in the inode, as it's followed by a loop over all remaining swap slots, calling try_to_free_swap() on stragglers.
Unless I missed something, it's still merely an optimization, and we can delete it for simplicity:
---
From fc9dcaf68c8b54baf365cd670fb5780c7f0d243f Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Mon, 11 May 2020 12:59:08 -0400
Subject: [PATCH] mm: shmem: remove rare optimization when swapin races with
hole punching
Commit 215c02bc33bb ("tmpfs: fix shmem_getpage_gfp() VM_BUG_ON") recognized that hole punching can race with swapin and removed the BUG_ON() for a truncated entry from the swapin path.
The patch also added a swapcache deletion to optimize this rare case:
Since swapin has the page locked, and free_swap_and_cache() merely trylocks, this situation can leave the page stranded in swapcache. Usually, page reclaim picks up stale swapcache pages, and the race can happen at any other time when the page is locked. (The same happens for non-shmem swapin racing with page table zapping.) The thinking here was: we already observed the race and we have the page locked, we may as well do the cleanup instead of waiting for reclaim.
However, this optimization complicates the next patch which moves the cgroup charging code around. As this is just a minor speedup for a race condition that is so rare that it required a fuzzer to trigger the original BUG_ON(), it's no longer worth the complications.
--- mm/shmem.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-)
mm/shmem.c +7 -18
@@ -1665,27 +1665,16 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, } error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg); -if (!error) { - error =shmem_add_to_page_cache (page, mapping, index, -swp_to_radix_entry (swap), gfp); -/* -* We already confirmed swap under page lock, and make -* no memory allocation here, so usually no possibility -* of error; but free_swap_and_cache() only trylocks a -* page, so it is just possible that the entry has been -* truncated or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. -*/ -if (error) { -mem_cgroup_cancel_charge (page, memcg); -delete_from_swap_cache (page); - } - }if (error) goto failed; + error =shmem_add_to_page_cache (page, mapping, index, +swp_to_radix_entry (swap), gfp); +if (error) { +mem_cgroup_cancel_charge (page, memcg); + goto failed; + } +mem_cgroup_commit_charge (page, memcg, true);spin_lock_irq (&info->lock);
Signature · 1 line
2.26.2
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
Quoted 5 lines · Johannes Weiner
From
fc9dcaf68c8b54baf365cd670fb5780c7f0d243fMon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Mon, 11 May 2020 12:59:08 -0400
Subject: [PATCH] mm: shmem: remove rare optimization when swapin races with
hole punching
And a new, conflict-resolved version of the patch this thread is attached to:
---
From 7f630d9bc5d6f692298fd906edd5f48070b257c7 Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Thu, 16 Apr 2020 15:08:07 -0400
Subject: [PATCH] mm: memcontrol: convert page cache to a new
mem_cgroup_charge() API
The try/commit/cancel protocol that memcg uses dates back to when pages used to be uncharged upon removal from the page cache, and thus couldn't be committed before the insertion had succeeded. Nowadays, pages are uncharged when they are physically freed; it doesn't matter whether the insertion was successful or not. For the page cache, the transaction dance has become unnecessary.
Introduce a mem_cgroup_charge() function that simply charges a newly allocated page to a cgroup and sets up page->mem_cgroup in one single step. If the insertion fails, the caller doesn't have to do anything but free/put the page.
Then switch the page cache over to this new API.
Subsequent patches will also convert anon pages, but it needs a bit more prep work. Right now, memcg depends on page->mapping being already set up at the time of charging, so that it can maintain its own MEMCG_CACHE and MEMCG_RSS counters. For anon, page->mapping is set under the same pte lock under which the page is publishd, so a single charge point that can block doesn't work there just yet.
The following prep patches will replace the private memcg counters with the generic vmstat counters, thus removing the page->mapping dependency, then complete the transition to the new single-point charge API and delete the old transactional scheme.
v2: leave shmem swapcache when charging fails to avoid double IO (Joonsoo)
v3: rebase on preceeding shmem simplification patch
--- include/linux/memcontrol.h | 10 ++++++ mm/filemap.c | 24 ++++++------- mm/memcontrol.c | 29 +++++++++++++-- mm/shmem.c | 73 ++++++++++++++++---------------------- 4 files changed, 77 insertions(+), 59 deletions(-)
include/linux/memcontrol.h +10 -0
@@ -379,6 +379,10 @@ int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm,void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg,bool lrucare);void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); + +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare); +void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);
@@ -893,6 +897,12 @@ static inline void mem_cgroup_cancel_charge(struct page *page, { } +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask,bool lrucare) +{ +return 0 ; +} +static inline void mem_cgroup_uncharge (struct page *page) { }
mm/filemap.c +10 -14
@@ -832,7 +832,6 @@ static int __add_to_page_cache_locked(struct page *page, {XA_STATE (xas, &mapping->i_pages, offset);int huge =PageHuge (page); -struct mem_cgroup *memcg;int error;void *old;
@@ -840,17 +839,16 @@ static int __add_to_page_cache_locked(struct page *page,VM_BUG_ON_PAGE (PageSwapBacked (page), page);mapping_set_update (&xas, mapping); -if (!huge) { - error =mem_cgroup_try_charge (page, current->mm, - gfp_mask, &memcg); -if (error) -return error; - } -get_page (page); page->mapping = mapping; page->index = offset; +if (!huge) { + error =mem_cgroup_charge (page, current->mm, gfp_mask, false); +if (error) + goto error; + } + do { xas_lock_irq(&xas); old = xas_load(&xas);
@@ -874,20 +872,18 @@ static int __add_to_page_cache_locked(struct page *page,xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK )); -if (xas_error (&xas)) +if (xas_error (&xas)) { + error =xas_error (&xas); goto error; + } -if (!huge) -mem_cgroup_commit_charge (page, memcg, false);trace_mm_filemap_add_to_page_cache (page);return 0 ; error: page->mapping =NULL ;/* Leave page->index set: truncation relies upon it */ -if (!huge) -mem_cgroup_cancel_charge (page, memcg);put_page (page); -return xas_error (&xas); +return error; }ALLOW_ERROR_INJECTION (__add_to_page_cache_locked,ERRNO );
mm/memcontrol.c +27 -2
@@ -6578,6 +6578,33 @@ void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg)cancel_charge (memcg, nr_pages); } +/** +* mem_cgroup_charge - charge a newly allocated page to a cgroup +* @page: page to charge +* @mm: mm context of the victim +* @gfp_mask: reclaim mode +* @lrucare: page might be on the LRU already +* +* Try to charge @page to the memcg that @mm belongs to, reclaiming +* pages according to @gfp_mask if necessary. +* +* Returns 0 on success. Otherwise, an error code is returned. +*/ +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, +bool lrucare) +{ +struct mem_cgroup *memcg; +int ret; + +VM_BUG_ON_PAGE (!page->mapping, page); + + ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg); +if (ret) +return ret; +mem_cgroup_commit_charge (page, memcg, lrucare); +return 0 ; +} +struct uncharge_gather {struct mem_cgroup *memcg;unsigned long pgpgout;
@@ -6625,8 +6652,6 @@ static void uncharge_batch(const struct uncharge_gather *ug)static void uncharge_page (struct page *page,struct uncharge_gather *ug) {VM_BUG_ON_PAGE (PageLRU (page), page); -VM_BUG_ON_PAGE (page_count (page) && !is_zone_device_page (page) && - !PageHWPoison (page) , page);if (!page->mem_cgroup)return ;
mm/shmem.c +30 -43
158 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH 05/18] mm: memcontrol: convert page cache to a new mem_cgroup_charge() API
In reply to Johannes WeinerQuoted 11 lines · Johannes Weiner
Since commit
b56a2d8af914("mm: rid swapoff of quadratic complexity"),shmem_unuse_inode()doesn't have its own copy anymore - it usesshmem_swapin_page().However, that commit appears to have made shmem's private call to
delete_from_swap_cache()obsolete as well. Whereas before this change we fully relied onshmem_unuse()to find and clear a shmem swap entry and its swapcache page, we now only need it to clean out shmem's private state in the inode, as it's followed by a loop over all remaining swap slots, callingtry_to_free_swap()on stragglers.
Great, you've looked deeper into the current situation than I had.
Quoted 3 lines
Unless I missed something, it's still merely an optimization, and we can delete it for simplicity:
Yes, nice ---s, simpler code, and a good idea to separate it out as a precursor: thanks, Hannes.
Quoted 22 lines
---
From
fc9dcaf68c8b54baf365cd670fb5780c7f0d243fMon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Mon, 11 May 2020 12:59:08 -0400
Subject: [PATCH] mm: shmem: remove rare optimization when swapin races with
hole punchingCommit
215c02bc33bb("tmpfs: fixshmem_getpage_gfp()VM_BUG_ON") recognized that hole punching can race with swapin and removed theBUG_ON()for a truncated entry from the swapin path.The patch also added a swapcache deletion to optimize this rare case:
Since swapin has the page locked, andfree_swap_and_cache()merely trylocks, this situation can leave the page stranded in swapcache. Usually, page reclaim picks up stale swapcache pages, and the race can happen at any other time when the page is locked. (The same happens for non-shmem swapin racing with page table zapping.) The thinking here was: we already observed the race and we have the page locked, we may as well do the cleanup instead of waiting for reclaim.
Quoted 7 lines
However, this optimization complicates the next patch which moves the cgroup charging code around. As this is just a minor speedup for a race condition that is so rare that it required a fuzzer to trigger the original
BUG_ON(), it's no longer worth the complications.Suggested-by Hugh Dickins <[email protected]>Signed-off-by Johannes Weiner <[email protected]>
(if one is allowed to suggest and to ack)
Quoted 47 lines
--- mm/shmem.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-)mm/shmem.c +7
-18@@ -1665,27 +1665,16 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, } error =mem_cgroup_try_charge_delay (page, charge_mm, gfp, &memcg); -if (!error) { - error =shmem_add_to_page_cache (page, mapping, index, -swp_to_radix_entry (swap), gfp); -/* -* We already confirmed swap under page lock, and make -* no memory allocation here, so usually no possibility -* of error; but free_swap_and_cache() only trylocks a -* page, so it is just possible that the entry has been -* truncated or holepunched since swap was confirmed. -* shmem_undo_range() will have done some of the -* unaccounting, now delete_from_swap_cache() will do -* the rest. -*/ -if (error) { -mem_cgroup_cancel_charge (page, memcg); -delete_from_swap_cache (page); - } - }if (error) goto failed; + error =shmem_add_to_page_cache (page, mapping, index, +swp_to_radix_entry (swap), gfp); +if (error) { +mem_cgroup_cancel_charge (page, memcg); + goto failed; + } +mem_cgroup_commit_charge (page, memcg, true);spin_lock_irq (&info->lock);Signature · 1 line
2.26.2
Johannes Weiner
maintainerPATCH 6/18 mm: memcontrol: prepare uncharging for removal of private page type counters
In reply to Johannes WeinerThe uncharge batching code adds up the anon, file, kmem counts to determine the total number of pages to uncharge and references to drop. But the next patches will remove the anon and file counters.
Maintain an aggregate nr_pages in the uncharge_gather struct.
--- mm/memcontrol.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
mm/memcontrol.c +12 -11
@@ -6606,6 +6606,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask,struct uncharge_gather {struct mem_cgroup *memcg; +unsigned long nr_pages;unsigned long pgpgout;unsigned long nr_anon;unsigned long nr_file;
@@ -6622,13 +6623,12 @@ static inline void uncharge_gather_clear(struct uncharge_gather *ug)static void uncharge_batch (const struct uncharge_gather *ug) { -unsigned long nr_pages = ug->nr_anon + ug->nr_file + ug->nr_kmem;unsigned long flags;if (!mem_cgroup_is_root (ug->memcg)) { -page_counter_uncharge (&ug->memcg->memory, nr_pages); +page_counter_uncharge (&ug->memcg->memory, ug->nr_pages);if (do_memsw_account ()) -page_counter_uncharge (&ug->memcg->memsw, nr_pages); +page_counter_uncharge (&ug->memcg->memsw, ug->nr_pages);if (!cgroup_subsys_on_dfl (memory_cgrp_subsys) && ug->nr_kmem)page_counter_uncharge (&ug->memcg->kmem, ug->nr_kmem);memcg_oom_recover (ug->memcg);
@@ -6640,16 +6640,18 @@ static void uncharge_batch(const struct uncharge_gather *ug)__mod_memcg_state (ug->memcg,MEMCG_RSS_HUGE , -ug->nr_huge);__mod_memcg_state (ug->memcg,NR_SHMEM , -ug->nr_shmem);__count_memcg_events (ug->memcg,PGPGOUT , ug->pgpgout); -__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, nr_pages); +__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);memcg_check_events (ug->memcg, ug->dummy_page);local_irq_restore (flags);if (!mem_cgroup_is_root (ug->memcg)) -css_put_many (&ug->memcg->css, nr_pages); +css_put_many (&ug->memcg->css, ug->nr_pages); }static void uncharge_page (struct page *page,struct uncharge_gather *ug) { +unsigned long nr_pages; +VM_BUG_ON_PAGE (PageLRU (page), page);VM_BUG_ON_PAGE (page_count (page) && !is_zone_device_page (page) && !PageHWPoison (page) , page);
@@ -6671,13 +6673,12 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->memcg = page->mem_cgroup; } -if (!PageKmemcg (page)) { -unsigned int nr_pages =1 ; + nr_pages =compound_nr (page); + ug->nr_pages += nr_pages; -if (PageTransHuge (page)) { - nr_pages =compound_nr (page); +if (!PageKmemcg (page)) { +if (PageTransHuge (page)) ug->nr_huge += nr_pages; - }if (PageAnon (page)) ug->nr_anon += nr_pages;else {
@@ -6687,7 +6688,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) } ug->pgpgout++; }else { - ug->nr_kmem +=compound_nr (page); + ug->nr_kmem += nr_pages;__ClearPageKmemcg (page); }
Signature · 1 line
2.26.0
Re: [PATCH 06/18] mm: memcontrol: prepare uncharging for removal of private page type counters
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 7 lines
The uncharge batching code adds up the anon, file, kmem counts to determine the total number of pages to uncharge and references to drop. But the next patches will remove the anon and file counters.
Maintain an aggregate
nr_pagesin theuncharge_gatherstruct.Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 06/18] mm: memcontrol: prepare uncharging for removal of private page type counters
In reply to Johannes WeinerQuoted 7 lines · Johannes Weiner
The uncharge batching code adds up the anon, file, kmem counts to determine the total number of pages to uncharge and references to drop. But the next patches will remove the anon and file counters.
Maintain an aggregate
nr_pagesin theuncharge_gatherstruct.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 7/18 mm: memcontrol: prepare move_account for removal of private page type counters
In reply to Johannes WeinerWhen memcg uses the generic vmstat counters, it doesn't need to do anything at charging and uncharging time. It does, however, need to migrate counts when pages move to a different cgroup in move_account.
Prepare the move_account function for the arrival of NR_FILE_PAGES, NR_ANON_MAPPED, NR_ANON_THPS etc. by having a branch for files and a branch for anon, which can then divided into sub-branches.
--- mm/memcontrol.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
mm/memcontrol.c +13 -12
@@ -5374,7 +5374,6 @@ static int mem_cgroup_move_account(struct page *page,struct pglist_data *pgdat;unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ;int ret; -bool anon;VM_BUG_ON (from == to);VM_BUG_ON_PAGE (PageLRU (page), page);
@@ -5392,25 +5391,27 @@ static int mem_cgroup_move_account(struct page *page,if (page->mem_cgroup != from) goto out_unlock; - anon =PageAnon (page); - pgdat =page_pgdat (page); from_vec =mem_cgroup_lruvec (from, pgdat); to_vec =mem_cgroup_lruvec (to, pgdat);lock_page_memcg (page); -if (!anon &&page_mapped (page)) { -__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages); -__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages); - } +if (!PageAnon (page)) { +if (page_mapped (page)) { +__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages); +__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages); + } -if (!anon &&PageDirty (page)) { -struct address_space *mapping =page_mapping (page); +if (PageDirty (page)) { +struct address_space *mapping =page_mapping (page); -if (mapping_cap_account_dirty (mapping)) { -__mod_lruvec_state (from_vec,NR_FILE_DIRTY , -nr_pages); -__mod_lruvec_state (to_vec,NR_FILE_DIRTY , nr_pages); +if (mapping_cap_account_dirty (mapping)) { +__mod_lruvec_state (from_vec,NR_FILE_DIRTY , + -nr_pages); +__mod_lruvec_state (to_vec,NR_FILE_DIRTY , + nr_pages); + } } }
Signature · 1 line
2.26.0
Re: [PATCH 07/18] mm: memcontrol: prepare move_account for removal of private page type counters
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 9 lines
When memcg uses the generic vmstat counters, it doesn't need to do anything at charging and uncharging time. It does, however, need to migrate counts when pages move to a different cgroup in
move_account.Prepare the
move_accountfunction for the arrival ofNR_FILE_PAGES,NR_ANON_MAPPED,NR_ANON_THPSetc. by having a branch for files and a branch for anon, which can then divided into sub-branches.Signed-off-by Johannes Weiner <[email protected]>
Re: [PATCH 07/18] mm: memcontrol: prepare move_account for removal of private page type counters
In reply to Johannes WeinerQuoted 9 lines · Johannes Weiner
When memcg uses the generic vmstat counters, it doesn't need to do anything at charging and uncharging time. It does, however, need to migrate counts when pages move to a different cgroup in
move_account.Prepare the
move_accountfunction for the arrival ofNR_FILE_PAGES,NR_ANON_MAPPED,NR_ANON_THPSetc. by having a branch for files and a branch for anon, which can then divided into sub-branches.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 8/18 mm: memcontrol: prepare cgroup vmstat infrastructure for native anon counters
In reply to Johannes WeinerAnonymous compound pages can be mapped by ptes, which means that if we want to track NR_MAPPED_ANON, NR_ANON_THPS on a per-cgroup basis, we have to be prepared to see tail pages in our accounting functions.
Make mod_lruvec_page_state() and lock_page_memcg() deal with tail pages correctly, namely by redirecting to the head page which has the page->mem_cgroup set up.
--- include/linux/memcontrol.h | 5 +++-- mm/memcontrol.c | 9 ++++++--- 2 files changed, 9 insertions(+), 5 deletions(-)
include/linux/memcontrol.h +3 -2
@@ -711,16 +711,17 @@ static inline void mod_lruvec_state(struct lruvec *lruvec,static inline void __mod_lruvec_page_state (struct page *page,enum node_stat_item idx,int val) { +struct page *head =compound_head (page);/* rmap on tail pages */ pg_data_t *pgdat =page_pgdat (page);struct lruvec *lruvec;/* Untracked pages have no memcg, no lruvec. Update only the node */ -if (!page->mem_cgroup) { +if (!head->mem_cgroup) {__mod_node_page_state (pgdat, idx, val);return ; } - lruvec =mem_cgroup_lruvec (page->mem_cgroup, pgdat); + lruvec =mem_cgroup_lruvec (head->mem_cgroup, pgdat);__mod_lruvec_state (lruvec, idx, val); }
mm/memcontrol.c +6 -3
@@ -1979,6 +1979,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)*/ struct mem_cgroup *lock_page_memcg (struct page *page) { +struct page *head =compound_head (page);/* rmap on tail pages */ struct mem_cgroup *memcg;unsigned long flags;
@@ -1998,7 +1999,7 @@ struct mem_cgroup *lock_page_memcg(struct page *page)if (mem_cgroup_disabled ())return NULL ; again: - memcg = page->mem_cgroup; + memcg = head->mem_cgroup;if (unlikely (!memcg))return NULL ;
@@ -2006,7 +2007,7 @@ struct mem_cgroup *lock_page_memcg(struct page *page)return memcg;spin_lock_irqsave (&memcg->move_lock, flags); -if (memcg != page->mem_cgroup) { +if (memcg != head->mem_cgroup) {spin_unlock_irqrestore (&memcg->move_lock, flags); goto again; }
@@ -2049,7 +2050,9 @@ void __unlock_page_memcg(struct mem_cgroup *memcg)*/ void unlock_page_memcg (struct page *page) { -__unlock_page_memcg (page->mem_cgroup); +struct page *head =compound_head (page); + +__unlock_page_memcg (head->mem_cgroup); }EXPORT_SYMBOL (unlock_page_memcg);
Signature · 1 line
2.26.0
Re: [PATCH 08/18] mm: memcontrol: prepare cgroup vmstat infrastructure for native anon counters
Quoted 9 lines · Johannes Weiner
Anonymous compound pages can be mapped by ptes, which means that if we want to track
NR_MAPPED_ANON,NR_ANON_THPSon a per-cgroup basis, we have to be prepared to see tail pages in our accounting functions.Make
mod_lruvec_page_state()andlock_page_memcg()deal with tail pages correctly, namely by redirecting to the head page which has thepage->mem_cgroupset up.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 9/18 mm: memcontrol: switch to native NR_FILE_PAGES and NR_SHMEM counters
In reply to Johannes WeinerMemcg maintains private MEMCG_CACHE and NR_SHMEM counters. This divergence from the generic VM accounting means unnecessary code overhead, and creates a dependency for memcg that page->mapping is set up at the time of charging, so that page types can be told apart.
Convert the generic accounting sites to mod_lruvec_page_state and friends to maintain the per-cgroup vmstat counters of NR_FILE_PAGES and NR_SHMEM. The page is already locked in these places, so page->mem_cgroup is stable; we only need minimal tweaks of two mem_cgroup_migrate() calls to ensure it's set up in time.
Then replace MEMCG_CACHE with NR_FILE_PAGES and delete the private NR_SHMEM accounting sites.
--- include/linux/memcontrol.h | 3 +-- mm/filemap.c | 17 +++++++++-------- mm/khugepaged.c | 16 +++++++++++----- mm/memcontrol.c | 28 +++++++++++----------------- mm/migrate.c | 15 +++++++++++---- mm/shmem.c | 14 +++++++------- 6 files changed, 50 insertions(+), 43 deletions(-)
include/linux/memcontrol.h +1 -2
@@ -29,8 +29,7 @@ struct kmem_cache;/* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { -MEMCG_CACHE =NR_VM_NODE_STAT_ITEMS , -MEMCG_RSS , +MEMCG_RSS =NR_VM_NODE_STAT_ITEMS ,MEMCG_RSS_HUGE ,MEMCG_SWAP ,MEMCG_SOCK ,
mm/filemap.c +9 -8
@@ -199,9 +199,9 @@ static void unaccount_page_cache_page(struct address_space *mapping, nr =hpage_nr_pages (page); -__mod_node_page_state (page_pgdat (page),NR_FILE_PAGES , -nr); +__mod_lruvec_page_state (page,NR_FILE_PAGES , -nr);if (PageSwapBacked (page)) { -__mod_node_page_state (page_pgdat (page),NR_SHMEM , -nr); +__mod_lruvec_page_state (page,NR_SHMEM , -nr);if (PageTransHuge (page))__dec_node_page_state (page,NR_SHMEM_THPS ); }else if (PageTransHuge (page)) {
@@ -802,21 +802,22 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask) new->mapping = mapping; new->index = offset; +mem_cgroup_migrate (old, new); +xas_lock_irqsave (&xas, flags);xas_store (&xas, new); old->mapping =NULL ;/* hugetlb pages do not participate in page cache accounting. */ if (!PageHuge (old)) -__dec_node_page_state (old,NR_FILE_PAGES ); +__dec_lruvec_page_state (old,NR_FILE_PAGES );if (!PageHuge (new)) -__inc_node_page_state (new,NR_FILE_PAGES ); +__inc_lruvec_page_state (new,NR_FILE_PAGES );if (PageSwapBacked (old)) -__dec_node_page_state (old,NR_SHMEM ); +__dec_lruvec_page_state (old,NR_SHMEM );if (PageSwapBacked (new)) -__inc_node_page_state (new,NR_SHMEM ); +__inc_lruvec_page_state (new,NR_SHMEM );xas_unlock_irqrestore (&xas, flags); -mem_cgroup_migrate (old, new);if (freepage)freepage (old);put_page (old);
@@ -867,7 +868,7 @@ static int __add_to_page_cache_locked(struct page *page,/* hugetlb pages do not participate in page cache accounting */ if (!huge) -__inc_node_page_state (page,NR_FILE_PAGES ); +__inc_lruvec_page_state (page,NR_FILE_PAGES ); unlock:xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp_mask &GFP_RECLAIM_MASK ));
mm/khugepaged.c +11 -5
@@ -1740,12 +1740,18 @@ static void collapse_file(struct mm_struct *mm, }if (nr_none) { -struct zone *zone =page_zone (new_page); - -__mod_node_page_state (zone->zone_pgdat,NR_FILE_PAGES , nr_none); +struct lruvec *lruvec; +/* +* XXX: We have started try_charge and pinned the +* memcg, but the page isn't committed yet so we +* cannot use mod_lruvec_page_state(). This hackery +* will be cleaned up when remove the page->mapping +* dependency from memcg and fully charge above. +*/ + lruvec =mem_cgroup_lruvec (memcg,page_pgdat (new_page)); +__mod_lruvec_state (lruvec,NR_FILE_PAGES , nr_none);if (is_shmem) -__mod_node_page_state (zone->zone_pgdat, -NR_SHMEM , nr_none); +__mod_lruvec_state (lruvec,NR_SHMEM , nr_none); } xa_locked:
mm/memcontrol.c +11 -17
@@ -842,11 +842,6 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg,*/ if (PageAnon (page))__mod_memcg_state (memcg,MEMCG_RSS , nr_pages); -else { -__mod_memcg_state (memcg,MEMCG_CACHE , nr_pages); -if (PageSwapBacked (page)) -__mod_memcg_state (memcg,NR_SHMEM , nr_pages); - }if (abs (nr_pages) >1 ) {VM_BUG_ON_PAGE (!PageTransHuge (page), page);
@@ -1392,7 +1387,7 @@ static char *memory_stat_format(struct mem_cgroup *memcg) (u64 )memcg_page_state (memcg,MEMCG_RSS ) * PAGE_SIZE);seq_buf_printf (&s,"file %llu\n" , - (u64 )memcg_page_state (memcg,MEMCG_CACHE ) * + (u64 )memcg_page_state (memcg,NR_FILE_PAGES ) *PAGE_SIZE ); seq_buf_printf(&s, "kernel_stack %llu\n", (u64)memcg_page_state(memcg, MEMCG_KERNEL_STACK_KB) *
@@ -3302,7 +3297,7 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)unsigned long val;if (mem_cgroup_is_root (memcg)) { - val =memcg_page_state (memcg,MEMCG_CACHE ) + + val =memcg_page_state (memcg,NR_FILE_PAGES ) +memcg_page_state (memcg,MEMCG_RSS );if (swap) val +=memcg_page_state (memcg,MEMCG_SWAP );
@@ -3772,7 +3767,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)#endif /* CONFIG_NUMA */ static const unsigned int memcg1_stats[] = { -MEMCG_CACHE , +NR_FILE_PAGES ,MEMCG_RSS ,MEMCG_RSS_HUGE ,NR_SHMEM ,
@@ -5401,6 +5396,14 @@ static int mem_cgroup_move_account(struct page *page,lock_page_memcg (page);if (!PageAnon (page)) { +__mod_lruvec_state (from_vec,NR_FILE_PAGES , -nr_pages); +__mod_lruvec_state (to_vec,NR_FILE_PAGES , nr_pages); + +if (PageSwapBacked (page)) { +__mod_lruvec_state (from_vec,NR_SHMEM , -nr_pages); +__mod_lruvec_state (to_vec,NR_SHMEM , nr_pages); + } +if (page_mapped (page)) {__mod_lruvec_state (from_vec,NR_FILE_MAPPED , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_MAPPED , nr_pages);
@@ -6613,10 +6616,8 @@ struct uncharge_gather {unsigned long nr_pages;unsigned long pgpgout;unsigned long nr_anon; -unsigned long nr_file;unsigned long nr_kmem;unsigned long nr_huge; -unsigned long nr_shmem;struct page *dummy_page; };
@@ -6640,9 +6641,7 @@ static void uncharge_batch(const struct uncharge_gather *ug)local_irq_save (flags);__mod_memcg_state (ug->memcg,MEMCG_RSS , -ug->nr_anon); -__mod_memcg_state (ug->memcg,MEMCG_CACHE , -ug->nr_file);__mod_memcg_state (ug->memcg,MEMCG_RSS_HUGE , -ug->nr_huge); -__mod_memcg_state (ug->memcg,NR_SHMEM , -ug->nr_shmem);__count_memcg_events (ug->memcg,PGPGOUT , ug->pgpgout);__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);memcg_check_events (ug->memcg, ug->dummy_page);
@@ -6685,11 +6684,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->nr_huge += nr_pages;if (PageAnon (page)) ug->nr_anon += nr_pages; -else { - ug->nr_file += nr_pages; -if (PageSwapBacked (page)) - ug->nr_shmem += nr_pages; - } ug->pgpgout++; }else { ug->nr_kmem += nr_pages;
mm/migrate.c +11 -4
@@ -490,11 +490,18 @@ int migrate_page_move_mapping(struct address_space *mapping,* are mapped to swap space. */ if (newzone != oldzone) { -__dec_node_state (oldzone->zone_pgdat,NR_FILE_PAGES ); -__inc_node_state (newzone->zone_pgdat,NR_FILE_PAGES ); +struct lruvec *old_lruvec, *new_lruvec; +struct mem_cgroup *memcg; + + memcg =page_memcg (page); + old_lruvec =mem_cgroup_lruvec (memcg, oldzone->zone_pgdat); + new_lruvec =mem_cgroup_lruvec (memcg, newzone->zone_pgdat); + +__dec_lruvec_state (old_lruvec,NR_FILE_PAGES ); +__inc_lruvec_state (new_lruvec,NR_FILE_PAGES );if (PageSwapBacked (page) && !PageSwapCache (page)) { -__dec_node_state (oldzone->zone_pgdat,NR_SHMEM ); -__inc_node_state (newzone->zone_pgdat,NR_SHMEM ); +__dec_lruvec_state (old_lruvec,NR_SHMEM ); +__inc_lruvec_state (new_lruvec,NR_SHMEM ); }if (dirty &&mapping_cap_account_dirty (mapping)) {__dec_node_state (oldzone->zone_pgdat,NR_FILE_DIRTY );
mm/shmem.c +7 -7
@@ -653,8 +653,8 @@ static int shmem_add_to_page_cache(struct page *page,__inc_node_page_state (page,NR_SHMEM_THPS ); } mapping->nrpages += nr; -__mod_node_page_state (page_pgdat (page),NR_FILE_PAGES , nr); -__mod_node_page_state (page_pgdat (page),NR_SHMEM , nr); +__mod_lruvec_page_state (page,NR_FILE_PAGES , nr); +__mod_lruvec_page_state (page,NR_SHMEM , nr); unlock:xas_unlock_irq (&xas); }while (xas_nomem (&xas, gfp));
@@ -685,8 +685,8 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap) error =shmem_replace_entry (mapping, page->index, page, radswap); page->mapping =NULL ; mapping->nrpages--; -__dec_node_page_state (page,NR_FILE_PAGES ); -__dec_node_page_state (page,NR_SHMEM ); +__dec_lruvec_page_state (page,NR_FILE_PAGES ); +__dec_lruvec_page_state (page,NR_SHMEM );xa_unlock_irq (&mapping->i_pages);put_page (page);BUG_ON (error);
@@ -1593,8 +1593,9 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,xa_lock_irq (&swap_mapping->i_pages); error =shmem_replace_entry (swap_mapping, swap_index, oldpage, newpage);if (!error) { -__inc_node_page_state (newpage,NR_FILE_PAGES ); -__dec_node_page_state (oldpage,NR_FILE_PAGES ); +mem_cgroup_migrate (oldpage, newpage); +__inc_lruvec_page_state (newpage,NR_FILE_PAGES ); +__dec_lruvec_page_state (oldpage,NR_FILE_PAGES ); }xa_unlock_irq (&swap_mapping->i_pages);
@@ -1606,7 +1607,6 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,*/ oldpage = newpage; }else { -mem_cgroup_migrate (oldpage, newpage);lru_cache_add_anon (newpage); *pagep = newpage; }
Signature · 1 line
2.26.0
Re: [PATCH 09/18] mm: memcontrol: switch to native NR_FILE_PAGES and NR_SHMEM counters
Quoted 11 lines · Johannes Weiner
Memcg maintains private
MEMCG_CACHEandNR_SHMEMcounters. This divergence from the generic VM accounting means unnecessary code overhead, and creates a dependency for memcg thatpage->mappingis set up at the time of charging, so that page types can be told apart.Convert the generic accounting sites to
mod_lruvec_page_stateand friends to maintain the per-cgroup vmstat counters ofNR_FILE_PAGESandNR_SHMEM. The page is already locked in these places, sopage->mem_cgroupis stable; we only need minimal tweaks of twomem_cgroup_migrate()calls to ensure it's set up in time.
Quoted 4 lines
Then replace
MEMCG_CACHEwithNR_FILE_PAGESand delete the privateNR_SHMEMaccounting sites.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 10/18 mm: memcontrol: switch to native NR_ANON_MAPPED counter
In reply to Johannes WeinerMemcg maintains a private MEMCG_RSS counter. This divergence from the generic VM accounting means unnecessary code overhead, and creates a dependency for memcg that page->mapping is set up at the time of charging, so that page types can be told apart.
Convert the generic accounting sites to mod_lruvec_page_state and friends to maintain the per-cgroup vmstat counter of NR_ANON_MAPPED. We use lock_page_memcg() to stabilize page->mem_cgroup during rmap changes, the same way we do for NR_FILE_MAPPED.
With the previous patch removing MEMCG_CACHE and the private NR_SHMEM counter, this patch finally eliminates the need to have page->mapping set up at charge time.
--- include/linux/memcontrol.h | 3 +-- mm/memcontrol.c | 27 ++++++++-------------- mm/rmap.c | 47 +++++++++++++++++++++++--------------- 3 files changed, 39 insertions(+), 38 deletions(-)
include/linux/memcontrol.h +1 -2
@@ -29,8 +29,7 @@ struct kmem_cache;/* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { -MEMCG_RSS =NR_VM_NODE_STAT_ITEMS , -MEMCG_RSS_HUGE , +MEMCG_RSS_HUGE =NR_VM_NODE_STAT_ITEMS ,MEMCG_SWAP ,MEMCG_SOCK ,/* XXX: why are these zone and not node counters? */
mm/memcontrol.c +9 -18
@@ -836,13 +836,6 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, int nr_pages) { -/* -* Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is -* counted as CACHE even if it's on ANON LRU. -*/ -if (PageAnon (page)) -__mod_memcg_state (memcg,MEMCG_RSS , nr_pages); - if (abs(nr_pages) > 1) {VM_BUG_ON_PAGE (!PageTransHuge (page), page);__mod_memcg_state (memcg,MEMCG_RSS_HUGE , nr_pages);
@@ -1384,7 +1377,7 @@ static char *memory_stat_format(struct mem_cgroup *memcg)*/ seq_buf_printf (&s,"anon %llu\n" , - (u64 )memcg_page_state (memcg,MEMCG_RSS ) * + (u64 )memcg_page_state (memcg,NR_ANON_MAPPED ) *PAGE_SIZE ); seq_buf_printf(&s, "file %llu\n", (u64)memcg_page_state(memcg, NR_FILE_PAGES) *
@@ -3298,7 +3291,7 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)if (mem_cgroup_is_root (memcg)) { val =memcg_page_state (memcg,NR_FILE_PAGES ) + -memcg_page_state (memcg,MEMCG_RSS ); +memcg_page_state (memcg,NR_ANON_MAPPED );if (swap) val +=memcg_page_state (memcg,MEMCG_SWAP ); }else {
@@ -3768,7 +3761,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)static const unsigned int memcg1_stats[] = {NR_FILE_PAGES , -MEMCG_RSS , +NR_ANON_MAPPED ,MEMCG_RSS_HUGE ,NR_SHMEM ,NR_FILE_MAPPED ,
@@ -5395,7 +5388,12 @@ static int mem_cgroup_move_account(struct page *page,lock_page_memcg (page); -if (!PageAnon (page)) { +if (PageAnon (page)) { +if (page_mapped (page)) { +__mod_lruvec_state (from_vec,NR_ANON_MAPPED , -nr_pages); +__mod_lruvec_state (to_vec,NR_ANON_MAPPED , nr_pages); + } + }else {__mod_lruvec_state (from_vec,NR_FILE_PAGES , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_PAGES , nr_pages);
@@ -6529,7 +6527,6 @@ void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg, {unsigned int nr_pages =hpage_nr_pages (page); -VM_BUG_ON_PAGE (!page->mapping, page);VM_BUG_ON_PAGE (PageLRU (page) && !lrucare, page);if (mem_cgroup_disabled ())
@@ -6602,8 +6599,6 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask,struct mem_cgroup *memcg;int ret; -VM_BUG_ON_PAGE (!page->mapping, page); - ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg);if (ret)return ret;
@@ -6615,7 +6610,6 @@ struct uncharge_gather {struct mem_cgroup *memcg;unsigned long nr_pages;unsigned long pgpgout; -unsigned long nr_anon;unsigned long nr_kmem;unsigned long nr_huge;struct page *dummy_page;
@@ -6640,7 +6634,6 @@ static void uncharge_batch(const struct uncharge_gather *ug) }local_irq_save (flags); -__mod_memcg_state (ug->memcg,MEMCG_RSS , -ug->nr_anon);__mod_memcg_state (ug->memcg,MEMCG_RSS_HUGE , -ug->nr_huge);__count_memcg_events (ug->memcg,PGPGOUT , ug->pgpgout);__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);
@@ -6682,8 +6675,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)if (!PageKmemcg (page)) {if (PageTransHuge (page)) ug->nr_huge += nr_pages; -if (PageAnon (page)) - ug->nr_anon += nr_pages; ug->pgpgout++; }else { ug->nr_kmem += nr_pages;
mm/rmap.c +29 -18
124 lines not on this page · raw message
Signature · 1 line
2.26.0
Re: [PATCH 10/18] mm: memcontrol: switch to native NR_ANON_MAPPED counter
Quoted 81 lines · Johannes Weiner
Memcg maintains a private
MEMCG_RSScounter. This divergence from the generic VM accounting means unnecessary code overhead, and creates a dependency for memcg thatpage->mappingis set up at the time of charging, so that page types can be told apart.Convert the generic accounting sites to
mod_lruvec_page_stateand friends to maintain the per-cgroup vmstat counter ofNR_ANON_MAPPED. We uselock_page_memcg()to stabilizepage->mem_cgroupduring rmap changes, the same way we do forNR_FILE_MAPPED.With the previous patch removing
MEMCG_CACHEand the privateNR_SHMEMcounter, this patch finally eliminates the need to havepage->mappingset up at charge time.Signed-off-by Johannes Weiner <[email protected]>--- include/linux/memcontrol.h | 3 +-- mm/memcontrol.c | 27 ++++++++-------------- mm/rmap.c | 47 +++++++++++++++++++++++--------------- 3 files changed, 39 insertions(+), 38 deletions(-)include/linux/memcontrol.h +1
-2@@ -29,8 +29,7 @@ struct kmem_cache;/* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { -MEMCG_RSS =NR_VM_NODE_STAT_ITEMS , -MEMCG_RSS_HUGE , +MEMCG_RSS_HUGE =NR_VM_NODE_STAT_ITEMS ,MEMCG_SWAP ,MEMCG_SOCK ,/* XXX: why are these zone and not node counters? */ mm/memcontrol.c +3
-10@@ -836,13 +836,6 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, int nr_pages) { -/* -* Here, RSS means 'mapped anon' and anon's SwapCache. Shmem/tmpfs is -* counted as CACHE even if it's on ANON LRU. -*/ -if (PageAnon (page)) -__mod_memcg_state (memcg,MEMCG_RSS , nr_pages); - if (abs(nr_pages) > 1) {VM_BUG_ON_PAGE (!PageTransHuge (page), page);__mod_memcg_state (memcg,MEMCG_RSS_HUGE , nr_pages);@@ -1384,7 +1377,7 @@ static char *memory_stat_format(struct mem_cgroup *memcg)*/ seq_buf_printf (&s,"anon %llu\n" , - (u64 )memcg_page_state (memcg,MEMCG_RSS ) * + (u64 )memcg_page_state (memcg,NR_ANON_MAPPED ) *PAGE_SIZE ); seq_buf_printf(&s, "file %llu\n", (u64)memcg_page_state(memcg, NR_FILE_PAGES) *@@ -3298,7 +3291,7 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)if (mem_cgroup_is_root (memcg)) { val =memcg_page_state (memcg,NR_FILE_PAGES ) + -memcg_page_state (memcg,MEMCG_RSS ); +memcg_page_state (memcg,NR_ANON_MAPPED );if (swap) val +=memcg_page_state (memcg,MEMCG_SWAP ); }else {@@ -3768,7 +3761,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)static const unsigned int memcg1_stats[] = {NR_FILE_PAGES , -MEMCG_RSS , +NR_ANON_MAPPED ,MEMCG_RSS_HUGE ,NR_SHMEM ,NR_FILE_MAPPED ,
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5392
Quoted 6 lines
mm/memcontrol.c +2
-1@@ -5395,7 +5388,12 @@ static int mem_cgroup_move_account(struct page *page,lock_page_memcg (page); -if (!PageAnon (page)) { +if (PageAnon (page)) { +if (page_mapped (page)) {
This page_mapped() check is newly inserted. Could you elaborate more on why mem_cgroup_charge_statistics() doesn't need this check?
Quoted 84 lines
mm/memcontrol.c +4
-7+__mod_lruvec_state (from_vec,NR_ANON_MAPPED , -nr_pages); +__mod_lruvec_state (to_vec,NR_ANON_MAPPED , nr_pages); + } + }else {__mod_lruvec_state (from_vec,NR_FILE_PAGES , -nr_pages);__mod_lruvec_state (to_vec,NR_FILE_PAGES , nr_pages);@@ -6529,7 +6527,6 @@ void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg, {unsigned int nr_pages =hpage_nr_pages (page); -VM_BUG_ON_PAGE (!page->mapping, page);VM_BUG_ON_PAGE (PageLRU (page) && !lrucare, page);if (mem_cgroup_disabled ())@@ -6602,8 +6599,6 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask,struct mem_cgroup *memcg;int ret; -VM_BUG_ON_PAGE (!page->mapping, page); - ret =mem_cgroup_try_charge (page, mm, gfp_mask, &memcg);if (ret)return ret;@@ -6615,7 +6610,6 @@ struct uncharge_gather {struct mem_cgroup *memcg;unsigned long nr_pages;unsigned long pgpgout; -unsigned long nr_anon;unsigned long nr_kmem;unsigned long nr_huge;struct page *dummy_page;@@ -6640,7 +6634,6 @@ static void uncharge_batch(const struct uncharge_gather *ug) }local_irq_save (flags); -__mod_memcg_state (ug->memcg,MEMCG_RSS , -ug->nr_anon);__mod_memcg_state (ug->memcg,MEMCG_RSS_HUGE , -ug->nr_huge);__count_memcg_events (ug->memcg,PGPGOUT , ug->pgpgout);__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);@@ -6682,8 +6675,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)if (!PageKmemcg (page)) {if (PageTransHuge (page)) ug->nr_huge += nr_pages; -if (PageAnon (page)) - ug->nr_anon += nr_pages; ug->pgpgout++; }else { ug->nr_kmem += nr_pages;mm/rmap.c +10
-4@@ -1114,6 +1114,11 @@ void do_page_add_anon_rmap(struct page *page,bool compound = flags &RMAP_COMPOUND ;bool first; +if (unlikely (PageKsm (page))) +lock_page_memcg (page); +else +VM_BUG_ON_PAGE (!PageLocked (page), page); +if (compound) {atomic_t *mapcount;VM_BUG_ON_PAGE (!PageLocked (page), page);@@ -1134,12 +1139,13 @@ void do_page_add_anon_rmap(struct page *page,*/ if (compound)__inc_node_page_state (page,NR_ANON_THPS ); -__mod_node_page_state (page_pgdat (page),NR_ANON_MAPPED , nr); +__mod_lruvec_page_state (page,NR_ANON_MAPPED , nr); } -if (unlikely (PageKsm (page))) -return ; -VM_BUG_ON_PAGE (!PageLocked (page), page); +if (unlikely (PageKsm (page))) { +unlock_page_memcg (page); +return ; + }/* address might be in next vma when migration races vma_adjust */ if (first)
mm/rmap.c void page_add_new_anon_rmap(struct page *page, · line 1192
Quoted 7 lines
mm/rmap.c +1
-1@@ -1181,7 +1187,7 @@ void page_add_new_anon_rmap(struct page *page,/* increment count (starts at -1) */ atomic_set (&page->_mapcount,0 ); } -__mod_node_page_state (page_pgdat (page),NR_ANON_MAPPED , nr); +__mod_lruvec_page_state (page,NR_ANON_MAPPED , nr);__page_set_anon_rmap (page, vma, address,1 ); }
memcg isn't setup yet and accounting isn't applied to proper memcg. Maybe, it would be applied to root memcg. With this change, we don't need the mapping to commit the charge so switching the order of page_add_new_anon_rmap() and mem_cgroup_commit_charge() will solve the issue.
Thanks.
Johannes Weiner
maintainerRe: [PATCH 10/18] mm: memcontrol: switch to native NR_ANON_MAPPED counter
Hello Joonsoo,
Quoted 12 lines · Joonsoo Kim
Quoted 11 lines · Johannes Weiner
mm/memcontrol.c +1
-1@@ -3768,7 +3761,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)static const unsigned int memcg1_stats[] = {NR_FILE_PAGES , -MEMCG_RSS , +NR_ANON_MAPPED ,MEMCG_RSS_HUGE ,NR_SHMEM ,NR_FILE_MAPPED ,@@ -5395,7 +5388,12 @@ static int mem_cgroup_move_account(struct page *page,
Quoted 8 lines
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5392
Quoted 5 lines
mm/memcontrol.c +2
-1lock_page_memcg (page); -if (!PageAnon (page)) { +if (PageAnon (page)) { +if (page_mapped (page)) {This
page_mapped()check is newly inserted. Could you elaborate more on whymem_cgroup_charge_statistics()doesn't need this check?
MEMCG_RSS extended from when the page was charged until it was uncharged, but NR_ANON_MAPPED is only counted while the page is really mapped into page tables. That starts shortly after we charge and ends shortly before we uncharge, so pages could move between cgroups before or after they are mapped, while they aren't counted in NR_ANON_MAPPED.
So to know that the page is counted, charge_statistics() only needed to know that the page is charged and Anon; move_account() also needs to know that the page is mapped.
Quoted 6 lines
Quoted 6 lines
mm/rmap.c +1
-1@@ -1181,7 +1187,7 @@ void page_add_new_anon_rmap(struct page *page,/* increment count (starts at -1) */ atomic_set (&page->_mapcount,0 ); } -__mod_node_page_state (page_pgdat (page),NR_ANON_MAPPED , nr); +__mod_lruvec_page_state (page,NR_ANON_MAPPED , nr);
Quoted 8 lines
mm/rmap.c void page_add_new_anon_rmap(struct page *page, · line 1192
Quoted 2 lines
mm/rmap.c +0
-0__page_set_anon_rmap (page, vma, address,1 ); }memcg isn't setup yet and accounting isn't applied to proper memcg. Maybe, it would be applied to root memcg. With this change, we don't need the mapping to commit the charge so switching the order of
page_add_new_anon_rmap()andmem_cgroup_commit_charge()will solve the issue.
Good catch, it's that dreaded circular dependency. It's fixed two patches down when I charge anon pages earlier as well. But I'll change the rmap<->commit order in this patch to avoid the temporary bug.
Thanks for your thorough review!
Re: [PATCH 10/18] mm: memcontrol: switch to native NR_ANON_MAPPED counter
Quoted 30 lines · Johannes Weiner
Hello Joonsoo,
Quoted 12 lines · Joonsoo Kim
Quoted 11 lines · Johannes Weiner
mm/memcontrol.c +1
-1@@ -3768,7 +3761,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)static const unsigned int memcg1_stats[] = {NR_FILE_PAGES , -MEMCG_RSS , +NR_ANON_MAPPED ,MEMCG_RSS_HUGE ,NR_SHMEM ,NR_FILE_MAPPED ,@@ -5395,7 +5388,12 @@ static int mem_cgroup_move_account(struct page *page,Quoted 8 lines
mm/memcontrol.c static int mem_cgroup_move_account(struct page *page, · line 5392
Quoted 5 lines
mm/memcontrol.c +2
-1lock_page_memcg (page); -if (!PageAnon (page)) { +if (PageAnon (page)) { +if (page_mapped (page)) {This
page_mapped()check is newly inserted. Could you elaborate more on whymem_cgroup_charge_statistics()doesn't need this check?
MEMCG_RSSextended from when the page was charged until it was uncharged, butNR_ANON_MAPPEDis only counted while the page is really mapped into page tables. That starts shortly after we charge and ends shortly before we uncharge, so pages could move between cgroups before or after they are mapped, while they aren't counted inNR_ANON_MAPPED.
Quoted 3 lines
So to know that the page is counted,
charge_statistics()only needed to know that the page is charged and Anon;move_account()also needs to know that the page is mapped.
Got it!
Quoted 16 lines
Quoted 15 lines
mm/rmap.c void page_add_new_anon_rmap(struct page *page, · line 1192
Quoted 9 lines
mm/rmap.c +1
-1@@ -1181,7 +1187,7 @@ void page_add_new_anon_rmap(struct page *page,/* increment count (starts at -1) */ atomic_set (&page->_mapcount,0 ); } -__mod_node_page_state (page_pgdat (page),NR_ANON_MAPPED , nr); +__mod_lruvec_page_state (page,NR_ANON_MAPPED , nr);__page_set_anon_rmap (page, vma, address,1 ); }memcg isn't setup yet and accounting isn't applied to proper memcg. Maybe, it would be applied to root memcg. With this change, we don't need the mapping to commit the charge so switching the order of
page_add_new_anon_rmap()andmem_cgroup_commit_charge()will solve the issue.
Quoted 3 lines
Good catch, it's that dreaded circular dependency. It's fixed two patches down when I charge anon pages earlier as well. But I'll change the rmap<->commit order in this patch to avoid the temporary bug.
Okay.
Quoted 1 line
Thanks for your thorough review!
Thanks.
Johannes Weiner
maintainerPATCH 11/18 mm: memcontrol: switch to native NR_ANON_THPS counter
In reply to Johannes WeinerWith rmap memcg locking already in place for NR_ANON_MAPPED, it's just a small step to remove the MEMCG_RSS_HUGE wart and switch memcg to the native NR_ANON_THPS accounting sites.
--- include/linux/memcontrol.h | 3 +-- mm/huge_memory.c | 4 +++- mm/memcontrol.c | 39 ++++++++++++++++---------------------- mm/rmap.c | 6 +++--- 4 files changed, 23 insertions(+), 29 deletions(-)
include/linux/memcontrol.h +1 -2
@@ -29,8 +29,7 @@ struct kmem_cache;/* Cgroup-specific page state, on top of universal node page state */ enum memcg_stat_item { -MEMCG_RSS_HUGE =NR_VM_NODE_STAT_ITEMS , -MEMCG_SWAP , +MEMCG_SWAP =NR_VM_NODE_STAT_ITEMS ,MEMCG_SOCK ,/* XXX: why are these zone and not node counters? */ MEMCG_KERNEL_STACK_KB ,
mm/huge_memory.c +3 -1
@@ -2359,15 +2359,17 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,atomic_inc (&page[i]._mapcount); } +lock_page_memcg (page);if (atomic_add_negative (-1 ,compound_mapcount_ptr (page))) {/* Last compound_mapcount is gone. */ -__dec_node_page_state (page,NR_ANON_THPS ); +__dec_lruvec_page_state (page,NR_ANON_THPS );if (TestClearPageDoubleMap (page)) {/* No need in mapcount reference anymore */ for (i =0 ; i <HPAGE_PMD_NR ; i++)atomic_dec (&page[i]._mapcount); } } +unlock_page_memcg (page);smp_wmb ();/* make pte visible before pmd */ pmd_populate (mm, pmd, pgtable);
mm/memcontrol.c +16 -23
@@ -836,11 +836,6 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, struct page *page, int nr_pages) { - if (abs(nr_pages) > 1) { -VM_BUG_ON_PAGE (!PageTransHuge (page), page); -__mod_memcg_state (memcg,MEMCG_RSS_HUGE , nr_pages); - } -/* pagein of a big page is an event. So, ignore page size */ if (nr_pages >0 )__count_memcg_events (memcg,PGPGIN ,1 );
@@ -1406,15 +1401,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg) (u64 )memcg_page_state (memcg,NR_WRITEBACK ) * PAGE_SIZE); -/* -* TODO: We should eventually replace our own MEMCG_RSS_HUGE counter -* with the NR_ANON_THP vm counter, but right now it's a pain in the -* arse because it requires migrating the work out of rmap to a place -* where the page->mem_cgroup is set up and stable. -*/ seq_buf_printf (&s,"anon_thp %llu\n" , - (u64 )memcg_page_state (memcg,MEMCG_RSS_HUGE ) * -PAGE_SIZE ); + (u64 )memcg_page_state (memcg,NR_ANON_THPS ) * +HPAGE_PMD_NR *PAGE_SIZE );for (i =0 ; i <NR_LRU_LISTS ; i++) seq_buf_printf(&s, "%s %llu\n", lru_list_name(i),
@@ -3006,8 +2995,6 @@ void mem_cgroup_split_huge_fixup(struct page *head)for (i =1 ; i <HPAGE_PMD_NR ; i++) head[i].mem_cgroup = head->mem_cgroup; - - __mod_memcg_state(head->mem_cgroup, MEMCG_RSS_HUGE, -HPAGE_PMD_NR); }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
@@ -3762,7 +3749,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)static const unsigned int memcg1_stats[] = {NR_FILE_PAGES ,NR_ANON_MAPPED , -MEMCG_RSS_HUGE , +NR_ANON_THPS ,NR_SHMEM ,NR_FILE_MAPPED ,NR_FILE_DIRTY ,
@@ -3799,11 +3786,14 @@ static int memcg_stat_show(struct seq_file *m, void *v)BUILD_BUG_ON (ARRAY_SIZE (memcg1_stat_names) !=ARRAY_SIZE (memcg1_stats));for (i =0 ; i <ARRAY_SIZE (memcg1_stats); i++) { +unsigned long nr; +if (memcg1_stats[i] ==MEMCG_SWAP && !do_memsw_account ())continue ; -seq_printf (m,"%s %lu\n" , memcg1_stat_names[i], -memcg_page_state_local (memcg, memcg1_stats[i]) * -PAGE_SIZE ); + nr =memcg_page_state_local (memcg, memcg1_stats[i]); +if (memcg1_stats[i] ==NR_ANON_THPS ) + nr *=HPAGE_PMD_NR ; +seq_printf (m,"%s %lu\n" , memcg1_stat_names[i], nr *PAGE_SIZE ); }for (i =0 ; i <ARRAY_SIZE (memcg1_events); i++)
@@ -5392,6 +5382,13 @@ static int mem_cgroup_move_account(struct page *page,if (page_mapped (page)) {__mod_lruvec_state (from_vec,NR_ANON_MAPPED , -nr_pages);__mod_lruvec_state (to_vec,NR_ANON_MAPPED , nr_pages); +if (PageTransHuge (page)) { +__mod_lruvec_state (from_vec,NR_ANON_THPS , + -nr_pages); +__mod_lruvec_state (to_vec,NR_ANON_THPS , + nr_pages); + } + } }else {__mod_lruvec_state (from_vec,NR_FILE_PAGES , -nr_pages);
@@ -6611,7 +6608,6 @@ struct uncharge_gather {unsigned long nr_pages;unsigned long pgpgout;unsigned long nr_kmem; -unsigned long nr_huge;struct page *dummy_page; };
@@ -6634,7 +6630,6 @@ static void uncharge_batch(const struct uncharge_gather *ug) }local_irq_save (flags); -__mod_memcg_state (ug->memcg,MEMCG_RSS_HUGE , -ug->nr_huge);__count_memcg_events (ug->memcg,PGPGOUT , ug->pgpgout);__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);memcg_check_events (ug->memcg, ug->dummy_page);
@@ -6673,8 +6668,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->nr_pages += nr_pages;if (!PageKmemcg (page)) { -if (PageTransHuge (page)) - ug->nr_huge += nr_pages; ug->pgpgout++; }else { ug->nr_kmem += nr_pages;
mm/rmap.c +3 -3
@@ -1138,7 +1138,7 @@ void do_page_add_anon_rmap(struct page *page,* disabled. */ if (compound) -__inc_node_page_state (page,NR_ANON_THPS ); +__inc_lruvec_page_state (page,NR_ANON_THPS );__mod_lruvec_page_state (page,NR_ANON_MAPPED , nr); }
@@ -1180,7 +1180,7 @@ void page_add_new_anon_rmap(struct page *page,if (hpage_pincount_available (page))atomic_set (compound_pincount_ptr (page),0 ); -__inc_node_page_state (page,NR_ANON_THPS ); +__inc_lruvec_page_state (page,NR_ANON_THPS ); }else {/* Anon THP always mapped first with PMD */ VM_BUG_ON_PAGE (PageTransCompound (page), page);
@@ -1286,7 +1286,7 @@ static void page_remove_anon_compound_rmap(struct page *page)if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ))return ; -__dec_node_page_state (page,NR_ANON_THPS ); +__dec_lruvec_page_state (page,NR_ANON_THPS );if (TestClearPageDoubleMap (page)) {/*
Signature · 1 line
2.26.0
Re: [PATCH 11/18] mm: memcontrol: switch to native NR_ANON_THPS counter
Quoted 5 lines · Johannes Weiner
With rmap memcg locking already in place for
NR_ANON_MAPPED, it's just a small step to remove theMEMCG_RSS_HUGEwart and switch memcg to the nativeNR_ANON_THPSaccounting sites.Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 12/18 mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API
In reply to Johannes WeinerWith the page->mapping requirement gone from memcg, we can charge anon and file-thp pages in one single step, right after they're allocated.
This removes two out of three API calls - especially the tricky commit step that needed to happen at just the right time between when the page is "set up" and when it's "published" - somewhat vague and fluid concepts that varied by page type. All we need is a freshly allocated page and a memcg context to charge.
--- include/linux/mm.h | 4 +--- kernel/events/uprobes.c | 11 +++-------- mm/filemap.c | 2 +- mm/huge_memory.c | 41 ++++++++++++----------------------------- mm/khugepaged.c | 31 ++++++------------------------- mm/memory.c | 36 ++++++++++-------------------------- mm/migrate.c | 5 +---- mm/swapfile.c | 6 +----- mm/userfaultfd.c | 5 +---- 9 files changed, 36 insertions(+), 105 deletions(-)
include/linux/mm.h +1 -3
@@ -505,7 +505,6 @@ struct vm_fault {pte_t orig_pte;/* Value of PTE at the time of fault */ struct page *cow_page;/* Page handler may use for COW fault */ -struct mem_cgroup *memcg;/* Cgroup cow_page belongs to */ struct page *page;/* ->fault handlers should return a * page here, unless VM_FAULT_NOPAGE * is set (which is also implied by
@@ -939,8 +938,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) return pte; } -vm_fault_t alloc_set_pte (struct vm_fault *vmf,struct mem_cgroup *memcg, -struct page *page); +vm_fault_t alloc_set_pte (struct vm_fault *vmf,struct page *page);vm_fault_t finish_fault (struct vm_fault *vmf);vm_fault_t finish_mkwrite_fault (struct vm_fault *vmf);#endif
kernel/events/uprobes.c +3 -8
@@ -162,14 +162,13 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, };int err;struct mmu_notifier_range range; -struct mem_cgroup *memcg;mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, mm, addr, addr +PAGE_SIZE );if (new_page) { - err =mem_cgroup_try_charge (new_page, vma->vm_mm,GFP_KERNEL , - &memcg); + err =mem_cgroup_charge (new_page, vma->vm_mm,GFP_KERNEL , + false);if (err)return err; }
@@ -179,17 +178,13 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,mmu_notifier_invalidate_range_start (&range); err = -EAGAIN ; -if (!page_vma_mapped_walk (&pvmw)) { -if (new_page) -mem_cgroup_cancel_charge (new_page, memcg); +if (!page_vma_mapped_walk (&pvmw)) goto unlock; - }VM_BUG_ON_PAGE (addr != pvmw.address, old_page);if (new_page) {get_page (new_page);page_add_new_anon_rmap (new_page, vma, addr, false); -mem_cgroup_commit_charge (new_page, memcg, false);lru_cache_add_active_or_unevictable (new_page, vma); }else /* no new page, just dec_mm_counter for old_page */
mm/filemap.c +1 -1
@@ -2633,7 +2633,7 @@ void filemap_map_pages(struct vm_fault *vmf,if (vmf->pte) vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; -if (alloc_set_pte (vmf,NULL , page)) +if (alloc_set_pte (vmf, page)) goto unlock;unlock_page (page); goto next;
mm/huge_memory.c +12 -29
147 lines not on this page · raw message
mm/khugepaged.c +6 -25
@@ -951,7 +951,6 @@ static void collapse_huge_page(struct mm_struct *mm,struct page *new_page;spinlock_t *pmd_ptl, *pte_ptl;int isolated =0 , result =0 ; -struct mem_cgroup *memcg;struct vm_area_struct *vma;struct mmu_notifier_range range;gfp_t gfp;
@@ -974,15 +973,15 @@ static void collapse_huge_page(struct mm_struct *mm, goto out_nolock; } -if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg))) { +if (unlikely (mem_cgroup_charge (new_page, mm, gfp, false))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out_nolock; } +count_memcg_page_event (new_page,THP_COLLAPSE_ALLOC );down_read (&mm->mmap_sem); result =hugepage_vma_revalidate (mm, address, &vma);if (result) { -mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -990,7 +989,6 @@ static void collapse_huge_page(struct mm_struct *mm, pmd =mm_find_pmd (mm, address);if (!pmd) { result =SCAN_PMD_NULL ; -mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -1001,7 +999,6 @@ static void collapse_huge_page(struct mm_struct *mm,* Continuing to collapse causes inconsistency. */ if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { -mem_cgroup_cancel_charge (new_page, memcg);up_read (&mm->mmap_sem); goto out_nolock; }
@@ -1087,8 +1084,6 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd));page_add_new_anon_rmap (new_page, vma, address, true); -mem_cgroup_commit_charge (new_page, memcg, false); -count_memcg_events (memcg,THP_COLLAPSE_ALLOC ,1 );lru_cache_add_active_or_unevictable (new_page, vma);pgtable_trans_huge_deposit (mm, pmd, pgtable);set_pmd_at (mm, address, pmd, _pmd);
@@ -1105,7 +1100,6 @@ static void collapse_huge_page(struct mm_struct *mm,trace_mm_collapse_huge_page (mm, isolated, result);return ; out: -mem_cgroup_cancel_charge (new_page, memcg); goto out_up_write; }
@@ -1515,7 +1509,6 @@ static void collapse_file(struct mm_struct *mm,struct address_space *mapping = file->f_mapping;gfp_t gfp;struct page *new_page; -struct mem_cgroup *memcg;pgoff_t index, end = start +HPAGE_PMD_NR ;LIST_HEAD (pagelist);XA_STATE_ORDER (xas, &mapping->i_pages, start,HPAGE_PMD_ORDER );
@@ -1534,10 +1527,11 @@ static void collapse_file(struct mm_struct *mm, goto out; } -if (unlikely (mem_cgroup_try_charge (new_page, mm, gfp, &memcg))) { +if (unlikely (mem_cgroup_charge (new_page, mm, gfp, false))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out; } +count_memcg_page_event (new_page,THP_COLLAPSE_ALLOC );/* This will be less messy when we use multi-index entries */ do {
@@ -1547,7 +1541,6 @@ static void collapse_file(struct mm_struct *mm,break ;xas_unlock_irq (&xas);if (!xas_nomem (&xas,GFP_KERNEL )) { -mem_cgroup_cancel_charge (new_page, memcg); result =SCAN_FAIL ; goto out; }
@@ -1740,18 +1733,9 @@ static void collapse_file(struct mm_struct *mm, }if (nr_none) { -struct lruvec *lruvec; -/* -* XXX: We have started try_charge and pinned the -* memcg, but the page isn't committed yet so we -* cannot use mod_lruvec_page_state(). This hackery -* will be cleaned up when remove the page->mapping -* dependency from memcg and fully charge above. -*/ - lruvec =mem_cgroup_lruvec (memcg,page_pgdat (new_page)); -__mod_lruvec_state (lruvec,NR_FILE_PAGES , nr_none); +__mod_lruvec_page_state (new_page,NR_FILE_PAGES , nr_none);if (is_shmem) -__mod_lruvec_state (lruvec,NR_SHMEM , nr_none); +__mod_lruvec_page_state (new_page,NR_SHMEM , nr_none); } xa_locked:
@@ -1789,7 +1773,6 @@ static void collapse_file(struct mm_struct *mm,SetPageUptodate (new_page);page_ref_add (new_page,HPAGE_PMD_NR -1 ); -mem_cgroup_commit_charge (new_page, memcg, false);if (is_shmem) {set_page_dirty (new_page);
@@ -1797,7 +1780,6 @@ static void collapse_file(struct mm_struct *mm, }else {lru_cache_add_file (new_page); } -count_memcg_events (memcg,THP_COLLAPSE_ALLOC ,1 );/* * Remove pte page tables, so we can re-fault the page as huge.
@@ -1844,7 +1826,6 @@ static void collapse_file(struct mm_struct *mm,VM_BUG_ON (nr_none);xas_unlock_irq (&xas); -mem_cgroup_cancel_charge (new_page, memcg); new_page->mapping =NULL ; }
mm/memory.c +10 -26
@@ -2647,7 +2647,6 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)struct page *new_page =NULL ;pte_t entry;int page_copied =0 ; -struct mem_cgroup *memcg;struct mmu_notifier_range range;if (unlikely (anon_vma_prepare (vma)))
@@ -2678,8 +2677,9 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } } -if (mem_cgroup_try_charge_delay (new_page, mm,GFP_KERNEL , &memcg)) +if (mem_cgroup_charge (new_page, mm,GFP_KERNEL , false)) goto oom_free_new; +cgroup_throttle_swaprate (new_page,GFP_KERNEL );__SetPageUptodate (new_page);
@@ -2713,7 +2713,6 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)*/ ptep_clear_flush_notify (vma, vmf->address, vmf->pte);page_add_new_anon_rmap (new_page, vma, vmf->address, false); -mem_cgroup_commit_charge (new_page, memcg, false);lru_cache_add_active_or_unevictable (new_page, vma);/* * We call the notify macro here because, when using secondary
@@ -2751,8 +2750,6 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)/* Free the old page.. */ new_page = old_page; page_copied =1 ; - }else { -mem_cgroup_cancel_charge (new_page, memcg); }if (new_page)
@@ -3090,7 +3087,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;struct page *page =NULL , *swapcache; -struct mem_cgroup *memcg;swp_entry_t entry;pte_t pte;int locked;
@@ -3195,10 +3191,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_page; } -if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , &memcg)) { +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , true)) { ret =VM_FAULT_OOM ; goto out_page; } +cgroup_throttle_swaprate (page,GFP_KERNEL );/* * Back out if somebody else already faulted in this pte.
@@ -3246,11 +3243,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* ksm created a completely new copy */ if (unlikely (page != swapcache && swapcache)) {page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }else {do_page_add_anon_rmap (page, vma, vmf->address, exclusive); -mem_cgroup_commit_charge (page, memcg, true);activate_page (page); }
@@ -3286,7 +3281,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out:return ret; out_nomap: -mem_cgroup_cancel_charge (page, memcg);pte_unmap_unlock (vmf->pte, vmf->ptl); out_page:unlock_page (page);
@@ -3307,7 +3301,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)static vm_fault_t do_anonymous_page (struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma; -struct mem_cgroup *memcg;struct page *page;vm_fault_t ret =0 ;pte_t entry;
@@ -3360,8 +3353,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (!page) goto oom; -if (mem_cgroup_try_charge_delay (page, vma->vm_mm,GFP_KERNEL , &memcg)) +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , false)) goto oom_free_page; +cgroup_throttle_swaprate (page,GFP_KERNEL );/* * The memory barrier inside __SetPageUptodate makes sure that
@@ -3386,14 +3380,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing (vma)) {pte_unmap_unlock (vmf->pte, vmf->ptl); -mem_cgroup_cancel_charge (page, memcg);put_page (page);return handle_userfault (vmf,VM_UFFD_MISSING ); }inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); setpte:set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry);
@@ -3404,7 +3396,6 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return ret; release: -mem_cgroup_cancel_charge (page, memcg);put_page (page); goto unlock; oom_free_page:
@@ -3609,7 +3600,6 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)* mapping. If needed, the fucntion allocates page table or use pre-allocated. * * @vmf: fault environment -* @memcg: memcg to charge page (only for private mappings) * @page: page to map * * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
@@ -3620,8 +3610,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)* * Return: %0 on success, %VM_FAULT_ code in case of error. */ -vm_fault_t alloc_set_pte (struct vm_fault *vmf,struct mem_cgroup *memcg, -struct page *page) +vm_fault_t alloc_set_pte (struct vm_fault *vmf,struct page *page) {struct vm_area_struct *vma = vmf->vma;bool write = vmf->flags &FAULT_FLAG_WRITE ;
@@ -3629,9 +3618,6 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,vm_fault_t ret;if (pmd_none (*vmf->pmd) &&PageTransCompound (page)) { -/* THP on COW? */ -VM_BUG_ON_PAGE (memcg, page); - ret =do_set_pmd (vmf, page);if (ret !=VM_FAULT_FALLBACK )return ret;
@@ -3655,7 +3641,6 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, vmf->address, false); -mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }else {inc_mm_counter_fast (vma->vm_mm,mm_counter_file (page));
@@ -3704,7 +3689,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf)if (!(vmf->vma->vm_flags &VM_SHARED )) ret =check_stable_address_space (vmf->vma->vm_mm);if (!ret) - ret =alloc_set_pte (vmf, vmf->memcg, page); + ret =alloc_set_pte (vmf, page);if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;
@@ -3864,11 +3849,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf)if (!vmf->cow_page)return VM_FAULT_OOM ; -if (mem_cgroup_try_charge_delay (vmf->cow_page, vma->vm_mm, -GFP_KERNEL , &vmf->memcg)) { +if (mem_cgroup_charge (vmf->cow_page, vma->vm_mm,GFP_KERNEL , false)) {put_page (vmf->cow_page);return VM_FAULT_OOM ; } +cgroup_throttle_swaprate (vmf->cow_page,GFP_KERNEL ); ret =__do_fault (vmf);if (unlikely (ret & (VM_FAULT_ERROR |VM_FAULT_NOPAGE |VM_FAULT_RETRY )))
@@ -3886,7 +3871,6 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) goto uncharge_out;return ret; uncharge_out: -mem_cgroup_cancel_charge (vmf->cow_page, vmf->memcg);put_page (vmf->cow_page);return ret; }
mm/migrate.c +1 -4
@@ -2746,7 +2746,6 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, {struct vm_area_struct *vma = migrate->vma;struct mm_struct *mm = vma->vm_mm; -struct mem_cgroup *memcg;bool flush = false;spinlock_t *ptl;pte_t entry;
@@ -2793,7 +2792,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,if (unlikely (anon_vma_prepare (vma))) goto abort; -if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , &memcg)) +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , false)) goto abort;/*
@@ -2839,7 +2838,6 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,inc_mm_counter (mm,MM_ANONPAGES );page_add_new_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, false);if (!is_zone_device_page (page))lru_cache_add_active_or_unevictable (page, vma);get_page (page);
@@ -2861,7 +2859,6 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, unlock_abort:pte_unmap_unlock (ptep, ptl); -mem_cgroup_cancel_charge (page, memcg); abort: *src &= ~MIGRATE_PFN_MIGRATE ; }
mm/swapfile.c +1 -5
@@ -1854,7 +1854,6 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, swp_entry_t entry, struct page *page) { struct page *swapcache; -struct mem_cgroup *memcg;spinlock_t *ptl;pte_t *pte;int ret =1 ;
@@ -1864,14 +1863,13 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,if (unlikely (!page))return -ENOMEM ; -if (mem_cgroup_try_charge (page, vma->vm_mm,GFP_KERNEL , &memcg)) { +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , true)) { ret = -ENOMEM ; goto out_nolock; } pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);if (unlikely (!pte_same_as_swp (*pte,swp_entry_to_pte (entry)))) { -mem_cgroup_cancel_charge (page, memcg); ret =0 ; goto out; }
@@ -1883,10 +1881,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, pte_mkold(mk_pte(page, vma->vm_page_prot)));if (page == swapcache) {page_add_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, true); }else {/* ksm created a completely new copy */ page_add_new_anon_rmap (page, vma, addr, false); -mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, vma); }swap_free (entry);
mm/userfaultfd.c +1 -4
@@ -56,7 +56,6 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, struct page **pagep, bool wp_copy) { -struct mem_cgroup *memcg;pte_t _dst_pte, *dst_pte;spinlock_t *ptl;void *page_kaddr;
@@ -97,7 +96,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,__SetPageUptodate (page); ret = -ENOMEM ; -if (mem_cgroup_try_charge (page, dst_mm,GFP_KERNEL , &memcg)) +if (mem_cgroup_charge (page, dst_mm,GFP_KERNEL , false)) goto out_release; _dst_pte =pte_mkdirty (mk_pte (page, dst_vma->vm_page_prot));
@@ -124,7 +123,6 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,inc_mm_counter (dst_mm,MM_ANONPAGES );page_add_new_anon_rmap (page, dst_vma, dst_addr, false); -mem_cgroup_commit_charge (page, memcg, false);lru_cache_add_active_or_unevictable (page, dst_vma);set_pte_at (dst_mm, dst_addr, dst_pte, _dst_pte);
@@ -138,7 +136,6 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,return ret; out_release_uncharge_unlock:pte_unmap_unlock (dst_pte, ptl); -mem_cgroup_cancel_charge (page, memcg); out_release:put_page (page); goto out;
Signature · 1 line
2.26.0
Re: [PATCH 12/18] mm: memcontrol: convert anon and file-thp to new mem_cgroup_charge() API
Quoted 10 lines · Johannes Weiner
With the
page->mappingrequirement gone from memcg, we can charge anon and file-thp pages in one single step, right after they're allocated.This removes two out of three API calls - especially the tricky commit step that needed to happen at just the right time between when the page is "set up" and when it's "published" - somewhat vague and fluid concepts that varied by page type. All we need is a freshly allocated page and a memcg context to charge.
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 13/18 mm: memcontrol: drop unused try/commit/cancel charge API
In reply to Johannes WeinerThere are no more users. RIP in peace.
--- include/linux/memcontrol.h | 36 ----------- mm/memcontrol.c | 126 +++++-------------------------------- 2 files changed, 15 insertions(+), 147 deletions(-)
include/linux/memcontrol.h +0 -36
@@ -357,14 +357,6 @@ static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg,enum mem_cgroup_protection mem_cgroup_protected (struct mem_cgroup *root,struct mem_cgroup *memcg); -int mem_cgroup_try_charge (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp); -int mem_cgroup_try_charge_delay (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,struct mem_cgroup **memcgp); -void mem_cgroup_commit_charge (struct page *page,struct mem_cgroup *memcg, -bool lrucare); -void mem_cgroup_cancel_charge (struct page *page,struct mem_cgroup *memcg); -int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask,bool lrucare);
@@ -846,34 +838,6 @@ static inline enum mem_cgroup_protection mem_cgroup_protected(return MEMCG_PROT_NONE ; } -static inline int mem_cgroup_try_charge (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask, -struct mem_cgroup **memcgp) -{ - *memcgp =NULL ; -return 0 ; -} - -static inline int mem_cgroup_try_charge_delay (struct page *page, -struct mm_struct *mm, -gfp_t gfp_mask, -struct mem_cgroup **memcgp) -{ - *memcgp =NULL ; -return 0 ; -} - -static inline void mem_cgroup_commit_charge (struct page *page, -struct mem_cgroup *memcg, -bool lrucare) -{ -} - -static inline void mem_cgroup_cancel_charge (struct page *page, -struct mem_cgroup *memcg) -{ -} -static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask,bool lrucare) {
mm/memcontrol.c +15 -111
170 lines not on this page · raw message
Signature · 1 line
2.26.0
Re: [PATCH 13/18] mm: memcontrol: drop unused try/commit/cancel charge API
Quoted 3 lines · Johannes Weiner
There are no more users. RIP in peace.
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 14/18 mm: memcontrol: prepare swap controller setup for integration
In reply to Johannes WeinerA few cleanups to streamline the swap controller setup:
- Replace the do_swap_account flag with cgroup_memory_noswap. This brings it in line with other functionality that is usually available unless explicitly opted out of - nosocket, nokmem.
- Remove the really_do_swap_account flag that stores the boot option and is later used to switch the do_swap_account. It's not clear why this indirection is/was necessary. Use do_swap_account directly.
- Minor coding style polishing
--- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 59 ++++++++++++++++++-------------------- mm/swap_cgroup.c | 4 +-- 3 files changed, 31 insertions(+), 34 deletions(-)
include/linux/memcontrol.h +1 -1
@@ -560,7 +560,7 @@ struct mem_cgroup *mem_cgroup_get_oom_group(struct task_struct *victim,void mem_cgroup_print_oom_group (struct mem_cgroup *memcg);#ifdef CONFIG_MEMCG_SWAP -extern int do_swap_account; +extern bool cgroup_memory_noswap;#endif struct mem_cgroup *lock_page_memcg (struct page *page);
mm/memcontrol.c +28 -31
133 lines not on this page · raw message
mm/swap_cgroup.c +2 -2
@@ -171,7 +171,7 @@ int swap_cgroup_swapon(int type, unsigned long max_pages)unsigned long length;struct swap_cgroup_ctrl *ctrl; -if (!do_swap_account) +if (cgroup_memory_noswap)return 0 ; length =DIV_ROUND_UP (max_pages,SC_PER_PAGE );
@@ -209,7 +209,7 @@ void swap_cgroup_swapoff(int type)unsigned long i, length;struct swap_cgroup_ctrl *ctrl; -if (!do_swap_account) +if (cgroup_memory_noswap)return ;mutex_lock (&swap_cgroup_mutex);
Signature · 1 line
2.26.0
Re: [PATCH 14/18] mm: memcontrol: prepare swap controller setup for integration
Quoted 6 lines · Johannes Weiner
A few cleanups to streamline the swap controller setup:
- Replace the
do_swap_accountflag withcgroup_memory_noswap. This brings it in line with other functionality that is usually available unless explicitly opted out of - nosocket, nokmem.
Quoted 7 lines
- Remove the
really_do_swap_accountflag that stores the boot option and is later used to switch thedo_swap_account. It's not clear why this indirection is/was necessary. Usedo_swap_accountdirectly.- Minor coding style polishing
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerPATCH 15/18 mm: memcontrol: make swap tracking an integral part of memory control
In reply to Johannes WeinerWithout swap page tracking, users that are otherwise memory controlled can easily escape their containment and allocate significant amounts of memory that they're not being charged for. That's because swap does readahead, but without the cgroup records of who owned the page at swapout, readahead pages don't get charged until somebody actually faults them into their page table and we can identify an owner task. This can be maliciously exploited with MADV_WILLNEED, which triggers arbitrary readahead allocations without charging the pages.
Make swap swap page tracking an integral part of memcg and remove the Kconfig options. In the first place, it was only made configurable to allow users to save some memory. But the overhead of tracking cgroup ownership per swap page is minimal - 2 byte per page, or 512k per 1G of swap, or 0.04%. Saving that at the expense of broken containment semantics is not something we should present as a coequal option.
The swapaccount=0 boot option will continue to exist, and it will eliminate the page_counter overhead and hide the swap control files, but it won't disable swap slot ownership tracking.
This patch makes sure we always have the cgroup records at swapin time; the next patch will fix the actual bug by charging readahead swap pages at swapin time rather than at fault time.
--- init/Kconfig | 17 +---------------- mm/memcontrol.c | 48 +++++++++++++++++------------------------------- mm/swap_cgroup.c | 6 ------ 3 files changed, 18 insertions(+), 53 deletions(-)
init/Kconfig +1 -16
@@ -835,24 +835,9 @@ config MEMCG Provides control over the memory footprint of tasks in a cgroup. config MEMCG_SWAP - bool "Swap controller" + bool depends on MEMCG && SWAP - help - Provides control over the swap space consumed by tasks in a cgroup. - -config MEMCG_SWAP_ENABLED - bool "Swap controller enabled by default" - depends on MEMCG_SWAP default y - help - Memory Resource Controller Swap Extension comes with its price in - a bigger memory consumption. General purpose distribution kernels - which want to enable the feature but keep it disabled by default - and let the user enable it by swapaccount=1 boot command line - parameter should have this option unselected. - For those who want to have the feature enabled by default should - select this option (if, for some reason, they need to disable it - then swapaccount=0 does the trick). config MEMCG_KMEM bool
mm/memcontrol.c +17 -31
121 lines not on this page · raw message
mm/swap_cgroup.c +0 -6
@@ -171,9 +171,6 @@ int swap_cgroup_swapon(int type, unsigned long max_pages)unsigned long length;struct swap_cgroup_ctrl *ctrl; -if (cgroup_memory_noswap) -return 0 ; - length =DIV_ROUND_UP (max_pages,SC_PER_PAGE ); array_size = length *sizeof (void *);
@@ -209,9 +206,6 @@ void swap_cgroup_swapoff(int type)unsigned long i, length;struct swap_cgroup_ctrl *ctrl; -if (cgroup_memory_noswap) -return ; -mutex_lock (&swap_cgroup_mutex); ctrl = &swap_cgroup_ctrl[type]; map = ctrl->map;
Signature · 1 line
2.26.0
Re: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 3 lines
The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.
May we add extra explanation for this change to user? and the default memsw limitations?
Quoted 6 lines
This patch makes sure we always have the cgroup records at swapin time; the next patch will fix the actual bug by charging readahead swap pages at swapin time rather than at fault time.
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerRe: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
Hi Alex,
thanks for your quick review so far, I'll add the tags to the patches.
Quoted 9 lines · Alex Shi
Quoted 3 lines · Johannes Weiner
The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.May we add extra explanation for this change to user? and the default memsw limitations?
Can you elaborate what you think is missing and where you would like to see it documented?
From a semantics POV, nothing changes with this patch. The memsw limit defaults to "max", so it doesn't exert any control per default. The only difference is whether we maintain swap records or not.
Re: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
Quoted 12 lines · Johannes Weiner
Hi Alex,
thanks for your quick review so far, I'll add the tags to the patches.
Quoted 7 lines · Alex Shi
Quoted 3 lines · Johannes Weiner
The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.
Quoted 6 lines
Quoted 2 lines
May we add extra explanation for this change to user? and the default memsw limitations?
Can you elaborate what you think is missing and where you would like to see it documented?
Maybe the following doc change is better after whole patchset? Guess users would would happy to know details of this change.
Also as to the RSS account name change, I don't know if it's good to polish them in docs.
Thanks
Alex
Documentation/admin-guide/cgroup-v1/memory.rst +6 -6
@@ -199,11 +199,11 @@ An RSS page is unaccounted when it's fully unmapped. A PageCache page is unaccounted when it's removed from radix-tree. Even if RSS pages are fully unmapped (by kswapd), they may exist as SwapCache in the system until they are really freed. Such SwapCaches are also accounted. -A swapped-in page is not accounted until it's mapped. +A swapped-in page is accounted after adding into swapcache. Note: The kernel does swapin-readahead and reads multiple swaps at once. -This means swapped-in pages may contain pages for other tasks than a task -causing page fault. So, we avoid accounting at swap-in I/O. +Since page's memcg recorded into swap whatever memsw enabled, the page will +be accounted after swapin. At page migration, accounting information is kept.
@@ -230,10 +230,10 @@ caller of swapoff rather than the users of shmem. 2.4 Swap Extension (CONFIG_MEMCG_SWAP) -------------------------------------- -Swap Extension allows you to record charge for swap. A swapped-in page is -charged back to original page allocator if possible. +Swap usage is always recorded for each of cgroup. Swap Extension allows you to +read and limit it. -When swap is accounted, following files are added. +When swap is limited, following files are added. - memory.memsw.usage_in_bytes. - memory.memsw.limit_in_bytes.
Quoted 4 lines
From a semantics POV, nothing changes with this patch. The memsw limit defaults to "max", so it doesn't exert any control per default. The only difference is whether we maintain swap records or not.
Johannes Weiner
maintainerRe: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
Quoted 15 lines · Alex Shi
Quoted 12 lines · Johannes Weiner
Hi Alex,
thanks for your quick review so far, I'll add the tags to the patches.
Quoted 7 lines · Alex Shi
Quoted 3 lines · Johannes Weiner
The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
May we add extra explanation for this change to user? and the default memsw limitations?
Can you elaborate what you think is missing and where you would like to see it documented?
Maybe the following doc change is better after whole patchset? Guess users would would happy to know details of this change.
Thanks, I stole your patch and extended/tweaked it a little. Would you mind providing your Signed-off-by:?
From 589d3c1b505e6671b4a9b424436c9eda88a0b08c Mon Sep 17 00:00:00 2001
From: Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/[email protected]>
Date: Wed, 22 Apr 2020 11:14:40 +0800
Subject: [PATCH] mm: memcontrol: document the new swap control behavior
--- .../admin-guide/cgroup-v1/memory.rst | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
Documentation/admin-guide/cgroup-v1/memory.rst +7 -12
@@ -199,11 +199,11 @@ An RSS page is unaccounted when it's fully unmapped. A PageCache page is unaccounted when it's removed from radix-tree. Even if RSS pages are fully unmapped (by kswapd), they may exist as SwapCache in the system until they are really freed. Such SwapCaches are also accounted. -A swapped-in page is not accounted until it's mapped. +A swapped-in page is accounted after adding into swapcache. Note: The kernel does swapin-readahead and reads multiple swaps at once. -This means swapped-in pages may contain pages for other tasks than a task -causing page fault. So, we avoid accounting at swap-in I/O. +Since page's memcg recorded into swap whatever memsw enabled, the page will +be accounted after swapin. At page migration, accounting information is kept.
@@ -222,18 +222,13 @@ the cgroup that brought it in -- this will happen on memory pressure). But see section 8.2: when moving a task to another cgroup, its pages may be recharged to the new cgroup, if move_charge_at_immigrate has been chosen. -Exception: If CONFIG_MEMCG_SWAP is not used. -When you do swapoff and make swapped-out pages of shmem(tmpfs) to -be backed into memory in force, charges for pages are accounted against the -caller of swapoff rather than the users of shmem. - -2.4 Swap Extension (CONFIG_MEMCG_SWAP) +2.4 Swap Extension -------------------------------------- -Swap Extension allows you to record charge for swap. A swapped-in page is -charged back to original page allocator if possible. +Swap usage is always recorded for each of cgroup. Swap Extension allows you to +read and limit it. -When swap is accounted, following files are added. +When CONFIG_SWAP is enabled, following files are added. - memory.memsw.usage_in_bytes. - memory.memsw.limit_in_bytes.
Quoted 2 lines
Also as to the RSS account name change, I don't know if it's good to polish them in docs.
I didn't actually change anything user-visible, just the internal name of the counters:
static const unsigned int memcg1_stats[] = {
NR_FILE_PAGES, /* was MEMCG_CACHE */
NR_ANON_MAPPED, /* was MEMCG_RSS */
NR_ANON_THPS, /* was MEMCG_RSS_HUGE */
NR_SHMEM,
NR_FILE_MAPPED,
NR_FILE_DIRTY,
NR_WRITEBACK,
MEMCG_SWAP,
};static const char *const memcg1_stat_names[] = {
"cache",
"rss",
"rss_huge",
"shmem",
"mapped_file",
"dirty",
"writeback",
"swap",
};Or did you refer to something else?
Re: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
Quoted 19 lines · Johannes Weiner
Quoted 18 lines · Alex Shi
Quoted 15 lines · Johannes Weiner
Hi Alex,
thanks for your quick review so far, I'll add the tags to the patches.
Quoted 9 lines · Alex Shi
Quoted 3 lines · Johannes Weiner
The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.May we add extra explanation for this change to user? and the default memsw limitations?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Can you elaborate what you think is missing and where you would like to see it documented?
Maybe the following doc change is better after whole patchset? Guess users would would happy to know details of this change.
Thanks, I stole your patch and extended/tweaked it a little. Would you mind providing your Signed-off-by:?
My pleasure. :)
Quoted 54 lines
From
589d3c1b505e6671b4a9b424436c9eda88a0b08cMon Sep 17 00:00:00 2001
From: Alex Shi <[email protected]>
Date: Wed, 22 Apr 2020 11:14:40 +0800
Subject: [PATCH] mm: memcontrol: document the new swap control behaviorSigned-off-by Johannes Weiner <[email protected]>--- .../admin-guide/cgroup-v1/memory.rst | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)Documentation/admin-guide/cgroup-v1/memory.rst +7
-12@@ -199,11 +199,11 @@ An RSS page is unaccounted when it's fully unmapped. A PageCache page is unaccounted when it's removed from radix-tree. Even if RSS pages are fully unmapped (by kswapd), they may exist as SwapCache in the system until they are really freed. Such SwapCaches are also accounted. -A swapped-in page is not accounted until it's mapped. +A swapped-in page is accounted after adding into swapcache. Note: The kernel does swapin-readahead and reads multiple swaps at once. -This means swapped-in pages may contain pages for other tasks than a task -causing page fault. So, we avoid accounting at swap-in I/O. +Since page's memcg recorded into swap whatever memsw enabled, the page will +be accounted after swapin. At page migration, accounting information is kept.@@ -222,18 +222,13 @@ the cgroup that brought it in -- this will happen on memory pressure). But see section 8.2: when moving a task to another cgroup, its pages may be recharged to the new cgroup, if move_charge_at_immigrate has been chosen. -Exception: If CONFIG_MEMCG_SWAP is not used. -When you do swapoff and make swapped-out pages of shmem(tmpfs) to -be backed into memory in force, charges for pages are accounted against the -caller of swapoff rather than the users of shmem. - -2.4 Swap Extension (CONFIG_MEMCG_SWAP) +2.4 Swap Extension -------------------------------------- -Swap Extension allows you to record charge for swap. A swapped-in page is -charged back to original page allocator if possible. +Swap usage is always recorded for each of cgroup. Swap Extension allows you to +read and limit it. -When swap is accounted, following files are added. +When CONFIG_SWAP is enabled, following files are added. - memory.memsw.usage_in_bytes. - memory.memsw.limit_in_bytes.
Re: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
In reply to Johannes WeinerQuoted 27 lines · Johannes Weiner
Quoted 2 lines
Also as to the RSS account name change, I don't know if it's good to polish them in docs.
I didn't actually change anything user-visible, just the internal name of the counters:
static const unsigned int memcg1_stats[] = { NR_FILE_PAGES, /* was MEMCG_CACHE */ NR_ANON_MAPPED, /* was MEMCG_RSS */ NR_ANON_THPS, /* was MEMCG_RSS_HUGE */ NR_SHMEM, NR_FILE_MAPPED, NR_FILE_DIRTY, NR_WRITEBACK, MEMCG_SWAP, };static const char *const memcg1_stat_names[] = { "cache", "rss", "rss_huge", "shmem", "mapped_file", "dirty", "writeback", "swap", };
Quoted 1 line
Or did you refer to something else?
With about 'was MEMCG_RSS' etc. I believe curious user would know where the concept from. :)
Thanks for this comments!
Aelx
Re: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
In reply to Johannes WeinerQuoted 20 lines · Johannes Weiner
Without swap page tracking, users that are otherwise memory controlled can easily escape their containment and allocate significant amounts of memory that they're not being charged for. That's because swap does readahead, but without the cgroup records of who owned the page at swapout, readahead pages don't get charged until somebody actually faults them into their page table and we can identify an owner task. This can be maliciously exploited with
MADV_WILLNEED, which triggers arbitrary readahead allocations without charging the pages.Make swap swap page tracking an integral part of memcg and remove the
Kconfigoptions. In the first place, it was only made configurable to allow users to save some memory. But the overhead of tracking cgroup ownership per swap page is minimal - 2 byte per page, or 512k per 1G of swap, or 0.04%. Saving that at the expense of broken containment semantics is not something we should present as a coequal option.The swapaccount=0 boot option will continue to exist, and it will eliminate the
page_counteroverhead and hide the swap control files, but it won't disable swap slot ownership tracking.
Quoted 5 lines
This patch makes sure we always have the cgroup records at swapin time; the next patch will fix the actual bug by charging readahead swap pages at swapin time rather than at fault time.
Signed-off-by Johannes Weiner <[email protected]>
Johannes Weiner
maintainerRe: [PATCH 15/18] mm: memcontrol: make swap tracking an integral part of memory control
In reply to Johannes Weinermm/memcontrol.c void mem_cgroup_swapout(struct page *page, swp_entry_t entry) · line 6881
Quoted 10 lines · Johannes Weiner
mm/memcontrol.c +0
-3@@ -6884,9 +6876,6 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)VM_BUG_ON_PAGE (PageLRU (page), page);VM_BUG_ON_PAGE (page_count (page), page); -if (!do_memsw_account ()) -return ; - memcg = page->mem_cgroup;/* Readahead page, never charged */
I messed up here.
mem_cgroup_swapout() must not run on cgroup2, because cgroup2 uses mem_cgroup_try_charge_swap() instead. Both record a swap entry and running them both will trigger a VM_BUG_ON() on an existing record.
I'm actually somewhat baffled why this didn't trigger in my MADV_PAGEOUT -> MADV_WILLNEED swap test. memory.max driven swapout triggered it right away.
!do_memsw_account() needs to be !cgroup_subsys_on_dfl(memory_cgrp_subsys)
mm/memcontrol.c void mem_cgroup_swapout(struct page *page, swp_entry_t entry) · line 6908
Quoted 9 lines
mm/memcontrol.c +1
-1@@ -6913,7 +6902,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)if (!mem_cgroup_is_root (memcg))page_counter_uncharge (&memcg->memory, nr_entries); -if (memcg != swap_memcg) { +if (do_memsw_account () && memcg != swap_memcg) {if (!mem_cgroup_is_root (swap_memcg))page_counter_charge (&swap_memcg->memsw, nr_entries);page_counter_uncharge (&memcg->memsw, nr_entries);
And this can be !cgroup_memory_noswap instead. It'll do the same thing, but will be clearer.
I'll have it fixed in version 2.
Johannes Weiner
maintainerPATCH 16/18 mm: memcontrol: charge swapin pages on instantiation
In reply to Johannes WeinerRight now, users that are otherwise memory controlled can easily escape their containment and allocate significant amounts of memory that they're not being charged for. That's because swap readahead pages are not being charged until somebody actually faults them into their page table. This can be exploited with MADV_WILLNEED, which triggers arbitrary readahead allocations without charging the pages.
There are additional problems with the delayed charging of swap pages:
1. To implement refault/workingset detection for anonymous pages, we need to have a target LRU available at swapin time, but the LRU is not determinable until the page has been charged.
2. To implement per-cgroup LRU locking, we need page->mem_cgroup to be stable when the page is isolated from the LRU; otherwise, the locks change under us. But swapcache gets charged after it's already on the LRU, and even if we cannot isolate it ourselves (since charging is not exactly optional).
The previous patch ensured we always maintain cgroup ownership records for swap pages. This patch moves the swapcache charging point from the fault handler to swapin time to fix all of the above problems.
--- mm/memory.c | 15 ++++++--- mm/shmem.c | 14 ++++---- mm/swap_state.c | 89 ++++++++++++++++++++++++++----------------------- mm/swapfile.c | 6 ---- 4 files changed, 67 insertions(+), 57 deletions(-)
mm/memory.c +11 -4
@@ -3127,9 +3127,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page =alloc_page_vma (GFP_HIGHUSER_MOVABLE , vma, vmf->address);if (page) { +int err; +__SetPageLocked (page);__SetPageSwapBacked (page);set_page_private (page, entry.val); + +/* Tell memcg to use swap ownership records */ +SetPageSwapCache (page); + err =mem_cgroup_charge (page, vma->vm_mm, +GFP_KERNEL , false); +ClearPageSwapCache (page); +if (err) + goto out_page; +lru_cache_add_anon (page);swap_readpage (page, true); }
@@ -3191,10 +3202,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_page; } -if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , true)) { - ret =VM_FAULT_OOM ; - goto out_page; - }cgroup_throttle_swaprate (page,GFP_KERNEL );/*
mm/shmem.c +8 -6
@@ -623,13 +623,15 @@ static int shmem_add_to_page_cache(struct page *page, page->mapping = mapping; page->index = index; - error =mem_cgroup_charge (page, charge_mm, gfp,PageSwapCache (page)); -if (error) { -if (!PageSwapCache (page) &&PageTransHuge (page)) { -count_vm_event (THP_FILE_FALLBACK ); -count_vm_event (THP_FILE_FALLBACK_CHARGE ); +if (!PageSwapCache (page)) { + error =mem_cgroup_charge (page, charge_mm, gfp, false); +if (error) { +if (PageTransHuge (page)) { +count_vm_event (THP_FILE_FALLBACK ); +count_vm_event (THP_FILE_FALLBACK_CHARGE ); + } + goto error; } - goto error; }cgroup_throttle_swaprate (page, gfp);
mm/swap_state.c +48 -41
124 lines not on this page · raw message
mm/swapfile.c +0 -6
@@ -1863,11 +1863,6 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,if (unlikely (!page))return -ENOMEM ; -if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , true)) { - ret = -ENOMEM ; - goto out_nolock; - } - pte =pte_offset_map_lock (vma->vm_mm, pmd, addr, &ptl);if (unlikely (!pte_same_as_swp (*pte,swp_entry_to_pte (entry)))) { ret =0 ;
@@ -1893,7 +1888,6 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,activate_page (page); out:pte_unmap_unlock (pte, ptl); -out_nolock:if (page != swapcache) {unlock_page (page);put_page (page);
Signature · 1 line
2.26.0
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 19 lines
Right now, users that are otherwise memory controlled can easily escape their containment and allocate significant amounts of memory that they're not being charged for. That's because swap readahead pages are not being charged until somebody actually faults them into their page table. This can be exploited with
MADV_WILLNEED, which triggers arbitrary readahead allocations without charging the pages.There are additional problems with the delayed charging of swap pages:
1. To implement refault/workingset detection for anonymous pages, we need to have a target LRU available at swapin time, but the LRU is not determinable until the page has been charged.
2. To implement per-cgroup LRU locking, we need
page->mem_cgroupto be stable when the page is isolated from the LRU; otherwise, the locks change under us. But swapcache gets charged after it's already on the LRU, and even if we cannot isolate it ourselves (since charging is not exactly optional).
Quoted 5 lines
The previous patch ensured we always maintain cgroup ownership records for swap pages. This patch moves the swapcache charging point from the fault handler to swapin time to fix all of the above problems.
Signed-off-by Johannes Weiner <[email protected]>
Quoted 175 lines · Johannes Weiner
Right now, users that are otherwise memory controlled can easily escape their containment and allocate significant amounts of memory that they're not being charged for. That's because swap readahead pages are not being charged until somebody actually faults them into their page table. This can be exploited with
MADV_WILLNEED, which triggers arbitrary readahead allocations without charging the pages.There are additional problems with the delayed charging of swap pages:
1. To implement refault/workingset detection for anonymous pages, we need to have a target LRU available at swapin time, but the LRU is not determinable until the page has been charged.
2. To implement per-cgroup LRU locking, we need
page->mem_cgroupto be stable when the page is isolated from the LRU; otherwise, the locks change under us. But swapcache gets charged after it's already on the LRU, and even if we cannot isolate it ourselves (since charging is not exactly optional).The previous patch ensured we always maintain cgroup ownership records for swap pages. This patch moves the swapcache charging point from the fault handler to swapin time to fix all of the above problems.
Signed-off-by Johannes Weiner <[email protected]>--- mm/memory.c | 15 ++++++--- mm/shmem.c | 14 ++++---- mm/swap_state.c | 89 ++++++++++++++++++++++++++----------------------- mm/swapfile.c | 6 ---- 4 files changed, 67 insertions(+), 57 deletions(-)mm/memory.c +11
-4@@ -3127,9 +3127,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page =alloc_page_vma (GFP_HIGHUSER_MOVABLE , vma, vmf->address);if (page) { +int err; +__SetPageLocked (page);__SetPageSwapBacked (page);set_page_private (page, entry.val); + +/* Tell memcg to use swap ownership records */ +SetPageSwapCache (page); + err =mem_cgroup_charge (page, vma->vm_mm, +GFP_KERNEL , false); +ClearPageSwapCache (page); +if (err) + goto out_page; +lru_cache_add_anon (page);swap_readpage (page, true); }@@ -3191,10 +3202,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_page; } -if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , true)) { - ret =VM_FAULT_OOM ; - goto out_page; - }cgroup_throttle_swaprate (page,GFP_KERNEL );/* mm/shmem.c +8
-6@@ -623,13 +623,15 @@ static int shmem_add_to_page_cache(struct page *page, page->mapping = mapping; page->index = index; - error =mem_cgroup_charge (page, charge_mm, gfp,PageSwapCache (page)); -if (error) { -if (!PageSwapCache (page) &&PageTransHuge (page)) { -count_vm_event (THP_FILE_FALLBACK ); -count_vm_event (THP_FILE_FALLBACK_CHARGE ); +if (!PageSwapCache (page)) { + error =mem_cgroup_charge (page, charge_mm, gfp, false); +if (error) { +if (PageTransHuge (page)) { +count_vm_event (THP_FILE_FALLBACK ); +count_vm_event (THP_FILE_FALLBACK_CHARGE ); + } + goto error; } - goto error; }cgroup_throttle_swaprate (page, gfp);mm/swap_state.c +13
-31@@ -360,12 +360,13 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, bool *new_page_allocated) { - struct page *found_page = NULL, *new_page = NULL; struct swap_info_struct *si; -int err; +struct page *page; + *new_page_allocated = false; - do { +for (;;) { +int err;/* * First check the swap cache. Since this is normally * called after lookup_swap_cache() failed, re-calling @@ -373,12 +374,12 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,*/ si =get_swap_device (entry);if (!si) -break ; - found_page =find_get_page (swap_address_space (entry), -swp_offset (entry)); +return NULL ; + page =find_get_page (swap_address_space (entry), +swp_offset (entry));put_swap_device (si); -if (found_page) -break ; +if (page) +return page;/* * Just skip read ahead for unused swap slot. @@ -389,21 +390,15 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,* else swap_off will be aborted if we return NULL. */ if (!__swp_swapcount (entry) && swap_slot_cache_enabled) -break ; - -/* -* Get a new page to read into from swap. -*/ -if (!new_page) { - new_page =alloc_page_vma (gfp_mask, vma, addr); -if (!new_page) -break ;/* Out of memory */ - } +return NULL ;/* * Swap entry may have been freed since our caller observed it. */ err =swapcache_prepare (entry); +if (!err) +break ; +if (err == -EEXIST ) {/* * We might race against get_swap_page() and stumble @@ -412,31 +407,43 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,*/ cond_resched ();continue ; - }else if (err)/* swp entry is obsolete ? */ -break ; - -/* May fail (-ENOMEM) if XArray node allocation failed. */ -__SetPageLocked (new_page); -__SetPageSwapBacked (new_page); - err =add_to_swap_cache (new_page, entry, gfp_mask &GFP_KERNEL ); -if (likely (!err)) { -/* Initiate read into locked page */ -SetPageWorkingset (new_page); -lru_cache_add_anon (new_page); - *new_page_allocated = true; -return new_page;
mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 412
Quoted 10 lines
mm/swap_state.c +2
-7} -__ClearPageLocked (new_page); -/* -* add_to_swap_cache() doesn't return -EEXIST, so we can safely -* clear SWAP_HAS_CACHE flag. -*/ -put_swap_page (new_page, entry); - }while (err != -ENOMEM ); +if (err)/* swp entry is obsolete ? */ +return NULL ;
"if (err)" is not needed since "!err" is already exiting the loop.
Quoted 10 lines
mm/swap_state.c +10
-0+ } + +/* +* The swap entry is ours to swap in. Prepare a new page. +*/ + + page =alloc_page_vma (gfp_mask, vma, addr); +if (!page) + goto fail_free; +
mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 432
Quoted 10 lines
mm/swap_state.c +10
-0+__SetPageLocked (page); +__SetPageSwapBacked (page); + +/* May fail (-ENOMEM) if XArray node allocation failed. */ +if (add_to_swap_cache (page, entry, gfp_mask &GFP_KERNEL )) + goto fail_unlock; + +if (mem_cgroup_charge (page,NULL , gfp_mask &GFP_KERNEL , false)) + goto fail_delete; +
I think that following order of operations is better than yours.
1. page alloc
2. memcg charge
3. swapcache_prepare
4. add_to_swap_cache
Reason is that page allocation and memcg charging could take for a long time due to reclaim and other tasks waiting this swapcache page could be blocked inbetween swapcache_prepare() and add_to_swap_cache().
Thanks.
Johannes Weiner
maintainerQuoted 22 lines · Joonsoo Kim
Quoted 21 lines · Johannes Weiner
mm/swap_state.c +0
-16@@ -412,31 +407,43 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,*/ cond_resched ();continue ; - }else if (err)/* swp entry is obsolete ? */ -break ; - -/* May fail (-ENOMEM) if XArray node allocation failed. */ -__SetPageLocked (new_page); -__SetPageSwapBacked (new_page); - err =add_to_swap_cache (new_page, entry, gfp_mask &GFP_KERNEL ); -if (likely (!err)) { -/* Initiate read into locked page */ -SetPageWorkingset (new_page); -lru_cache_add_anon (new_page); - *new_page_allocated = true; -return new_page; } -__ClearPageLocked (new_page); -/* -* add_to_swap_cache() doesn't return -EEXIST, so we can safely
Quoted 8 lines
mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 412
Quoted 6 lines
mm/swap_state.c +2
-4-* clear SWAP_HAS_CACHE flag. -*/ -put_swap_page (new_page, entry); - }while (err != -ENOMEM ); +if (err)/* swp entry is obsolete ? */ +return NULL ;"if (err)" is not needed since "!err" is already exiting the loop.
But we don't want to leave the loop, we want to leave the function. For example, if swapcache_prepare() says the entry is gone (-ENOENT), we don't want to exit the loop and allocate a page for it.
Quoted 22 lines
Quoted 9 lines
mm/swap_state.c +9
-0+ +/* +* The swap entry is ours to swap in. Prepare a new page. +*/ + + page =alloc_page_vma (gfp_mask, vma, addr); +if (!page) + goto fail_free; +mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 432
Quoted 10 lines
mm/swap_state.c +10
-0+__SetPageLocked (page); +__SetPageSwapBacked (page); + +/* May fail (-ENOMEM) if XArray node allocation failed. */ +if (add_to_swap_cache (page, entry, gfp_mask &GFP_KERNEL )) + goto fail_unlock; + +if (mem_cgroup_charge (page,NULL , gfp_mask &GFP_KERNEL , false)) + goto fail_delete; +I think that following order of operations is better than yours.
Quoted 8 lines
1. page alloc
2. memcg charge
3.swapcache_prepare
4.add_to_swap_cacheReason is that page allocation and memcg charging could take for a long time due to reclaim and other tasks waiting this swapcache page could be blocked inbetween
swapcache_prepare()andadd_to_swap_cache().
I see how that would be preferable, but memcg charging actually needs the swap(cache) information to figure out the cgroup that owned it at swapout, then uncharge the swapcache and drop the swap cgroup record.
Maybe it could be done, but I'm not sure that level of surgery would be worth the benefits? Whoever else would be trying to swap the page in at the same time is likely in the same memory situation, and would not necessarily be able to allocate pages any faster.
2020년 4월 24일 (금) 오전 11:51, Johannes Weiner <[email protected]>님이 작성:
Quoted 33 lines · Joonsoo Kim
Quoted 22 lines · Johannes Weiner
Quoted 21 lines
mm/swap_state.c +0
-16@@ -412,31 +407,43 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,*/ cond_resched ();continue ; - }else if (err)/* swp entry is obsolete ? */ -break ; - -/* May fail (-ENOMEM) if XArray node allocation failed. */ -__SetPageLocked (new_page); -__SetPageSwapBacked (new_page); - err =add_to_swap_cache (new_page, entry, gfp_mask &GFP_KERNEL ); -if (likely (!err)) { -/* Initiate read into locked page */ -SetPageWorkingset (new_page); -lru_cache_add_anon (new_page); - *new_page_allocated = true; -return new_page; } -__ClearPageLocked (new_page); -/* -* add_to_swap_cache() doesn't return -EEXIST, so we can safely Quoted 8 lines
mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 412
Quoted 6 lines
mm/swap_state.c +2
-4-* clear SWAP_HAS_CACHE flag. -*/ -put_swap_page (new_page, entry); - }while (err != -ENOMEM ); +if (err)/* swp entry is obsolete ? */ +return NULL ;"if (err)" is not needed since "!err" is already exiting the loop.
Quoted 3 lines
But we don't want to leave the loop, we want to leave the function. For example, if
swapcache_prepare()says the entry is gone (-ENOENT), we don't want to exit the loop and allocate a page for it.
Yes, so I said "if (err)" is not needed.
Just "return NULL;" would be enough.
Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
mm/swap_state.c +9
-0+ +/* +* The swap entry is ours to swap in. Prepare a new page. +*/ + + page =alloc_page_vma (gfp_mask, vma, addr); +if (!page) + goto fail_free; +mm/swap_state.c struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · line 432
Quoted 10 lines
mm/swap_state.c +10
-0+__SetPageLocked (page); +__SetPageSwapBacked (page); + +/* May fail (-ENOMEM) if XArray node allocation failed. */ +if (add_to_swap_cache (page, entry, gfp_mask &GFP_KERNEL )) + goto fail_unlock; + +if (mem_cgroup_charge (page,NULL , gfp_mask &GFP_KERNEL , false)) + goto fail_delete; +I think that following order of operations is better than yours.
Quoted 8 lines
1. page alloc
2. memcg charge
3.swapcache_prepare
4.add_to_swap_cacheReason is that page allocation and memcg charging could take for a long time due to reclaim and other tasks waiting this swapcache page could be blocked inbetween
swapcache_prepare()andadd_to_swap_cache().
Quoted 8 lines
I see how that would be preferable, but memcg charging actually needs the swap(cache) information to figure out the cgroup that owned it at swapout, then uncharge the swapcache and drop the swap cgroup record.
Maybe it could be done, but I'm not sure that level of surgery would be worth the benefits? Whoever else would be trying to swap the page in at the same time is likely in the same memory situation, and would not necessarily be able to allocate pages any faster.
Hmm, at least, some modifications are needed since waiting task would do busy waiting in the loop and it wastes overall system cpu time.
I still think that changing operation order is better since it's possible that later task allocates a page faster though it's not usual case. However, I also agree your reasoning so will not insist more.
Thanks.
Johannes Weiner
maintainerPATCH 17/18 mm: memcontrol: delete unused lrucare handling
In reply to Johannes Weiner--- include/linux/memcontrol.h | 5 ++-- kernel/events/uprobes.c | 3 +- mm/filemap.c | 2 +- mm/huge_memory.c | 7 ++--- mm/khugepaged.c | 4 +-- mm/memcontrol.c | 57 +++----------------------------------- mm/memory.c | 8 +++--- mm/migrate.c | 2 +- mm/shmem.c | 2 +- mm/swap_state.c | 2 +- mm/userfaultfd.c | 2 +- 11 files changed, 21 insertions(+), 73 deletions(-)
include/linux/memcontrol.h +2 -3
@@ -357,8 +357,7 @@ static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg,enum mem_cgroup_protection mem_cgroup_protected (struct mem_cgroup *root,struct mem_cgroup *memcg); -int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask, -bool lrucare); +int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask);void mem_cgroup_uncharge (struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);
@@ -839,7 +838,7 @@ static inline enum mem_cgroup_protection mem_cgroup_protected( }static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, -gfp_t gfp_mask,bool lrucare) +gfp_t gfp_mask) {return 0 ; }
kernel/events/uprobes.c +1 -2
@@ -167,8 +167,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, addr + PAGE_SIZE);if (new_page) { - err =mem_cgroup_charge (new_page, vma->vm_mm,GFP_KERNEL , - false); + err =mem_cgroup_charge (new_page, vma->vm_mm,GFP_KERNEL );if (err)return err; }
mm/filemap.c +1 -1
@@ -845,7 +845,7 @@ static int __add_to_page_cache_locked(struct page *page, page->index = offset;if (!huge) { - error =mem_cgroup_charge (page, current->mm, gfp_mask, false); + error =mem_cgroup_charge (page, current->mm, gfp_mask);if (error) goto error; }
mm/huge_memory.c +3 -4
@@ -593,7 +593,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,VM_BUG_ON_PAGE (!PageCompound (page), page); -if (mem_cgroup_charge (page, vma->vm_mm, gfp, false)) { +if (mem_cgroup_charge (page, vma->vm_mm, gfp)) {put_page (page);count_vm_event (THP_FAULT_FALLBACK );count_vm_event (THP_FAULT_FALLBACK_CHARGE );
@@ -1276,7 +1276,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, vmf->address, page_to_nid(page));if (unlikely (!pages[i] ||mem_cgroup_charge (pages[i], vma->vm_mm, -GFP_KERNEL , false))) { +GFP_KERNEL ))) {if (pages[i])put_page (pages[i]);while (--i >=0 )
@@ -1430,8 +1430,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) goto out; } -if (unlikely (mem_cgroup_charge (new_page, vma->vm_mm, huge_gfp, - false))) { +if (unlikely (mem_cgroup_charge (new_page, vma->vm_mm, huge_gfp))) {put_page (new_page);split_huge_pmd (vma, vmf->pmd, vmf->address);if (page)
mm/khugepaged.c +2 -2
@@ -973,7 +973,7 @@ static void collapse_huge_page(struct mm_struct *mm, goto out_nolock; } -if (unlikely (mem_cgroup_charge (new_page, mm, gfp, false))) { +if (unlikely (mem_cgroup_charge (new_page, mm, gfp))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out_nolock; }
@@ -1527,7 +1527,7 @@ static void collapse_file(struct mm_struct *mm, goto out; } -if (unlikely (mem_cgroup_charge (new_page, mm, gfp, false))) { +if (unlikely (mem_cgroup_charge (new_page, mm, gfp))) { result =SCAN_CGROUP_CHARGE_FAIL ; goto out; }
mm/memcontrol.c +4 -53
100 lines not on this page · raw message
mm/memory.c +4 -4
@@ -2677,7 +2677,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } } -if (mem_cgroup_charge (new_page, mm,GFP_KERNEL , false)) +if (mem_cgroup_charge (new_page, mm,GFP_KERNEL )) goto oom_free_new;cgroup_throttle_swaprate (new_page,GFP_KERNEL );
@@ -3136,7 +3136,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* Tell memcg to use swap ownership records */ SetPageSwapCache (page); err =mem_cgroup_charge (page, vma->vm_mm, -GFP_KERNEL , false); +GFP_KERNEL );ClearPageSwapCache (page);if (err) goto out_page;
@@ -3360,7 +3360,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (!page) goto oom; -if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , false)) +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;cgroup_throttle_swaprate (page,GFP_KERNEL );
@@ -3856,7 +3856,7 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf)if (!vmf->cow_page)return VM_FAULT_OOM ; -if (mem_cgroup_charge (vmf->cow_page, vma->vm_mm,GFP_KERNEL , false)) { +if (mem_cgroup_charge (vmf->cow_page, vma->vm_mm,GFP_KERNEL )) {put_page (vmf->cow_page);return VM_FAULT_OOM ; }
mm/migrate.c +1 -1
@@ -2792,7 +2792,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,if (unlikely (anon_vma_prepare (vma))) goto abort; -if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL , false)) +if (mem_cgroup_charge (page, vma->vm_mm,GFP_KERNEL )) goto abort;/*
mm/shmem.c +1 -1
@@ -624,7 +624,7 @@ static int shmem_add_to_page_cache(struct page *page, page->index = index;if (!PageSwapCache (page)) { - error =mem_cgroup_charge (page, charge_mm, gfp, false); + error =mem_cgroup_charge (page, charge_mm, gfp);if (error) {if (PageTransHuge (page)) {count_vm_event (THP_FILE_FALLBACK );
mm/swap_state.c +1 -1
@@ -427,7 +427,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,if (add_to_swap_cache (page, entry, gfp_mask &GFP_KERNEL )) goto fail_unlock; -if (mem_cgroup_charge (page,NULL , gfp_mask &GFP_KERNEL , false)) +if (mem_cgroup_charge (page,NULL , gfp_mask &GFP_KERNEL )) goto fail_delete;/* Initiate read into locked page */
mm/userfaultfd.c +1 -1
@@ -96,7 +96,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm,__SetPageUptodate (page); ret = -ENOMEM ; -if (mem_cgroup_charge (page, dst_mm,GFP_KERNEL , false)) +if (mem_cgroup_charge (page, dst_mm,GFP_KERNEL )) goto out_release; _dst_pte =pte_mkdirty (mk_pte (page, dst_vma->vm_page_prot));
Signature · 1 line
2.26.0
Re: [PATCH 17/18] mm: memcontrol: delete unused lrucare handling
Quoted 1 line · Johannes Weiner
Signed-off-by Johannes Weiner <[email protected]>
Code looks fine to me. With proper commit message,
Thanks.
Johannes Weiner
maintainerPATCH 18/18 mm: memcontrol: update page->mem_cgroup stability rules
In reply to Johannes WeinerThe previous patches have simplified the access rules around page->mem_cgroup somewhat:
1. We never change page->mem_cgroup while the page is isolated by somebody else. This was by far the biggest exception to our rules and it didn't stop at lock_page() or lock_page_memcg().
2. We charge pages before they get put into page tables now, so the somewhat fishy rule about "can be in page table as long as it's still locked" is now gone and boiled down to having an exclusive reference to the page.
Document the new rules. Any of the following will stabilize the page->mem_cgroup association:
- the page lock
- LRU isolation
- lock_page_memcg()
- exclusive access to the page
--- mm/memcontrol.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
mm/memcontrol.c +7 -14
@@ -1201,9 +1201,8 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg,* @page: the page * @pgdat: pgdat of the page * -* This function is only safe when following the LRU page isolation -* and putback protocol: the LRU lock must be held, and the page must -* either be PageLRU() or the caller must have isolated/allocated it. +* This function relies on page->mem_cgroup being stable - see the +* access rules in commit_charge(). */ struct lruvec *mem_cgroup_page_lruvec (struct page *page,struct pglist_data *pgdat) {
@@ -2605,18 +2604,12 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) {VM_BUG_ON_PAGE (page->mem_cgroup, page);/* -* Nobody should be changing or seriously looking at -* page->mem_cgroup at this point: -* -* - the page is uncharged -* -* - the page is off-LRU -* -* - an anonymous fault has exclusive page access, except for -* a locked page table +* Any of the following ensures page->mem_cgroup stability: * -* - a page cache insertion, a swapin fault, or a migration -* have the page locked +* - the page lock +* - LRU isolation +* - lock_page_memcg() +* - exclusive reference */ page->mem_cgroup = memcg; }
Signature · 1 line
2.26.0
Re: [PATCH 18/18] mm: memcontrol: update page->mem_cgroup stability rules
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 15 lines
The previous patches have simplified the access rules around
page->mem_cgroupsomewhat:1. We never change
page->mem_cgroupwhile the page is isolated by somebody else. This was by far the biggest exception to our rules and it didn't stop atlock_page()orlock_page_memcg().2. We charge pages before they get put into page tables now, so the somewhat fishy rule about "can be in page table as long as it's still locked" is now gone and boiled down to having an exclusive reference to the page.
Document the new rules. Any of the following will stabilize the
page->mem_cgroupassociation:
Quoted 6 lines
- the page lock
- LRU isolation
-lock_page_memcg()
- exclusive access to the pageSigned-off-by Johannes Weiner <[email protected]>
Quoted 45 lines
--- mm/memcontrol.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)mm/memcontrol.c +7
-14@@ -1201,9 +1201,8 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg,* @page: the page * @pgdat: pgdat of the page * -* This function is only safe when following the LRU page isolation -* and putback protocol: the LRU lock must be held, and the page must -* either be PageLRU() or the caller must have isolated/allocated it. +* This function relies on page->mem_cgroup being stable - see the +* access rules in commit_charge(). */ struct lruvec *mem_cgroup_page_lruvec (struct page *page,struct pglist_data *pgdat) {@@ -2605,18 +2604,12 @@ static void commit_charge(struct page *page, struct mem_cgroup *memcg) {VM_BUG_ON_PAGE (page->mem_cgroup, page);/* -* Nobody should be changing or seriously looking at -* page->mem_cgroup at this point: -* -* - the page is uncharged -* -* - the page is off-LRU -* -* - an anonymous fault has exclusive page access, except for -* a locked page table +* Any of the following ensures page->mem_cgroup stability: * -* - a page cache insertion, a swapin fault, or a migration -* have the page locked +* - the page lock +* - LRU isolation +* - lock_page_memcg() +* - exclusive reference */ page->mem_cgroup = memcg; }
Re: [PATCH 18/18] mm: memcontrol: update page->mem_cgroup stability rules
In reply to Johannes WeinerQuoted 15 lines · Johannes Weiner
The previous patches have simplified the access rules around
page->mem_cgroupsomewhat:1. We never change
page->mem_cgroupwhile the page is isolated by somebody else. This was by far the biggest exception to our rules and it didn't stop atlock_page()orlock_page_memcg().2. We charge pages before they get put into page tables now, so the somewhat fishy rule about "can be in page table as long as it's still locked" is now gone and boiled down to having an exclusive reference to the page.
Document the new rules. Any of the following will stabilize the
page->mem_cgroupassociation:
Quoted 6 lines
- the page lock
- LRU isolation
-lock_page_memcg()
- exclusive access to the pageSigned-off-by Johannes Weiner <[email protected]>
ÔÚ 2020/4/21 ÉÏÎç6:11, Johannes Weiner дµÀ:
Quoted 11 lines
This patch series reworks memcg to charge swapin pages directly at swapin time, rather than at fault time, which may be much later, or not happen at all.
The delayed charging scheme we have right now causes problems:
- Alex's per-cgroup
lru_lockpatches rely on pages that have been isolated from the LRU to have a stablepage->mem_cgroup; otherwise the lock may change underneath him. Swapcache pages are charged only after they are added to the LRU, and charging doesn't follow the LRU isolation protocol.
Hi Johannes,
Thanks a lot!
It looks all fine for me. I will rebase per cgroup lru_lock on this. Thanks!
Alex
Quoted 58 lines
- Joonsoo's anon workingset patches need a suitable LRU at the time the page enters the swap cache and displaces the non-resident info. But the correct LRU is only available after charging.
- It's a containment hole / DoS vector. Users can trigger arbitrarily large swap readahead using
MADV_WILLNEED. The memory is never charged unless somebody actually touches it.- It complicates the
page->mem_cgroupstabilization rulesIn order to charge pages directly at swapin time, the memcg code base needs to be prepared, and several overdue cleanups become a necessity:
To charge pages at swapin time, we need to always have cgroup ownership tracking of swap records. We also cannot rely on
page->mappingto tell apart page types at charge time, because that's only set up during a page fault.To eliminate the
page->mappingdependency, memcg needs to ditch its private page type counters (MEMCG_CACHE,MEMCG_RSS,NR_SHMEM) in favor of the generic vmstat counters and accounting sites, such asNR_FILE_PAGES,NR_ANON_MAPPEDetc.To switch to generic vmstat counters, the charge sequence must be adjusted such that
page->mem_cgroupis set up by the time these counters are modified.The series is structured as follows:
1. Bug fixes
2. Decoupling charging from rmap
3. Swap controller integration into memcg
4. Direct swapin chargingThe patches survive a simple
swapout->swapintest inside a virtual machine. Because this is blocking two major patch sets, I'm sending these out early and will continue testing in parallel to the review.include/linux/memcontrol.h | 53 +---- include/linux/mm.h | 4 +- include/linux/swap.h | 6 +- init/Kconfig | 17 +- kernel/events/uprobes.c | 10 +- mm/filemap.c | 43 ++--- mm/huge_memory.c | 45 ++--- mm/khugepaged.c | 25 +-- mm/memcontrol.c | 448 ++++++++++++++----------------------------- mm/memory.c | 51 ++--- mm/migrate.c | 20 +- mm/rmap.c | 53 +++-- mm/shmem.c | 117 +++++------ mm/swap_cgroup.c | 6 - mm/swap_state.c | 89 +++++---- mm/swapfile.c | 25 +-- mm/userfaultfd.c | 5 +- 17 files changed, 367 insertions(+), 650 deletions(-)
Re: [PATCH 18/18] mm: memcontrol: update page->mem_cgroup stability rules
In reply to Johannes WeinerQuoted 13 lines · On Mon, 20 Apr 2020 18:11:26 -0400 Johannes Weiner wrote:
The previous patches have simplified the access rules around
page->mem_cgroupsomewhat:1. We never change
page->mem_cgroupwhile the page is isolated by somebody else. This was by far the biggest exception to our rules and it didn't stop atlock_page()orlock_page_memcg().2. We charge pages before they get put into page tables now, so the somewhat fishy rule about "can be in page table as long as it's still locked" is now gone and boiled down to having an exclusive reference to the page.
Quoted 7 lines
Document the new rules. Any of the following will stabilize the
page->mem_cgroupassociation:- the page lock
- LRU isolation
-lock_page_memcg()
- exclusive access to the page
Then rule-1 makes rule-3 no longer needed in mem_cgroup_move_account()?
Johannes Weiner
maintainerRe: [PATCH 18/18] mm: memcontrol: update page->mem_cgroup stability rules
Quoted 18 lines · Hillf Danton
Quoted 16 lines · On Mon, 20 Apr 2020 18:11:26 -0400 Johannes Weiner wrote:
The previous patches have simplified the access rules around
page->mem_cgroupsomewhat:1. We never change
page->mem_cgroupwhile the page is isolated by somebody else. This was by far the biggest exception to our rules and it didn't stop atlock_page()orlock_page_memcg().2. We charge pages before they get put into page tables now, so the somewhat fishy rule about "can be in page table as long as it's still locked" is now gone and boiled down to having an exclusive reference to the page.
Document the new rules. Any of the following will stabilize the
page->mem_cgroupassociation:
Quoted 6 lines
Quoted 4 lines
- the page lock
- LRU isolation
-lock_page_memcg()
- exclusive access to the pageThen rule-1 makes rule-3 no longer needed in
mem_cgroup_move_account()?
Well, mem_cgroup_move_account() is the write side. It's the function that changes page->mem_cgroup. So it needs to take all these locks in order for the readside / fastpath to be okay with any one of them.