mm/memcontrol: recharge mlocked pages
Subsystems (13)
cover mm/memcontrol: recharge mlocked pages
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache. Also removed cgroup stays pinned by these mlocked pages.
This patchset implements recharging pages to cgroup of mlock user.
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlock
To keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
---
Konstantin Khlebnikov (7):
mm/memcontrol: move locking page out of mem_cgroup_move_account mm/memcontrol: add mem_cgroup_recharge
mm/mlock: add vma argument for mlock_vma_page() mm/mlock: recharge memory accounting to first mlock user mm/mlock: recharge memory accounting to second mlock user at munlock mm/vmscan: allow changing page memory cgroup during reclaim mm/mlock: recharge mlocked pages at culling by vmscan
Documentation/admin-guide/cgroup-v1/memory.rst | 5 + include/linux/memcontrol.h | 9 ++ include/linux/rmap.h | 3 - mm/gup.c | 2 mm/huge_memory.c | 4 - mm/internal.h | 6 + mm/ksm.c | 2 mm/memcontrol.c | 104 ++++++++++++++++-------- mm/migrate.c | 2 mm/mlock.c | 14 +++ mm/rmap.c | 5 + mm/vmscan.c | 17 ++-- 12 files changed, 121 insertions(+), 52 deletions(-)
--
Signature
PATCH 1/7 mm/memcontrol: move locking page out of mem_cgroup_move_account
Required for calling mem_cgroup_move_account() for already locked page.
--- mm/memcontrol.c | 64 +++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 33 deletions(-)
mm/memcontrol.c +31 -33
@@ -5135,7 +5135,8 @@ static struct page *mc_handle_file_pte(struct vm_area_struct *vma,* @from: mem_cgroup which the page is moved from. * @to: mem_cgroup which the page is moved to. @from != @to. * -* The caller must make sure the page is not on LRU (isolate_page() is useful.) +* The caller must lock the page and make sure it is not on LRU +* (isolate_page() is useful.) * * This function doesn't do "charge" to new cgroup and doesn't do "uncharge" * from old cgroup.
@@ -5147,24 +5148,15 @@ static int mem_cgroup_move_account(struct page *page, {unsigned long flags;unsigned int nr_pages = compound ?hpage_nr_pages (page) :1 ; -int ret;bool anon;VM_BUG_ON (from == to); +VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE (PageLRU (page), page);VM_BUG_ON (compound && !PageTransHuge (page)); -/* -* Prevent mem_cgroup_migrate() from looking at -* page->mem_cgroup of its source page while we change it. -*/ - ret = -EBUSY ; -if (!trylock_page (page)) - goto out; - - ret = -EINVAL ;if (page->mem_cgroup != from) - goto out_unlock; +return -EINVAL ; anon =PageAnon (page);
@@ -5204,18 +5196,14 @@ static int mem_cgroup_move_account(struct page *page, page->mem_cgroup = to;spin_unlock_irqrestore (&from->move_lock, flags); - ret =0 ; -local_irq_disable ();mem_cgroup_charge_statistics (to, page, compound, nr_pages);memcg_check_events (to, page);mem_cgroup_charge_statistics (from, page, compound, -nr_pages);memcg_check_events (from, page);local_irq_enable (); -out_unlock: -unlock_page (page); -out: -return ret; + +return 0 ; }/**
@@ -5535,36 +5523,42 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,struct vm_area_struct *vma = walk->vma;pte_t *pte;spinlock_t *ptl; -enum mc_target_type target_type;union mc_target target;struct page *page; ptl =pmd_trans_huge_lock (pmd, vma);if (ptl) { +bool device = false; +if (mc.precharge <HPAGE_PMD_NR ) {spin_unlock (ptl);return 0 ; } - target_type =get_mctgt_type_thp (vma, addr, *pmd, &target); -if (target_type ==MC_TARGET_PAGE ) { - page = target.page; -if (!isolate_lru_page (page)) { -if (!mem_cgroup_move_account (page, true, - mc.from, mc.to)) { - mc.precharge -=HPAGE_PMD_NR ; - mc.moved_charge +=HPAGE_PMD_NR ; - } -putback_lru_page (page); - } -put_page (page); - }else if (target_type ==MC_TARGET_DEVICE ) { + +switch (get_mctgt_type_thp (vma, addr, *pmd, &target)) { +case MC_TARGET_DEVICE : + device = true; +/* fall through */ +case MC_TARGET_PAGE : page = target.page; +if (!trylock_page (page)) + goto put_huge; +if (!device &&isolate_lru_page (page)) + goto unlock_huge;if (!mem_cgroup_move_account (page, true, mc.from, mc.to)) { mc.precharge -=HPAGE_PMD_NR ; mc.moved_charge +=HPAGE_PMD_NR ; } +if (!device) +putback_lru_page (page); +unlock_huge: +unlock_page (page); +put_huge:put_page (page); +break ; +default : +break ; }spin_unlock (ptl);return 0 ;
@@ -5596,8 +5590,10 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd,*/ if (PageTransCompound (page)) goto put; -if (!device &&isolate_lru_page (page)) +if (!trylock_page (page)) goto put; +if (!device &&isolate_lru_page (page)) + goto unlock;if (!mem_cgroup_move_account (page, false, mc.from, mc.to)) { mc.precharge--;
@@ -5606,6 +5602,8 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, }if (!device)putback_lru_page (page); +unlock: +unlock_page (page); put:/* get_mctgt_type() gets the page */ put_page (page);break ;
PATCH 2/7 mm/memcontrol: add mem_cgroup_recharge
In reply to Konstantin KhlebnikovThis function tries to move page into other cgroup. Caller must lock page and isolate it from LRU.
--- include/linux/memcontrol.h | 9 +++++++++ mm/memcontrol.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+)
include/linux/memcontrol.h +9 -0
@@ -352,6 +352,8 @@ void mem_cgroup_uncharge(struct page *page);void mem_cgroup_uncharge_list (struct list_head *page_list);void mem_cgroup_migrate (struct page *oldpage,struct page *newpage); +int mem_cgroup_try_recharge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask);static struct mem_cgroup_per_node *mem_cgroup_nodeinfo (struct mem_cgroup *memcg,int nid)
@@ -857,6 +859,13 @@ static inline void mem_cgroup_migrate(struct page *old, struct page *new) { } +static inline int mem_cgroup_try_recharge (struct page *page, +struct mm_struct *mm, +gfp_t gfp_mask) +{ +return 0 ; +} +static inline struct lruvec *mem_cgroup_lruvec (struct pglist_data *pgdat,struct mem_cgroup *memcg) {
mm/memcontrol.c +40 -0
@@ -6507,6 +6507,46 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)local_irq_restore (flags); } +/* +* mem_cgroup_try_recharge - try to recharge page to mm's memcg. +* +* Page must be locked and isolated. +*/ +int mem_cgroup_try_recharge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask) +{ +struct mem_cgroup *from, *to; +int nr_pages; +int err =0 ; + +VM_BUG_ON_PAGE (!PageLocked (page), page); +VM_BUG_ON_PAGE (PageLRU (page), page); + +if (mem_cgroup_disabled ()) +return 0 ; + + from = page->mem_cgroup; + to =get_mem_cgroup_from_mm (mm); + +if (likely (from == to) || !from) + goto out; + + nr_pages =hpage_nr_pages (page); + err =try_charge (to, gfp_mask, nr_pages); +if (err) + goto out; + + err =mem_cgroup_move_account (page, nr_pages >1 , from, to); +if (err) +cancel_charge (to, nr_pages); +else +cancel_charge (from, nr_pages); +out: +css_put (&to->css); + +return err; +} +DEFINE_STATIC_KEY_FALSE (memcg_sockets_enabled_key);EXPORT_SYMBOL (memcg_sockets_enabled_key);
PATCH 3/7 mm/mlock: add vma argument for mlock_vma_page()
In reply to Konstantin KhlebnikovThis will be used for recharging memory cgroup accounting.
--- mm/gup.c | 2 +- mm/huge_memory.c | 4 ++-- mm/internal.h | 4 ++-- mm/ksm.c | 2 +- mm/migrate.c | 2 +- mm/mlock.c | 2 +- mm/rmap.c | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-)
mm/gup.c +1 -1
@@ -306,7 +306,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,* know the page is still mapped, we don't even * need to check for file-cache page truncation. */ -mlock_vma_page (page); +mlock_vma_page (vma, page);unlock_page (page); } }
mm/huge_memory.c +2 -2
@@ -1513,7 +1513,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, goto skip_mlock;lru_add_drain ();if (page->mapping && !PageDoubleMap (page)) -mlock_vma_page (page); +mlock_vma_page (vma, page);unlock_page (page); } skip_mlock:
@@ -3009,7 +3009,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)page_add_file_rmap (new, true);set_pmd_at (mm, mmun_start, pvmw->pmd, pmde);if ((vma->vm_flags &VM_LOCKED ) && !PageDoubleMap (new)) -mlock_vma_page (new); +mlock_vma_page (vma, new); update_mmu_cache_pmd(vma, address, pvmw->pmd); }#endif
mm/internal.h +2 -2
@@ -305,7 +305,7 @@ static inline void munlock_vma_pages_all(struct vm_area_struct *vma)/* * must be called with vma's mmap_sem held for read or write, and page locked. */ -extern void mlock_vma_page (struct page *page); +extern void mlock_vma_page (struct vm_area_struct *vma,struct page *page);extern unsigned int munlock_vma_page (struct page *page);/*
@@ -364,7 +364,7 @@ vma_address(struct page *page, struct vm_area_struct *vma)#else /* !CONFIG_MMU */ static inline void clear_page_mlock (struct page *page) { } -static inline void mlock_vma_page (struct page *page) { } +static inline void mlock_vma_page (struct vm_area_struct *,struct page *) { }static inline void mlock_migrate_page (struct page *new,struct page *old) { }#endif /* !CONFIG_MMU */
mm/ksm.c +1 -1
@@ -1274,7 +1274,7 @@ static int try_to_merge_one_page(struct vm_area_struct *vma,if (!PageMlocked (kpage)) {unlock_page (page);lock_page (kpage); -mlock_vma_page (kpage); +mlock_vma_page (vma, kpage); page = kpage;/* for final unlock */ } }
mm/migrate.c +1 -1
@@ -269,7 +269,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,page_add_file_rmap (new, false); }if (vma->vm_flags &VM_LOCKED && !PageTransCompound (new)) -mlock_vma_page (new); +mlock_vma_page (vma, new);if (PageTransHuge (page) &&PageMlocked (page))clear_page_mlock (page);
mm/mlock.c +1 -1
@@ -85,7 +85,7 @@ void clear_page_mlock(struct page *page)* Mark page as mlocked if not already. * If page on LRU, isolate and putback to move to unevictable list. */ -void mlock_vma_page (struct page *page) +void mlock_vma_page (struct vm_area_struct *vma,struct page *page) {/* Serialize with page migration */ BUG_ON (!PageLocked (page));
mm/rmap.c +1 -1
@@ -1410,7 +1410,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,* Holding pte lock, we do *not* need * mmap_sem here */ -mlock_vma_page (page); +mlock_vma_page (vma, page); } ret = false;page_vma_mapped_walk_done (&pvmw);
PATCH 4/7 mm/mlock: recharge memory accounting to first mlock user
In reply to Konstantin KhlebnikovCurrently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of file.
This patch recharges accounting to cgroup which owns mm for mlock vma. Recharging happens at first mlock, when PageMlocked is set.
Mlock moves pages into unevictable LRU under pte lock thus in this place we cannot call reclaimer. To keep things simple just charge using force. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
--- Documentation/admin-guide/cgroup-v1/memory.rst | 5 +++++ mm/mlock.c | 9 ++++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
Documentation/admin-guide/cgroup-v1/memory.rst +5 -0
@@ -220,6 +220,11 @@ the cgroup that brought it in -- this will happen on memory pressure). But see section 8.2: when moving a task to another cgroup, its pages may be recharged to the new cgroup, if move_charge_at_immigrate has been chosen. +Locking pages in memory with mlock() or mmap(MAP_LOCKED) recharges pages +into current memory cgroup. This recharge ignores memory limit thus memory +usage could temporary become higher than limit. After that any allocation +will reclaim memory down to limit or trigger oom if mlock size does not fit. + Exception: If CONFIG_MEMCG_SWAP is not used. When you do swapoff and make swapped-out pages of shmem(tmpfs) to be backed into memory in force, charges for pages are accounted against the
mm/mlock.c +8 -1
@@ -97,8 +97,15 @@ void mlock_vma_page(struct vm_area_struct *vma, struct page *page)mod_zone_page_state (page_zone (page),NR_MLOCK ,hpage_nr_pages (page));count_vm_event (UNEVICTABLE_PGMLOCKED ); -if (!isolate_lru_page (page)) +if (!isolate_lru_page (page)) { +/* +* Force memory recharge to mlock user. Cannot +* reclaim memory because called under pte lock. +*/ +mem_cgroup_try_recharge (page, vma->vm_mm, +GFP_NOWAIT | __GFP_NOFAIL);putback_lru_page (page); + } } }
PATCH 5/7 mm/mlock: recharge memory accounting to second mlock user at munlock
In reply to Konstantin KhlebnikovMunlock isolates page from LRU and then looks for another mlock vma. Thus we could could rechange page to second mlock without isolating.
This patch adds argument 'isolated' to mlock_vma_page() and passes this flag through try_to_ummap as TTU_LRU_ISOLATED.
--- include/linux/rmap.h | 3 ++- mm/gup.c | 2 +- mm/huge_memory.c | 4 ++-- mm/internal.h | 6 ++++-- mm/ksm.c | 2 +- mm/migrate.c | 2 +- mm/mlock.c | 21 ++++++++++++--------- mm/rmap.c | 5 +++-- 8 files changed, 26 insertions(+), 19 deletions(-)
include/linux/rmap.h +2 -1
@@ -98,7 +98,8 @@ enum ttu_flags {* do a final flush if necessary */ TTU_RMAP_LOCKED =0x80 ,/* do not grab rmap lock: * caller holds it */ - TTU_SPLIT_FREEZE = 0x100, /* freeze pte under splitting thp */ +TTU_SPLIT_FREEZE =0x100 ,/* freeze pte under splitting thp */ + TTU_LRU_ISOLATED = 0x200, /* caller isolated page from LRU */ };#ifdef CONFIG_MMU
mm/gup.c +1 -1
@@ -306,7 +306,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,* know the page is still mapped, we don't even * need to check for file-cache page truncation. */ -mlock_vma_page (vma, page); +mlock_vma_page (vma, page, false);unlock_page (page); } }
mm/huge_memory.c +2 -2
@@ -1513,7 +1513,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, goto skip_mlock;lru_add_drain ();if (page->mapping && !PageDoubleMap (page)) -mlock_vma_page (vma, page); +mlock_vma_page (vma, page, false);unlock_page (page); } skip_mlock:
@@ -3009,7 +3009,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)page_add_file_rmap (new, true);set_pmd_at (mm, mmun_start, pvmw->pmd, pmde);if ((vma->vm_flags &VM_LOCKED ) && !PageDoubleMap (new)) -mlock_vma_page (vma, new); +mlock_vma_page (vma, new, false); update_mmu_cache_pmd(vma, address, pvmw->pmd); }#endif
mm/internal.h +4 -2
@@ -305,7 +305,8 @@ static inline void munlock_vma_pages_all(struct vm_area_struct *vma)/* * must be called with vma's mmap_sem held for read or write, and page locked. */ -extern void mlock_vma_page (struct vm_area_struct *vma,struct page *page); +extern void mlock_vma_page (struct vm_area_struct *vma,struct page *page, +bool isolated);extern unsigned int munlock_vma_page (struct page *page);/*
@@ -364,7 +365,8 @@ vma_address(struct page *page, struct vm_area_struct *vma)#else /* !CONFIG_MMU */ static inline void clear_page_mlock (struct page *page) { } -static inline void mlock_vma_page (struct vm_area_struct *,struct page *) { } +static inline void mlock_vma_page (struct vm_area_struct *vma, +struct page *page,bool isolated) { }static inline void mlock_migrate_page (struct page *new,struct page *old) { }#endif /* !CONFIG_MMU */
mm/ksm.c +1 -1
@@ -1274,7 +1274,7 @@ static int try_to_merge_one_page(struct vm_area_struct *vma,if (!PageMlocked (kpage)) {unlock_page (page);lock_page (kpage); -mlock_vma_page (vma, kpage); +mlock_vma_page (vma, kpage, false); page = kpage;/* for final unlock */ } }
mm/migrate.c +1 -1
@@ -269,7 +269,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,page_add_file_rmap (new, false); }if (vma->vm_flags &VM_LOCKED && !PageTransCompound (new)) -mlock_vma_page (vma, new); +mlock_vma_page (vma, new, false);if (PageTransHuge (page) &&PageMlocked (page))clear_page_mlock (page);
mm/mlock.c +12 -9
@@ -85,7 +85,8 @@ void clear_page_mlock(struct page *page)* Mark page as mlocked if not already. * If page on LRU, isolate and putback to move to unevictable list. */ -void mlock_vma_page (struct vm_area_struct *vma,struct page *page) +void mlock_vma_page (struct vm_area_struct *vma,struct page *page, +bool isolated) {/* Serialize with page migration */ BUG_ON (!PageLocked (page));
@@ -97,15 +98,17 @@ void mlock_vma_page(struct vm_area_struct *vma, struct page *page)mod_zone_page_state (page_zone (page),NR_MLOCK ,hpage_nr_pages (page));count_vm_event (UNEVICTABLE_PGMLOCKED ); -if (!isolate_lru_page (page)) { -/* -* Force memory recharge to mlock user. Cannot -* reclaim memory because called under pte lock. -*/ -mem_cgroup_try_recharge (page, vma->vm_mm, -GFP_NOWAIT | __GFP_NOFAIL); + +if (!isolated &&isolate_lru_page (page)) +return ; +/* +* Force memory recharge to mlock user. +* Cannot reclaim memory because called under pte lock. +*/ +mem_cgroup_try_recharge (page, vma->vm_mm, +GFP_NOWAIT | __GFP_NOFAIL); +if (!isolated)putback_lru_page (page); - } } }
mm/rmap.c +3 -2
@@ -1410,7 +1410,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,* Holding pte lock, we do *not* need * mmap_sem here */ -mlock_vma_page (vma, page); +mlock_vma_page (vma, page, + !!(flags &TTU_LRU_ISOLATED )); } ret = false;page_vma_mapped_walk_done (&pvmw);
@@ -1752,7 +1753,7 @@ void try_to_munlock(struct page *page) {struct rmap_walk_control rwc = { .rmap_one = try_to_unmap_one, - .arg = (void *)TTU_MUNLOCK , + .arg = (void *)(TTU_MUNLOCK |TTU_LRU_ISOLATED ), .done = page_not_mapped, .anon_lock = page_lock_anon_vma_read,
PATCH 6/7 mm/vmscan: allow changing page memory cgroup during reclaim
In reply to Konstantin KhlebnikovAll LRU lists in one numa node are protected with one spin-lock and right now move_pages_to_lru() re-evaluates lruvec for each page. This allows to change page cgroup while page is isolated by reclaimer, but nobody use that for now. This patch makes this feature clear and passes into move_pages_to_lru pgdat rather than lruvec pointer.
--- mm/vmscan.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
mm/vmscan.c +8 -6
@@ -1873,15 +1873,15 @@ static int too_many_isolated(struct pglist_data *pgdat, int file,* The downside is that we have to touch page->_refcount against each page. * But we had to alter page->flags anyway. * -* Returns the number of pages moved to the given lruvec. +* Returns the number of pages moved to LRU lists. */ -static unsigned noinline_for_stack move_pages_to_lru (struct lruvec *lruvec, +static unsigned noinline_for_stack move_pages_to_lru (struct pglist_data *pgdat,struct list_head *list) { -struct pglist_data *pgdat =lruvec_pgdat (lruvec);int nr_pages, nr_moved =0 ;LIST_HEAD (pages_to_free); +struct lruvec *lruvec;struct page *page;enum lru_list lru;
@@ -1895,6 +1895,8 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec,spin_lock_irq (&pgdat->lru_lock);continue ; } + +/* Re-evaluate lru: isolated page could be moved */ lruvec =mem_cgroup_page_lruvec (page, pgdat);SetPageLRU (page);
@@ -2005,7 +2007,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, reclaim_stat->recent_rotated[0 ] += stat.nr_activate[0 ]; reclaim_stat->recent_rotated[1 ] += stat.nr_activate[1 ]; -move_pages_to_lru (lruvec, &page_list); +move_pages_to_lru (pgdat, &page_list);__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, -nr_taken);
@@ -2128,8 +2130,8 @@ static void shrink_active_list(unsigned long nr_to_scan,*/ reclaim_stat->recent_rotated[file] += nr_rotated; - nr_activate =move_pages_to_lru (lruvec, &l_active); - nr_deactivate =move_pages_to_lru (lruvec, &l_inactive); + nr_activate =move_pages_to_lru (pgdat, &l_active); + nr_deactivate =move_pages_to_lru (pgdat, &l_inactive);/* Keep all free pages in l_active list */ list_splice (&l_inactive, &l_active);
PATCH 7/7 mm/mlock: recharge mlocked pages at culling by vmscan
In reply to Konstantin KhlebnikovIf mlock cannot catch page in LRU then it isn't moved into unevictable lru. These pages are 'culled' by reclaimer and moved into unevictable lru. It seems pages locked with MLOCK_ONFAULT always go through this path.
Reclaimer calls try_to_unmap for already isolated pages, thus on this path we could freely change page to owner of any mlock vma we found in rmap.
This patch passes flag TTU_LRU_ISOLATED into try_to_ummap to move pages in mmlock_vma_page().
--- mm/vmscan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/vmscan.c +2 -1
@@ -1345,7 +1345,8 @@ static unsigned long shrink_page_list(struct list_head *page_list,* processes. Try to unmap it here. */ if (page_mapped (page)) { -enum ttu_flags flags = ttu_flags |TTU_BATCH_FLUSH ; +enum ttu_flags flags = ttu_flags |TTU_BATCH_FLUSH | +TTU_LRU_ISOLATED ;if (unlikely (PageTransHuge (page))) flags |=TTU_SPLIT_HUGE_PMD ;
Michal Hocko
maintainerQuoted 8 lines · Konstantin Khlebnikov
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache. Also removed cgroup stays pinned by these mlocked pages.
This patchset implements recharging pages to cgroup of mlock user.
Quoted 8 lines
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlockTo keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
I assume that this is mlock specific because the pagecache which has the same problem is reclaimable and the problem tends to resolve itself after some time.
Anyway, how big of a problem this really is? A lingering memcg is certainly not nice but pages are usually not mlocked for ever. Or is this a way to protect from an hostile actor?
Quoted 26 lines
Konstantin Khlebnikov (7):
mm/memcontrol: move locking page out ofmem_cgroup_move_accountmm/memcontrol: addmem_cgroup_recharge
mm/mlock: add vma argument formlock_vma_page()mm/mlock: recharge memory accounting to first mlock user mm/mlock: recharge memory accounting to second mlock user at munlock mm/vmscan: allow changing page memory cgroup during reclaim mm/mlock: recharge mlocked pages at culling by vmscanDocumentation/admin-guide/cgroup-v1/memory.rst | 5 + include/linux/memcontrol.h | 9 ++ include/linux/rmap.h | 3 - mm/gup.c | 2 mm/huge_memory.c | 4 - mm/internal.h | 6 + mm/ksm.c | 2 mm/memcontrol.c | 104 ++++++++++++++++-------- mm/migrate.c | 2 mm/mlock.c | 14 +++ mm/rmap.c | 5 + mm/vmscan.c | 17 ++-- 12 files changed, 121 insertions(+), 52 deletions(-)--
Signature
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 18 lines · Michal Hocko
Quoted 8 lines · Konstantin Khlebnikov
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache. Also removed cgroup stays pinned by these mlocked pages.
This patchset implements recharging pages to cgroup of mlock user.
Quoted 8 lines
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlockTo keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
Quoted 7 lines
I assume that this is mlock specific because the pagecache which has the same problem is reclaimable and the problem tends to resolve itself after some time.
Anyway, how big of a problem this really is? A lingering memcg is certainly not nice but pages are usually not mlocked for ever. Or is this a way to protect from an hostile actor?
We're using mlock mostly to avoid non-deterministic behaviour in cache. For example some of our applications mlock index structures in databases to limit count of major faults in worst case.
Surprisingly mlock fixates unwanted effects of non-predictable cache sharing.
So, it seems makes sense to make mlock behaviour simple and completely deterministic because this isn't cheap operation and needs careful resource planning.
Quoted 14 lines · Konstantin Khlebnikov
Quoted 4 lines
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache.
Yeah, it's a valid problem, and it's not about mlocked pages only, the same thing is true for generic pagecache. The only difference is that in theory memory pressure should fix everything. But in reality pagecache used by the second container can be very hot, so the first once can't really get rid of it.
In other words, there is no way to pass a pagecache page between cgroups without evicting it and re-reading from a storage, which is sub-optimal in many cases.
Quoted 3 lines
We thought about new
madvise(), which will uncharge pagecache but set a new page flag, which will mean something like "whoever first starts using the page, should be charged for it". But it never materialized in a patchset.
I've implemented something similar in OpenVZ kernel - "shadow" LRU sets for abandoned cache which automatically changes ownership at first activation.
I'm thinking about fadvise() or fcntl() for moving cache into current memory cgroup. This should give enough control to solve all our problems.
Quoted 5 lines
Quoted 2 lines
Also removed cgroup stays pinned by these mlocked pages.
Tbh, I don't think it's a big issue here. If only there is a huge number of 1-page sized mlock areas, but this seems to be unlikely.
Yep, not so big problem, tmpfs generates much more issues in this area.
Quoted 9 lines
Quoted 9 lines
This patchset implements recharging pages to cgroup of mlock user.
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlock
Quoted 8 lines
Quoted 3 lines
To keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
OOM is a concern here. If quitting an application will cause an immediate OOM in an other cgroup, that's not so good. Ideally it should work like memory.high, forcing all threads in the second cgroup into direct reclaim.
Mlock requires careful resource planning. Since sharing always been non-deterministic each user should be ready to take all locked memory.
It's hard to inject direct reclaim into another thread for sure. All we could do is starting background reclaim in kernel thread. Doing this in task who calls munlock() is not fair.
At mlock it's possible to force direct claim for memory usage over high limit:
https://lore.kernel.org/linux-mm/156431697805.3170.6377599347542228221.stgit@buzz/
Quoted 28 lines
Quoted 27 lines
Konstantin Khlebnikov (7):
mm/memcontrol: move locking page out ofmem_cgroup_move_accountmm/memcontrol: addmem_cgroup_recharge
mm/mlock: add vma argument formlock_vma_page()mm/mlock: recharge memory accounting to first mlock user mm/mlock: recharge memory accounting to second mlock user at munlock mm/vmscan: allow changing page memory cgroup during reclaim mm/mlock: recharge mlocked pages at culling by vmscanDocumentation/admin-guide/cgroup-v1/memory.rst | 5 + include/linux/memcontrol.h | 9 ++ include/linux/rmap.h | 3 - mm/gup.c | 2 mm/huge_memory.c | 4 - mm/internal.h | 6 + mm/ksm.c | 2 mm/memcontrol.c | 104 ++++++++++++++++-------- mm/migrate.c | 2 mm/mlock.c | 14 +++ mm/rmap.c | 5 + mm/vmscan.c | 17 ++-- 12 files changed, 121 insertions(+), 52 deletions(-)--
Signature
Quoted 31 lines · Konstantin Khlebnikov
Quoted 18 lines · Michal Hocko
Quoted 8 lines · Konstantin Khlebnikov
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache. Also removed cgroup stays pinned by these mlocked pages.
This patchset implements recharging pages to cgroup of mlock user.
Quoted 8 lines
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlockTo keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
Quoted 7 lines
I assume that this is mlock specific because the pagecache which has the same problem is reclaimable and the problem tends to resolve itself after some time.
Anyway, how big of a problem this really is? A lingering memcg is certainly not nice but pages are usually not mlocked for ever. Or is this a way to protect from an hostile actor?
We're using mlock mostly to avoid non-deterministic behaviour in cache. For example some of our applications mlock index structures in databases to limit count of major faults in worst case.
Quoted 6 lines
Surprisingly mlock fixates unwanted effects of non-predictable cache sharing.
So, it seems makes sense to make mlock behaviour simple and completely deterministic because this isn't cheap operation and needs careful resource planning.
Totally agree.
Quoted 21 lines · Konstantin Khlebnikov
Quoted 14 lines
Quoted 4 lines
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache.
Yeah, it's a valid problem, and it's not about mlocked pages only, the same thing is true for generic pagecache. The only difference is that in theory memory pressure should fix everything. But in reality pagecache used by the second container can be very hot, so the first once can't really get rid of it.
In other words, there is no way to pass a pagecache page between cgroups without evicting it and re-reading from a storage, which is sub-optimal in many cases.Quoted 3 lines
We thought about new
madvise(), which will uncharge pagecache but set a new page flag, which will mean something like "whoever first starts using the page, should be charged for it". But it never materialized in a patchset.
Quoted 5 lines
I've implemented something similar in OpenVZ kernel - "shadow" LRU sets for abandoned cache which automatically changes ownership at first activation.
I'm thinking about
fadvise()orfcntl()for moving cache into current memory cgroup. This should give enough control to solve all our problems.
Idk, it feels a bit fragile: because only one cgroup can own a page, there should be a strong coordination, otherwise cgroups may just spend non-trivial amount of cpu time stealing pages back and forth.
I'm not strictly against such fadvise() though, it can be useful in certain cases. But in general the abandoning semantics makes more sense to me. If a cgroup doesn't plan to use the page anymore, it marks it in a special way, so that the next one who want to use it pays the whole price. So it works exactly as if the page had been evicted, but more efficiently.
Thanks!
Quoted 4 lines · Konstantin Khlebnikov
Currently mlock keeps pages in cgroups where they were accounted. This way one container could affect another if they share file cache. Typical case is writing (downloading) file in one container and then locking in another. After that first container cannot get rid of cache.
Yeah, it's a valid problem, and it's not about mlocked pages only, the same thing is true for generic pagecache. The only difference is that in theory memory pressure should fix everything. But in reality pagecache used by the second container can be very hot, so the first once can't really get rid of it.
In other words, there is no way to pass a pagecache page between cgroups without evicting it and re-reading from a storage, which is sub-optimal in many cases.
We thought about new madvise(), which will uncharge pagecache but set a new page flag, which will mean something like "whoever first starts using the page, should be charged for it". But it never materialized in a patchset.
Quoted 1 line
Also removed cgroup stays pinned by these mlocked pages.
Tbh, I don't think it's a big issue here. If only there is a huge number of 1-page sized mlock areas, but this seems to be unlikely.
Quoted 11 lines
This patchset implements recharging pages to cgroup of mlock user.
There are three cases:
* recharging at first mlock
* recharging at munlock to any remaining mlock
* recharging at 'culling' in reclaimer to any existing mlockTo keep things simple recharging ignores memory limit. After that memory usage temporary could be higher than limit but cgroup will reclaim memory later or trigger oom, which is valid outcome when somebody mlock too much.
OOM is a concern here. If quitting an application will cause an immediate OOM in an other cgroup, that's not so good. Ideally it should work like memory.high, forcing all threads in the second cgroup into direct reclaim.
Thanks!