Use obj_cgroup APIs to charge kmem pages
Subsystems (3)
v4 cover Use obj_cgroup APIs to charge kmem pages
Since Roman series "The new cgroup slab memory controller" applied. All slab objects are charged with the new APIs of obj_cgroup. The new APIs introduce a struct obj_cgroup to charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory. But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged with the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup.
E.g. We know that the kernel stack is charged as kmem pages because the size of the kernel stack can be greater than 2 pages (e.g. 16KB on x86_64 or arm64). If we create a thread (suppose the thread stack is charged to memory cgroup A) and then move it from memory cgroup A to memory cgroup B. Because the kernel stack of the thread hold a reference to the memory cgroup A. The thread can pin the memory cgroup A in the memory even if we remove the cgroup A. If we want to see this scenario by using the following script. We can see that the system has added 500 dying cgroups (This is not a real world issue, just a script to show that the large kmallocs are charged as kmem pages which can pin the memory cgroup in the memory).
#!/bin/bash
cat /proc/cgroups | grep memory
cd /sys/fs/cgroup/memory
echo 1 > memory.move_charge_at_immigrate
for i in range{1..500}
do
mkdir kmem_test
echo $$ > kmem_test/cgroup.procs
sleep 3600 &
echo $$ > cgroup.procs
echo cat kmem_test/cgroup.procs > cgroup.procs rmdir kmem_test
done
cat /proc/cgroups | grep memory
This patchset aims to make those kmem pages to drop the reference to memory cgroup by using the APIs of obj_cgroup. Finally, we can see that the number of the dying cgroups will not increase if we run the above test script.
Changlogs in v4:
1. Do not change behavior of page_memcg() and page_memcg_rcu().
2. Rework uncharge_page() and uncharge_batch().
3. Add two patches (patch #2 and patch #3).
Thanks to Johannes and Shakeel and Roman's review and suggestions.
Changlogs in v3:
1. Drop "remote objcg charging APIs" patch.
2. Rename obj_cgroup_{un}charge_page to obj_cgroup_{un}charge_pages.
3. Make page_memcg/page_memcg_rcu safe for adding new memcg_data flags.
4. Reuse the ug infrastructure to uncharge the kmem pages.
5. Add a new patch to move PageMemcgKmem to the scope of CONFIG_MEMCG_KMEM.
Thanks to Roman's review and suggestions.
Changlogs in v2:
1. Fix some types in the commit log (Thanks Roman).
2. Do not introduce page_memcg_kmem helper (Thanks to Johannes and Shakeel).
3. Reduce the CC list to mm/memcg folks (Thanks to Johannes).
4. Introduce remote objcg charging APIs instead of convert "remote memcg charging APIs" to "remote objcg charging APIs".
Muchun Song (5):
mm: memcontrol: introduce obj_cgroup_{un}charge_pages
mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c
mm: memcontrol: change ug->dummy_page only if memcg changed
mm: memcontrol: use obj_cgroup APIs to charge kmem pages
mm: memcontrol: move PageMemcgKmem to the scope of CONFIG_MEMCG_KMEM
include/linux/memcontrol.h | 123 ++++++++++++++++++++++++++++++++--------- mm/memcontrol.c | 133 ++++++++++++++++++++++++++++++--------------- mm/page_alloc.c | 4 +- 3 files changed, 188 insertions(+), 72 deletions(-)
Signature · 1 line
2.11.0
PATCH v4 1/5 mm: memcontrol: introduce obj_cgroup_{un}charge_pages
We know that the unit of slab object charging is bytes, the unit of kmem page charging is PAGE_SIZE. If we want to reuse obj_cgroup APIs to charge the kmem pages, we should pass PAGE_SIZE (as third parameter) to obj_cgroup_charge(). Because the size is already PAGE_SIZE, we can skip touch the objcg stock. And obj_cgroup_{un}charge_pages() are introduced to charge in units of page level.
In the later patch, we also can reuse those two helpers to charge or uncharge a number of kernel pages to a object cgroup. This is just a code movement without any functional changes.
--- mm/memcontrol.c | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-)
mm/memcontrol.c +31 -15
@@ -3056,6 +3056,34 @@ static void memcg_free_cache_id(int id)ida_simple_remove (&memcg_cache_ida, id); } +static inline void obj_cgroup_uncharge_pages (struct obj_cgroup *objcg, +unsigned int nr_pages) +{ +rcu_read_lock (); +__memcg_kmem_uncharge (obj_cgroup_memcg (objcg), nr_pages); +rcu_read_unlock (); +} + +static int obj_cgroup_charge_pages (struct obj_cgroup *objcg,gfp_t gfp, +unsigned int nr_pages) +{ +struct mem_cgroup *memcg; +int ret; + +rcu_read_lock (); +retry: + memcg =obj_cgroup_memcg (objcg); +if (unlikely (!css_tryget (&memcg->css))) + goto retry; +rcu_read_unlock (); + + ret =__memcg_kmem_charge (memcg, gfp, nr_pages); + +css_put (&memcg->css); + +return ret; +} +/** * __memcg_kmem_charge: charge a number of kernel pages to a memcg * @memcg: memory cgroup to charge
@@ -3180,11 +3208,8 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock)unsigned int nr_pages = stock->nr_bytes >>PAGE_SHIFT ;unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE -1 ); -if (nr_pages) { -rcu_read_lock (); -__memcg_kmem_uncharge (obj_cgroup_memcg (old), nr_pages); -rcu_read_unlock (); - } +if (nr_pages) +obj_cgroup_uncharge_pages (old, nr_pages);/* * The leftover is flushed to the centralized per-memcg value.
@@ -3242,7 +3267,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)int obj_cgroup_charge (struct obj_cgroup *objcg,gfp_t gfp,size_t size) { -struct mem_cgroup *memcg;unsigned int nr_pages, nr_bytes;int ret;
@@ -3259,24 +3283,16 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size)* refill_obj_stock(), called from this function or * independently later. */ -rcu_read_lock (); -retry: - memcg =obj_cgroup_memcg (objcg); -if (unlikely (!css_tryget (&memcg->css))) - goto retry; -rcu_read_unlock (); - nr_pages = size >>PAGE_SHIFT ; nr_bytes = size & (PAGE_SIZE -1 );if (nr_bytes) nr_pages +=1 ; - ret =__memcg_kmem_charge (memcg, gfp, nr_pages); + ret =obj_cgroup_charge_pages (objcg, gfp, nr_pages);if (!ret && nr_bytes)refill_obj_stock (objcg,PAGE_SIZE - nr_bytes); -css_put (&memcg->css);return ret; }
Signature · 1 line
2.11.0
PATCH v4 2/5 mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c
In reply to Muchun SongThe page_memcg() is not suitable for use by page_expected_state() and page_bad_reason(). Because it can BUG_ON() for the slab pages when CONFIG_DEBUG_VM is enabled. As neither lru, nor kmem, nor slab page should have anything left in there by the time the page is freed, what we care about is whether the value of page->memcg_data is 0. So just directly access page->memcg_data here.
--- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c +2 -2
@@ -1124,7 +1124,7 @@ static inline bool page_expected_state(struct page *page,if (unlikely ((unsigned long )page->mapping |page_ref_count (page) | #ifdef CONFIG_MEMCG - (unsigned long)page_memcg(page) | + page->memcg_data | #endif (page->flags & check_flags)))return false;
@@ -1149,7 +1149,7 @@ static const char *page_bad_reason(struct page *page, unsigned long flags) bad_reason ="PAGE_FLAGS_CHECK_AT_FREE flag(s) set" ; }#ifdef CONFIG_MEMCG -if (unlikely (page_memcg (page))) +if (unlikely (page->memcg_data)) bad_reason ="page still charged to cgroup" ;#endif return bad_reason;
Signature · 1 line
2.11.0
Re: [PATCH v4 2/5] mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c
Quoted 8 lines · Muchun Song
The
page_memcg()is not suitable for use bypage_expected_state()andpage_bad_reason(). Because it canBUG_ON()for the slab pages whenCONFIG_DEBUG_VMis enabled. As neither lru, nor kmem, nor slab page should have anything left in there by the time the page is freed, what we care about is whether the value ofpage->memcg_datais 0. So just directly accesspage->memcg_datahere.Signed-off-by Muchun Song <[email protected]>
Re: [PATCH v4 2/5] mm: memcontrol: directly access page->memcg_data in mm/page_alloc.c
In reply to Muchun SongQuoted 9 lines · Muchun Song
The
page_memcg()is not suitable for use bypage_expected_state()andpage_bad_reason(). Because it canBUG_ON()for the slab pages whenCONFIG_DEBUG_VMis enabled. As neither lru, nor kmem, nor slab page should have anything left in there by the time the page is freed, what we care about is whether the value ofpage->memcg_datais 0. So just directly accesspage->memcg_datahere.Signed-off-by Muchun Song <[email protected]>
PATCH v4 3/5 mm: memcontrol: change ug->dummy_page only if memcg changed
In reply to Muchun SongJust like assignment to ug->memcg, we only need to update ug->dummy_page if memcg changed. So move it to there. This is a very small optimization.
--- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/memcontrol.c +1 -1
@@ -6845,6 +6845,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)uncharge_gather_clear (ug); } ug->memcg =page_memcg (page); + ug->dummy_page = page;/* pairs with css_put in uncharge_batch */ css_get (&ug->memcg->css);
@@ -6858,7 +6859,6 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug)else ug->pgpgout++; - ug->dummy_page = page; page->memcg_data =0 ;css_put (&ug->memcg->css); }
Signature · 1 line
2.11.0
Re: [PATCH v4 3/5] mm: memcontrol: change ug->dummy_page only if memcg changed
Quoted 4 lines · Muchun Song
Just like assignment to
ug->memcg, we only need to updateug->dummy_pageif memcg changed. So move it to there. This is a very small optimization.Signed-off-by Muchun Song <[email protected]>
Re: [PATCH v4 3/5] mm: memcontrol: change ug->dummy_page only if memcg changed
In reply to Muchun SongQuoted 5 lines · Muchun Song
Just like assignment to
ug->memcg, we only need to updateug->dummy_pageif memcg changed. So move it to there. This is a very small optimization.Signed-off-by Muchun Song <[email protected]>
PATCH v4 4/5 mm: memcontrol: use obj_cgroup APIs to charge kmem pages
In reply to Muchun SongSince Roman series "The new cgroup slab memory controller" applied. All slab objects are charged via the new APIs of obj_cgroup. The new APIs introduce a struct obj_cgroup to charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory. But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged via the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup.
We want to reuse the obj_cgroup APIs to charge the kmem pages. If we do that, we should store an object cgroup pointer to page->memcg_data for the kmem pages.
Finally, page->memcg_data will have 3 different meanings.
1) For the slab pages, page->memcg_data points to an object cgroups vector.
2) For the kmem pages (exclude the slab pages), page->memcg_data points to an object cgroup.
3) For the user pages (e.g. the LRU pages), page->memcg_data points to a memory cgroup.
We do not change the behavior of page_memcg() and page_memcg_rcu(). They are also suitable for LRU pages and kmem pages. Why?
Because memory allocations pinning memcgs for a long time - it exists at a larger scale and is causing recurring problems in the real world:
page cache doesn't get reclaimed for a long time, or is used by the second, third, fourth, ... instance of the same job that was restarted into a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, and make page reclaim very inefficient.
We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. At that time, LRU pages and kmem pages will be treated the same. The implementation of page_memcg() will remove the kmem page check.
This patch aims to charge the kmem pages by using the new APIs of obj_cgroup. Finally, the page->memcg_data of the kmem page points to an object cgroup. We can use the __page_objcg() to get the object cgroup associated with a kmem page. Or we can use page_memcg() to get the memory cgroup associated with a kmem page, but caller must ensure that the returned memcg won't be released (e.g. acquire the rcu_read_lock or css_set_lock).
--- include/linux/memcontrol.h | 116 +++++++++++++++++++++++++++++++++++---------- mm/memcontrol.c | 101 ++++++++++++++++++++++++--------------- 2 files changed, 156 insertions(+), 61 deletions(-)
include/linux/memcontrol.h +92 -24
@@ -358,6 +358,62 @@ enum page_memcg_data_flags {#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +static inline bool PageMemcgKmem (struct page *page); + +/* +* After the initialization objcg->memcg is always pointing at +* a valid memcg, but can be atomically swapped to the parent memcg. +* +* The caller must ensure that the returned memcg won't be released: +* e.g. acquire the rcu_read_lock or css_set_lock. +*/ +static inline struct mem_cgroup *obj_cgroup_memcg (struct obj_cgroup *objcg) +{ +return READ_ONCE (objcg->memcg); +} + +/* +* __page_memcg - get the memory cgroup associated with a non-kmem page +* @page: a pointer to the page struct +* +* Returns a pointer to the memory cgroup associated with the page, +* or NULL. This function assumes that the page is known to have a +* proper memory cgroup pointer. It's not safe to call this function +* against some type of pages, e.g. slab pages or ex-slab pages or +* kmem pages. +*/ +static inline struct mem_cgroup *__page_memcg (struct page *page) +{ +unsigned long memcg_data = page->memcg_data; + +VM_BUG_ON_PAGE (PageSlab (page), page); +VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_OBJCGS , page); +VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_KMEM , page); + +return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +} + +/* +* __page_objcg - get the object cgroup associated with a kmem page +* @page: a pointer to the page struct +* +* Returns a pointer to the object cgroup associated with the page, +* or NULL. This function assumes that the page is known to have a +* proper object cgroup pointer. It's not safe to call this function +* against some type of pages, e.g. slab pages or ex-slab pages or +* LRU pages. +*/ +static inline struct obj_cgroup *__page_objcg (struct page *page) +{ +unsigned long memcg_data = page->memcg_data; + +VM_BUG_ON_PAGE (PageSlab (page), page); +VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_OBJCGS , page); +VM_BUG_ON_PAGE (!(memcg_data &MEMCG_DATA_KMEM ), page); + +return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +} +/* * page_memcg - get the memory cgroup associated with a page * @page: a pointer to the page struct
@@ -367,20 +423,23 @@ enum page_memcg_data_flags {* proper memory cgroup pointer. It's not safe to call this function * against some type of pages, e.g. slab pages or ex-slab pages. * -* Any of the following ensures page and memcg binding stability: +* For a non-kmem page any of the following ensures page and memcg binding +* stability: +* * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference +* +* For a kmem page a caller should hold an rcu read lock to protect memcg +* associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg (struct page *page) { -unsigned long memcg_data = page->memcg_data; - -VM_BUG_ON_PAGE (PageSlab (page), page); -VM_BUG_ON_PAGE (memcg_data &MEMCG_DATA_OBJCGS , page); - -return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +if (PageMemcgKmem (page)) +return obj_cgroup_memcg (__page_objcg (page)); +else +return __page_memcg (page); }/*
@@ -394,11 +453,19 @@ static inline struct mem_cgroup *page_memcg(struct page *page)*/ static inline struct mem_cgroup *page_memcg_rcu (struct page *page) { +unsigned long memcg_data =READ_ONCE (page->memcg_data); +VM_BUG_ON_PAGE (PageSlab (page), page);WARN_ON_ONCE (!rcu_read_lock_held ()); -return (struct mem_cgroup *)(READ_ONCE (page->memcg_data) & - ~MEMCG_DATA_FLAGS_MASK ); +if (memcg_data &MEMCG_DATA_KMEM ) { +struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return obj_cgroup_memcg (objcg); + } + +return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); }/*
@@ -406,15 +473,21 @@ static inline struct mem_cgroup *page_memcg_rcu(struct page *page)* @page: a pointer to the page struct * * Returns a pointer to the memory cgroup associated with the page, -* or NULL. This function unlike page_memcg() can take any page +* or NULL. This function unlike page_memcg() can take any page * as an argument. It has to be used in cases when it's not known if a page -* has an associated memory cgroup pointer or an object cgroups vector. +* has an associated memory cgroup pointer or an object cgroups vector or +* an object cgroup. +* +* For a non-kmem page any of the following ensures page and memcg binding +* stability: * -* Any of the following ensures page and memcg binding stability: * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference +* +* For a kmem page a caller should hold an rcu read lock to protect memcg +* associated with a kmem page from being released. */ static inline struct mem_cgroup *page_memcg_check (struct page *page) {
@@ -427,6 +500,13 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page)if (memcg_data &MEMCG_DATA_OBJCGS )return NULL ; +if (memcg_data &MEMCG_DATA_KMEM ) { +struct obj_cgroup *objcg; + + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); +return obj_cgroup_memcg (objcg); + } +return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); }
@@ -713,18 +793,6 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg)percpu_ref_put (&objcg->refcnt); } -/* -* After the initialization objcg->memcg is always pointing at -* a valid memcg, but can be atomically swapped to the parent memcg. -* -* The caller must ensure that the returned memcg won't be released: -* e.g. acquire the rcu_read_lock or css_set_lock. -*/ -static inline struct mem_cgroup *obj_cgroup_memcg (struct obj_cgroup *objcg) -{ -return READ_ONCE (objcg->memcg); -} -static inline void mem_cgroup_put (struct mem_cgroup *memcg) {if (memcg)
mm/memcontrol.c +64 -37
201 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages
Quoted 49 lines · Muchun Song
Since Roman series "The new cgroup slab memory controller" applied. All slab objects are charged via the new APIs of
obj_cgroup. The new APIs introduce a structobj_cgroupto charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory. But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged via the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup.We want to reuse the
obj_cgroupAPIs to charge the kmem pages. If we do that, we should store an object cgroup pointer topage->memcg_datafor the kmem pages.Finally,
page->memcg_datawill have 3 different meanings.1) For the slab pages,
page->memcg_datapoints to an object cgroups vector.2) For the kmem pages (exclude the slab pages),
page->memcg_datapoints to an object cgroup.3) For the user pages (e.g. the LRU pages),
page->memcg_datapoints to a memory cgroup.We do not change the behavior of
page_memcg()andpage_memcg_rcu(). They are also suitable for LRU pages and kmem pages. Why?Because memory allocations pinning memcgs for a long time - it exists at a larger scale and is causing recurring problems in the real world:
page cache doesn't get reclaimed for a long time, or is used by the second, third, fourth, ... instance of the same job that was restarted into a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, and make page reclaim very inefficient.We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the
page->memcgwill always point to an object cgroup pointer. At that time, LRU pages and kmem pages will be treated the same. The implementation ofpage_memcg()will remove the kmem page check.This patch aims to charge the kmem pages by using the new APIs of
obj_cgroup. Finally, thepage->memcg_dataof the kmem page points to an object cgroup. We can use the__page_objcg()to get the object cgroup associated with a kmem page. Or we can usepage_memcg()to get the memory cgroup associated with a kmem page, but caller must ensure that the returned memcg won't be released (e.g. acquire thercu_read_lockorcss_set_lock).
Quoted 1 line
Signed-off-by Muchun Song <[email protected]>
Thanks for the updated version, looks good to me!
Re: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages
In reply to Muchun SongQuoted 1 line · Muchun Song
[...]
Quoted 2 lines
+static inline struct
mem_cgroup*get_obj_cgroup_memcg(structobj_cgroup*objcg)
I would prefer get_mem_cgroup_from_objcg().
Quoted 19 lines
mm/memcontrol.c +13
-0+{ +struct mem_cgroup *memcg; + +rcu_read_lock (); +retry: + memcg =obj_cgroup_memcg (objcg); +if (unlikely (!css_tryget (&memcg->css))) + goto retry; +rcu_read_unlock (); + +return memcg; +} +#ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page)@@ -3070,15 +3088,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,struct mem_cgroup *memcg;int ret;
mm/memcontrol.c static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · line 3092
Quoted 10 lines
mm/memcontrol.c +1
-7-rcu_read_lock (); -retry: - memcg =obj_cgroup_memcg (objcg); -if (unlikely (!css_tryget (&memcg->css))) - goto retry; -rcu_read_unlock (); - + memcg =get_obj_cgroup_memcg (objcg); ret =__memcg_kmem_charge (memcg, gfp, nr_pages);
Why not manually inline __memcg_kmem_charge() here? This is the only user.
Similarly manually inline __memcg_kmem_uncharge() into obj_cgroup_uncharge_pages() and call obj_cgroup_uncharge_pages() in obj_cgroup_release().
mm/memcontrol.c static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page · line 3149
Quoted 13 lines
mm/memcontrol.c +1
-3-css_put (&memcg->css);return ret;@@ -3143,18 +3154,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page*/ int __memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order) { -struct mem_cgroup *memcg; +struct obj_cgroup *objcg;int ret =0 ; - memcg =get_mem_cgroup_from_current ();
This was the only use of get_mem_cgroup_from_current(). Why not remove it?
Quoted 16 lines
mm/memcontrol.c +5
-4-if (memcg && !mem_cgroup_is_root (memcg)) { - ret =__memcg_kmem_charge (memcg, gfp,1 << order); + objcg =get_obj_cgroup_from_current (); +if (objcg) { + ret =obj_cgroup_charge_pages (objcg, gfp,1 << order);if (!ret) { - page->memcg_data = (unsigned long )memcg | + page->memcg_data = (unsigned long )objcg |MEMCG_DATA_KMEM ;return 0 ; } -css_put (&memcg->css); +obj_cgroup_put (objcg); }return ret; }
[...]
Quoted 10 lines
static void
uncharge_page(struct page *page, struct uncharge_gather *ug){ unsigned long nr_pages;mm/memcontrol.c +2
-2+struct mem_cgroup *memcg; +struct obj_cgroup *objcg;VM_BUG_ON_PAGE (PageLRU (page), page); -if (!page_memcg (page)) -return ;
mm/memcontrol.c static void uncharge_batch(const struct uncharge_gather *ug) · line 6853
Quoted 10 lines
mm/memcontrol.c +4
-2-/* * Nobody should be changing or seriously looking at -* page_memcg(page) at this point, we have fully +* page memcg or objcg at this point, we have fully * exclusive access to the page. */ +if (PageMemcgKmem (page)) { + objcg =__page_objcg (page); + memcg =get_obj_cgroup_memcg (objcg);
Can you add a comment that this get matches the put at the end of the function and kmem pages do not hold memcg references anymore.
Quoted 51 lines
mm/memcontrol.c +22
-9+ }else { + memcg =__page_memcg (page); + } + +if (!memcg) +return ; -if (ug->memcg !=page_memcg (page)) { +if (ug->memcg != memcg) {if (ug->memcg) {uncharge_batch (ug);uncharge_gather_clear (ug); } - ug->memcg =page_memcg (page); + ug->memcg = memcg; ug->dummy_page = page;/* pairs with css_put in uncharge_batch */ -css_get (&ug->memcg->css); +css_get (&memcg->css); } nr_pages =compound_nr (page); - ug->nr_pages += nr_pages; -if (PageMemcgKmem (page)) +if (PageMemcgKmem (page)) { + ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; -else + + page->memcg_data =0 ; +obj_cgroup_put (objcg); + }else { +/* LRU pages aren't accounted at the root level */ +if (!mem_cgroup_is_root (memcg)) + ug->nr_memory += nr_pages; ug->pgpgout++; - page->memcg_data =0 ; -css_put (&ug->memcg->css); + page->memcg_data =0 ; + } + +css_put (&memcg->css); }/** -- 2.11.0
Re: [External] Re: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages
Quoted 8 lines · Shakeel Butt
Quoted 1 line · Muchun Song
[...]
Quoted 2 lines
+static inline struct
mem_cgroup*get_obj_cgroup_memcg(structobj_cgroup*objcg)I would prefer
get_mem_cgroup_from_objcg().
Inspired by obj_cgroup_memcg() which returns the memcg from objcg. So I introduce get_obj_cgroup_memcg() which obtains a reference of memcg on the basis of obj_cgroup_memcg().
So that the names are more consistent. Just my thought.
So should I rename it to get_mem_cgroup_from_objcg?
Quoted 33 lines
Quoted 20 lines
mm/memcontrol.c +13
-0+{ +struct mem_cgroup *memcg; + +rcu_read_lock (); +retry: + memcg =obj_cgroup_memcg (objcg); +if (unlikely (!css_tryget (&memcg->css))) + goto retry; +rcu_read_unlock (); + +return memcg; +} +#ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page)@@ -3070,15 +3088,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,struct mem_cgroup *memcg;int ret;mm/memcontrol.c static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · line 3092
Quoted 10 lines
mm/memcontrol.c +1
-7-rcu_read_lock (); -retry: - memcg =obj_cgroup_memcg (objcg); -if (unlikely (!css_tryget (&memcg->css))) - goto retry; -rcu_read_unlock (); - + memcg =get_obj_cgroup_memcg (objcg); ret =__memcg_kmem_charge (memcg, gfp, nr_pages);Why not manually inline
__memcg_kmem_charge()here? This is the only user.
Quoted 3 lines
Similarly manually inline
__memcg_kmem_uncharge()intoobj_cgroup_uncharge_pages()and callobj_cgroup_uncharge_pages()inobj_cgroup_release().
Good point. I will do this.
Quoted 13 lines
Quoted 13 lines
mm/memcontrol.c +1
-2-css_put (&memcg->css);return ret;@@ -3143,18 +3154,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page*/ int __memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order) { -struct mem_cgroup *memcg; +struct obj_cgroup *objcg;int ret =0 ;
Quoted 3 lines
mm/memcontrol.c static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page · line 3149
Quoted 1 line
mm/memcontrol.c +0
-1- memcg =get_mem_cgroup_from_current ();This was the only use of
get_mem_cgroup_from_current(). Why not remove it?
I saw a potential user.
[PATCH v10 0/3] Charge loop device i/o to issuing cgroup
To avoid reintroducing them. So I did not remove it.
Quoted 39 lines
Quoted 17 lines
mm/memcontrol.c +5
-4-if (memcg && !mem_cgroup_is_root (memcg)) { - ret =__memcg_kmem_charge (memcg, gfp,1 << order); + objcg =get_obj_cgroup_from_current (); +if (objcg) { + ret =obj_cgroup_charge_pages (objcg, gfp,1 << order);if (!ret) { - page->memcg_data = (unsigned long )memcg | + page->memcg_data = (unsigned long )objcg |MEMCG_DATA_KMEM ;return 0 ; } -css_put (&memcg->css); +obj_cgroup_put (objcg); }return ret; }[...]
Quoted 10 lines
static void
uncharge_page(struct page *page, struct uncharge_gather *ug){ unsigned long nr_pages;mm/memcontrol.c +2
-2+struct mem_cgroup *memcg; +struct obj_cgroup *objcg;VM_BUG_ON_PAGE (PageLRU (page), page); -if (!page_memcg (page)) -return ;mm/memcontrol.c static void uncharge_batch(const struct uncharge_gather *ug) · line 6853
Quoted 10 lines
mm/memcontrol.c +4
-2-/* * Nobody should be changing or seriously looking at -* page_memcg(page) at this point, we have fully +* page memcg or objcg at this point, we have fully * exclusive access to the page. */ +if (PageMemcgKmem (page)) { + objcg =__page_objcg (page); + memcg =get_obj_cgroup_memcg (objcg);
Quoted 2 lines
Can you add a comment that this get matches the put at the end of the function and kmem pages do not hold memcg references anymore.
Make sense. Will do.
Thanks for your suggestions.
Quoted 52 lines
Quoted 52 lines
mm/memcontrol.c +22
-9+ }else { + memcg =__page_memcg (page); + } + +if (!memcg) +return ; -if (ug->memcg !=page_memcg (page)) { +if (ug->memcg != memcg) {if (ug->memcg) {uncharge_batch (ug);uncharge_gather_clear (ug); } - ug->memcg =page_memcg (page); + ug->memcg = memcg; ug->dummy_page = page;/* pairs with css_put in uncharge_batch */ -css_get (&ug->memcg->css); +css_get (&memcg->css); } nr_pages =compound_nr (page); - ug->nr_pages += nr_pages; -if (PageMemcgKmem (page)) +if (PageMemcgKmem (page)) { + ug->nr_memory += nr_pages; ug->nr_kmem += nr_pages; -else + + page->memcg_data =0 ; +obj_cgroup_put (objcg); + }else { +/* LRU pages aren't accounted at the root level */ +if (!mem_cgroup_is_root (memcg)) + ug->nr_memory += nr_pages; ug->pgpgout++; - page->memcg_data =0 ; -css_put (&ug->memcg->css); + page->memcg_data =0 ; + } + +css_put (&memcg->css); }/** -- 2.11.0
Re: [External] Re: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages
Quoted 11 lines · Muchun Song
Quoted 8 lines · Shakeel Butt
Quoted 1 line · Muchun Song
[...]
Quoted 2 lines
+static inline struct
mem_cgroup*get_obj_cgroup_memcg(structobj_cgroup*objcg)I would prefer
get_mem_cgroup_from_objcg().
Quoted 8 lines
Inspired by
obj_cgroup_memcg()which returns the memcg from objcg. So I introduceget_obj_cgroup_memcg()which obtains a reference of memcg on the basis ofobj_cgroup_memcg().So that the names are more consistent. Just my thought.
So should I rename it to
get_mem_cgroup_from_objcg?
If you look at other functions which get reference on mem_cgroup, they have the format of get_mem_cgroup_*. Similarly the current function to get a reference on obj_cgroup is get_obj_cgroup_from_current().
So, from the name get_obj_cgroup_memcg(), it seems like we are getting reference on obj_cgroup but the function is getting reference on mem_cgroup.
Quoted 56 lines
Quoted 33 lines
Quoted 20 lines
mm/memcontrol.c +13
-0+{ +struct mem_cgroup *memcg; + +rcu_read_lock (); +retry: + memcg =obj_cgroup_memcg (objcg); +if (unlikely (!css_tryget (&memcg->css))) + goto retry; +rcu_read_unlock (); + +return memcg; +} +#ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page)@@ -3070,15 +3088,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,struct mem_cgroup *memcg;int ret;mm/memcontrol.c static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · line 3092
Quoted 10 lines
mm/memcontrol.c +1
-7-rcu_read_lock (); -retry: - memcg =obj_cgroup_memcg (objcg); -if (unlikely (!css_tryget (&memcg->css))) - goto retry; -rcu_read_unlock (); - + memcg =get_obj_cgroup_memcg (objcg); ret =__memcg_kmem_charge (memcg, gfp, nr_pages);Why not manually inline
__memcg_kmem_charge()here? This is the only user.Quoted 3 lines
Similarly manually inline
__memcg_kmem_uncharge()intoobj_cgroup_uncharge_pages()and callobj_cgroup_uncharge_pages()inobj_cgroup_release().Good point. I will do this.
Quoted 13 lines
Quoted 13 lines
mm/memcontrol.c +1
-2-css_put (&memcg->css);return ret;@@ -3143,18 +3154,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page*/ int __memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order) { -struct mem_cgroup *memcg; +struct obj_cgroup *objcg;int ret =0 ;Quoted 3 lines
mm/memcontrol.c static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page · line 3149
Quoted 1 line
mm/memcontrol.c +0
-1- memcg =get_mem_cgroup_from_current ();This was the only use of
get_mem_cgroup_from_current(). Why not remove it?
Quoted 6 lines
I saw a potential user.
[PATCH v10 0/3] Charge loop device i/o to issuing cgroup
To avoid reintroducing them. So I did not remove it.
Don't worry about that. Most probably that user would be changing this function, so it would to better to introduce from scratch.
Re: [External] Re: [PATCH v4 4/5] mm: memcontrol: use obj_cgroup APIs to charge kmem pages
Quoted 22 lines · Shakeel Butt
Quoted 11 lines · Muchun Song
Quoted 8 lines · Shakeel Butt
Quoted 1 line · Muchun Song
[...]
Quoted 2 lines
+static inline struct
mem_cgroup*get_obj_cgroup_memcg(structobj_cgroup*objcg)I would prefer
get_mem_cgroup_from_objcg().Quoted 8 lines
Inspired by
obj_cgroup_memcg()which returns the memcg from objcg. So I introduceget_obj_cgroup_memcg()which obtains a reference of memcg on the basis ofobj_cgroup_memcg().So that the names are more consistent. Just my thought.
So should I rename it to
get_mem_cgroup_from_objcg?
Quoted 7 lines
If you look at other functions which get reference on
mem_cgroup, they have the format of get_mem_cgroup_*. Similarly the current function to get a reference onobj_cgroupisget_obj_cgroup_from_current().So, from the name
get_obj_cgroup_memcg(), it seems like we are getting reference onobj_cgroupbut the function is getting reference onmem_cgroup.
Make sense. I will use get_mem_cgroup_from_objcg(). Thanks.
Quoted 59 lines
Quoted 59 lines
Quoted 34 lines
Quoted 21 lines
mm/memcontrol.c +13
-0+{ +struct mem_cgroup *memcg; + +rcu_read_lock (); +retry: + memcg =obj_cgroup_memcg (objcg); +if (unlikely (!css_tryget (&memcg->css))) + goto retry; +rcu_read_unlock (); + +return memcg; +} +#ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp,bool new_page)@@ -3070,15 +3088,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,struct mem_cgroup *memcg;int ret;mm/memcontrol.c static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · line 3092
Quoted 10 lines
mm/memcontrol.c +1
-7-rcu_read_lock (); -retry: - memcg =obj_cgroup_memcg (objcg); -if (unlikely (!css_tryget (&memcg->css))) - goto retry; -rcu_read_unlock (); - + memcg =get_obj_cgroup_memcg (objcg); ret =__memcg_kmem_charge (memcg, gfp, nr_pages);Why not manually inline
__memcg_kmem_charge()here? This is the only user.Quoted 3 lines
Similarly manually inline
__memcg_kmem_uncharge()intoobj_cgroup_uncharge_pages()and callobj_cgroup_uncharge_pages()inobj_cgroup_release().Good point. I will do this.
Quoted 13 lines
Quoted 13 lines
mm/memcontrol.c +1
-2-css_put (&memcg->css);return ret;@@ -3143,18 +3154,18 @@ static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page*/ int __memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order) { -struct mem_cgroup *memcg; +struct obj_cgroup *objcg;int ret =0 ;Quoted 3 lines
mm/memcontrol.c static void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_page · line 3149
Quoted 1 line
mm/memcontrol.c +0
-1- memcg =get_mem_cgroup_from_current ();This was the only use of
get_mem_cgroup_from_current(). Why not remove it?I saw a potential user.
Quoted 7 lines
Quoted 4 lines
[PATCH v10 0/3] Charge loop device i/o to issuing cgroup
To avoid reintroducing them. So I did not remove it.
Don't worry about that. Most probably that user would be changing this function, so it would to better to introduce from scratch.
OK. I will remove get_mem_cgroup_from_current(). Thanks for your suggestions.
PATCH v4 5/5 mm: memcontrol: move PageMemcgKmem to the scope of CONFIG_MEMCG_KMEM
In reply to Muchun SongThe page only can be marked as kmem when CONFIG_MEMCG_KMEM is enabled. So move PageMemcgKmem() to the scope of the CONFIG_MEMCG_KMEM.
As a bonus, on !CONFIG_MEMCG_KMEM build some code can be compiled out.
--- include/linux/memcontrol.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
include/linux/memcontrol.h +6 -1
@@ -510,6 +510,7 @@ static inline struct mem_cgroup *page_memcg_check(struct page *page)return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK ); } +#ifdef CONFIG_MEMCG_KMEM /* * PageMemcgKmem - check if the page has MemcgKmem flag set * @page: a pointer to the page struct
@@ -524,7 +525,6 @@ static inline bool PageMemcgKmem(struct page *page)return page->memcg_data &MEMCG_DATA_KMEM ; } -#ifdef CONFIG_MEMCG_KMEM /* * page_objcgs - get the object cgroups vector associated with a page * @page: a pointer to the page struct
@@ -566,6 +566,11 @@ static inline struct obj_cgroup **page_objcgs_check(struct page *page) }#else +static inline bool PageMemcgKmem (struct page *page) +{ +return false; +} +static inline struct obj_cgroup **page_objcgs (struct page *page) {return NULL ;
Signature · 1 line
2.11.0