The new slab memory controller
Subsystems (14)
cover The new slab memory controller
The existing slab memory controller is based on the idea of replicating slab allocator internals for each memory cgroup. This approach promises a low memory overhead (one pointer per page), and isn't adding too much code on hot allocation and release paths. But is has a very serious flaw:
it leads to a low slab utilization.
Using a drgn* script I've got an estimation of slab utilization on a number of machines running different production workloads. In most cases it was between 45% and 65%, and the best number I've seen was around 85%. Turning kmem accounting off brings it to high 90s. Also it brings back 30-50% of slab memory. It means that the real price of the existing slab memory controller is way bigger than a pointer per page.
The real reason why the existing design leads to a low slab utilization is simple: slab pages are used exclusively by one memory cgroup. If there are only few allocations of certain size made by a cgroup, or if some active objects (e.g. dentries) are left after the cgroup is deleted, or the cgroup contains a single-threaded application which is barely allocating any kernel objects, but does it every time on a new CPU:
in all these cases the resulting slab utilization is very low. If kmem accounting is off, the kernel is able to use free space on slab pages for other allocations.
Arguably it wasn't an issue back to days when the kmem controller was introduced and was an opt-in feature, which had to be turned on individually for each memory cgroup. But now it's turned on by default on both cgroup v1 and v2. And modern systemd-based systems tend to create a large number of cgroups.
This patchset provides a new implementation of the slab memory controller, which aims to reach a much better slab utilization by sharing slab pages between multiple memory cgroups. Below is the short description of the new design (more details in commit messages).
Accounting is performed per-object instead of per-page. Slab-related vmstat counters are converted to bytes. Charging is performed on page-basis, with rounding up and remembering leftovers.
Memcg ownership data is stored in a per-slab-page vector: for each slab page a vector of corresponding size is allocated. To keep slab memory reparenting working, instead of saving a pointer to the memory cgroup directly an intermediate object is used. It's simply a pointer to a memcg (which can be easily changed to the parent) with a built-in reference counter. This scheme allows to reparent all allocated objects without walking them over and changing memcg pointer to the parent.
Instead of creating an individual set of kmem_caches for each memory cgroup, two global sets are used: the root set for non-accounted and root-cgroup allocations and the second set for all other allocations. This allows to simplify the lifetime management of individual kmem_caches: they are destroyed with root counterparts. It allows to remove a good amount of code and make things generally simpler.
The patchset contains a couple of semi-independent parts, which can find their usage outside of the slab memory controller too:
1) subpage charging API, which can be used in the future for accounting of other non-page-sized objects, e.g. percpu allocations.
2) mem_cgroup_ptr API (refcounted pointers to a memcg, can be reused for the efficient reparenting of other objects, e.g. pagecache.
The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memory
So far I haven't found any regression on all tested workloads, but potential CPU regression caused by more precise accounting is a concern.
Obviously the amount of saved memory depend on the number of memory cgroups, uptime and specific workloads, but overall it feels like the new controller saves 30-40% of slab memory, sometimes more. Additionally, it should lead to a lower memory fragmentation, just because of a smaller number of non-movable pages and also because there is no more need to move all slab objects to a new set of pages when a workload is restarted in a new memory cgroup.
* https://github.com/osandov/drgn
Roman Gushchin (14):
mm: memcg: subpage charging API
mm: memcg: introduce mem_cgroup_ptr
mm: vmstat: use s32 for vm_node_stat_diff in struct per_cpu_nodestat
mm: vmstat: convert slab vmstat counter to bytes
mm: memcg/slab: allocate space for memcg ownership data for non-root
slabs
mm: slub: implement SLUB version of obj_to_index()
mm: memcg/slab: save memcg ownership data for non-root slab objects
mm: memcg: move memcg_kmem_bypass() to memcontrol.h
mm: memcg: introduce __mod_lruvec_memcg_state()
mm: memcg/slab: charge individual slab objects instead of pages
mm: memcg: move get_mem_cgroup_from_current() to memcontrol.h
mm: memcg/slab: replace memcg_from_slab_page() with
memcg_from_slab_obj()
mm: memcg/slab: use one set of kmem_caches for all memory cgroups
mm: slab: remove redundant check in memcg_accumulate_slabinfo()
drivers/base/node.c | 11 +- fs/proc/meminfo.c | 4 +- include/linux/memcontrol.h | 102 ++++++++- include/linux/mm_types.h | 5 +- include/linux/mmzone.h | 12 +- include/linux/slab.h | 3 +- include/linux/slub_def.h | 9 + include/linux/vmstat.h | 8 + kernel/power/snapshot.c | 2 +- mm/list_lru.c | 12 +- mm/memcontrol.c | 431 +++++++++++++++++++++-------------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 +- mm/slab.c | 37 ++- mm/slab.h | 300 +++++++++++++------------ mm/slab_common.c | 449 ++++--------------------------------- mm/slob.c | 12 +- mm/slub.c | 63 ++---- mm/vmscan.c | 3 +- mm/vmstat.c | 38 +++- mm/workingset.c | 6 +- 21 files changed, 683 insertions(+), 834 deletions(-)
Signature · 1 line
2.21.0
RFC PATCH 1/14 mm: memcg: subpage charging API
Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting.
In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting.
The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock.
Please, note, that unlike the generic page charging API, a subpage object is not holding a reference to the memory cgroup. It's because a more complicated indirect scheme is required in order to implement cheap reparenting. The percpu stock holds a single reference to the cached cgroup though.
--- include/linux/memcontrol.h | 4 ++ mm/memcontrol.c | 129 +++++++++++++++++++++++++++++++++---- 2 files changed, 119 insertions(+), 14 deletions(-)
include/linux/memcontrol.h +4 -0
@@ -214,6 +214,7 @@ struct mem_cgroup {/* Accounted resources */ struct page_counter memory;struct page_counter swap; +atomic_t nr_stocked_bytes;/* Legacy consumer-oriented counters */ struct page_counter memsw;
@@ -1370,6 +1371,9 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg);void __memcg_kmem_uncharge_memcg (struct mem_cgroup *memcg,unsigned int nr_pages); +int __memcg_kmem_charge_subpage (struct mem_cgroup *memcg,size_t size, +gfp_t gfp); +void __memcg_kmem_uncharge_subpage (struct mem_cgroup *memcg,size_t size);extern struct static_key_false memcg_kmem_enabled_key;extern struct workqueue_struct *memcg_kmem_cache_wq;
mm/memcontrol.c +115 -14
226 lines not on this page · raw message
Signature · 1 line
2.21.0
Johannes Weiner
maintainerRe: [PATCH RFC 01/14] mm: memcg: subpage charging API
Quoted 10 lines · Roman Gushchin
Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting.
In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting.The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock.
Did you just implement a slab allocator for page_counter to track memory consumed by the slab allocator?
Quoted 12 lines
mm/memcontrol.c +2
-1@@ -2500,8 +2577,9 @@ void mem_cgroup_handle_over_high(void) }static int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, -unsigned int nr_pages) +unsigned int amount,bool subpage) { +unsigned int nr_pages = subpage ? ((amount >>PAGE_SHIFT ) +1 ) : amount;unsigned int batch =max (MEMCG_CHARGE_BATCH , nr_pages);int nr_retries =MEM_CGROUP_RECLAIM_RETRIES ;struct mem_cgroup *mem_over_limit;
mm/memcontrol.c static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2598
Quoted 8 lines
mm/memcontrol.c +3
-1@@ -2514,7 +2592,9 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,if (mem_cgroup_is_root (memcg))return 0 ; retry: -if (consume_stock (memcg, nr_pages)) +if (subpage &&consume_subpage_stock (memcg, amount)) +return 0 ; +else if (!subpage &&consume_stock (memcg, nr_pages))return 0 ;
The layering here isn't clean. We have an existing per-cpu cache to batch-charge the page counter. Why does the new subpage allocator not sit on *top* of this, instead of wedged in between?
I think what it should be is a try_charge_bytes() that simply gets one page from try_charge() and then does its byte tracking, regardless of how try_charge() chooses to implement its own page tracking.
That would avoid the awkward @amount + @subpage multiplexing, as well as annotating all existing callsites of try_charge() with a non-descript "false" parameter.
You can still reuse the stock data structures, use the lower bits of stock->nr_bytes for a different cgroup etc., but the charge API should really be separate.
Re: [PATCH RFC 01/14] mm: memcg: subpage charging API
Quoted 8 lines · Johannes Weiner
Quoted 7 lines · Roman Gushchin
Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting.
In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting.
Quoted 6 lines
Quoted 3 lines
The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock.
Did you just implement a slab allocator for
page_counterto track memory consumed by the slab allocator?
:)
Quoted 30 lines
Quoted 13 lines
mm/memcontrol.c +2
-1@@ -2500,8 +2577,9 @@ void mem_cgroup_handle_over_high(void) }static int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, -unsigned int nr_pages) +unsigned int amount,bool subpage) { +unsigned int nr_pages = subpage ? ((amount >>PAGE_SHIFT ) +1 ) : amount;unsigned int batch =max (MEMCG_CHARGE_BATCH , nr_pages);int nr_retries =MEM_CGROUP_RECLAIM_RETRIES ;struct mem_cgroup *mem_over_limit;mm/memcontrol.c static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2598
Quoted 8 lines
mm/memcontrol.c +3
-1@@ -2514,7 +2592,9 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,if (mem_cgroup_is_root (memcg))return 0 ; retry: -if (consume_stock (memcg, nr_pages)) +if (subpage &&consume_subpage_stock (memcg, amount)) +return 0 ; +else if (!subpage &&consume_stock (memcg, nr_pages))return 0 ;The layering here isn't clean. We have an existing per-cpu cache to batch-charge the page counter. Why does the new subpage allocator not sit on *top* of this, instead of wedged in between?
I think what it should be is a
try_charge_bytes()that simply gets one page fromtry_charge()and then does its byte tracking, regardless of howtry_charge()chooses to implement its own page tracking.
Quoted 7 lines
That would avoid the awkward @amount + @subpage multiplexing, as well as annotating all existing callsites of
try_charge()with a non-descript "false" parameter.You can still reuse the stock data structures, use the lower bits of
stock->nr_bytesfor a different cgroup etc., but the charge API should really be separate.
Hm, I kinda like the idea, however there is a complication: for the subpage accounting the css reference management is done in a different way, so that all existing code should avoid changing the css refcounter. So I'd need to pass a boolean argument anyway.
But let me try to write this down, hopefully v2 will be cleaner.
Thank you!
Johannes Weiner
maintainerRe: [PATCH RFC 01/14] mm: memcg: subpage charging API
Quoted 52 lines · Roman Gushchin
Quoted 8 lines · Johannes Weiner
Quoted 7 lines · Roman Gushchin
Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting.
In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting.Quoted 6 lines
Quoted 3 lines
The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock.
Did you just implement a slab allocator for
page_counterto track memory consumed by the slab allocator?:)
Quoted 34 lines
Quoted 13 lines
mm/memcontrol.c +2
-1@@ -2500,8 +2577,9 @@ void mem_cgroup_handle_over_high(void) }static int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, -unsigned int nr_pages) +unsigned int amount,bool subpage) { +unsigned int nr_pages = subpage ? ((amount >>PAGE_SHIFT ) +1 ) : amount;unsigned int batch =max (MEMCG_CHARGE_BATCH , nr_pages);int nr_retries =MEM_CGROUP_RECLAIM_RETRIES ;struct mem_cgroup *mem_over_limit;mm/memcontrol.c static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2598
Quoted 8 lines
mm/memcontrol.c +3
-1@@ -2514,7 +2592,9 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,if (mem_cgroup_is_root (memcg))return 0 ; retry: -if (consume_stock (memcg, nr_pages)) +if (subpage &&consume_subpage_stock (memcg, amount)) +return 0 ; +else if (!subpage &&consume_stock (memcg, nr_pages))return 0 ;The layering here isn't clean. We have an existing per-cpu cache to batch-charge the page counter. Why does the new subpage allocator not sit on *top* of this, instead of wedged in between?
I think what it should be is a
try_charge_bytes()that simply gets one page fromtry_charge()and then does its byte tracking, regardless of howtry_charge()chooses to implement its own page tracking.That would avoid the awkward @amount + @subpage multiplexing, as well as annotating all existing callsites of
try_charge()with a non-descript "false" parameter.
Quoted 8 lines
Quoted 3 lines
You can still reuse the stock data structures, use the lower bits of
stock->nr_bytesfor a different cgroup etc., but the charge API should really be separate.Hm, I kinda like the idea, however there is a complication: for the subpage accounting the css reference management is done in a different way, so that all existing code should avoid changing the css refcounter. So I'd need to pass a boolean argument anyway.
Can you elaborate on the refcounting scheme? I don't quite understand how there would be complications with that.
Generally, references are held for each page that is allocated in the page_counter. try_charge() allocates a batch of css references, returns one and keeps the rest in stock.
So couldn't the following work?
When somebody allocates a subpage, the css reference returned by try_charge() is shared by the allocated subpage object and the remainder that is kept via stock->subpage_cache and stock->nr_bytes (or memcg->nr_stocked_bytes when the percpu cache is reset).
When the subpage objects are freed, you'll eventually have a full page again in stock->nr_bytes, at which point you page_counter_uncharge() paired with css_put(_many) as per usual.
A remainder left in old->nr_stocked_bytes would continue to hold on to one css reference. (I don't quite understand who is protecting this remainder in your current version, actually. A bug?)
Instead of doing your own batched page_counter uncharging in refill_subpage_stock() -> drain_subpage_stock(), you should be able to call refill_stock() when stock->nr_bytes adds up to a whole page again.
Again, IMO this would be much cleaner architecture if there was a try_charge_bytes() byte allocator that would sit on top of a cleanly abstracted try_charge() page allocator, just like the slab allocator is sitting on top of the page allocator - instead of breaking through the abstraction layer of the underlying page allocator.
Re: [PATCH RFC 01/14] mm: memcg: subpage charging API
Quoted 69 lines · Johannes Weiner
Quoted 52 lines · Roman Gushchin
Quoted 8 lines · Johannes Weiner
Quoted 7 lines · Roman Gushchin
Introduce an API to charge subpage objects to the memory cgroup. The API will be used by the new slab memory controller. Later it can also be used to implement percpu memory accounting.
In both cases, a single page can be shared between multiple cgroups (and in percpu case a single allocation is split over multiple pages), so it's not possible to use page-based accounting.Quoted 6 lines
Quoted 3 lines
The implementation is based on percpu stocks. Memory cgroups are still charged in pages, and the residue is stored in perpcu stock, or on the memcg itself, when it's necessary to flush the stock.
Did you just implement a slab allocator for
page_counterto track memory consumed by the slab allocator?:)
Quoted 34 lines
Quoted 13 lines
mm/memcontrol.c +2
-1@@ -2500,8 +2577,9 @@ void mem_cgroup_handle_over_high(void) }static int try_charge (struct mem_cgroup *memcg,gfp_t gfp_mask, -unsigned int nr_pages) +unsigned int amount,bool subpage) { +unsigned int nr_pages = subpage ? ((amount >>PAGE_SHIFT ) +1 ) : amount;unsigned int batch =max (MEMCG_CHARGE_BATCH , nr_pages);int nr_retries =MEM_CGROUP_RECLAIM_RETRIES ;struct mem_cgroup *mem_over_limit;mm/memcontrol.c static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2598
Quoted 8 lines
mm/memcontrol.c +3
-1@@ -2514,7 +2592,9 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,if (mem_cgroup_is_root (memcg))return 0 ; retry: -if (consume_stock (memcg, nr_pages)) +if (subpage &&consume_subpage_stock (memcg, amount)) +return 0 ; +else if (!subpage &&consume_stock (memcg, nr_pages))return 0 ;The layering here isn't clean. We have an existing per-cpu cache to batch-charge the page counter. Why does the new subpage allocator not sit on *top* of this, instead of wedged in between?
I think what it should be is a
try_charge_bytes()that simply gets one page fromtry_charge()and then does its byte tracking, regardless of howtry_charge()chooses to implement its own page tracking.That would avoid the awkward @amount + @subpage multiplexing, as well as annotating all existing callsites of
try_charge()with a non-descript "false" parameter.Quoted 8 lines
Quoted 3 lines
You can still reuse the stock data structures, use the lower bits of
stock->nr_bytesfor a different cgroup etc., but the charge API should really be separate.Hm, I kinda like the idea, however there is a complication: for the subpage accounting the css reference management is done in a different way, so that all existing code should avoid changing the css refcounter. So I'd need to pass a boolean argument anyway.
Can you elaborate on the refcounting scheme? I don't quite understand how there would be complications with that.
Generally, references are held for each page that is allocated in the
page_counter.try_charge()allocates a batch of css references, returns one and keeps the rest in stock.
Quoted 6 lines
So couldn't the following work?
When somebody allocates a subpage, the css reference returned by
try_charge()is shared by the allocated subpage object and the remainder that is kept viastock->subpage_cacheandstock->nr_bytes(ormemcg->nr_stocked_byteswhen the percpu cache is reset).
Because individual objects are a subject of reparenting and can outlive the origin memory cgroup, they shouldn't hold a direct reference to the memory cgroup. Instead they hold a reference to the mem_cgroup_ptr object, and this objects holds a single reference to the memory cgroup. Underlying pages shouldn't hold a reference too.
Btw, it's already true, just kmem_cache plays the role of such intermediate object, and we do an explicit transfer of charge (look at memcg_charge_slab()). So we initially associate a page with the memcg, and almost immediately after break this association and insert kmem_cache in between.
But with subpage accounting it's not possible, as a page is shared between multiple cgroups, and it can't be attributed to any specific cgroup at any time.
Quoted 13 lines
When the subpage objects are freed, you'll eventually have a full page again in
stock->nr_bytes, at which point youpage_counter_uncharge()paired withcss_put(_many)as per usual.A remainder left in
old->nr_stocked_byteswould continue to hold on to one css reference. (I don't quite understand who is protecting this remainder in your current version, actually. A bug?)Instead of doing your own batched
page_counteruncharging inrefill_subpage_stock()->drain_subpage_stock(), you should be able to callrefill_stock()whenstock->nr_bytesadds up to a whole page again.
Quoted 6 lines
Again, IMO this would be much cleaner architecture if there was a
try_charge_bytes()byte allocator that would sit on top of a cleanly abstractedtry_charge()page allocator, just like the slab allocator is sitting on top of the page allocator - instead of breaking through the abstraction layer of the underlying page allocator.
As I said, I like the idea to put it on top, but it can't be put on top without changes in css refcounting (or I don't see how). I don't know how to mix stocks which are holding css references and which are not, so I might end up with two stocks as in current implementation. Then the idea of having another layer of caching on top looks slightly less appealing, but maybe still worth a try.
Thanks!
RFC PATCH 2/14 mm: memcg: introduce mem_cgroup_ptr
In reply to Roman GushchinThis commit introduces mem_cgroup_ptr structure and corresponding API. It implements a pointer to a memory cgroup with a built-in reference counter. The main goal of it is to implement reparenting efficiently.
If a number of objects (e.g. slab pages) have to keep a pointer and a reference to a memory cgroup, they can use mem_cgroup_ptr instead. On reparenting, only one mem_cgroup_ptr->memcg pointer has to be changed, instead of walking over all accounted objects.
mem_cgroup_ptr holds a single reference to the corresponding memory cgroup. Because it's initialized before the css reference counter, css's refcounter can't be bumped at allocation time. Instead, it's bumped on reparenting which happens during offlining. A cgroup is never released online, so it's fine.
mem_cgroup_ptr is released using rcu, so memcg->kmem_memcg_ptr can be accessed in a rcu read section. On reparenting it's atomically switched to NULL. If the reader gets NULL, it can just read parent's kmem_memcg_ptr instead.
Each memory cgroup contains a list of kmem_memcg_ptrs. On reparenting the list is spliced into the parent's list. The list is protected using the css set lock.
--- include/linux/memcontrol.h | 50 ++++++++++++++++++++++ mm/memcontrol.c | 87 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +50 -0
@@ -23,6 +23,7 @@#include <linux/page-flags.h> struct mem_cgroup ; +struct mem_cgroup_ptr ;struct page ;struct mm_struct ;struct kmem_cache ;
@@ -197,6 +198,22 @@ struct memcg_cgwb_frn {int memcg_id;/* memcg->css.id of foreign inode */ u64 at;/* jiffies_64 at the time of dirtying */ struct wb_completion done;/* tracks in-flight foreign writebacks */ +} + +/* +* A pointer to a memory cgroup with a built-in reference counter. +* For a use as an intermediate object to simplify reparenting of +* objects charged to the cgroup. The memcg pointer can be switched +* to the parent cgroup without a need to modify all objects +* which hold the reference to the cgroup. +*/ +struct mem_cgroup_ptr { +struct percpu_ref refcnt; +struct mem_cgroup *memcg; +union { +struct list_head list; +struct rcu_head rcu; + }; };/*
@@ -312,6 +329,8 @@ struct mem_cgroup {int kmemcg_id;enum memcg_kmem_state kmem_state;struct list_head kmem_caches; +struct mem_cgroup_ptr __rcu *kmem_memcg_ptr; +struct list_head kmem_memcg_ptr_list;#endif int last_scanned_node;
@@ -440,6 +459,21 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; } +static inline bool mem_cgroup_ptr_tryget (struct mem_cgroup_ptr *ptr) +{ +return percpu_ref_tryget (&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_get (struct mem_cgroup_ptr *ptr) +{ +percpu_ref_get (&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_put (struct mem_cgroup_ptr *ptr) +{ +percpu_ref_put (&ptr->refcnt); +} +static inline void mem_cgroup_put (struct mem_cgroup *memcg) {if (memcg)
@@ -1433,6 +1467,22 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return memcg ? memcg->kmemcg_id : -1; } +static inline struct mem_cgroup_ptr * +mem_cgroup_get_kmem_ptr (struct mem_cgroup *memcg) +{ +struct mem_cgroup_ptr *memcg_ptr; + +rcu_read_lock (); +do { + memcg_ptr =rcu_dereference (memcg->kmem_memcg_ptr); +if (memcg_ptr &&mem_cgroup_ptr_tryget (memcg_ptr)) +break ; + }while ((memcg =parent_mem_cgroup (memcg))); +rcu_read_unlock (); + +return memcg_ptr; +} +#else static inline int memcg_kmem_charge (struct page *page,gfp_t gfp,int order)
mm/memcontrol.c +83 -4
120 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH RFC 02/14] mm: memcg: introduce mem_cgroup_ptr
Quoted 43 lines · Roman Gushchin
This commit introduces
mem_cgroup_ptrstructure and corresponding API. It implements a pointer to a memory cgroup with a built-in reference counter. The main goal of it is to implement reparenting efficiently.If a number of objects (e.g. slab pages) have to keep a pointer and a reference to a memory cgroup, they can use
mem_cgroup_ptrinstead. On reparenting, only onemem_cgroup_ptr->memcgpointer has to be changed, instead of walking over all accounted objects.
mem_cgroup_ptrholds a single reference to the corresponding memory cgroup. Because it's initialized before the css reference counter, css's refcounter can't be bumped at allocation time. Instead, it's bumped on reparenting which happens during offlining. A cgroup is never released online, so it's fine.
mem_cgroup_ptris released using rcu, somemcg->kmem_memcg_ptrcan be accessed in a rcu read section. On reparenting it's atomically switched to NULL. If the reader gets NULL, it can just read parent'skmem_memcg_ptrinstead.Each memory cgroup contains a list of
kmem_memcg_ptrs. On reparenting the list is spliced into the parent's list. The list is protected using the css set lock.Signed-off-by Roman Gushchin <[email protected]>--- include/linux/memcontrol.h | 50 ++++++++++++++++++++++ mm/memcontrol.c | 87 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 4 deletions(-)include/linux/memcontrol.h +1
-0@@ -23,6 +23,7 @@#include <linux/page-flags.h> struct mem_cgroup ; +struct mem_cgroup_ptr ;struct page ;struct mm_struct ;struct kmem_cache ;
include/linux/memcontrol.h struct memcg_cgwb_frn { · line 201
Quoted 4 lines
include/linux/memcontrol.h +1
-0@@ -197,6 +198,22 @@ struct memcg_cgwb_frn {int memcg_id;/* memcg->css.id of foreign inode */ u64 at;/* jiffies_64 at the time of dirtying */ struct wb_completion done;/* tracks in-flight foreign writebacks */ +}
Oops, a semicolon has been lost during the final rebase. I'll send a correct version of this patch separately.
Sorry for the mess.
RFC PATCH 2/14 mm: memcg: introduce mem_cgroup_ptr
In reply to Roman GushchinThis commit introduces mem_cgroup_ptr structure and corresponding API. It implements a pointer to a memory cgroup with a built-in reference counter. The main goal of it is to implement reparenting efficiently.
If a number of objects (e.g. slab pages) have to keep a pointer and a reference to a memory cgroup, they can use mem_cgroup_ptr instead. On reparenting, only one mem_cgroup_ptr->memcg pointer has to be changed, instead of walking over all accounted objects.
mem_cgroup_ptr holds a single reference to the corresponding memory cgroup. Because it's initialized before the css reference counter, css's refcounter can't be bumped at allocation time. Instead, it's bumped on reparenting which happens during offlining. A cgroup is never released online, so it's fine.
mem_cgroup_ptr is released using rcu, so memcg->kmem_memcg_ptr can be accessed in a rcu read section. On reparenting it's atomically switched to NULL. If the reader gets NULL, it can just read parent's kmem_memcg_ptr instead.
Each memory cgroup contains a list of kmem_memcg_ptrs. On reparenting the list is spliced into the parent's list. The list is protected using the css set lock.
--- include/linux/memcontrol.h | 50 ++++++++++++++++++++++ mm/memcontrol.c | 87 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 133 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +50 -0
@@ -23,6 +23,7 @@#include <linux/page-flags.h> struct mem_cgroup ; +struct mem_cgroup_ptr ;struct page ;struct mm_struct ;struct kmem_cache ;
@@ -199,6 +200,22 @@ struct memcg_cgwb_frn {struct wb_completion done;/* tracks in-flight foreign writebacks */ }; +/* +* A pointer to a memory cgroup with a built-in reference counter. +* For a use as an intermediate object to simplify reparenting of +* objects charged to the cgroup. The memcg pointer can be switched +* to the parent cgroup without a need to modify all objects +* which hold the reference to the cgroup. +*/ +struct mem_cgroup_ptr { +struct percpu_ref refcnt; +struct mem_cgroup *memcg; +union { +struct list_head list; +struct rcu_head rcu; + }; +}; +/* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide
@@ -312,6 +329,8 @@ struct mem_cgroup {int kmemcg_id;enum memcg_kmem_state kmem_state;struct list_head kmem_caches; +struct mem_cgroup_ptr __rcu *kmem_memcg_ptr; +struct list_head kmem_memcg_ptr_list;#endif int last_scanned_node;
@@ -440,6 +459,21 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; } +static inline bool mem_cgroup_ptr_tryget (struct mem_cgroup_ptr *ptr) +{ +return percpu_ref_tryget (&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_get (struct mem_cgroup_ptr *ptr) +{ +percpu_ref_get (&ptr->refcnt); +} + +static inline void mem_cgroup_ptr_put (struct mem_cgroup_ptr *ptr) +{ +percpu_ref_put (&ptr->refcnt); +} +static inline void mem_cgroup_put (struct mem_cgroup *memcg) {if (memcg)
@@ -1433,6 +1467,22 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) return memcg ? memcg->kmemcg_id : -1; } +static inline struct mem_cgroup_ptr * +mem_cgroup_get_kmem_ptr (struct mem_cgroup *memcg) +{ +struct mem_cgroup_ptr *memcg_ptr; + +rcu_read_lock (); +do { + memcg_ptr =rcu_dereference (memcg->kmem_memcg_ptr); +if (memcg_ptr &&mem_cgroup_ptr_tryget (memcg_ptr)) +break ; + }while ((memcg =parent_mem_cgroup (memcg))); +rcu_read_unlock (); + +return memcg_ptr; +} +#else static inline int memcg_kmem_charge (struct page *page,gfp_t gfp,int order)
mm/memcontrol.c +83 -4
120 lines not on this page · raw message
Signature · 1 line
2.21.0
RFC PATCH 3/14 mm: vmstat: use s32 for vm_node_stat_diff in struct per_cpu_nodestat
In reply to Roman GushchinCurrently s8 type is used for per-cpu caching of per-node statistics. It works fine because the overfill threshold can't exceed 125.
But if some counters are in bytes (and the next commit in the series will convert slab counters to bytes), it's not gonna work:
value in bytes can easily exceed s8 without exceeding the threshold converted to bytes. So to avoid overfilling per-cpu caches and breaking vmstats correctness, let's use s32 instead.
This doesn't affect per-zone statistics. There are no plans to use zone-level byte-sized counters, so no reasons to change anything.
--- include/linux/mmzone.h | 2 +- mm/vmstat.c | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-)
include/linux/mmzone.h +1 -1
@@ -353,7 +353,7 @@ struct per_cpu_pageset {struct per_cpu_nodestat {s8 stat_threshold; -s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS ]; +s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS ]; };#endif /* !__GENERATING_BOUNDS.H */
mm/vmstat.c +8 -8
@@ -337,7 +337,7 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, long delta) {struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; -s8 __percpu *p = pcp->vm_node_stat_diff + item; +s32 __percpu *p = pcp->vm_node_stat_diff + item;long x;long t;
@@ -395,13 +395,13 @@ void __inc_zone_state(struct zone *zone, enum zone_stat_item item)void __inc_node_state (struct pglist_data *pgdat,enum node_stat_item item) {struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; -s8 __percpu *p = pcp->vm_node_stat_diff + item; -s8 v, t; +s32 __percpu *p = pcp->vm_node_stat_diff + item; +s32 v, t; v =__this_cpu_inc_return (*p); t =__this_cpu_read (pcp->stat_threshold);if (unlikely (v > t)) { -s8 overstep = t >>1 ; +s32 overstep = t >>1 ;node_page_state_add (v + overstep, pgdat, item);__this_cpu_write (*p, -overstep);
@@ -439,13 +439,13 @@ void __dec_zone_state(struct zone *zone, enum zone_stat_item item)void __dec_node_state (struct pglist_data *pgdat,enum node_stat_item item) {struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; -s8 __percpu *p = pcp->vm_node_stat_diff + item; -s8 v, t; +s32 __percpu *p = pcp->vm_node_stat_diff + item; +s32 v, t; v =__this_cpu_dec_return (*p); t =__this_cpu_read (pcp->stat_threshold);if (unlikely (v < - t)) { -s8 overstep = t >>1 ; +s32 overstep = t >>1 ;node_page_state_add (v - overstep, pgdat, item);__this_cpu_write (*p, overstep);
@@ -538,7 +538,7 @@ static inline void mod_node_state(struct pglist_data *pgdat, enum node_stat_item item, int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; -s8 __percpu *p = pcp->vm_node_stat_diff + item; +s32 __percpu *p = pcp->vm_node_stat_diff + item;long o, n, t, z; do {
Signature · 1 line
2.21.0
RFC PATCH 4/14 mm: vmstat: convert slab vmstat counter to bytes
In reply to Roman GushchinIn order to prepare for per-object slab memory accounting, convert NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE vmstat items to bytes.
To make sure that these vmstats are in bytes, rename them to NR_SLAB_RECLAIMABLE_B and NR_SLAB_UNRECLAIMABLE_B (similar to NR_KERNEL_STACK_KB).
The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into atomic_long_t we use for vmstats.
--- drivers/base/node.c | 11 ++++++++--- fs/proc/meminfo.c | 4 ++-- include/linux/mmzone.h | 10 ++++++++-- include/linux/vmstat.h | 8 ++++++++ kernel/power/snapshot.c | 2 +- mm/memcontrol.c | 29 ++++++++++++++++------------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 ++++---- mm/slab.h | 15 ++++++++------- mm/slab_common.c | 4 ++-- mm/slob.c | 12 ++++++------ mm/slub.c | 8 ++++---- mm/vmscan.c | 3 ++- mm/vmstat.c | 22 +++++++++++++++++++--- mm/workingset.c | 6 ++++-- 15 files changed, 93 insertions(+), 51 deletions(-)
drivers/base/node.c +8 -3
@@ -368,8 +368,8 @@ static ssize_t node_read_meminfo(struct device *dev,unsigned long sreclaimable, sunreclaimable;si_meminfo_node (&i, nid); - sreclaimable =node_page_state (pgdat,NR_SLAB_RECLAIMABLE ); - sunreclaimable =node_page_state (pgdat,NR_SLAB_UNRECLAIMABLE ); + sreclaimable =node_page_state_pages (pgdat,NR_SLAB_RECLAIMABLE_B ); + sunreclaimable =node_page_state_pages (pgdat,NR_SLAB_UNRECLAIMABLE_B ); n =sprintf (buf,"Node %d MemTotal: %8lu kB\n" "Node %d MemFree: %8lu kB\n"
@@ -495,9 +495,14 @@ static ssize_t node_read_vmstat(struct device *dev,int i;int n =0 ; -for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++) +for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++) { +unsigned long x =sum_zone_node_page_state (nid, i); + +if (vmstat_item_in_bytes (i)) + x >>=PAGE_SHIFT ; n +=sprintf (buf+n,"%s %lu\n" , vmstat_text[i],sum_zone_node_page_state (nid, i)); + }#ifdef CONFIG_NUMA for (i =0 ; i <NR_VM_NUMA_STAT_ITEMS ; i++)
fs/proc/meminfo.c +2 -2
@@ -53,8 +53,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v) pages[lru] =global_node_page_state (NR_LRU_BASE + lru); available =si_mem_available (); - sreclaimable =global_node_page_state (NR_SLAB_RECLAIMABLE ); - sunreclaim =global_node_page_state (NR_SLAB_UNRECLAIMABLE ); + sreclaimable =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ); + sunreclaim =global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B );show_val_kb (m,"MemTotal: " , i.totalram);show_val_kb (m,"MemFree: " , i.freeram);
include/linux/mmzone.h +8 -2
@@ -215,8 +215,8 @@ enum node_stat_item {NR_INACTIVE_FILE ,/* " " " " " */ NR_ACTIVE_FILE ,/* " " " " " */ NR_UNEVICTABLE ,/* " " " " " */ -NR_SLAB_RECLAIMABLE ,/* Please do not reorder this item */ -NR_SLAB_UNRECLAIMABLE ,/* and this one without looking at +NR_SLAB_RECLAIMABLE_B ,/* Please do not reorder this item */ +NR_SLAB_UNRECLAIMABLE_B ,/* and this one without looking at * memcg_flush_percpu_vmstats() first. */ NR_ISOLATED_ANON ,/* Temporary isolated pages from anon lru */ NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */
@@ -247,6 +247,12 @@ enum node_stat_item {NR_VM_NODE_STAT_ITEMS }; +static __always_inline bool vmstat_item_in_bytes(enum node_stat_item item) +{ +return (item ==NR_SLAB_RECLAIMABLE_B || + item ==NR_SLAB_UNRECLAIMABLE_B ); +} +/* * We do arithmetic on the LRU lists in various places in the code, * so it is important to keep the active lists LRU_ACTIVE higher in
include/linux/vmstat.h +8 -0
@@ -194,6 +194,12 @@ static inline unsigned long global_node_page_state(enum node_stat_item item)return x; } +static inline +unsigned long global_node_page_state_pages (enum node_stat_item item) +{ +return global_node_page_state (item) >>PAGE_SHIFT ; +} +static inline unsigned long zone_page_state (struct zone *zone,enum zone_stat_item item) {
@@ -234,6 +240,8 @@ extern unsigned long sum_zone_node_page_state(int node,extern unsigned long sum_zone_numa_state (int node,enum numa_stat_item item);extern unsigned long node_page_state (struct pglist_data *pgdat,enum node_stat_item item); +extern unsigned long node_page_state_pages (struct pglist_data *pgdat, +enum node_stat_item item);#else #define sum_zone_node_page_state (node, item) global_zone_page_state(item)#define node_page_state (node, item) global_node_page_state(item)
kernel/power/snapshot.c +1 -1
@@ -1659,7 +1659,7 @@ static unsigned long minimum_image_size(unsigned long saveable) {unsigned long size; - size =global_node_page_state (NR_SLAB_RECLAIMABLE ) + size =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ) +global_node_page_state (NR_ACTIVE_ANON ) +global_node_page_state (NR_INACTIVE_ANON ) +global_node_page_state (NR_ACTIVE_FILE )
mm/memcontrol.c +16 -13
@@ -759,13 +759,16 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)*/ void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val) { -long x; +long x, threshold =MEMCG_CHARGE_BATCH ;if (mem_cgroup_disabled ())return ; +if (vmstat_item_in_bytes (idx)) + threshold <<=PAGE_SHIFT ; + x = val +__this_cpu_read (memcg->vmstats_percpu->stat[idx]); -if (unlikely (abs (x) >MEMCG_CHARGE_BATCH )) { +if (unlikely (abs (x) > threshold)) {struct mem_cgroup *mi;/*
@@ -807,7 +810,7 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,pg_data_t *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup_per_node *pn;struct mem_cgroup *memcg; -long x; +long x, threshold =MEMCG_CHARGE_BATCH ;/* Update node */ __mod_node_page_state (pgdat, idx, val);
@@ -824,8 +827,11 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,/* Update lruvec */ __this_cpu_add (pn->lruvec_stat_local->count[idx], val); +if (vmstat_item_in_bytes (idx)) + threshold <<=PAGE_SHIFT ; + x = val +__this_cpu_read (pn->lruvec_stat_cpu->count[idx]); -if (unlikely (abs (x) >MEMCG_CHARGE_BATCH )) { +if (unlikely (abs (x) > threshold)) {struct mem_cgroup_per_node *pi;for (pi = pn; pi; pi =parent_nodeinfo (pi, pgdat->node_id))
@@ -1478,9 +1484,8 @@ static char *memory_stat_format(struct mem_cgroup *memcg) (u64 )memcg_page_state (memcg,MEMCG_KERNEL_STACK_KB ) * 1024);seq_buf_printf (&s,"slab %llu\n" , - (u64)(memcg_page_state (memcg,NR_SLAB_RECLAIMABLE ) + -memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE )) * -PAGE_SIZE ); + (u64)(memcg_page_state (memcg,NR_SLAB_RECLAIMABLE_B ) + +memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE_B )));seq_buf_printf (&s,"sock %llu\n" , (u64 )memcg_page_state (memcg,MEMCG_SOCK ) *PAGE_SIZE );
@@ -1514,11 +1519,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg) PAGE_SIZE);seq_buf_printf (&s,"slab_reclaimable %llu\n" , - (u64 )memcg_page_state (memcg,NR_SLAB_RECLAIMABLE ) * -PAGE_SIZE ); + (u64 )memcg_page_state (memcg,NR_SLAB_RECLAIMABLE_B ));seq_buf_printf (&s,"slab_unreclaimable %llu\n" , - (u64 )memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE ) * -PAGE_SIZE ); + (u64 )memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE_B ));/* Accumulated memory events */
@@ -3564,8 +3567,8 @@ static void memcg_flush_percpu_vmstats(struct mem_cgroup *memcg, bool slab_only)int min_idx, max_idx;if (slab_only) { - min_idx =NR_SLAB_RECLAIMABLE ; - max_idx =NR_SLAB_UNRECLAIMABLE ; + min_idx =NR_SLAB_RECLAIMABLE_B ; + max_idx =NR_SLAB_UNRECLAIMABLE_B ; }else { min_idx =0 ; max_idx =MEMCG_NR_STAT ;
mm/oom_kill.c +1 -1
@@ -183,7 +183,7 @@ static bool is_dump_unreclaim_slabs(void)global_node_page_state (NR_ISOLATED_FILE ) +global_node_page_state (NR_UNEVICTABLE ); -return (global_node_page_state (NR_SLAB_UNRECLAIMABLE ) > nr_lru); +return (global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B ) > nr_lru); }/**
mm/page_alloc.c +4 -4
@@ -5100,8 +5100,8 @@ long si_mem_available(void)* items that are in use, and cannot be freed. Cap this estimate at the * low watermark. */ - reclaimable =global_node_page_state (NR_SLAB_RECLAIMABLE ) + -global_node_page_state (NR_KERNEL_MISC_RECLAIMABLE ); + reclaimable =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ) + +global_node_page_state (NR_KERNEL_MISC_RECLAIMABLE ); available += reclaimable -min (reclaimable /2 , wmark_low);if (available <0 )
@@ -5245,8 +5245,8 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)global_node_page_state (NR_FILE_DIRTY ),global_node_page_state (NR_WRITEBACK ),global_node_page_state (NR_UNSTABLE_NFS ), -global_node_page_state (NR_SLAB_RECLAIMABLE ), -global_node_page_state (NR_SLAB_UNRECLAIMABLE ), +global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ), +global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B ),global_node_page_state (NR_FILE_MAPPED ),global_node_page_state (NR_SHMEM ), global_zone_page_state(NR_PAGETABLE),
mm/slab.h +8 -7
@@ -272,7 +272,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **);static inline int cache_vmstat_idx (struct kmem_cache *s) {return (s->flags &SLAB_RECLAIM_ACCOUNT ) ? -NR_SLAB_RECLAIMABLE :NR_SLAB_UNRECLAIMABLE ; +NR_SLAB_RECLAIMABLE_B :NR_SLAB_UNRECLAIMABLE_B ; }#ifdef CONFIG_MEMCG_KMEM
@@ -360,7 +360,7 @@ static __always_inline int memcg_charge_slab(struct page *page,if (unlikely (!memcg ||mem_cgroup_is_root (memcg))) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - (1 << order)); + (PAGE_SIZE << order));percpu_ref_get_many (&s->memcg_params.refcnt,1 << order);return 0 ; }
@@ -370,7 +370,7 @@ static __always_inline int memcg_charge_slab(struct page *page, goto out; lruvec =mem_cgroup_lruvec (page_pgdat (page), memcg); -mod_lruvec_state (lruvec,cache_vmstat_idx (s),1 << order); +mod_lruvec_state (lruvec,cache_vmstat_idx (s),PAGE_SIZE << order);/* transer try_charge() page references to kmem_cache */ percpu_ref_get_many (&s->memcg_params.refcnt,1 << order);
@@ -394,11 +394,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, memcg =READ_ONCE (s->memcg_params.memcg);if (likely (!mem_cgroup_is_root (memcg))) { lruvec =mem_cgroup_lruvec (page_pgdat (page), memcg); -mod_lruvec_state (lruvec,cache_vmstat_idx (s), -(1 << order)); +mod_lruvec_state (lruvec,cache_vmstat_idx (s), + -(PAGE_SIZE << order));memcg_kmem_uncharge_memcg (page, order, memcg); }else {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -(1 << order)); + -(PAGE_SIZE << order)); }rcu_read_unlock ();
@@ -482,7 +483,7 @@ static __always_inline int charge_slab_page(struct page *page, {if (is_root_cache (s)) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), -1 << order); +PAGE_SIZE << order);return 0 ; }
@@ -494,7 +495,7 @@ static __always_inline void uncharge_slab_page(struct page *page, int order, {if (is_root_cache (s)) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -(1 << order)); + -(PAGE_SIZE << order));return ; }
mm/slab_common.c +2 -2
@@ -1303,8 +1303,8 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) page =alloc_pages (flags, order);if (likely (page)) { ret =page_address (page); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order); } ret =kasan_kmalloc_large (ret, size, flags);/* As ret might get tagged, call kmemleak hook after KASAN. */
mm/slob.c +6 -6
@@ -202,8 +202,8 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)if (!page)return NULL ; -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order);return page_address (page); }
@@ -214,8 +214,8 @@ static void slob_free_pages(void *b, int order)if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order; -mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (sp, order); }
@@ -550,8 +550,8 @@ void kfree(const void *block)slob_free (m, *m + align); }else {unsigned int order =compound_order (sp); -mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (sp, order); }
mm/slub.c +4 -4
@@ -3825,8 +3825,8 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node) page =alloc_pages_node (node, flags, order);if (page) { ptr =page_address (page); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order); }return kmalloc_large_node_hook (ptr, size, flags);
@@ -3957,8 +3957,8 @@ void kfree(const void *x)BUG_ON (!PageCompound (page));kfree_hook (object); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (page, order);return ; }
mm/vmscan.c +2 -1
@@ -4272,7 +4272,8 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)* unmapped file backed pages. */ if (node_pagecache_reclaimable (pgdat) <= pgdat->min_unmapped_pages && -node_page_state (pgdat,NR_SLAB_RECLAIMABLE ) <= pgdat->min_slab_pages) +node_page_state_pages (pgdat,NR_SLAB_RECLAIMABLE_B ) <= + pgdat->min_slab_pages)return NODE_RECLAIM_FULL ;/*
mm/vmstat.c +19 -3
@@ -344,6 +344,8 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, x = delta +__this_cpu_read (*p); t =__this_cpu_read (pcp->stat_threshold); +if (vmstat_item_in_bytes (item)) + t <<=PAGE_SHIFT ;if (unlikely (x > t || x < -t)) {node_page_state_add (x, pgdat, item);
@@ -555,6 +557,8 @@ static inline void mod_node_state(struct pglist_data *pgdat,* for all cpus in a node. */ t =this_cpu_read (pcp->stat_threshold); +if (vmstat_item_in_bytes (item)) + t <<=PAGE_SHIFT ; o =this_cpu_read (*p); n = delta + o;
@@ -999,6 +1003,12 @@ unsigned long node_page_state(struct pglist_data *pgdat,#endif return x; } + +unsigned long node_page_state_pages (struct pglist_data *pgdat, +enum node_stat_item item) +{ +return node_page_state (pgdat, item) >>PAGE_SHIFT ; +}#endif #ifdef CONFIG_COMPACTION
@@ -1547,10 +1557,13 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,if (is_zone_first_populated (pgdat, zone)) {seq_printf (m,"\n per-node stats" );for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) { +unsigned long x =node_page_state (pgdat, i); + +if (vmstat_item_in_bytes (i)) + x >>=PAGE_SHIFT ;seq_printf (m,"\n %-12s %lu" , vmstat_text[i +NR_VM_ZONE_STAT_ITEMS + -NR_VM_NUMA_STAT_ITEMS ], -node_page_state (pgdat, i)); +NR_VM_NUMA_STAT_ITEMS ], x); } } seq_printf(m,
@@ -1679,8 +1692,11 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos) v +=NR_VM_NUMA_STAT_ITEMS ;#endif -for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) +for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) { v[i] =global_node_page_state (i); +if (vmstat_item_in_bytes (i)) + v[i] >>=PAGE_SHIFT ; + } v +=NR_VM_NODE_STAT_ITEMS ; global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD,
mm/workingset.c +4 -2
@@ -430,8 +430,10 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,for (pages =0 , i =0 ; i <NR_LRU_LISTS ; i++) pages +=lruvec_page_state_local (lruvec,NR_LRU_BASE + i); - pages +=lruvec_page_state_local (lruvec,NR_SLAB_RECLAIMABLE ); - pages +=lruvec_page_state_local (lruvec,NR_SLAB_UNRECLAIMABLE ); + pages +=lruvec_page_state_local ( + lruvec,NR_SLAB_RECLAIMABLE_B ) >>PAGE_SHIFT ; + pages +=lruvec_page_state_local ( + lruvec,NR_SLAB_UNRECLAIMABLE_B ) >>PAGE_SHIFT ; } else#endif pages =node_present_pages (sc->nid);
Signature · 1 line
2.21.0
Johannes Weiner
maintainerRe: [PATCH RFC 04/14] mm: vmstat: convert slab vmstat counter to bytes
Quoted 4 lines · Roman Gushchin
In order to prepare for per-object slab memory accounting, convert
NR_SLAB_RECLAIMABLEandNR_SLAB_UNRECLAIMABLEvmstat items to bytes.
Quoted 8 lines
To make sure that these vmstats are in bytes, rename them to
NR_SLAB_RECLAIMABLE_BandNR_SLAB_UNRECLAIMABLE_B(similar toNR_KERNEL_STACK_KB).The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into
atomic_long_twe use for vmstats.Signed-off-by Roman Gushchin <[email protected]>
Maybe a crazy idea, but instead of mixing bytes and pages, would it be difficult to account all vmstat items in bytes internally? And provide two general apis, byte and page based, to update and query the counts, instead of tying the unit it to individual items?
The vmstat_item_in_bytes() conditional shifting is pretty awkward in code that has a recent history littered with subtle breakages.
The translation helper node_page_state_pages() will yield garbage if used with the page-based counters, which is another easy to misuse interface.
We already have many places that multiply with PAGE_SIZE to get the stats in bytes or kb units.
And _B/_KB suffixes are kinda clunky.
The stats use atomic_long_t, so switching to atomic64_t doesn't make a difference on 64-bit and is backward compatible with 32-bit.
The per-cpu batch size you have to raise from s8 either way.
It seems to me that would make the code and API a lot simpler and easier to use / harder to misuse.
Re: [PATCH RFC 04/14] mm: vmstat: convert slab vmstat counter to bytes
Quoted 5 lines · Johannes Weiner
Quoted 4 lines · Roman Gushchin
In order to prepare for per-object slab memory accounting, convert
NR_SLAB_RECLAIMABLEandNR_SLAB_UNRECLAIMABLEvmstat items to bytes.
Quoted 8 lines
Quoted 8 lines
To make sure that these vmstats are in bytes, rename them to
NR_SLAB_RECLAIMABLE_BandNR_SLAB_UNRECLAIMABLE_B(similar toNR_KERNEL_STACK_KB).The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into
atomic_long_twe use for vmstats.Signed-off-by Roman Gushchin <[email protected]>
Hello Johannes!
Thank you for looking into the patchset!
Quoted 13 lines
Maybe a crazy idea, but instead of mixing bytes and pages, would it be difficult to account all vmstat items in bytes internally? And provide two general apis, byte and page based, to update and query the counts, instead of tying the unit it to individual items?
The
vmstat_item_in_bytes()conditional shifting is pretty awkward in code that has a recent history littered with subtle breakages.The translation helper
node_page_state_pages()will yield garbage if used with the page-based counters, which is another easy to misuse interface.
Quoted 7 lines
We already have many places that multiply with
PAGE_SIZEto get the stats in bytes or kb units.And _B/_KB suffixes are kinda clunky.
The stats use
atomic_long_t, so switching toatomic64_tdoesn't make a difference on 64-bit and is backward compatible with 32-bit.
I fully agree here, that having different stats in different units adds a lot of mess to the code. But I always thought that 64-bit atomics are slow on a 32-bit machine, so it might be a noticeable performance regression. Don't you think so?
I'm happy to prepare such a patch(set), only I'd prefer to keep it separately from this one. It can precede or follow the slab controller rework, either way will work. Slab controller rework is already not so small, so adding more code (and potential issues) here will only make the review more complex.
Quoted 2 lines
The per-cpu batch size you have to raise from s8 either way.
Yeah, tbh I don't know why those are just not unsigned long by default. Space savings are miserable here, and I don't see any other reasons. It could be even slightly faster to use a larger type.
I kinda tried to keep the patchset as small as possible (at least for the RFC version), so tried to avoid any non-necessary changes. But overall using s8 or s16 here doesn't make much sense to me.
Quoted 3 lines
It seems to me that would make the code and API a lot simpler and easier to use / harder to misuse.
RFC PATCH 5/14 mm: memcg/slab: allocate space for memcg ownership data for non-root slabs
In reply to Roman GushchinAllocate and release memory for storing the memcg ownership data. For each slab page allocate space sufficient for number_of_objects pointers to struct mem_cgroup_vec.
The mem_cgroup field of the struct page isn't used for slab pages, so let's use the space for storing the pointer for the allocated space.
This commit makes sure that the space is ready for use, but nobody is actually using it yet. Following commits in the series will fix it.
--- include/linux/mm_types.h | 5 ++++- mm/slab.c | 3 ++- mm/slab.h | 37 ++++++++++++++++++++++++++++++++++++- mm/slub.c | 2 +- 4 files changed, 43 insertions(+), 4 deletions(-)
include/linux/mm_types.h +4 -1
@@ -199,7 +199,10 @@ struct page {atomic_t _refcount;#ifdef CONFIG_MEMCG -struct mem_cgroup *mem_cgroup; +union { +struct mem_cgroup *mem_cgroup; +struct mem_cgroup_ptr **mem_cgroup_vec; + };#endif /*
mm/slab.c +2 -1
@@ -1369,7 +1369,8 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,return NULL ; } -if (charge_slab_page (page, flags, cachep->gfporder, cachep)) { +if (charge_slab_page (page, flags, cachep->gfporder, cachep, + cachep->num)) {__free_pages (page, cachep->gfporder);return NULL ; }
mm/slab.h +36 -1
@@ -406,6 +406,23 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,percpu_ref_put_many (&s->memcg_params.refcnt,1 << order); } +static inline int memcg_alloc_page_memcg_vec (struct page *page,gfp_t gfp, +unsigned int objects) +{ + page->mem_cgroup_vec =kmalloc (sizeof (struct mem_cgroup_ptr *) * + objects, gfp | __GFP_ZERO); +if (!page->mem_cgroup_vec) +return -ENOMEM ; + +return 0 ; +} + +static inline void memcg_free_page_memcg_vec (struct page *page) +{ +kfree (page->mem_cgroup_vec); + page->mem_cgroup_vec =NULL ; +} +extern void slab_init_memcg_params (struct kmem_cache *);extern void memcg_link_cache (struct kmem_cache *s,struct mem_cgroup *memcg);
@@ -455,6 +472,16 @@ static inline void memcg_uncharge_slab(struct page *page, int order, { } +static inline int memcg_alloc_page_memcg_vec (struct page *page,gfp_t gfp, +unsigned int objects) +{ +return 0 ; +} + +static inline void memcg_free_page_memcg_vec (struct page *page) +{ +} +static inline void slab_init_memcg_params (struct kmem_cache *s) { }
@@ -479,14 +506,21 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) static __always_inline int charge_slab_page(struct page *page,gfp_t gfp,int order, -struct kmem_cache *s) +struct kmem_cache *s, +unsigned int objects) { +int ret; +if (is_root_cache (s)) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s),PAGE_SIZE << order);return 0 ; } + ret =memcg_alloc_page_memcg_vec (page, gfp, objects); +if (ret) +return ret; +return memcg_charge_slab (page, gfp, order, s); }
@@ -499,6 +533,7 @@ static __always_inline void uncharge_slab_page(struct page *page, int order,return ; } +memcg_free_page_memcg_vec (page);memcg_uncharge_slab (page, order, s); }
mm/slub.c +1 -1
@@ -1517,7 +1517,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,else page =__alloc_pages_node (node, flags, order); -if (page &&charge_slab_page (page, flags, order, s)) { +if (page &&charge_slab_page (page, flags, order, s,oo_objects (oo))) {__free_pages (page, order); page =NULL ; }
Signature · 1 line
2.21.0
RFC PATCH 6/14 mm: slub: implement SLUB version of obj_to_index()
In reply to Roman GushchinThis commit implements SLUB version of the obj_to_index() function, which will be required to calculate the offset of memcg_ptr in the mem_cgroup_vec to store/obtain the memcg ownership data.
To make it faster, let's repeat the SLAB's trick introduced by commit 6a2d7a955d8d ("[PATCH] SLAB: use a multiply instead of a divide in obj_to_index()") and avoid an expensive division.
--- include/linux/slub_def.h | 9 +++++++++ mm/slub.c | 1 + 2 files changed, 10 insertions(+)
include/linux/slub_def.h +9 -0
@@ -8,6 +8,7 @@* (C) 2007 SGI, Christoph Lameter */ #include <linux/kobject.h> +#include <linux/reciprocal_div.h> enum stat_item {ALLOC_FASTPATH ,/* Allocation from cpu slab */
@@ -86,6 +87,7 @@ struct kmem_cache {unsigned long min_partial;unsigned int size;/* The size of an object including metadata */ unsigned int object_size;/* The size of an object without metadata */ +struct reciprocal_value reciprocal_size;unsigned int offset;/* Free pointer offset */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */
@@ -182,4 +184,11 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, return result; } +static inline unsigned int obj_to_index (const struct kmem_cache *cache, +const struct page *page,void *obj) +{ +return reciprocal_divide (kasan_reset_tag (obj) -page_address (page), + cache->reciprocal_size); +} +#endif /* _LINUX_SLUB_DEF_H */
mm/slub.c +1 -0
@@ -3587,6 +3587,7 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order)*/ size =ALIGN (size, s->align); s->size = size; + s->reciprocal_size =reciprocal_value (size);if (forced_order >=0 ) order = forced_order;else
Signature · 1 line
2.21.0
RFC PATCH 7/14 mm: memcg/slab: save memcg ownership data for non-root slab objects
In reply to Roman GushchinStore a memcg_ptr in the corresponding place of the mem_cgroup_vec for each allocated non-root slab object. Make sure that each allocated object holds a reference to the mem_cgroup_ptr.
To get the memcg_ptr in the post alloc hook, we need a memcg pointer. Because all memory cgroup will soon share the same set of kmem_caches, let's not use the kmem_cache->memcg_params.memcg. Instead, let's pass the pointer directly from memcg_kmem_get_cache(). This will guarantee that we will use the same cgroup in pre- and post-alloc hooks.
Please, note that the code is a bit bulky now, because we have to manage 3 types of objects with reference counters: memcg, kmem_cache and memcg_ptr. The following commits in the series will simplify it.
--- include/linux/memcontrol.h | 3 +- mm/memcontrol.c | 8 +++-- mm/slab.c | 18 ++++++----- mm/slab.h | 64 ++++++++++++++++++++++++++++++++++---- mm/slub.c | 14 ++++++--- 5 files changed, 86 insertions(+), 21 deletions(-)
include/linux/memcontrol.h +2 -1
@@ -1395,7 +1395,8 @@ static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, }#endif -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep, +struct mem_cgroup **memcgp);void memcg_kmem_put_cache (struct kmem_cache *cachep);#ifdef CONFIG_MEMCG_KMEM
mm/memcontrol.c +6 -2
@@ -3015,7 +3015,8 @@ static inline bool memcg_kmem_bypass(void)* done with it, memcg_kmem_put_cache() must be called to release the * reference. */ -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep) +struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep, +struct mem_cgroup **memcgp) {struct mem_cgroup *memcg;struct kmem_cache *memcg_cachep;
@@ -3071,8 +3072,11 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)*/ if (unlikely (!memcg_cachep))memcg_schedule_kmem_cache_create (memcg, cachep); -else if (percpu_ref_tryget (&memcg_cachep->memcg_params.refcnt)) +else if (percpu_ref_tryget (&memcg_cachep->memcg_params.refcnt)) { +css_get (&memcg->css); + *memcgp = memcg; cachep = memcg_cachep; + } out_unlock:rcu_read_unlock ();return cachep;
mm/slab.c +11 -7
@@ -3222,9 +3222,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,unsigned long save_flags;void *ptr;int slab_node =numa_mem_id (); +struct mem_cgroup *memcg =NULL ; flags &= gfp_allowed_mask; - cachep =slab_pre_alloc_hook (cachep, flags); + cachep =slab_pre_alloc_hook (cachep, &memcg,1 , flags);if (unlikely (!cachep))return NULL ;
@@ -3260,7 +3261,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size); -slab_post_alloc_hook (cachep, flags,1 , &ptr); +slab_post_alloc_hook (cachep, memcg, flags,1 , &ptr);return ptr; }
@@ -3301,9 +3302,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) {unsigned long save_flags;void *objp; +struct mem_cgroup *memcg =NULL ; flags &= gfp_allowed_mask; - cachep =slab_pre_alloc_hook (cachep, flags); + cachep =slab_pre_alloc_hook (cachep, &memcg,1 , flags);if (unlikely (!cachep))return NULL ;
@@ -3317,7 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size); -slab_post_alloc_hook (cachep, flags,1 , &objp); +slab_post_alloc_hook (cachep, memcg, flags,1 , &objp);return objp; }
@@ -3439,6 +3441,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller); +memcg_slab_free_hook (cachep,virt_to_head_page (objp), objp);/* * Skip calling cache_free_alien() when the platform is not numa.
@@ -3504,8 +3507,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) {size_t i; +struct mem_cgroup *memcg =NULL ; - s =slab_pre_alloc_hook (s, flags); + s =slab_pre_alloc_hook (s, &memcg, size, flags);if (!s)return 0 ;
@@ -3528,13 +3532,13 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size); -slab_post_alloc_hook (s, flags, size, p); +slab_post_alloc_hook (s, memcg, flags, size, p);/* FIXME: Trace call missing. Christoph would like a bulk variant */ return size; error:local_irq_enable ();cache_alloc_debugcheck_after_bulk (s, flags, i, p, _RET_IP_); -slab_post_alloc_hook (s, flags, i, p); +slab_post_alloc_hook (s, memcg, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ; }
mm/slab.h +58 -6
98 lines not on this page · raw message
mm/slub.c +9 -5
@@ -2686,8 +2686,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,struct kmem_cache_cpu *c;struct page *page;unsigned long tid; +struct mem_cgroup *memcg =NULL ; - s =slab_pre_alloc_hook (s, gfpflags); + s =slab_pre_alloc_hook (s, &memcg,1 , gfpflags);if (!s)return NULL ; redo:
@@ -2767,7 +2768,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size); -slab_post_alloc_hook (s, gfpflags,1 , &object); +slab_post_alloc_hook (s, memcg, gfpflags,1 , &object);return object; }
@@ -2972,6 +2973,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s,void *tail_obj = tail ? : head;struct kmem_cache_cpu *c;unsigned long tid; + +memcg_slab_free_hook (s, page, head); redo:/* * Determine the currently cpus per cpu slab.
@@ -3149,9 +3152,10 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, {struct kmem_cache_cpu *c;int i; +struct mem_cgroup *memcg =NULL ;/* memcg and kmem_cache debug support */ - s =slab_pre_alloc_hook (s, flags); + s =slab_pre_alloc_hook (s, &memcg, size, flags);if (unlikely (!s))return false;/*
@@ -3193,11 +3197,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, }/* memcg and kmem_cache debug support */ -slab_post_alloc_hook (s, flags, size, p); +slab_post_alloc_hook (s, memcg, flags, size, p);return i; error:local_irq_enable (); -slab_post_alloc_hook (s, flags, i, p); +slab_post_alloc_hook (s, memcg, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ; }
Signature · 1 line
2.21.0
RFC PATCH 8/14 mm: memcg: move memcg_kmem_bypass() to memcontrol.h
In reply to Roman GushchinTo make the memcg_kmem_bypass() function available outside of the memcontrol.c, let's move it to memcontrol.h. The function is small and nicely fits into static inline sort of functions.
It will be used from the slab code.
--- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 7 ------- 2 files changed, 7 insertions(+), 7 deletions(-)
include/linux/memcontrol.h +7 -0
@@ -1430,6 +1430,13 @@ static inline bool memcg_kmem_enabled(void)return static_branch_unlikely (&memcg_kmem_enabled_key); } +static inline bool memcg_kmem_bypass (void ) +{ +if (in_interrupt () || !current->mm || (current->flags &PF_KTHREAD )) +return true; +return false; +} +static inline int memcg_kmem_charge (struct page *page,gfp_t gfp,int order) {if (memcg_kmem_enabled ())
mm/memcontrol.c +0 -7
@@ -2992,13 +2992,6 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg,queue_work (memcg_kmem_cache_wq, &cw->work); } -static inline bool memcg_kmem_bypass (void ) -{ -if (in_interrupt () || !current->mm || (current->flags &PF_KTHREAD )) -return true; -return false; -} -/** * memcg_kmem_get_cache: select the correct per-memcg cache for allocation * @cachep: the original global kmem cache
Signature · 1 line
2.21.0
RFC PATCH 9/14 mm: memcg: introduce __mod_lruvec_memcg_state()
In reply to Roman GushchinTo prepare for per-object accounting of slab objects, let's introduce __mod_lruvec_memcg_state() and mod_lruvec_memcg_state() helpers, which are similar to mod_lruvec_state(), but do not update global node counters, only lruvec and per-cgroup.
It's necessary because soon node slab counters will be used for accounting of all memory used by slab pages, however on memcg level only the actually used memory will be counted. Free space will be shared between all cgroups, so it can't be accounted to any.
--- include/linux/memcontrol.h | 22 ++++++++++++++++++++++ mm/memcontrol.c | 37 +++++++++++++++++++++++++++---------- 2 files changed, 49 insertions(+), 10 deletions(-)
include/linux/memcontrol.h +22 -0
@@ -739,6 +739,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,void __mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val); +void __mod_lruvec_memcg_state (struct lruvec *lruvec,enum node_stat_item idx, +int val);void __mod_lruvec_slab_state (void *p,enum node_stat_item idx,int val); static inline void mod_lruvec_state(struct lruvec *lruvec,
@@ -751,6 +753,16 @@ static inline void mod_lruvec_state(struct lruvec *lruvec,local_irq_restore (flags); } +static inline void mod_lruvec_memcg_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +unsigned long flags; + +local_irq_save (flags); +__mod_lruvec_memcg_state (lruvec, idx, val); +local_irq_restore (flags); +} +static inline void __mod_lruvec_page_state (struct page *page,enum node_stat_item idx,int val) {
@@ -1143,6 +1155,16 @@ static inline void mod_lruvec_state(struct lruvec *lruvec,mod_node_page_state (lruvec_pgdat (lruvec), idx, val); } +static inline void __mod_lruvec_memcg_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +} + +static inline void mod_lruvec_memcg_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +} +static inline void __mod_lruvec_page_state (struct page *page,enum node_stat_item idx,int val) {
mm/memcontrol.c +27 -10
@@ -795,16 +795,16 @@ parent_nodeinfo(struct mem_cgroup_per_node *pn, int nid) }/** -* __mod_lruvec_state - update lruvec memory statistics +* __mod_lruvec_memcg_state - update lruvec memory statistics * @lruvec: the lruvec * @idx: the stat item * @val: delta to add to the counter, can be negative * * The lruvec is the intersection of the NUMA node and a cgroup. This -* function updates the all three counters that are affected by a -* change of state at this level: per-node, per-cgroup, per-lruvec. +* function updates the two of three counters that are affected by a +* change of state at this level: per-cgroup and per-lruvec. */ -void __mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx, +void __mod_lruvec_memcg_state (struct lruvec *lruvec,enum node_stat_item idx,int val) {pg_data_t *pgdat =lruvec_pgdat (lruvec);
@@ -812,12 +812,6 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,struct mem_cgroup *memcg;long x, threshold =MEMCG_CHARGE_BATCH ; -/* Update node */ -__mod_node_page_state (pgdat, idx, val); - -if (mem_cgroup_disabled ()) -return ; - pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg;
@@ -841,6 +835,29 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,__this_cpu_write (pn->lruvec_stat_cpu->count[idx], x); } +/** +* __mod_lruvec_state - update lruvec memory statistics +* @lruvec: the lruvec +* @idx: the stat item +* @val: delta to add to the counter, can be negative +* +* The lruvec is the intersection of the NUMA node and a cgroup. This +* function updates the all three counters that are affected by a +* change of state at this level: per-node, per-cgroup, per-lruvec. +*/ +void __mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx, +int val) +{ +pg_data_t *pgdat =lruvec_pgdat (lruvec); + +/* Update node */ +__mod_node_page_state (pgdat, idx, val); + +/* Update per-cgroup and per-lruvec stats */ +if (!mem_cgroup_disabled ()) +__mod_lruvec_memcg_state (lruvec, idx, val); +} +void __mod_lruvec_slab_state (void *p,enum node_stat_item idx,int val) {struct page *page =virt_to_head_page (p);
Signature · 1 line
2.21.0
Quoted 113 lines · Roman Gushchin
The existing slab memory controller is based on the idea of replicating slab allocator internals for each memory cgroup. This approach promises a low memory overhead (one pointer per page), and isn't adding too much code on hot allocation and release paths. But is has a very serious flaw:
it leads to a low slab utilization.Using a drgn* script I've got an estimation of slab utilization on a number of machines running different production workloads. In most cases it was between 45% and 65%, and the best number I've seen was around 85%. Turning kmem accounting off brings it to high 90s. Also it brings back 30-50% of slab memory. It means that the real price of the existing slab memory controller is way bigger than a pointer per page.
The real reason why the existing design leads to a low slab utilization is simple: slab pages are used exclusively by one memory cgroup. If there are only few allocations of certain size made by a cgroup, or if some active objects (e.g. dentries) are left after the cgroup is deleted, or the cgroup contains a single-threaded application which is barely allocating any kernel objects, but does it every time on a new CPU:
in all these cases the resulting slab utilization is very low. If kmem accounting is off, the kernel is able to use free space on slab pages for other allocations.Arguably it wasn't an issue back to days when the kmem controller was introduced and was an opt-in feature, which had to be turned on individually for each memory cgroup. But now it's turned on by default on both cgroup v1 and v2. And modern systemd-based systems tend to create a large number of cgroups.
This patchset provides a new implementation of the slab memory controller, which aims to reach a much better slab utilization by sharing slab pages between multiple memory cgroups. Below is the short description of the new design (more details in commit messages).
Accounting is performed per-object instead of per-page. Slab-related vmstat counters are converted to bytes. Charging is performed on page-basis, with rounding up and remembering leftovers.
Memcg ownership data is stored in a per-slab-page vector: for each slab page a vector of corresponding size is allocated. To keep slab memory reparenting working, instead of saving a pointer to the memory cgroup directly an intermediate object is used. It's simply a pointer to a memcg (which can be easily changed to the parent) with a built-in reference counter. This scheme allows to reparent all allocated objects without walking them over and changing memcg pointer to the parent.
Instead of creating an individual set of
kmem_cachesfor each memory cgroup, two global sets are used: the root set for non-accounted and root-cgroup allocations and the second set for all other allocations. This allows to simplify the lifetime management of individualkmem_caches: they are destroyed with root counterparts. It allows to remove a good amount of code and make things generally simpler.The patchset contains a couple of semi-independent parts, which can find their usage outside of the slab memory controller too:
1) subpage charging API, which can be used in the future for accounting of other non-page-sized objects, e.g. percpu allocations.
2)mem_cgroup_ptrAPI (refcounted pointers to a memcg, can be reused for the efficient reparenting of other objects, e.g. pagecache.The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memorySo far I haven't found any regression on all tested workloads, but potential CPU regression caused by more precise accounting is a concern.
Obviously the amount of saved memory depend on the number of memory cgroups, uptime and specific workloads, but overall it feels like the new controller saves 30-40% of slab memory, sometimes more. Additionally, it should lead to a lower memory fragmentation, just because of a smaller number of non-movable pages and also because there is no more need to move all slab objects to a new set of pages when a workload is restarted in a new memory cgroup.
* https://github.com/osandov/drgn
Roman Gushchin (14):
mm: memcg: subpage charging API
mm: memcg: introducemem_cgroup_ptr
mm: vmstat: use s32 forvm_node_stat_diffin structper_cpu_nodestat
mm: vmstat: convert slab vmstat counter to bytes
mm: memcg/slab: allocate space for memcg ownership data for non-root
slabs
mm: slub: implement SLUB version ofobj_to_index()
mm: memcg/slab: save memcg ownership data for non-root slab objects
mm: memcg: movememcg_kmem_bypass()tomemcontrol.h
mm: memcg: introduce__mod_lruvec_memcg_state()
mm: memcg/slab: charge individual slab objects instead of pages
mm: memcg: moveget_mem_cgroup_from_current()tomemcontrol.h
mm: memcg/slab: replacememcg_from_slab_page()with
memcg_from_slab_obj()
mm: memcg/slab: use one set ofkmem_cachesfor all memory cgroups
mm: slab: remove redundant check inmemcg_accumulate_slabinfo()drivers/base/node.c | 11 +- fs/proc/meminfo.c | 4 +- include/linux/memcontrol.h | 102 ++++++++- include/linux/mm_types.h | 5 +- include/linux/mmzone.h | 12 +- include/linux/slab.h | 3 +- include/linux/slub_def.h | 9 + include/linux/vmstat.h | 8 + kernel/power/snapshot.c | 2 +- mm/list_lru.c | 12 +- mm/memcontrol.c | 431 +++++++++++++++++++++-------------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 +- mm/slab.c | 37 ++-
Quoted 9 lines
mm/slab.h | 300 +++++++++++++------------ mm/slab_common.c | 449 ++++--------------------------------- mm/slob.c | 12 +- mm/slub.c | 63 ++---- mm/vmscan.c | 3 +- mm/vmstat.c | 38 +++- mm/workingset.c | 6 +- 21 files changed, 683 insertions(+), 834 deletions(-)
I can only see the first 9 patches. Patches 10-14 are not there.
Cheers,
Longman
Quoted 123 lines · Waiman Long
Quoted 113 lines · Roman Gushchin
The existing slab memory controller is based on the idea of replicating slab allocator internals for each memory cgroup. This approach promises a low memory overhead (one pointer per page), and isn't adding too much code on hot allocation and release paths. But is has a very serious flaw:
it leads to a low slab utilization.Using a drgn* script I've got an estimation of slab utilization on a number of machines running different production workloads. In most cases it was between 45% and 65%, and the best number I've seen was around 85%. Turning kmem accounting off brings it to high 90s. Also it brings back 30-50% of slab memory. It means that the real price of the existing slab memory controller is way bigger than a pointer per page.
The real reason why the existing design leads to a low slab utilization is simple: slab pages are used exclusively by one memory cgroup. If there are only few allocations of certain size made by a cgroup, or if some active objects (e.g. dentries) are left after the cgroup is deleted, or the cgroup contains a single-threaded application which is barely allocating any kernel objects, but does it every time on a new CPU:
in all these cases the resulting slab utilization is very low. If kmem accounting is off, the kernel is able to use free space on slab pages for other allocations.Arguably it wasn't an issue back to days when the kmem controller was introduced and was an opt-in feature, which had to be turned on individually for each memory cgroup. But now it's turned on by default on both cgroup v1 and v2. And modern systemd-based systems tend to create a large number of cgroups.
This patchset provides a new implementation of the slab memory controller, which aims to reach a much better slab utilization by sharing slab pages between multiple memory cgroups. Below is the short description of the new design (more details in commit messages).
Accounting is performed per-object instead of per-page. Slab-related vmstat counters are converted to bytes. Charging is performed on page-basis, with rounding up and remembering leftovers.
Memcg ownership data is stored in a per-slab-page vector: for each slab page a vector of corresponding size is allocated. To keep slab memory reparenting working, instead of saving a pointer to the memory cgroup directly an intermediate object is used. It's simply a pointer to a memcg (which can be easily changed to the parent) with a built-in reference counter. This scheme allows to reparent all allocated objects without walking them over and changing memcg pointer to the parent.
Instead of creating an individual set of
kmem_cachesfor each memory cgroup, two global sets are used: the root set for non-accounted and root-cgroup allocations and the second set for all other allocations. This allows to simplify the lifetime management of individualkmem_caches: they are destroyed with root counterparts. It allows to remove a good amount of code and make things generally simpler.The patchset contains a couple of semi-independent parts, which can find their usage outside of the slab memory controller too:
1) subpage charging API, which can be used in the future for accounting of other non-page-sized objects, e.g. percpu allocations.
2)mem_cgroup_ptrAPI (refcounted pointers to a memcg, can be reused for the efficient reparenting of other objects, e.g. pagecache.The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memorySo far I haven't found any regression on all tested workloads, but potential CPU regression caused by more precise accounting is a concern.
Obviously the amount of saved memory depend on the number of memory cgroups, uptime and specific workloads, but overall it feels like the new controller saves 30-40% of slab memory, sometimes more. Additionally, it should lead to a lower memory fragmentation, just because of a smaller number of non-movable pages and also because there is no more need to move all slab objects to a new set of pages when a workload is restarted in a new memory cgroup.
* https://github.com/osandov/drgn
Roman Gushchin (14):
mm: memcg: subpage charging API
mm: memcg: introducemem_cgroup_ptr
mm: vmstat: use s32 forvm_node_stat_diffin structper_cpu_nodestat
mm: vmstat: convert slab vmstat counter to bytes
mm: memcg/slab: allocate space for memcg ownership data for non-root
slabs
mm: slub: implement SLUB version ofobj_to_index()
mm: memcg/slab: save memcg ownership data for non-root slab objects
mm: memcg: movememcg_kmem_bypass()tomemcontrol.h
mm: memcg: introduce__mod_lruvec_memcg_state()
mm: memcg/slab: charge individual slab objects instead of pages
mm: memcg: moveget_mem_cgroup_from_current()tomemcontrol.h
mm: memcg/slab: replacememcg_from_slab_page()with
memcg_from_slab_obj()
mm: memcg/slab: use one set ofkmem_cachesfor all memory cgroups
mm: slab: remove redundant check inmemcg_accumulate_slabinfo()drivers/base/node.c | 11 +- fs/proc/meminfo.c | 4 +- include/linux/memcontrol.h | 102 ++++++++- include/linux/mm_types.h | 5 +- include/linux/mmzone.h | 12 +- include/linux/slab.h | 3 +- include/linux/slub_def.h | 9 + include/linux/vmstat.h | 8 + kernel/power/snapshot.c | 2 +- mm/list_lru.c | 12 +- mm/memcontrol.c | 431 +++++++++++++++++++++-------------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 +- mm/slab.c | 37 ++-Quoted 9 lines
mm/slab.h | 300 +++++++++++++------------ mm/slab_common.c | 449 ++++--------------------------------- mm/slob.c | 12 +- mm/slub.c | 63 ++---- mm/vmscan.c | 3 +- mm/vmstat.c | 38 +++- mm/workingset.c | 6 +- 21 files changed, 683 insertions(+), 834 deletions(-)
Quoted 1 line
I can only see the first 9 patches. Patches 10-14 are not there.
Hm, strange. I'll rebase the patchset on top of the current mm tree and resend.
In the meantime you can find the original patchset here:
https://github.com/rgushchin/linux/tree/new_slab.rfc
or on top of the 5.3 release, which might be better for testing here:
https://github.com/rgushchin/linux/tree/new_slab.rfc.v5.3
Thanks!
Quoted 5 lines · Roman Gushchin
The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memory
Do these workloads cycle through a lot of different memcgs?
For workloads that don't, wouldn't this approach potentially use more memory? For example, a workload where everything is in one or two memcgs, and those memcgs last forever.
-- Suleiman
Quoted 8 lines · Suleiman Souhlal
Quoted 5 lines · Roman Gushchin
The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memoryDo these workloads cycle through a lot of different memcgs?
Not really, those are just plain services managed by systemd. They aren't restarted too often, maybe several times per day at most.
Also, there is nothing fb-specific. You can take any new modern distributive (I've tried Fedora 30), boot it up and look at the amount of slab memory. Numbers are roughly the same.
Quoted 5 lines
For workloads that don't, wouldn't this approach potentially use more memory? For example, a workload where everything is in one or two memcgs, and those memcgs last forever.
Yes, it's true, if you have a very small and fixed number of memory cgroups, in theory the new approach can take ~10% more memory.
I don't think it's such a big problem though: it seems that the majority of cgroup users have a lot of them, and they are dynamically created and destroyed by systemd/kubernetes/whatever else.
And if somebody has a very special setup with only 1-2 cgroups, arguably kernel memory accounting isn't such a big thing for them, so it can be simple disabled. Am I wrong and do you have a real-life example?
Thanks!
Roman
Quoted 9 lines · Roman Gushchin
Quoted 7 lines · Suleiman Souhlal
Quoted 5 lines · Roman Gushchin
The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memory
Quoted 8 lines
Quoted 1 line
Do these workloads cycle through a lot of different memcgs?
Not really, those are just plain services managed by systemd. They aren't restarted too often, maybe several times per day at most.
Also, there is nothing fb-specific. You can take any new modern distributive (I've tried Fedora 30), boot it up and look at the amount of slab memory. Numbers are roughly the same.
Ah, ok.
These numbers are kind of surprising to me.
Do you know if the savings are similar if you use CONFIG_SLAB instead of CONFIG_SLUB?
Quoted 8 lines
Quoted 4 lines
For workloads that don't, wouldn't this approach potentially use more memory? For example, a workload where everything is in one or two memcgs, and those memcgs last forever.
Yes, it's true, if you have a very small and fixed number of memory cgroups, in theory the new approach can take ~10% more memory.
Quoted 7 lines
I don't think it's such a big problem though: it seems that the majority of cgroup users have a lot of them, and they are dynamically created and destroyed by systemd/kubernetes/whatever else.
And if somebody has a very special setup with only 1-2 cgroups, arguably kernel memory accounting isn't such a big thing for them, so it can be simple disabled. Am I wrong and do you have a real-life example?
No, I don't have any specific examples.
-- Suleiman
Quoted 15 lines · Suleiman Souhlal
Quoted 14 lines · Roman Gushchin
Quoted 8 lines · Suleiman Souhlal
Quoted 5 lines · Roman Gushchin
The patchset has been tested on a number of different workloads in our production. In all cases, it saved hefty amounts of memory:
1) web frontend, 650-700 Mb, ~42% of slab memory
2) database cache, 750-800 Mb, ~35% of slab memory
3) dns server, 700 Mb, ~36% of slab memoryDo these workloads cycle through a lot of different memcgs?
Not really, those are just plain services managed by systemd. They aren't restarted too often, maybe several times per day at most.
Quoted 8 lines
Quoted 3 lines
Also, there is nothing fb-specific. You can take any new modern distributive (I've tried Fedora 30), boot it up and look at the amount of slab memory. Numbers are roughly the same.
Ah, ok.
These numbers are kind of surprising to me.
Do you know if the savings are similar if you useCONFIG_SLABinstead ofCONFIG_SLUB?
I did only a brief testing of the SLAB version: savings were there, numbers were slightly less impressive, but still in a double digit number of percents.
Quoted 20 lines
Quoted 9 lines
Quoted 5 lines
For workloads that don't, wouldn't this approach potentially use more memory? For example, a workload where everything is in one or two memcgs, and those memcgs last forever.
Yes, it's true, if you have a very small and fixed number of memory cgroups, in theory the new approach can take ~10% more memory.
Quoted 7 lines
I don't think it's such a big problem though: it seems that the majority of cgroup users have a lot of them, and they are dynamically created and destroyed by systemd/kubernetes/whatever else.
And if somebody has a very special setup with only 1-2 cgroups, arguably kernel memory accounting isn't such a big thing for them, so it can be simple disabled. Am I wrong and do you have a real-life example?
No, I don't have any specific examples.
-- Suleiman
Quoted 3 lines · Roman Gushchin
Roman Gushchin (14):
[...]
mm: memcg/slab: use one set ofkmem_cachesfor all memory cgroups
From that commit's message:
Quoted 2 lines
6) obsoletes kmem.slabinfo cgroup v1 interface file, as there are no per-memcg
kmem_cachesanymore (empty output is printed)
The empty file means no allocations took place in the particular cgroup. I find this quite a surprising change for consumers of these stats.
I understand obtaining the same data efficiently from the proposed structures is difficult, however, such a change should be avoided. (In my understanding, obsoleted file ~ not available in v2, however, it should not disappear from v1.)
Michal
Quoted 9 lines · Michal Koutný
Quoted 3 lines · Roman Gushchin
Roman Gushchin (14):
[...]
mm: memcg/slab: use one set ofkmem_cachesfor all memory cgroupsFrom that commit's message:
Quoted 2 lines
6) obsoletes kmem.slabinfo cgroup v1 interface file, as there are no per-memcg
kmem_cachesanymore (empty output is printed)
Quoted 7 lines
The empty file means no allocations took place in the particular cgroup. I find this quite a surprising change for consumers of these stats.
I understand obtaining the same data efficiently from the proposed structures is difficult, however, such a change should be avoided. (In my understanding, obsoleted file ~ not available in v2, however, it should not disappear from v1.)
Well, my assumption is that nobody is using this file for anything except debugging purposes (I might be wrong, if somebody has an automation based on it, please, let me know). A number of allocations of each type per memory cgroup is definitely a useful debug information, but currently it barely works (displayed numbers show mostly the number of allocated pages, not the number of active objects). We can support it, but it comes with the price, and most users don't really need it. So I don't think it worth it to make all allocations slower just to keep some debug interface working for some cgroup v1 users. Do you have examples when it's really useful and worth extra cpu cost?
Unfortunately, we can't enable it conditionally, as a user can switch between cgroup v1 and cgroup v2 memory controllers dynamically.
Thanks!
Quoted 30 lines · Roman Gushchin
Quoted 9 lines · Michal Koutný
Quoted 3 lines · Roman Gushchin
Roman Gushchin (14):
[...]
mm: memcg/slab: use one set ofkmem_cachesfor all memory cgroupsFrom that commit's message:
Quoted 2 lines
6) obsoletes kmem.slabinfo cgroup v1 interface file, as there are no per-memcg
kmem_cachesanymore (empty output is printed)Quoted 7 lines
The empty file means no allocations took place in the particular cgroup. I find this quite a surprising change for consumers of these stats.
I understand obtaining the same data efficiently from the proposed structures is difficult, however, such a change should be avoided. (In my understanding, obsoleted file ~ not available in v2, however, it should not disappear from v1.)
Well, my assumption is that nobody is using this file for anything except debugging purposes (I might be wrong, if somebody has an automation based on it, please, let me know). A number of allocations of each type per memory cgroup is definitely a useful debug information, but currently it barely works (displayed numbers show mostly the number of allocated pages, not the number of active objects). We can support it, but it comes with the price, and most users don't really need it. So I don't think it worth it to make all allocations slower just to keep some debug interface working for some cgroup v1 users. Do you have examples when it's really useful and worth extra cpu cost?
Quoted 2 lines
Unfortunately, we can't enable it conditionally, as a user can switch between cgroup v1 and cgroup v2 memory controllers dynamically.
kmem.slabinfo has been absolutely invaluable for debugging, in my experience. I am however not aware of any automation based on it.
Maybe it might be worth adding it to cgroup v2 and have a CONFIG option to enable it?
-- Suleiman
Quoted 2 lines · Suleiman Souhlal
kmem.slabinfo has been absolutely invaluable for debugging, in my experience. I am however not aware of any automation based on it.
My experience is the same. However, the point is that this has been exposed since ages, so the safe assumption is that there may be users.
Quoted 2 lines
Maybe it might be worth adding it to cgroup v2 and have a CONFIG option to enable it?
I don't think v2 file is necessary given the cost of obtaining the information. But I concur the idea of making the per-object tracking switchable at boot time or at least CONFIGurable.
Michal
Quoted 5 lines · Michal Koutný
Quoted 2 lines · Suleiman Souhlal
kmem.slabinfo has been absolutely invaluable for debugging, in my experience. I am however not aware of any automation based on it.
My experience is the same. However, the point is that this has been exposed since ages, so the safe assumption is that there may be users.
Yes, but kernel memory accounting was an opt-in feature for years, and also it can be disabled on boot time, so displaying an empty memory.slabinfo file doesn't break the interface.
Quoted 6 lines
Quoted 3 lines
Maybe it might be worth adding it to cgroup v2 and have a CONFIG option to enable it?
I don't think v2 file is necessary given the cost of obtaining the information. But I concur the idea of making the per-object tracking switchable at boot time or at least CONFIGurable.
As I said, the cost is the same and should be paid in any case, no matter if cgroup v1 or v2 is used. A user can dynamically switch between v1 and v2, and there is no way to obtain this information afterwards, so we need to collect it from scratch.
Another concern I have is that it will require adding a non-trivial amount of new code (as we don't have dynamically creating and destroying kmem_caches anymore). It's perfectly doable, but I'm not sure we need it so much to postpone the merging of the main thing. But I'm happy to hear any arguments why it's not true.
Thanks!
Hi,
I see the below crash during early boot when I try this patchset on PowerPC host. I am on new_slab.rfc.v5.3 branch.
BUG: Unable to handle kernel data access at 0x81030236d1814578 Faulting instruction address: 0xc0000000002cc314 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA PowerNV Modules linked in: ip_tables x_tables autofs4 sr_mod cdrom usbhid bnx2x crct10dif_vpmsum crct10dif_common mdio libcrc32c crc32c_vpmsum CPU: 31 PID: 1752 Comm: keyboard-setup. Not tainted 5.3.0-g9bd85fd72a0c #155 NIP: c0000000002cc314 LR: c0000000002cc2e8 CTR: 0000000000000000 REGS: c000001e40f378b0 TRAP: 0380 Not tainted (5.3.0-g9bd85fd72a0c) MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 44022224 XER: 00000000
CFAR: c0000000002c6ad4 IRQMASK: 1
GPR00: c0000000000b8a40 c000001e40f37b40 c000000000ed9600 0000000000000000
GPR04: 0000000000000023 0000000000000010 c000001e40f37b24 c000001e3cba3400
GPR08: 0000000000000020 81030218815f4578 0000001e50220000 0000000000000030
GPR12: 0000000000002200 c000001fff774d80 0000000000000000 00000001072600d8
GPR16: 0000000000000000 c0000000000bbaac 0000000000000000 0000000000000000
GPR20: c000001e40f37c48 0000000000000001 0000000000000000 c000001e3cba3400
GPR24: c000001e40f37dd8 0000000000000000 c000000000fa0d58 0000000000000000
GPR28: c000001e3a080080 c000001e32da0100 0000000000000118 0000000000000010 NIP [c0000000002cc314] __mod_memcg_state+0x58/0xd0
LR [c0000000002cc2e8] __mod_memcg_state+0x2c/0xd0
Call Trace: [c000001e40f37b90] [c0000000000b8a40] account_kernel_stack+0xa4/0xe4 [c000001e40f37bd0] [c0000000000ba4a4] copy_process+0x2b4/0x16f0 [c000001e40f37cf0] [c0000000000bbaac] _do_fork+0x9c/0x3e4 [c000001e40f37db0] [c0000000000bc030] sys_clone+0x74/0xa8 [c000001e40f37e20] [c00000000000bb34] ppc_clone+0x8/0xc
Instruction dump:4bffa7e9 2fa30000 409e007c 395efffb 3d000020 2b8a0001 409d0008 39000020 e93d0718 e94d0028 7bde1f24 7d29f214 <7ca9502a> 7fff2a14 7fe9fe76 7d27fa78
Looks like page->mem_cgroup_vec is allocated but not yet initialized with memcg pointers when we try to access them.
I did get past the crash by initializing the pointers like this in account_kernel_stack(), but I am pretty sure that this is not the place to do this:
kernel/fork.c +15 -2
@@ -380,13 +380,26 @@ static void account_kernel_stack(struct task_struct *tsk, int account)* All stack pages are in the same zone and belong to the * same memcg. */ -struct page *first_page =virt_to_page (stack); +struct page *first_page =virt_to_head_page ((stack)); +unsigned long off; +struct mem_cgroup_ptr *memcg_ptr; +struct mem_cgroup *memcg;mod_zone_page_state (page_zone (first_page),NR_KERNEL_STACK_KB ,THREAD_SIZE /1024 * account); -mod_memcg_page_state (first_page,MEMCG_KERNEL_STACK_KB , +if (!first_page->mem_cgroup_vec) +return ; + off =obj_to_index (task_struct_cachep, first_page, stack); + memcg_ptr = first_page->mem_cgroup_vec[off]; +if (!memcg_ptr) +return ; +rcu_read_lock (); + memcg = memcg_ptr->memcg; +if (memcg) +__mod_memcg_state (memcg,MEMCG_KERNEL_STACK_KB , account * (THREAD_SIZE /1024 )); +rcu_read_unlock (); } }
Regards,
Bharata.
Quoted 35 lines · Bharata B Rao
Hi,
I see the below crash during early boot when I try this patchset on PowerPC host. I am on new_slab.rfc.v5.3 branch.
BUG: Unable to handle kernel data access at 0x81030236d1814578 Faulting instruction address: 0xc0000000002cc314 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA PowerNV Modules linked in: ip_tables x_tables autofs4 sr_mod cdrom usbhid bnx2x crct10dif_vpmsum crct10dif_common mdio libcrc32c crc32c_vpmsum CPU: 31 PID: 1752 Comm: keyboard-setup. Not tainted 5.3.0-g9bd85fd72a0c #155 NIP: c0000000002cc314 LR: c0000000002cc2e8 CTR: 0000000000000000 REGS: c000001e40f378b0 TRAP: 0380 Not tainted (5.3.0-g9bd85fd72a0c) MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 44022224 XER: 00000000CFAR:
c0000000002c6ad4IRQMASK: 1
GPR00:c0000000000b8a40c000001e40f37b40c000000000ed96000000000000000000
GPR04: 0000000000000023 0000000000000010c000001e40f37b24c000001e3cba3400
GPR08: 000000000000002081030218815f45780000001e502200000000000000000030
GPR12: 0000000000002200c000001fff774d80000000000000000000000001072600d8
GPR16: 0000000000000000c0000000000bbaac0000000000000000 0000000000000000
GPR20:c000001e40f37c480000000000000001 0000000000000000c000001e3cba3400
GPR24:c000001e40f37dd80000000000000000c000000000fa0d580000000000000000
GPR28:c000001e3a080080c000001e32da01000000000000000118 0000000000000010 NIP [c0000000002cc314]__mod_memcg_state+0x58/0xd0
LR [c0000000002cc2e8]__mod_memcg_state+0x2c/0xd0Call Trace: [c000001e40f37b90] [c0000000000b8a40] account_kernel_stack+0xa4/0xe4 [c000001e40f37bd0] [c0000000000ba4a4] copy_process+0x2b4/0x16f0 [c000001e40f37cf0] [c0000000000bbaac] _do_fork+0x9c/0x3e4 [c000001e40f37db0] [c0000000000bc030] sys_clone+0x74/0xa8 [c000001e40f37e20] [c00000000000bb34] ppc_clone+0x8/0xcInstruction dump:
4bffa7e92fa30000409e007c395efffb3d0000202b8a0001409d000839000020e93d0718e94d00287bde1f247d29f214<7ca9502a>7fff2a147fe9fe767d27fa78
Quoted 5 lines
Looks like
page->mem_cgroup_vecis allocated but not yet initialized with memcg pointers when we try to access them.I did get past the crash by initializing the pointers like this in
account_kernel_stack(),
The above is not an accurate description of the hack I showed below. Essentially I am making sure that I get to the memcg corresponding to task_struct_cachep object in the page.
But that still doesn't explain why we don't hit this problem on x86.
Quoted 38 lines
but I am pretty sure that this is not the
place to do this:kernel/fork.c +15
-2@@ -380,13 +380,26 @@ static void account_kernel_stack(struct task_struct *tsk, int account)* All stack pages are in the same zone and belong to the * same memcg. */ -struct page *first_page =virt_to_page (stack); +struct page *first_page =virt_to_head_page ((stack)); +unsigned long off; +struct mem_cgroup_ptr *memcg_ptr; +struct mem_cgroup *memcg;mod_zone_page_state (page_zone (first_page),NR_KERNEL_STACK_KB ,THREAD_SIZE /1024 * account); -mod_memcg_page_state (first_page,MEMCG_KERNEL_STACK_KB , +if (!first_page->mem_cgroup_vec) +return ; + off =obj_to_index (task_struct_cachep, first_page, stack); + memcg_ptr = first_page->mem_cgroup_vec[off]; +if (!memcg_ptr) +return ; +rcu_read_lock (); + memcg = memcg_ptr->memcg; +if (memcg) +__mod_memcg_state (memcg,MEMCG_KERNEL_STACK_KB , account * (THREAD_SIZE /1024 )); +rcu_read_unlock (); } }Regards,
Bharata.
Quoted 39 lines · Bharata B Rao
Quoted 38 lines · Bharata B Rao
Hi,
I see the below crash during early boot when I try this patchset on PowerPC host. I am on new_slab.rfc.v5.3 branch.
BUG: Unable to handle kernel data access at 0x81030236d1814578 Faulting instruction address: 0xc0000000002cc314 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA PowerNV Modules linked in: ip_tables x_tables autofs4 sr_mod cdrom usbhid bnx2x crct10dif_vpmsum crct10dif_common mdio libcrc32c crc32c_vpmsum CPU: 31 PID: 1752 Comm: keyboard-setup. Not tainted 5.3.0-g9bd85fd72a0c #155 NIP: c0000000002cc314 LR: c0000000002cc2e8 CTR: 0000000000000000 REGS: c000001e40f378b0 TRAP: 0380 Not tainted (5.3.0-g9bd85fd72a0c) MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 44022224 XER: 00000000CFAR:
c0000000002c6ad4IRQMASK: 1
GPR00:c0000000000b8a40c000001e40f37b40c000000000ed96000000000000000000
GPR04: 0000000000000023 0000000000000010c000001e40f37b24c000001e3cba3400
GPR08: 000000000000002081030218815f45780000001e502200000000000000000030
GPR12: 0000000000002200c000001fff774d80000000000000000000000001072600d8
GPR16: 0000000000000000c0000000000bbaac0000000000000000 0000000000000000
GPR20:c000001e40f37c480000000000000001 0000000000000000c000001e3cba3400
GPR24:c000001e40f37dd80000000000000000c000000000fa0d580000000000000000
GPR28:c000001e3a080080c000001e32da01000000000000000118 0000000000000010 NIP [c0000000002cc314]__mod_memcg_state+0x58/0xd0
LR [c0000000002cc2e8]__mod_memcg_state+0x2c/0xd0Call Trace: [c000001e40f37b90] [c0000000000b8a40] account_kernel_stack+0xa4/0xe4 [c000001e40f37bd0] [c0000000000ba4a4] copy_process+0x2b4/0x16f0 [c000001e40f37cf0] [c0000000000bbaac] _do_fork+0x9c/0x3e4 [c000001e40f37db0] [c0000000000bc030] sys_clone+0x74/0xa8 [c000001e40f37e20] [c00000000000bb34] ppc_clone+0x8/0xcInstruction dump:
4bffa7e92fa30000409e007c395efffb3d0000202b8a0001409d000839000020e93d0718e94d00287bde1f247d29f214<7ca9502a>7fff2a147fe9fe767d27fa78Looks like
page->mem_cgroup_vecis allocated but not yet initialized with memcg pointers when we try to access them.
Quoted 6 lines
Quoted 2 lines
I did get past the crash by initializing the pointers like this in
account_kernel_stack(),The above is not an accurate description of the hack I showed below. Essentially I am making sure that I get to the memcg corresponding to
task_struct_cachepobject in the page.
Hello, Bharata!
Thank you very much for the report and the patch, it's a good catch, and the code looks good to me. I'll include the fix into the next version of the patchset (I can't keep it as a separate fix due to massive renamings/rewrites).
Quoted 2 lines
But that still doesn't explain why we don't hit this problem on x86.
On x86 (and arm64) we're using vmap-based stacks, so the underlying memory is allocated directly by the page allocator, bypassing the slab allocator. It depends on CONFIG_VMAP_STACK.
Btw, thank you for looking into the patchset and trying it on powerpc. Would you mind to share some results?
Thank you!
Roman
Quoted 8 lines · Roman Gushchin
Quoted 1 line · Bharata B Rao
On Mon, Dec 09, 2019 at 02:47:52PM +0530, Bharata B Rao wrote:
Hello, Bharata!
Thank you very much for the report and the patch, it's a good catch, and the code looks good to me. I'll include the fix into the next version of the patchset (I can't keep it as a separate fix due to massive renamings/rewrites).
Sure, but please note that I did post only the core change without the associated header includes etc, where I took some short cuts.
Quoted 7 lines
Quoted 3 lines
But that still doesn't explain why we don't hit this problem on x86.
On x86 (and arm64) we're using vmap-based stacks, so the underlying memory is allocated directly by the page allocator, bypassing the slab allocator. It depends on
CONFIG_VMAP_STACK.
I turned off CONFIG_VMAP_STACK on x86, but still don't hit any problems.
$ grep VMAP .config CONFIG_HAVE_ARCH_HUGE_VMAP=y CONFIG_HAVE_ARCH_VMAP_STACK=y # CONFIG_VMAP_STACK is not set
May be something else prevents this particular crash on x86?
Quoted 3 lines
Btw, thank you for looking into the patchset and trying it on powerpc. Would you mind to share some results?
Sure, I will get back with more results, but initial numbers when running a small alpine docker image look promising.
With slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 24bc99d94d91 sleek 0.00% 1MiB / 25MiB 4.00% 1.81kB / 0B 0B / 0B 0
Without slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 52382f8aaa13 sleek 0.00% 8.688MiB / 25MiB 34.75% 1.53kB / 0B 0B / 0B 0
So that's an improvement of MEM USAGE from 8.688MiB to 1MiB. Note that this docker container isn't doing anything useful and hence the numbers aren't representative of any workload.
Regards,
Bharata.
Quoted 5 lines · Bharata B Rao
Quoted 4 lines · Roman Gushchin
Quoted 1 line · Bharata B Rao
On Mon, Dec 09, 2019 at 02:47:52PM +0530, Bharata B Rao wrote:
Hello, Bharata!
Quoted 7 lines
Quoted 4 lines
Thank you very much for the report and the patch, it's a good catch, and the code looks good to me. I'll include the fix into the next version of the patchset (I can't keep it as a separate fix due to massive renamings/rewrites).
Sure, but please note that I did post only the core change without the associated header includes etc, where I took some short cuts.
Sure, I'll adopt the code to the next version.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
But that still doesn't explain why we don't hit this problem on x86.
On x86 (and arm64) we're using vmap-based stacks, so the underlying memory is allocated directly by the page allocator, bypassing the slab allocator. It depends on
CONFIG_VMAP_STACK.I turned off
CONFIG_VMAP_STACKon x86, but still don't hit any problems.
If you'll look at kernel/fork.c (~ :184), there are two ORed conditions to bypass the slab allocator:
1) THREAD_SIZE >= PAGE_SIZE
2) CONFIG_VMAP_STACK
I guess the first one is what saves x86 in your case, while on ppc you might have 64k pages (hard to say without looking at your config).
Quoted 7 lines
$ grep VMAP .config CONFIG_HAVE_ARCH_HUGE_VMAP=y CONFIG_HAVE_ARCH_VMAP_STACK=y # CONFIG_VMAP_STACK is not setMay be something else prevents this particular crash on x86?
I'm pretty sure it will crash, have stack been allocated using the slab allocator. I bet everybody are using vmap-based stacks.
Quoted 13 lines
Quoted 4 lines
Btw, thank you for looking into the patchset and trying it on powerpc. Would you mind to share some results?
Sure, I will get back with more results, but initial numbers when running a small alpine docker image look promising.
With slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 24bc99d94d91 sleek 0.00% 1MiB / 25MiB 4.00% 1.81kB / 0B 0B / 0B 0
Quoted 8 lines
Without slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 52382f8aaa13 sleek 0.00% 8.688MiB / 25MiB 34.75% 1.53kB / 0B 0B / 0B 0So that's an improvement of MEM USAGE from 8.688MiB to 1MiB. Note that this docker container isn't doing anything useful and hence the numbers aren't representative of any workload.
Cool, that's great!
Small containers is where the relative win is the biggest. Of course, it will decrease with the size of containers, but it's expected.
If you'll get any additional numbers, please, share them. It's really interesting, especially if you have larger-than-4k pages.
Thank you!
Roman
Quoted 15 lines · Roman Gushchin
Quoted 6 lines
With slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 24bc99d94d91 sleek 0.00% 1MiB / 25MiB 4.00% 1.81kB / 0B 0B / 0B 0Quoted 8 lines
Without slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 52382f8aaa13 sleek 0.00% 8.688MiB / 25MiB 34.75% 1.53kB / 0B 0B / 0B 0So that's an improvement of MEM USAGE from 8.688MiB to 1MiB. Note that this docker container isn't doing anything useful and hence the numbers aren't representative of any workload.
Quoted 7 lines
Cool, that's great!
Small containers is where the relative win is the biggest. Of course, it will decrease with the size of containers, but it's expected.
If you'll get any additional numbers, please, share them. It's really interesting, especially if you have larger-than-4k pages.
I run a couple of workloads contained within a memory cgroup and measured memory.kmem.usage_in_bytes and memory.usage_in_bytes with and without this patchset on PowerPC host. I see significant reduction in memory.kmem.usage_in_bytes and some reduction in memory.usage_in_bytes. Before posting the numbers, would like to get the following clarified:
In the original case, the memory cgroup is charged (including kmem charging) when a new slab page is allocated. In your patch, the subpage charging is done in slab_pre_alloc_hook routine. However in this case, I couldn't find where exactly kmem counters are being charged/updated. Hence wanted to make sure that the reduction in memory.kmem.usage_in_bytes that I am seeing is indeed real and not because kmem accounting was missed out for slab usage?
Also, I see all non-root allocations are coming from a single set of kmem_caches. Guess <kmemcache_name>-memcg caches don't yet show up in /proc/slabinfo and nor their stats is accumulated into /proc/slabinfo?
Regards,
Bharata.
Quoted 38 lines · Bharata B Rao
Quoted 15 lines · Roman Gushchin
Quoted 6 lines
With slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 24bc99d94d91 sleek 0.00% 1MiB / 25MiB 4.00% 1.81kB / 0B 0B / 0B 0Quoted 8 lines
Without slab patches
# docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 52382f8aaa13 sleek 0.00% 8.688MiB / 25MiB 34.75% 1.53kB / 0B 0B / 0B 0So that's an improvement of MEM USAGE from 8.688MiB to 1MiB. Note that this docker container isn't doing anything useful and hence the numbers aren't representative of any workload.
Quoted 7 lines
Cool, that's great!
Small containers is where the relative win is the biggest. Of course, it will decrease with the size of containers, but it's expected.
If you'll get any additional numbers, please, share them. It's really interesting, especially if you have larger-than-4k pages.
I run a couple of workloads contained within a memory cgroup and measured memory.kmem.usage_in_bytes and memory.usage_in_bytes with and without this patchset on PowerPC host. I see significant reduction in memory.kmem.usage_in_bytes and some reduction in memory.usage_in_bytes. Before posting the numbers, would like to get the following clarified:
In the original case, the memory cgroup is charged (including kmem charging) when a new slab page is allocated. In your patch, the subpage charging is done in
slab_pre_alloc_hookroutine. However in this case, I couldn't find where exactly kmem counters are being charged/updated. Hence wanted to make sure that the reduction in memory.kmem.usage_in_bytes that I am seeing is indeed real and not because kmem accounting was missed out for slab usage?
Quoted 3 lines
Also, I see all non-root allocations are coming from a single set of
kmem_caches. Guess <kmemcache_name>-memcg caches don't yet show up in/proc/slabinfoand nor their stats is accumulated into/proc/slabinfo?
Hello Bharata!
First I'd look at global slab counters in /proc/meminfo (or vmstat). These are reflecting the total system-wide amount of pages used by all slab memory and they are accurate.
What about cgroup-level counters, they are not perfect in the version which I posted. In general on cgroup v1 kernel memory is accounted twice: as a part of total memory (memory.usage_in_bytes) and as a separate value (memory.kmem.usage_in_bytes). The version of the slab controller which you test doesn't support the second one. Also, it doesn't include the space used by the accounting meta-data (1 pointer per object) into the accounting. But after all the difference in memory.usage_in_bytes values beside some margin (~10% of the difference) is meaningful.
The next version which I'm working on right now (and hope to post in a week or so) will address these issues.
Thanks!