The new cgroup slab memory controller
Subsystems (14)
v5 cover The new cgroup slab memory controller
This is a fifth version of the slab cgroup controller rework.
The patchset moves the accounting from the page level to the object level. It allows to share slab pages between memory cgroups. This leads to a significant win in the slab utilization (up to 45%) and the corresponding drop in the total kernel memory footprint. The reduced number of unmovable slab pages should also have a positive effect on the memory fragmentation.
The patchset makes the slab accounting code simpler: there is no more need in the complicated dynamic creation and destruction of per-cgroup slab caches, all memory cgroups use a global set of shared slab caches. The lifetime of slab caches is not more connected to the lifetime of memory cgroups.
The more precise accounting does require more CPU, however in practice the difference seems to be negligible. We've been using the new slab controller in Facebook production for several months with different workloads and haven't seen any noticeable regressions. What we've seen were memory savings in order of 1 GB per host (it varied heavily depending on the actual workload, size of RAM, number of CPUs, memory pressure, etc).
The third version of the patchset added yet another step towards the simplification of the code: sharing of slab caches between accounted and non-accounted allocations. It comes with significant upsides (most noticeable, a complete elimination of dynamic slab caches creation) but not without some regression risks, so this change sits on top of the patchset and is not completely merged in. So in the unlikely event of a noticeable performance regression it can be reverted separately.
v5:
1) fixed a build error, spotted by Vlastimil
2) added a comment about memcg->nr_charged_bytes, asked by Johannes
3) added missed acks and reviews
v4:
1) rebased on top of the mm tree, some fixes here and there
2) merged obj_to_index() with slab_index(), suggested by Vlastimil
3) changed objects_per_slab() to a better objects_per_slab_page(), suggested by Vlastimil
4) other minor fixes and changes
v3:
1) added a patch that switches to a global single set of kmem_caches
2) kmem API clean up dropped, because if has been already merged
3) byte-sized slab vmstat API over page-sized global counters and bytes-sized memcg/lruvec counters
3) obj_cgroup refcounting simplifications and other minor fixes
4) other minor changes
v2:
1) implemented re-layering and renaming suggested by Johannes, added his patch to the set. Thanks!
2) fixed the issue discovered by Bharata B Rao. Thanks!
3) added kmem API clean up part
4) added slab/memcg follow-up clean up part
5) fixed a couple of issues discovered by internal testing on FB fleet.
6) added kselftests
7) included metadata into the charge calculation
8) refreshed commit logs, regrouped patches, rebased onto mm tree, etc
v1:
1) fixed a bug in zoneinfo_show_print()
2) added some comments to the subpage charging API, a minor fix
3) separated memory.kmem.slabinfo deprecation into a separate patch, provided a drgn-based replacement
4) rebased on top of the current mm tree
RFC:
https://lwn.net/Articles/798605/
Johannes Weiner (1):
mm: memcontrol: decouple reference counting from page accounting
Roman Gushchin (18):
mm: memcg: factor out memcg- and lruvec-level changes out of
__mod_lruvec_state()
mm: memcg: prepare for byte-sized vmstat items
mm: memcg: convert vmstat slab counters to bytes
mm: slub: implement SLUB version of obj_to_index()
mm: memcg/slab: obj_cgroup API
mm: memcg/slab: allocate obj_cgroups for non-root slab pages
mm: memcg/slab: save obj_cgroup for non-root slab objects
mm: memcg/slab: charge individual slab objects instead of pages
mm: memcg/slab: deprecate memory.kmem.slabinfo
mm: memcg/slab: move memcg_kmem_bypass() to memcontrol.h
mm: memcg/slab: use a single set of kmem_caches for all accounted
allocations
mm: memcg/slab: simplify memcg cache creation
mm: memcg/slab: remove memcg_kmem_get_cache()
mm: memcg/slab: deprecate slab_root_caches
mm: memcg/slab: remove redundant check in memcg_accumulate_slabinfo()
mm: memcg/slab: use a single set of kmem_caches for all allocations
kselftests: cgroup: add kernel memory accounting tests tools/cgroup: add memcg_slabinfo.py tool
drivers/base/node.c | 6 +- fs/proc/meminfo.c | 4 +- include/linux/memcontrol.h | 80 ++- include/linux/mm_types.h | 5 +- include/linux/mmzone.h | 24 +- include/linux/slab.h | 5 - include/linux/slab_def.h | 9 +- include/linux/slub_def.h | 31 +- include/linux/vmstat.h | 14 +- kernel/power/snapshot.c | 2 +- mm/memcontrol.c | 603 +++++++++++-------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 +- mm/slab.c | 70 +-- mm/slab.h | 365 +++++------- mm/slab_common.c | 643 +-------------------- mm/slob.c | 12 +- mm/slub.c | 229 +------- mm/vmscan.c | 3 +- mm/vmstat.c | 30 +- mm/workingset.c | 6 +- tools/cgroup/memcg_slabinfo.py | 226 ++++++++ tools/testing/selftests/cgroup/.gitignore | 1 + tools/testing/selftests/cgroup/Makefile | 2 + tools/testing/selftests/cgroup/test_kmem.c | 382 ++++++++++++ 25 files changed, 1369 insertions(+), 1393 deletions(-) create mode 100755 tools/cgroup/memcg_slabinfo.py create mode 100644 tools/testing/selftests/cgroup/test_kmem.c
Signature · 1 line
2.25.4
PATCH v5 1/19 mm: memcg: factor out memcg- and lruvec-level changes out of __mod_lruvec_state()
To convert memcg and lruvec slab counters to bytes there must be a way to change these counters without touching node counters. Factor out __mod_memcg_lruvec_state() out of __mod_lruvec_state().
--- include/linux/memcontrol.h | 17 +++++++++++++++ mm/memcontrol.c | 43 +++++++++++++++++++++----------------- 2 files changed, 41 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +17 -0
@@ -681,11 +681,23 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,return x; } +void __mod_memcg_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx, +int val);void __mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val);void __mod_lruvec_slab_state (void *p,enum node_stat_item idx,int val);void mod_memcg_obj_state (void *p,int idx,int val); +static inline void mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +unsigned long flags; + +local_irq_save (flags); +__mod_memcg_lruvec_state (lruvec, idx, val); +local_irq_restore (flags); +} +static inline void mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val) {
@@ -1059,6 +1071,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,return node_page_state (lruvec_pgdat (lruvec), idx); } +static inline void __mod_memcg_lruvec_state (struct lruvec *lruvec, +enum node_stat_item idx,int val) +{ +} +static inline void __mod_lruvec_state (struct lruvec *lruvec,enum node_stat_item idx,int val) {
mm/memcontrol.c +24 -19
66 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v5 2/19 mm: memcg: prepare for byte-sized vmstat items
In reply to Roman GushchinTo implement per-object slab memory accounting, we need to convert slab vmstat counters to bytes. Actually, out of 4 levels of counters: global, per-node, per-memcg and per-lruvec only two last levels will require byte-sized counters. It's because global and per-node counters will be counting the number of slab pages, and per-memcg and per-lruvec will be counting the amount of memory taken by charged slab objects.
Converting all vmstat counters to bytes or even all slab counters to bytes would introduce an additional overhead. So instead let's store global and per-node counters in pages, and memcg and lruvec counters in bytes.
To make the API clean all access helpers (both on the read and write sides) are dealing with bytes.
To avoid back-and-forth conversions a new flavor of read-side helpers is introduced, which always returns values in pages:node_page_state_pages() and global_node_page_state_pages().
Actually new helpers are just reading raw values. Old helpers are simple wrappers, which will complain on an attempt to read byte value, because at the moment no one actually needs bytes.
Thanks to Johannes Weiner for the idea of having the byte-sized API on top of the page-sized internal storage.
--- drivers/base/node.c | 2 +- include/linux/mmzone.h | 10 ++++++++++ include/linux/vmstat.h | 14 +++++++++++++- mm/memcontrol.c | 14 ++++++++++---- mm/vmstat.c | 30 ++++++++++++++++++++++++++---- 5 files changed, 60 insertions(+), 10 deletions(-)
drivers/base/node.c +1 -1
@@ -513,7 +513,7 @@ static ssize_t node_read_vmstat(struct device *dev,for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) n +=sprintf (buf+n,"%s %lu\n" ,node_stat_name (i), -node_page_state (pgdat, i)); +node_page_state_pages (pgdat, i));return n; }
include/linux/mmzone.h +10 -0
@@ -207,6 +207,16 @@ enum node_stat_item {NR_VM_NODE_STAT_ITEMS }; +/* +* Returns true if the value is measured in bytes (most vmstat values are +* measured in pages). This defines the API part, the internal representation +* might be different. +*/ +static __always_inline bool vmstat_item_in_bytes(enum node_stat_item item) +{ +return false; +} +/* * We do arithmetic on the LRU lists in various places in the code, * so it is important to keep the active lists LRU_ACTIVE higher in
include/linux/vmstat.h +13 -1
@@ -8,6 +8,7 @@#include <linux/vm_event_item.h> #include <linux/atomic.h> #include <linux/static_key.h> +#include <linux/mmdebug.h> extern int sysctl_stat_interval;
@@ -192,7 +193,8 @@ static inline unsigned long global_zone_page_state(enum zone_stat_item item)return x; } -static inline unsigned long global_node_page_state (enum node_stat_item item) +static inline +unsigned long global_node_page_state_pages (enum node_stat_item item) {long x =atomic_long_read (&vm_node_stat[item]);#ifdef CONFIG_SMP
@@ -202,6 +204,13 @@ static inline unsigned long global_node_page_state(enum node_stat_item item)return x; } +static inline unsigned long global_node_page_state (enum node_stat_item item) +{ +VM_WARN_ON_ONCE (vmstat_item_in_bytes (item)); + +return global_node_page_state_pages (item); +} +static inline unsigned long zone_page_state (struct zone *zone,enum zone_stat_item item) {
@@ -242,9 +251,12 @@ extern unsigned long sum_zone_node_page_state(int node,extern unsigned long sum_zone_numa_state (int node,enum numa_stat_item item);extern unsigned long node_page_state (struct pglist_data *pgdat,enum node_stat_item item); +extern unsigned long node_page_state_pages (struct pglist_data *pgdat, +enum node_stat_item item);#else #define sum_zone_node_page_state (node, item) global_zone_page_state(item)#define node_page_state (node, item) global_node_page_state(item) +#define node_page_state_pages (node, item) global_node_page_state_pages(item)#endif /* CONFIG_NUMA */ #ifdef CONFIG_SMP
mm/memcontrol.c +10 -4
@@ -681,13 +681,16 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz)*/ void __mod_memcg_state (struct mem_cgroup *memcg,int idx,int val) { -long x; +long x, threshold =MEMCG_CHARGE_BATCH ;if (mem_cgroup_disabled ())return ; +if (vmstat_item_in_bytes (idx)) + threshold <<=PAGE_SHIFT ; + x = val +__this_cpu_read (memcg->vmstats_percpu->stat[idx]); -if (unlikely (abs (x) >MEMCG_CHARGE_BATCH )) { +if (unlikely (abs (x) > threshold)) {struct mem_cgroup *mi;/*
@@ -718,7 +721,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, {struct mem_cgroup_per_node *pn;struct mem_cgroup *memcg; -long x; +long x, threshold =MEMCG_CHARGE_BATCH ; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg;
@@ -729,8 +732,11 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,/* Update lruvec */ __this_cpu_add (pn->lruvec_stat_local->count[idx], val); +if (vmstat_item_in_bytes (idx)) + threshold <<=PAGE_SHIFT ; + x = val +__this_cpu_read (pn->lruvec_stat_cpu->count[idx]); -if (unlikely (abs (x) >MEMCG_CHARGE_BATCH )) { +if (unlikely (abs (x) > threshold)) {pg_data_t *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup_per_node *pi;
mm/vmstat.c +26 -4
@@ -341,6 +341,11 @@ void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,long x;long t; +if (vmstat_item_in_bytes (item)) { +VM_WARN_ON_ONCE (delta & (PAGE_SIZE -1 )); + delta >>=PAGE_SHIFT ; + } + x = delta +__this_cpu_read (*p); t =__this_cpu_read (pcp->stat_threshold);
@@ -398,6 +403,8 @@ void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)s8 __percpu *p = pcp->vm_node_stat_diff + item;s8 v, t; +VM_WARN_ON_ONCE (vmstat_item_in_bytes (item)); + v =__this_cpu_inc_return (*p); t =__this_cpu_read (pcp->stat_threshold);if (unlikely (v > t)) {
@@ -442,6 +449,8 @@ void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)s8 __percpu *p = pcp->vm_node_stat_diff + item;s8 v, t; +VM_WARN_ON_ONCE (vmstat_item_in_bytes (item)); + v =__this_cpu_dec_return (*p); t =__this_cpu_read (pcp->stat_threshold);if (unlikely (v < - t)) {
@@ -541,6 +550,11 @@ static inline void mod_node_state(struct pglist_data *pgdat,s8 __percpu *p = pcp->vm_node_stat_diff + item;long o, n, t, z; +if (vmstat_item_in_bytes (item)) { +VM_WARN_ON_ONCE (delta & (PAGE_SIZE -1 )); + delta >>=PAGE_SHIFT ; + } + do { z = 0; /* overflow to node counters */
@@ -989,8 +1003,8 @@ unsigned long sum_zone_numa_state(int node,/* * Determine the per node value of a stat item. */ -unsigned long node_page_state (struct pglist_data *pgdat, -enum node_stat_item item) +unsigned long node_page_state_pages (struct pglist_data *pgdat, +enum node_stat_item item) {long x =atomic_long_read (&pgdat->vm_stat[item]);#ifdef CONFIG_SMP
@@ -999,6 +1013,14 @@ unsigned long node_page_state(struct pglist_data *pgdat,#endif return x; } + +unsigned long node_page_state (struct pglist_data *pgdat, +enum node_stat_item item) +{ +VM_WARN_ON_ONCE (vmstat_item_in_bytes (item)); + +return node_page_state_pages (pgdat, item); +}#endif #ifdef CONFIG_COMPACTION
@@ -1582,7 +1604,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,seq_printf (m,"\n per-node stats" );for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) {seq_printf (m,"\n %-12s %lu" ,node_stat_name (i), -node_page_state (pgdat, i)); +node_page_state_pages (pgdat, i)); } } seq_printf(m,
@@ -1703,7 +1725,7 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos)#endif for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) - v[i] =global_node_page_state (i); + v[i] =global_node_page_state_pages (i); v +=NR_VM_NODE_STAT_ITEMS ; global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD,
Signature · 1 line
2.25.4
PATCH v5 3/19 mm: memcg: convert vmstat slab counters to bytes
In reply to Roman GushchinIn order to prepare for per-object slab memory accounting, convert NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE vmstat items to bytes.
To make it obvious, rename them to NR_SLAB_RECLAIMABLE_B and NR_SLAB_UNRECLAIMABLE_B (similar to NR_KERNEL_STACK_KB).
Internally global and per-node counters are stored in pages, however memcg and lruvec counters are stored in bytes. This scheme may look weird, but only for now. As soon as slab pages will be shared between multiple cgroups, global and node counters will reflect the total number of slab pages. However memcg and lruvec counters will be used for per-memcg slab memory tracking, which will take separate kernel objects in the account. Keeping global and node counters in pages helps to avoid additional overhead.
The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into atomic_long_t we use for vmstats.
--- drivers/base/node.c | 4 ++-- fs/proc/meminfo.c | 4 ++-- include/linux/mmzone.h | 16 +++++++++++++--- kernel/power/snapshot.c | 2 +- mm/memcontrol.c | 11 ++++------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 ++++---- mm/slab.h | 15 ++++++++------- mm/slab_common.c | 4 ++-- mm/slob.c | 12 ++++++------ mm/slub.c | 8 ++++---- mm/vmscan.c | 3 ++- mm/workingset.c | 6 ++++-- 13 files changed, 53 insertions(+), 42 deletions(-)
drivers/base/node.c +2 -2
@@ -368,8 +368,8 @@ static ssize_t node_read_meminfo(struct device *dev,unsigned long sreclaimable, sunreclaimable;si_meminfo_node (&i, nid); - sreclaimable =node_page_state (pgdat,NR_SLAB_RECLAIMABLE ); - sunreclaimable =node_page_state (pgdat,NR_SLAB_UNRECLAIMABLE ); + sreclaimable =node_page_state_pages (pgdat,NR_SLAB_RECLAIMABLE_B ); + sunreclaimable =node_page_state_pages (pgdat,NR_SLAB_UNRECLAIMABLE_B ); n =sprintf (buf,"Node %d MemTotal: %8lu kB\n" "Node %d MemFree: %8lu kB\n"
fs/proc/meminfo.c +2 -2
@@ -52,8 +52,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v) pages[lru] =global_node_page_state (NR_LRU_BASE + lru); available =si_mem_available (); - sreclaimable =global_node_page_state (NR_SLAB_RECLAIMABLE ); - sunreclaim =global_node_page_state (NR_SLAB_UNRECLAIMABLE ); + sreclaimable =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ); + sunreclaim =global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B );show_val_kb (m,"MemTotal: " , i.totalram);show_val_kb (m,"MemFree: " , i.freeram);
include/linux/mmzone.h +13 -3
@@ -174,8 +174,8 @@ enum node_stat_item {NR_INACTIVE_FILE ,/* " " " " " */ NR_ACTIVE_FILE ,/* " " " " " */ NR_UNEVICTABLE ,/* " " " " " */ -NR_SLAB_RECLAIMABLE , -NR_SLAB_UNRECLAIMABLE , +NR_SLAB_RECLAIMABLE_B , +NR_SLAB_UNRECLAIMABLE_B ,NR_ISOLATED_ANON ,/* Temporary isolated pages from anon lru */ NR_ISOLATED_FILE ,/* Temporary isolated pages from file lru */ WORKINGSET_NODES,
@@ -214,7 +214,17 @@ enum node_stat_item {*/ static __always_inline bool vmstat_item_in_bytes(enum node_stat_item item) { -return false; +/* +* Global and per-node slab counters track slab pages. +* It's expected that changes are multiples of PAGE_SIZE. +* Internally values are stored in pages. +* +* Per-memcg and per-lruvec counters track memory, consumed +* by individual slab objects. These counters are actually +* byte-precise. +*/ +return (item ==NR_SLAB_RECLAIMABLE_B || + item ==NR_SLAB_UNRECLAIMABLE_B ); }/*
kernel/power/snapshot.c +1 -1
@@ -1663,7 +1663,7 @@ static unsigned long minimum_image_size(unsigned long saveable) {unsigned long size; - size =global_node_page_state (NR_SLAB_RECLAIMABLE ) + size =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ) +global_node_page_state (NR_ACTIVE_ANON ) +global_node_page_state (NR_INACTIVE_ANON ) +global_node_page_state (NR_ACTIVE_FILE )
mm/memcontrol.c +4 -7
@@ -1391,9 +1391,8 @@ static char *memory_stat_format(struct mem_cgroup *memcg) (u64 )memcg_page_state (memcg,MEMCG_KERNEL_STACK_KB ) * 1024);seq_buf_printf (&s,"slab %llu\n" , - (u64)(memcg_page_state (memcg,NR_SLAB_RECLAIMABLE ) + -memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE )) * -PAGE_SIZE ); + (u64)(memcg_page_state (memcg,NR_SLAB_RECLAIMABLE_B ) + +memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE_B )));seq_buf_printf (&s,"sock %llu\n" , (u64 )memcg_page_state (memcg,MEMCG_SOCK ) *PAGE_SIZE );
@@ -1423,11 +1422,9 @@ static char *memory_stat_format(struct mem_cgroup *memcg) PAGE_SIZE);seq_buf_printf (&s,"slab_reclaimable %llu\n" , - (u64 )memcg_page_state (memcg,NR_SLAB_RECLAIMABLE ) * -PAGE_SIZE ); + (u64 )memcg_page_state (memcg,NR_SLAB_RECLAIMABLE_B ));seq_buf_printf (&s,"slab_unreclaimable %llu\n" , - (u64 )memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE ) * -PAGE_SIZE ); + (u64 )memcg_page_state (memcg,NR_SLAB_UNRECLAIMABLE_B ));/* Accumulated memory events */
mm/oom_kill.c +1 -1
@@ -184,7 +184,7 @@ static bool is_dump_unreclaim_slabs(void)global_node_page_state (NR_ISOLATED_FILE ) +global_node_page_state (NR_UNEVICTABLE ); -return (global_node_page_state (NR_SLAB_UNRECLAIMABLE ) > nr_lru); +return (global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B ) > nr_lru); }/**
mm/page_alloc.c +4 -4
@@ -5248,8 +5248,8 @@ long si_mem_available(void)* items that are in use, and cannot be freed. Cap this estimate at the * low watermark. */ - reclaimable =global_node_page_state (NR_SLAB_RECLAIMABLE ) + -global_node_page_state (NR_KERNEL_MISC_RECLAIMABLE ); + reclaimable =global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ) + +global_node_page_state (NR_KERNEL_MISC_RECLAIMABLE ); available += reclaimable -min (reclaimable /2 , wmark_low);if (available <0 )
@@ -5393,8 +5393,8 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)global_node_page_state (NR_FILE_DIRTY ),global_node_page_state (NR_WRITEBACK ),global_node_page_state (NR_UNSTABLE_NFS ), -global_node_page_state (NR_SLAB_RECLAIMABLE ), -global_node_page_state (NR_SLAB_UNRECLAIMABLE ), +global_node_page_state_pages (NR_SLAB_RECLAIMABLE_B ), +global_node_page_state_pages (NR_SLAB_UNRECLAIMABLE_B ),global_node_page_state (NR_FILE_MAPPED ),global_node_page_state (NR_SHMEM ), global_zone_page_state(NR_PAGETABLE),
mm/slab.h +8 -7
@@ -272,7 +272,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **);static inline int cache_vmstat_idx (struct kmem_cache *s) {return (s->flags &SLAB_RECLAIM_ACCOUNT ) ? -NR_SLAB_RECLAIMABLE :NR_SLAB_UNRECLAIMABLE ; +NR_SLAB_RECLAIMABLE_B :NR_SLAB_UNRECLAIMABLE_B ; }#ifdef CONFIG_MEMCG_KMEM
@@ -361,7 +361,7 @@ static __always_inline int memcg_charge_slab(struct page *page,if (unlikely (!memcg ||mem_cgroup_is_root (memcg))) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - nr_pages); + nr_pages <<PAGE_SHIFT );percpu_ref_get_many (&s->memcg_params.refcnt, nr_pages);return 0 ; }
@@ -371,7 +371,7 @@ static __always_inline int memcg_charge_slab(struct page *page, goto out; lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); -mod_lruvec_state (lruvec,cache_vmstat_idx (s), nr_pages); +mod_lruvec_state (lruvec,cache_vmstat_idx (s), nr_pages <<PAGE_SHIFT );/* transer try_charge() page references to kmem_cache */ percpu_ref_get_many (&s->memcg_params.refcnt, nr_pages);
@@ -396,11 +396,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, memcg =READ_ONCE (s->memcg_params.memcg);if (likely (!mem_cgroup_is_root (memcg))) { lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); -mod_lruvec_state (lruvec,cache_vmstat_idx (s), -nr_pages); +mod_lruvec_state (lruvec,cache_vmstat_idx (s), + -(nr_pages <<PAGE_SHIFT ));memcg_kmem_uncharge (memcg, nr_pages); }else {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -nr_pages); + -(nr_pages <<PAGE_SHIFT )); }rcu_read_unlock ();
@@ -484,7 +485,7 @@ static __always_inline int charge_slab_page(struct page *page, {if (is_root_cache (s)) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), -1 << order); +PAGE_SIZE << order);return 0 ; }
@@ -496,7 +497,7 @@ static __always_inline void uncharge_slab_page(struct page *page, int order, {if (is_root_cache (s)) {mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -(1 << order)); + -(PAGE_SIZE << order));return ; }
mm/slab_common.c +2 -2
@@ -1325,8 +1325,8 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) page =alloc_pages (flags, order);if (likely (page)) { ret =page_address (page); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order); } ret =kasan_kmalloc_large (ret, size, flags);/* As ret might get tagged, call kmemleak hook after KASAN. */
mm/slob.c +6 -6
@@ -202,8 +202,8 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)if (!page)return NULL ; -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order);return page_address (page); }
@@ -214,8 +214,8 @@ static void slob_free_pages(void *b, int order)if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order; -mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (sp, order); }
@@ -552,8 +552,8 @@ void kfree(const void *block)slob_free (m, *m + align); }else {unsigned int order =compound_order (sp); -mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (sp),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (sp, order); }
mm/slub.c +4 -4
@@ -3923,8 +3923,8 @@ static void *kmalloc_large_node(size_t size, gfp_t flags, int node) page =alloc_pages_node (node, flags, order);if (page) { ptr =page_address (page); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , -1 << order); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , +PAGE_SIZE << order); }return kmalloc_large_node_hook (ptr, size, flags);
@@ -4055,8 +4055,8 @@ void kfree(const void *x)BUG_ON (!PageCompound (page));kfree_hook (object); -mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE , - -(1 << order)); +mod_node_page_state (page_pgdat (page),NR_SLAB_UNRECLAIMABLE_B , + -(PAGE_SIZE << order));__free_pages (page, order);return ; }
mm/vmscan.c +2 -1
@@ -4219,7 +4219,8 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)* unmapped file backed pages. */ if (node_pagecache_reclaimable (pgdat) <= pgdat->min_unmapped_pages && -node_page_state (pgdat,NR_SLAB_RECLAIMABLE ) <= pgdat->min_slab_pages) +node_page_state_pages (pgdat,NR_SLAB_RECLAIMABLE_B ) <= + pgdat->min_slab_pages)return NODE_RECLAIM_FULL ;/*
mm/workingset.c +4 -2
@@ -478,8 +478,10 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,for (pages =0 , i =0 ; i <NR_LRU_LISTS ; i++) pages +=lruvec_page_state_local (lruvec,NR_LRU_BASE + i); - pages +=lruvec_page_state_local (lruvec,NR_SLAB_RECLAIMABLE ); - pages +=lruvec_page_state_local (lruvec,NR_SLAB_UNRECLAIMABLE ); + pages +=lruvec_page_state_local ( + lruvec,NR_SLAB_RECLAIMABLE_B ) >>PAGE_SHIFT ; + pages +=lruvec_page_state_local ( + lruvec,NR_SLAB_UNRECLAIMABLE_B ) >>PAGE_SHIFT ; } else#endif pages =node_present_pages (sc->nid);
Signature · 1 line
2.25.4
PATCH v5 4/19 mm: slub: implement SLUB version of obj_to_index()
In reply to Roman GushchinThis commit implements SLUB version of the obj_to_index() function, which will be required to calculate the offset of obj_cgroup in the obj_cgroups vector to store/obtain the objcg ownership data.
To make it faster, let's repeat the SLAB's trick introduced by commit 6a2d7a955d8d ("[PATCH] SLAB: use a multiply instead of a divide in obj_to_index()") and avoid an expensive division.
Vlastimil Babka noticed, that SLUB does have already a similar function called slab_index(), which is defined only if SLUB_DEBUG is enabled. The function does a similar math, but with a division, and it also takes a page address instead of a page pointer.
Let's remove slab_index() and replace it with the new helper __obj_to_index(), which takes a page address. obj_to_index() will be a simple wrapper taking a page pointer and passing page_address(page) into __obj_to_index().
--- include/linux/slub_def.h | 16 ++++++++++++++++ mm/slub.c | 15 +++++---------- 2 files changed, 21 insertions(+), 10 deletions(-)
include/linux/slub_def.h +16 -0
@@ -8,6 +8,7 @@* (C) 2007 SGI, Christoph Lameter */ #include <linux/kobject.h> +#include <linux/reciprocal_div.h> enum stat_item {ALLOC_FASTPATH ,/* Allocation from cpu slab */
@@ -86,6 +87,7 @@ struct kmem_cache {unsigned long min_partial;unsigned int size;/* The size of an object including metadata */ unsigned int object_size;/* The size of an object without metadata */ +struct reciprocal_value reciprocal_size;unsigned int offset;/* Free pointer offset */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */
@@ -182,4 +184,18 @@ static inline void *nearest_obj(struct kmem_cache *cache, struct page *page, return result; } +/* Determine object index from a given position */ +static inline unsigned int __obj_to_index (const struct kmem_cache *cache, +void *addr,void *obj) +{ +return reciprocal_divide (kasan_reset_tag (obj) - addr, + cache->reciprocal_size); +} + +static inline unsigned int obj_to_index (const struct kmem_cache *cache, +const struct page *page,void *obj) +{ +return __obj_to_index (cache,page_address (page), obj); +} +#endif /* _LINUX_SLUB_DEF_H */
mm/slub.c +5 -10
@@ -313,12 +313,6 @@ static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) __p < (__addr ) + (__objects) * (__s)->size; \ __p += (__s)->size) -/* Determine object index from a given position */ -static inline unsigned int slab_index (void *p,struct kmem_cache *s,void *addr) -{ -return (kasan_reset_tag (p) - addr) / s->size; -} -static inline unsigned int order_objects (unsigned int order,unsigned int size) {return ((unsigned int )PAGE_SIZE << order) / size;
@@ -461,7 +455,7 @@ static unsigned long *get_map(struct kmem_cache *s, struct page *page)bitmap_zero (object_map, page->objects);for (p = page->freelist; p; p =get_freepointer (s, p)) -set_bit (slab_index (p, s, addr), object_map); +set_bit (__obj_to_index (s, addr, p), object_map);return object_map; }
@@ -3675,6 +3669,7 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order)*/ size =ALIGN (size, s->align); s->size = size; + s->reciprocal_size =reciprocal_value (size);if (forced_order >=0 ) order = forced_order;else
@@ -3781,7 +3776,7 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, map =get_map (s, page);for_each_object (p, s, addr, page->objects) { -if (!test_bit (slab_index (p, s, addr), map)) { +if (!test_bit (__obj_to_index (s, addr, p), map)) {pr_err ("INFO: Object 0x%p @offset=%tu\n" , p, p - addr);print_tracking (s, p); }
@@ -4506,7 +4501,7 @@ static void validate_slab(struct kmem_cache *s, struct page *page)/* Now we know that a valid freelist exists */ map =get_map (s, page);for_each_object (p, s, addr, page->objects) { -u8 val =test_bit (slab_index (p, s, addr), map) ? +u8 val =test_bit (__obj_to_index (s, addr, p), map) ?SLUB_RED_INACTIVE :SLUB_RED_ACTIVE ;if (!check_object (s, page, p, val))
@@ -4697,7 +4692,7 @@ static void process_slab(struct loc_track *t, struct kmem_cache *s, map =get_map (s, page);for_each_object (p, s, addr, page->objects) -if (!test_bit (slab_index (p, s, addr), map)) +if (!test_bit (__obj_to_index (s, addr, p), map))add_location (t, s,get_track (s, p, alloc));put_map (map); }
Signature · 1 line
2.25.4
PATCH v5 5/19 mm: memcontrol: decouple reference counting from page accounting
In reply to Roman GushchinFrom: Johannes Weiner <[email protected]>
The reference counting of a memcg is currently coupled directly to how many 4k pages are charged to it. This doesn't work well with Roman's new slab controller, which maintains pools of objects and doesn't want to keep an extra balance sheet for the pages backing those objects.
This unusual refcounting design (reference counts usually track pointers to an object) is only for historical reasons: memcg used to not take any css references and simply stalled offlining until all charges had been reparented and the page counters had dropped to zero. When we got rid of the reparenting requirement, the simple mechanical translation was to take a reference for every charge.
More historical context can be found in commit e8ea14cc6ead ("mm:
memcontrol: take a css reference for each charged page"), commit 64f219938941 ("mm: memcontrol: remove obsolete kmemcg pinning tricks") and commit b2052564e66d ("mm: memcontrol: continue cache reclaim from offlined groups").
The new slab controller exposes the limitations in this scheme, so let's switch it to a more idiomatic reference counting model based on actual kernel pointers to the memcg:
- The per-cpu stock holds a reference to the memcg its caching
- User pages hold a reference for their page->mem_cgroup. Transparent huge pages will no longer acquire tail references in advance, we'll get them if needed during the split.
- Kernel pages hold a reference for their page->mem_cgroup
- Pages allocated in the root cgroup will acquire and release css references for simplicity. css_get() and css_put() optimize that.
- The current memcg_charge_slab() already hacked around the per-charge references; this change gets rid of that as well.
Roman:
1) Rebased on top of the current mm tree: added css_get() in mem_cgroup_charge(), dropped mem_cgroup_try_charge() part
2) I've reformatted commit references in the commit log to make checkpatch.pl happy.
--- mm/memcontrol.c | 37 +++++++++++++++++++++---------------- mm/slab.h | 2 -- 2 files changed, 21 insertions(+), 18 deletions(-)
mm/memcontrol.c +21 -16
@@ -2094,13 +2094,17 @@ static void drain_stock(struct memcg_stock_pcp *stock) {struct mem_cgroup *old = stock->cached; +if (!old) +return ; +if (stock->nr_pages) {page_counter_uncharge (&old->memory, stock->nr_pages);if (do_memsw_account ())page_counter_uncharge (&old->memsw, stock->nr_pages); -css_put_many (&old->css, stock->nr_pages); stock->nr_pages =0 ; } + +css_put (&old->css); stock->cached =NULL ; }
@@ -2136,6 +2140,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) stock =this_cpu_ptr (&memcg_stock);if (stock->cached != memcg) {/* reset if necessary */ drain_stock (stock); +css_get (&memcg->css); stock->cached = memcg; } stock->nr_pages += nr_pages;
@@ -2566,12 +2571,10 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,page_counter_charge (&memcg->memory, nr_pages);if (do_memsw_account ())page_counter_charge (&memcg->memsw, nr_pages); -css_get_many (&memcg->css, nr_pages);return 0 ; done_restock: -css_get_many (&memcg->css, batch);if (batch > nr_pages)refill_stock (memcg, batch - nr_pages);
@@ -2608,8 +2611,6 @@ static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)page_counter_uncharge (&memcg->memory, nr_pages);if (do_memsw_account ())page_counter_uncharge (&memcg->memsw, nr_pages); - -css_put_many (&memcg->css, nr_pages); }static void commit_charge (struct page *page,struct mem_cgroup *memcg)
@@ -2909,6 +2910,7 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order)if (!ret) { page->mem_cgroup = memcg;__SetPageKmemcg (page); +return 0 ; } }css_put (&memcg->css);
@@ -2931,12 +2933,11 @@ void __memcg_kmem_uncharge_page(struct page *page, int order)VM_BUG_ON_PAGE (mem_cgroup_is_root (memcg), page);__memcg_kmem_uncharge (memcg, nr_pages); page->mem_cgroup =NULL ; +css_put (&memcg->css);/* slab pages do not have PageKmemcg flag set */ if (PageKmemcg (page)) __ClearPageKmemcg(page); - - css_put_many(&memcg->css, nr_pages); }#endif /* CONFIG_MEMCG_KMEM */
@@ -2948,13 +2949,16 @@ void __memcg_kmem_uncharge_page(struct page *page, int order)*/ void mem_cgroup_split_huge_fixup (struct page *head) { +struct mem_cgroup *memcg = head->mem_cgroup;int i;if (mem_cgroup_disabled ())return ; -for (i =1 ; i <HPAGE_PMD_NR ; i++) - head[i].mem_cgroup = head->mem_cgroup; +for (i =1 ; i <HPAGE_PMD_NR ; i++) { +css_get (&memcg->css); + head[i].mem_cgroup = memcg; + } }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
@@ -5398,7 +5402,10 @@ static int mem_cgroup_move_account(struct page *page,*/ smp_mb (); - page->mem_cgroup = to;/* caller should have done css_get */ +css_get (&to->css); +css_put (&from->css); + + page->mem_cgroup = to;__unlock_page_memcg (from);
@@ -6483,6 +6490,7 @@ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask)if (ret) goto out_put; +css_get (&memcg->css);commit_charge (page, memcg);local_irq_disable ();
@@ -6537,9 +6545,6 @@ static void uncharge_batch(const struct uncharge_gather *ug)__this_cpu_add (ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages);memcg_check_events (ug->memcg, ug->dummy_page);local_irq_restore (flags); - -if (!mem_cgroup_is_root (ug->memcg)) -css_put_many (&ug->memcg->css, ug->nr_pages); }static void uncharge_page (struct page *page,struct uncharge_gather *ug)
@@ -6577,6 +6582,7 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) ug->dummy_page = page; page->mem_cgroup =NULL ; +css_put (&ug->memcg->css); }static void uncharge_list (struct list_head *page_list)
@@ -6682,8 +6688,8 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage)page_counter_charge (&memcg->memory, nr_pages);if (do_memsw_account ())page_counter_charge (&memcg->memsw, nr_pages); -css_get_many (&memcg->css, nr_pages); +css_get (&memcg->css);commit_charge (newpage, memcg);local_irq_save (flags);
@@ -6920,8 +6926,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry)mem_cgroup_charge_statistics (memcg, page, -nr_entries);memcg_check_events (memcg, page); -if (!mem_cgroup_is_root (memcg)) -css_put_many (&memcg->css, nr_entries); +css_put (&memcg->css); }/**
mm/slab.h +0 -2
@@ -373,9 +373,7 @@ static __always_inline int memcg_charge_slab(struct page *page, lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page));mod_lruvec_state (lruvec,cache_vmstat_idx (s), nr_pages <<PAGE_SHIFT ); -/* transer try_charge() page references to kmem_cache */ percpu_ref_get_many (&s->memcg_params.refcnt, nr_pages); -css_put_many (&memcg->css, nr_pages); out:css_put (&memcg->css);return ret;
Signature · 1 line
2.25.4
PATCH v5 6/19 mm: memcg/slab: obj_cgroup API
In reply to Roman GushchinObj_cgroup API provides an ability to account sub-page sized kernel objects, which potentially outlive the original memory cgroup.
The top-level API consists of the following functions:
bool obj_cgroup_tryget(struct obj_cgroup *objcg); void obj_cgroup_get(struct obj_cgroup *objcg); void obj_cgroup_put(struct obj_cgroup *objcg);
int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg); struct obj_cgroup *get_obj_cgroup_from_current(void);
Object cgroup is basically a pointer to a memory cgroup with a per-cpu reference counter. It substitutes a memory cgroup in places where it's necessary to charge a custom amount of bytes instead of pages.
All charged memory rounded down to pages is charged to the corresponding memory cgroup using __memcg_kmem_charge().
It implements reparenting: on memcg offlining it's getting reattached to the parent memory cgroup. Each online memory cgroup has an associated active object cgroup to handle new allocations and the list of all attached object cgroups. On offlining of a cgroup this list is reparented and for each object cgroup in the list the memcg pointer is swapped to the parent memory cgroup. It prevents long-living objects from pinning the original memory cgroup in the memory.
The implementation is based on byte-sized per-cpu stocks. A sub-page sized leftover is stored in an atomic field, which is a part of obj_cgroup object. So on cgroup offlining the leftover is automatically reparented.
memcg->objcg is rcu protected.objcg->memcg is a raw pointer, which is always pointing at a memory cgroup, but can be atomically swapped to the parent memory cgroup. So the caller must ensure the lifetime of the cgroup, e.g. grab rcu_read_lock or css_set_lock.
--- include/linux/memcontrol.h | 51 +++++++ mm/memcontrol.c | 288 ++++++++++++++++++++++++++++++++++++- 2 files changed, 338 insertions(+), 1 deletion(-)
include/linux/memcontrol.h +51 -0
@@ -23,6 +23,7 @@#include <linux/page-flags.h> struct mem_cgroup ; +struct obj_cgroup ;struct page ;struct mm_struct ;struct kmem_cache ;
@@ -191,6 +192,22 @@ struct memcg_cgwb_frn {struct wb_completion done;/* tracks in-flight foreign writebacks */ }; +/* +* Bucket for arbitrarily byte-sized objects charged to a memory +* cgroup. The bucket can be reparented in one piece when the cgroup +* is destroyed, without having to round up the individual references +* of all live memory objects in the wild. +*/ +struct obj_cgroup { +struct percpu_ref refcnt; +struct mem_cgroup *memcg; +atomic_t nr_charged_bytes; +union { +struct list_head list; +struct rcu_head rcu; + }; +}; +/* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide
@@ -303,6 +320,8 @@ struct mem_cgroup {int kmemcg_id;enum memcg_kmem_state kmem_state;struct list_head kmem_caches; +struct obj_cgroup __rcu *objcg; +struct list_head objcg_list;#endif #ifdef CONFIG_CGROUP_WRITEBACK
@@ -418,6 +437,33 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; } +static inline bool obj_cgroup_tryget (struct obj_cgroup *objcg) +{ +return percpu_ref_tryget (&objcg->refcnt); +} + +static inline void obj_cgroup_get (struct obj_cgroup *objcg) +{ +percpu_ref_get (&objcg->refcnt); +} + +static inline void obj_cgroup_put (struct obj_cgroup *objcg) +{ +percpu_ref_put (&objcg->refcnt); +} + +/* +* After the initialization objcg->memcg is always pointing at +* a valid memcg, but can be atomically swapped to the parent memcg. +* +* The caller must ensure that the returned memcg won't be released: +* e.g. acquire the rcu_read_lock or css_set_lock. +*/ +static inline struct mem_cgroup *obj_cgroup_memcg (struct obj_cgroup *objcg) +{ +return READ_ONCE (objcg->memcg); +} +static inline void mem_cgroup_put (struct mem_cgroup *memcg) {if (memcg)
@@ -1370,6 +1416,11 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages);int __memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order);void __memcg_kmem_uncharge_page (struct page *page,int order); +struct obj_cgroup *get_obj_cgroup_from_current (void ); + +int obj_cgroup_charge (struct obj_cgroup *objcg,gfp_t gfp,size_t size); +void obj_cgroup_uncharge (struct obj_cgroup *objcg,size_t size); +extern struct static_key_false memcg_kmem_enabled_key;extern struct workqueue_struct *memcg_kmem_cache_wq;
mm/memcontrol.c +287 -1
356 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v5 7/19 mm: memcg/slab: allocate obj_cgroups for non-root slab pages
In reply to Roman GushchinAllocate and release memory to store obj_cgroup pointers for each non-root slab page. Reuse page->mem_cgroup pointer to store a pointer to the allocated space.
To distinguish between obj_cgroups and memcg pointers in case when it's not obvious which one is used (as in page_cgroup_ino()), let's always set the lowest bit in the obj_cgroup case.
--- include/linux/mm_types.h | 5 +++- include/linux/slab_def.h | 6 +++++ include/linux/slub_def.h | 5 ++++ mm/memcontrol.c | 17 +++++++++++--- mm/slab.h | 49 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 78 insertions(+), 4 deletions(-)
include/linux/mm_types.h +4 -1
@@ -198,7 +198,10 @@ struct page {atomic_t _refcount;#ifdef CONFIG_MEMCG -struct mem_cgroup *mem_cgroup; +union { +struct mem_cgroup *mem_cgroup; +struct obj_cgroup **obj_cgroups; + };#endif /*
include/linux/slab_def.h +6 -0
@@ -114,4 +114,10 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache, return reciprocal_divide(offset, cache->reciprocal_buffer_size); } +static inline int objs_per_slab_page (const struct kmem_cache *cache, +const struct page *page) +{ +return cache->num; +} +#endif /* _LINUX_SLAB_DEF_H */
include/linux/slub_def.h +5 -0
@@ -198,4 +198,9 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache, return __obj_to_index(cache, page_address(page), obj); } +static inline int objs_per_slab_page (const struct kmem_cache *cache, +const struct page *page) +{ +return page->objects; +}#endif /* _LINUX_SLUB_DEF_H */
mm/memcontrol.c +14 -3
@@ -569,10 +569,21 @@ ino_t page_cgroup_ino(struct page *page)unsigned long ino =0 ;rcu_read_lock (); -if (PageSlab (page) && !PageTail (page)) +if (PageSlab (page) && !PageTail (page)) { memcg =memcg_from_slab_page (page); -else - memcg =READ_ONCE (page->mem_cgroup); + }else { + memcg = page->mem_cgroup; + +/* +* The lowest bit set means that memcg isn't a valid +* memcg pointer, but a obj_cgroups pointer. +* In this case the page is shared and doesn't belong +* to any specific memory cgroup. +*/ +if ((unsigned long ) memcg &0x1UL ) + memcg =NULL ; + } +while (memcg && !(memcg->css.flags &CSS_ONLINE )) memcg =parent_mem_cgroup (memcg);if (memcg)
mm/slab.h +49 -0
85 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v5 8/19 mm: memcg/slab: save obj_cgroup for non-root slab objects
In reply to Roman GushchinStore the obj_cgroup pointer in the corresponding place of page->obj_cgroups for each allocated non-root slab object. Make sure that each allocated object holds a reference to obj_cgroup.
Objcg pointer is obtained from the memcg->objcg dereferencing in memcg_kmem_get_cache() and passed from pre_alloc_hook to post_alloc_hook. Then in case of successful allocation(s) it's getting stored in the page->obj_cgroups vector.
The objcg obtaining part look a bit bulky now, but it will be simplified by next commits in the series.
--- include/linux/memcontrol.h | 3 +- mm/memcontrol.c | 14 +++++++-- mm/slab.c | 18 +++++++----- mm/slab.h | 60 ++++++++++++++++++++++++++++++++++---- mm/slub.c | 14 +++++---- 5 files changed, 88 insertions(+), 21 deletions(-)
include/linux/memcontrol.h +2 -1
@@ -1406,7 +1406,8 @@ static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, }#endif -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep, +struct obj_cgroup **objcgp);void memcg_kmem_put_cache (struct kmem_cache *cachep);#ifdef CONFIG_MEMCG_KMEM
mm/memcontrol.c +12 -2
@@ -2916,7 +2916,8 @@ static inline bool memcg_kmem_bypass(void)* done with it, memcg_kmem_put_cache() must be called to release the * reference. */ -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep) +struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep, +struct obj_cgroup **objcgp) {struct mem_cgroup *memcg;struct kmem_cache *memcg_cachep;
@@ -2972,8 +2973,17 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)*/ if (unlikely (!memcg_cachep))memcg_schedule_kmem_cache_create (memcg, cachep); -else if (percpu_ref_tryget (&memcg_cachep->memcg_params.refcnt)) +else if (percpu_ref_tryget (&memcg_cachep->memcg_params.refcnt)) { +struct obj_cgroup *objcg =rcu_dereference (memcg->objcg); + +if (!objcg || !obj_cgroup_tryget (objcg)) { +percpu_ref_put (&memcg_cachep->memcg_params.refcnt); + goto out_unlock; + } + + *objcgp = objcg; cachep = memcg_cachep; + } out_unlock:rcu_read_unlock ();return cachep;
mm/slab.c +11 -7
@@ -3222,9 +3222,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,unsigned long save_flags;void *ptr;int slab_node =numa_mem_id (); +struct obj_cgroup *objcg =NULL ; flags &= gfp_allowed_mask; - cachep =slab_pre_alloc_hook (cachep, flags); + cachep =slab_pre_alloc_hook (cachep, &objcg,1 , flags);if (unlikely (!cachep))return NULL ;
@@ -3260,7 +3261,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size); -slab_post_alloc_hook (cachep, flags,1 , &ptr); +slab_post_alloc_hook (cachep, objcg, flags,1 , &ptr);return ptr; }
@@ -3301,9 +3302,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) {unsigned long save_flags;void *objp; +struct obj_cgroup *objcg =NULL ; flags &= gfp_allowed_mask; - cachep =slab_pre_alloc_hook (cachep, flags); + cachep =slab_pre_alloc_hook (cachep, &objcg,1 , flags);if (unlikely (!cachep))return NULL ;
@@ -3317,7 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size); -slab_post_alloc_hook (cachep, flags,1 , &objp); +slab_post_alloc_hook (cachep, objcg, flags,1 , &objp);return objp; }
@@ -3439,6 +3441,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller); +memcg_slab_free_hook (cachep,virt_to_head_page (objp), objp);/* * Skip calling cache_free_alien() when the platform is not numa.
@@ -3504,8 +3507,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) {size_t i; +struct obj_cgroup *objcg =NULL ; - s =slab_pre_alloc_hook (s, flags); + s =slab_pre_alloc_hook (s, &objcg, size, flags);if (!s)return 0 ;
@@ -3528,13 +3532,13 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size); -slab_post_alloc_hook (s, flags, size, p); +slab_post_alloc_hook (s, objcg, flags, size, p);/* FIXME: Trace call missing. Christoph would like a bulk variant */ return size; error:local_irq_enable ();cache_alloc_debugcheck_after_bulk (s, flags, i, p, _RET_IP_); -slab_post_alloc_hook (s, flags, i, p); +slab_post_alloc_hook (s, objcg, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ; }
mm/slab.h +54 -6
94 lines not on this page · raw message
mm/slub.c +9 -5
@@ -2738,8 +2738,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,struct kmem_cache_cpu *c;struct page *page;unsigned long tid; +struct obj_cgroup *objcg =NULL ; - s =slab_pre_alloc_hook (s, gfpflags); + s =slab_pre_alloc_hook (s, &objcg,1 , gfpflags);if (!s)return NULL ; redo:
@@ -2815,7 +2816,7 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size); -slab_post_alloc_hook (s, gfpflags,1 , &object); +slab_post_alloc_hook (s, objcg, gfpflags,1 , &object);return object; }
@@ -3020,6 +3021,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s,void *tail_obj = tail ? : head;struct kmem_cache_cpu *c;unsigned long tid; + +memcg_slab_free_hook (s, page, head); redo:/* * Determine the currently cpus per cpu slab.
@@ -3199,9 +3202,10 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, {struct kmem_cache_cpu *c;int i; +struct obj_cgroup *objcg =NULL ;/* memcg and kmem_cache debug support */ - s =slab_pre_alloc_hook (s, flags); + s =slab_pre_alloc_hook (s, &objcg, size, flags);if (unlikely (!s))return false;/*
@@ -3255,11 +3259,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, }/* memcg and kmem_cache debug support */ -slab_post_alloc_hook (s, flags, size, p); +slab_post_alloc_hook (s, objcg, flags, size, p);return i; error:local_irq_enable (); -slab_post_alloc_hook (s, flags, i, p); +slab_post_alloc_hook (s, objcg, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ; }
Signature · 1 line
2.25.4
PATCH v5 9/19 mm: memcg/slab: charge individual slab objects instead of pages
In reply to Roman GushchinSwitch to per-object accounting of non-root slab objects.
Charging is performed using obj_cgroup API in the pre_alloc hook. Obj_cgroup is charged with the size of the object and the size of metadata: as now it's the size of an obj_cgroup pointer. If the amount of memory has been charged successfully, the actual allocation code is executed. Otherwise, -ENOMEM is returned.
In the post_alloc hook if the actual allocation succeeded, corresponding vmstats are bumped and the obj_cgroup pointer is saved. Otherwise, the charge is canceled.
On the free path obj_cgroup pointer is obtained and used to uncharge the size of the releasing object.
Memcg and lruvec counters are now representing only memory used by active slab objects and do not include the free space. The free space is shared and doesn't belong to any specific cgroup.
Global per-node slab vmstats are still modified from (un)charge_slab_page() functions. The idea is to keep all slab pages accounted as slab pages on system level.
--- mm/slab.h | 173 ++++++++++++++++++++++++------------------------------ 1 file changed, 77 insertions(+), 96 deletions(-)
mm/slab.h +77 -96
@@ -352,72 +352,6 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)return NULL ; } -/* -* Charge the slab page belonging to the non-root kmem_cache. -* Can be called for non-root kmem_caches only. -*/ -static __always_inline int memcg_charge_slab(struct page *page, -gfp_t gfp,int order, -struct kmem_cache *s) -{ -unsigned int nr_pages =1 << order; -struct mem_cgroup *memcg; -struct lruvec *lruvec; -int ret; - -rcu_read_lock (); - memcg =READ_ONCE (s->memcg_params.memcg); -while (memcg && !css_tryget_online (&memcg->css)) - memcg =parent_mem_cgroup (memcg); -rcu_read_unlock (); - -if (unlikely (!memcg ||mem_cgroup_is_root (memcg))) { -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - nr_pages <<PAGE_SHIFT ); -percpu_ref_get_many (&s->memcg_params.refcnt, nr_pages); -return 0 ; - } - - ret =memcg_kmem_charge (memcg, gfp, nr_pages); -if (ret) - goto out; - - lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); -mod_lruvec_state (lruvec,cache_vmstat_idx (s), nr_pages <<PAGE_SHIFT ); - -percpu_ref_get_many (&s->memcg_params.refcnt, nr_pages); -out: -css_put (&memcg->css); -return ret; -} - -/* -* Uncharge a slab page belonging to a non-root kmem_cache. -* Can be called for non-root kmem_caches only. -*/ -static __always_inline void memcg_uncharge_slab(struct page *page, int order, -struct kmem_cache *s) -{ -unsigned int nr_pages =1 << order; -struct mem_cgroup *memcg; -struct lruvec *lruvec; - -rcu_read_lock (); - memcg =READ_ONCE (s->memcg_params.memcg); -if (likely (!mem_cgroup_is_root (memcg))) { - lruvec =mem_cgroup_lruvec (memcg,page_pgdat (page)); -mod_lruvec_state (lruvec,cache_vmstat_idx (s), - -(nr_pages <<PAGE_SHIFT )); -memcg_kmem_uncharge (memcg, nr_pages); - }else { -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -(nr_pages <<PAGE_SHIFT )); - } -rcu_read_unlock (); - -percpu_ref_put_many (&s->memcg_params.refcnt, nr_pages); -} -static inline int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp) {
@@ -438,6 +372,47 @@ static inline void memcg_free_page_obj_cgroups(struct page *page) page->obj_cgroups =NULL ; } +static inline size_t obj_full_size (struct kmem_cache *s) +{ +/* +* For each accounted object there is an extra space which is used +* to store obj_cgroup membership. Charge it too. +*/ +return s->size +sizeof (struct obj_cgroup *); +} + +static inline struct kmem_cache *memcg_slab_pre_alloc_hook (struct kmem_cache *s, +struct obj_cgroup **objcgp, +size_t objects,gfp_t flags) +{ +struct kmem_cache *cachep; + + cachep =memcg_kmem_get_cache (s, objcgp); +if (is_root_cache (cachep)) +return s; + +if (obj_cgroup_charge (*objcgp, flags, objects *obj_full_size (s))) { +memcg_kmem_put_cache (cachep); + cachep =NULL ; + } + +return cachep; +} + +static inline void mod_objcg_state (struct obj_cgroup *objcg, +struct pglist_data *pgdat, +int idx,int nr) +{ +struct mem_cgroup *memcg; +struct lruvec *lruvec; + +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg); + lruvec =mem_cgroup_lruvec (memcg, pgdat); +mod_memcg_lruvec_state (lruvec, idx, nr); +rcu_read_unlock (); +} +static inline void memcg_slab_post_alloc_hook (struct kmem_cache *s,struct obj_cgroup *objcg,size_t size,void **p)
@@ -452,6 +427,10 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, off =obj_to_index (s, page, p[i]);obj_cgroup_get (objcg);page_obj_cgroups (page)[off] = objcg; +mod_objcg_state (objcg,page_pgdat (page), +cache_vmstat_idx (s),obj_full_size (s)); + }else { +obj_cgroup_uncharge (objcg,obj_full_size (s)); } }obj_cgroup_put (objcg);
@@ -470,6 +449,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, off =obj_to_index (s, page, p); objcg =page_obj_cgroups (page)[off];page_obj_cgroups (page)[off] =NULL ; + +obj_cgroup_uncharge (objcg,obj_full_size (s)); +mod_objcg_state (objcg,page_pgdat (page),cache_vmstat_idx (s), + -obj_full_size (s)); +obj_cgroup_put (objcg); }
@@ -511,17 +495,6 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)return NULL ; } -static inline int memcg_charge_slab (struct page *page,gfp_t gfp,int order, -struct kmem_cache *s) -{ -return 0 ; -} - -static inline void memcg_uncharge_slab (struct page *page,int order, -struct kmem_cache *s) -{ -} -static inline int memcg_alloc_page_obj_cgroups (struct page *page,struct kmem_cache *s,gfp_t gfp) {
@@ -532,6 +505,13 @@ static inline void memcg_free_page_obj_cgroups(struct page *page) { } +static inline struct kmem_cache *memcg_slab_pre_alloc_hook (struct kmem_cache *s, +struct obj_cgroup **objcgp, +size_t objects,gfp_t flags) +{ +return NULL ; +} +static inline void memcg_slab_post_alloc_hook (struct kmem_cache *s,struct obj_cgroup *objcg,size_t size,void **p)
@@ -569,32 +549,33 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - int ret; - -if (is_root_cache (s)) { -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), -PAGE_SIZE << order); -return 0 ; - } +#ifdef CONFIG_MEMCG_KMEM + if (memcg_kmem_enabled() && !is_root_cache(s)) { + int ret; - ret =memcg_alloc_page_obj_cgroups (page, s, gfp); -if (ret) -return ret; + ret =memcg_alloc_page_obj_cgroups (page, s, gfp); +if (ret) +return ret; -return memcg_charge_slab (page, gfp, order, s); +percpu_ref_get_many (&s->memcg_params.refcnt,1 << order); + } +#endif +mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), +PAGE_SIZE << order); +return 0 ; } static __always_inline void uncharge_slab_page(struct page *page, int order,struct kmem_cache *s) { -if (is_root_cache (s)) { -mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), - -(PAGE_SIZE << order)); -return ; +#ifdef CONFIG_MEMCG_KMEM +if (memcg_kmem_enabled () && !is_root_cache (s)) { +memcg_free_page_obj_cgroups (page); +percpu_ref_put_many (&s->memcg_params.refcnt,1 << order); } - -memcg_free_page_obj_cgroups (page); -memcg_uncharge_slab (page, order, s); +#endif +mod_node_page_state (page_pgdat (page),cache_vmstat_idx (s), + -(PAGE_SIZE << order)); }static inline struct kmem_cache *cache_from_obj (struct kmem_cache *s,void *x)
@@ -666,7 +647,7 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s,if (memcg_kmem_enabled () && ((flags & __GFP_ACCOUNT) || (s->flags &SLAB_ACCOUNT ))) -return memcg_kmem_get_cache (s, objcgp); +return memcg_slab_pre_alloc_hook (s, objcgp, size, flags);return s; }
Signature · 1 line
2.25.4
PATCH v5 10/19 mm: memcg/slab: deprecate memory.kmem.slabinfo
In reply to Roman GushchinDeprecate memory.kmem.slabinfo.
An empty file will be presented if corresponding config options are enabled.
The interface is implementation dependent, isn't present in cgroup v2, and is generally useful only for core mm debugging purposes. In other words, it doesn't provide any value for the absolute majority of users.
A drgn-based replacement can be found in tools/cgroup/slabinfo.py . It does support cgroup v1 and v2, mimics memory.kmem.slabinfo output and also allows to get any additional information without a need to recompile the kernel.
If a drgn-based solution is too slow for a task, a bpf-based tracing tool can be used, which can easily keep track of all slab allocations belonging to a memory cgroup.
--- mm/memcontrol.c | 3 --- mm/slab_common.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 30 deletions(-)
mm/memcontrol.c +0 -3
@@ -5062,9 +5062,6 @@ static struct cftype mem_cgroup_legacy_files[] = { (defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG)) { .name = "kmem.slabinfo", - .seq_start = memcg_slab_start, - .seq_next = memcg_slab_next, - .seq_stop = memcg_slab_stop, .seq_show = memcg_slab_show, },#endif
mm/slab_common.c +4 -27
@@ -1523,35 +1523,12 @@ void dump_unreclaimable_slab(void) }#if defined(CONFIG_MEMCG_KMEM ) -void *memcg_slab_start (struct seq_file *m,loff_t *pos) -{ -struct mem_cgroup *memcg =mem_cgroup_from_seq (m); - -mutex_lock (&slab_mutex); -return seq_list_start (&memcg->kmem_caches, *pos); -} - -void *memcg_slab_next (struct seq_file *m,void *p,loff_t *pos) -{ -struct mem_cgroup *memcg =mem_cgroup_from_seq (m); - -return seq_list_next (p, &memcg->kmem_caches, pos); -} - -void memcg_slab_stop (struct seq_file *m,void *p) -{ -mutex_unlock (&slab_mutex); -} -int memcg_slab_show (struct seq_file *m,void *p) { - struct kmem_cache *s = list_entry(p, struct kmem_cache, - memcg_params.kmem_caches_node); -struct mem_cgroup *memcg =mem_cgroup_from_seq (m); - -if (p == memcg->kmem_caches.next) -print_slabinfo_header (m); -cache_show (s, m); +/* +* Deprecated. +* Please, take a look at tools/cgroup/slabinfo.py . +*/ return 0 ; }#endif
Signature · 1 line
2.25.4
PATCH v5 11/19 mm: memcg/slab: move memcg_kmem_bypass() to memcontrol.h
In reply to Roman GushchinTo make the memcg_kmem_bypass() function available outside of the memcontrol.c, let's move it to memcontrol.h. The function is small and nicely fits into static inline sort of functions.
It will be used from the slab code.
--- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 7 ------- 2 files changed, 7 insertions(+), 7 deletions(-)
include/linux/memcontrol.h +7 -0
@@ -1442,6 +1442,13 @@ static inline bool memcg_kmem_enabled(void)return static_branch_unlikely (&memcg_kmem_enabled_key); } +static inline bool memcg_kmem_bypass (void ) +{ +if (in_interrupt () || !current->mm || (current->flags &PF_KTHREAD )) +return true; +return false; +} +static inline int memcg_kmem_charge_page (struct page *page,gfp_t gfp,int order) {
mm/memcontrol.c +0 -7
@@ -2893,13 +2893,6 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg,queue_work (memcg_kmem_cache_wq, &cw->work); } -static inline bool memcg_kmem_bypass (void ) -{ -if (in_interrupt () || !current->mm || (current->flags &PF_KTHREAD )) -return true; -return false; -} -/** * memcg_kmem_get_cache: select the correct per-memcg cache for allocation * @cachep: the original global kmem cache
Signature · 1 line
2.25.4
PATCH v5 12/19 mm: memcg/slab: use a single set of kmem_caches for all accounted allocations
In reply to Roman GushchinThis is fairly big but mostly red patch, which makes all accounted slab allocations use a single set of kmem_caches instead of creating a separate set for each memory cgroup.
Because the number of non-root kmem_caches is now capped by the number of root kmem_caches, there is no need to shrink or destroy them prematurely. They can be perfectly destroyed together with their root counterparts. This allows to dramatically simplify the management of non-root kmem_caches and delete a ton of code.
This patch performs the following changes:
1) introduces memcg_params.memcg_cache pointer to represent the kmem_cache which will be used for all non-root allocations
2) reuses the existing memcg kmem_cache creation mechanism to create memcg kmem_cache on the first allocation attempt
3) memcg kmem_caches are named <kmemcache_name>-memcg, e.g. dentry-memcg
4) simplifies memcg_kmem_get_cache() to just return memcg kmem_cache or schedule it's creation and return the root cache
5) removes almost all non-root kmem_cache management code (separate refcounter, reparenting, shrinking, etc)
6) makes slab debugfs to display root_mem_cgroup css id and never show :dead and :deact flags in the memcg_slabinfo attribute.
Following patches in the series will simplify the kmem_cache creation.
--- include/linux/memcontrol.h | 5 +- include/linux/slab.h | 5 +- mm/memcontrol.c | 163 +++----------- mm/slab.c | 16 +- mm/slab.h | 145 ++++--------- mm/slab_common.c | 426 ++++--------------------------------- mm/slub.c | 38 +--- 7 files changed, 128 insertions(+), 670 deletions(-)
include/linux/memcontrol.h +1 -4
@@ -319,7 +319,6 @@ struct mem_cgroup {/* Index in the kmem_cache->memcg_params.memcg_caches array */ int kmemcg_id;enum memcg_kmem_state kmem_state; -struct list_head kmem_caches;struct obj_cgroup __rcu *objcg;struct list_head objcg_list;#endif
@@ -1406,9 +1405,7 @@ static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, }#endif -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep, -struct obj_cgroup **objcgp); -void memcg_kmem_put_cache (struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep);#ifdef CONFIG_MEMCG_KMEM int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp,
include/linux/slab.h +1 -4
@@ -155,8 +155,7 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,void kmem_cache_destroy (struct kmem_cache *);int kmem_cache_shrink (struct kmem_cache *); -void memcg_create_kmem_cache (struct mem_cgroup *,struct kmem_cache *); -void memcg_deactivate_kmem_caches (struct mem_cgroup *,struct mem_cgroup *); +void memcg_create_kmem_cache (struct kmem_cache *cachep);/* * Please use this macro to create slab caches. Simply specify the
@@ -578,8 +577,6 @@ static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)return __kmalloc_node (size, flags, node); } -int memcg_update_all_caches (int num_memcgs); -/** * kmalloc_array - allocate memory for an array. * @n: number of elements.
mm/memcontrol.c +32 -131
268 lines not on this page · raw message
mm/slab.c +3 -13
@@ -1239,7 +1239,7 @@ void __init kmem_cache_init(void) nr_node_ids *sizeof (struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0);list_add (&kmem_cache->list, &slab_caches); -memcg_link_cache (kmem_cache,NULL ); +memcg_link_cache (kmem_cache); slab_state =PARTIAL ;/*
@@ -2243,17 +2243,6 @@ int __kmem_cache_shrink(struct kmem_cache *cachep)return (ret ?1 :0 ); } -#ifdef CONFIG_MEMCG -void __kmemcg_cache_deactivate (struct kmem_cache *cachep) -{ -__kmem_cache_shrink (cachep); -} - -void __kmemcg_cache_deactivate_after_rcu (struct kmem_cache *s) -{ -} -#endif -int __kmem_cache_shutdown (struct kmem_cache *cachep) {return __kmem_cache_shrink (cachep);
@@ -3861,7 +3850,8 @@ static int do_tune_cpucache(struct kmem_cache *cachep, int limit,return ret;lockdep_assert_held (&slab_mutex); -for_each_memcg_cache (c, cachep) { + c =memcg_cache (cachep); +if (c) {/* return value determined by the root cache only */ __do_tune_cpucache (c, limit, batchcount, shared, gfp); }
mm/slab.h +45 -100
256 lines not on this page · raw message
mm/slab_common.c +41 -385
588 lines not on this page · raw message
mm/slub.c +5 -33
@@ -4136,36 +4136,6 @@ int __kmem_cache_shrink(struct kmem_cache *s)return ret; } -#ifdef CONFIG_MEMCG -void __kmemcg_cache_deactivate_after_rcu (struct kmem_cache *s) -{ -/* -* Called with all the locks held after a sched RCU grace period. -* Even if @s becomes empty after shrinking, we can't know that @s -* doesn't have allocations already in-flight and thus can't -* destroy @s until the associated memcg is released. -* -* However, let's remove the sysfs files for empty caches here. -* Each cache has a lot of interface files which aren't -* particularly useful for empty draining caches; otherwise, we can -* easily end up with millions of unnecessary sysfs files on -* systems which have a lot of memory and transient cgroups. -*/ -if (!__kmem_cache_shrink (s)) -sysfs_slab_remove (s); -} - -void __kmemcg_cache_deactivate (struct kmem_cache *s) -{ -/* -* Disable empty slabs caching. Used to avoid pinning offline -* memory cgroups by kmem pages that can be freed. -*/ -slub_set_cpu_partial (s,0 ); - s->min_partial =0 ; -} -#endif /* CONFIG_MEMCG */ -static int slab_mem_going_offline_callback (void *arg) {struct kmem_cache *s;
@@ -4322,7 +4292,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) }slab_init_memcg_params (s);list_add (&s->list, &slab_caches); -memcg_link_cache (s,NULL ); +memcg_link_cache (s);return s; }
@@ -4390,7 +4360,8 @@ __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, s->object_size =max (s->object_size, size); s->inuse =max (s->inuse,ALIGN (size,sizeof (void *))); -for_each_memcg_cache (c, s) { + c =memcg_cache (s); +if (c) { c->object_size = s->object_size; c->inuse =max (c->inuse,ALIGN (size,sizeof (void *))); }
@@ -5645,7 +5616,8 @@ static ssize_t slab_attr_store(struct kobject *kobj,* directly either failed or succeeded, in which case we loop * through the descendants with best-effort propagation. */ -for_each_memcg_cache (c, s) + c =memcg_cache (s); +if (c) attribute->store (c, buf, len);mutex_unlock (&slab_mutex); }
Signature · 1 line
2.25.4
PATCH v5 13/19 mm: memcg/slab: simplify memcg cache creation
In reply to Roman GushchinBecause the number of non-root kmem_caches doesn't depend on the number of memory cgroups anymore and is generally not very big, there is no more need for a dedicated workqueue.
Also, as there is no more need to pass any arguments to the memcg_create_kmem_cache() except the root kmem_cache, it's possible to just embed the work structure into the kmem_cache and avoid the dynamic allocation of the work structure.
This will also simplify the synchronization: for each root kmem_cache there is only one work. So there will be no more concurrent attempts to create a non-root kmem_cache for a root kmem_cache: the second and all following attempts to queue the work will fail.
On the kmem_cache destruction path there is no more need to call the expensive flush_workqueue() and wait for all pending works to be finished. Instead, cancel_work_sync() can be used to cancel/wait for only one work.
--- include/linux/memcontrol.h | 1 - mm/memcontrol.c | 48 +------------------------------------- mm/slab.h | 2 ++ mm/slab_common.c | 22 +++++++++-------- 4 files changed, 15 insertions(+), 58 deletions(-)
include/linux/memcontrol.h +0 -1
@@ -1420,7 +1420,6 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size);void obj_cgroup_uncharge (struct obj_cgroup *objcg,size_t size);extern struct static_key_false memcg_kmem_enabled_key; -extern struct workqueue_struct *memcg_kmem_cache_wq;extern int memcg_nr_cache_ids;void memcg_get_cache_ids (void );
mm/memcontrol.c +1 -47
72 lines not on this page · raw message
mm/slab.h +2 -0
@@ -45,12 +45,14 @@ struct kmem_cache {* @memcg_cache: pointer to memcg kmem cache, used by all non-root memory * cgroups. * @root_caches_node: list node for slab_root_caches list. +* @work: work struct used to create the non-root cache. */ struct memcg_cache_params {struct kmem_cache *root_cache;struct kmem_cache *memcg_cache;struct list_head __root_caches_node; +struct work_struct work; };#endif /* CONFIG_SLOB */
mm/slab_common.c +12 -10
@@ -132,10 +132,18 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,LIST_HEAD (slab_root_caches); +static void memcg_kmem_cache_create_func (struct work_struct *work) +{ + struct kmem_cache *cachep = container_of(work, struct kmem_cache, + memcg_params.work); +memcg_create_kmem_cache (cachep); +} +void slab_init_memcg_params (struct kmem_cache *s) { s->memcg_params.root_cache =NULL ; s->memcg_params.memcg_cache =NULL ; +INIT_WORK (&s->memcg_params.work, memcg_kmem_cache_create_func); } static void init_memcg_params(struct kmem_cache *s,
@@ -584,15 +592,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s) return 0; } -static void flush_memcg_workqueue (struct kmem_cache *s) +static void cancel_memcg_cache_creation (struct kmem_cache *s) { -/* -* SLAB and SLUB create memcg kmem_caches through workqueue and SLUB -* deactivates the memcg kmem_caches through workqueue. Make sure all -* previous workitems on workqueue are processed. -*/ -if (likely (memcg_kmem_cache_wq)) -flush_workqueue (memcg_kmem_cache_wq); +cancel_work_sync (&s->memcg_params.work); }#else static inline int shutdown_memcg_caches (struct kmem_cache *s)
@@ -600,7 +602,7 @@ static inline int shutdown_memcg_caches(struct kmem_cache *s) return 0; } -static inline void flush_memcg_workqueue (struct kmem_cache *s) +static inline void cancel_memcg_cache_creation (struct kmem_cache *s) { }#endif /* CONFIG_MEMCG_KMEM */
@@ -619,7 +621,7 @@ void kmem_cache_destroy(struct kmem_cache *s)if (unlikely (!s))return ; -flush_memcg_workqueue (s); +cancel_memcg_cache_creation (s);get_online_cpus ();get_online_mems ();
Signature · 1 line
2.25.4
PATCH v5 14/19 mm: memcg/slab: remove memcg_kmem_get_cache()
In reply to Roman GushchinThe memcg_kmem_get_cache() function became really trivial, so let's just inline it into the single call point:memcg_slab_pre_alloc_hook().
It will make the code less bulky and can also help the compiler to generate a better code.
--- include/linux/memcontrol.h | 2 -- mm/memcontrol.c | 25 +------------------------ mm/slab.h | 11 +++++++++-- mm/slab_common.c | 2 +- 4 files changed, 11 insertions(+), 29 deletions(-)
include/linux/memcontrol.h +0 -2
@@ -1405,8 +1405,6 @@ static inline void memcg_set_shrinker_bit(struct mem_cgroup *memcg, }#endif -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep); -#ifdef CONFIG_MEMCG_KMEM int __memcg_kmem_charge (struct mem_cgroup *memcg,gfp_t gfp,unsigned int nr_pages);
mm/memcontrol.c +1 -24
@@ -393,7 +393,7 @@ void memcg_put_cache_ids(void)/* * A lot of the calls to the cache allocation functions are expected to be -* inlined by the compiler. Since the calls to memcg_kmem_get_cache are +* inlined by the compiler. Since the calls to memcg_slab_pre_alloc_hook() are * conditional to this static branch, we'll have to allow modules that does * kmem_cache_alloc and the such to see this symbol as well */
@@ -2850,29 +2850,6 @@ static void memcg_free_cache_id(int id)ida_simple_remove (&memcg_cache_ida, id); } -/** -* memcg_kmem_get_cache: select memcg or root cache for allocation -* @cachep: the original global kmem cache -* -* Return the kmem_cache we're supposed to use for a slab allocation. -* -* If the cache does not exist yet, if we are the first user of it, we -* create it asynchronously in a workqueue and let the current allocation -* go through with the original cache. -*/ -struct kmem_cache *memcg_kmem_get_cache (struct kmem_cache *cachep) -{ -struct kmem_cache *memcg_cachep; - - memcg_cachep =READ_ONCE (cachep->memcg_params.memcg_cache); -if (unlikely (!memcg_cachep)) { -queue_work (system_wq, &cachep->memcg_params.work); -return cachep; - } - -return memcg_cachep; -} -/** * __memcg_kmem_charge: charge a number of kernel pages to a memcg * @memcg: memory cgroup to charge
mm/slab.h +9 -2
@@ -333,9 +333,16 @@ static inline struct kmem_cache *memcg_slab_pre_alloc_hook(struct kmem_cache *s,if (memcg_kmem_bypass ())return s; - cachep =memcg_kmem_get_cache (s); -if (is_root_cache (cachep)) + cachep =READ_ONCE (s->memcg_params.memcg_cache); +if (unlikely (!cachep)) { +/* +* If memcg cache does not exist yet, we schedule it's +* asynchronous creation and let the current allocation +* go through with the root cache. +*/ +queue_work (system_wq, &s->memcg_params.work);return s; + } objcg =get_obj_cgroup_from_current ();if (!objcg)
mm/slab_common.c +1 -1
@@ -568,7 +568,7 @@ void memcg_create_kmem_cache(struct kmem_cache *root_cache) }/* -* Since readers won't lock (see memcg_kmem_get_cache()), we need a +* Since readers won't lock (see memcg_slab_pre_alloc_hook()), we need a * barrier here to ensure nobody will see the kmem_cache partially * initialized. */
Signature · 1 line
2.25.4
PATCH v5 15/19 mm: memcg/slab: deprecate slab_root_caches
In reply to Roman GushchinCurrently there are two lists of kmem_caches:
1) slab_caches, which contains all kmem_caches,
2) slab_root_caches, which contains only root kmem_caches.
And there is some preprocessor magic to have a single list if CONFIG_MEMCG_KMEM isn't enabled.
It was required earlier because the number of non-root kmem_caches was proportional to the number of memory cgroups and could reach really big values. Now, when it cannot exceed the number of root kmem_caches, there is really no reason to maintain two lists.
We never iterate over the slab_root_caches list on any hot paths, so it's perfectly fine to iterate over slab_caches and filter out non-root kmem_caches.
It allows to remove a lot of config-dependent code and two pointers from the kmem_cache structure.
--- mm/slab.c | 1 - mm/slab.h | 17 ----------------- mm/slab_common.c | 37 ++++++++----------------------------- mm/slub.c | 1 - 4 files changed, 8 insertions(+), 48 deletions(-)
mm/slab.c +0 -1
@@ -1239,7 +1239,6 @@ void __init kmem_cache_init(void) nr_node_ids *sizeof (struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0);list_add (&kmem_cache->list, &slab_caches); -memcg_link_cache (kmem_cache); slab_state =PARTIAL ;/*
mm/slab.h +0 -17
@@ -44,14 +44,12 @@ struct kmem_cache {* * @memcg_cache: pointer to memcg kmem cache, used by all non-root memory * cgroups. -* @root_caches_node: list node for slab_root_caches list. * @work: work struct used to create the non-root cache. */ struct memcg_cache_params {struct kmem_cache *root_cache;struct kmem_cache *memcg_cache; -struct list_head __root_caches_node;struct work_struct work; };#endif /* CONFIG_SLOB */
@@ -235,11 +233,6 @@ static inline int cache_vmstat_idx(struct kmem_cache *s) }#ifdef CONFIG_MEMCG_KMEM - -/* List of all root caches. */ -extern struct list_head slab_root_caches; -#define root_caches_node memcg_params.__root_caches_node -static inline bool is_root_cache (struct kmem_cache *s) {return !s->memcg_params.root_cache;
@@ -415,14 +408,8 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, }extern void slab_init_memcg_params (struct kmem_cache *); -extern void memcg_link_cache (struct kmem_cache *s);#else /* CONFIG_MEMCG_KMEM */ - -/* If !memcg, all caches are root. */ -#define slab_root_caches slab_caches -#define root_caches_node list -static inline bool is_root_cache (struct kmem_cache *s) {return true;
@@ -491,10 +478,6 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache (struct kmem_cache *s) -{ -} -#endif /* CONFIG_MEMCG_KMEM */ static inline struct kmem_cache *virt_to_cache (const void *obj)
mm/slab_common.c +8 -29
@@ -129,9 +129,6 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, }#ifdef CONFIG_MEMCG_KMEM - -LIST_HEAD (slab_root_caches); -static void memcg_kmem_cache_create_func (struct work_struct *work) { struct kmem_cache *cachep = container_of(work, struct kmem_cache,
@@ -154,27 +151,11 @@ static void init_memcg_params(struct kmem_cache *s, else slab_init_memcg_params(s); } - -void memcg_link_cache (struct kmem_cache *s) -{ -if (is_root_cache (s)) -list_add (&s->root_caches_node, &slab_root_caches); -} - -static void memcg_unlink_cache (struct kmem_cache *s) -{ -if (is_root_cache (s)) -list_del (&s->root_caches_node); -}#else static inline void init_memcg_params (struct kmem_cache *s,struct kmem_cache *root_cache) { } - -static inline void memcg_unlink_cache (struct kmem_cache *s) -{ -}#endif /* CONFIG_MEMCG_KMEM */ /*
@@ -251,7 +232,7 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,if (flags &SLAB_NEVER_MERGE )return NULL ; -list_for_each_entry_reverse (s, &slab_root_caches, root_caches_node) { +list_for_each_entry_reverse (s, &slab_caches, list) {if (slab_unmergeable (s))continue ;
@@ -310,7 +291,6 @@ static struct kmem_cache *create_cache(const char *name, s->refcount =1 ;list_add (&s->list, &slab_caches); -memcg_link_cache (s); out:if (err)return ERR_PTR (err);
@@ -505,7 +485,6 @@ static int shutdown_cache(struct kmem_cache *s)if (__kmem_cache_shutdown (s) !=0 )return -EBUSY ; -memcg_unlink_cache (s);list_del (&s->list);if (s->flags &SLAB_TYPESAFE_BY_RCU ) {
@@ -749,7 +728,6 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name,create_boot_cache (s, name, size, flags, useroffset, usersize);list_add (&s->list, &slab_caches); -memcg_link_cache (s); s->refcount =1 ;return s; }
@@ -1090,12 +1068,12 @@ static void print_slabinfo_header(struct seq_file *m)void *slab_start (struct seq_file *m,loff_t *pos) {mutex_lock (&slab_mutex); -return seq_list_start (&slab_root_caches, *pos); +return seq_list_start (&slab_caches, *pos); }void *slab_next (struct seq_file *m,void *p,loff_t *pos) { -return seq_list_next (p, &slab_root_caches, pos); +return seq_list_next (p, &slab_caches, pos); }void slab_stop (struct seq_file *m,void *p)
@@ -1148,11 +1126,12 @@ static void cache_show(struct kmem_cache *s, struct seq_file *m)static int slab_show (struct seq_file *m,void *p) { - struct kmem_cache *s = list_entry(p, struct kmem_cache, root_caches_node); + struct kmem_cache *s = list_entry(p, struct kmem_cache, list); -if (p == slab_root_caches.next) +if (p == slab_caches.next)print_slabinfo_header (m); -cache_show (s, m); +if (is_root_cache (s)) +cache_show (s, m);return 0 ; }
@@ -1254,7 +1233,7 @@ static int memcg_slabinfo_show(struct seq_file *m, void *unused)mutex_lock (&slab_mutex);seq_puts (m,"# <name> <css_id[:dead|deact]> <active_objs> <num_objs>" );seq_puts (m," <active_slabs> <num_slabs>\n" ); -list_for_each_entry (s, &slab_root_caches, root_caches_node) { +list_for_each_entry (s, &slab_caches, list) {/* * Skip kmem caches that don't have the memcg cache. */
mm/slub.c +0 -1
@@ -4292,7 +4292,6 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) }slab_init_memcg_params (s);list_add (&s->list, &slab_caches); -memcg_link_cache (s);return s; }
Signature · 1 line
2.25.4
PATCH v5 16/19 mm: memcg/slab: remove redundant check in memcg_accumulate_slabinfo()
In reply to Roman Gushchinmemcg_accumulate_slabinfo() is never called with a non-root kmem_cache as a first argument, so the is_root_cache(s) check is redundant and can be removed without any functional change.
--- mm/slab_common.c | 3 --- 1 file changed, 3 deletions(-)
mm/slab_common.c +0 -3
@@ -1087,9 +1087,6 @@ memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info)struct kmem_cache *c;struct slabinfo sinfo; -if (!is_root_cache (s)) -return ; - c =memcg_cache (s);if (c) {memset (&sinfo,0 ,sizeof (sinfo));
Signature · 1 line
2.25.4
PATCH v5 17/19 mm: memcg/slab: use a single set of kmem_caches for all allocations
In reply to Roman GushchinInstead of having two sets of kmem_caches: one for system-wide and non-accounted allocations and the second one shared by all accounted allocations, we can use just one.
The idea is simple: space for obj_cgroup metadata can be allocated on demand and filled only for accounted allocations.
It allows to remove a bunch of code which is required to handle kmem_cache clones for accounted allocations. There is no more need to create them, accumulate statistics, propagate attributes, etc. It's a quite significant simplification.
Also, because the total number of slab_caches is reduced almost twice (not all kmem_caches have a memcg clone), some additional memory savings are expected. On my devvm it additionally saves about 3.5% of slab memory.
--- include/linux/slab.h | 2 - include/linux/slab_def.h | 3 - include/linux/slub_def.h | 10 -- mm/memcontrol.c | 5 +- mm/slab.c | 41 +------ mm/slab.h | 179 +++++++----------------------- mm/slab_common.c | 230 +-------------------------------------- mm/slub.c | 163 +-------------------------- 8 files changed, 57 insertions(+), 576 deletions(-)
include/linux/slab.h +0 -2
@@ -155,8 +155,6 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name,void kmem_cache_destroy (struct kmem_cache *);int kmem_cache_shrink (struct kmem_cache *); -void memcg_create_kmem_cache (struct kmem_cache *cachep); -/* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above.
include/linux/slab_def.h +0 -3
@@ -72,9 +72,6 @@ struct kmem_cache {int obj_offset;#endif /* CONFIG_DEBUG_SLAB */ -#ifdef CONFIG_MEMCG -struct memcg_cache_params memcg_params; -#endif #ifdef CONFIG_KASAN struct kasan_cache kasan_info;#endif
include/linux/slub_def.h +0 -10
@@ -108,17 +108,7 @@ struct kmem_cache {struct list_head list;/* List of slab caches */ #ifdef CONFIG_SYSFS struct kobject kobj;/* For sysfs */ -struct work_struct kobj_remove_work;#endif -#ifdef CONFIG_MEMCG -struct memcg_cache_params memcg_params; -/* For propagation, maximum size of a stored attr */ -unsigned int max_attr_size; -#ifdef CONFIG_SYSFS -struct kset *memcg_kset; -#endif -#endif -#ifdef CONFIG_SLAB_FREELIST_HARDENED unsigned long random;#endif
mm/memcontrol.c +4 -1
@@ -2776,7 +2776,10 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p) off =obj_to_index (page->slab_cache, page, p); objcg =page_obj_cgroups (page)[off]; -return obj_cgroup_memcg (objcg); +if (objcg) +return obj_cgroup_memcg (objcg); + +return NULL ; }/* All other pages use page->mem_cgroup */
mm/slab.c +3 -38
65 lines not on this page · raw message
mm/slab.h +41 -138
304 lines not on this page · raw message
mm/slab_common.c +5 -225
326 lines not on this page · raw message
mm/slub.c +4 -159
254 lines not on this page · raw message
Signature · 1 line
2.25.4
Re: [PATCH v5 17/19] mm: memcg/slab: use a single set of kmem_caches for all allocations
Quoted 4 lines · Roman Gushchin
mm/slab.h +0
-0
...
mm/slab.h static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) · line 435
Quoted 9 lines
mm/slab.h +1
-2@@ -526,8 +430,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)* When kmemcg is not being used, both assignments should return the * same value. but we don't want to pay the assignment price in that * case. If it is not compiled in, the compiler should be smart enough -* to not do even the assignment. In that case, slab_equal_or_root -* will also be a constant. +* to not do even the assignment. */ if (!memcg_kmem_enabled() &&
Just realized that this test can go away - we don't have to call virt_to_cache() due to kmemcg if there is just a single cache.
Quoted 10 lines
mm/slab.h +1
-1!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) &&@@ -535,7 +438,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)return s; cachep =virt_to_cache (x); -WARN_ONCE (cachep && !slab_equal_or_root (cachep, s), +WARN_ONCE (cachep && cachep != s,"%s: Wrong slab cache. %s but object is from %s\n" , __func__, s->name, cachep->name);return cachep;
Re: [PATCH v5 17/19] mm: memcg/slab: use a single set of kmem_caches for all allocations
Quoted 6 lines · Vlastimil Babka
Quoted 4 lines · Roman Gushchin
mm/slab.h +0
-0...
Quoted 12 lines
mm/slab.h static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) · line 435
Quoted 9 lines
mm/slab.h +1
-2@@ -526,8 +430,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)* When kmemcg is not being used, both assignments should return the * same value. but we don't want to pay the assignment price in that * case. If it is not compiled in, the compiler should be smart enough -* to not do even the assignment. In that case, slab_equal_or_root -* will also be a constant. +* to not do even the assignment. */ if (!memcg_kmem_enabled() &&Just realized that this test can go away - we don't have to call
virt_to_cache()due to kmemcg if there is just a single cache.
Good point, will remove it in v6.
Thanks!
Quoted 12 lines
Quoted 11 lines
mm/slab.h +1
-1!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) &&@@ -535,7 +438,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)return s; cachep =virt_to_cache (x); -WARN_ONCE (cachep && !slab_equal_or_root (cachep, s), +WARN_ONCE (cachep && cachep != s,"%s: Wrong slab cache. %s but object is from %s\n" , __func__, s->name, cachep->name);return cachep;
PATCH v5 18/19 kselftests: cgroup: add kernel memory accounting tests
In reply to Roman GushchinAdd some tests to cover the kernel memory accounting functionality. These are covering some issues (and changes) we had recently.
1) A test which allocates a lot of negative dentries, checks memcg slab statistics, creates memory pressure by setting memory.max to some low value and checks that some number of slabs was reclaimed.
2) A test which covers side effects of memcg destruction: it creates and destroys a large number of sub-cgroups, each containing a multi-threaded workload which allocates and releases some kernel memory. Then it checks that the charge ans memory.stats do add up on the parent level.
3) A test which reads /proc/kpagecgroup and implicitly checks that it doesn't crash the system.
4) A test which spawns a large number of threads and checks that the kernel stacks accounting works as expected.
5) A test which checks that living charged slab objects are not preventing the memory cgroup from being released after being deleted by a user.
--- tools/testing/selftests/cgroup/.gitignore | 1 + tools/testing/selftests/cgroup/Makefile | 2 + tools/testing/selftests/cgroup/test_kmem.c | 382 +++++++++++++++++++++ 3 files changed, 385 insertions(+) create mode 100644 tools/testing/selftests/cgroup/test_kmem.c
tools/testing/selftests/cgroup/.gitignore +1 -0
@@ -2,3 +2,4 @@ test_memcontrol test_core test_freezer +test_kmem
tools/testing/selftests/cgroup/Makefile +2 -0
@@ -6,11 +6,13 @@ all: TEST_FILES := with_stress.sh TEST_PROGS := test_stress.sh TEST_GEN_PROGS = test_memcontrol +TEST_GEN_PROGS += test_kmem TEST_GEN_PROGS += test_core TEST_GEN_PROGS += test_freezer include ../lib.mk $(OUTPUT)/test_memcontrol: cgroup_util.c ../clone3/clone3_selftests.h +$(OUTPUT)/test_kmem: cgroup_util.c ../clone3/clone3_selftests.h $(OUTPUT)/test_core: cgroup_util.c ../clone3/clone3_selftests.h $(OUTPUT)/test_freezer: cgroup_util.c ../clone3/clone3_selftests.h
tools/testing/selftests/cgroup/test_kmem.c new file +382 -0
382 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v5 19/19 tools/cgroup: add memcg_slabinfo.py tool
In reply to Roman GushchinAdd a drgn-based tool to display slab information for a given memcg. Can replace cgroup v1 memory.kmem.slabinfo interface on cgroup v2, but in a more flexiable way.
Currently supports only SLUB configuration, but SLAB can be trivially added later.
Output example:
$ sudo ./tools/cgroup/memcg_slabinfo.py /sys/fs/cgroup/user.slice/user-111017.slice/user\@111017.service shmem_inode_cache 92 92 704 46 8 : tunables 0 0 0 : slabdata 2 2 0 eventpoll_pwq 56 56 72 56 1 : tunables 0 0 0 : slabdata 1 1 0 eventpoll_epi 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0 kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-64 128 128 64 64 1 : tunables 0 0 0 : slabdata 2 2 0 mm_struct 160 160 1024 32 8 : tunables 0 0 0 : slabdata 5 5 0 signal_cache 96 96 1024 32 8 : tunables 0 0 0 : slabdata 3 3 0 sighand_cache 45 45 2112 15 8 : tunables 0 0 0 : slabdata 3 3 0 files_cache 138 138 704 46 8 : tunables 0 0 0 : slabdata 3 3 0 task_delay_info 153 153 80 51 1 : tunables 0 0 0 : slabdata 3 3 0 task_struct 27 27 3520 9 8 : tunables 0 0 0 : slabdata 3 3 0 radix_tree_node 56 56 584 28 4 : tunables 0 0 0 : slabdata 2 2 0 btrfs_inode 140 140 1136 28 8 : tunables 0 0 0 : slabdata 5 5 0 kmalloc-1024 64 64 1024 32 8 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-192 84 84 192 42 2 : tunables 0 0 0 : slabdata 2 2 0 inode_cache 54 54 600 27 4 : tunables 0 0 0 : slabdata 2 2 0 kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-512 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0 skbuff_head_cache 32 32 256 32 2 : tunables 0 0 0 : slabdata 1 1 0 sock_inode_cache 46 46 704 46 8 : tunables 0 0 0 : slabdata 1 1 0 cred_jar 378 378 192 42 2 : tunables 0 0 0 : slabdata 9 9 0 proc_inode_cache 96 96 672 24 4 : tunables 0 0 0 : slabdata 4 4 0 dentry 336 336 192 42 2 : tunables 0 0 0 : slabdata 8 8 0 filp 697 864 256 32 2 : tunables 0 0 0 : slabdata 27 27 0 anon_vma 644 644 88 46 1 : tunables 0 0 0 : slabdata 14 14 0 pid 1408 1408 64 64 1 : tunables 0 0 0 : slabdata 22 22 0 vm_area_struct 1200 1200 200 40 2 : tunables 0 0 0 : slabdata 30 30 0
--- tools/cgroup/memcg_slabinfo.py | 226 +++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100755 tools/cgroup/memcg_slabinfo.py
tools/cgroup/memcg_slabinfo.py new file +226 -0
@@ -0,0 +1,226 @@ +#!/usr/bin/env drgn +# +# Copyright (C) 2020 Roman Gushchin <[email protected]> +# Copyright (C) 2020 Facebook + +from os import stat +import argparse +import sys + +from drgn.helpers.linux import list_for_each_entry, list_empty +from drgn.helpers.linux import for_each_page +from drgn.helpers.linux.cpumask import for_each_online_cpu +from drgn.helpers.linux.percpu import per_cpu_ptr +from drgn import container_of, FaultError, Object + + +DESC = """ +This is a drgn script to provide slab statistics for memory cgroups. +It supports cgroup v2 and v1 and can emulate memory.kmem.slabinfo +interface of cgroup v1. +For drgn, visit https://github.com/osandov/drgn. +""" + + +MEMCGS = {} + +OO_SHIFT = 16 +OO_MASK = ((1 << OO_SHIFT) - 1) + + +def err(s): + print('slabinfo.py: error: %s' % s, file=sys.stderr, flush=True) + sys.exit(1) + + +def find_memcg_ids(css=prog['root_mem_cgroup'].css, prefix=''): + if not list_empty(css.children.address_of_()): + for css in list_for_each_entry('struct cgroup_subsys_state', + css.children.address_of_(), + 'sibling'): + name = prefix + '/' + css.cgroup.kn.name.string_().decode('utf-8') + memcg = container_of(css, 'struct mem_cgroup', 'css') + MEMCGS[css.cgroup.kn.id.value_()] = memcg + find_memcg_ids(css, name) + + +def is_root_cache(s): + try: + return False if s.memcg_params.root_cache else True + except AttributeError: + return True + + +def cache_name(s): + if is_root_cache(s): + return s.name.string_().decode('utf-8') + else: + return s.memcg_params.root_cache.name.string_().decode('utf-8') + + +# SLUB + +def oo_order(s): + return s.oo.x >> OO_SHIFT + + +def oo_objects(s): + return s.oo.x & OO_MASK + + +def count_partial(n, fn): + nr_pages = 0 + for page in list_for_each_entry('struct page', n.partial.address_of_(), + 'lru'): + nr_pages += fn(page) + return nr_pages + + +def count_free(page): + return page.objects - page.inuse + + +def slub_get_slabinfo(s, cfg): + nr_slabs = 0 + nr_objs = 0 + nr_free = 0 + + for node in range(cfg['nr_nodes']): + n = s.node[node] + nr_slabs += n.nr_slabs.counter.value_() + nr_objs += n.total_objects.counter.value_() + nr_free += count_partial(n, count_free) + + return {'active_objs': nr_objs - nr_free, + 'num_objs': nr_objs, + 'active_slabs': nr_slabs, + 'num_slabs': nr_slabs, + 'objects_per_slab': oo_objects(s), + 'cache_order': oo_order(s), + 'limit': 0, + 'batchcount': 0, + 'shared': 0, + 'shared_avail': 0} + + +def cache_show(s, cfg, objs): + if cfg['allocator'] == 'SLUB': + sinfo = slub_get_slabinfo(s, cfg) + else: + err('SLAB isn\'t supported yet') + + if cfg['shared_slab_pages']: + sinfo['active_objs'] = objs + sinfo['num_objs'] = objs + + print('%-17s %6lu %6lu %6u %4u %4d' + ' : tunables %4u %4u %4u' + ' : slabdata %6lu %6lu %6lu' % ( + cache_name(s), sinfo['active_objs'], sinfo['num_objs'], + s.size, sinfo['objects_per_slab'], 1 << sinfo['cache_order'], + sinfo['limit'], sinfo['batchcount'], sinfo['shared'], + sinfo['active_slabs'], sinfo['num_slabs'], + sinfo['shared_avail'])) + + +def detect_kernel_config(): + cfg = {} + + cfg['nr_nodes'] = prog['nr_online_nodes'].value_() + + if prog.type('struct kmem_cache').members[1][1] == 'flags': + cfg['allocator'] = 'SLUB' + elif prog.type('struct kmem_cache').members[1][1] == 'batchcount': + cfg['allocator'] = 'SLAB' + else: + err('Can\'t determine the slab allocator') + + cfg['shared_slab_pages'] = False + try: + if prog.type('struct obj_cgroup'): + cfg['shared_slab_pages'] = True + except: + pass + + return cfg + + +def for_each_slab_page(prog): + PGSlab = 1 << prog.constant('PG_slab') + PGHead = 1 << prog.constant('PG_head') + + for page in for_each_page(prog): + try: + if page.flags.value_() & PGSlab: + yield page + except FaultError: + pass + + +def main(): + parser = argparse.ArgumentParser(description=DESC, + formatter_class= + argparse.RawTextHelpFormatter) + parser.add_argument('cgroup', metavar='CGROUP', + help='Target memory cgroup') + args = parser.parse_args() + + try: + cgroup_id = stat(args.cgroup).st_ino + find_memcg_ids() + memcg = MEMCGS[cgroup_id] + except KeyError: + err('Can\'t find the memory cgroup') + + cfg = detect_kernel_config() + + print('# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>' + ' : tunables <limit> <batchcount> <sharedfactor>' + ' : slabdata <active_slabs> <num_slabs> <sharedavail>') + + if cfg['shared_slab_pages']: + obj_cgroups = set() + stats = {} + caches = {} + + # find memcg pointers belonging to the specified cgroup + obj_cgroups.add(memcg.objcg.value_()) + for ptr in list_for_each_entry('struct obj_cgroup', + memcg.objcg_list.address_of_(), + 'list'): + obj_cgroups.add(ptr.value_()) + + # look over all slab pages, belonging to non-root memcgs + # and look for objects belonging to the given memory cgroup + for page in for_each_slab_page(prog): + objcg_vec_raw = page.obj_cgroups.value_() + if objcg_vec_raw == 0: + continue + cache = page.slab_cache + if not cache: + continue + addr = cache.value_() + caches[addr] = cache + # clear the lowest bit to get the true obj_cgroups + objcg_vec = Object(prog, page.obj_cgroups.type_, + value=objcg_vec_raw & ~1) + + if addr not in stats: + stats[addr] = 0 + + for i in range(oo_objects(cache)): + if objcg_vec[i].value_() in obj_cgroups: + stats[addr] += 1 + + for addr in caches: + if stats[addr] > 0: + cache_show(caches[addr], cfg, stats[addr]) + + else: + for s in list_for_each_entry('struct kmem_cache', + memcg.kmem_caches.address_of_(), + 'memcg_params.kmem_caches_node'): + cache_show(s, cfg, None) + + +main()
Signature · 1 line
2.25.4