memcg targeted shrinking
Subsystems (10)
cover memcg targeted shrinking
This patchset implements targeted shrinking for memcg when kmem limits are present. So far, we've been accounting kernel objects but failing allocations when short of memory. This is because our only option would be to call the global shrinker, depleting objects from all caches and breaking isolation.
This patchset builds upon the recent work from David Chinner
(http://oss.sgi.com/archives/xfs/2012-11/msg00643.html) to implement NUMA aware per-node LRUs. I build heavily on its API, and its presence is implied.
The main idea is to associate per-memcg lists with each of the LRUs. The main LRU still provides a single entry point and when adding or removing an element from the LRU, we use the page information to figure out which memcg it belongs to and relay it to the right list.
This patchset is still not perfect, and some uses cases still need to be dealt with. But I wanted to get this out in the open sooner rather than later. In particular, I have the following (noncomprehensive) todo list:
TODO:
* shrink dead memcgs when global pressure kicks in.
* balance global reclaim among memcgs.
* improve testing and reliability (I am still seeing some stalls in some cases)
Glauber Costa (7):
vmscan: also shrink slab in memcg pressure memcg,list_lru: duplicate LRUs upon kmemcg creation
lru: add an element to a memcg list
list_lru: also include memcg lists in counts and scans
list_lru: per-memcg walks
super: targeted memcg reclaim
memcg: per-memcg kmem shrinking
fs/dcache.c | 7 +- fs/inode.c | 6 +- fs/internal.h | 5 +- fs/super.c | 37 ++++-- include/linux/list_lru.h | 81 +++++++++++- include/linux/memcontrol.h | 34 +++++ include/linux/shrinker.h | 4 + include/linux/swap.h | 2 + lib/list_lru.c | 301 ++++++++++++++++++++++++++++++++++++++------- mm/memcontrol.c | 271 ++++++++++++++++++++++++++++++++++++++-- mm/slab_common.c | 1 - mm/vmscan.c | 78 +++++++++++- 12 files changed, 747 insertions(+), 80 deletions(-)
Signature · 1 line
1.8.1
PATCH 1/7 vmscan: also shrink slab in memcg pressure
Without the surrounding infrastructure, this patch is a bit of a hammer:
it will basically shrink objects from all memcgs under memcg pressure. At least, however, we will keep the scan limited to the shrinkers marked as per-memcg.
Future patches will implement the in-shrinker logic to filter objects based on its memcg association.
--- include/linux/memcontrol.h | 16 ++++++++++++++++ include/linux/shrinker.h | 4 ++++ mm/memcontrol.c | 11 ++++++++++- mm/vmscan.c | 41 ++++++++++++++++++++++++++++++++++++++--- 4 files changed, 68 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +16 -0
@@ -200,6 +200,9 @@ void mem_cgroup_split_huge_fixup(struct page *head);bool mem_cgroup_bad_page_check (struct page *page);void mem_cgroup_print_bad_page (struct page *page);#endif + +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone);#else /* CONFIG_MEMCG */ struct mem_cgroup ;
@@ -384,6 +387,11 @@ static inline void mem_cgroup_replace_page_cache(struct page *oldpage, struct page *newpage) { } + +static inline unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +}#endif /* CONFIG_MEMCG */ #if !defined(CONFIG_MEMCG ) || !defined(CONFIG_DEBUG_VM )
@@ -436,6 +444,8 @@ static inline bool memcg_kmem_enabled(void)return static_key_false (&memcg_kmem_enabled_key); } +bool memcg_kmem_is_active (struct mem_cgroup *memcg); +/* * In general, we'll do everything in our power to not incur in any overhead * for non-memcg users for the kmem functions. Not even a function call, if we
@@ -569,6 +579,12 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp) return __memcg_kmem_get_cache(cachep, gfp); }#else + +static inline bool memcg_kmem_is_active (struct mem_cgroup *memcg) +{ +return false; +} +#define for_each_memcg_cache_index (_idx) \ for (; NULL; )
include/linux/shrinker.h +4 -0
@@ -20,6 +20,9 @@ struct shrink_control {/* shrink from these nodes */ nodemask_t nodes_to_scan; + +/* reclaim from this memcg only (if not NULL) */ +struct mem_cgroup *target_mem_cgroup; };/*
@@ -45,6 +48,7 @@ struct shrinker {int seeks;/* seeks to recreate an obj */ long batch;/* reclaim batch size, 0 = default */ +bool memcg_shrinker;/* These are for internal use */ struct list_head list;
mm/memcontrol.c +10 -1
@@ -442,7 +442,7 @@ static inline void memcg_kmem_set_active(struct mem_cgroup *memcg)set_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); } -static bool memcg_kmem_is_active (struct mem_cgroup *memcg) +bool memcg_kmem_is_active (struct mem_cgroup *memcg) {return test_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); }
@@ -991,6 +991,15 @@ mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid,return ret; } +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +int nid =zone_to_nid (zone); +int zid =zone_idx (zone); + +return mem_cgroup_zone_nr_lru_pages (memcg, nid, zid,LRU_ALL ); +} +static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg,int nid,unsigned int lru_mask)
mm/vmscan.c +38 -3
77 lines not on this page · raw message
Signature · 1 line
1.8.1
Re: [PATCH 1/7] vmscan: also shrink slab in memcg pressure
Quoted 39 lines · Glauber Costa
Without the surrounding infrastructure, this patch is a bit of a hammer:
it will basically shrink objects from all memcgs under memcg pressure. At least, however, we will keep the scan limited to the shrinkers marked as per-memcg.Future patches will implement the in-shrinker logic to filter objects based on its memcg association.
Signed-off-by Glauber Costa <[email protected]>Cc Dave Chinner <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kamezawa Hiroyuki <[email protected]>Cc Andrew Morton <[email protected]>--- include/linux/memcontrol.h | 16 ++++++++++++++++ include/linux/shrinker.h | 4 ++++ mm/memcontrol.c | 11 ++++++++++- mm/vmscan.c | 41 ++++++++++++++++++++++++++++++++++++++--- 4 files changed, 68 insertions(+), 4 deletions(-)include/linux/memcontrol.h +3
-0@@ -200,6 +200,9 @@ void mem_cgroup_split_huge_fixup(struct page *head);bool mem_cgroup_bad_page_check (struct page *page);void mem_cgroup_print_bad_page (struct page *page);#endif + +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone);#else /* CONFIG_MEMCG */ struct mem_cgroup ;
include/linux/memcontrol.h static inline void mem_cgroup_replace_page_cache(struct page *oldpage, · line 393
Quoted 7 lines
include/linux/memcontrol.h +4
-0@@ -384,6 +387,11 @@ static inline void mem_cgroup_replace_page_cache(struct page *oldpage, struct page *newpage) { } + +static inline unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{
return 0;
Quoted 62 lines
include/linux/memcontrol.h +9
-0+}#endif /* CONFIG_MEMCG */ #if !defined(CONFIG_MEMCG ) || !defined(CONFIG_DEBUG_VM )@@ -436,6 +444,8 @@ static inline bool memcg_kmem_enabled(void)return static_key_false (&memcg_kmem_enabled_key); } +bool memcg_kmem_is_active (struct mem_cgroup *memcg); +/* * In general, we'll do everything in our power to not incur in any overhead * for non-memcg users for the kmem functions. Not even a function call, if we @@ -569,6 +579,12 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp) return __memcg_kmem_get_cache(cachep, gfp); }#else + +static inline bool memcg_kmem_is_active (struct mem_cgroup *memcg) +{ +return false; +} +#define for_each_memcg_cache_index (_idx) \ for (; NULL; )include/linux/shrinker.h +4
-0@@ -20,6 +20,9 @@ struct shrink_control {/* shrink from these nodes */ nodemask_t nodes_to_scan; + +/* reclaim from this memcg only (if not NULL) */ +struct mem_cgroup *target_mem_cgroup; };/* @@ -45,6 +48,7 @@ struct shrinker {int seeks;/* seeks to recreate an obj */ long batch;/* reclaim batch size, 0 = default */ +bool memcg_shrinker;/* These are for internal use */ struct list_head list;mm/memcontrol.c +1
-1@@ -442,7 +442,7 @@ static inline void memcg_kmem_set_active(struct mem_cgroup *memcg)set_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); } -static bool memcg_kmem_is_active (struct mem_cgroup *memcg) +bool memcg_kmem_is_active (struct mem_cgroup *memcg) {return test_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); }
mm/memcontrol.c mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid, · line 1000
Quoted 10 lines
mm/memcontrol.c +7
-0@@ -991,6 +991,15 @@ mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid,return ret; } +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +int nid =zone_to_nid (zone); +int zid =zone_idx (zone); + +return mem_cgroup_zone_nr_lru_pages (memcg, nid, zid,LRU_ALL );
Without swap enabled it seems like LRU_ALL_FILE is more appropriate. Maybe something like test_mem_cgroup_node_reclaimable().
Quoted 13 lines
mm/memcontrol.c +2
-0+} +static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg,int nid,unsigned int lru_mask)mm/vmscan.c +0
-0@@ -138,11 +138,42 @@ static bool global_reclaim(struct scan_control *sc) {return !sc->target_mem_cgroup; }
mm/vmscan.c static bool global_reclaim(struct scan_control *sc) · line 150
Quoted 10 lines
mm/vmscan.c +10
-0+ +/* +* kmem reclaim should usually not be triggered when we are doing targetted +* reclaim. It is only valid when global reclaim is triggered, or when the +* underlying memcg has kmem objects. +*/ +static bool has_kmem_reclaim (struct scan_control *sc) +{ +return !sc->target_mem_cgroup || + (sc->target_mem_cgroup &&memcg_kmem_is_active (sc->target_mem_cgroup));
Isn't this the same as:
return !sc->target_mem_cgroup || memcg_kmem_is_active(sc->target_mem_cgroup);
[...]
Re: [PATCH 1/7] vmscan: also shrink slab in memcg pressure
Quoted 12 lines
include/linux/memcontrol.h static inline void mem_cgroup_replace_page_cache(struct page *oldpage, · line 393
Quoted 9 lines
include/linux/memcontrol.h +4
-0@@ -384,6 +387,11 @@ static inline void mem_cgroup_replace_page_cache(struct page *oldpage, struct page *newpage) { } + +static inline unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{return 0;
ok
Quoted 8 lines
Quoted 8 lines
mm/memcontrol.c +1
-0+bool memcg_kmem_is_active (struct mem_cgroup *memcg) {return test_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); }@@ -991,6 +991,15 @@ mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid,return ret; }
Quoted 11 lines
mm/memcontrol.c mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid, · line 1000
Quoted 7 lines
mm/memcontrol.c +7
-0+unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +int nid =zone_to_nid (zone); +int zid =zone_idx (zone); + +return mem_cgroup_zone_nr_lru_pages (memcg, nid, zid,LRU_ALL );Without swap enabled it seems like
LRU_ALL_FILEis more appropriate. Maybe something liketest_mem_cgroup_node_reclaimable().
You are right, I will look into it.
Quoted 25 lines
Quoted 13 lines
mm/memcontrol.c +2
-0+} +static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg,int nid,unsigned int lru_mask)mm/vmscan.c +0
-0@@ -138,11 +138,42 @@ static bool global_reclaim(struct scan_control *sc) {return !sc->target_mem_cgroup; }mm/vmscan.c static bool global_reclaim(struct scan_control *sc) · line 150
Quoted 10 lines
mm/vmscan.c +10
-0+ +/* +* kmem reclaim should usually not be triggered when we are doing targetted +* reclaim. It is only valid when global reclaim is triggered, or when the +* underlying memcg has kmem objects. +*/ +static bool has_kmem_reclaim (struct scan_control *sc) +{ +return !sc->target_mem_cgroup || + (sc->target_mem_cgroup &&memcg_kmem_is_active (sc->target_mem_cgroup));Isn't this the same as:
Quoted 3 lines
return !sc->target_mem_cgroup || memcg_kmem_is_active(sc->target_mem_cgroup);
Yes, it is.
Re: [PATCH 1/7] vmscan: also shrink slab in memcg pressure
In reply to Glauber CostaQuoted 87 lines · Glauber Costa
Without the surrounding infrastructure, this patch is a bit of a hammer:
it will basically shrink objects from all memcgs under memcg pressure. At least, however, we will keep the scan limited to the shrinkers marked as per-memcg.Future patches will implement the in-shrinker logic to filter objects based on its memcg association.
Signed-off-by Glauber Costa <[email protected]>Cc Dave Chinner <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kamezawa Hiroyuki <[email protected]>Cc Andrew Morton <[email protected]>--- include/linux/memcontrol.h | 16 ++++++++++++++++ include/linux/shrinker.h | 4 ++++ mm/memcontrol.c | 11 ++++++++++- mm/vmscan.c | 41 ++++++++++++++++++++++++++++++++++++++--- 4 files changed, 68 insertions(+), 4 deletions(-)include/linux/memcontrol.h +16
-0@@ -200,6 +200,9 @@ void mem_cgroup_split_huge_fixup(struct page *head);bool mem_cgroup_bad_page_check (struct page *page);void mem_cgroup_print_bad_page (struct page *page);#endif + +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone);#else /* CONFIG_MEMCG */ struct mem_cgroup ;@@ -384,6 +387,11 @@ static inline void mem_cgroup_replace_page_cache(struct page *oldpage, struct page *newpage) { } + +static inline unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +}#endif /* CONFIG_MEMCG */ #if !defined(CONFIG_MEMCG ) || !defined(CONFIG_DEBUG_VM )@@ -436,6 +444,8 @@ static inline bool memcg_kmem_enabled(void)return static_key_false (&memcg_kmem_enabled_key); } +bool memcg_kmem_is_active (struct mem_cgroup *memcg); +/* * In general, we'll do everything in our power to not incur in any overhead * for non-memcg users for the kmem functions. Not even a function call, if we @@ -569,6 +579,12 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp) return __memcg_kmem_get_cache(cachep, gfp); }#else + +static inline bool memcg_kmem_is_active (struct mem_cgroup *memcg) +{ +return false; +} +#define for_each_memcg_cache_index (_idx) \ for (; NULL; )include/linux/shrinker.h +3
-0@@ -20,6 +20,9 @@ struct shrink_control {/* shrink from these nodes */ nodemask_t nodes_to_scan; + +/* reclaim from this memcg only (if not NULL) */ +struct mem_cgroup *target_mem_cgroup; };/*
include/linux/shrinker.h struct shrinker { · line 52
Quoted 5 lines
include/linux/shrinker.h +1
-0@@ -45,6 +48,7 @@ struct shrinker {int seeks;/* seeks to recreate an obj */ long batch;/* reclaim batch size, 0 = default */ +bool memcg_shrinker;
What is this boolean for ? When is this set ?
Quoted 66 lines
include/linux/shrinker.h +0
-0/* These are for internal use */ struct list_head list;mm/memcontrol.c +10
-1@@ -442,7 +442,7 @@ static inline void memcg_kmem_set_active(struct mem_cgroup *memcg)set_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); } -static bool memcg_kmem_is_active (struct mem_cgroup *memcg) +bool memcg_kmem_is_active (struct mem_cgroup *memcg) {return test_bit (KMEM_ACCOUNTED_ACTIVE , &memcg->kmem_account_flags); }@@ -991,6 +991,15 @@ mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid,return ret; } +unsigned long +memcg_zone_reclaimable_pages (struct mem_cgroup *memcg,struct zone *zone) +{ +int nid =zone_to_nid (zone); +int zid =zone_idx (zone); + +return mem_cgroup_zone_nr_lru_pages (memcg, nid, zid,LRU_ALL ); +} +static unsigned long mem_cgroup_node_nr_lru_pages (struct mem_cgroup *memcg,int nid,unsigned int lru_mask)mm/vmscan.c +22
-0@@ -138,11 +138,42 @@ static bool global_reclaim(struct scan_control *sc) {return !sc->target_mem_cgroup; } + +/* +* kmem reclaim should usually not be triggered when we are doing targetted +* reclaim. It is only valid when global reclaim is triggered, or when the +* underlying memcg has kmem objects. +*/ +static bool has_kmem_reclaim (struct scan_control *sc) +{ +return !sc->target_mem_cgroup || + (sc->target_mem_cgroup &&memcg_kmem_is_active (sc->target_mem_cgroup)); +} + +static unsigned long +zone_nr_reclaimable_pages (struct scan_control *sc,struct zone *zone) +{ +if (global_reclaim (sc)) +return zone_reclaimable_pages (zone); +return memcg_zone_reclaimable_pages (sc->target_mem_cgroup, zone); +} +#else static bool global_reclaim (struct scan_control *sc) {return true; } + +static bool has_kmem_reclaim (struct scan_control *sc)
mm/vmscan.c static bool global_reclaim(struct scan_control *sc) · line 177
Quoted 10 lines
mm/vmscan.c +9
-0+{ +return true; +} + +static unsigned long +zone_nr_reclaimable_pages (struct scan_control *sc,struct zone *zone) +{ +return zone_reclaimable_pages (zone); +}#endif
Can't be in a devided patch ?
mm/vmscan.c unsigned long shrink_slab(struct shrink_control *sc, · line 257
Quoted 9 lines
mm/vmscan.c +3
-0static unsigned long get_lru_size (struct lruvec *lruvec,enum lru_list lru)@@ -221,6 +252,9 @@ unsigned long shrink_slab(struct shrink_control *sc,long batch_size = shrinker->batch ? shrinker->batch :SHRINK_BATCH ; +if (!shrinker->memcg_shrinker && sc->target_mem_cgroup) +continue ; +
What does this mean ?
Quoted 26 lines
mm/vmscan.c +3
-3max_pass = shrinker->count_objects (shrinker, sc);WARN_ON (max_pass <0 );if (max_pass <=0 )@@ -2170,9 +2204,9 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,/* * Don't shrink slabs when reclaiming memory from -* over limit cgroups +* over limit cgroups, unless we know they have kmem objects */ -if (global_reclaim (sc)) { +if (has_kmem_reclaim (sc)) {unsigned long lru_pages =0 ;nodes_clear (shrink->nodes_to_scan);@@ -2181,7 +2215,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,if (!cpuset_zone_allowed_hardwall (zone,GFP_KERNEL ))continue ; - lru_pages +=zone_reclaimable_pages (zone); + lru_pages +=zone_nr_reclaimable_pages (sc, zone);node_set (zone_to_nid (zone), shrink->nodes_to_scan); }
mm/vmscan.c unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, · line 2483
Quoted 7 lines
mm/vmscan.c +1
-0@@ -2443,6 +2477,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, };struct shrink_control shrink = { .gfp_mask = sc.gfp_mask, + .target_mem_cgroup = memcg, };/*
Thanks,
-Kame
Re: [PATCH 1/7] vmscan: also shrink slab in memcg pressure
Quoted 8 lines
include/linux/shrinker.h struct shrinker { · line 52
Quoted 6 lines
include/linux/shrinker.h +1
-0@@ -45,6 +48,7 @@ struct shrinker {int seeks;/* seeks to recreate an obj */ long batch;/* reclaim batch size, 0 = default */ +bool memcg_shrinker;What is this boolean for ? When is this set ?
It is set when a subsystem declares that its shrinker is memcg capable. Therefore, it won't be done until all infrastructure is in place. Take a look at the super.c patches at the end of the series.
Quoted 10 lines
Quoted 10 lines
mm/vmscan.c +0
-0static bool global_reclaim (struct scan_control *sc) {return true; }mm/vmscan.c +6
-0+ +static bool has_kmem_reclaim (struct scan_control *sc) +{ +return true; +} +
Quoted 9 lines
mm/vmscan.c static bool global_reclaim(struct scan_control *sc) · line 177
Quoted 6 lines
mm/vmscan.c +5
-0+static unsigned long +zone_nr_reclaimable_pages (struct scan_control *sc,struct zone *zone) +{ +return zone_reclaimable_pages (zone); +}#endif Can't be in a devided patch ?
if you prefer this way, sure, I can separate it.
Quoted 10 lines
mm/vmscan.c unsigned long shrink_slab(struct shrink_control *sc, · line 257
Quoted 8 lines
static unsigned long
get_lru_size(struct lruvec *lruvec, enum lru_list lru)mm/vmscan.c +3
-0@@ -221,6 +252,9 @@ unsigned long shrink_slab(struct shrink_control *sc,long batch_size = shrinker->batch ? shrinker->batch :SHRINK_BATCH ; +if (!shrinker->memcg_shrinker && sc->target_mem_cgroup) +continue ; +What does this mean ?
It means that if target_mem_cgroup is set, we should skip all the shrinkers that are not memcg capable. Maybe if I invert the order it will be clearer?
PATCH 2/7 memcg,list_lru: duplicate LRUs upon kmemcg creation
In reply to Glauber CostaWhen a new memcg is created, we need to open up room for its descriptors in all of the list_lrus that are marked per-memcg. The process is quite similar to the one we are using for the kmem caches: we initialize the new structures in an array indexed by kmemcg_id, and grow the array if needed. Key data like the size of the array will be shared between the kmem cache code and the list_lru code (they basically describe the same thing)
--- include/linux/list_lru.h | 47 +++++++++++++++++ include/linux/memcontrol.h | 6 +++ lib/list_lru.c | 115 +++++++++++++++++++++++++++++++++++++--- mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++++++++++--- mm/slab_common.c | 1 - 5 files changed, 283 insertions(+), 14 deletions(-)
include/linux/list_lru.h +47 -0
@@ -16,11 +16,58 @@ struct list_lru_node {long nr_items; }____cacheline_aligned_in_smp ; +struct list_lru_array { +struct list_lru_node node[1 ]; +}; +struct list_lru { +struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ];nodemask_t active_nodes; +#ifdef CONFIG_MEMCG_KMEM +struct list_lru_array **memcg_lrus; +#endif }; +struct mem_cgroup ; +#ifdef CONFIG_MEMCG_KMEM +/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/ +static inline void lru_memcg_enable (struct list_lru *lru) +{ + lru->memcg_lrus = (void *)0x1ULL ; +} + +/* +* This will return true if we have already allocated and assignment a memcg +* pointer set to the LRU. Therefore, we need to mask the first bit out +*/ +static inline bool lru_memcg_is_assigned (struct list_lru *lru) +{ +return (unsigned long )lru->memcg_lrus & ~0x1ULL ; +} + +struct list_lru_array *lru_alloc_array (void ); +int memcg_update_all_lrus (unsigned long num); +void list_lru_destroy (struct list_lru *lru); +void list_lru_destroy_memcg (struct mem_cgroup *memcg); +#else +static inline void lru_memcg_enable (struct list_lru *lru) +{ +} + +static inline bool lru_memcg_is_assigned (struct list_lru *lru) +{ +return false; +} + +static inline void list_lru_destroy (struct list_lru *lru) +{ +} +#endif +int list_lru_init (struct list_lru *lru);int list_lru_add (struct list_lru *lru,struct list_head *item);int list_lru_del (struct list_lru *lru,struct list_head *item);
include/linux/memcontrol.h +6 -0
@@ -23,6 +23,7 @@#include <linux/vm_event_item.h> #include <linux/hardirq.h> #include <linux/jump_label.h> +#include <linux/list_lru.h> struct mem_cgroup ;struct page_cgroup ;
@@ -475,6 +476,11 @@ void memcg_update_array_size(int num_groups);struct kmem_cache *__memcg_kmem_get_cache (struct kmem_cache *cachep,gfp_t gfp); +int memcg_new_lru (struct list_lru *lru); + +int memcg_kmem_update_lru_size (struct list_lru *lru,int num_groups, +bool new_lru); +void mem_cgroup_destroy_cache (struct kmem_cache *cachep);void kmem_cache_destroy_memcg_children (struct kmem_cache *s);
lib/list_lru.c +108 -7
132 lines not on this page · raw message
mm/memcontrol.c +122 -6
152 lines not on this page · raw message
mm/slab_common.c +0 -1
@@ -102,7 +102,6 @@ int memcg_update_all_caches(int num_memcgs) goto out; } -memcg_update_array_size (num_memcgs); out:mutex_unlock (&slab_mutex);return ret;
Signature · 1 line
1.8.1
Re: [PATCH 2/7] memcg,list_lru: duplicate LRUs upon kmemcg creation
Quoted 33 lines · Glauber Costa
When a new memcg is created, we need to open up room for its descriptors in all of the
list_lrusthat are marked per-memcg. The process is quite similar to the one we are using for the kmem caches: we initialize the new structures in an array indexed bykmemcg_id, and grow the array if needed. Key data like the size of the array will be shared between the kmem cache code and thelist_lrucode (they basically describe the same thing)Signed-off-by Glauber Costa <[email protected]>Cc Dave Chinner <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kamezawa Hiroyuki <[email protected]>Cc Andrew Morton <[email protected]>--- include/linux/list_lru.h | 47 +++++++++++++++++ include/linux/memcontrol.h | 6 +++ lib/list_lru.c | 115 +++++++++++++++++++++++++++++++++++++--- mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++++++++++--- mm/slab_common.c | 1 - 5 files changed, 283 insertions(+), 14 deletions(-)include/linux/list_lru.h +0
-0@@ -16,11 +16,58 @@ struct list_lru_node {long nr_items; }____cacheline_aligned_in_smp ;
include/linux/list_lru.h struct list_lru_node { · line 28
Quoted 10 lines
include/linux/list_lru.h +7
-0+struct list_lru_array { +struct list_lru_node node[1 ]; +}; +struct list_lru { +struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ];nodemask_t active_nodes; +#ifdef CONFIG_MEMCG_KMEM +struct list_lru_array **memcg_lrus;
Probably need a comment regarding that 0x1 is a magic value and describing what indexes this lazily constructed array. Is the primary index memcg_kmem_id and the secondary index a nid?
include/linux/list_lru.h struct list_lru_node { · line 37
Quoted 9 lines
include/linux/list_lru.h +7
-0+#endif }; +struct mem_cgroup ; +#ifdef CONFIG_MEMCG_KMEM +/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/
From this patch it seems like 0x1 is a magic value rather than bit 0 being special. memcg_lrus is either 0x1 or a pointer to an array of struct list_lru_array. The array is indexed by memcg_kmem_id.
include/linux/list_lru.h struct list_lru_node { · line 38
Quoted 1 line
include/linux/list_lru.h +1
-0+static inline void lru_memcg_enable (struct list_lru *lru)
This function is not called yet. Hmm.
include/linux/list_lru.h struct list_lru_node { · line 49
Quoted 11 lines
include/linux/list_lru.h +11
-0+{ + lru->memcg_lrus = (void *)0x1ULL ; +} + +/* +* This will return true if we have already allocated and assignment a memcg +* pointer set to the LRU. Therefore, we need to mask the first bit out +*/ +static inline bool lru_memcg_is_assigned (struct list_lru *lru) +{ +return (unsigned long )lru->memcg_lrus & ~0x1ULL ;
Is this equivalent to?
return lru->memcg_lrus != NULL && lru->memcg_lrus != 0x1
include/linux/list_lru.h struct list_lru_node { · line 51
Quoted 2 lines
include/linux/list_lru.h +2
-0+} +
[...]
/* comment the meaning of "num" */
include/linux/list_lru.h struct list_lru_node { · line 56
Quoted 5 lines
include/linux/list_lru.h +5
-0+int memcg_update_all_lrus (unsigned long num) +{ +int ret =0 ; +struct list_lru *lru; +
[...]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/7] memcg,list_lru: duplicate LRUs upon kmemcg creation
Quoted 10 lines
Quoted 10 lines
include/linux/list_lru.h +0
-0@@ -16,11 +16,58 @@ struct list_lru_node {long nr_items; }____cacheline_aligned_in_smp ;
Quoted 13 lines
include/linux/list_lru.h struct list_lru_node { · line 28
Quoted 10 lines
include/linux/list_lru.h +7
-0+struct list_lru_array { +struct list_lru_node node[1 ]; +}; +struct list_lru { +struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ];nodemask_t active_nodes; +#ifdef CONFIG_MEMCG_KMEM +struct list_lru_array **memcg_lrus;Probably need a comment regarding that
0x1is a magic value and describing what indexes this lazily constructed array.
Ok.
Quoted 3 lines
Is the primary
indexmemcg_kmem_idand the secondary index a nid?
Precisely. The first level is an array of pointers to list_lru_array. And each list_lru_array is an array of nids.
Quoted 11 lines
include/linux/list_lru.h struct list_lru_node { · line 37
Quoted 6 lines
include/linux/list_lru.h +6
-0+struct mem_cgroup ; +#ifdef CONFIG_MEMCG_KMEM +/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/ From this patch it seems like
0x1is a magic value rather than bit 0 being special.memcg_lrusis either0x1or a pointer to an array of structlist_lru_array. The array is indexed bymemcg_kmem_id.
Well, I thought in terms of "set the last bit". To be honest, when I first designed this, I figured it could possibly be useful to keep the bit set at all times, and that is why I used the LSB. Since I turned out not using it, maybe we could actually resort to a fully fledged magical to avoid the confusion?
Quoted 10 lines
include/linux/list_lru.h struct list_lru_node { · line 49
Quoted 8 lines
include/linux/list_lru.h +8
-0+static inline void lru_memcg_enable (struct list_lru *lru) +/* +* This will return true if we have already allocated and assignment a memcg +* pointer set to the LRU. Therefore, we need to mask the first bit out +*/ +static inline bool lru_memcg_is_assigned(struct list_lru *lru) +{ +return (unsigned long )lru->memcg_lrus & ~0x1ULL ;Is this equivalent to?
Quoted 2 lines
return lru->memcg_lrus != NULL && lru->memcg_lrus != 0x1
yes. What I've explained above should help clarifying why I wrote it this way. But if we use an actual magical (0x1 is a bad magical, IMHO), the intentions become a lot clearer.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/7] memcg,list_lru: duplicate LRUs upon kmemcg creation
Quoted 12 lines · Glauber Costa
Quoted 11 lines
include/linux/list_lru.h struct list_lru_node { · line 37
Quoted 6 lines
include/linux/list_lru.h +6
-0+struct mem_cgroup ; +#ifdef CONFIG_MEMCG_KMEM +/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/ From this patch it seems like
0x1is a magic value rather than bit 0 being special.memcg_lrusis either0x1or a pointer to an array of structlist_lru_array. The array is indexed bymemcg_kmem_id.
Quoted 5 lines
Well, I thought in terms of "set the last bit". To be honest, when I first designed this, I figured it could possibly be useful to keep the bit set at all times, and that is why I used the LSB. Since I turned out not using it, maybe we could actually resort to a fully fledged magical to avoid the confusion?
To avoid confusion, I'd prefer a magic value. This allows callers to not worrying about having to strip off the low order bit, if it's later always set for some reason. But I'm not even sure we need a magic value or magic bit (see below).
Quoted 12 lines
Quoted 10 lines
include/linux/list_lru.h struct list_lru_node { · line 49
Quoted 8 lines
include/linux/list_lru.h +8
-0+static inline void lru_memcg_enable (struct list_lru *lru) +/* +* This will return true if we have already allocated and assignment a memcg +* pointer set to the LRU. Therefore, we need to mask the first bit out +*/ +static inline bool lru_memcg_is_assigned(struct list_lru *lru) +{ +return (unsigned long )lru->memcg_lrus & ~0x1ULL ;Is this equivalent to?
Quoted 2 lines
return lru->memcg_lrus != NULL && lru->memcg_lrus != 0x1
Quoted 3 lines
yes. What I've explained above should help clarifying why I wrote it this way. But if we use an actual magical (
0x1is a bad magical, IMHO), the intentions become a lot clearer.
Does the following work and yield simpler code?
1. add a 'bool memcg_enabled' parameter to list_lru_init()
2. rename all_lrus to all_memcg_lrus
3. only add lru to all_memcg_lrus if memcg_enabled is set
4. delete lru_memcg_enable()
5. redefine lru_memcg_is_assigned() to just test (lru->memcg_lrus == NULL)
Then we don't need a magic valid (or LSB) to identify memcg enabled lrus. Any lru in the all_memcg_lrus list is memcg enabled.
Re: [PATCH 2/7] memcg,list_lru: duplicate LRUs upon kmemcg creation
In reply to Glauber CostaQuoted 30 lines · Glauber Costa
When a new memcg is created, we need to open up room for its descriptors in all of the
list_lrusthat are marked per-memcg. The process is quite similar to the one we are using for the kmem caches: we initialize the new structures in an array indexed bykmemcg_id, and grow the array if needed. Key data like the size of the array will be shared between the kmem cache code and thelist_lrucode (they basically describe the same thing)Signed-off-by Glauber Costa <[email protected]>Cc Dave Chinner <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kamezawa Hiroyuki <[email protected]>Cc Andrew Morton <akpm-de/[email protected]>--- include/linux/list_lru.h | 47 +++++++++++++++++ include/linux/memcontrol.h | 6 +++ lib/list_lru.c | 115 +++++++++++++++++++++++++++++++++++++--- mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++++++++++--- mm/slab_common.c | 1 - 5 files changed, 283 insertions(+), 14 deletions(-)include/linux/list_lru.h +0
-0
include/linux/list_lru.h struct list_lru_node { · line 21
Quoted 6 lines
include/linux/list_lru.h +3
-0@@ -16,11 +16,58 @@ struct list_lru_node {long nr_items; }____cacheline_aligned_in_smp ; +struct list_lru_array { +struct list_lru_node node[1 ]; +};
size is up to nr_node_ids ?
include/linux/list_lru.h struct list_lru_node { · line 30
Quoted 9 lines
include/linux/list_lru.h +5
-0+struct list_lru { +struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ];nodemask_t active_nodes; +#ifdef CONFIG_MEMCG_KMEM +struct list_lru_array **memcg_lrus; +#endif };
size is up to memcg_limited_groups_array_size ?
include/linux/list_lru.h struct list_lru_node { · line 42
Quoted 12 lines
include/linux/list_lru.h +11
-0+struct mem_cgroup ; +#ifdef CONFIG_MEMCG_KMEM +/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/ +static inline void lru_memcg_enable (struct list_lru *lru) +{ + lru->memcg_lrus = (void *)0x1ULL ; +} +
This "enable" is not used in this patch itself, right ?
Quoted 106 lines
include/linux/list_lru.h +28
-0+/* +* This will return true if we have already allocated and assignment a memcg +* pointer set to the LRU. Therefore, we need to mask the first bit out +*/ +static inline bool lru_memcg_is_assigned (struct list_lru *lru) +{ +return (unsigned long )lru->memcg_lrus & ~0x1ULL ; +} + +struct list_lru_array *lru_alloc_array (void ); +int memcg_update_all_lrus (unsigned long num); +void list_lru_destroy (struct list_lru *lru); +void list_lru_destroy_memcg (struct mem_cgroup *memcg); +#else +static inline void lru_memcg_enable (struct list_lru *lru) +{ +} + +static inline bool lru_memcg_is_assigned (struct list_lru *lru) +{ +return false; +} + +static inline void list_lru_destroy (struct list_lru *lru) +{ +} +#endif +int list_lru_init (struct list_lru *lru);int list_lru_add (struct list_lru *lru,struct list_head *item);int list_lru_del (struct list_lru *lru,struct list_head *item);include/linux/memcontrol.h +6
-0@@ -23,6 +23,7 @@#include <linux/vm_event_item.h> #include <linux/hardirq.h> #include <linux/jump_label.h> +#include <linux/list_lru.h> struct mem_cgroup ;struct page_cgroup ;@@ -475,6 +476,11 @@ void memcg_update_array_size(int num_groups);struct kmem_cache *__memcg_kmem_get_cache (struct kmem_cache *cachep,gfp_t gfp); +int memcg_new_lru (struct list_lru *lru); + +int memcg_kmem_update_lru_size (struct list_lru *lru,int num_groups, +bool new_lru); +void mem_cgroup_destroy_cache (struct kmem_cache *cachep);void kmem_cache_destroy_memcg_children (struct kmem_cache *s);lib/list_lru.c +33
-3@@ -8,6 +8,7 @@#include <linux/module.h> #include <linux/mm.h> #include <linux/list_lru.h> +#include <linux/memcontrol.h> int list_lru_add(@@ -184,18 +185,118 @@ list_lru_dispose_all(return total; } -int -list_lru_init ( -struct list_lru *lru) +/* +* This protects the list of all LRU in the system. One only needs +* to take when registering an LRU, or when duplicating the list of lrus. +* Transversing an LRU can and should be done outside the lock +*/ +static DEFINE_MUTEX (all_lrus_mutex); +static LIST_HEAD (all_lrus); + +static void list_lru_init_one (struct list_lru_node *lru) +{ +spin_lock_init (&lru->lock); +INIT_LIST_HEAD (&lru->list); + lru->nr_items =0 ; +} + +struct list_lru_array *lru_alloc_array (void ) +{ +struct list_lru_array *lru_array; +int i; + + lru_array =kzalloc (nr_node_ids *sizeof (struct list_lru_node ), +GFP_KERNEL ); +if (!lru_array) +return NULL ; + +for (i =0 ; i < nr_node_ids ; i++) +list_lru_init_one (&lru_array->node[i]); + +return lru_array; +} + +int __list_lru_init (struct list_lru *lru)
lib/list_lru.c list_lru_dispose_all( · line 225
Quoted 10 lines
lib/list_lru.c +2
-4{int i;nodes_clear (lru->active_nodes); -for (i =0 ; i <MAX_NUMNODES ; i++) { -spin_lock_init (&lru->node[i].lock); -INIT_LIST_HEAD (&lru->node[i].list); - lru->node[i].nr_items =0 ; +for (i =0 ; i <MAX_NUMNODES ; i++) +list_lru_init_one (&lru->node[i]);
Hmm. lru_list is up to MAX_NUMNODES, your new one is up to nr_node_ids...
Quoted 9 lines
lib/list_lru.c +9
-0+ +return 0 ; +} + +#ifdef CONFIG_MEMCG_KMEM +static int memcg_init_lru (struct list_lru *lru) +{ +int ret; +
lib/list_lru.c list_lru_dispose_all( · line 244
Quoted 10 lines
lib/list_lru.c +10
-0+if (!lru->memcg_lrus) +return 0 ; + +INIT_LIST_HEAD (&lru->lrus); +mutex_lock (&all_lrus_mutex); +list_add (&lru->lrus, &all_lrus); + ret =memcg_new_lru (lru); +mutex_unlock (&all_lrus_mutex); +return ret; +}
only writer takes this mutex ?
Quoted 27 lines
lib/list_lru.c +27
-0+ +int memcg_update_all_lrus (unsigned long num) +{ +int ret =0 ; +struct list_lru *lru; + +mutex_lock (&all_lrus_mutex); +list_for_each_entry (lru, &all_lrus, lrus) { +if (!lru->memcg_lrus) +continue ; + + ret =memcg_kmem_update_lru_size (lru, num, false); +if (ret) + goto out; + } +out: +mutex_unlock (&all_lrus_mutex); +return ret; +} + +void list_lru_destroy (struct list_lru *lru) +{ +if (!lru->memcg_lrus) +return ; + +mutex_lock (&all_lrus_mutex); +list_del (&lru->lrus);
lib/list_lru.c list_lru_dispose_all( · line 281
Quoted 10 lines
lib/list_lru.c +10
-0+mutex_unlock (&all_lrus_mutex); +} + +void list_lru_destroy_memcg (struct mem_cgroup *memcg) +{ +struct list_lru *lru; +mutex_lock (&all_lrus_mutex); +list_for_each_entry (lru, &all_lrus, lrus) { + lru->memcg_lrus[memcg_cache_id (memcg)] =NULL ; +/* everybody must beaware that this memcg is no longer valid */
Hm, the object pointed by this array entry will be freed by some other func ?
Quoted 77 lines
lib/list_lru.c +17
-0+ wmb(); } +mutex_unlock (&all_lrus_mutex); +} +#else +static int memcg_init_lru (struct list_lru *lru) +{return 0 ; } +#endif + +int list_lru_init (struct list_lru *lru) +{ +int ret; + ret =__list_lru_init (lru); +if (ret) +return ret; + +return memcg_init_lru (lru); +}EXPORT_SYMBOL_GPL (list_lru_init);mm/memcontrol.c +31
-5@@ -3032,16 +3032,30 @@ int memcg_update_cache_sizes(struct mem_cgroup *memcg)memcg_kmem_set_activated (memcg); ret =memcg_update_all_caches (num+1 ); -if (ret) { -ida_simple_remove (&kmem_limited_groups, num); -memcg_kmem_clear_activated (memcg); -return ret; - } +if (ret) + goto out; + +/* +* We should make sure that the array size is not updated until we are +* done; otherwise we have no easy way to know whether or not we should +* grow the array. +*/ + ret =memcg_update_all_lrus (num +1 ); +if (ret) + goto out; memcg->kmemcg_id = num; + +memcg_update_array_size (num +1 ); +INIT_LIST_HEAD (&memcg->memcg_slab_caches);mutex_init (&memcg->slab_caches_mutex); +return 0 ; +out: +ida_simple_remove (&kmem_limited_groups, num); +memcg_kmem_clear_activated (memcg); +return ret; }static size_t memcg_caches_array_size (int num_groups)@@ -3121,6 +3135,106 @@ int memcg_update_cache_size(struct kmem_cache *s, int num_groups)return 0 ; } +/* +* memcg_kmem_update_lru_size - fill in kmemcg info into a list_lru +* +* @lru: the lru we are operating with +* @num_groups: how many kmem-limited cgroups we have +* @new_lru: true if this is a new_lru being created, false if this +* was triggered from the memcg side +* +* Returns 0 on success, and an error code otherwise. +* +* This function can be called either when a new kmem-limited memcg appears, +* or when a new list_lru is created. The work is roughly the same in two cases,
mm/memcontrol.c int memcg_update_cache_size(struct kmem_cache *s, int num_groups) · line 3159
Quoted 10 lines
mm/memcontrol.c +10
-0+* but in the later we never have to expand the array size. +* +* This is always protected by the all_lrus_mutex from the list_lru side. +*/ +int memcg_kmem_update_lru_size (struct list_lru *lru,int num_groups, +bool new_lru) +{ +struct list_lru_array **new_lru_array; +struct list_lru_array *lru_array; +
Both are named as array ...confusing ;)
Quoted 34 lines
mm/memcontrol.c +34
-0+ lru_array =lru_alloc_array (); +if (!lru_array) +return -ENOMEM ; + +/* need some fucked up locking around the list acquisition */ +if ((num_groups > memcg_limited_groups_array_size) || new_lru) { +int i; +struct list_lru_array **old_array; +size_t size =memcg_caches_array_size (num_groups); + + new_lru_array =kzalloc (size *sizeof (void *),GFP_KERNEL ); +if (!new_lru_array) { +kfree (lru_array); +return -ENOMEM ; + } + +for (i =0 ; i < memcg_limited_groups_array_size; i++) { +if (!lru_memcg_is_assigned (lru) || lru->memcg_lrus[i]) +continue ; + new_lru_array[i] = lru->memcg_lrus[i]; + } + + old_array = lru->memcg_lrus; + lru->memcg_lrus = new_lru_array; +/* +* We don't need a barrier here because we are just copying +* information over. Anybody operating in memcg_lrus will +* either follow the new array or the old one and they contain +* exactly the same information. The new space in the end is +* always empty anyway. +* +* We do have to make sure that no more users of the old +* memcg_lrus array exist before we free, and this is achieved +* by the synchronize_lru below.
mm/memcontrol.c int memcg_update_cache_size(struct kmem_cache *s, int num_groups) · line 3203
Quoted 10 lines
mm/memcontrol.c +10
-0+*/ +if (lru_memcg_is_assigned (lru)) { +synchronize_rcu (); +kfree (old_array); + } + + } + +if (lru_memcg_is_assigned (lru)) { + lru->memcg_lrus[num_groups -1 ] = lru_array;
Can't this pointer already set ?
Quoted 50 lines
mm/memcontrol.c +37
-1+/* +* Here we do need the barrier, because of the state transition +* implied by the assignment of the array. All users should be +* able to see it +*/ +wmb (); + } + +return 0 ; + +} + +int memcg_new_lru (struct list_lru *lru) +{ +struct mem_cgroup *iter; + +if (!memcg_kmem_enabled ()) +return 0 ; + +for_each_mem_cgroup (iter) { +int ret; +int memcg_id =memcg_cache_id (iter); +if (memcg_id <0 ) +continue ; + + ret =memcg_kmem_update_lru_size (lru, memcg_id +1 , true); +if (ret) { +mem_cgroup_iter_break (root_mem_cgroup, iter); +return ret; + } + } +return 0 ; +} +int memcg_register_cache (struct mem_cgroup *memcg,struct kmem_cache *s,struct kmem_cache *root_cache) {@@ -5914,8 +6028,10 @@ static void kmem_cgroup_destroy(struct mem_cgroup *memcg)* possible that the charges went down to 0 between mark_dead and the * res_counter read, so in that case, we don't need the put */ -if (memcg_kmem_test_and_clear_dead (memcg)) +if (memcg_kmem_test_and_clear_dead (memcg)) { +list_lru_destroy_memcg (memcg);mem_cgroup_put (memcg); + } }#else static int memcg_init_kmem (struct mem_cgroup *memcg,struct cgroup_subsys *ss)mm/slab_common.c +0
-0
Quoted 1 line
indek
x 3f3cd97..2470d11 100644
mm/slab_common.c int memcg_update_all_caches(int num_memcgs) · line 107
Quoted 11 lines
mm/slab_common.c +0
-1@@ -102,7 +102,6 @@ int memcg_update_all_caches(int num_memcgs) goto out; } -memcg_update_array_size (num_memcgs); out:mutex_unlock (&slab_mutex);return ret;
Thanks,
-Kame
Re: [PATCH 2/7] memcg,list_lru: duplicate LRUs upon kmemcg creation
Quoted 34 lines · Kamezawa Hiroyuki
Quoted 33 lines · Glauber Costa
When a new memcg is created, we need to open up room for its descriptors in all of the
list_lrusthat are marked per-memcg. The process is quite similar to the one we are using for the kmem caches: we initialize the new structures in an array indexed bykmemcg_id, and grow the array if needed. Key data like the size of the array will be shared between the kmem cache code and thelist_lrucode (they basically describe the same thing)Signed-off-by Glauber Costa <[email protected]>Cc Dave Chinner <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Johannes Weiner <[email protected]>Cc Michal Hocko <[email protected]>Cc Hugh Dickins <[email protected]>Cc Kamezawa Hiroyuki <[email protected]>Cc Andrew Morton <[email protected]>--- include/linux/list_lru.h | 47 +++++++++++++++++ include/linux/memcontrol.h | 6 +++ lib/list_lru.c | 115 +++++++++++++++++++++++++++++++++++++--- mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++++++++++--- mm/slab_common.c | 1 - 5 files changed, 283 insertions(+), 14 deletions(-)include/linux/list_lru.h +0
-0@@ -16,11 +16,58 @@ struct list_lru_node {long nr_items; }____cacheline_aligned_in_smp ;
Quoted 6 lines
include/linux/list_lru.h struct list_lru_node { · line 21
Quoted 3 lines
include/linux/list_lru.h +3
-0+struct list_lru_array { +struct list_lru_node node[1 ]; +};size is up to
nr_node_ids?
This is a dynamic quantity, so the correct way to do it is to size it to 1 (or 0 for that matter), have it be the last element of the struct, and then allocate the right size at allocation time.
Quoted 11 lines
include/linux/list_lru.h struct list_lru_node { · line 30
Quoted 9 lines
include/linux/list_lru.h +5
-0+struct list_lru { +struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ];nodemask_t active_nodes; +#ifdef CONFIG_MEMCG_KMEM +struct list_lru_array **memcg_lrus; +#endif };size is up to
memcg_limited_groups_array_size?
ditto. This one not only is a dynamic quantity, but also changes as new memcgs are created.
Quoted 12 lines
include/linux/list_lru.h struct list_lru_node { · line 42
Quoted 9 lines
include/linux/list_lru.h +9
-0+/* +* We will reuse the last bit of the pointer to tell the lru subsystem that +* this particular lru should be replicated when a memcg comes in. +*/ +static inline void lru_memcg_enable (struct list_lru *lru) +{ + lru->memcg_lrus = (void *)0x1ULL ; +} +This "enable" is not used in this patch itself, right ?
I am not sure. It is definitely used later on, I can check and move it if necessary.
Quoted 4 lines
Quoted 4 lines
include/linux/list_lru.h +1
-0+int __list_lru_init (struct list_lru *lru) {int i;
Quoted 10 lines
include/linux/list_lru.h struct list_lru_node {
Quoted 7 lines
include/linux/list_lru.h +2
-4nodes_clear (lru->active_nodes); -for (i =0 ; i <MAX_NUMNODES ; i++) { -spin_lock_init (&lru->node[i].lock); -INIT_LIST_HEAD (&lru->node[i].list); - lru->node[i].nr_items =0 ; +for (i =0 ; i <MAX_NUMNODES ; i++) +list_lru_init_one (&lru->node[i]);Hmm.
lru_listis up toMAX_NUMNODES, your new one is up tonr_node_ids...
well spotted.
Thanks.
Quoted 10 lines
include/linux/list_lru.h
Quoted 7 lines
include/linux/list_lru.h +7
-0+INIT_LIST_HEAD (&lru->lrus); +mutex_lock (&all_lrus_mutex); +list_add (&lru->lrus, &all_lrus); + ret =memcg_new_lru (lru); +mutex_unlock (&all_lrus_mutex); +return ret; +}only writer takes this mutex ?
yes. IIRC, I documented that. But I might be wrong (will check)
Quoted 9 lines
include/linux/list_lru.h
Quoted 7 lines
include/linux/list_lru.h +7
-0+void list_lru_destroy_memcg (struct mem_cgroup *memcg) +{ +struct list_lru *lru; +mutex_lock (&all_lrus_mutex); +list_for_each_entry (lru, &all_lrus, lrus) { + lru->memcg_lrus[memcg_cache_id (memcg)] =NULL ; +/* everybody must beaware that this memcg is no longer valid */ Hm, the object pointed by this array entry will be freed by some other func ?
They should be destroyed before we get here, but I am skimming through the code now, and I see they are not. On a second thought, I think it would be simpler and less error prone if I would just free them here...
Quoted 28 lines
Quoted 28 lines
include/linux/list_lru.h +28
-0+ new_lru_array =kzalloc (size *sizeof (void *),GFP_KERNEL ); +if (!new_lru_array) { +kfree (lru_array); +return -ENOMEM ; + } + +for (i =0 ; i < memcg_limited_groups_array_size; i++) { +if (!lru_memcg_is_assigned (lru) || lru->memcg_lrus[i]) +continue ; + new_lru_array[i] = lru->memcg_lrus[i]; + } + + old_array = lru->memcg_lrus; + lru->memcg_lrus = new_lru_array; +/* +* We don't need a barrier here because we are just copying +* information over. Anybody operating in memcg_lrus will +* either follow the new array or the old one and they contain +* exactly the same information. The new space in the end is +* always empty anyway. +* +* We do have to make sure that no more users of the old +* memcg_lrus array exist before we free, and this is achieved +* by the synchronize_lru below. +*/ +if (lru_memcg_is_assigned (lru)) { +synchronize_rcu (); +kfree (old_array);
Quoted 9 lines
include/linux/list_lru.h
Quoted 6 lines
include/linux/list_lru.h +6
-0+ } + + } + +if (lru_memcg_is_assigned (lru)) { + lru->memcg_lrus[num_groups -1 ] = lru_array;Can't this pointer already set ?
If it is, it is a bug. I can set VM_BUG_ON here to catch those cases.
PATCH 3/7 lru: add an element to a memcg list
In reply to Glauber CostaWith the infrastructure we now have, we can add an element to a memcg LRU list instead of the global list. The memcg lists are still per-node.
Technically, we will never trigger per-node shrinking in the memcg is short of memory. Therefore an alternative to this would be to add the element to *both* a single-node memcg array and a per-node global array.
There are two main reasons for this design choice:
1) adding an extra list_head to each of the objects would waste 16-bytes per object, always remembering that we are talking about 1 dentry + 1 inode in the common case. This means a close to 10 % increase in the dentry size, and a lower yet significant increase in the inode size. In terms of total memory, this design pays 32-byte per-superblock-per-node (size of struct list_lru_node), which means that in any scenario where we have more than 10 dentries + inodes, we would already be paying more memory in the two-list-heads approach than we will here with 1 node x 10 superblocks. The turning point of course depends on the workload, but I hope the figures above would convince you that the memory footprint is in my side in any workload that matters.
2) The main drawback of this, namely, that we loose global LRU order, is not really seen by me as a disadvantage: if we are using memcg to isolate the workloads, global pressure should try to balance the amount reclaimed from all memcgs the same way the shrinkers will already naturally balance the amount reclaimed from each superblock. (This patchset needs some love in this regard, btw).
To help us easily tracking down which nodes have and which nodes doesn't have elements in the list, we will count on an auxiliary node bitmap in the global level.
--- include/linux/list_lru.h | 10 ++++++++ include/linux/memcontrol.h | 10 ++++++++ lib/list_lru.c | 63 ++++++++++++++++++++++++++++++++++++++++------ mm/memcontrol.c | 30 ++++++++++++++++++++++ 4 files changed, 105 insertions(+), 8 deletions(-)
include/linux/list_lru.h +10 -0
@@ -23,6 +23,7 @@ struct list_lru_array {struct list_lru {struct list_head lrus;struct list_lru_node node[MAX_NUMNODES ]; +atomic_long_t node_totals[MAX_NUMNODES ];nodemask_t active_nodes;#ifdef CONFIG_MEMCG_KMEM struct list_lru_array **memcg_lrus;
@@ -53,6 +54,8 @@ struct list_lru_array *lru_alloc_array(void);int memcg_update_all_lrus (unsigned long num);void list_lru_destroy (struct list_lru *lru);void list_lru_destroy_memcg (struct mem_cgroup *memcg); +struct list_lru_node * +lru_node_of_index (struct list_lru *lru,int index,int nid);#else static inline void lru_memcg_enable (struct list_lru *lru) {
@@ -66,6 +69,13 @@ static inline bool lru_memcg_is_assigned(struct list_lru *lru)static inline void list_lru_destroy (struct list_lru *lru) { } + +static inline struct list_lru_node * +lru_node_of_index (struct list_lru *lru,int index,int nid); +{ +BUG_ON (index <0 );/* index != -1 with !MEMCG_KMEM. Impossible */ +return &lru->node[nid]; +}#endif int list_lru_init (struct list_lru *lru);
include/linux/memcontrol.h +10 -0
@@ -24,6 +24,7 @@#include <linux/hardirq.h> #include <linux/jump_label.h> #include <linux/list_lru.h> +#include <linux/mm.h> struct mem_cgroup ;struct page_cgroup ;
@@ -478,6 +479,9 @@ __memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp);int memcg_new_lru (struct list_lru *lru); +struct list_lru_node * +memcg_kmem_lru_of_page (struct list_lru *lru,struct page *page); +int memcg_kmem_update_lru_size (struct list_lru *lru,int num_groups,bool new_lru);
@@ -644,6 +648,12 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp)static inline void kmem_cache_destroy_memcg_children (struct kmem_cache *s) { } + +static inline struct list_lru_node * +memcg_kmem_lru_of_page (struct list_lru *lru,struct page *page) +{ +return &lru->node[page_to_nid (page)]; +}#endif /* CONFIG_MEMCG_KMEM */ #endif /* _LINUX_MEMCONTROL_H */
lib/list_lru.c +55 -8
104 lines not on this page · raw message
mm/memcontrol.c +30 -0
@@ -3319,6 +3319,36 @@ static inline void memcg_resume_kmem_account(void) current->memcg_kmem_skip_account--; } +static struct mem_cgroup *mem_cgroup_from_kmem_page (struct page *page) +{ +struct page_cgroup *pc; +struct mem_cgroup *memcg =NULL ; + + pc =lookup_page_cgroup (page); +if (!PageCgroupUsed (pc)) +return NULL ; + +lock_page_cgroup (pc); +if (PageCgroupUsed (pc)) + memcg = pc->mem_cgroup; +unlock_page_cgroup (pc); +return memcg; +} + +struct list_lru_node * +memcg_kmem_lru_of_page (struct list_lru *lru,struct page *page) +{ +struct mem_cgroup *memcg =mem_cgroup_from_kmem_page (page); +int nid =page_to_nid (page); +int memcg_id; + +if (!memcg_kmem_enabled ()) +return &lru->node[nid]; + + memcg_id =memcg_cache_id (memcg); +return lru_node_of_index (lru, memcg_id, nid); +} +static void kmem_cache_destroy_work_func (struct work_struct *w) {struct kmem_cache *cachep;
Signature · 1 line
1.8.1
Re: [PATCH 3/7] lru: add an element to a memcg list
Quoted 3 lines · Glauber Costa
With the infrastructure we now have, we can add an element to a memcg LRU list instead of the global list. The memcg lists are still per-node.
[...]
Quoted 11 lines
mm/memcontrol.c +3
-0@@ -3319,6 +3319,36 @@ static inline void memcg_resume_kmem_account(void) current->memcg_kmem_skip_account--; } +static struct mem_cgroup *mem_cgroup_from_kmem_page (struct page *page) +{ +struct page_cgroup *pc;
mm/memcontrol.c static inline void memcg_resume_kmem_account(void) · line 3334
Quoted 10 lines
mm/memcontrol.c +10
-0+struct mem_cgroup *memcg =NULL ; + + pc =lookup_page_cgroup (page); +if (!PageCgroupUsed (pc)) +return NULL ; + +lock_page_cgroup (pc); +if (PageCgroupUsed (pc)) + memcg = pc->mem_cgroup; +unlock_page_cgroup (pc);
Once we drop the lock, is there anything that needs protection (e.g. PageCgroupUsed)? If there's no problem, then what's the point of taking the lock?
mm/memcontrol.c static inline void memcg_resume_kmem_account(void) · line 3337
Quoted 3 lines
mm/memcontrol.c +3
-0+return memcg; +} +
[...]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/7] lru: add an element to a memcg list
Quoted 25 lines · Greg Thelen
Quoted 3 lines · Glauber Costa
With the infrastructure we now have, we can add an element to a memcg LRU list instead of the global list. The memcg lists are still per-node.
[...]
Quoted 17 lines
mm/memcontrol.c +9
-0@@ -3319,6 +3319,36 @@ static inline void memcg_resume_kmem_account(void) current->memcg_kmem_skip_account--; } +static struct mem_cgroup *mem_cgroup_from_kmem_page (struct page *page) +{ +struct page_cgroup *pc; +struct mem_cgroup *memcg =NULL ; + + pc =lookup_page_cgroup (page); +if (!PageCgroupUsed (pc)) +return NULL ; +
Quoted 9 lines
mm/memcontrol.c static inline void memcg_resume_kmem_account(void) · line 3334
Quoted 4 lines
mm/memcontrol.c +4
-0+lock_page_cgroup (pc); +if (PageCgroupUsed (pc)) + memcg = pc->mem_cgroup; +unlock_page_cgroup (pc);Once we drop the lock, is there anything that needs protection (e.g. PageCgroupUsed)? If there's no problem, then what's the point of taking the lock?
This is the same pattern already used in the rest of memcg, and I just transposing it here. From my understanding, we need to make sure that if the Used bit is not set, we don't rely on the memcg information. So we take the lock to guarantee that the big is not cleared in the meantime. But after that, we should be fine.
Kame, you have any input?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/7 list_lru: also include memcg lists in counts and scans
In reply to Glauber CostaAs elements are added to per-memcg lists, they will be invisible to global reclaimers. This patch mainly modifies list_lru walk and count functions to take that into account.
Counting is very simple: since we already have total figures for the node, which we use to figure out when to set or clear the node in the bitmap, we can just use that.
For walking, we need to walk the memcg lists as well as the global list. To achieve that, this patch introduces the helper macro for_each_memcg_lru_index. Locking semantics are simple, since introducing a new LRU in the list does not influence the memcg walkers.
The only operation we race against is memcg creation and teardown. For those, barriers should be enough to guarantee that we are seeing up-to-date information and not accessing invalid pointers.
--- include/linux/memcontrol.h | 2 ++ lib/list_lru.c | 90 ++++++++++++++++++++++++++++++++++------------ 2 files changed, 69 insertions(+), 23 deletions(-)
include/linux/memcontrol.h +2 -0
@@ -598,6 +598,8 @@ static inline bool memcg_kmem_is_active(struct mem_cgroup *memcg)#define for_each_memcg_cache_index (_idx) \ for (; NULL; ) +#define memcg_limited_groups_array_size 0 +static inline bool memcg_kmem_enabled (void ) {return false;
lib/list_lru.c +67 -23
132 lines not on this page · raw message
Signature · 1 line
1.8.1
PATCH 5/7 list_lru: per-memcg walks
In reply to Glauber CostaThis patch extend the list_lru interfaces to allow for a memcg parameter. Because most of its users won't need it, instead of modifying the function signatures we create a new set of _memcg() functions and write the old API ontop of that.
At this point, the infrastructure is mostly in place. We already walk the nodes using all memcg indexes, so we just need to make sure we skip all but the one we're interested in. We could just go directly to the memcg of interest, but I am assuming that given the gained simplicity, spending a few cycles here won't hurt *that* much (but that can be improved if needed, of course).
--- include/linux/list_lru.h | 24 ++++++++++++++++++++---- lib/list_lru.c | 41 ++++++++++++++++++++++++++++++++--------- 2 files changed, 52 insertions(+), 13 deletions(-)
include/linux/list_lru.h +20 -4
@@ -81,20 +81,36 @@ lru_node_of_index(struct list_lru *lru, int index, int nid);int list_lru_init (struct list_lru *lru);int list_lru_add (struct list_lru *lru,struct list_head *item);int list_lru_del (struct list_lru *lru,struct list_head *item); -long list_lru_count_nodemask (struct list_lru *lru,nodemask_t *nodes_to_count); + +long list_lru_count_nodemask_memcg (struct list_lru *lru, +nodemask_t *nodes_to_count,struct mem_cgroup *memcg); + +static inline long +list_lru_count_nodemask (struct list_lru *lru,nodemask_t *nodes_to_count) +{ +return list_lru_count_nodemask_memcg (lru, nodes_to_count,NULL ); +}static inline long list_lru_count (struct list_lru *lru) {return list_lru_count_nodemask (lru, &lru->active_nodes); } -typedef int (*list_lru_walk_cb )(struct list_head *item,spinlock_t *lock,void *cb_arg);typedef void (*list_lru_dispose_cb )(struct list_head *dispose_list); -long list_lru_walk_nodemask (struct list_lru *lru,list_lru_walk_cb isolate, -void *cb_arg,long nr_to_walk,nodemask_t *nodes_to_walk); +long list_lru_walk_nodemask_memcg (struct list_lru *lru, +list_lru_walk_cb isolate,void *cb_arg,long nr_to_walk, +nodemask_t *nodes_to_walk,struct mem_cgroup *memcg); + +static inline long list_lru_walk_nodemask (struct list_lru *lru, +list_lru_walk_cb isolate,void *cb_arg,long nr_to_walk, +nodemask_t *nodes_to_walk) +{ +return list_lru_walk_nodemask_memcg (lru, isolate, cb_arg, nr_to_walk, + &lru->active_nodes,NULL ); +}static inline long list_lru_walk (struct list_lru *lru,list_lru_walk_cb isolate,void *cb_arg,long nr_to_walk)
lib/list_lru.c +32 -9
86 lines not on this page · raw message
Signature · 1 line
1.8.1
PATCH 6/7 super: targeted memcg reclaim
In reply to Glauber CostaWe now have all our dentries and inodes placed in memcg-specific LRU lists. All we have to do, is to restrict the reclaim to the said lists in case of memcg pressure.
Marking the superblock shrinker and its LRUs as memcg-aware will guarantee that the shrinkers will get invoked during targetted reclaim.
--- fs/dcache.c | 7 ++++--- fs/inode.c | 6 +++--- fs/internal.h | 5 +++-- fs/super.c | 37 +++++++++++++++++++++++++++---------- 4 files changed, 37 insertions(+), 18 deletions(-)
fs/dcache.c +4 -3
@@ -907,14 +907,15 @@ static int dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock,* This function may fail to free any resources if all the dentries are in * use. */ +long prune_dcache_sb (struct super_block *sb,long nr_to_scan, -nodemask_t *nodes_to_walk) +nodemask_t *nodes_to_walk,struct mem_cgroup *memcg) {LIST_HEAD (dispose);long freed; - freed =list_lru_walk_nodemask (&sb->s_dentry_lru, dentry_lru_isolate, - &dispose, nr_to_scan, nodes_to_walk); + freed =list_lru_walk_nodemask_memcg (&sb->s_dentry_lru, + dentry_lru_isolate, &dispose, nr_to_scan, nodes_to_walk, memcg);shrink_dentry_list (&dispose);return freed; }
fs/inode.c +3 -3
@@ -746,13 +746,13 @@ static int inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock,* then are freed outside inode_lock by dispose_list(). */ long prune_icache_sb (struct super_block *sb,long nr_to_scan, -nodemask_t *nodes_to_walk) +nodemask_t *nodes_to_walk,struct mem_cgroup *memcg) {LIST_HEAD (freeable);long freed; - freed =list_lru_walk_nodemask (&sb->s_inode_lru, inode_lru_isolate, - &freeable, nr_to_scan, nodes_to_walk); + freed =list_lru_walk_nodemask_memcg (&sb->s_inode_lru, + inode_lru_isolate, &freeable, nr_to_scan, nodes_to_walk, memcg);dispose_list (&freeable);return freed; }
fs/internal.h +3 -2
@@ -16,6 +16,7 @@ struct file_system_type;struct linux_binprm ;struct path ;struct mount ; +struct mem_cgroup ;/* * block_dev.c
@@ -111,7 +112,7 @@ extern int open_check_o_direct(struct file *f);*/ extern spinlock_t inode_sb_list_lock;extern long prune_icache_sb (struct super_block *sb,long nr_to_scan, -nodemask_t *nodes_to_scan); +nodemask_t *nodes_to_scan,struct mem_cgroup *memcg);extern void inode_add_lru (struct inode *inode);/*
@@ -128,4 +129,4 @@ extern int invalidate_inodes(struct super_block *, bool);*/ extern struct dentry *__d_alloc (struct super_block *,const struct qstr *);extern long prune_dcache_sb (struct super_block *sb,long nr_to_scan, -nodemask_t *nodes_to_scan); +nodemask_t *nodes_to_scan,struct mem_cgroup *memcg);
fs/super.c +27 -10
@@ -34,6 +34,7 @@#include <linux/cleancache.h> #include <linux/fsnotify.h> #include <linux/lockdep.h> +#include <linux/memcontrol.h> #include "internal.h"
@@ -56,6 +57,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = {static long super_cache_scan (struct shrinker *shrink,struct shrink_control *sc) {struct super_block *sb; +struct mem_cgroup *memcg = sc->target_mem_cgroup;long fs_objects =0 ;long total_objects;long freed =0 ;
@@ -74,11 +76,13 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)if (!grab_super_passive (sb))return -1 ; -if (sb->s_op && sb->s_op->nr_cached_objects) +if (sb->s_op && sb->s_op->nr_cached_objects && !memcg) fs_objects = sb->s_op->nr_cached_objects (sb, &sc->nodes_to_scan); - inodes =list_lru_count_nodemask (&sb->s_inode_lru, &sc->nodes_to_scan); - dentries =list_lru_count_nodemask (&sb->s_dentry_lru, &sc->nodes_to_scan); + inodes =list_lru_count_nodemask_memcg (&sb->s_inode_lru, + &sc->nodes_to_scan, memcg); + dentries =list_lru_count_nodemask_memcg (&sb->s_dentry_lru, + &sc->nodes_to_scan, memcg); total_objects = dentries + inodes + fs_objects +1 ;/* proportion the scan between the caches */
@@ -89,8 +93,8 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)* prune the dcache first as the icache is pinned by it, then * prune the icache, followed by the filesystem specific caches */ - freed =prune_dcache_sb (sb, dentries, &sc->nodes_to_scan); - freed +=prune_icache_sb (sb, inodes, &sc->nodes_to_scan); + freed =prune_dcache_sb (sb, dentries, &sc->nodes_to_scan, memcg); + freed +=prune_icache_sb (sb, inodes, &sc->nodes_to_scan, memcg);if (fs_objects) { fs_objects = (sc->nr_to_scan * fs_objects) / total_objects;
@@ -106,20 +110,26 @@ static long super_cache_count(struct shrinker *shrink, struct shrink_control *sc {struct super_block *sb;long total_objects =0 ; +struct mem_cgroup *memcg = sc->target_mem_cgroup; sb = container_of(shrink, struct super_block, s_shrink);if (!grab_super_passive (sb))return 0 ; -if (sb->s_op && sb->s_op->nr_cached_objects) +/* +* Ideally we would pass memcg to nr_cached_objects, and +* let the underlying filesystem decide. Most likely the +* path will be if (!memcg) return;, but even then. +*/ +if (sb->s_op && sb->s_op->nr_cached_objects && !memcg) total_objects = sb->s_op->nr_cached_objects (sb, &sc->nodes_to_scan); - total_objects +=list_lru_count_nodemask (&sb->s_dentry_lru, - &sc->nodes_to_scan); - total_objects +=list_lru_count_nodemask (&sb->s_inode_lru, - &sc->nodes_to_scan); + total_objects +=list_lru_count_nodemask_memcg (&sb->s_dentry_lru, + &sc->nodes_to_scan, memcg); + total_objects +=list_lru_count_nodemask_memcg (&sb->s_inode_lru, + &sc->nodes_to_scan, memcg); total_objects =vfs_pressure_ratio (total_objects);drop_super (sb);
@@ -198,8 +208,12 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)INIT_HLIST_NODE (&s->s_instances);INIT_HLIST_BL_HEAD (&s->s_anon);INIT_LIST_HEAD (&s->s_inodes); + +lru_memcg_enable (&s->s_dentry_lru);list_lru_init (&s->s_dentry_lru); +lru_memcg_enable (&s->s_inode_lru);list_lru_init (&s->s_inode_lru); +INIT_LIST_HEAD (&s->s_mounts);init_rwsem (&s->s_umount);lockdep_set_class (&s->s_umount, &type->s_umount_key);
@@ -235,6 +249,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags) s->s_shrink.scan_objects = super_cache_scan; s->s_shrink.count_objects = super_cache_count; s->s_shrink.batch =1024 ; + s->s_shrink.memcg_shrinker = true; } out:return s;
@@ -317,6 +332,8 @@ void deactivate_locked_super(struct super_block *s)/* caches are now gone, we can safely kill the shrinker now */ unregister_shrinker (&s->s_shrink); +list_lru_destroy (&s->s_dentry_lru); +list_lru_destroy (&s->s_inode_lru);put_filesystem (fs);put_super (s); }else {
Signature · 1 line
1.8.1
PATCH 7/7 memcg: per-memcg kmem shrinking
In reply to Glauber CostaIf the kernel limit is smaller than the user limit, we will have situations in which our allocations fail but freeing user pages will buy us nothing. In those, we would like to call a specialized memcg reclaimer that only frees kernel memory and leave the user memory alone. Those are also expected to fail when we account memcg->kmem, instead of when we account memcg->res. Based on that, this patch implements a memcg-specific reclaimer, that only shrinks kernel objects, withouth touching user pages.
There might be situations in which there are plenty of objects to shrink, but we can't do it because the __GFP_FS flag is not set. Although they can happen with user pages, they are a lot more common with fs-metadata: this is the case with almost all inode allocation.
Those allocations are, however, capable of waiting. So we can just span a worker, let it finish its job and proceed with the allocation. As slow as it is, at this point we are already past any hopes anyway.
--- include/linux/swap.h | 2 + mm/memcontrol.c | 102 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/vmscan.c | 37 ++++++++++++++++++- 3 files changed, 137 insertions(+), 4 deletions(-)
include/linux/swap.h +2 -0
@@ -259,6 +259,8 @@ extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,extern int __isolate_lru_page (struct page *page,isolate_mode_t mode);extern unsigned long try_to_free_mem_cgroup_pages (struct mem_cgroup *mem,gfp_t gfp_mask,bool noswap); +extern unsigned long try_to_free_mem_cgroup_kmem (struct mem_cgroup *mem, +gfp_t gfp_mask); extern unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *mem, gfp_t gfp_mask, bool noswap, struct zone *zone,
mm/memcontrol.c +99 -3
180 lines not on this page · raw message
mm/vmscan.c +36 -1
@@ -2499,7 +2499,42 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, return nr_reclaimed; } -#endif + +#ifdef CONFIG_MEMCG_KMEM +/* +* This function is called when we are under kmem-specific pressure. It will +* only trigger in environments with kmem.limit_in_bytes < limit_in_bytes, IOW, +* with a lower kmem allowance than the memory allowance. +* +* In this situation, freeing user pages from the cgroup won't do us any good. +* What we really need is to call the memcg-aware shrinkers, in the hope of +* freeing pages holding kmem objects. It may also be that we won't be able to +* free any pages, but will get rid of old objects opening up space for new +* ones. +*/ +unsigned long try_to_free_mem_cgroup_kmem (struct mem_cgroup *memcg, +gfp_t gfp_mask) +{ +struct shrink_control shrink = { + .gfp_mask = gfp_mask, + .target_mem_cgroup = memcg, + }; + +if (!(gfp_mask & __GFP_WAIT)) +return 0 ; + +nodes_setall (shrink.nodes_to_scan); + +/* +* We haven't scanned any user LRU, so we basically come up with +* crafted values of nr_scanned and LRU page (1 and 0 respectively). +* This should be enough to tell shrink_slab that the freeing +* responsibility is all on himself. +*/ +return shrink_slab (&shrink,1 ,0 ); +} +#endif /* CONFIG_MEMCG_KMEM */ +#endif /* CONFIG_MEMCG */ static void age_active_anon (struct zone *zone,struct scan_control *sc) {
Signature · 1 line
1.8.1
Quoted 14 lines · Glauber Costa
This patchset implements targeted shrinking for memcg when kmem limits are present. So far, we've been accounting kernel objects but failing allocations when short of memory. This is because our only option would be to call the global shrinker, depleting objects from all caches and breaking isolation.
This patchset builds upon the recent work from David Chinner
(http://oss.sgi.com/archives/xfs/2012-11/msg00643.html) to implement NUMA aware per-node LRUs. I build heavily on its API, and its presence is implied.The main idea is to associate per-memcg lists with each of the LRUs. The main LRU still provides a single entry point and when adding or removing an element from the LRU, we use the page information to figure out which memcg it belongs to and relay it to the right list.
Quoted 8 lines
This patchset is still not perfect, and some uses cases still need to be dealt with. But I wanted to get this out in the open sooner rather than later. In particular, I have the following (noncomprehensive) todo list:
TODO:
* shrink dead memcgs when global pressure kicks in.
* balance global reclaim among memcgs.
* improve testing and reliability (I am still seeing some stalls in some cases)
Do you have a git tree with these changes so I can see Dave's numa LRUs plus these changes?
Quoted 20 lines · Greg Thelen
Quoted 18 lines · Glauber Costa
This patchset implements targeted shrinking for memcg when kmem limits are present. So far, we've been accounting kernel objects but failing allocations when short of memory. This is because our only option would be to call the global shrinker, depleting objects from all caches and breaking isolation.
This patchset builds upon the recent work from David Chinner
(http://oss.sgi.com/archives/xfs/2012-11/msg00643.html) to implement NUMA aware per-node LRUs. I build heavily on its API, and its presence is implied.The main idea is to associate per-memcg lists with each of the LRUs. The main LRU still provides a single entry point and when adding or removing an element from the LRU, we use the page information to figure out which memcg it belongs to and relay it to the right list.
This patchset is still not perfect, and some uses cases still need to be dealt with. But I wanted to get this out in the open sooner rather than later. In particular, I have the following (noncomprehensive) todo list:
Quoted 8 lines
Quoted 4 lines
TODO:
* shrink dead memcgs when global pressure kicks in.
* balance global reclaim among memcgs.
* improve testing and reliability (I am still seeing some stalls in some cases)Do you have a git tree with these changes so I can see Dave's numa LRUs plus these changes?
I've just uploaded the exact same thing I have sent here to:
git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git
The branch is kmemcg-lru-shrinker. Note that there is also another branch kmemcg-shrinker that contains some other simple patches that were not yet taken and are more stable. I eventually have to merge the two.
I also still need to incorporate the feedback from you and Kame into that. I will be traveling until next Wednesday, so expect changes in there around Thursday.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>