protect page cache from freeing inode
Subsystems (6)
v6 cover protect page cache from freeing inode
On my server there're some running MEMCGs protected by memory.{min, low}, but I found the usage of these MEMCGs abruptly became very small, which were far less than the protect limit. It confused me and finally I found that was because of inode stealing.
Once an inode is freed, all its belonging page caches will be dropped as well, no matter how may page caches it has. So if we intend to protect the page caches in a memcg, we must protect their host (the inode) first. Otherwise the memcg protection can be easily bypassed with freeing inode, especially if there're big files in this memcg.
The inherent mismatch between memcg and inode is a trouble. One inode can be shared by different MEMCGs, but it is a very rare case. If an inode is shared, its belonging page caches may be charged to different MEMCGs. Currently there's no perfect solution to fix this kind of issue, but the inode majority-writer ownership switching can help it more or less.
After this patch, it may take extra time to skip these inodes when workload outside of a memcg protected by memory.min or memory.low is trying to do page reclaim, especially if there're lots of inodes pinned by pagecache in this protected memcg. In order to measure the potential regressions, I constructed bellow test case on my server.
My server is a machine with two nodes, and each of these nodes has 64GB memory. I created two memcgs, and memory.low of these memcgs are both set with 1G. Then I generated more than 500 thousand inodes in each of them, and pagacaches of these inodes are from 4K to 4M. IOW, there're totally more than 1 million xfs_inode in the memory and the total pagecache of them are nearly 128GB. Then I run a workload outside of these two protected memcgs. That workload is usemem in Mel's mmtests with a little modification to alloc almost all the memory and iterate only once. Bellow is the compared result of the Amean of elapsed time and sys%.
5.6.0-rc4 patched
Amean syst-4 65.75 ( 0.00%) 68.08 * -3.54%* Amean elsp-4 32.14 ( 0.00%) 32.63 * -1.52%* Amean syst-7 67.47 ( 0.00%) 66.71 * 1.13%* Amean elsp-7 19.83 ( 0.00%) 18.41 * 7.16%* Amean syst-12 98.27 ( 0.00%) 99.29 * -1.04%* Amean elsp-12 15.60 ( 0.00%) 16.00 * -2.56%* Amean syst-21 174.69 ( 0.00%) 172.92 * 1.01%* Amean elsp-21 14.63 ( 0.00%) 14.75 * -0.82%* Amean syst-30 195.78 ( 0.00%) 205.90 * -5.17%* Amean elsp-30 12.42 ( 0.00%) 12.73 * -2.50%* Amean syst-40 249.85 ( 0.00%) 250.81 * -0.38%* Amean elsp-40 12.19 ( 0.00%) 12.25 * -0.49%*
I did many times. Each time I run this test, I got different result. But the differece is not too big.
Furthmore, this behavior only occurs when memory.min or memory.low is set, and the user already knows that memory.{min, low} can protect the pages at the cost of taking more CPU times, so small extra time is expected by the user.
While if the workload trying to reclaim these protected inodes is inside of a protected memcg, then this workload will not be effected at all because memory.{min, low} doesn't take effect under this condition.
- Changes against v5:
Fix an error pointed by Matthew.
- Changes against v4:
Update with the test result to measure the potential regression. And rebase this patchset on 5.6.0-rc4.
- Changes against v3:
Fix the possible risk pointed by Johannes in another patchset [1]. Per discussion with Johannes in that mail thread, I found that the issue Johannes is trying to fix is different with the issue I'm trying to fix. That's why I update this patchset and post it again. This specific memcg protection issue should be addressed.
- Changes against v2:
1. Seperates memcg patches from this patchset, suggested by Roman.
2. Improves code around the usage of for_each_mem_cgroup(), suggested by Dave
3. Use memcg_low_reclaim passed from scan_control, instead of introducing a new member in struct mem_cgroup.
4. Some other code improvement suggested by Dave.
- Changes against v1:
Use the memcg passed from the shrink_control, instead of getting it from inode itself, suggested by Dave. That could make the laying better.
[1]. https://lore.kernel.org/linux-mm/[email protected]/
Yafang Shao (3):
mm, list_lru: make memcg visible to lru walker isolation function mm, shrinker: make memcg low reclaim visible to lru walker isolation
function
inode: protect page cache from freeing inode
fs/inode.c | 76 ++++++++++++++++++++++++++++++++++++++++++++-- include/linux/memcontrol.h | 18 +++++++++++ include/linux/shrinker.h | 3 ++ mm/list_lru.c | 47 ++++++++++++++++------------ mm/memcontrol.c | 15 --------- mm/vmscan.c | 27 +++++++++------- 6 files changed, 138 insertions(+), 48 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v6 1/3 mm, list_lru: make memcg visible to lru walker isolation function
The lru walker isolation function may use this memcg to do something, e.g. the inode isolatation function will use the memcg to do inode protection in followup patch. So make memcg visible to the lru walker isolation function.
Something should be emphasized in this patch is it replaces for_each_memcg_cache_index() with for_each_mem_cgroup() in list_lru_walk_node(). Because there's a gap between these two MACROs that for_each_mem_cgroup() depends on CONFIG_MEMCG while the other one depends on CONFIG_MEMCG_KMEM. But as list_lru_memcg_aware() returns false if CONFIG_MEMCG_KMEM is not configured, it is safe to this replacement. Another difference between for_each_memcg_cache_index() and for_each_mem_cgroup() is that for_each_memcg_cache_index() excludes the root_mem_cgroup because its kmemcg_id is -1, while for_each_mem_cgroup() includes the root_mem_cgroup. So we need to skip the root_mem_cgroup explicitly in the for loop.
--- include/linux/memcontrol.h | 18 ++++++++++++++++++ mm/list_lru.c | 47 +++++++++++++++++++++++++++------------------- mm/memcontrol.c | 15 --------------- 3 files changed, 46 insertions(+), 34 deletions(-)
include/linux/memcontrol.h +18 -0
@@ -445,6 +445,21 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *,int mem_cgroup_scan_tasks (struct mem_cgroup *,int (*)(struct task_struct *,void *),void *); +/* +* Iteration constructs for visiting all cgroups (under a tree). If +* loops are exited prematurely (break), mem_cgroup_iter_break() must +* be used for reference counting. +*/ +#define for_each_mem_cgroup_tree (iter, root) \ + for (iter = mem_cgroup_iter(root, NULL, NULL); \ + iter != NULL; \ + iter = mem_cgroup_iter(root, iter, NULL)) + +#define for_each_mem_cgroup (iter) \ + for (iter = mem_cgroup_iter(NULL, NULL, NULL); \ + iter != NULL; \ + iter = mem_cgroup_iter(NULL, iter, NULL)) +static inline unsigned short mem_cgroup_id (struct mem_cgroup *memcg) {if (mem_cgroup_disabled ())
@@ -945,6 +960,9 @@ static inline int mem_cgroup_scan_tasks(struct mem_cgroup *memcg,return 0 ; } +#define for_each_mem_cgroup (iter) \ + for (iter = NULL; iter; ) +static inline unsigned short mem_cgroup_id (struct mem_cgroup *memcg) {return 0 ;
mm/list_lru.c +28 -19
78 lines not on this page · raw message
mm/memcontrol.c +0 -15
@@ -222,21 +222,6 @@ enum res_type {/* Used for OOM nofiier */ #define OOM_CONTROL (0) -/* -* Iteration constructs for visiting all cgroups (under a tree). If -* loops are exited prematurely (break), mem_cgroup_iter_break() must -* be used for reference counting. -*/ -#define for_each_mem_cgroup_tree (iter, root) \ - for (iter = mem_cgroup_iter(root, NULL, NULL); \ - iter != NULL; \ - iter = mem_cgroup_iter(root, iter, NULL)) - -#define for_each_mem_cgroup (iter) \ - for (iter = mem_cgroup_iter(NULL, NULL, NULL); \ - iter != NULL; \ - iter = mem_cgroup_iter(NULL, iter, NULL)) -static inline bool should_force_charge (void ) { return tsk_is_oom_victim(current) || fatal_signal_pending(current) ||
Signature · 1 line
1.8.3.1
PATCH v6 2/3 mm, shrinker: make memcg low reclaim visible to lru walker isolation function
In reply to Yafang ShaoA new member memcg_low_reclaim is introduced in shrink_control struct, which is derived from scan_control struct, in order to tell the shrinker whether the reclaim session is under memcg low reclaim or not. The followup patch will use this new member.
--- include/linux/shrinker.h | 3 +++ mm/vmscan.c | 27 ++++++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-)
include/linux/shrinker.h +3 -0
@@ -31,6 +31,9 @@ struct shrink_control {/* current memcg being shrunk (for memcg aware shrinkers) */ struct mem_cgroup *memcg; + +/* derived from struct scan_control */ +bool memcg_low_reclaim; };#define SHRINK_STOP (~0UL)
mm/vmscan.c +16 -11
@@ -625,10 +625,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,/** * shrink_slab - shrink slab caches -* @gfp_mask: allocation context -* @nid: node whose slab caches to target * @memcg: memory cgroup whose slab caches to target -* @priority: the reclaim priority +* @sc: scan_control struct for this reclaim session +* @nid: node whose slab caches to target * * Call the shrink functions to age shrinkable caches. *
@@ -638,15 +637,18 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,* @memcg specifies the memory cgroup to target. Unaware shrinkers * are called only if it is the root cgroup. * -* @priority is sc->priority, we take the number of objects and >> by priority -* in order to get the scan target. +* @sc is the scan_control struct, we take the number of objects +* and >> by sc->priority in order to get the scan target. * * Returns the number of reclaimed slab objects. */ -static unsigned long shrink_slab (gfp_t gfp_mask,int nid, -struct mem_cgroup *memcg, -int priority) +static unsigned long shrink_slab (struct mem_cgroup *memcg, +struct scan_control *sc, +int nid) { +bool memcg_low_reclaim = sc->memcg_low_reclaim; +gfp_t gfp_mask = sc->gfp_mask; +int priority = sc->priority;unsigned long ret, freed =0 ;struct shrinker *shrinker;
@@ -668,6 +670,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, .gfp_mask = gfp_mask, .nid = nid, .memcg = memcg, + .memcg_low_reclaim = memcg_low_reclaim, }; ret =do_shrink_slab (&sc, shrinker, priority);
@@ -694,6 +697,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid,void drop_slab_node (int nid) {unsigned long freed; +struct scan_control sc = { + .gfp_mask =GFP_KERNEL , + }; do { struct mem_cgroup *memcg = NULL;
@@ -701,7 +707,7 @@ void drop_slab_node(int nid) freed =0 ; memcg =mem_cgroup_iter (NULL ,NULL ,NULL );do { - freed +=shrink_slab (GFP_KERNEL , nid, memcg,0 ); + freed +=shrink_slab (memcg, &sc, nid); }while ((memcg =mem_cgroup_iter (NULL , memcg,NULL )) !=NULL ); }while (freed >10 ); }
@@ -2673,8 +2679,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)shrink_lruvec (lruvec, sc); -shrink_slab (sc->gfp_mask, pgdat->node_id, memcg, - sc->priority); +shrink_slab (memcg, sc, pgdat->node_id);/* Record the group's reclaim efficiency */ vmpressure(sc->gfp_mask, memcg, false,
Signature · 1 line
1.8.3.1
PATCH v6 3/3 inode: protect page cache from freeing inode
In reply to Yafang ShaoOn my server there're some running MEMCGs protected by memory.{min, low}, but I found the usage of these MEMCGs abruptly became very small, which were far less than the protect limit. It confused me and finally I found that was because of inode stealing.
Once an inode is freed, all its belonging page caches will be dropped as well, no matter how may page caches it has. So if we intend to protect the page caches in a memcg, we must protect their host (the inode) first. Otherwise the memcg protection can be easily bypassed with freeing inode, especially if there're big files in this memcg.
Supposes we have a memcg, and the stat of this memcg is,
memory.current = 1024M
memory.min = 512MAnd in this memcg there's a inode with 800M page caches.
Once this memcg is scanned by kswapd or other regular reclaimers,
kswapd <<<< It can be either of the regular reclaimers.
shrink_node_memcgs
switch (mem_cgroup_protected()) <<<< Not protected
case MEMCG_PROT_NONE: <<<< Will scan this memcg
beak;
shrink_lruvec() <<<< Reclaim the page caches
shrink_slab() <<<< It may free this inode and drop all its
page caches(800M).
So we must protect the inode first if we want to protect page caches. Note that this inode may be a cold inode (in the tail of list lru), because memcg protection protects all slabs and page cache pages whatever they are cold or hot. IOW, this is a memcg-protection-specific issue.
The inherent mismatch between memcg and inode is a trouble. One inode can be shared by different MEMCGs, but it is a very rare case. If an inode is shared, its belonging page caches may be charged to different MEMCGs. Currently there's no perfect solution to fix this kind of issue, but the inode majority-writer ownership switching can help it more or less.
--- fs/inode.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 72 insertions(+), 3 deletions(-)
fs/inode.c +72 -3
@@ -55,6 +55,12 @@* inode_hash_lock */ +struct inode_isolate_control { +struct list_head *freeable; +struct mem_cgroup *memcg;/* derived from shrink_control */ +bool memcg_low_reclaim;/* derived from scan_control */ +}; +static unsigned int i_hash_mask__read_mostly ;static unsigned int i_hash_shift__read_mostly ;static struct hlist_head *inode_hashtable__read_mostly ;
@@ -715,6 +721,58 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)return busy; } +#ifdef CONFIG_MEMCG_KMEM +/* +* Once an inode is freed, all its belonging page caches will be dropped as +* well, even if there're lots of page caches. So if we intend to protect +* page caches in a memcg, we must protect their host(the inode) first. +* Otherwise the memcg protection can be easily bypassed with freeing inode, +* especially if there're big files in this memcg. +* Note that it may happen that the page caches are already charged to the +* memcg, but the inode hasn't been added to this memcg yet. In this case, +* this inode is not protected. +* The inherent mismatch between memcg and inode is a trouble. One inode +* can be shared by different MEMCGs, but it is a very rare case. If +* an inode is shared, its belonging page caches may be charged to +* different MEMCGs. Currently there's no perfect solution to fix this +* kind of issue, but the inode majority-writer ownership switching can +* help it more or less. +*/ +static bool memcg_can_reclaim_inode (struct inode *inode, +struct inode_isolate_control *iic) +{ +unsigned long protection; +struct mem_cgroup *memcg; +bool reclaimable = true; + +if (!inode->i_data.nrpages) + goto out; + +/* Excludes freeing inode via drop_caches */ +if (!current->reclaim_state) + goto out; + + memcg = iic->memcg; +if (!memcg || memcg == root_mem_cgroup) + goto out; + + protection =mem_cgroup_protection (memcg, iic->memcg_low_reclaim); +if (!protection) + goto out; + + reclaimable = false; + +out: +return reclaimable; +} +#else /* CONFIG_MEMCG_KMEM */ +static bool memcg_can_reclaim_inode (struct inode *inode, +struct inode_isolate_control *iic) +{ +return true; +} +#endif /* CONFIG_MEMCG_KMEM */ +/* * Isolate the inode from the LRU in preparation for freeing it. *
@@ -733,8 +791,9 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)static enum lru_status inode_lru_isolate (struct list_head *item,struct list_lru_one *lru,spinlock_t *lru_lock,void *arg) { -struct list_head *freeable = arg; - struct inode *inode = container_of(item, struct inode, i_lru); +struct inode_isolate_control *iic = arg; +struct list_head *freeable = iic->freeable; + struct inode *inode = container_of(item, struct inode, i_lru);/* * we are inverting the lru lock/inode->i_lock here, so use a trylock.
@@ -743,6 +802,11 @@ static enum lru_status inode_lru_isolate(struct list_head *item,if (!spin_trylock (&inode->i_lock))return LRU_SKIP ; +if (!memcg_can_reclaim_inode (inode, iic)) { +spin_unlock (&inode->i_lock); +return LRU_ROTATE ; + } +/* * Referenced or dirty inodes are still in use. Give them another pass * through the LRU as we canot reclaim them now.
@@ -800,9 +864,14 @@ long prune_icache_sb(struct super_block *sb, struct shrink_control *sc) {LIST_HEAD (freeable);long freed; +struct inode_isolate_control iic = { + .freeable = &freeable, + .memcg = sc->memcg, + .memcg_low_reclaim = sc->memcg_low_reclaim, + }; freed =list_lru_shrink_walk (&sb->s_inode_lru, sc, - inode_lru_isolate, &freeable); + inode_lru_isolate, &iic);dispose_list (&freeable);return freed; }
Signature · 1 line
1.8.3.1