Skip to contentSkip to conversation

kmemcg shrinkers

PATCH Superseded Glauber Costa 6 participants 63 messages May 12, 2013 – May 18, 2013

Subsystems (32)

v6 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v6 cover kmemcg shrinkers

Initial notes:
==============

Mel, Dave, this is the last round of fixes I have for the series. The fixes are few, and I was mostly interested in getting this out based on an up2date tree so Dave can verify it. This should apply fine ontop of Friday's linux-next. Alternatively, please grab from branch "kmemcg-lru-shrinker" at:

git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git

Main changes from *v5:
* Rebased to linux-next, and fix the conflicts with the dcache.
* Make sure LRU_RETRY only retry once
* Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function)
* Fix i915 return code when mutex cannot be acquired.
* Only scan less-than-batch objects in memcg scenarios

Hi,

This patchset implements targeted shrinking for memcg when kmem limits are present. So far, we've been accounting kernel objects but failing allocations when short of memory. This is because our only option would be to call the global shrinker, depleting objects from all caches and breaking isolation.

The main idea is to associate per-memcg lists with each of the LRUs. The main LRU still provides a single entry point and when adding or removing an element from the LRU, we use the page information to figure out which memcg it belongs to and relay it to the right list.

Base work:
==========

Please note that this builds upon the recent work from Dave Chinner that sanitizes the LRU shrinking API and make the shrinkers node aware. Node awareness is not *strictly* needed for my work, but I still perceive it as an advantage. The API unification is a major need, and I build upon it heavily. That allows us to manipulate the LRUs without knowledge of the underlying objects with ease. This time, I am including that work here as a baseline.

Main changes from *v5:
* Rebased to linux-next, and fix the conflicts with the dcache.
* Make sure LRU_RETRY only retry once
* Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function)
* Fix i915 return code when mutex cannot be acquired.
* Only scan less-than-batch objects in memcg scenarios

Main changes from *v4:
* Fixed a bug in user-generated memcg pressure
* Fixed overly-agressive slab shrinker behavior spotted by Mel Gorman
* Various other fixes and comments by Mel Gorman

Main changes from *v3:
* Merged suggestions from mailing list.
* Removed the memcg-walking code from LRU. vmscan now drives all the hierarchy decisions, which makes more sense
* lazily free the old memcg arrays (needs now to be saved in struct lru). Since we need to call synchronize_rcu, calling it for every LRU can become expensive
* Moved the dead memcg shrinker to vmpressure. Already independently sent to linux-mm for review.
* Changed locking convention for LRU_RETRY. It now needs to return locked, which silents warnings about possible lock unbalance (although previous code was correct)

Main changes from *v2:
* shrink dead memcgs when global pressure kicks in. Uses the new lru API.
* bugfixes and comments from the mailing list.
* proper hierarchy-aware walk in shrink_slab.

Main changes from *v1:
* merged comments from the mailing list
* reworked lru-memcg API
* effective proportional shrinking
* sanitized locking on the memcg side
* bill user memory first when kmem == umem
* various bugfixes

Dave Chinner (17):
dcache: convert dentry_stat.nr_unused to per-cpu counters
dentry: move to per-sb LRU locks
dcache: remove dentries from LRU before putting on dispose list
mm: new shrinker API
shrinker: convert superblock shrinkers to new API
list: add a new LRU list type
inode: convert inode lru list to generic lru list code.
dcache: convert to use new lru list infrastructure
list_lru: per-node list infrastructure
shrinker: add node awareness
fs: convert inode and dentry shrinking to be node aware
xfs: convert buftarg LRU to generic code
xfs: convert dquot cache lru to list_lru
fs: convert fs shrinkers to new scan/count API
drivers: convert shrinkers to new count/scan API
shrinker: convert remaining shrinkers to count/scan API
shrinker: Kill old ->shrink API.

Glauber Costa (14):
super: fix calculation of shrinkable objects for small numbers
i915: bail out earlier when shrinker cannot acquire mutex
hugepage: convert huge zero page shrinker to new shrinker API
vmscan: also shrink slab in memcg pressure memcg,list_lru: duplicate LRUs upon kmemcg creation
lru: add an element to a memcg list
list_lru: per-memcg walks
memcg: per-memcg kmem shrinking
memcg: scan cache objects hierarchically
vmscan: take at least one pass with shrinkers
super: targeted memcg reclaim
memcg: move initialization to memcg creation
vmpressure: in-kernel notifications
memcg: reap dead memcgs upon global memory pressure.

 arch/x86/kvm/mmu.c                        |  28 +-
 drivers/gpu/drm/i915/i915_dma.c           |   4 +-
 drivers/gpu/drm/i915/i915_gem.c           |  71 +++--
 drivers/gpu/drm/ttm/ttm_page_alloc.c      |  48 ++--
 drivers/gpu/drm/ttm/ttm_page_alloc_dma.c  |  55 ++--
 drivers/md/bcache/btree.c                 |  43 +--
 drivers/md/bcache/sysfs.c                 |   2 +-
 drivers/md/dm-bufio.c                     |  65 +++--
 drivers/staging/android/ashmem.c          |  46 +++-
 drivers/staging/android/lowmemorykiller.c |  40 +--
 drivers/staging/zcache/zcache-main.c      |  29 +-
 fs/dcache.c                               | 240 ++++++++++-------
 fs/drop_caches.c                          |   1 +
 fs/ext4/extents_status.c                  |  30 ++-
 fs/gfs2/glock.c                           |  30 ++-
 fs/gfs2/main.c                            |   3 +-
 fs/gfs2/quota.c                           |  14 +-
 fs/gfs2/quota.h                           |   4 +-
 fs/inode.c                                | 175 +++++-------
 fs/internal.h                             |   5 +
 fs/mbcache.c                              |  53 ++--
 fs/nfs/dir.c                              |  20 +-
 fs/nfs/internal.h                         |   4 +-
 fs/nfs/super.c                            |   3 +-
 fs/nfsd/nfscache.c                        |  31 ++-
 fs/quota/dquot.c                          |  39 ++-
 fs/super.c                                | 107 +++++---
 fs/ubifs/shrinker.c                       |  20 +-
 fs/ubifs/super.c                          |   3 +-
 fs/ubifs/ubifs.h                          |   3 +-
 fs/xfs/xfs_buf.c                          | 169 ++++++------
 fs/xfs/xfs_buf.h                          |   5 +-
 fs/xfs/xfs_dquot.c                        |   7 +-
 fs/xfs/xfs_icache.c                       |   4 +-
 fs/xfs/xfs_icache.h                       |   2 +-
 fs/xfs/xfs_qm.c                           | 275 +++++++++----------
 fs/xfs/xfs_qm.h                           |   4 +-
 fs/xfs/xfs_super.c                        |  12 +-
 include/linux/dcache.h                    |   4 +
 include/linux/fs.h                        |  25 +-
 include/linux/list_lru.h                  | 145 ++++++++++
 include/linux/memcontrol.h                |  45 ++++
 include/linux/shrinker.h                  |  44 ++-
 include/linux/swap.h                      |   2 +
 include/linux/vmpressure.h                |   6 +
 include/trace/events/vmscan.h             |   4 +-
 lib/Makefile                              |   2 +-
 lib/list_lru.c                            | 433 ++++++++++++++++++++++++++++++
 mm/huge_memory.c                          |  17 +-
 mm/memcontrol.c                           | 430 +++++++++++++++++++++++++----
 mm/memory-failure.c                       |   2 +
 mm/slab_common.c                          |   1 -
 mm/vmpressure.c                           |  52 +++-
 mm/vmscan.c                               | 325 +++++++++++++++-------
 net/sunrpc/auth.c                         |  45 +++-
 55 files changed, 2339 insertions(+), 937 deletions(-)
 create mode 100644 include/linux/list_lru.h
 create mode 100644 lib/list_lru.c
Signature · 1 line
1.8.1.4
#1 lore raw

PATCH v6 1/31 super: fix calculation of shrinkable objects for small numbers

The sysctl knob sysctl_vfs_cache_pressure is used to determine which percentage of the shrinkable objects in our cache we should actively try to shrink.

It works great in situations in which we have many objects (at least more than 100), because the aproximation errors will be negligible. But if this is not the case, specially when total_objects < 100, we may end up concluding that we have no objects at all (total / 100 = 0, if total < 100).

This is certainly not the biggest killer in the world, but may matter in very low kernel memory situations.

[ v2: fix it for all occurrences of sysctl_vfs_cache_pressure ]

---
 fs/gfs2/glock.c        |  2 +-
 fs/gfs2/quota.c        |  2 +-
 fs/mbcache.c           |  2 +-
 fs/nfs/dir.c           |  2 +-
 fs/quota/dquot.c       |  5 ++---
 fs/super.c             | 14 +++++++-------
 fs/xfs/xfs_qm.c        |  2 +-
 include/linux/dcache.h |  4 ++++
 8 files changed, 18 insertions(+), 15 deletions(-)
fs/gfs2/glock.c +1 -1
index 9435384..3bd2748 100644
@@ -1463,7 +1463,7 @@ static int gfs2_shrink_glock_memory(struct shrinker *shrink,
 		gfs2_scan_glock_lru(sc->nr_to_scan);
 	}
 
-	return (atomic_read(&lru_count) / 100) * sysctl_vfs_cache_pressure;
+	return vfs_pressure_ratio(atomic_read(&lru_count));
 }
 
 static struct shrinker glock_shrinker = {
fs/gfs2/quota.c +1 -1
index c7c840e..5c14206 100644
@@ -114,7 +114,7 @@ int gfs2_shrink_qd_memory(struct shrinker *shrink, struct shrink_control *sc)
 	spin_unlock(&qd_lru_lock);
 
 out:
-	return (atomic_read(&qd_lru_count) * sysctl_vfs_cache_pressure) / 100;
+	return vfs_pressure_ratio(atomic_read(&qd_lru_count));
 }
 
 static u64 qd2index(struct gfs2_quota_data *qd)
fs/mbcache.c +1 -1
index 8c32ef3..5eb0476 100644
@@ -189,7 +189,7 @@ mb_cache_shrink_fn(struct shrinker *shrink, struct shrink_control *sc)
 	list_for_each_entry_safe(entry, tmp, &free_list, e_lru_list) {
 		__mb_cache_entry_forget(entry, gfp_mask);
 	}
-	return (count / 100) * sysctl_vfs_cache_pressure;
+	return vfs_pressure_ratio(count);
 }
 
 
fs/nfs/dir.c +1 -1
index e093e73..54d7c47 100644
@@ -1998,7 +1998,7 @@ remove_lru_entry:
 	}
 	spin_unlock(&nfs_access_lru_lock);
 	nfs_access_free_list(&head);
-	return (atomic_long_read(&nfs_access_nr_entries) / 100) * sysctl_vfs_cache_pressure;
+	return vfs_pressure_ratio(atomic_long_read(&nfs_access_nr_entries));
 }
 
 static void __nfs_access_zap_cache(struct nfs_inode *nfsi, struct list_head *head)
fs/quota/dquot.c +2 -3
index 3e64169..762b09c 100644
@@ -719,9 +719,8 @@ static int shrink_dqcache_memory(struct shrinker *shrink,
 		prune_dqcache(nr);
 		spin_unlock(&dq_list_lock);
 	}
-	return ((unsigned)
-		percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS])
-		/100) * sysctl_vfs_cache_pressure;
+	return vfs_pressure_ratio(
+	percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));
 }
 
 static struct shrinker dqcache_shrinker = {
fs/super.c +7 -7
index 7465d43..2a37fd6 100644
@@ -82,13 +82,13 @@ static int prune_super(struct shrinker *shrink, struct shrink_control *sc)
 		int	inodes;
 
 		/* proportion the scan between the caches */
-		dentries = (sc->nr_to_scan * sb->s_nr_dentry_unused) /
-							total_objects;
-		inodes = (sc->nr_to_scan * sb->s_nr_inodes_unused) /
-							total_objects;
+		dentries = mult_frac(sc->nr_to_scan, sb->s_nr_dentry_unused,
+							total_objects);
+		inodes = mult_frac(sc->nr_to_scan, sb->s_nr_inodes_unused,
+							total_objects);
 		if (fs_objects)
-			fs_objects = (sc->nr_to_scan * fs_objects) /
-							total_objects;
+			fs_objects = mult_frac(sc->nr_to_scan, fs_objects,
+							total_objects);
 		/*
 		 * prune the dcache first as the icache is pinned by it, then
 		 * prune the icache, followed by the filesystem specific caches
@@ -104,7 +104,7 @@ static int prune_super(struct shrinker *shrink, struct shrink_control *sc)
 				sb->s_nr_inodes_unused + fs_objects;
 	}
 
-	total_objects = (total_objects / 100) * sysctl_vfs_cache_pressure;
+	total_objects = vfs_pressure_ratio(total_objects);
 	drop_super(sb);
 	return total_objects;
 }
fs/xfs/xfs_qm.c +1 -1
index f41702b..7ade175 100644
@@ -1585,7 +1585,7 @@ xfs_qm_shake(
 	}
 
 out:
-	return (qi->qi_lru_count / 100) * sysctl_vfs_cache_pressure;
+	return vfs_pressure_ratio(qi->qi_lru_count);
 }
 
 /*
include/linux/dcache.h +4 -0
index 1a6bb81..4d24a12 100644
@@ -411,4 +411,8 @@ static inline bool d_mountpoint(struct dentry *dentry)
 
 extern int sysctl_vfs_cache_pressure;
 
+static inline unsigned long vfs_pressure_ratio(unsigned long val)
+{
+	return mult_frac(val, sysctl_vfs_cache_pressure, 100);
+}
 #endif	/* __LINUX_DCACHE_H */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

PATCH v6 2/31 dcache: convert dentry_stat.nr_unused to per-cpu counters

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Before we split up the dcache_lru_lock, the unused dentry counter needs to be made independent of the global dcache_lru_lock. Convert it to per-cpu counters to do this.

[ v5: comment about possible cpus ]
---
 fs/dcache.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
fs/dcache.c +15 -3
index f09b908..cb304f4 100644
@@ -118,8 +118,10 @@ struct dentry_stat_t dentry_stat = {
 };
 
 static DEFINE_PER_CPU(unsigned int, nr_dentry);
+static DEFINE_PER_CPU(unsigned int, nr_dentry_unused);
 
 #if defined(CONFIG_SYSCTL) && defined(CONFIG_PROC_FS)
+/* scan possible cpus instead of online and avoid worrying about CPU hotplug. */
 static int get_nr_dentry(void)
 {
 	int i;
@@ -129,10 +131,20 @@ static int get_nr_dentry(void)
 	return sum < 0 ? 0 : sum;
 }
 
+static int get_nr_dentry_unused(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_dentry_unused, i);
+	return sum < 0 ? 0 : sum;
+}
+
 int proc_nr_dentry(ctl_table *table, int write, void __user *buffer,
 		   size_t *lenp, loff_t *ppos)
 {
 	dentry_stat.nr_dentry = get_nr_dentry();
+	dentry_stat.nr_unused = get_nr_dentry_unused();
 	return proc_dointvec(table, write, buffer, lenp, ppos);
 }
 #endif
@@ -312,7 +324,7 @@ static void dentry_lru_add(struct dentry *dentry)
 		spin_lock(&dcache_lru_lock);
 		list_add(&dentry->d_lru, &dentry->d_sb->s_dentry_lru);
 		dentry->d_sb->s_nr_dentry_unused++;
-		dentry_stat.nr_unused++;
+		this_cpu_inc(nr_dentry_unused);
 		spin_unlock(&dcache_lru_lock);
 	}
 }
@@ -322,7 +334,7 @@ static void __dentry_lru_del(struct dentry *dentry)
 	list_del_init(&dentry->d_lru);
 	dentry->d_flags &= ~DCACHE_SHRINK_LIST;
 	dentry->d_sb->s_nr_dentry_unused--;
-	dentry_stat.nr_unused--;
+	this_cpu_dec(nr_dentry_unused);
 }
 
 /*
@@ -343,7 +355,7 @@ static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 	if (list_empty(&dentry->d_lru)) {
 		list_add_tail(&dentry->d_lru, list);
 		dentry->d_sb->s_nr_dentry_unused++;
-		dentry_stat.nr_unused++;
+		this_cpu_inc(nr_dentry_unused);
 	} else {
 		list_move_tail(&dentry->d_lru, list);
 	}
Signature · 1 line
1.8.1.4
#3 lore raw

PATCH v6 3/31 dentry: move to per-sb LRU locks

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

With the dentry LRUs being per-sb structures, there is no real need for a global dentry_lru_lock. The locking can be made more fine-grained by moving to a per-sb LRU lock, isolating the LRU operations of different filesytsems completely from each other.

---
 fs/dcache.c        | 33 ++++++++++++++++-----------------
 fs/super.c         |  1 +
 include/linux/fs.h |  4 +++-
 3 files changed, 20 insertions(+), 18 deletions(-)
fs/dcache.c +16 -17
index cb304f4..795c15d 100644
@@ -48,7 +48,7 @@
  *   - the dcache hash table
  * s_anon bl list spinlock protects:
  *   - the s_anon list (see __d_drop)
- * dcache_lru_lock protects:
+ * dentry->d_sb->s_dentry_lru_lock protects:
  *   - the dcache lru lists and counters
  * d_lock protects:
  *   - d_flags
@@ -63,7 +63,7 @@
  * Ordering:
  * dentry->d_inode->i_lock
  *   dentry->d_lock
- *     dcache_lru_lock
+ *     dentry->d_sb->s_dentry_lru_lock
  *     dcache_hash_bucket lock
  *     s_anon lock
  *
@@ -81,7 +81,6 @@
 int sysctl_vfs_cache_pressure __read_mostly = 100;
 EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
 
-static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
 __cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
 
 EXPORT_SYMBOL(rename_lock);
@@ -321,11 +320,11 @@ static void dentry_unlink_inode(struct dentry * dentry)
 static void dentry_lru_add(struct dentry *dentry)
 {
 	if (list_empty(&dentry->d_lru)) {
-		spin_lock(&dcache_lru_lock);
+		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		list_add(&dentry->d_lru, &dentry->d_sb->s_dentry_lru);
 		dentry->d_sb->s_nr_dentry_unused++;
 		this_cpu_inc(nr_dentry_unused);
-		spin_unlock(&dcache_lru_lock);
+		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 	}
 }
 
@@ -343,15 +342,15 @@ static void __dentry_lru_del(struct dentry *dentry)
 static void dentry_lru_del(struct dentry *dentry)
 {
 	if (!list_empty(&dentry->d_lru)) {
-		spin_lock(&dcache_lru_lock);
+		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		__dentry_lru_del(dentry);
-		spin_unlock(&dcache_lru_lock);
+		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 	}
 }
 
 static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 {
-	spin_lock(&dcache_lru_lock);
+	spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 	if (list_empty(&dentry->d_lru)) {
 		list_add_tail(&dentry->d_lru, list);
 		dentry->d_sb->s_nr_dentry_unused++;
@@ -359,7 +358,7 @@ static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 	} else {
 		list_move_tail(&dentry->d_lru, list);
 	}
-	spin_unlock(&dcache_lru_lock);
+	spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 }
 
 /**
@@ -839,14 +838,14 @@ void prune_dcache_sb(struct super_block *sb, int count)
 	LIST_HEAD(tmp);
 
 relock:
-	spin_lock(&dcache_lru_lock);
+	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
 		dentry = list_entry(sb->s_dentry_lru.prev,
 				struct dentry, d_lru);
 		BUG_ON(dentry->d_sb != sb);
 
 		if (!spin_trylock(&dentry->d_lock)) {
-			spin_unlock(&dcache_lru_lock);
+			spin_unlock(&sb->s_dentry_lru_lock);
 			cpu_relax();
 			goto relock;
 		}
@@ -862,11 +861,11 @@ relock:
 			if (!--count)
 				break;
 		}
-		cond_resched_lock(&dcache_lru_lock);
+		cond_resched_lock(&sb->s_dentry_lru_lock);
 	}
 	if (!list_empty(&referenced))
 		list_splice(&referenced, &sb->s_dentry_lru);
-	spin_unlock(&dcache_lru_lock);
+	spin_unlock(&sb->s_dentry_lru_lock);
 
 	shrink_dentry_list(&tmp);
 }
@@ -882,14 +881,14 @@ void shrink_dcache_sb(struct super_block *sb)
 {
 	LIST_HEAD(tmp);
 
-	spin_lock(&dcache_lru_lock);
+	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
 		list_splice_init(&sb->s_dentry_lru, &tmp);
-		spin_unlock(&dcache_lru_lock);
+		spin_unlock(&sb->s_dentry_lru_lock);
 		shrink_dentry_list(&tmp);
-		spin_lock(&dcache_lru_lock);
+		spin_lock(&sb->s_dentry_lru_lock);
 	}
-	spin_unlock(&dcache_lru_lock);
+	spin_unlock(&sb->s_dentry_lru_lock);
 }
 EXPORT_SYMBOL(shrink_dcache_sb);
 
fs/super.c +1 -0
index 2a37fd6..0be75fb 100644
@@ -182,6 +182,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)
 		INIT_HLIST_BL_HEAD(&s->s_anon);
 		INIT_LIST_HEAD(&s->s_inodes);
 		INIT_LIST_HEAD(&s->s_dentry_lru);
+		spin_lock_init(&s->s_dentry_lru_lock);
 		INIT_LIST_HEAD(&s->s_inode_lru);
 		spin_lock_init(&s->s_inode_lru_lock);
 		INIT_LIST_HEAD(&s->s_mounts);
include/linux/fs.h +3 -1
index 0a9a6766..67e1040 100644
@@ -1263,7 +1263,9 @@ struct super_block {
 	struct list_head	s_files;
 #endif
 	struct list_head	s_mounts;	/* list of mounts; _not_ for fs use */
-	/* s_dentry_lru, s_nr_dentry_unused protected by dcache.c lru locks */
+
+	/* s_dentry_lru_lock protects s_dentry_lru and s_nr_dentry_unused */
+	spinlock_t		s_dentry_lru_lock ____cacheline_aligned_in_smp;
 	struct list_head	s_dentry_lru;	/* unused dentry lru */
 	int			s_nr_dentry_unused;	/* # of dentry on lru */
 
Signature · 1 line
1.8.1.4
#4 lore raw

PATCH v6 4/31 dcache: remove dentries from LRU before putting on dispose list

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

One of the big problems with modifying the way the dcache shrinker and LRU implementation works is that the LRU is abused in several ways. One of these is shrink_dentry_list().

Basically, we can move a dentry off the LRU onto a different list without doing any accounting changes, and then use dentry_lru_prune() to remove it from what-ever list it is now on to do the LRU accounting at that point.

This makes it -really hard- to change the LRU implementation. The use of the per-sb LRU lock serialises movement of the dentries between the different lists and the removal of them, and this is the only reason that it works. If we want to break up the dentry LRU lock and lists into, say, per-node lists, we remove the only serialisation that allows this lru list/dispose list abuse to work.

To make this work effectively, the dispose list has to be isolated from the LRU list - dentries have to be removed from the LRU *before* being placed on the dispose list. This means that the LRU accounting and isolation is completed before disposal is started, and that means we can change the LRU implementation freely in future.

This means that dentries *must* be marked with DCACHE_SHRINK_LIST when they are placed on the dispose list so that we don't think that parent dentries found in try_prune_one_dentry() are on the LRU when the are actually on the dispose list. This would result in accounting the dentry to the LRU a second time. Hence dentry_lru_prune() has to handle the DCACHE_SHRINK_LIST case differently because the dentry isn't on the LRU list.

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ]
---
 fs/dcache.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 71 insertions(+), 22 deletions(-)
fs/dcache.c +71 -22
index 795c15d..868abf9 100644
@@ -331,7 +331,6 @@ static void dentry_lru_add(struct dentry *dentry)
 static void __dentry_lru_del(struct dentry *dentry)
 {
 	list_del_init(&dentry->d_lru);
-	dentry->d_flags &= ~DCACHE_SHRINK_LIST;
 	dentry->d_sb->s_nr_dentry_unused--;
 	this_cpu_dec(nr_dentry_unused);
 }
@@ -341,6 +340,8 @@ static void __dentry_lru_del(struct dentry *dentry)
  */
 static void dentry_lru_del(struct dentry *dentry)
 {
+	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
+
 	if (!list_empty(&dentry->d_lru)) {
 		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		__dentry_lru_del(dentry);
@@ -348,15 +349,39 @@ static void dentry_lru_del(struct dentry *dentry)
 	}
 }
 
+static void dentry_lru_prune(struct dentry *dentry)
+{
+	/*
+	 * inform the fs via d_prune that this dentry is about to be
+	 * unhashed and destroyed.
+	 */
+	if (dentry->d_flags & DCACHE_OP_PRUNE)
+		dentry->d_op->d_prune(dentry);
+
+	if (list_empty(&dentry->d_lru))
+		return;
+
+	if ((dentry->d_flags & DCACHE_SHRINK_LIST)) {
+		list_del_init(&dentry->d_lru);
+		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+	} else {
+		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
+		__dentry_lru_del(dentry);
+		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
+	}
+}
+
 static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 {
+	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
+
 	spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 	if (list_empty(&dentry->d_lru)) {
 		list_add_tail(&dentry->d_lru, list);
-		dentry->d_sb->s_nr_dentry_unused++;
-		this_cpu_inc(nr_dentry_unused);
 	} else {
 		list_move_tail(&dentry->d_lru, list);
+		dentry->d_sb->s_nr_dentry_unused--;
+		this_cpu_dec(nr_dentry_unused);
 	}
 	spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 }
@@ -479,14 +504,8 @@ relock:
 
 	if (ref)
 		dentry->d_count--;
-	/*
-	 * inform the fs via d_prune that this dentry is about to be
-	 * unhashed and destroyed.
-	 */
-	if (dentry->d_flags & DCACHE_OP_PRUNE)
-		dentry->d_op->d_prune(dentry);
 
-	dentry_lru_del(dentry);
+	dentry_lru_prune(dentry);
 	/* if it was on the hash then remove it */
 	__d_drop(dentry);
 	return d_kill(dentry, parent);
@@ -800,12 +819,18 @@ static void shrink_dentry_list(struct list_head *list)
 		}
 
 		/*
+		 * The dispose list is isolated and dentries are not accounted
+		 * to the LRU here, so we can simply remove it from the list
+		 * here regardless of whether it is referenced or not.
+		 */
+		list_del_init(&dentry->d_lru);
+
+		/*
 		 * We found an inuse dentry which was not removed from
-		 * the LRU because of laziness during lookup.  Do not free
-		 * it - just keep it off the LRU list.
+		 * the LRU because of laziness during lookup. Do not free it.
 		 */
 		if (dentry->d_count) {
-			dentry_lru_del(dentry);
+			dentry->d_flags &= ~DCACHE_SHRINK_LIST;
 			spin_unlock(&dentry->d_lock);
 			continue;
 		}
@@ -857,6 +882,8 @@ relock:
 		} else {
 			list_move_tail(&dentry->d_lru, &tmp);
 			dentry->d_flags |= DCACHE_SHRINK_LIST;
+			this_cpu_dec(nr_dentry_unused);
+			sb->s_nr_dentry_unused--;
 			spin_unlock(&dentry->d_lock);
 			if (!--count)
 				break;
@@ -870,6 +897,27 @@ relock:
 	shrink_dentry_list(&tmp);
 }
 
+/*
+ * Mark all the dentries as on being the dispose list so we don't think they are
+ * still on the LRU if we try to kill them from ascending the parent chain in
+ * try_prune_one_dentry() rather than directly from the dispose list.
+ */
+static void
+shrink_dcache_list(
+	struct list_head *dispose)
+{
+	struct dentry *dentry;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(dentry, dispose, d_lru) {
+		spin_lock(&dentry->d_lock);
+		dentry->d_flags |= DCACHE_SHRINK_LIST;
+		spin_unlock(&dentry->d_lock);
+	}
+	rcu_read_unlock();
+	shrink_dentry_list(dispose);
+}
+
 /**
  * shrink_dcache_sb - shrink dcache for a superblock
  * @sb: superblock
@@ -884,8 +932,16 @@ void shrink_dcache_sb(struct super_block *sb)
 	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
 		list_splice_init(&sb->s_dentry_lru, &tmp);
+
+		/*
+		 * account for removal here so we don't need to handle it later
+		 * even though the dentry is no longer on the lru list.
+		 */
+		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
+		sb->s_nr_dentry_unused = 0;
+
 		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dentry_list(&tmp);
+		shrink_dcache_list(&tmp);
 		spin_lock(&sb->s_dentry_lru_lock);
 	}
 	spin_unlock(&sb->s_dentry_lru_lock);
@@ -914,14 +970,7 @@ static void shrink_dcache_for_umount_subtree(struct dentry *dentry)
 		do {
 			struct inode *inode;
 
-			/*
-			 * inform the fs that this dentry is about to be
-			 * unhashed and destroyed.
-			 */
-			if (dentry->d_flags & DCACHE_OP_PRUNE)
-				dentry->d_op->d_prune(dentry);
-
-			dentry_lru_del(dentry);
+			dentry_lru_prune(dentry);
 			__d_shrink(dentry);
 
 			if (dentry->d_count != 0) {
Signature · 1 line
1.8.1.4
maintainer
#5 lore raw

Re: [PATCH v6 04/31] dcache: remove dentries from LRU before putting on dispose list

Quoted 77 lines · Glauber Costa

From: Dave Chinner <[email protected]>

One of the big problems with modifying the way the dcache shrinker and LRU implementation works is that the LRU is abused in several ways. One of these is shrink_dentry_list().

Basically, we can move a dentry off the LRU onto a different list without doing any accounting changes, and then use dentry_lru_prune() to remove it from what-ever list it is now on to do the LRU accounting at that point.

This makes it -really hard- to change the LRU implementation. The use of the per-sb LRU lock serialises movement of the dentries between the different lists and the removal of them, and this is the only reason that it works. If we want to break up the dentry LRU lock and lists into, say, per-node lists, we remove the only serialisation that allows this lru list/dispose list abuse to work.

To make this work effectively, the dispose list has to be isolated from the LRU list - dentries have to be removed from the LRU *before* being placed on the dispose list. This means that the LRU accounting and isolation is completed before disposal is started, and that means we can change the LRU implementation freely in future.

This means that dentries *must* be marked with DCACHE_SHRINK_LIST when they are placed on the dispose list so that we don't think that parent dentries found in try_prune_one_dentry() are on the LRU when the are actually on the dispose list. This would result in accounting the dentry to the LRU a second time. Hence dentry_lru_prune() has to handle the DCACHE_SHRINK_LIST case differently because the dentry isn't on the LRU list.

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ]
---
 fs/dcache.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 71 insertions(+), 22 deletions(-)
fs/dcache.c +13 -1
index 795c15d..868abf9 100644
@@ -331,7 +331,6 @@ static void dentry_lru_add(struct dentry *dentry)
 static void __dentry_lru_del(struct dentry *dentry)
 {
 	list_del_init(&dentry->d_lru);
-	dentry->d_flags &= ~DCACHE_SHRINK_LIST;
 	dentry->d_sb->s_nr_dentry_unused--;
 	this_cpu_dec(nr_dentry_unused);
 }
@@ -341,6 +340,8 @@ static void __dentry_lru_del(struct dentry *dentry)
  */
 static void dentry_lru_del(struct dentry *dentry)
 {
+	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
+
 	if (!list_empty(&dentry->d_lru)) {
 		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		__dentry_lru_del(dentry);
@@ -348,15 +349,39 @@ static void dentry_lru_del(struct dentry *dentry)
 	}
 }
 
+static void dentry_lru_prune(struct dentry *dentry)
+{
+	/*
+	 * inform the fs via d_prune that this dentry is about to be
+	 * unhashed and destroyed.
+	 */
+	if (dentry->d_flags & DCACHE_OP_PRUNE)
+		dentry->d_op->d_prune(dentry);
+
+	if (list_empty(&dentry->d_lru))
+		return;

fs/dcache.c static void dentry_lru_del(struct dentry *dentry) · line 372

Quoted 10 lines
fs/dcache.c +10 -0
+
+	if ((dentry->d_flags & DCACHE_SHRINK_LIST)) {
+		list_del_init(&dentry->d_lru);
+		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+	} else {
+		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
+		__dentry_lru_del(dentry);
+		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
+	}
+}

Re-adding this function is wrong - it went away because the act of calling ->d_prune is independent of the presence of the dentry on the LRU.

Quoted 6 lines
fs/dcache.c +0 -2
@@ -479,14 +504,8 @@ relock:
 
 	if (ref)
 		dentry->d_count--;
-	/*
-	 * inform the fs via d_prune that this dentry is about to be

fs/dcache.c relock: · line 511

Quoted 10 lines
fs/dcache.c +1 -5
-	 * unhashed and destroyed.
-	 */
-	if (dentry->d_flags & DCACHE_OP_PRUNE)
-		dentry->d_op->d_prune(dentry);
 
-	dentry_lru_del(dentry);
+	dentry_lru_prune(dentry);
 	/* if it was on the hash then remove it */
 	__d_drop(dentry);
 	return d_kill(dentry, parent);

So this change is not necessary.

Quoted 6 lines
fs/dcache.c +0 -2
@@ -914,14 +970,7 @@ static void shrink_dcache_for_umount_subtree(struct dentry *dentry)
 		do {
 			struct inode *inode;
 
-			/*
-			 * inform the fs that this dentry is about to be

fs/dcache.c static void shrink_dcache_for_umount_subtree(struct dentry *dentry) · line 976

Quoted 10 lines
fs/dcache.c +1 -6
-			 * unhashed and destroyed.
-			 */
-			if (dentry->d_flags & DCACHE_OP_PRUNE)
-				dentry->d_op->d_prune(dentry);
-
-			dentry_lru_del(dentry);
+			dentry_lru_prune(dentry);
 			__d_shrink(dentry);
 
 			if (dentry->d_count != 0) {

Nor is this one.

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer
#6 lore raw

PATCH v7 4/31 dcache: remove dentries from LRU before putting on dispose list

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

One of the big problems with modifying the way the dcache shrinker and LRU implementation works is that the LRU is abused in several ways. One of these is shrink_dentry_list().

Basically, we can move a dentry off the LRU onto a different list without doing any accounting changes, and then use dentry_lru_prune() to remove it from what-ever list it is now on to do the LRU accounting at that point.

This makes it -really hard- to change the LRU implementation. The use of the per-sb LRU lock serialises movement of the dentries between the different lists and the removal of them, and this is the only reason that it works. If we want to break up the dentry LRU lock and lists into, say, per-node lists, we remove the only serialisation that allows this lru list/dispose list abuse to work.

To make this work effectively, the dispose list has to be isolated from the LRU list - dentries have to be removed from the LRU *before* being placed on the dispose list. This means that the LRU accounting and isolation is completed before disposal is started, and that means we can change the LRU implementation freely in future.

This means that dentries *must* be marked with DCACHE_SHRINK_LIST when they are placed on the dispose list so that we don't think that parent dentries found in try_prune_one_dentry() are on the LRU when the are actually on the dispose list. This would result in accounting the dentry to the LRU a second time. Hence dentry_lru_del() has to handle the DCACHE_SHRINK_LIST case differently because the dentry isn't on the LRU list.

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ] [ v7: (dchinner)
- shrink list leaks dentries when inode/parent can't be locked in dentry_kill().
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readdition of dentry_lru_prune(). ]

---
 fs/dcache.c |   91 ++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 69 insertions(+), 22 deletions(-)
fs/dcache.c +69 -22
index 795c15d..df37740 100644
@@ -315,7 +315,7 @@ static void dentry_unlink_inode(struct dentry * dentry)
 }
 
 /*
- * dentry_lru_(add|del|prune|move_tail) must be called with d_lock held.
+ * dentry_lru_(add|del|move_list) must be called with d_lock held.
  */
 static void dentry_lru_add(struct dentry *dentry)
 {
@@ -341,7 +341,8 @@ static void __dentry_lru_del(struct dentry *dentry)
  */
 static void dentry_lru_del(struct dentry *dentry)
 {
-	if (!list_empty(&dentry->d_lru)) {
+	if (!list_empty(&dentry->d_lru) &&
+	    !(dentry->d_flags & DCACHE_SHRINK_LIST)) {
 		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		__dentry_lru_del(dentry);
 		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
@@ -350,13 +351,15 @@ static void dentry_lru_del(struct dentry *dentry)
 
 static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 {
+	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
+
 	spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 	if (list_empty(&dentry->d_lru)) {
 		list_add_tail(&dentry->d_lru, list);
-		dentry->d_sb->s_nr_dentry_unused++;
-		this_cpu_inc(nr_dentry_unused);
 	} else {
 		list_move_tail(&dentry->d_lru, list);
+		dentry->d_sb->s_nr_dentry_unused--;
+		this_cpu_dec(nr_dentry_unused);
 	}
 	spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 }
@@ -454,7 +457,8 @@ EXPORT_SYMBOL(d_drop);
  * If ref is non-zero, then decrement the refcount too.
  * Returns dentry requiring refcount drop, or NULL if we're done.
  */
-static inline struct dentry *dentry_kill(struct dentry *dentry, int ref)
+static inline struct dentry *
+dentry_kill(struct dentry *dentry, int ref, int unlock_on_failure)
 	__releases(dentry->d_lock)
 {
 	struct inode *inode;
@@ -463,8 +467,10 @@ static inline struct dentry *dentry_kill(struct dentry *dentry, int ref)
 	inode = dentry->d_inode;
 	if (inode && !spin_trylock(&inode->i_lock)) {
 relock:
-		spin_unlock(&dentry->d_lock);
-		cpu_relax();
+		if (unlock_on_failure) {
+			spin_unlock(&dentry->d_lock);
+			cpu_relax();
+		}
 		return dentry; /* try again with same dentry */
 	}
 	if (IS_ROOT(dentry))
@@ -551,7 +557,7 @@ repeat:
 	return;
 
 kill_it:
-	dentry = dentry_kill(dentry, 1);
+	dentry = dentry_kill(dentry, 1, 1);
 	if (dentry)
 		goto repeat;
 }
@@ -750,12 +756,12 @@ EXPORT_SYMBOL(d_prune_aliases);
  *
  * This may fail if locks cannot be acquired no problem, just try again.
  */
-static void try_prune_one_dentry(struct dentry *dentry)
+static struct dentry * try_prune_one_dentry(struct dentry *dentry)
 	__releases(dentry->d_lock)
 {
 	struct dentry *parent;
 
-	parent = dentry_kill(dentry, 0);
+	parent = dentry_kill(dentry, 0, 0);
 	/*
 	 * If dentry_kill returns NULL, we have nothing more to do.
 	 * if it returns the same dentry, trylocks failed. In either
@@ -767,9 +773,9 @@ static void try_prune_one_dentry(struct dentry *dentry)
 	 * fragmentation.
 	 */
 	if (!parent)
-		return;
+		return NULL;
 	if (parent == dentry)
-		return;
+		return dentry;
 
 	/* Prune ancestors. */
 	dentry = parent;
@@ -778,10 +784,11 @@ static void try_prune_one_dentry(struct dentry *dentry)
 		if (dentry->d_count > 1) {
 			dentry->d_count--;
 			spin_unlock(&dentry->d_lock);
-			return;
+			return NULL;
 		}
-		dentry = dentry_kill(dentry, 1);
+		dentry = dentry_kill(dentry, 1, 1);
 	}
+	return NULL;
 }
 
 static void shrink_dentry_list(struct list_head *list)
@@ -800,21 +807,31 @@ static void shrink_dentry_list(struct list_head *list)
 		}
 
 		/*
+		 * The dispose list is isolated and dentries are not accounted
+		 * to the LRU here, so we can simply remove it from the list
+		 * here regardless of whether it is referenced or not.
+		 */
+		list_del_init(&dentry->d_lru);
+		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+
+		/*
 		 * We found an inuse dentry which was not removed from
-		 * the LRU because of laziness during lookup.  Do not free
-		 * it - just keep it off the LRU list.
+		 * the LRU because of laziness during lookup. Do not free it.
 		 */
 		if (dentry->d_count) {
-			dentry_lru_del(dentry);
 			spin_unlock(&dentry->d_lock);
 			continue;
 		}
-
 		rcu_read_unlock();
 
-		try_prune_one_dentry(dentry);
+		dentry = try_prune_one_dentry(dentry);
 
 		rcu_read_lock();
+		if (dentry) {
+			dentry->d_flags |= DCACHE_SHRINK_LIST;
+			list_add(&dentry->d_lru, list);
+			spin_unlock(&dentry->d_lock);
+		}
 	}
 	rcu_read_unlock();
 }
@@ -855,8 +872,10 @@ relock:
 			list_move(&dentry->d_lru, &referenced);
 			spin_unlock(&dentry->d_lock);
 		} else {
-			list_move_tail(&dentry->d_lru, &tmp);
+			list_move(&dentry->d_lru, &tmp);
 			dentry->d_flags |= DCACHE_SHRINK_LIST;
+			this_cpu_dec(nr_dentry_unused);
+			sb->s_nr_dentry_unused--;
 			spin_unlock(&dentry->d_lock);
 			if (!--count)
 				break;
@@ -870,6 +889,27 @@ relock:
 	shrink_dentry_list(&tmp);
 }
 
+/*
+ * Mark all the dentries as on being the dispose list so we don't think they are
+ * still on the LRU if we try to kill them from ascending the parent chain in
+ * try_prune_one_dentry() rather than directly from the dispose list.
+ */
+static void
+shrink_dcache_list(
+	struct list_head *dispose)
+{
+	struct dentry *dentry;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(dentry, dispose, d_lru) {
+		spin_lock(&dentry->d_lock);
+		dentry->d_flags |= DCACHE_SHRINK_LIST;
+		spin_unlock(&dentry->d_lock);
+	}
+	rcu_read_unlock();
+	shrink_dentry_list(dispose);
+}
+
 /**
  * shrink_dcache_sb - shrink dcache for a superblock
  * @sb: superblock
@@ -883,9 +923,16 @@ void shrink_dcache_sb(struct super_block *sb)
 
 	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
+		/*
+		 * account for removal here so we don't need to handle it later
+		 * even though the dentry is no longer on the lru list.
+		 */
 		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dentry_list(&tmp);
+		list_splice_init(&sb->s_dentry_lru, &tmp);
+		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
+		sb->s_nr_dentry_unused = 0;
+
+		shrink_dcache_list(&tmp);
 		spin_lock(&sb->s_dentry_lru_lock);
 	}
 	spin_unlock(&sb->s_dentry_lru_lock);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#7 lore raw

Re: [PATCH v7 04/31] dcache: remove dentries from LRU before putting on dispose list

Quoted 33 lines · Dave Chinner

From: Dave Chinner <[email protected]>

One of the big problems with modifying the way the dcache shrinker and LRU implementation works is that the LRU is abused in several ways. One of these is shrink_dentry_list().

Basically, we can move a dentry off the LRU onto a different list without doing any accounting changes, and then use dentry_lru_prune() to remove it from what-ever list it is now on to do the LRU accounting at that point.

This makes it -really hard- to change the LRU implementation. The use of the per-sb LRU lock serialises movement of the dentries between the different lists and the removal of them, and this is the only reason that it works. If we want to break up the dentry LRU lock and lists into, say, per-node lists, we remove the only serialisation that allows this lru list/dispose list abuse to work.

To make this work effectively, the dispose list has to be isolated from the LRU list - dentries have to be removed from the LRU *before* being placed on the dispose list. This means that the LRU accounting and isolation is completed before disposal is started, and that means we can change the LRU implementation freely in future.

This means that dentries *must* be marked with DCACHE_SHRINK_LIST when they are placed on the dispose list so that we don't think that parent dentries found in try_prune_one_dentry() are on the LRU when the are actually on the dispose list. This would result in accounting the dentry to the LRU a second time. Hence dentry_lru_del() has to handle the DCACHE_SHRINK_LIST case differently because the dentry isn't on the LRU list.

Quoted 5 lines

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ] [ v7: (dchinner)
- shrink list leaks dentries when inode/parent can't be locked in dentry_kill().
- fix the scope of the sb locking inside shrink_dcache_sb()

<sigh>

I need find a dealer that sells better crack.

Quoted 7 lines
fs/dcache.c +2 -1
@@ -883,9 +923,16 @@ void shrink_dcache_sb(struct super_block *sb)
 
 	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
+		/*
+		 * account for removal here so we don't need to handle it later

fs/dcache.c void shrink_dcache_sb(struct super_block *sb) · line 936

Quoted 10 lines
fs/dcache.c +7 -1
+		 * even though the dentry is no longer on the lru list.
+		 */
 		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dentry_list(&tmp);
+		list_splice_init(&sb->s_dentry_lru, &tmp);
+		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
+		sb->s_nr_dentry_unused = 0;
+
+		shrink_dcache_list(&tmp);
 		spin_lock(&sb->s_dentry_lru_lock);

This is now completely wrong. It should end up like this:

	while (!list_empty(&sb->s_dentry_lru)) {
		/*
		 * account for removal here so we don't need to handle it later
		 * even though the dentry is no longer on the lru list.
		 */
		list_splice_init(&sb->s_dentry_lru, &tmp);
		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
		sb->s_nr_dentry_unused = 0;
		spin_unlock(&sb->s_dentry_lru_lock);

		shrink_dcache_list(&tmp);

		spin_lock(&sb->s_dentry_lru_lock);
	}
	spin_unlock(&sb->s_dentry_lru_lock);

-Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw

Re: [PATCH v7 04/31] dcache: remove dentries from LRU before putting on dispose list

In reply to Dave Chinner
Quoted 6 lines · Dave Chinner

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ] [ v7: (dchinner)
- shrink list leaks dentries when inode/parent can't be locked in dentry_kill().
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readdition of dentry_lru_prune(). ]

Dave,

dentry_lru_prune was removed because it would only prune the dentry if it was in the LRU list, and it has to be always pruned (61572bb1).

You don't reintroduce dentry_lru_prune here, so the two locations which prune dentries read as follows:

        if (dentry->d_flags & DCACHE_OP_PRUNE)
                dentry->d_op->d_prune(dentry);

        dentry_lru_del(dentry);

I believe this is wrong. My old version would do:

+static void dentry_lru_prune(struct dentry *dentry) +{
+ /*
+ * inform the fs via d_prune that this dentry is about to be
+ * unhashed and destroyed.
+ */
+ if (dentry->d_flags & DCACHE_OP_PRUNE)
+ dentry->d_op->d_prune(dentry);
+
+ if (list_empty(&dentry->d_lru))
+ return;
+
+ if ((dentry->d_flags & DCACHE_SHRINK_LIST)) {
+ list_del_init(&dentry->d_lru);
+ dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+ } else {
+ spin_lock(&dentry->d_sb->s_dentry_lru_lock);
+ __dentry_lru_del(dentry);
+ spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
+ }
+}

Which is SHRINK_LIST aware. The code as it reads today after your patch will only be correct if it is totally impossible for a dentry to be in the shrink list before we reach both sites that call d_op->d_prune. They
are: dentry_kill and shrink_dcache_for_umount_subtree. So it seems to me that we do really need to reintroduce dentry_lru_prune or just patch both call sites with shrik-list aware code.

Comments ?

maintainer
#9 lore raw

Re: [PATCH v7 04/31] dcache: remove dentries from LRU before putting on dispose list

Quoted 46 lines · Glauber Costa
Quoted 6 lines · Dave Chinner

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ] [ v7: (dchinner)
- shrink list leaks dentries when inode/parent can't be locked in dentry_kill().
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readdition of dentry_lru_prune(). ]

Dave,

dentry_lru_prune was removed because it would only prune the dentry if it was in the LRU list, and it has to be always pruned (61572bb1).

You don't reintroduce dentry_lru_prune here, so the two locations which prune dentries read as follows:

        if (dentry->d_flags & DCACHE_OP_PRUNE)
                dentry->d_op->d_prune(dentry);

        dentry_lru_del(dentry);

I believe this is wrong. My old version would do:

(quoted diff) +21 -0
+static void dentry_lru_prune(struct dentry *dentry)
+{
+	/*
+	 * inform the fs via d_prune that this dentry is about to be
+	 * unhashed and destroyed.
+	 */
+	if (dentry->d_flags & DCACHE_OP_PRUNE)
+		dentry->d_op->d_prune(dentry);
+
+	if (list_empty(&dentry->d_lru))
+		return;
+
+	if ((dentry->d_flags & DCACHE_SHRINK_LIST)) {
+		list_del_init(&dentry->d_lru);
+		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+	} else {
+		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
+		__dentry_lru_del(dentry);
+		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
+	}
+}
Quoted 1 line

Which is SHRINK_LIST aware.

Right. Did I mention I was seeing a panic on a later patch? That's because this patch removed the DCACHE_SHRINK_LIST check. So, the DCACHE_SHRINK_LIST needs to move into dentry_lru_del(), and dentry_lru_prune() can still go away. That's what my current version of this patch does - I just didn't post it last night because it was still running tests when I went to bed....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw

PATCH v6 5/31 mm: new shrinker API

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

The current shrinker callout API uses an a single shrinker call for multiple functions. To determine the function, a special magical value is passed in a parameter to change the behaviour. This complicates the implementation and return value specification for the different behaviours.

Separate the two different behaviours into separate operations, one to return a count of freeable objects in the cache, and another to scan a certain number of objects in the cache for freeing. In defining these new operations, ensure the return values and resultant behaviours are clearly defined and documented.

Modify shrink_slab() to use the new API and implement the callouts for all the existing shrinkers.

---
 include/linux/shrinker.h | 36 ++++++++++++++++++++++++----------
 mm/vmscan.c              | 50 +++++++++++++++++++++++++++++++-----------------
 2 files changed, 58 insertions(+), 28 deletions(-)
include/linux/shrinker.h +26 -10
index ac6b8ee..c277b4e 100644
@@ -4,31 +4,47 @@
 /*
  * This struct is used to pass information from page reclaim to the shrinkers.
  * We consolidate the values for easier extention later.
+ *
+ * The 'gfpmask' refers to the allocation we are currently trying to
+ * fulfil.
+ *
+ * Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
+ * querying the cache size, so a fastpath for that case is appropriate.
  */
 struct shrink_control {
 	gfp_t gfp_mask;
 
 	/* How many slab objects shrinker() should scan and try to reclaim */
-	unsigned long nr_to_scan;
+	long nr_to_scan;
 };
 
 /*
  * A callback you can register to apply pressure to ageable caches.
  *
- * 'sc' is passed shrink_control which includes a count 'nr_to_scan'
- * and a 'gfpmask'.  It should look through the least-recently-used
- * 'nr_to_scan' entries and attempt to free them up.  It should return
- * the number of objects which remain in the cache.  If it returns -1, it means
- * it cannot do any scanning at this time (eg. there is a risk of deadlock).
+ * @shrink() should look through the least-recently-used 'nr_to_scan' entries
+ * and attempt to free them up.  It should return the number of objects which
+ * remain in the cache.  If it returns -1, it means it cannot do any scanning at
+ * this time (eg. there is a risk of deadlock).
  *
- * The 'gfpmask' refers to the allocation we are currently trying to
- * fulfil.
+ * @count_objects should return the number of freeable items in the cache. If
+ * there are no objects to free or the number of freeable items cannot be
+ * determined, it should return 0. No deadlock checks should be done during the
+ * count callback - the shrinker relies on aggregating scan counts that couldn't
+ * be executed due to potential deadlocks to be run at a later call when the
+ * deadlock condition is no longer pending.
  *
- * Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
- * querying the cache size, so a fastpath for that case is appropriate.
+ * @scan_objects will only be called if @count_objects returned a positive
+ * value for the number of freeable objects. The callout should scan the cache
+ * and attempt to free items from the cache. It should then return the number of
+ * objects freed during the scan, or -1 if progress cannot be made due to
+ * potential deadlocks. If -1 is returned, then no further attempts to call the
+ * @scan_objects will be made from the current reclaim context.
  */
 struct shrinker {
 	int (*shrink)(struct shrinker *, struct shrink_control *sc);
+	long (*count_objects)(struct shrinker *, struct shrink_control *sc);
+	long (*scan_objects)(struct shrinker *, struct shrink_control *sc);
+
 	int seeks;	/* seeks to recreate an obj */
 	long batch;	/* reclaim batch size, 0 = default */
 
mm/vmscan.c +32 -18

104 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#11 lore raw

PATCH v6 6/31 shrinker: convert superblock shrinkers to new API

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert superblock shrinker to use the new count/scan API, and propagate the API changes through to the filesystem callouts. The filesystem callouts already use a count/scan API, so it's just changing counters to longs to match the VM API.

This requires the dentry and inode shrinker callouts to be converted to the count/scan API. This is mainly a mechanical change.

[ glommer: use mult_frac for fractional proportions, build fixes ]
---
 fs/dcache.c         | 10 +++++---
 fs/inode.c          |  7 +++--
 fs/internal.h       |  2 ++
 fs/super.c          | 74 ++++++++++++++++++++++++++++++++---------------------
 fs/xfs/xfs_icache.c |  4 +--
 fs/xfs/xfs_icache.h |  2 +-
 fs/xfs/xfs_super.c  |  8 +++---
 include/linux/fs.h  |  8 ++----
 8 files changed, 67 insertions(+), 48 deletions(-)
fs/dcache.c +6 -4
index 868abf9..dc0069c 100644
@@ -856,11 +856,12 @@ static void shrink_dentry_list(struct list_head *list)
  * This function may fail to free any resources if all the dentries are in
  * use.
  */
-void prune_dcache_sb(struct super_block *sb, int count)
+long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan)
 {
 	struct dentry *dentry;
 	LIST_HEAD(referenced);
 	LIST_HEAD(tmp);
+	long freed = 0;
 
 relock:
 	spin_lock(&sb->s_dentry_lru_lock);
@@ -885,7 +886,8 @@ relock:
 			this_cpu_dec(nr_dentry_unused);
 			sb->s_nr_dentry_unused--;
 			spin_unlock(&dentry->d_lock);
-			if (!--count)
+			freed++;
+			if (!--nr_to_scan)
 				break;
 		}
 		cond_resched_lock(&sb->s_dentry_lru_lock);
@@ -895,6 +897,7 @@ relock:
 	spin_unlock(&sb->s_dentry_lru_lock);
 
 	shrink_dentry_list(&tmp);
+	return freed;
 }
 
 /*
@@ -1275,9 +1278,8 @@ rename_retry:
 void shrink_dcache_parent(struct dentry * parent)
 {
 	LIST_HEAD(dispose);
-	int found;
 
-	while ((found = select_parent(parent, &dispose)) != 0) {
+	while (select_parent(parent, &dispose)) {
 		shrink_dentry_list(&dispose);
 		cond_resched();
 	}
fs/inode.c +5 -2
index 00d5fc3..730aea7 100644
@@ -704,10 +704,11 @@ static int can_unuse(struct inode *inode)
  * LRU does not have strict ordering. Hence we don't want to reclaim inodes
  * with this flag set because they are the inodes that are out of order.
  */
-void prune_icache_sb(struct super_block *sb, int nr_to_scan)
+long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
 {
 	LIST_HEAD(freeable);
-	int nr_scanned;
+	long nr_scanned;
+	long freed = 0;
 	unsigned long reap = 0;
 
 	spin_lock(&sb->s_inode_lru_lock);
@@ -777,6 +778,7 @@ void prune_icache_sb(struct super_block *sb, int nr_to_scan)
 		list_move(&inode->i_lru, &freeable);
 		sb->s_nr_inodes_unused--;
 		this_cpu_dec(nr_unused);
+		freed++;
 	}
 	if (current_is_kswapd())
 		__count_vm_events(KSWAPD_INODESTEAL, reap);
@@ -787,6 +789,7 @@ void prune_icache_sb(struct super_block *sb, int nr_to_scan)
 		current->reclaim_state->reclaimed_slab += reap;
 
 	dispose_list(&freeable);
+	return freed;
 }
 
 static void __wait_on_freeing_inode(struct inode *inode);
fs/internal.h +2 -0
index eaa75f7..bb7b6e4 100644
@@ -110,6 +110,7 @@ extern int open_check_o_direct(struct file *f);
  * inode.c
  */
 extern spinlock_t inode_sb_list_lock;
+extern long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan);
 extern void inode_add_lru(struct inode *inode);
 
 /*
@@ -125,6 +126,7 @@ extern int invalidate_inodes(struct super_block *, bool);
  * dcache.c
  */
 extern struct dentry *__d_alloc(struct super_block *, const struct qstr *);
+extern long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan);
 
 /*
  * read_write.c
fs/super.c +45 -29

103 lines not on this page · raw message

fs/xfs/xfs_icache.c +2 -2
index 96e344e..b35c311 100644
@@ -1164,7 +1164,7 @@ xfs_reclaim_inodes(
  * them to be cleaned, which we hope will not be very long due to the
  * background walker having already kicked the IO off on those dirty inodes.
  */
-void
+long
 xfs_reclaim_inodes_nr(
 	struct xfs_mount	*mp,
 	int			nr_to_scan)
@@ -1173,7 +1173,7 @@ xfs_reclaim_inodes_nr(
 	xfs_reclaim_work_queue(mp);
 	xfs_ail_push_all(mp->m_ail);
 
-	xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan);
+	return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan);
 }
 
 /*
fs/xfs/xfs_icache.h +1 -1
index e0f138c..2d6d2d3 100644
@@ -31,7 +31,7 @@ void xfs_reclaim_worker(struct work_struct *work);
 
 int xfs_reclaim_inodes(struct xfs_mount *mp, int mode);
 int xfs_reclaim_inodes_count(struct xfs_mount *mp);
-void xfs_reclaim_inodes_nr(struct xfs_mount *mp, int nr_to_scan);
+long xfs_reclaim_inodes_nr(struct xfs_mount *mp, int nr_to_scan);
 
 void xfs_inode_set_reclaim_tag(struct xfs_inode *ip);
 
fs/xfs/xfs_super.c +4 -4
index ea341ce..1ff991b 100644
@@ -1523,19 +1523,19 @@ xfs_fs_mount(
 	return mount_bdev(fs_type, flags, dev_name, data, xfs_fs_fill_super);
 }
 
-static int
+static long
 xfs_fs_nr_cached_objects(
 	struct super_block	*sb)
 {
 	return xfs_reclaim_inodes_count(XFS_M(sb));
 }
 
-static void
+static long
 xfs_fs_free_cached_objects(
 	struct super_block	*sb,
-	int			nr_to_scan)
+	long			nr_to_scan)
 {
-	xfs_reclaim_inodes_nr(XFS_M(sb), nr_to_scan);
+	return xfs_reclaim_inodes_nr(XFS_M(sb), nr_to_scan);
 }
 
 static const struct super_operations xfs_super_operations = {
include/linux/fs.h +2 -6
index 67e1040..f5eabf9 100644
@@ -1326,10 +1326,6 @@ struct super_block {
 	int s_readonly_remount;
 };
 
-/* superblock cache pruning functions */
-extern void prune_icache_sb(struct super_block *sb, int nr_to_scan);
-extern void prune_dcache_sb(struct super_block *sb, int nr_to_scan);
-
 extern struct timespec current_fs_time(struct super_block *sb);
 
 /*
@@ -1616,8 +1612,8 @@ struct super_operations {
 	ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t);
 #endif
 	int (*bdev_try_to_free_page)(struct super_block*, struct page*, gfp_t);
-	int (*nr_cached_objects)(struct super_block *);
-	void (*free_cached_objects)(struct super_block *, int);
+	long (*nr_cached_objects)(struct super_block *);
+	long (*free_cached_objects)(struct super_block *, long);
 };
 
 /*
Signature · 1 line
1.8.1.4
#12 lore raw

PATCH v6 7/31 list: add a new LRU list type

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Several subsystems use the same construct for LRU lists - a list head, a spin lock and and item count. They also use exactly the same code for adding and removing items from the LRU. Create a generic type for these LRU lists.

This is the beginning of generic, node aware LRUs for shrinkers to work with.

[ glommer: enum defined constants for lru. Suggested by gthelen,
don't relock over retry ]

---
 include/linux/list_lru.h |  46 ++++++++++++++++++
 lib/Makefile             |   2 +-
 lib/list_lru.c           | 121 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 168 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/list_lru.h
 create mode 100644 lib/list_lru.c
include/linux/list_lru.h new file +46 -0
new file mode 100644
index 0000000..4f82a57
@@ -0,0 +1,46 @@
+/*
+ * Copyright (c) 2010-2012 Red Hat, Inc. All rights reserved.
+ * Author: David Chinner
+ *
+ * Generic LRU infrastructure
+ */
+#ifndef _LRU_LIST_H
+#define _LRU_LIST_H
+
+#include <linux/list.h>
+
+enum lru_status {
+	LRU_REMOVED,		/* item removed from list */
+	LRU_ROTATE,		/* item referenced, give another pass */
+	LRU_SKIP,		/* item cannot be locked, skip */
+	LRU_RETRY,		/* item not freeable. May drop the lock
+				   internally, but has to return locked. */
+};
+
+struct list_lru {
+	spinlock_t		lock;
+	struct list_head	list;
+	long			nr_items;
+};
+
+int list_lru_init(struct list_lru *lru);
+int list_lru_add(struct list_lru *lru, struct list_head *item);
+int list_lru_del(struct list_lru *lru, struct list_head *item);
+
+static inline unsigned long list_lru_count(struct list_lru *lru)
+{
+	return lru->nr_items;
+}
+
+typedef enum lru_status
+(*list_lru_walk_cb)(struct list_head *item, spinlock_t *lock, void *cb_arg);
+
+typedef void (*list_lru_dispose_cb)(struct list_head *dispose_list);
+
+unsigned long list_lru_walk(struct list_lru *lru, list_lru_walk_cb isolate,
+		   void *cb_arg, unsigned long nr_to_walk);
+
+unsigned long
+list_lru_dispose_all(struct list_lru *lru, list_lru_dispose_cb dispose);
+
+#endif /* _LRU_LIST_H */
lib/Makefile +1 -1
index 3eacb2c..dbeb899 100644
@@ -13,7 +13,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 sha1.o md5.o irq_regs.o reciprocal_div.o argv_split.o \
 	 proportions.o flex_proportions.o prio_heap.o ratelimit.o show_mem.o \
 	 is_single_threaded.o plist.o decompress.o kobject_uevent.o \
-	 earlycpio.o percpu-refcount.o
+	 earlycpio.o percpu-refcount.o list_lru.o
 
 obj-$(CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS) += usercopy.o
 lib-$(CONFIG_MMU) += ioremap.o
lib/list_lru.c new file +121 -0

121 lines not on this page · raw message

Signature · 1 line
1.8.1.4
Acked-by on v6 7/31
#13 lore raw

Re: [PATCH v6 07/31] list: add a new LRU list type

Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Several subsystems use the same construct for LRU lists - a list head, a spin lock and and item count. They also use exactly the same code for adding and removing items from the LRU. Create a generic type for these LRU lists.

Quoted 8 lines

This is the beginning of generic, node aware LRUs for shrinkers to work with.

[ glommer: enum defined constants for lru. Suggested by gthelen,
don't relock over retry ]

Signature · 8 lines
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#14 lore raw

PATCH v6 8/31 inode: convert inode lru list to generic lru list code.

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

[ glommer: adapted for new LRU return codes ]
---
 fs/inode.c         | 175 +++++++++++++++++++++--------------------------------
 fs/super.c         |  12 ++--
 include/linux/fs.h |   6 +-
 3 files changed, 77 insertions(+), 116 deletions(-)
fs/inode.c +70 -105
index 730aea7..ff66f49 100644
@@ -17,6 +17,7 @@
 #include <linux/prefetch.h>
 #include <linux/buffer_head.h> /* for inode_has_buffers */
 #include <linux/ratelimit.h>
+#include <linux/list_lru.h>
 #include "internal.h"
 
 /*
@@ -24,7 +25,7 @@
  *
  * inode->i_lock protects:
  *   inode->i_state, inode->i_hash, __iget()
- * inode->i_sb->s_inode_lru_lock protects:
+ * Inode LRU list locks protect:
  *   inode->i_sb->s_inode_lru, inode->i_lru
  * inode_sb_list_lock protects:
  *   sb->s_inodes, inode->i_sb_list
@@ -37,7 +38,7 @@
  *
  * inode_sb_list_lock
  *   inode->i_lock
- *     inode->i_sb->s_inode_lru_lock
+ *     Inode LRU list locks
  *
  * bdi->wb.list_lock
  *   inode->i_lock
@@ -399,13 +400,8 @@ EXPORT_SYMBOL(ihold);
 
 static void inode_lru_list_add(struct inode *inode)
 {
-	spin_lock(&inode->i_sb->s_inode_lru_lock);
-	if (list_empty(&inode->i_lru)) {
-		list_add(&inode->i_lru, &inode->i_sb->s_inode_lru);
-		inode->i_sb->s_nr_inodes_unused++;
+	if (list_lru_add(&inode->i_sb->s_inode_lru, &inode->i_lru))
 		this_cpu_inc(nr_unused);
-	}
-	spin_unlock(&inode->i_sb->s_inode_lru_lock);
 }
 
 /*
@@ -423,13 +419,9 @@ void inode_add_lru(struct inode *inode)
 
 static void inode_lru_list_del(struct inode *inode)
 {
-	spin_lock(&inode->i_sb->s_inode_lru_lock);
-	if (!list_empty(&inode->i_lru)) {
-		list_del_init(&inode->i_lru);
-		inode->i_sb->s_nr_inodes_unused--;
+
+	if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
 		this_cpu_dec(nr_unused);
-	}
-	spin_unlock(&inode->i_sb->s_inode_lru_lock);
 }
 
 /**
@@ -673,24 +665,8 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty)
 	return busy;
 }
 
-static int can_unuse(struct inode *inode)
-{
-	if (inode->i_state & ~I_REFERENCED)
-		return 0;
-	if (inode_has_buffers(inode))
-		return 0;
-	if (atomic_read(&inode->i_count))
-		return 0;
-	if (inode->i_data.nrpages)
-		return 0;
-	return 1;
-}
-
 /*
- * Walk the superblock inode LRU for freeable inodes and attempt to free them.
- * This is called from the superblock shrinker function with a number of inodes
- * to trim from the LRU. Inodes to be freed are moved to a temporary list and
- * then are freed outside inode_lock by dispose_list().
+ * Isolate the inode from the LRU in preparation for freeing it.
  *
  * Any inodes which are pinned purely because of attached pagecache have their
  * pagecache removed.  If the inode has metadata buffers attached to
@@ -704,90 +680,79 @@ static int can_unuse(struct inode *inode)
  * LRU does not have strict ordering. Hence we don't want to reclaim inodes
  * with this flag set because they are the inodes that are out of order.
  */
-long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
+static enum lru_status
+inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
 {
-	LIST_HEAD(freeable);
-	long nr_scanned;
-	long freed = 0;
-	unsigned long reap = 0;
+	struct list_head *freeable = arg;
+	struct inode	*inode = container_of(item, struct inode, i_lru);
 
-	spin_lock(&sb->s_inode_lru_lock);
-	for (nr_scanned = nr_to_scan; nr_scanned >= 0; nr_scanned--) {
-		struct inode *inode;
+	/*
+	 * we are inverting the lru lock/inode->i_lock here, so use a trylock.
+	 * If we fail to get the lock, just skip it.
+	 */
+	if (!spin_trylock(&inode->i_lock))
+		return LRU_SKIP;
 
-		if (list_empty(&sb->s_inode_lru))
-			break;
+	/*
+	 * Referenced or dirty inodes are still in use. Give them another pass
+	 * through the LRU as we canot reclaim them now.
+	 */
+	if (atomic_read(&inode->i_count) ||
+	    (inode->i_state & ~I_REFERENCED)) {
+		list_del_init(&inode->i_lru);
+		spin_unlock(&inode->i_lock);
+		this_cpu_dec(nr_unused);
+		return LRU_REMOVED;
+	}
 
-		inode = list_entry(sb->s_inode_lru.prev, struct inode, i_lru);
+	/* recently referenced inodes get one more pass */
+	if (inode->i_state & I_REFERENCED) {
+		inode->i_state &= ~I_REFERENCED;
+		spin_unlock(&inode->i_lock);
+		return LRU_ROTATE;
+	}
 
-		/*
-		 * we are inverting the sb->s_inode_lru_lock/inode->i_lock here,
-		 * so use a trylock. If we fail to get the lock, just move the
-		 * inode to the back of the list so we don't spin on it.
-		 */
-		if (!spin_trylock(&inode->i_lock)) {
-			list_move(&inode->i_lru, &sb->s_inode_lru);
-			continue;
+	if (inode_has_buffers(inode) || inode->i_data.nrpages) {
+		__iget(inode);
+		spin_unlock(&inode->i_lock);
+		spin_unlock(lru_lock);
+		if (remove_inode_buffers(inode)) {
+			unsigned long reap;
+			reap = invalidate_mapping_pages(&inode->i_data, 0, -1);
+			if (current_is_kswapd())
+				__count_vm_events(KSWAPD_INODESTEAL, reap);
+			else
+				__count_vm_events(PGINODESTEAL, reap);
+			if (current->reclaim_state)
+				current->reclaim_state->reclaimed_slab += reap;
 		}
+		iput(inode);
+		spin_lock(lru_lock);
+		return LRU_RETRY;
+	}
 
-		/*
-		 * Referenced or dirty inodes are still in use. Give them
-		 * another pass through the LRU as we canot reclaim them now.
-		 */
-		if (atomic_read(&inode->i_count) ||
-		    (inode->i_state & ~I_REFERENCED)) {
-			list_del_init(&inode->i_lru);
-			spin_unlock(&inode->i_lock);
-			sb->s_nr_inodes_unused--;
-			this_cpu_dec(nr_unused);
-			continue;
-		}
+	WARN_ON(inode->i_state & I_NEW);
+	inode->i_state |= I_FREEING;
+	spin_unlock(&inode->i_lock);
 
-		/* recently referenced inodes get one more pass */
-		if (inode->i_state & I_REFERENCED) {
-			inode->i_state &= ~I_REFERENCED;
-			list_move(&inode->i_lru, &sb->s_inode_lru);
-			spin_unlock(&inode->i_lock);
-			continue;
-		}
-		if (inode_has_buffers(inode) || inode->i_data.nrpages) {
-			__iget(inode);
-			spin_unlock(&inode->i_lock);
-			spin_unlock(&sb->s_inode_lru_lock);
-			if (remove_inode_buffers(inode))
-				reap += invalidate_mapping_pages(&inode->i_data,
-								0, -1);
-			iput(inode);
-			spin_lock(&sb->s_inode_lru_lock);
-
-			if (inode != list_entry(sb->s_inode_lru.next,
-						struct inode, i_lru))
-				continue;	/* wrong inode or list_empty */
-			/* avoid lock inversions with trylock */
-			if (!spin_trylock(&inode->i_lock))
-				continue;
-			if (!can_unuse(inode)) {
-				spin_unlock(&inode->i_lock);
-				continue;
-			}
-		}
-		WARN_ON(inode->i_state & I_NEW);
-		inode->i_state |= I_FREEING;
-		spin_unlock(&inode->i_lock);
+	list_move(&inode->i_lru, freeable);
+	this_cpu_dec(nr_unused);
+	return LRU_REMOVED;
+}
 
-		list_move(&inode->i_lru, &freeable);
-		sb->s_nr_inodes_unused--;
-		this_cpu_dec(nr_unused);
-		freed++;
-	}
-	if (current_is_kswapd())
-		__count_vm_events(KSWAPD_INODESTEAL, reap);
-	else
-		__count_vm_events(PGINODESTEAL, reap);
-	spin_unlock(&sb->s_inode_lru_lock);
-	if (current->reclaim_state)
-		current->reclaim_state->reclaimed_slab += reap;
+/*
+ * Walk the superblock inode LRU for freeable inodes and attempt to free them.
+ * This is called from the superblock shrinker function with a number of inodes
+ * to trim from the LRU. Inodes to be freed are moved to a temporary list and
+ * then are freed outside inode_lock by dispose_list().
+ */
+long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
+{
+	LIST_HEAD(freeable);
+	long freed;
 
+	freed = list_lru_walk(&sb->s_inode_lru, inode_lru_isolate,
+						&freeable, nr_to_scan);
 	dispose_list(&freeable);
 	return freed;
 }
fs/super.c +5 -7
index 9d2f2e9..9049110 100644
@@ -77,14 +77,13 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 	if (sb->s_op && sb->s_op->nr_cached_objects)
 		fs_objects = sb->s_op->nr_cached_objects(sb);
 
-	total_objects = sb->s_nr_dentry_unused +
-			sb->s_nr_inodes_unused + fs_objects + 1;
+	inodes = list_lru_count(&sb->s_inode_lru);
+	total_objects = sb->s_nr_dentry_unused + inodes + fs_objects + 1;
 
 	/* proportion the scan between the caches */
 	dentries = mult_frac(sc->nr_to_scan, sb->s_nr_dentry_unused,
 								total_objects);
-	inodes = mult_frac(sc->nr_to_scan, sb->s_nr_inodes_unused,
-								total_objects);
+	inodes = mult_frac(sc->nr_to_scan, inodes, total_objects);
 
 	/*
 	 * prune the dcache first as the icache is pinned by it, then
@@ -117,7 +116,7 @@ static long super_cache_count(struct shrinker *shrink, struct shrink_control *sc
 		total_objects = sb->s_op->nr_cached_objects(sb);
 
 	total_objects += sb->s_nr_dentry_unused;
-	total_objects += sb->s_nr_inodes_unused;
+	total_objects += list_lru_count(&sb->s_inode_lru);
 
 	total_objects = vfs_pressure_ratio(total_objects);
 	drop_super(sb);
@@ -198,8 +197,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)
 		INIT_LIST_HEAD(&s->s_inodes);
 		INIT_LIST_HEAD(&s->s_dentry_lru);
 		spin_lock_init(&s->s_dentry_lru_lock);
-		INIT_LIST_HEAD(&s->s_inode_lru);
-		spin_lock_init(&s->s_inode_lru_lock);
+		list_lru_init(&s->s_inode_lru);
 		INIT_LIST_HEAD(&s->s_mounts);
 		init_rwsem(&s->s_umount);
 		lockdep_set_class(&s->s_umount, &type->s_umount_key);
include/linux/fs.h +2 -4
index f5eabf9..4442613 100644
@@ -10,6 +10,7 @@
 #include <linux/stat.h>
 #include <linux/cache.h>
 #include <linux/list.h>
+#include <linux/list_lru.h>
 #include <linux/radix-tree.h>
 #include <linux/rbtree.h>
 #include <linux/init.h>
@@ -1269,10 +1270,7 @@ struct super_block {
 	struct list_head	s_dentry_lru;	/* unused dentry lru */
 	int			s_nr_dentry_unused;	/* # of dentry on lru */
 
-	/* s_inode_lru_lock protects s_inode_lru and s_nr_inodes_unused */
-	spinlock_t		s_inode_lru_lock ____cacheline_aligned_in_smp;
-	struct list_head	s_inode_lru;		/* unused inode lru */
-	int			s_nr_inodes_unused;	/* # of inodes on lru */
+	struct list_lru		s_inode_lru ____cacheline_aligned_in_smp;
 
 	struct block_device	*s_bdev;
 	struct backing_dev_info *s_bdi;
Signature · 1 line
1.8.1.4
#15 lore raw

PATCH v6 9/31 dcache: convert to use new lru list infrastructure

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

[ glommer: don't reintroduce double decrement of nr_unused_dentries,
adapted for new LRU return codes ]

---
 fs/dcache.c        | 180 +++++++++++++++++++++++------------------------------
 fs/super.c         |  11 ++--
 include/linux/fs.h |  15 +++--
 3 files changed, 91 insertions(+), 115 deletions(-)
fs/dcache.c +79 -101
index dc0069c..ba14661 100644
@@ -37,6 +37,7 @@
 #include <linux/rculist_bl.h>
 #include <linux/prefetch.h>
 #include <linux/ratelimit.h>
+#include <linux/list_lru.h>
 #include "internal.h"
 #include "mount.h"
 
@@ -319,20 +320,8 @@ static void dentry_unlink_inode(struct dentry * dentry)
  */
 static void dentry_lru_add(struct dentry *dentry)
 {
-	if (list_empty(&dentry->d_lru)) {
-		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
-		list_add(&dentry->d_lru, &dentry->d_sb->s_dentry_lru);
-		dentry->d_sb->s_nr_dentry_unused++;
+	if (list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru))
 		this_cpu_inc(nr_dentry_unused);
-		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
-	}
-}
-
-static void __dentry_lru_del(struct dentry *dentry)
-{
-	list_del_init(&dentry->d_lru);
-	dentry->d_sb->s_nr_dentry_unused--;
-	this_cpu_dec(nr_dentry_unused);
 }
 
 /*
@@ -342,11 +331,8 @@ static void dentry_lru_del(struct dentry *dentry)
 {
 	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
 
-	if (!list_empty(&dentry->d_lru)) {
-		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
-		__dentry_lru_del(dentry);
-		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
-	}
+	if (list_lru_del(&dentry->d_sb->s_dentry_lru, &dentry->d_lru))
+		this_cpu_dec(nr_dentry_unused);
 }
 
 static void dentry_lru_prune(struct dentry *dentry)
@@ -358,32 +344,13 @@ static void dentry_lru_prune(struct dentry *dentry)
 	if (dentry->d_flags & DCACHE_OP_PRUNE)
 		dentry->d_op->d_prune(dentry);
 
-	if (list_empty(&dentry->d_lru))
-		return;
-
-	if ((dentry->d_flags & DCACHE_SHRINK_LIST)) {
+	if (!list_empty(&dentry->d_lru) &&
+	    (dentry->d_flags & DCACHE_SHRINK_LIST))
 		list_del_init(&dentry->d_lru);
-		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
-	} else {
-		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
-		__dentry_lru_del(dentry);
-		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
-	}
-}
-
-static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
-{
-	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
-
-	spin_lock(&dentry->d_sb->s_dentry_lru_lock);
-	if (list_empty(&dentry->d_lru)) {
-		list_add_tail(&dentry->d_lru, list);
-	} else {
-		list_move_tail(&dentry->d_lru, list);
-		dentry->d_sb->s_nr_dentry_unused--;
+	else if (list_lru_del(&dentry->d_sb->s_dentry_lru, &dentry->d_lru))
 		this_cpu_dec(nr_dentry_unused);
-	}
-	spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
+
+	dentry->d_flags &= ~DCACHE_SHRINK_LIST;
 }
 
 /**
@@ -844,12 +811,72 @@ static void shrink_dentry_list(struct list_head *list)
 	rcu_read_unlock();
 }
 
+static enum lru_status
+dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
+{
+	struct list_head *freeable = arg;
+	struct dentry	*dentry = container_of(item, struct dentry, d_lru);
+
+
+	/*
+	 * we are inverting the lru lock/dentry->d_lock here,
+	 * so use a trylock. If we fail to get the lock, just skip
+	 * it
+	 */
+	if (!spin_trylock(&dentry->d_lock))
+		return LRU_SKIP;
+
+	/*
+	 * Referenced dentries are still in use. If they have active
+	 * counts, just remove them from the LRU. Otherwise give them
+	 * another pass through the LRU.
+	 */
+	if (dentry->d_count) {
+		list_del_init(&dentry->d_lru);
+		spin_unlock(&dentry->d_lock);
+		return LRU_REMOVED;
+	}
+
+	if (dentry->d_flags & DCACHE_REFERENCED) {
+		dentry->d_flags &= ~DCACHE_REFERENCED;
+		spin_unlock(&dentry->d_lock);
+
+		/*
+		 * The list move itself will be made by the common LRU code. At
+		 * this point, we've dropped the dentry->d_lock but keep the
+		 * lru lock. This is safe to do, since every list movement is
+		 * protected by the lru lock even if both locks are held.
+		 *
+		 * This is guaranteed by the fact that all LRU management
+		 * functions are intermediated by the LRU API calls like
+		 * list_lru_add and list_lru_del. List movement in this file
+		 * only ever occur through this functions or through callbacks
+		 * like this one, that are called from the LRU API.
+		 *
+		 * The only exceptions to this are functions like
+		 * shrink_dentry_list, and code that first checks for the
+		 * DCACHE_SHRINK_LIST flag.  Those are guaranteed to be
+		 * operating only with stack provided lists after they are
+		 * properly isolated from the main list.  It is thus, always a
+		 * local access.
+		 */
+		return LRU_ROTATE;
+	}
+
+	dentry->d_flags |= DCACHE_SHRINK_LIST;
+	list_move_tail(&dentry->d_lru, freeable);
+	this_cpu_dec(nr_dentry_unused);
+	spin_unlock(&dentry->d_lock);
+
+	return LRU_REMOVED;
+}
+
 /**
  * prune_dcache_sb - shrink the dcache
  * @sb: superblock
- * @count: number of entries to try to free
+ * @nr_to_scan : number of entries to try to free
  *
- * Attempt to shrink the superblock dcache LRU by @count entries. This is
+ * Attempt to shrink the superblock dcache LRU by @nr_to_scan entries. This is
  * done when we need more memory an called from the superblock shrinker
  * function.
  *
@@ -858,45 +885,12 @@ static void shrink_dentry_list(struct list_head *list)
  */
 long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan)
 {
-	struct dentry *dentry;
-	LIST_HEAD(referenced);
-	LIST_HEAD(tmp);
-	long freed = 0;
-
-relock:
-	spin_lock(&sb->s_dentry_lru_lock);
-	while (!list_empty(&sb->s_dentry_lru)) {
-		dentry = list_entry(sb->s_dentry_lru.prev,
-				struct dentry, d_lru);
-		BUG_ON(dentry->d_sb != sb);
-
-		if (!spin_trylock(&dentry->d_lock)) {
-			spin_unlock(&sb->s_dentry_lru_lock);
-			cpu_relax();
-			goto relock;
-		}
-
-		if (dentry->d_flags & DCACHE_REFERENCED) {
-			dentry->d_flags &= ~DCACHE_REFERENCED;
-			list_move(&dentry->d_lru, &referenced);
-			spin_unlock(&dentry->d_lock);
-		} else {
-			list_move_tail(&dentry->d_lru, &tmp);
-			dentry->d_flags |= DCACHE_SHRINK_LIST;
-			this_cpu_dec(nr_dentry_unused);
-			sb->s_nr_dentry_unused--;
-			spin_unlock(&dentry->d_lock);
-			freed++;
-			if (!--nr_to_scan)
-				break;
-		}
-		cond_resched_lock(&sb->s_dentry_lru_lock);
-	}
-	if (!list_empty(&referenced))
-		list_splice(&referenced, &sb->s_dentry_lru);
-	spin_unlock(&sb->s_dentry_lru_lock);
+	LIST_HEAD(dispose);
+	long freed;
 
-	shrink_dentry_list(&tmp);
+	freed = list_lru_walk(&sb->s_dentry_lru, dentry_lru_isolate,
+			      &dispose, nr_to_scan);
+	shrink_dentry_list(&dispose);
 	return freed;
 }
 
@@ -930,24 +924,7 @@ shrink_dcache_list(
  */
 void shrink_dcache_sb(struct super_block *sb)
 {
-	LIST_HEAD(tmp);
-
-	spin_lock(&sb->s_dentry_lru_lock);
-	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
-
-		/*
-		 * account for removal here so we don't need to handle it later
-		 * even though the dentry is no longer on the lru list.
-		 */
-		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
-		sb->s_nr_dentry_unused = 0;
-
-		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dcache_list(&tmp);
-		spin_lock(&sb->s_dentry_lru_lock);
-	}
-	spin_unlock(&sb->s_dentry_lru_lock);
+	list_lru_dispose_all(&sb->s_dentry_lru, shrink_dcache_list);
 }
 EXPORT_SYMBOL(shrink_dcache_sb);
 
@@ -1213,7 +1190,8 @@ resume:
 		if (dentry->d_count) {
 			dentry_lru_del(dentry);
 		} else if (!(dentry->d_flags & DCACHE_SHRINK_LIST)) {
-			dentry_lru_move_list(dentry, dispose);
+			dentry_lru_del(dentry);
+			list_add_tail(&dentry->d_lru, dispose);
 			dentry->d_flags |= DCACHE_SHRINK_LIST;
 			found++;
 		}
fs/super.c +5 -6
index 9049110..66f5cde 100644
@@ -78,11 +78,11 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 		fs_objects = sb->s_op->nr_cached_objects(sb);
 
 	inodes = list_lru_count(&sb->s_inode_lru);
-	total_objects = sb->s_nr_dentry_unused + inodes + fs_objects + 1;
+	dentries = list_lru_count(&sb->s_dentry_lru);
+	total_objects = dentries + inodes + fs_objects + 1;
 
 	/* proportion the scan between the caches */
-	dentries = mult_frac(sc->nr_to_scan, sb->s_nr_dentry_unused,
-								total_objects);
+	dentries = mult_frac(sc->nr_to_scan, dentries, total_objects);
 	inodes = mult_frac(sc->nr_to_scan, inodes, total_objects);
 
 	/*
@@ -115,7 +115,7 @@ static long super_cache_count(struct shrinker *shrink, struct shrink_control *sc
 	if (sb->s_op && sb->s_op->nr_cached_objects)
 		total_objects = sb->s_op->nr_cached_objects(sb);
 
-	total_objects += sb->s_nr_dentry_unused;
+	total_objects += list_lru_count(&sb->s_dentry_lru);
 	total_objects += list_lru_count(&sb->s_inode_lru);
 
 	total_objects = vfs_pressure_ratio(total_objects);
@@ -195,8 +195,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)
 		INIT_HLIST_NODE(&s->s_instances);
 		INIT_HLIST_BL_HEAD(&s->s_anon);
 		INIT_LIST_HEAD(&s->s_inodes);
-		INIT_LIST_HEAD(&s->s_dentry_lru);
-		spin_lock_init(&s->s_dentry_lru_lock);
+		list_lru_init(&s->s_dentry_lru);
 		list_lru_init(&s->s_inode_lru);
 		INIT_LIST_HEAD(&s->s_mounts);
 		init_rwsem(&s->s_umount);
include/linux/fs.h +7 -8
index 4442613..02b3612 100644
@@ -1264,14 +1264,6 @@ struct super_block {
 	struct list_head	s_files;
 #endif
 	struct list_head	s_mounts;	/* list of mounts; _not_ for fs use */
-
-	/* s_dentry_lru_lock protects s_dentry_lru and s_nr_dentry_unused */
-	spinlock_t		s_dentry_lru_lock ____cacheline_aligned_in_smp;
-	struct list_head	s_dentry_lru;	/* unused dentry lru */
-	int			s_nr_dentry_unused;	/* # of dentry on lru */
-
-	struct list_lru		s_inode_lru ____cacheline_aligned_in_smp;
-
 	struct block_device	*s_bdev;
 	struct backing_dev_info *s_bdi;
 	struct mtd_info		*s_mtd;
@@ -1322,6 +1314,13 @@ struct super_block {
 
 	/* Being remounted read-only */
 	int s_readonly_remount;
+
+	/*
+	 * Keep the lru lists last in the structure so they always sit on their
+	 * own individual cachelines.
+	 */
+	struct list_lru		s_dentry_lru ____cacheline_aligned_in_smp;
+	struct list_lru		s_inode_lru ____cacheline_aligned_in_smp;
 };
 
 extern struct timespec current_fs_time(struct super_block *sb);
Signature · 1 line
1.8.1.4
maintainer
#16 lore raw

Re: [PATCH v6 09/31] dcache: convert to use new lru list infrastructure

Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

[ glommer: don't reintroduce double decrement of nr_unused_dentries,
adapted for new LRU return codes ]

---

I'm seeing a panic on startup in d_kill() with an invalid d_child list entry with this patch. I haven't got to the bottom of it yet.

.....

Quoted 13 lines

void shrink_dcache_sb(struct super_block *sb)
{

fs/dcache.c +0 -11
-	LIST_HEAD(tmp);
-
-	spin_lock(&sb->s_dentry_lru_lock);
-	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
-
-		/*
-		 * account for removal here so we don't need to handle it later
-		 * even though the dentry is no longer on the lru list.
-		 */
-		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);

fs/dcache.c shrink_dcache_list( · line 929

Quoted 10 lines
fs/dcache.c +1 -7
-		sb->s_nr_dentry_unused = 0;
-
-		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dcache_list(&tmp);
-		spin_lock(&sb->s_dentry_lru_lock);
-	}
-	spin_unlock(&sb->s_dentry_lru_lock);
+	list_lru_dispose_all(&sb->s_dentry_lru, shrink_dcache_list);
 }
 EXPORT_SYMBOL(shrink_dcache_sb);

And here comes the fun part. This doesn't account for the dentries that are freed from the superblock here.

So, it needs to be something like:

void shrink_dcache_sb(struct super_block *sb)

{
	unsigned long disposed;

	disposed = list_lru_dispose_all(&sb->s_dentry_lru,
					shrink_dcache_list);

	this_cpu_sub(nr_dentry_unused, disposed);
}

But, therein lies a problem. nr_dentry_unused is a 32 bit counter, and we can return a 64 bit value here. So that means we have to bump nr_dentry_unused to a long, not an int for these per-cpu counters to work.

And then there's the problem that the sum of these counters only uses an int. Which means if we get large numbers of negative values on different CPU from unmounts, the summation will end up overflowing and it'll all suck.

So, Glauber, what do you reckon? I've never likes this stupid hand-rolled per-cpu counter stuff, and it's causing issues. Should we just convert them to generic per-cpu counters because they are 64bit clean and just handle out-of-range sums in the /proc update code?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#17 lore raw

Re: [PATCH v6 09/31] dcache: convert to use new lru list infrastructure

Quoted 63 lines · Dave Chinner
Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

[ glommer: don't reintroduce double decrement of nr_unused_dentries,
adapted for new LRU return codes ]

---

I'm seeing a panic on startup in d_kill() with an invalid d_child list entry with this patch. I haven't got to the bottom of it yet.

.....

Quoted 13 lines

void shrink_dcache_sb(struct super_block *sb)
{

fs/dcache.c +0 -11
-	LIST_HEAD(tmp);
-
-	spin_lock(&sb->s_dentry_lru_lock);
-	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
-
-		/*
-		 * account for removal here so we don't need to handle it later
-		 * even though the dentry is no longer on the lru list.
-		 */
-		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);

fs/dcache.c shrink_dcache_list( · line 929

Quoted 10 lines
fs/dcache.c +1 -7
-		sb->s_nr_dentry_unused = 0;
-
-		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dcache_list(&tmp);
-		spin_lock(&sb->s_dentry_lru_lock);
-	}
-	spin_unlock(&sb->s_dentry_lru_lock);
+	list_lru_dispose_all(&sb->s_dentry_lru, shrink_dcache_list);
 }
 EXPORT_SYMBOL(shrink_dcache_sb);

And here comes the fun part. This doesn't account for the dentries that are freed from the superblock here.

So, it needs to be something like:

void shrink_dcache_sb(struct super_block *sb)

{
	unsigned long disposed;

	disposed = list_lru_dispose_all(&sb->s_dentry_lru,
					shrink_dcache_list);

	this_cpu_sub(nr_dentry_unused, disposed);
}

But, therein lies a problem. nr_dentry_unused is a 32 bit counter, and we can return a 64 bit value here. So that means we have to bump nr_dentry_unused to a long, not an int for these per-cpu counters to work.

And then there's the problem that the sum of these counters only uses an int. Which means if we get large numbers of negative values on different CPU from unmounts, the summation will end up overflowing and it'll all suck.

Quoted 6 lines

So, Glauber, what do you reckon? I've never likes this stupid hand-rolled per-cpu counter stuff, and it's causing issues. Should we just convert them to generic per-cpu counters because they are 64bit clean and just handle out-of-range sums in the /proc update code?

I am not against it, per se.
But assuming that whoever did it did it for a reason, can't we just propagate 64 bit quantities everywhere? Moving to long instead of int should already give us some room, and is actually a sensible thing to
do: There is no reason whatsoever to keep using ints around like this.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#18 lore raw

Re: [PATCH v6 09/31] dcache: convert to use new lru list infrastructure

In reply to Dave Chinner
Quoted 12 lines · Dave Chinner
Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

[ glommer: don't reintroduce double decrement of nr_unused_dentries,
adapted for new LRU return codes ]

---

I'm seeing a panic on startup in d_kill() with an invalid d_child list entry with this patch. I haven't got to the bottom of it yet.

My wild guess is that your patch does not prune correctly, as I described in my last message.

Quoted 51 lines

.....

Quoted 13 lines

void shrink_dcache_sb(struct super_block *sb)
{

(quoted diff) +0 -11
-	LIST_HEAD(tmp);
-
-	spin_lock(&sb->s_dentry_lru_lock);
-	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
-
-		/*
-		 * account for removal here so we don't need to handle it later
-		 * even though the dentry is no longer on the lru list.
-		 */
-		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);

Review

Quoted 10 lines
(quoted diff) +1 -7
-		sb->s_nr_dentry_unused = 0;
-
-		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dcache_list(&tmp);
-		spin_lock(&sb->s_dentry_lru_lock);
-	}
-	spin_unlock(&sb->s_dentry_lru_lock);
+	list_lru_dispose_all(&sb->s_dentry_lru, shrink_dcache_list);
 }
 EXPORT_SYMBOL(shrink_dcache_sb);

And here comes the fun part. This doesn't account for the dentries that are freed from the superblock here.

So, it needs to be something like:

void shrink_dcache_sb(struct super_block *sb)

{
	unsigned long disposed;

	disposed = list_lru_dispose_all(&sb->s_dentry_lru,
					shrink_dcache_list);

	this_cpu_sub(nr_dentry_unused, disposed);
}

But, therein lies a problem. nr_dentry_unused is a 32 bit counter, and we can return a 64 bit value here. So that means we have to bump nr_dentry_unused to a long, not an int for these per-cpu counters to work.

And then there's the problem that the sum of these counters only uses an int. Which means if we get large numbers of negative values on different CPU from unmounts, the summation will end up overflowing and it'll all suck.

Quoted 6 lines

So, Glauber, what do you reckon? I've never likes this stupid hand-rolled per-cpu counter stuff, and it's causing issues. Should we just convert them to generic per-cpu counters because they are 64bit clean and just handle out-of-range sums in the /proc update code?

One option would be to add the following patch to the beginning of the series.

fs/dcache.c +5 -5
index 3a3adc4..a8be4c9 100644
@@ -116,12 +116,12 @@ struct dentry_stat_t dentry_stat = {
 	.age_limit = 45,
 };
 
-static DEFINE_PER_CPU(unsigned int, nr_dentry);
-static DEFINE_PER_CPU(unsigned int, nr_dentry_unused);
+static DEFINE_PER_CPU(long, nr_dentry);
+static DEFINE_PER_CPU(long, nr_dentry_unused);
 
 #if defined(CONFIG_SYSCTL) && defined(CONFIG_PROC_FS)
 /* scan possible cpus instead of online and avoid worrying about CPU hotplug. */
-static int get_nr_dentry(void)
+static long get_nr_dentry(void)
 {
 	int i;
 	int sum = 0;
@@ -130,7 +130,7 @@ static int get_nr_dentry(void)
 	return sum < 0 ? 0 : sum;
 }
 
-static int get_nr_dentry_unused(void)
+static long get_nr_dentry_unused(void)
 {
 	int i;
 	int sum = 0;
@@ -144,7 +144,7 @@ int proc_nr_dentry(ctl_table *table, int write, void __user *buffer,
 {
 	dentry_stat.nr_dentry = get_nr_dentry();
 	dentry_stat.nr_unused = get_nr_dentry_unused();
-	return proc_dointvec(table, write, buffer, lenp, ppos);
+	return proc_doulongvec_minmax(table, write, buffer, lenp, ppos);
 }
 #endif
 
include/linux/dcache.h +5 -5
index 4d24a12..bd08285 100644
@@ -54,11 +54,11 @@ struct qstr {
 #define hashlen_len(hashlen)  ((u32)((hashlen) >> 32))
 
 struct dentry_stat_t {
-	int nr_dentry;
-	int nr_unused;
-	int age_limit;          /* age in seconds */
-	int want_pages;         /* pages requested by system */
-	int dummy[2];
+	long nr_dentry;
+	long nr_unused;
+	long age_limit;          /* age in seconds */
+	long want_pages;         /* pages requested by system */
+	long dummy[2];
 };
 extern struct dentry_stat_t dentry_stat;
 
include/linux/fs.h +2 -2
index 67e1040..e875f60 100644
@@ -1267,12 +1267,12 @@ struct super_block {
 	/* s_dentry_lru_lock protects s_dentry_lru and s_nr_dentry_unused */
 	spinlock_t		s_dentry_lru_lock ____cacheline_aligned_in_smp;
 	struct list_head	s_dentry_lru;	/* unused dentry lru */
-	int			s_nr_dentry_unused;	/* # of dentry on lru */
+	long			s_nr_dentry_unused;	/* # of dentry on lru */
 
 	/* s_inode_lru_lock protects s_inode_lru and s_nr_inodes_unused */
 	spinlock_t		s_inode_lru_lock ____cacheline_aligned_in_smp;
 	struct list_head	s_inode_lru;		/* unused inode lru */
-	int			s_nr_inodes_unused;	/* # of inodes on lru */
+	long			s_nr_inodes_unused;	/* # of inodes on lru */
 
 	struct block_device	*s_bdev;
 	struct backing_dev_info *s_bdi;
kernel/sysctl.c +1 -1
index 9edcf45..0dc51c0 100644
@@ -1493,7 +1493,7 @@ static struct ctl_table fs_table[] = {
 	{
 		.procname	= "dentry-state",
 		.data		= &dentry_stat,
-		.maxlen		= 6*sizeof(int),
+		.maxlen		= 6*sizeof(long),
 		.mode		= 0444,
 		.proc_handler	= proc_nr_dentry,
 	},
#19 lore raw

PATCH v6 10/31 list_lru: per-node list infrastructure

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Now that we have an LRU list API, we can start to enhance the implementation. This splits the single LRU list into per-node lists and locks to enhance scalability. Items are placed on lists according to the node the memory belongs to. To make scanning the lists efficient, also track whether the per-node lists have entries in them in a active nodemask.

Note:
We use a fixed-size array for the node LRU, this struct can be very big if MAX_NUMNODES is big. If this becomes a problem this is fixable by turning this into a pointer and dynamically allocating this to nr_node_ids. This quantity is firwmare-provided, and still would provide room for all nodes at the cost of a pointer lookup and an extra allocation. Because that allocation will most likely come from a different slab cache than the main structure holding this structure, we may very well fail.

[ glommer: fixed warnings, added note about node lru ]
---
 include/linux/list_lru.h |  24 +++++--
 lib/list_lru.c           | 161 +++++++++++++++++++++++++++++++++++------------
 2 files changed, 139 insertions(+), 46 deletions(-)
include/linux/list_lru.h +18 -6
index 4f82a57..668f1f1 100644
@@ -8,6 +8,7 @@
 #define _LRU_LIST_H
 
 #include <linux/list.h>
+#include <linux/nodemask.h>
 
 enum lru_status {
 	LRU_REMOVED,		/* item removed from list */
@@ -17,20 +18,31 @@ enum lru_status {
 				   internally, but has to return locked. */
 };
 
-struct list_lru {
+struct list_lru_node {
 	spinlock_t		lock;
 	struct list_head	list;
 	long			nr_items;
+} ____cacheline_aligned_in_smp;
+
+struct list_lru {
+	/*
+	 * Because we use a fixed-size array, this struct can be very big if
+	 * MAX_NUMNODES is big. If this becomes a problem this is fixable by
+	 * turning this into a pointer and dynamically allocating this to
+	 * nr_node_ids. This quantity is firwmare-provided, and still would
+	 * provide room for all nodes at the cost of a pointer lookup and an
+	 * extra allocation. Because that allocation will most likely come from
+	 * a different slab cache than the main structure holding this
+	 * structure, we may very well fail.
+	 */
+	struct list_lru_node	node[MAX_NUMNODES];
+	nodemask_t		active_nodes;
 };
 
 int list_lru_init(struct list_lru *lru);
 int list_lru_add(struct list_lru *lru, struct list_head *item);
 int list_lru_del(struct list_lru *lru, struct list_head *item);
-
-static inline unsigned long list_lru_count(struct list_lru *lru)
-{
-	return lru->nr_items;
-}
+unsigned long list_lru_count(struct list_lru *lru);
 
 typedef enum lru_status
 (*list_lru_walk_cb)(struct list_head *item, spinlock_t *lock, void *cb_arg);
lib/list_lru.c +121 -40

228 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#20 lore raw

PATCH v6 11/31 shrinker: add node awareness

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Pass the node of the current zone being reclaimed to shrink_slab(), allowing the shrinker control nodemask to be set appropriately for node aware shrinkers.

---
 drivers/staging/android/ashmem.c |  3 +++
 fs/drop_caches.c                 |  1 +
 include/linux/shrinker.h         |  3 +++
 mm/memory-failure.c              |  2 ++
 mm/vmscan.c                      | 12 +++++++++---
 5 files changed, 18 insertions(+), 3 deletions(-)
drivers/staging/android/ashmem.c +3 -0
index e681bdd..3240d34 100644
@@ -692,6 +692,9 @@ static long ashmem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 				.gfp_mask = GFP_KERNEL,
 				.nr_to_scan = 0,
 			};
+
+			nodes_setall(sc.nodes_to_scan);
+
 			ret = ashmem_shrink(&ashmem_shrinker, &sc);
 			sc.nr_to_scan = ret;
 			ashmem_shrink(&ashmem_shrinker, &sc);
fs/drop_caches.c +1 -0
index f23d2a7..c3f44e7 100644
@@ -44,6 +44,7 @@ static void drop_slab(void)
 		.gfp_mask = GFP_KERNEL,
 	};
 
+	nodes_setall(shrink.nodes_to_scan);
 	do {
 		nr_objects = shrink_slab(&shrink, 1000, 1000);
 	} while (nr_objects > 10);
include/linux/shrinker.h +3 -0
index c277b4e..98be3ab 100644
@@ -16,6 +16,9 @@ struct shrink_control {
 
 	/* How many slab objects shrinker() should scan and try to reclaim */
 	long nr_to_scan;
+
+	/* shrink from these nodes */
+	nodemask_t nodes_to_scan;
 };
 
 /*
mm/memory-failure.c +2 -0
index ceb0c7f..86788ff 100644
@@ -248,10 +248,12 @@ void shake_page(struct page *p, int access)
 	 */
 	if (access) {
 		int nr;
+		int nid = page_to_nid(p);
 		do {
 			struct shrink_control shrink = {
 				.gfp_mask = GFP_KERNEL,
 			};
+			node_set(nid, shrink.nodes_to_scan);
 
 			nr = shrink_slab(&shrink, 1000, 1000);
 			if (page_count(p) == 1)
mm/vmscan.c +9 -3
index fd4639c..35a6a9b 100644
@@ -2198,15 +2198,20 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 		 */
 		if (global_reclaim(sc)) {
 			unsigned long lru_pages = 0;
+
+			nodes_clear(shrink->nodes_to_scan);
 			for_each_zone_zonelist(zone, z, zonelist,
 					gfp_zone(sc->gfp_mask)) {
 				if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
 					continue;
 
 				lru_pages += zone_reclaimable_pages(zone);
+				node_set(zone_to_nid(zone),
+					 shrink->nodes_to_scan);
 			}
 
 			shrink_slab(shrink, sc->nr_scanned, lru_pages);
+
 			if (reclaim_state) {
 				sc->nr_reclaimed += reclaim_state->reclaimed_slab;
 				reclaim_state->reclaimed_slab = 0;
@@ -2782,6 +2787,8 @@ loop_again:
 				shrink_zone(zone, &sc);
 
 				reclaim_state->reclaimed_slab = 0;
+				nodes_clear(shrink.nodes_to_scan);
+				node_set(zone_to_nid(zone), shrink.nodes_to_scan);
 				nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages);
 				sc.nr_reclaimed += reclaim_state->reclaimed_slab;
 
@@ -3367,10 +3374,9 @@ static int __zone_reclaim(struct zone *zone, gfp_t gfp_mask, unsigned int order)
 		 * number of slab pages and shake the slab until it is reduced
 		 * by the same nr_pages that we used for reclaiming unmapped
 		 * pages.
-		 *
-		 * Note that shrink_slab will free memory on all zones and may
-		 * take a long time.
 		 */
+		nodes_clear(shrink.nodes_to_scan);
+		node_set(zone_to_nid(zone), shrink.nodes_to_scan);
 		for (;;) {
 			unsigned long lru_pages = zone_reclaimable_pages(zone);
 
Signature · 1 line
1.8.1.4
#21 lore raw

PATCH v6 12/31 fs: convert inode and dentry shrinking to be node aware

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Now that the shrinker is passing a nodemask in the scan control structure, we can pass this to the the generic LRU list code to isolate reclaim to the lists on matching nodes.

This requires a small amount of refactoring of the LRU list API, which might be best split out into a separate patch.

---
 fs/dcache.c              |  8 +++++---
 fs/inode.c               |  7 ++++---
 fs/internal.h            |  6 ++++--
 fs/super.c               | 22 +++++++++++++---------
 fs/xfs/xfs_super.c       |  6 ++++--
 include/linux/fs.h       |  4 ++--
 include/linux/list_lru.h | 22 +++++++++++++++++++---
 lib/list_lru.c           | 15 ++++++++-------
 8 files changed, 59 insertions(+), 31 deletions(-)
fs/dcache.c +5 -3
index ba14661..20ab9c0 100644
@@ -875,6 +875,7 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * prune_dcache_sb - shrink the dcache
  * @sb: superblock
  * @nr_to_scan : number of entries to try to free
+ * @nodes_to_walk: which nodes to scan for freeable entities
  *
  * Attempt to shrink the superblock dcache LRU by @nr_to_scan entries. This is
  * done when we need more memory an called from the superblock shrinker
@@ -883,13 +884,14 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * This function may fail to free any resources if all the dentries are in
  * use.
  */
-long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan)
+long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
+		     nodemask_t *nodes_to_walk)
 {
 	LIST_HEAD(dispose);
 	long freed;
 
-	freed = list_lru_walk(&sb->s_dentry_lru, dentry_lru_isolate,
-			      &dispose, nr_to_scan);
+	freed = list_lru_walk_nodemask(&sb->s_dentry_lru, dentry_lru_isolate,
+				       &dispose, nr_to_scan, nodes_to_walk);
 	shrink_dentry_list(&dispose);
 	return freed;
 }
fs/inode.c +4 -3
index ff66f49..3cf4cb0 100644
@@ -746,13 +746,14 @@ inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * to trim from the LRU. Inodes to be freed are moved to a temporary list and
  * then are freed outside inode_lock by dispose_list().
  */
-long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan)
+long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan,
+		     nodemask_t *nodes_to_walk)
 {
 	LIST_HEAD(freeable);
 	long freed;
 
-	freed = list_lru_walk(&sb->s_inode_lru, inode_lru_isolate,
-						&freeable, nr_to_scan);
+	freed = list_lru_walk_nodemask(&sb->s_inode_lru, inode_lru_isolate,
+				       &freeable, nr_to_scan, nodes_to_walk);
 	dispose_list(&freeable);
 	return freed;
 }
fs/internal.h +4 -2
index bb7b6e4..91900f2 100644
@@ -110,7 +110,8 @@ extern int open_check_o_direct(struct file *f);
  * inode.c
  */
 extern spinlock_t inode_sb_list_lock;
-extern long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan);
+extern long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan,
+			    nodemask_t *nodes_to_scan);
 extern void inode_add_lru(struct inode *inode);
 
 /*
@@ -126,7 +127,8 @@ extern int invalidate_inodes(struct super_block *, bool);
  * dcache.c
  */
 extern struct dentry *__d_alloc(struct super_block *, const struct qstr *);
-extern long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan);
+extern long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
+			    nodemask_t *nodes_to_scan);
 
 /*
  * read_write.c
fs/super.c +13 -9
index 66f5cde..5c7b879 100644
@@ -75,10 +75,10 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 		return -1;
 
 	if (sb->s_op && sb->s_op->nr_cached_objects)
-		fs_objects = sb->s_op->nr_cached_objects(sb);
+		fs_objects = sb->s_op->nr_cached_objects(sb, &sc->nodes_to_scan);
 
-	inodes = list_lru_count(&sb->s_inode_lru);
-	dentries = list_lru_count(&sb->s_dentry_lru);
+	inodes = list_lru_count_nodemask(&sb->s_inode_lru, &sc->nodes_to_scan);
+	dentries = list_lru_count_nodemask(&sb->s_dentry_lru, &sc->nodes_to_scan);
 	total_objects = dentries + inodes + fs_objects + 1;
 
 	/* proportion the scan between the caches */
@@ -89,13 +89,14 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 	 * prune the dcache first as the icache is pinned by it, then
 	 * prune the icache, followed by the filesystem specific caches
 	 */
-	freed = prune_dcache_sb(sb, dentries);
-	freed += prune_icache_sb(sb, inodes);
+	freed = prune_dcache_sb(sb, dentries, &sc->nodes_to_scan);
+	freed += prune_icache_sb(sb, inodes, &sc->nodes_to_scan);
 
 	if (fs_objects) {
 		fs_objects = mult_frac(sc->nr_to_scan, fs_objects,
 								total_objects);
-		freed += sb->s_op->free_cached_objects(sb, fs_objects);
+		freed += sb->s_op->free_cached_objects(sb, fs_objects,
+						       &sc->nodes_to_scan);
 	}
 
 	drop_super(sb);
@@ -113,10 +114,13 @@ static long super_cache_count(struct shrinker *shrink, struct shrink_control *sc
 		return -1;
 
 	if (sb->s_op && sb->s_op->nr_cached_objects)
-		total_objects = sb->s_op->nr_cached_objects(sb);
+		total_objects = sb->s_op->nr_cached_objects(sb,
+						 &sc->nodes_to_scan);
 
-	total_objects += list_lru_count(&sb->s_dentry_lru);
-	total_objects += list_lru_count(&sb->s_inode_lru);
+	total_objects += list_lru_count_nodemask(&sb->s_dentry_lru,
+						 &sc->nodes_to_scan);
+	total_objects += list_lru_count_nodemask(&sb->s_inode_lru,
+						 &sc->nodes_to_scan);
 
 	total_objects = vfs_pressure_ratio(total_objects);
 	drop_super(sb);
fs/xfs/xfs_super.c +4 -2
index 1ff991b..7fa60214 100644
@@ -1525,7 +1525,8 @@ xfs_fs_mount(
 
 static long
 xfs_fs_nr_cached_objects(
-	struct super_block	*sb)
+	struct super_block	*sb,
+	nodemask_t		*nodes_to_count)
 {
 	return xfs_reclaim_inodes_count(XFS_M(sb));
 }
@@ -1533,7 +1534,8 @@ xfs_fs_nr_cached_objects(
 static long
 xfs_fs_free_cached_objects(
 	struct super_block	*sb,
-	long			nr_to_scan)
+	long			nr_to_scan,
+	nodemask_t		*nodes_to_scan)
 {
 	return xfs_reclaim_inodes_nr(XFS_M(sb), nr_to_scan);
 }
include/linux/fs.h +2 -2
index 02b3612..d548724 100644
@@ -1609,8 +1609,8 @@ struct super_operations {
 	ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t);
 #endif
 	int (*bdev_try_to_free_page)(struct super_block*, struct page*, gfp_t);
-	long (*nr_cached_objects)(struct super_block *);
-	long (*free_cached_objects)(struct super_block *, long);
+	long (*nr_cached_objects)(struct super_block *, nodemask_t *);
+	long (*free_cached_objects)(struct super_block *, long, nodemask_t *);
 };
 
 /*
include/linux/list_lru.h +19 -3
index 668f1f1..f5e3c15 100644
@@ -42,15 +42,31 @@ struct list_lru {
 int list_lru_init(struct list_lru *lru);
 int list_lru_add(struct list_lru *lru, struct list_head *item);
 int list_lru_del(struct list_lru *lru, struct list_head *item);
-unsigned long list_lru_count(struct list_lru *lru);
+unsigned long
+list_lru_count_nodemask(struct list_lru *lru, nodemask_t *nodes_to_count);
+
+static inline unsigned long list_lru_count(struct list_lru *lru)
+{
+	return list_lru_count_nodemask(lru, &lru->active_nodes);
+}
+
 
 typedef enum lru_status
 (*list_lru_walk_cb)(struct list_head *item, spinlock_t *lock, void *cb_arg);
 
 typedef void (*list_lru_dispose_cb)(struct list_head *dispose_list);
 
-unsigned long list_lru_walk(struct list_lru *lru, list_lru_walk_cb isolate,
-		   void *cb_arg, unsigned long nr_to_walk);
+unsigned long
+list_lru_walk_nodemask(struct list_lru *lru, list_lru_walk_cb isolate,
+	   void *cb_arg, unsigned long nr_to_walk, nodemask_t *nodes_to_walk);
+
+static inline unsigned long
+list_lru_walk(struct list_lru *lru, list_lru_walk_cb isolate,
+	      void *cb_arg, unsigned long nr_to_walk)
+{
+	return list_lru_walk_nodemask(lru, isolate, cb_arg, nr_to_walk,
+				      &lru->active_nodes);
+}
 
 unsigned long
 list_lru_dispose_all(struct list_lru *lru, list_lru_dispose_cb dispose);
lib/list_lru.c +8 -7
index 6d43079..248be3b 100644
@@ -54,12 +54,12 @@ list_lru_del(
 EXPORT_SYMBOL_GPL(list_lru_del);
 
 unsigned long
-list_lru_count(struct list_lru *lru)
+list_lru_count_nodemask(struct list_lru *lru, nodemask_t *nodes_to_count)
 {
 	long count = 0;
 	int nid;
 
-	for_each_node_mask(nid, lru->active_nodes) {
+	for_each_node_mask(nid, *nodes_to_count) {
 		struct list_lru_node *nlru = &lru->node[nid];
 
 		spin_lock(&nlru->lock);
@@ -70,7 +70,7 @@ list_lru_count(struct list_lru *lru)
 
 	return count;
 }
-EXPORT_SYMBOL_GPL(list_lru_count);
+EXPORT_SYMBOL_GPL(list_lru_count_nodemask);
 
 static unsigned long
 list_lru_walk_node(
@@ -119,16 +119,17 @@ restart:
 }
 
 unsigned long
-list_lru_walk(
+list_lru_walk_nodemask(
 	struct list_lru	*lru,
 	list_lru_walk_cb isolate,
 	void		*cb_arg,
-	unsigned long	nr_to_walk)
+	unsigned long	nr_to_walk,
+	nodemask_t	*nodes_to_walk)
 {
 	long isolated = 0;
 	int nid;
 
-	for_each_node_mask(nid, lru->active_nodes) {
+	for_each_node_mask(nid, *nodes_to_walk) {
 		isolated += list_lru_walk_node(lru, nid, isolate,
 					       cb_arg, &nr_to_walk);
 		if (nr_to_walk <= 0)
@@ -136,7 +137,7 @@ list_lru_walk(
 	}
 	return isolated;
 }
-EXPORT_SYMBOL_GPL(list_lru_walk);
+EXPORT_SYMBOL_GPL(list_lru_walk_nodemask);
 
 static unsigned long
 list_lru_dispose_all_node(
Signature · 1 line
1.8.1.4
maintainer
#22 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 6 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Now that the shrinker is passing a nodemask in the scan control structure, we can pass this to the the generic LRU list code to isolate reclaim to the lists on matching nodes.

Quoted 6 lines

This requires a small amount of refactoring of the LRU list API, which might be best split out into a separate patch.

What I see at this point is that the superblock shrinkers appear to be way too agressive. As soon as memory pressure hits, the slab caches are getting emptied down to almost nothing. I'm testing on a 4 node fake-numa system here, so I initially suspected what is happening is that the amount of reclaim has just been multiplied by 4.

Peak performance is good, it's just that there is no stable steady state - the caches are either filling at maximum rate, or being emptied at maximum rate - and that implies that the shrinkers are doing more work than they need to.

I think this is the point at what we need to ensure that the balance between the dentry/inode caches and the page cache is balanced, so I'm going to see what happens when I tweak a few numbers. e.g. see what effect tweaking shrink.seeks has on the rate of change of the cache sizes.....

Changing the seek count doesn't change the fact that it's fundamentally unstable. More investigation needed. Trace points.

kswapd0-632 mm_shrink_slab_start: objects to shrink 945211

		gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7046
		cache items 600456 delta 1363 total_scan 300228

Bingo. We've got windup!

	objects to shrink = shrinker->nr_in_batch
			  = 945211
			  = large amount of deferred work
	cache items = max_pass
		    = current cache size
		    = 600456
	total_scan  = 300228
		    = (cache items) / 2.
	delta	    = 1363

And this code:

/*
* We need to avoid excessive windup on filesystem shrinkers
* due to large numbers of GFP_NOFS allocations causing the
* shrinkers to return -1 all the time. This results in a large
* nr being built up so when a shrink that can do some work
* comes along it empties the entire cache due to nr >>>
* max_pass. This is bad for sustaining a working set in
* memory.
*
* Hence only allow the shrinker to scan the entire cache when
* a large delta change is calculated directly.
*/
if (delta < max_pass / 4)
total_scan = min(total_scan, max_pass / 2);

Has obviously triggered.

So, it would seem to me that we have a relatively small amount of incremental memory pressure, but an awful lot of deferred work. When I see this:

kswapd0-632 1210443.469309: mm_shrink_slab_start: cache items 600456 delta 1363 total_scan 300228 kswapd3-635 1210443.510311: mm_shrink_slab_start: cache items 514885 delta 1250 total_scan 101025 kswapd1-633 1210443.517440: mm_shrink_slab_start: cache items 613824 delta 1357 total_scan 97727 kswapd2-634 1210443.527026: mm_shrink_slab_start: cache items 568610 delta 1331 total_scan 259185 kswapd3-635 1210443.573165: mm_shrink_slab_start: cache items 486408 delta 1277 total_scan 243204 kswapd1-633 1210443.697012: mm_shrink_slab_start: cache items 550827 delta 1224 total_scan 82231

in the space of 230ms, I can see why the caches are getting completely emptied. kswapds are making multiple, large scale scan passes on the caches. Looks like our problem is an impedence
mismatch: global windup counter, per-node cache scan calculations.

So, that's the mess we really need to cleaning up before going much further with this patchset. We need stable behaviour from the shrinkers - I'll look into this a bit deeper tomorrow.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#23 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 7 lines · Dave Chinner

kswapd0-632 1210443.469309: mm_shrink_slab_start: cache items 600456 delta 1363 total_scan 300228 kswapd3-635 1210443.510311: mm_shrink_slab_start: cache items 514885 delta 1250 total_scan 101025 kswapd1-633 1210443.517440: mm_shrink_slab_start: cache items 613824 delta 1357 total_scan 97727 kswapd2-634 1210443.527026: mm_shrink_slab_start: cache items 568610 delta 1331 total_scan 259185 kswapd3-635 1210443.573165: mm_shrink_slab_start: cache items 486408 delta 1277 total_scan 243204 kswapd1-633 1210443.697012: mm_shrink_slab_start: cache items 550827 delta 1224 total_scan 82231

Quoted 8 lines

in the space of 230ms, I can see why the caches are getting completely emptied. kswapds are making multiple, large scale scan passes on the caches. Looks like our problem is an impedence
mismatch: global windup counter, per-node cache scan calculations.

So, that's the mess we really need to cleaning up before going much further with this patchset. We need stable behaviour from the shrinkers - I'll look into this a bit deeper tomorrow.

That doesn't totally make sense to me.

Both our scan and count functions will be per-node now. This means we will always try to keep ourselves within reasonable maximums on a per-node basis as well.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#24 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 17 lines · Glauber Costa
Quoted 7 lines · Dave Chinner

kswapd0-632 1210443.469309: mm_shrink_slab_start: cache items 600456 delta 1363 total_scan 300228 kswapd3-635 1210443.510311: mm_shrink_slab_start: cache items 514885 delta 1250 total_scan 101025 kswapd1-633 1210443.517440: mm_shrink_slab_start: cache items 613824 delta 1357 total_scan 97727 kswapd2-634 1210443.527026: mm_shrink_slab_start: cache items 568610 delta 1331 total_scan 259185 kswapd3-635 1210443.573165: mm_shrink_slab_start: cache items 486408 delta 1277 total_scan 243204 kswapd1-633 1210443.697012: mm_shrink_slab_start: cache items 550827 delta 1224 total_scan 82231

Quoted 8 lines

in the space of 230ms, I can see why the caches are getting completely emptied. kswapds are making multiple, large scale scan passes on the caches. Looks like our problem is an impedence
mismatch: global windup counter, per-node cache scan calculations.

So, that's the mess we really need to cleaning up before going much further with this patchset. We need stable behaviour from the shrinkers - I'll look into this a bit deeper tomorrow.

Quoted 5 lines

That doesn't totally make sense to me.

Both our scan and count functions will be per-node now. This means we will always try to keep ourselves within reasonable maximums on a per-node basis as well.

Right, but if we have a bunch of GFP_NOFS reclaims on multiple nodes at the same time, we get:

	max_pass = shr->count_objects;
	nr = shr->nr_in_batch
	shr->nr_in_batch = 0
	/* total_scan has new delta added */
	/* nothing scanned */
	shr->nr_in_batch += total_scan;

And then the next node does the same, and so on.

What I cut from the above output was the shr->nr_in_batch values. They were:

kswapd1-633 [001] 1210443.500045: objects to shrink 4077

	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7096 cache
	 items 623079 delta 1404 total_scan 5481
 kswapd1-633   [001] 1210443.504315: objects to shrink 15138
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7224 cache
	 items 620936 delta 1375 total_scan 16513
 kswapd3-635   [007] 1210443.510311: objects to shrink 99775
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6587 cache
	 items 514885 delta 1250 total_scan 101025
 kswapd1-633   [001] 1210443.517440: objects to shrink 96370
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7236 cache
	 items 613824 delta 1357 total_scan 97727
 kswapd2-634   [006] 1210443.527026: objects to shrink 257854
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6831 cache
	 items 568610 delta 1331 total_scan 259185
 kswapd3-635   [007] 1210443.573165: objects to shrink 1034342
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6089 cache
	 items 486408 delta 1277 total_scan 243204

So you can see that the number of objects being deferred is driving the total_scan count completely in these cases.

This is over a period of 70ms - shr->nr_in_batch has gone from roughly zero to 1034342 because of deferred work. Between these traces are hundreds of GFP_NOFS reclaims from all 8 cpus (i.e. direct reclaim, every 300-400us on *each* CPU), each adding ~1200 to shr->nr_in_batch, and the only thing able to reclaim memory is kswapd as it does GFP_KERNEL context reclaim.

IOWs, each kswapd is taking the global windup and applying it to each per-node list, when in fact the windup is not distributed that way. The trace leading up to this kswapd scan:

kswapd1-633 [001] 1210443.517440: objects to shrink 96370

	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7236 cache
	 items 613824 delta 1357 total_scan 97727

Shows that most of the deferred work has come from CPUs 2, 3, 4 and a little from CPU 5. The cpu->node map shows that only CPU 5 is on node 1 (cpus 1 and 5 are on node 1), so this means that less than a quarter of the work that this node 1 shrinker is being asked to do was deferred from node 1. Most of it was deferred from nodes 0, 2 and 3, and so this work the shrinker is doing is doing nothing to relieve the memory pressure on those nodes. So direct reclaim on those nodes continues to wind up the nr_in_batch count.

And look at where the per-node cache count and windup is getting to at times in this process:

fs_mark-5561 [002] 1210443.555528: objects to shrink 2914436

	gfp_flags GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|GFP_NOTRACK
	pgs_scanned 32 lru_pgs 26591
	cache items 2085764 delta 1254 total_scan 1042882

What we see here is a node with more than 2 million filesystem items cached on it. The other nodes are around 500,000 at this point, indicating that we definitely have a per-node reclaim imbalance....

IOWs, shr->nr_in_batch can grow much larger than any single node LRU list, and the deffered count is only limited to (2 * max_pass). Hence if the same node is the one that keeps stealing the global shr->nr_in_batch calculation, it will always be a number related to the size of the cache on that node. All the other nodes will simply keep adding their delta counts to it.

Hence if you've got a node with less cache in it than others, and kswapd comes along, it will see a gigantic amount of deferred work in nr_in_batch, and then we end up removing a large amount of the cache on that node, even though it hasn't had a significant amount of pressure. And the node that has pressure continues to wind up nr_in_batch until it's the one that gets hit by a kswapd run with that wound up nr_in_batch....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#25 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 104 lines · Dave Chinner
Quoted 17 lines · Glauber Costa
Quoted 7 lines · Dave Chinner

kswapd0-632 1210443.469309: mm_shrink_slab_start: cache items 600456 delta 1363 total_scan 300228 kswapd3-635 1210443.510311: mm_shrink_slab_start: cache items 514885 delta 1250 total_scan 101025 kswapd1-633 1210443.517440: mm_shrink_slab_start: cache items 613824 delta 1357 total_scan 97727 kswapd2-634 1210443.527026: mm_shrink_slab_start: cache items 568610 delta 1331 total_scan 259185 kswapd3-635 1210443.573165: mm_shrink_slab_start: cache items 486408 delta 1277 total_scan 243204 kswapd1-633 1210443.697012: mm_shrink_slab_start: cache items 550827 delta 1224 total_scan 82231

Quoted 8 lines

in the space of 230ms, I can see why the caches are getting completely emptied. kswapds are making multiple, large scale scan passes on the caches. Looks like our problem is an impedence
mismatch: global windup counter, per-node cache scan calculations.

So, that's the mess we really need to cleaning up before going much further with this patchset. We need stable behaviour from the shrinkers - I'll look into this a bit deeper tomorrow.

Quoted 5 lines

That doesn't totally make sense to me.

Both our scan and count functions will be per-node now. This means we will always try to keep ourselves within reasonable maximums on a per-node basis as well.

Right, but if we have a bunch of GFP_NOFS reclaims on multiple nodes at the same time, we get:

	max_pass = shr->count_objects;
	nr = shr->nr_in_batch
	shr->nr_in_batch = 0
	/* total_scan has new delta added */
	/* nothing scanned */
	shr->nr_in_batch += total_scan;

And then the next node does the same, and so on.

What I cut from the above output was the shr->nr_in_batch values. They were:

kswapd1-633 [001] 1210443.500045: objects to shrink 4077

	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7096 cache
	 items 623079 delta 1404 total_scan 5481
 kswapd1-633   [001] 1210443.504315: objects to shrink 15138
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7224 cache
	 items 620936 delta 1375 total_scan 16513
 kswapd3-635   [007] 1210443.510311: objects to shrink 99775
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6587 cache
	 items 514885 delta 1250 total_scan 101025
 kswapd1-633   [001] 1210443.517440: objects to shrink 96370
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7236 cache
	 items 613824 delta 1357 total_scan 97727
 kswapd2-634   [006] 1210443.527026: objects to shrink 257854
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6831 cache
	 items 568610 delta 1331 total_scan 259185
 kswapd3-635   [007] 1210443.573165: objects to shrink 1034342
	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 6089 cache
	 items 486408 delta 1277 total_scan 243204

So you can see that the number of objects being deferred is driving the total_scan count completely in these cases.

This is over a period of 70ms - shr->nr_in_batch has gone from roughly zero to 1034342 because of deferred work. Between these traces are hundreds of GFP_NOFS reclaims from all 8 cpus (i.e. direct reclaim, every 300-400us on *each* CPU), each adding ~1200 to shr->nr_in_batch, and the only thing able to reclaim memory is kswapd as it does GFP_KERNEL context reclaim.

IOWs, each kswapd is taking the global windup and applying it to each per-node list, when in fact the windup is not distributed that way. The trace leading up to this kswapd scan:

kswapd1-633 [001] 1210443.517440: objects to shrink 96370

	 gfp_flags GFP_KERNEL pgs_scanned 32 lru_pgs 7236 cache
	 items 613824 delta 1357 total_scan 97727

Shows that most of the deferred work has come from CPUs 2, 3, 4 and a little from CPU 5. The cpu->node map shows that only CPU 5 is on node 1 (cpus 1 and 5 are on node 1), so this means that less than a quarter of the work that this node 1 shrinker is being asked to do was deferred from node 1. Most of it was deferred from nodes 0, 2 and 3, and so this work the shrinker is doing is doing nothing to relieve the memory pressure on those nodes. So direct reclaim on those nodes continues to wind up the nr_in_batch count.

And look at where the per-node cache count and windup is getting to at times in this process:

fs_mark-5561 [002] 1210443.555528: objects to shrink 2914436

	gfp_flags GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|GFP_NOTRACK
	pgs_scanned 32 lru_pgs 26591
	cache items 2085764 delta 1254 total_scan 1042882

What we see here is a node with more than 2 million filesystem items cached on it. The other nodes are around 500,000 at this point, indicating that we definitely have a per-node reclaim imbalance....

IOWs, shr->nr_in_batch can grow much larger than any single node LRU list, and the deffered count is only limited to (2 * max_pass). Hence if the same node is the one that keeps stealing the global shr->nr_in_batch calculation, it will always be a number related to the size of the cache on that node. All the other nodes will simply keep adding their delta counts to it.

Quoted 8 lines

Hence if you've got a node with less cache in it than others, and kswapd comes along, it will see a gigantic amount of deferred work in nr_in_batch, and then we end up removing a large amount of the cache on that node, even though it hasn't had a significant amount of pressure. And the node that has pressure continues to wind up nr_in_batch until it's the one that gets hit by a kswapd run with that wound up nr_in_batch....

That makes sense now, thanks. It doesn't seem, however, that any deep black magic is required to fix this here.

Any reason why proportional scanning wouldn't work ?

If we have a very large nr_in_batch, and many nodes to scan, we can try dividing them among present nodes in the proportion of existing objects in the cache.

Or maybe even better, we can make deferring work node aware, so when you defer, you don't defer to a global pool, but rather to a node copy.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#26 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

In reply to Dave Chinner
Quoted 16 lines

IOWs, shr->nr_in_batch can grow much larger than any single node LRU list, and the deffered count is only limited to (2 * max_pass). Hence if the same node is the one that keeps stealing the global shr->nr_in_batch calculation, it will always be a number related to the size of the cache on that node. All the other nodes will simply keep adding their delta counts to it.

Hence if you've got a node with less cache in it than others, and kswapd comes along, it will see a gigantic amount of deferred work in nr_in_batch, and then we end up removing a large amount of the cache on that node, even though it hasn't had a significant amount of pressure. And the node that has pressure continues to wind up nr_in_batch until it's the one that gets hit by a kswapd run with that wound up nr_in_batch....

Quoted 4 lines

Cheers,

Dave.

Ok Dave,

My system in general seems to behave quite differently than this. In special, I hardly see peaks and the caches fill up very slowly. They later are pruned but always down to the same level, and then they grow slowly again, in a triangular fashion. Always within a fairly reasonable range. This might be because my disks are slower than yours.

It may also be some glitch in my setup. I spent a fair amount of time today trying to see your behavior but I can't. I will try more tomorrow.

For the time being, what do you think about the following patch (that obviously need a lot more work, just a PoC) ?

If we are indeed deferring work to unrelated nodes, keeping the deferred work per-node should help. I don't want to make it a static array because the shrinker structure tend to be embedded in structures. In particular, the superblock already have two list_lrus with per-node static arrays. This will make the sb gigantic. But that is not the main thing.

include/linux/shrinker.h +1 -1
index 98be3ab..3edcd7f 100644
@@ -53,7 +53,7 @@ struct shrinker {
 
 	/* These are for internal use */
 	struct list_head list;
-	atomic_long_t nr_in_batch; /* objs pending delete */
+	atomic_long_t *nr_in_batch; /* objs pending delete, per node */
 };
 #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */
 extern void register_shrinker(struct shrinker *);
mm/vmscan.c +30 -17

78 lines not on this page · raw message

maintainer
#27 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 20 lines · Glauber Costa
Quoted 16 lines

IOWs, shr->nr_in_batch can grow much larger than any single node LRU list, and the deffered count is only limited to (2 * max_pass). Hence if the same node is the one that keeps stealing the global shr->nr_in_batch calculation, it will always be a number related to the size of the cache on that node. All the other nodes will simply keep adding their delta counts to it.

Hence if you've got a node with less cache in it than others, and kswapd comes along, it will see a gigantic amount of deferred work in nr_in_batch, and then we end up removing a large amount of the cache on that node, even though it hasn't had a significant amount of pressure. And the node that has pressure continues to wind up nr_in_batch until it's the one that gets hit by a kswapd run with that wound up nr_in_batch....

Quoted 4 lines

Cheers,

Dave.

Quoted 7 lines

Ok Dave,

My system in general seems to behave quite differently than this. In special, I hardly see peaks and the caches fill up very slowly. They later are pruned but always down to the same level, and then they grow slowly again, in a triangular fashion. Always within a fairly reasonable range. This might be because my disks are slower than yours.

Yes, that is the sort of pattern I'd expect from slower disks....

Quoted 6 lines

It may also be some glitch in my setup. I spent a fair amount of time today trying to see your behavior but I can't. I will try more tomorrow.

For the time being, what do you think about the following patch (that obviously need a lot more work, just a PoC) ?

Quoted 6 lines

If we are indeed deferring work to unrelated nodes, keeping the deferred work per-node should help. I don't want to make it a static array because the shrinker structure tend to be embedded in structures. In particular, the superblock already have two list_lrus with per-node static arrays. This will make the sb gigantic. But that is not the main thing.

I'm not concerned about the size of the superblock structure, to tell the truth. ;)

What I'd suggest is that the shrinker needs a flags/capabilities field that indicates we care about per-node accounting, and only use this method for the shrinkers that can actually do numa aware shrinking.

Quoted 20 lines
include/linux/shrinker.h +1 -1
index 98be3ab..3edcd7f 100644
@@ -53,7 +53,7 @@ struct shrinker {
 
 	/* These are for internal use */
 	struct list_head list;
-	atomic_long_t nr_in_batch; /* objs pending delete */
+	atomic_long_t *nr_in_batch; /* objs pending delete, per node */
 };
 #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */
 extern void register_shrinker(struct shrinker *);
mm/vmscan.c +0 -0
index 35a6a9b..6dddc8d 100644
@@ -159,7 +159,14 @@ static unsigned long get_lru_size(struct lruvec *lruvec, enum lru_list lru)
  */
 void register_shrinker(struct shrinker *shrinker)

mm/vmscan.c static unsigned long get_lru_size(struct lruvec *lruvec, enum lru_list lru) · line 169

Quoted 10 lines
mm/vmscan.c +8 -1
 {
-	atomic_long_set(&shrinker->nr_in_batch, 0);
+	int i = 0;
+
+	shrinker->nr_in_batch = kmalloc(sizeof(atomic_long_t) * nr_node_ids, GFP_KERNEL);
+	BUG_ON(!shrinker->nr_in_batch); /* obviously bogus */
+
+	for (i = 0; i < nr_node_ids; i++)
+		atomic_long_set(&shrinker->nr_in_batch[i], 0);
+

just use kzalloc....

Quoted 28 lines
mm/vmscan.c +2 -5
 	down_write(&shrinker_rwsem);
 	list_add_tail(&shrinker->list, &shrinker_list);
 	up_write(&shrinker_rwsem);
@@ -211,6 +218,7 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 {
 	struct shrinker *shrinker;
 	unsigned long freed = 0;
+	unsigned long nr_active_nodes = 0;
 
 	if (nr_pages_scanned == 0)
 		nr_pages_scanned = SWAP_CLUSTER_MAX;
@@ -229,6 +237,7 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 		long new_nr;
 		long batch_size = shrinker->batch ? shrinker->batch
 						  : SHRINK_BATCH;
+		int nid;
 
 		if (shrinker->scan_objects) {
 			max_pass = shrinker->count_objects(shrinker, shrinkctl);
@@ -238,12 +247,17 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 		if (max_pass <= 0)
 			continue;
 
-		/*
-		 * copy the current shrinker scan count into a local variable
-		 * and zero it so that other concurrent shrinker invocations
-		 * don't also do this scanning work.
-		 */

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 258

Quoted 10 lines
mm/vmscan.c +9 -1
-		nr = atomic_long_xchg(&shrinker->nr_in_batch, 0);
+		nr = 0;
+		for_each_node_mask(nid, shrinkctl->nodes_to_scan) {
+			/*
+			 * copy the current shrinker scan count into a local
+			 * variable and zero it so that other concurrent
+			 * shrinker invocations don't also do this scanning
+			 * work.
+			 */
+			nr += atomic_long_xchg(&shrinker->nr_in_batch[nid], 0);

Just a thought - let's rename "nr_in_batch" to
"deferred_scan"....

Quoted 2 lines
mm/vmscan.c +2 -0
+			nr_active_nodes++;
+		}

.....

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 333

Quoted 5 lines
mm/vmscan.c +5 -0
+		total_scan /= nr_active_nodes;
+		for_each_node_mask(nid, shrinkctl->nodes_to_scan) {
+			if (total_scan > 0)
+				new_nr += atomic_long_add_return(total_scan / nr_active_nodes,
+						&shrinker->nr_in_batch[nid]);

(you do the total_scan / nr_active_nodes twice here)

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 337

Quoted 4 lines
mm/vmscan.c +3 -0
+			else
+				new_nr += atomic_long_read(&shrinker->nr_in_batch[nid]);
 
+		}

I don't think this solves the problem entirely - it still aggregates multiple nodes together into the one count. It might be better, but it will still bleed the deferred count from a single node into other nodes that have no deferred count.

Perhaps we need to factor this code a little first - separate the calculation from the per-shrinker loop, so we can do something like:

shrink_slab_node(shr, sc, nid)

{
	nodemask_clear(sc->nodemask);
	nodemask_set(sc->nodemask, nid)
	for each shrinker {
		deferred_count = atomic_long_xchg(&shr->deferred_scan[nid], 0);

		deferred_count = __shrink_slab(shr, sc, deferred_count);

		atomic_long_add(deferred_count, &shr->deferred_scan[nid]);
	}
}

And the existing shrink_slab function becomes something like:

shrink_slab(shr, sc, nodemask)

{
	if (shr->flags & SHRINKER_NUMA_AWARE) {
		for_each_node_mask(nid, nodemask)
			shrink_slab_node(shr, sc, nid)
		return;
	}

	for each shrinker {
		deferred_count = atomic_long_xchg(&shr->deferred_scan[0], 0);

		deferred_count = __shrink_slab(shr, sc, deferred_count);

		atomic_long_add(deferred_count, &shr->deferred_scan[0]);
	}
}

This then makes the deferred count properly node aware when the underlying shrinker needs it to be, and prevents bleed from one node to another. I'd much prefer to see us move to an explicitly node based iteration like this than try to hack more stuff into shrink_slab() and confuse it further.

The only thing I don't like about this is the extra nodemask needed, which, like the scan control, would have to sit on the stack. Suggestions for avoiding that problem are welcome.. :)

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#28 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 7 lines · Dave Chinner
Quoted 5 lines
mm/vmscan.c +1 -0
+		total_scan /= nr_active_nodes;

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 333

Quoted 4 lines
mm/vmscan.c +4 -0
+		for_each_node_mask(nid, shrinkctl->nodes_to_scan) {
+			if (total_scan > 0)
+				new_nr += atomic_long_add_return(total_scan / nr_active_nodes,
+						&shrinker->nr_in_batch[nid]);

(you do the total_scan / nr_active_nodes twice here)

Thanks. Indeed. As I told you, I boot tested this, but since I haven't seen the behavior you are seeing, I didn't give it a lot of testing.

I am a lot more interested in finding out if this approach is worth it. So if you could give something like this a go, that would be awesome.

Quoted 9 lines
Quoted 4 lines

mm/vmscan.c unsigned long shrink_slab(struct shrink_control *shrinkctl, · line 337

Quoted 4 lines
mm/vmscan.c +3 -0
+			else
+				new_nr += atomic_long_read(&shrinker->nr_in_batch[nid]);
 
+		}

I don't think this solves the problem entirely - it still aggregates multiple nodes together into the one count. It might be better, but it will still bleed the deferred count from a single node into other nodes that have no deferred count.

Yes, but only the nodes that are being scan in this moment, no? If the shrinker is deferring because we returned -1, this means that nobody was able to shrink anything.

Quoted 18 lines

Perhaps we need to factor this code a little first - separate the calculation from the per-shrinker loop, so we can do something like:

shrink_slab_node(shr, sc, nid)

{
	nodemask_clear(sc->nodemask);
	nodemask_set(sc->nodemask, nid)
	for each shrinker {
		deferred_count = atomic_long_xchg(&shr->deferred_scan[nid], 0);

		deferred_count = __shrink_slab(shr, sc, deferred_count);

		atomic_long_add(deferred_count, &shr->deferred_scan[nid]);
	}
}

And the existing shrink_slab function becomes something like:

Quoted 7 lines

shrink_slab(shr, sc, nodemask)

{
	if (shr->flags & SHRINKER_NUMA_AWARE) {
		for_each_node_mask(nid, nodemask)
			shrink_slab_node(shr, sc, nid)
		return;
	}

I am fine with a numa aware flag.

Quoted 10 lines
	for each shrinker {
		deferred_count = atomic_long_xchg(&shr->deferred_scan[0], 0);

		deferred_count = __shrink_slab(shr, sc, deferred_count);

		atomic_long_add(deferred_count, &shr->deferred_scan[0]);
	}
}
Quoted 6 lines

This then makes the deferred count properly node aware when the underlying shrinker needs it to be, and prevents bleed from one node to another. I'd much prefer to see us move to an explicitly node based iteration like this than try to hack more stuff into shrink_slab() and confuse it further.

Except that shrink_slab_node would also defer work, right?

Quoted 4 lines

The only thing I don't like about this is the extra nodemask needed, which, like the scan control, would have to sit on the stack. Suggestions for avoiding that problem are welcome.. :)

I will try to come up with a patch to do all this, and then we can concretely discuss.
You are also of course welcome to do so as well =)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#29 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 9 lines · Glauber Costa

Except that shrink_slab_node would also defer work, right?

Quoted 4 lines
Quoted 4 lines

The only thing I don't like about this is the extra nodemask needed, which, like the scan control, would have to sit on the stack. Suggestions for avoiding that problem are welcome.. :)

I will try to come up with a patch to do all this, and then we can concretely discuss.
You are also of course welcome to do so as well =)

All right.

I played a bit today with variations of this patch that will keep the deferred count per node. I will rebase the whole series ontop of it (the changes can get quite disruptive) and post. I want to believe that after this, all our regression problems will be gone (famous last words).

As I have told you, I wasn't seeing problems like you are, and speculated that this was due to the disk speeds. While this is true, the patch I came up with makes my workload actually a lot better. While my caches weren't being emptied, they were being slightly depleted and then slowly filled again. With my new patch, it is almost a straight line throughout the whole find run. There is a dent here and there eventually, but it recovers quickly. It takes some time as well for steady state to be reached, but once it is, we have all variables in the equation (dentries, inodes, etc) basically flat. So I guess it works, and I am confident that it will make your workload better.

My strategy is to modify the shrinker structure like this:

struct shrinker {
        int (*shrink)(struct shrinker *, struct shrink_control *sc);
        long (*count_objects)(struct shrinker *, struct shrink_control *sc);
        long (*scan_objects)(struct shrinker *, struct shrink_control *sc);

        int seeks;      /* seeks to recreate an obj */
        long batch;     /* reclaim batch size, 0 = default */
        unsigned long flags;

        /* These are for internal use */
        struct list_head list;
        atomic_long_t *nr_deferred; /* objs pending delete, per node */

        /* nodes being currently shrunk, only makes sense for NUMA
shrinkers */
        nodemask_t *nodes_shrinking;

};

We need memory allocation now for nr_deferred and nodes_shrinking, but OTOH we use no stack, and can keep the size of this to be dynamically adjusted depending on whether or not your shrinker is NUMA aware.

Guess that is it. Expect news soon.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#30 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

Quoted 49 lines · Glauber Costa
Quoted 9 lines · Glauber Costa

Except that shrink_slab_node would also defer work, right?

Quoted 4 lines
Quoted 4 lines

The only thing I don't like about this is the extra nodemask needed, which, like the scan control, would have to sit on the stack. Suggestions for avoiding that problem are welcome.. :)

I will try to come up with a patch to do all this, and then we can concretely discuss.
You are also of course welcome to do so as well =)

All right.

I played a bit today with variations of this patch that will keep the deferred count per node. I will rebase the whole series ontop of it (the changes can get quite disruptive) and post. I want to believe that after this, all our regression problems will be gone (famous last words).

As I have told you, I wasn't seeing problems like you are, and speculated that this was due to the disk speeds. While this is true, the patch I came up with makes my workload actually a lot better. While my caches weren't being emptied, they were being slightly depleted and then slowly filled again. With my new patch, it is almost a straight line throughout the whole find run. There is a dent here and there eventually, but it recovers quickly. It takes some time as well for steady state to be reached, but once it is, we have all variables in the equation (dentries, inodes, etc) basically flat. So I guess it works, and I am confident that it will make your workload better.

My strategy is to modify the shrinker structure like this:

struct shrinker {
        int (*shrink)(struct shrinker *, struct shrink_control *sc);
        long (*count_objects)(struct shrinker *, struct shrink_control *sc);
        long (*scan_objects)(struct shrinker *, struct shrink_control *sc);

        int seeks;      /* seeks to recreate an obj */
        long batch;     /* reclaim batch size, 0 = default */
        unsigned long flags;

        /* These are for internal use */
        struct list_head list;
        atomic_long_t *nr_deferred; /* objs pending delete, per node */

        /* nodes being currently shrunk, only makes sense for NUMA
shrinkers */
        nodemask_t *nodes_shrinking;
Quoted 8 lines

};

We need memory allocation now for nr_deferred and nodes_shrinking, but OTOH we use no stack, and can keep the size of this to be dynamically adjusted depending on whether or not your shrinker is NUMA aware.

Guess that is it. Expect news soon.

Except of course that struct shrinker is obviously shared between runs, and this won't cut.

Right now I am inclined to really just put this in the stack. The alternative, if it becomes a problem, can be to extend the lru apis to allow us to go for a single node. This way we only need to use 1 extra word in the stack.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#31 lore raw

Re: [PATCH v6 12/31] fs: convert inode and dentry shrinking to be node aware

In reply to Glauber Costa
Quoted 19 lines · Glauber Costa
Quoted 9 lines · Glauber Costa

Except that shrink_slab_node would also defer work, right?

Quoted 4 lines
Quoted 4 lines

The only thing I don't like about this is the extra nodemask needed, which, like the scan control, would have to sit on the stack. Suggestions for avoiding that problem are welcome.. :)

I will try to come up with a patch to do all this, and then we can concretely discuss.
You are also of course welcome to do so as well =)

All right.

I played a bit today with variations of this patch that will keep the deferred count per node. I will rebase the whole series ontop of it (the changes can get quite disruptive) and post. I want to believe that after this, all our regression problems will be gone (famous last words).

Quoted 10 lines

As I have told you, I wasn't seeing problems like you are, and speculated that this was due to the disk speeds. While this is true, the patch I came up with makes my workload actually a lot better. While my caches weren't being emptied, they were being slightly depleted and then slowly filled again. With my new patch, it is almost a straight line throughout the whole find run. There is a dent here and there eventually, but it recovers quickly. It takes some time as well for steady state to be reached, but once it is, we have all variables in the equation (dentries, inodes, etc) basically flat. So I guess it works, and I am confident that it will make your workload better.

Sounds promising. A non-numa system gives dead flat linesonthat workload, and that's how I want a numa system to behave as well when there is even pressure applied across all nodes.

Sorry for not doing more here - I've been flat out the last couple of days with XFS CRC stuff. Let me know when you have a tree rebased as far as this patch and I'll run some tests here....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#33 lore raw

PATCH v6 13/31 xfs: convert buftarg LRU to generic code

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert the buftarg LRU to use the new generic LRU list and take advantage of the functionality it supplies to make the buffer cache shrinker node aware.

---
 fs/xfs/xfs_buf.c | 169 ++++++++++++++++++++++++++-----------------------------
 fs/xfs/xfs_buf.h |   5 +-
 2 files changed, 81 insertions(+), 93 deletions(-)
fs/xfs/xfs_buf.c +79 -90
index cee0e42..802b65b 100644
@@ -85,20 +85,14 @@ xfs_buf_vmap_len(
  * The LRU takes a new reference to the buffer so that it will only be freed
  * once the shrinker takes the buffer off the LRU.
  */
-STATIC void
+static void
 xfs_buf_lru_add(
 	struct xfs_buf	*bp)
 {
-	struct xfs_buftarg *btp = bp->b_target;
-
-	spin_lock(&btp->bt_lru_lock);
-	if (list_empty(&bp->b_lru)) {
-		atomic_inc(&bp->b_hold);
-		list_add_tail(&bp->b_lru, &btp->bt_lru);
-		btp->bt_lru_nr++;
+	if (list_lru_add(&bp->b_target->bt_lru, &bp->b_lru)) {
 		bp->b_lru_flags &= ~_XBF_LRU_DISPOSE;
+		atomic_inc(&bp->b_hold);
 	}
-	spin_unlock(&btp->bt_lru_lock);
 }
 
 /*
@@ -107,24 +101,13 @@ xfs_buf_lru_add(
  * The unlocked check is safe here because it only occurs when there are not
  * b_lru_ref counts left on the inode under the pag->pag_buf_lock. it is there
  * to optimise the shrinker removing the buffer from the LRU and calling
- * xfs_buf_free(). i.e. it removes an unnecessary round trip on the
- * bt_lru_lock.
+ * xfs_buf_free().
  */
-STATIC void
+static void
 xfs_buf_lru_del(
 	struct xfs_buf	*bp)
 {
-	struct xfs_buftarg *btp = bp->b_target;
-
-	if (list_empty(&bp->b_lru))
-		return;
-
-	spin_lock(&btp->bt_lru_lock);
-	if (!list_empty(&bp->b_lru)) {
-		list_del_init(&bp->b_lru);
-		btp->bt_lru_nr--;
-	}
-	spin_unlock(&btp->bt_lru_lock);
+	list_lru_del(&bp->b_target->bt_lru, &bp->b_lru);
 }
 
 /*
@@ -151,18 +134,10 @@ xfs_buf_stale(
 	bp->b_flags &= ~_XBF_DELWRI_Q;
 
 	atomic_set(&(bp)->b_lru_ref, 0);
-	if (!list_empty(&bp->b_lru)) {
-		struct xfs_buftarg *btp = bp->b_target;
-
-		spin_lock(&btp->bt_lru_lock);
-		if (!list_empty(&bp->b_lru) &&
-		    !(bp->b_lru_flags & _XBF_LRU_DISPOSE)) {
-			list_del_init(&bp->b_lru);
-			btp->bt_lru_nr--;
-			atomic_dec(&bp->b_hold);
-		}
-		spin_unlock(&btp->bt_lru_lock);
-	}
+	if (!(bp->b_lru_flags & _XBF_LRU_DISPOSE) &&
+	    (list_lru_del(&bp->b_target->bt_lru, &bp->b_lru)))
+		atomic_dec(&bp->b_hold);
+
 	ASSERT(atomic_read(&bp->b_hold) >= 1);
 }
 
@@ -1501,83 +1476,97 @@ xfs_buf_iomove(
  * returned. These buffers will have an elevated hold count, so wait on those
  * while freeing all the buffers only held by the LRU.
  */
-void
-xfs_wait_buftarg(
-	struct xfs_buftarg	*btp)
+static enum lru_status
+xfs_buftarg_wait_rele(
+	struct list_head	*item,
+	spinlock_t		*lru_lock,
+	void			*arg)
+
 {
-	struct xfs_buf		*bp;
+	struct xfs_buf		*bp = container_of(item, struct xfs_buf, b_lru);
 
-restart:
-	spin_lock(&btp->bt_lru_lock);
-	while (!list_empty(&btp->bt_lru)) {
-		bp = list_first_entry(&btp->bt_lru, struct xfs_buf, b_lru);
-		if (atomic_read(&bp->b_hold) > 1) {
-			trace_xfs_buf_wait_buftarg(bp, _RET_IP_);
-			list_move_tail(&bp->b_lru, &btp->bt_lru);
-			spin_unlock(&btp->bt_lru_lock);
-			delay(100);
-			goto restart;
-		}
+	if (atomic_read(&bp->b_hold) > 1) {
+		/* need to wait */
+		trace_xfs_buf_wait_buftarg(bp, _RET_IP_);
+		spin_unlock(lru_lock);
+		delay(100);
+	} else {
 		/*
 		 * clear the LRU reference count so the buffer doesn't get
 		 * ignored in xfs_buf_rele().
 		 */
 		atomic_set(&bp->b_lru_ref, 0);
-		spin_unlock(&btp->bt_lru_lock);
+		spin_unlock(lru_lock);
 		xfs_buf_rele(bp);
-		spin_lock(&btp->bt_lru_lock);
 	}
-	spin_unlock(&btp->bt_lru_lock);
+
+	spin_lock(lru_lock);
+	return LRU_RETRY;
 }
 
-int
-xfs_buftarg_shrink(
+void
+xfs_wait_buftarg(
+	struct xfs_buftarg	*btp)
+{
+	while (list_lru_count(&btp->bt_lru))
+		list_lru_walk(&btp->bt_lru, xfs_buftarg_wait_rele,
+			      NULL, LONG_MAX);
+}
+
+static enum lru_status
+xfs_buftarg_isolate(
+	struct list_head	*item,
+	spinlock_t		*lru_lock,
+	void			*arg)
+{
+	struct xfs_buf		*bp = container_of(item, struct xfs_buf, b_lru);
+	struct list_head	*dispose = arg;
+
+	/*
+	 * Decrement the b_lru_ref count unless the value is already
+	 * zero. If the value is already zero, we need to reclaim the
+	 * buffer, otherwise it gets another trip through the LRU.
+	 */
+	if (!atomic_add_unless(&bp->b_lru_ref, -1, 0))
+		return LRU_ROTATE;
+
+	bp->b_lru_flags |= _XBF_LRU_DISPOSE;
+	list_move(item, dispose);
+	return LRU_REMOVED;
+}
+
+static long
+xfs_buftarg_shrink_scan(
 	struct shrinker		*shrink,
 	struct shrink_control	*sc)
 {
 	struct xfs_buftarg	*btp = container_of(shrink,
 					struct xfs_buftarg, bt_shrinker);
-	struct xfs_buf		*bp;
-	int nr_to_scan = sc->nr_to_scan;
 	LIST_HEAD(dispose);
+	long			freed;
 
-	if (!nr_to_scan)
-		return btp->bt_lru_nr;
-
-	spin_lock(&btp->bt_lru_lock);
-	while (!list_empty(&btp->bt_lru)) {
-		if (nr_to_scan-- <= 0)
-			break;
-
-		bp = list_first_entry(&btp->bt_lru, struct xfs_buf, b_lru);
-
-		/*
-		 * Decrement the b_lru_ref count unless the value is already
-		 * zero. If the value is already zero, we need to reclaim the
-		 * buffer, otherwise it gets another trip through the LRU.
-		 */
-		if (!atomic_add_unless(&bp->b_lru_ref, -1, 0)) {
-			list_move_tail(&bp->b_lru, &btp->bt_lru);
-			continue;
-		}
-
-		/*
-		 * remove the buffer from the LRU now to avoid needing another
-		 * lock round trip inside xfs_buf_rele().
-		 */
-		list_move(&bp->b_lru, &dispose);
-		btp->bt_lru_nr--;
-		bp->b_lru_flags |= _XBF_LRU_DISPOSE;
-	}
-	spin_unlock(&btp->bt_lru_lock);
+	freed = list_lru_walk_nodemask(&btp->bt_lru, xfs_buftarg_isolate,
+				       &dispose, sc->nr_to_scan,
+				       &sc->nodes_to_scan);
 
 	while (!list_empty(&dispose)) {
+		struct xfs_buf *bp;
 		bp = list_first_entry(&dispose, struct xfs_buf, b_lru);
 		list_del_init(&bp->b_lru);
 		xfs_buf_rele(bp);
 	}
 
-	return btp->bt_lru_nr;
+	return freed;
+}
+
+static long
+xfs_buftarg_shrink_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
+{
+	struct xfs_buftarg	*btp = container_of(shrink,
+					struct xfs_buftarg, bt_shrinker);
+	return list_lru_count_nodemask(&btp->bt_lru, &sc->nodes_to_scan);
 }
 
 void
@@ -1659,11 +1648,11 @@ xfs_alloc_buftarg(
 	if (!btp->bt_bdi)
 		goto error;
 
-	INIT_LIST_HEAD(&btp->bt_lru);
-	spin_lock_init(&btp->bt_lru_lock);
+	list_lru_init(&btp->bt_lru);
 	if (xfs_setsize_buftarg_early(btp, bdev))
 		goto error;
-	btp->bt_shrinker.shrink = xfs_buftarg_shrink;
+	btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count;
+	btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan;
 	btp->bt_shrinker.seeks = DEFAULT_SEEKS;
 	register_shrinker(&btp->bt_shrinker);
 	return btp;
fs/xfs/xfs_buf.h +2 -3
index 433a12e..5ec7d35 100644
@@ -25,6 +25,7 @@
 #include <linux/fs.h>
 #include <linux/buffer_head.h>
 #include <linux/uio.h>
+#include <linux/list_lru.h>
 
 /*
  *	Base types
@@ -92,9 +93,7 @@ typedef struct xfs_buftarg {
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
-	struct list_head	bt_lru;
-	spinlock_t		bt_lru_lock;
-	unsigned int		bt_lru_nr;
+	struct list_lru		bt_lru;
 } xfs_buftarg_t;
 
 struct xfs_buf;
Signature · 1 line
1.8.1.4
#34 lore raw

PATCH v6 14/31 xfs: convert dquot cache lru to list_lru

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert the XFS dquot lru to use the list_lru construct and convert the shrinker to being node aware.

[ glommer: edited for conflicts + warning fixes ]
---
 fs/xfs/xfs_dquot.c |   7 +-
 fs/xfs/xfs_qm.c    | 275 +++++++++++++++++++++++++++--------------------------
 fs/xfs/xfs_qm.h    |   4 +-
 3 files changed, 142 insertions(+), 144 deletions(-)
fs/xfs/xfs_dquot.c +1 -6
index a41f8bf..4e9178d 100644
@@ -950,13 +950,8 @@ xfs_qm_dqput_final(
 
 	trace_xfs_dqput_free(dqp);
 
-	mutex_lock(&qi->qi_lru_lock);
-	if (list_empty(&dqp->q_lru)) {
-		list_add_tail(&dqp->q_lru, &qi->qi_lru_list);
-		qi->qi_lru_count++;
+	if (list_lru_add(&qi->qi_lru, &dqp->q_lru))
 		XFS_STATS_INC(xs_qm_dquot_unused);
-	}
-	mutex_unlock(&qi->qi_lru_lock);
 
 	/*
 	 * If we just added a udquot to the freelist, then we want to release
fs/xfs/xfs_qm.c +140 -135

314 lines not on this page · raw message

fs/xfs/xfs_qm.h +1 -3
index 5d16a6e..8173b5e 100644
@@ -47,9 +47,7 @@ typedef struct xfs_quotainfo {
 	struct mutex qi_tree_lock;
 	xfs_inode_t	*qi_uquotaip;	 /* user quota inode */
 	xfs_inode_t	*qi_gquotaip;	 /* group quota inode */
-	struct list_head qi_lru_list;
-	struct mutex	 qi_lru_lock;
-	int		 qi_lru_count;
+	struct list_lru	 qi_lru;
 	int		 qi_dquots;
 	time_t		 qi_btimelimit;	 /* limit for blks timer */
 	time_t		 qi_itimelimit;	 /* limit for inodes timer */
Signature · 1 line
1.8.1.4
#35 lore raw

PATCH v6 15/31 fs: convert fs shrinkers to new scan/count API

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert the filesystem shrinkers to use the new API, and standardise some of the behaviours of the shrinkers at the same time. For example, nr_to_scan means the number of objects to scan, not the number of objects to free.

I refactored the CIFS idmap shrinker a little - it really needs to be broken up into a shrinker per tree and keep an item count with the tree root so that we don't need to walk the tree every time the shrinker needs to count the number of objects in the tree (i.e. all the time under memory pressure).

[ glommer: fixes for ext4, ubifs, nfs, cifs and glock. Fixes are
needed mainly due to new code merged in the tree ]

CC Steven Whitehouse <[email protected]>
CC Artem Bityutskiy <[email protected]>
CC Adrian Hunter <[email protected]>
CC Jan Kara <[email protected]>
---
 fs/ext4/extents_status.c | 30 ++++++++++++++++------------
 fs/gfs2/glock.c          | 28 +++++++++++++++-----------
 fs/gfs2/main.c           |  3 ++-
 fs/gfs2/quota.c          | 12 +++++++-----
 fs/gfs2/quota.h          |  4 +++-
 fs/mbcache.c             | 51 ++++++++++++++++++++++++++++--------------------
 fs/nfs/dir.c             | 18 ++++++++++++++---
 fs/nfs/internal.h        |  4 +++-
 fs/nfs/super.c           |  3 ++-
 fs/nfsd/nfscache.c       | 31 ++++++++++++++++++++---------
 fs/quota/dquot.c         | 34 +++++++++++++++-----------------
 fs/ubifs/shrinker.c      | 20 +++++++++++--------
 fs/ubifs/super.c         |  3 ++-
 fs/ubifs/ubifs.h         |  3 ++-
 14 files changed, 151 insertions(+), 93 deletions(-)
fs/ext4/extents_status.c +18 -12
index e6941e6..4bce4f0 100644
@@ -878,20 +878,26 @@ int ext4_es_zeroout(struct inode *inode, struct ext4_extent *ex)
 				     EXTENT_STATUS_WRITTEN);
 }
 
-static int ext4_es_shrink(struct shrinker *shrink, struct shrink_control *sc)
+
+static long ext4_es_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long nr;
+	struct ext4_sb_info *sbi = container_of(shrink,
+					struct ext4_sb_info, s_es_shrinker);
+
+	nr = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
+	trace_ext4_es_shrink_enter(sbi->s_sb, sc->nr_to_scan, nr);
+	return nr;
+}
+
+static long ext4_es_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct ext4_sb_info *sbi = container_of(shrink,
 					struct ext4_sb_info, s_es_shrinker);
 	struct ext4_inode_info *ei;
 	struct list_head *cur, *tmp, scanned;
 	int nr_to_scan = sc->nr_to_scan;
-	int ret, nr_shrunk = 0;
-
-	ret = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
-	trace_ext4_es_shrink_enter(sbi->s_sb, nr_to_scan, ret);
-
-	if (!nr_to_scan)
-		return ret;
+	int ret = 0, nr_shrunk = 0;
 
 	INIT_LIST_HEAD(&scanned);
 
@@ -920,9 +926,8 @@ static int ext4_es_shrink(struct shrinker *shrink, struct shrink_control *sc)
 	list_splice_tail(&scanned, &sbi->s_es_lru);
 	spin_unlock(&sbi->s_es_lru_lock);
 
-	ret = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
 	trace_ext4_es_shrink_exit(sbi->s_sb, nr_shrunk, ret);
-	return ret;
+	return nr_shrunk;
 }
 
 void ext4_es_register_shrinker(struct super_block *sb)
@@ -932,7 +937,8 @@ void ext4_es_register_shrinker(struct super_block *sb)
 	sbi = EXT4_SB(sb);
 	INIT_LIST_HEAD(&sbi->s_es_lru);
 	spin_lock_init(&sbi->s_es_lru_lock);
-	sbi->s_es_shrinker.shrink = ext4_es_shrink;
+	sbi->s_es_shrinker.scan_objects = ext4_es_scan;
+	sbi->s_es_shrinker.count_objects = ext4_es_count;
 	sbi->s_es_shrinker.seeks = DEFAULT_SEEKS;
 	register_shrinker(&sbi->s_es_shrinker);
 }
@@ -973,7 +979,7 @@ static int __es_try_to_reclaim_extents(struct ext4_inode_info *ei,
 	struct ext4_es_tree *tree = &ei->i_es_tree;
 	struct rb_node *node;
 	struct extent_status *es;
-	int nr_shrunk = 0;
+	long nr_shrunk = 0;
 
 	if (ei->i_es_lru_nr == 0)
 		return 0;
fs/gfs2/glock.c +17 -11
index 3bd2748..4ddbccb 100644
@@ -1428,21 +1428,22 @@ __acquires(&lru_lock)
  * gfs2_dispose_glock_lru() above.
  */
 
-static void gfs2_scan_glock_lru(int nr)
+static long gfs2_scan_glock_lru(int nr)
 {
 	struct gfs2_glock *gl;
 	LIST_HEAD(skipped);
 	LIST_HEAD(dispose);
+	long freed = 0;
 
 	spin_lock(&lru_lock);
-	while(nr && !list_empty(&lru_list)) {
+	while ((nr-- >= 0) && !list_empty(&lru_list)) {
 		gl = list_entry(lru_list.next, struct gfs2_glock, gl_lru);
 
 		/* Test for being demotable */
 		if (!test_and_set_bit(GLF_LOCK, &gl->gl_flags)) {
 			list_move(&gl->gl_lru, &dispose);
 			atomic_dec(&lru_count);
-			nr--;
+			freed++;
 			continue;
 		}
 
@@ -1452,23 +1453,28 @@ static void gfs2_scan_glock_lru(int nr)
 	if (!list_empty(&dispose))
 		gfs2_dispose_glock_lru(&dispose);
 	spin_unlock(&lru_lock);
+
+	return freed;
 }
 
-static int gfs2_shrink_glock_memory(struct shrinker *shrink,
-				    struct shrink_control *sc)
+static long gfs2_glock_shrink_scan(struct shrinker *shrink,
+				   struct shrink_control *sc)
 {
-	if (sc->nr_to_scan) {
-		if (!(sc->gfp_mask & __GFP_FS))
-			return -1;
-		gfs2_scan_glock_lru(sc->nr_to_scan);
-	}
+	if (!(sc->gfp_mask & __GFP_FS))
+		return -1;
+	return gfs2_scan_glock_lru(sc->nr_to_scan);
+}
 
+static long gfs2_glock_shrink_count(struct shrinker *shrink,
+				    struct shrink_control *sc)
+{
 	return vfs_pressure_ratio(atomic_read(&lru_count));
 }
 
 static struct shrinker glock_shrinker = {
-	.shrink = gfs2_shrink_glock_memory,
 	.seeks = DEFAULT_SEEKS,
+	.count_objects = gfs2_glock_shrink_count,
+	.scan_objects = gfs2_glock_shrink_scan,
 };
 
 /**
fs/gfs2/main.c +2 -1
index e04d0e0..a105d84 100644
@@ -32,7 +32,8 @@
 struct workqueue_struct *gfs2_control_wq;
 
 static struct shrinker qd_shrinker = {
-	.shrink = gfs2_shrink_qd_memory,
+	.count_objects = gfs2_qd_shrink_count,
+	.scan_objects = gfs2_qd_shrink_scan,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/gfs2/quota.c +7 -5
index 5c14206..77b82f6 100644
@@ -75,14 +75,12 @@ static LIST_HEAD(qd_lru_list);
 static atomic_t qd_lru_count = ATOMIC_INIT(0);
 static DEFINE_SPINLOCK(qd_lru_lock);
 
-int gfs2_shrink_qd_memory(struct shrinker *shrink, struct shrink_control *sc)
+long gfs2_qd_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct gfs2_quota_data *qd;
 	struct gfs2_sbd *sdp;
 	int nr_to_scan = sc->nr_to_scan;
-
-	if (nr_to_scan == 0)
-		goto out;
+	long freed = 0;
 
 	if (!(sc->gfp_mask & __GFP_FS))
 		return -1;
@@ -110,10 +108,14 @@ int gfs2_shrink_qd_memory(struct shrinker *shrink, struct shrink_control *sc)
 		kmem_cache_free(gfs2_quotad_cachep, qd);
 		spin_lock(&qd_lru_lock);
 		nr_to_scan--;
+		freed++;
 	}
 	spin_unlock(&qd_lru_lock);
+	return freed;
+}
 
-out:
+long gfs2_qd_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
 	return vfs_pressure_ratio(atomic_read(&qd_lru_count));
 }
 
fs/gfs2/quota.h +3 -1
index 4f5e6e4..4f61708 100644
@@ -53,7 +53,9 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip)
 	return ret;
 }
 
-extern int gfs2_shrink_qd_memory(struct shrinker *shrink,
+extern long gfs2_qd_shrink_count(struct shrinker *shrink,
+				 struct shrink_control *sc);
+extern long gfs2_qd_shrink_scan(struct shrinker *shrink,
 				 struct shrink_control *sc);
 extern const struct quotactl_ops gfs2_quotactl_ops;
 
fs/mbcache.c +30 -21

91 lines not on this page · raw message

fs/nfs/dir.c +15 -3
index 54d7c47..b355905 100644
@@ -1958,17 +1958,20 @@ static void nfs_access_free_list(struct list_head *head)
 	}
 }
 
-int nfs_access_cache_shrinker(struct shrinker *shrink,
-			      struct shrink_control *sc)
+long
+nfs_access_cache_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	LIST_HEAD(head);
 	struct nfs_inode *nfsi, *next;
 	struct nfs_access_entry *cache;
 	int nr_to_scan = sc->nr_to_scan;
 	gfp_t gfp_mask = sc->gfp_mask;
+	long freed = 0;
 
 	if ((gfp_mask & GFP_KERNEL) != GFP_KERNEL)
-		return (nr_to_scan == 0) ? 0 : -1;
+		return -1;
 
 	spin_lock(&nfs_access_lru_lock);
 	list_for_each_entry_safe(nfsi, next, &nfs_access_lru_list, access_cache_inode_lru) {
@@ -1984,6 +1987,7 @@ int nfs_access_cache_shrinker(struct shrinker *shrink,
 				struct nfs_access_entry, lru);
 		list_move(&cache->lru, &head);
 		rb_erase(&cache->rb_node, &nfsi->access_cache);
+		freed++;
 		if (!list_empty(&nfsi->access_cache_entry_lru))
 			list_move_tail(&nfsi->access_cache_inode_lru,
 					&nfs_access_lru_list);
@@ -1998,6 +2002,14 @@ remove_lru_entry:
 	}
 	spin_unlock(&nfs_access_lru_lock);
 	nfs_access_free_list(&head);
+	return freed;
+}
+
+long
+nfs_access_cache_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
+{
 	return vfs_pressure_ratio(atomic_long_read(&nfs_access_nr_entries));
 }
 
fs/nfs/internal.h +3 -1
index 91e59a3..9651e20 100644
@@ -269,7 +269,9 @@ extern struct nfs_client *nfs_init_client(struct nfs_client *clp,
 			   const char *ip_addr, rpc_authflavor_t authflavour);
 
 /* dir.c */
-extern int nfs_access_cache_shrinker(struct shrinker *shrink,
+extern long nfs_access_cache_count(struct shrinker *shrink,
+					struct shrink_control *sc);
+extern long nfs_access_cache_scan(struct shrinker *shrink,
 					struct shrink_control *sc);
 struct dentry *nfs_lookup(struct inode *, struct dentry *, unsigned int);
 int nfs_create(struct inode *, struct dentry *, umode_t, bool);
fs/nfs/super.c +2 -1
index a366107..2fed70f 100644
@@ -359,7 +359,8 @@ static void unregister_nfs4_fs(void)
 #endif
 
 static struct shrinker acl_shrinker = {
-	.shrink		= nfs_access_cache_shrinker,
+	.count_objects	= nfs_access_cache_count,
+	.scan_objects	= nfs_access_cache_scan,
 	.seeks		= DEFAULT_SEEKS,
 };
 
fs/nfsd/nfscache.c +22 -9
index e76244e..5564c38 100644
@@ -59,11 +59,14 @@ static unsigned int		longest_chain_cachesize;
 
 static int	nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *vec);
 static void	cache_cleaner_func(struct work_struct *unused);
-static int 	nfsd_reply_cache_shrink(struct shrinker *shrink,
-					struct shrink_control *sc);
+static long	nfsd_reply_cache_count(struct shrinker *shrink,
+				       struct shrink_control *sc);
+static long	nfsd_reply_cache_scan(struct shrinker *shrink,
+				      struct shrink_control *sc);
 
 static struct shrinker nfsd_reply_cache_shrinker = {
-	.shrink	= nfsd_reply_cache_shrink,
+	.scan_objects = nfsd_reply_cache_scan,
+	.count_objects = nfsd_reply_cache_count,
 	.seeks	= 1,
 };
 
@@ -232,16 +235,18 @@ nfsd_cache_entry_expired(struct svc_cacherep *rp)
  * Walk the LRU list and prune off entries that are older than RC_EXPIRE.
  * Also prune the oldest ones when the total exceeds the max number of entries.
  */
-static void
+static long
 prune_cache_entries(void)
 {
 	struct svc_cacherep *rp, *tmp;
+	long freed = 0;
 
 	list_for_each_entry_safe(rp, tmp, &lru_head, c_lru) {
 		if (!nfsd_cache_entry_expired(rp) &&
 		    num_drc_entries <= max_drc_entries)
 			break;
 		nfsd_reply_cache_free_locked(rp);
+		freed++;
 	}
 
 	/*
@@ -254,6 +259,7 @@ prune_cache_entries(void)
 		cancel_delayed_work(&cache_cleaner);
 	else
 		mod_delayed_work(system_wq, &cache_cleaner, RC_EXPIRE);
+	return freed;
 }
 
 static void
@@ -264,20 +270,27 @@ cache_cleaner_func(struct work_struct *unused)
 	spin_unlock(&cache_lock);
 }
 
-static int
-nfsd_reply_cache_shrink(struct shrinker *shrink, struct shrink_control *sc)
+static long
+nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)
 {
-	unsigned int num;
+	long num;
 
 	spin_lock(&cache_lock);
-	if (sc->nr_to_scan)
-		prune_cache_entries();
 	num = num_drc_entries;
 	spin_unlock(&cache_lock);
 
 	return num;
 }
 
+static long
+nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long freed;
+	spin_lock(&cache_lock);
+	freed = prune_cache_entries();
+	spin_unlock(&cache_lock);
+	return freed;
+}
 /*
  * Walk an xdr_buf and get a CRC for at most the first RC_CSUMLEN bytes
  */
fs/quota/dquot.c +16 -18
index 762b09c..fd6b762 100644
@@ -687,44 +687,42 @@ int dquot_quota_sync(struct super_block *sb, int type)
 }
 EXPORT_SYMBOL(dquot_quota_sync);
 
-/* Free unused dquots from cache */
-static void prune_dqcache(int count)
+static long
+dqcache_shrink_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	struct list_head *head;
 	struct dquot *dquot;
+	long freed = 0;
 
 	head = free_dquots.prev;
-	while (head != &free_dquots && count) {
+	while (head != &free_dquots && sc->nr_to_scan) {
 		dquot = list_entry(head, struct dquot, dq_free);
 		remove_dquot_hash(dquot);
 		remove_free_dquot(dquot);
 		remove_inuse(dquot);
 		do_destroy_dquot(dquot);
-		count--;
+		sc->nr_to_scan--;
+		freed++;
 		head = free_dquots.prev;
 	}
+	return freed;
 }
 
-/*
- * This is called from kswapd when we think we need some
- * more memory
- */
-static int shrink_dqcache_memory(struct shrinker *shrink,
-				 struct shrink_control *sc)
-{
-	int nr = sc->nr_to_scan;
+static long
+dqcache_shrink_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 
-	if (nr) {
-		spin_lock(&dq_list_lock);
-		prune_dqcache(nr);
-		spin_unlock(&dq_list_lock);
-	}
+{
 	return vfs_pressure_ratio(
 	percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));
 }
 
 static struct shrinker dqcache_shrinker = {
-	.shrink = shrink_dqcache_memory,
+	.count_objects = dqcache_shrink_count,
+	.scan_objects = dqcache_shrink_scan,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/ubifs/shrinker.c +12 -8
index 9e1d056..669d8c0 100644
@@ -277,19 +277,23 @@ static int kick_a_thread(void)
 	return 0;
 }
 
-int ubifs_shrinker(struct shrinker *shrink, struct shrink_control *sc)
+long ubifs_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long clean_zn_cnt = atomic_long_read(&ubifs_clean_zn_cnt);
+
+	/*
+	 * Due to the way UBIFS updates the clean znode counter it may
+	 * temporarily be negative.
+	 */
+	return clean_zn_cnt >= 0 ? clean_zn_cnt : 1;
+}
+
+long ubifs_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	int nr = sc->nr_to_scan;
 	int freed, contention = 0;
 	long clean_zn_cnt = atomic_long_read(&ubifs_clean_zn_cnt);
 
-	if (nr == 0)
-		/*
-		 * Due to the way UBIFS updates the clean znode counter it may
-		 * temporarily be negative.
-		 */
-		return clean_zn_cnt >= 0 ? clean_zn_cnt : 1;
-
 	if (!clean_zn_cnt) {
 		/*
 		 * No clean znodes, nothing to reap. All we can do in this case
fs/ubifs/super.c +2 -1
index f21acf0..ff357e0 100644
@@ -49,7 +49,8 @@ struct kmem_cache *ubifs_inode_slab;
 
 /* UBIFS TNC shrinker description */
 static struct shrinker ubifs_shrinker_info = {
-	.shrink = ubifs_shrinker,
+	.scan_objects = ubifs_shrink_scan,
+	.count_objects = ubifs_shrink_count,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/ubifs/ubifs.h +2 -1
index b2babce..bcdafcc 100644
@@ -1624,7 +1624,8 @@ int ubifs_tnc_start_commit(struct ubifs_info *c, struct ubifs_zbranch *zroot);
 int ubifs_tnc_end_commit(struct ubifs_info *c);
 
 /* shrinker.c */
-int ubifs_shrinker(struct shrinker *shrink, struct shrink_control *sc);
+long ubifs_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+long ubifs_shrink_count(struct shrinker *shrink, struct shrink_control *sc);
 
 /* commit.c */
 int ubifs_bg_thread(void *info);
Signature · 1 line
1.8.1.4
#36 lore raw

Re: [PATCH v6 15/31] fs: convert fs shrinkers to new scan/count API

Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Convert the filesystem shrinkers to use the new API, and standardise some of the behaviours of the shrinkers at the same time. For example, nr_to_scan means the number of objects to scan, not the number of objects to free.

Quoted 5 lines

I refactored the CIFS idmap shrinker a little - it really needs to be broken up into a shrinker per tree and keep an item count with the tree root so that we don't need to walk the tree every time the shrinker needs to count the number of objects in the tree (i.e. all the time under memory pressure).

UBIFS part looks OK.

Signature · 8 lines
Best Regards,
Artem Bityutskiy

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#37 lore raw

Re: [PATCH v6 15/31] fs: convert fs shrinkers to new scan/count API

Quoted 8 lines · Artem Bityutskiy
Quoted 7 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Convert the filesystem shrinkers to use the new API, and standardise some of the behaviours of the shrinkers at the same time. For example, nr_to_scan means the number of objects to scan, not the number of objects to free.

Quoted 8 lines
Quoted 5 lines

I refactored the CIFS idmap shrinker a little - it really needs to be broken up into a shrinker per tree and keep an item count with the tree root so that we don't need to walk the tree every time the shrinker needs to count the number of objects in the tree (i.e. all the time under memory pressure).

UBIFS part looks OK.

Great. Could I add your acked or reviewed by tag ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#38 lore raw

Re: [PATCH v6 15/31] fs: convert fs shrinkers to new scan/count API

Quoted 15 lines · Glauber Costa
Quoted 14 lines · Artem Bityutskiy
Quoted 12 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Convert the filesystem shrinkers to use the new API, and standardise some of the behaviours of the shrinkers at the same time. For example, nr_to_scan means the number of objects to scan, not the number of objects to free.

I refactored the CIFS idmap shrinker a little - it really needs to be broken up into a shrinker per tree and keep an item count with the tree root so that we don't need to walk the tree every time the shrinker needs to count the number of objects in the tree (i.e. all the time under memory pressure).

Quoted 3 lines
Quoted 2 lines

UBIFS part looks OK.

Great. Could I add your acked or reviewed by tag ?

Signature · 8 lines
Best Regards,
Artem Bityutskiy

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
reviewer Acked-by on v6 15/31
#39 lore raw

Re: [PATCH v6 15/31] fs: convert fs shrinkers to new scan/count API

In reply to Glauber Costa
Quoted 15 lines · Glauber Costa

From: Dave Chinner <[email protected]>

Convert the filesystem shrinkers to use the new API, and standardise some of the behaviours of the shrinkers at the same time. For example, nr_to_scan means the number of objects to scan, not the number of objects to free.

I refactored the CIFS idmap shrinker a little - it really needs to be broken up into a shrinker per tree and keep an item count with the tree root so that we don't need to walk the tree every time the shrinker needs to count the number of objects in the tree (i.e. all the time under memory pressure).

[ glommer: fixes for ext4, ubifs, nfs, cifs and glock. Fixes are
needed mainly due to new code merged in the tree ]

Quoted 7 lines
CC Steven Whitehouse <[email protected]>
CC Artem Bityutskiy <[email protected]>
CC Adrian Hunter <[email protected]>
CC Jan Kara <[email protected]>

for ext4, quota, and mbcache parts...

Honza

Quoted 613 lines
---
 fs/ext4/extents_status.c | 30 ++++++++++++++++------------
 fs/gfs2/glock.c          | 28 +++++++++++++++-----------
 fs/gfs2/main.c           |  3 ++-
 fs/gfs2/quota.c          | 12 +++++++-----
 fs/gfs2/quota.h          |  4 +++-
 fs/mbcache.c             | 51 ++++++++++++++++++++++++++++--------------------
 fs/nfs/dir.c             | 18 ++++++++++++++---
 fs/nfs/internal.h        |  4 +++-
 fs/nfs/super.c           |  3 ++-
 fs/nfsd/nfscache.c       | 31 ++++++++++++++++++++---------
 fs/quota/dquot.c         | 34 +++++++++++++++-----------------
 fs/ubifs/shrinker.c      | 20 +++++++++++--------
 fs/ubifs/super.c         |  3 ++-
 fs/ubifs/ubifs.h         |  3 ++-
 14 files changed, 151 insertions(+), 93 deletions(-)
fs/ext4/extents_status.c +18 -12
index e6941e6..4bce4f0 100644
@@ -878,20 +878,26 @@ int ext4_es_zeroout(struct inode *inode, struct ext4_extent *ex)
 				     EXTENT_STATUS_WRITTEN);
 }
 
-static int ext4_es_shrink(struct shrinker *shrink, struct shrink_control *sc)
+
+static long ext4_es_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long nr;
+	struct ext4_sb_info *sbi = container_of(shrink,
+					struct ext4_sb_info, s_es_shrinker);
+
+	nr = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
+	trace_ext4_es_shrink_enter(sbi->s_sb, sc->nr_to_scan, nr);
+	return nr;
+}
+
+static long ext4_es_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct ext4_sb_info *sbi = container_of(shrink,
 					struct ext4_sb_info, s_es_shrinker);
 	struct ext4_inode_info *ei;
 	struct list_head *cur, *tmp, scanned;
 	int nr_to_scan = sc->nr_to_scan;
-	int ret, nr_shrunk = 0;
-
-	ret = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
-	trace_ext4_es_shrink_enter(sbi->s_sb, nr_to_scan, ret);
-
-	if (!nr_to_scan)
-		return ret;
+	int ret = 0, nr_shrunk = 0;
 
 	INIT_LIST_HEAD(&scanned);
 
@@ -920,9 +926,8 @@ static int ext4_es_shrink(struct shrinker *shrink, struct shrink_control *sc)
 	list_splice_tail(&scanned, &sbi->s_es_lru);
 	spin_unlock(&sbi->s_es_lru_lock);
 
-	ret = percpu_counter_read_positive(&sbi->s_extent_cache_cnt);
 	trace_ext4_es_shrink_exit(sbi->s_sb, nr_shrunk, ret);
-	return ret;
+	return nr_shrunk;
 }
 
 void ext4_es_register_shrinker(struct super_block *sb)
@@ -932,7 +937,8 @@ void ext4_es_register_shrinker(struct super_block *sb)
 	sbi = EXT4_SB(sb);
 	INIT_LIST_HEAD(&sbi->s_es_lru);
 	spin_lock_init(&sbi->s_es_lru_lock);
-	sbi->s_es_shrinker.shrink = ext4_es_shrink;
+	sbi->s_es_shrinker.scan_objects = ext4_es_scan;
+	sbi->s_es_shrinker.count_objects = ext4_es_count;
 	sbi->s_es_shrinker.seeks = DEFAULT_SEEKS;
 	register_shrinker(&sbi->s_es_shrinker);
 }
@@ -973,7 +979,7 @@ static int __es_try_to_reclaim_extents(struct ext4_inode_info *ei,
 	struct ext4_es_tree *tree = &ei->i_es_tree;
 	struct rb_node *node;
 	struct extent_status *es;
-	int nr_shrunk = 0;
+	long nr_shrunk = 0;
 
 	if (ei->i_es_lru_nr == 0)
 		return 0;
fs/gfs2/glock.c +17 -11
index 3bd2748..4ddbccb 100644
@@ -1428,21 +1428,22 @@ __acquires(&lru_lock)
  * gfs2_dispose_glock_lru() above.
  */
 
-static void gfs2_scan_glock_lru(int nr)
+static long gfs2_scan_glock_lru(int nr)
 {
 	struct gfs2_glock *gl;
 	LIST_HEAD(skipped);
 	LIST_HEAD(dispose);
+	long freed = 0;
 
 	spin_lock(&lru_lock);
-	while(nr && !list_empty(&lru_list)) {
+	while ((nr-- >= 0) && !list_empty(&lru_list)) {
 		gl = list_entry(lru_list.next, struct gfs2_glock, gl_lru);
 
 		/* Test for being demotable */
 		if (!test_and_set_bit(GLF_LOCK, &gl->gl_flags)) {
 			list_move(&gl->gl_lru, &dispose);
 			atomic_dec(&lru_count);
-			nr--;
+			freed++;
 			continue;
 		}
 
@@ -1452,23 +1453,28 @@ static void gfs2_scan_glock_lru(int nr)
 	if (!list_empty(&dispose))
 		gfs2_dispose_glock_lru(&dispose);
 	spin_unlock(&lru_lock);
+
+	return freed;
 }
 
-static int gfs2_shrink_glock_memory(struct shrinker *shrink,
-				    struct shrink_control *sc)
+static long gfs2_glock_shrink_scan(struct shrinker *shrink,
+				   struct shrink_control *sc)
 {
-	if (sc->nr_to_scan) {
-		if (!(sc->gfp_mask & __GFP_FS))
-			return -1;
-		gfs2_scan_glock_lru(sc->nr_to_scan);
-	}
+	if (!(sc->gfp_mask & __GFP_FS))
+		return -1;
+	return gfs2_scan_glock_lru(sc->nr_to_scan);
+}
 
+static long gfs2_glock_shrink_count(struct shrinker *shrink,
+				    struct shrink_control *sc)
+{
 	return vfs_pressure_ratio(atomic_read(&lru_count));
 }
 
 static struct shrinker glock_shrinker = {
-	.shrink = gfs2_shrink_glock_memory,
 	.seeks = DEFAULT_SEEKS,
+	.count_objects = gfs2_glock_shrink_count,
+	.scan_objects = gfs2_glock_shrink_scan,
 };
 
 /**
fs/gfs2/main.c +2 -1
index e04d0e0..a105d84 100644
@@ -32,7 +32,8 @@
 struct workqueue_struct *gfs2_control_wq;
 
 static struct shrinker qd_shrinker = {
-	.shrink = gfs2_shrink_qd_memory,
+	.count_objects = gfs2_qd_shrink_count,
+	.scan_objects = gfs2_qd_shrink_scan,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/gfs2/quota.c +7 -5
index 5c14206..77b82f6 100644
@@ -75,14 +75,12 @@ static LIST_HEAD(qd_lru_list);
 static atomic_t qd_lru_count = ATOMIC_INIT(0);
 static DEFINE_SPINLOCK(qd_lru_lock);
 
-int gfs2_shrink_qd_memory(struct shrinker *shrink, struct shrink_control *sc)
+long gfs2_qd_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct gfs2_quota_data *qd;
 	struct gfs2_sbd *sdp;
 	int nr_to_scan = sc->nr_to_scan;
-
-	if (nr_to_scan == 0)
-		goto out;
+	long freed = 0;
 
 	if (!(sc->gfp_mask & __GFP_FS))
 		return -1;
@@ -110,10 +108,14 @@ int gfs2_shrink_qd_memory(struct shrinker *shrink, struct shrink_control *sc)
 		kmem_cache_free(gfs2_quotad_cachep, qd);
 		spin_lock(&qd_lru_lock);
 		nr_to_scan--;
+		freed++;
 	}
 	spin_unlock(&qd_lru_lock);
+	return freed;
+}
 
-out:
+long gfs2_qd_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
 	return vfs_pressure_ratio(atomic_read(&qd_lru_count));
 }
 
fs/gfs2/quota.h +3 -1
index 4f5e6e4..4f61708 100644
@@ -53,7 +53,9 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip)
 	return ret;
 }
 
-extern int gfs2_shrink_qd_memory(struct shrinker *shrink,
+extern long gfs2_qd_shrink_count(struct shrinker *shrink,
+				 struct shrink_control *sc);
+extern long gfs2_qd_shrink_scan(struct shrinker *shrink,
 				 struct shrink_control *sc);
 extern const struct quotactl_ops gfs2_quotactl_ops;
 
fs/mbcache.c +30 -21
index 5eb0476..009a463 100644
@@ -86,18 +86,6 @@ static LIST_HEAD(mb_cache_list);
 static LIST_HEAD(mb_cache_lru_list);
 static DEFINE_SPINLOCK(mb_cache_spinlock);
 
-/*
- * What the mbcache registers as to get shrunk dynamically.
- */
-
-static int mb_cache_shrink_fn(struct shrinker *shrink,
-			      struct shrink_control *sc);
-
-static struct shrinker mb_cache_shrinker = {
-	.shrink = mb_cache_shrink_fn,
-	.seeks = DEFAULT_SEEKS,
-};
-
 static inline int
 __mb_cache_entry_is_hashed(struct mb_cache_entry *ce)
 {
@@ -151,7 +139,7 @@ forget:
 
 
 /*
- * mb_cache_shrink_fn()  memory pressure callback
+ * mb_cache_shrink_scan()  memory pressure callback
  *
  * This function is called by the kernel memory management when memory
  * gets low.
@@ -159,17 +147,18 @@ forget:
  * @shrink: (ignored)
  * @sc: shrink_control passed from reclaim
  *
- * Returns the number of objects which are present in the cache.
+ * Returns the number of objects freed.
  */
-static int
-mb_cache_shrink_fn(struct shrinker *shrink, struct shrink_control *sc)
+static long
+mb_cache_shrink_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	LIST_HEAD(free_list);
-	struct mb_cache *cache;
 	struct mb_cache_entry *entry, *tmp;
-	int count = 0;
 	int nr_to_scan = sc->nr_to_scan;
 	gfp_t gfp_mask = sc->gfp_mask;
+	long freed = 0;
 
 	mb_debug("trying to free %d entries", nr_to_scan);
 	spin_lock(&mb_cache_spinlock);
@@ -179,19 +168,39 @@ mb_cache_shrink_fn(struct shrinker *shrink, struct shrink_control *sc)
 				   struct mb_cache_entry, e_lru_list);
 		list_move_tail(&ce->e_lru_list, &free_list);
 		__mb_cache_entry_unhash(ce);
+		freed++;
+	}
+	spin_unlock(&mb_cache_spinlock);
+	list_for_each_entry_safe(entry, tmp, &free_list, e_lru_list) {
+		__mb_cache_entry_forget(entry, gfp_mask);
 	}
+	return freed;
+}
+
+static long
+mb_cache_shrink_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
+{
+	struct mb_cache *cache;
+	long count = 0;
+
+	spin_lock(&mb_cache_spinlock);
 	list_for_each_entry(cache, &mb_cache_list, c_cache_list) {
 		mb_debug("cache %s (%d)", cache->c_name,
 			  atomic_read(&cache->c_entry_count));
 		count += atomic_read(&cache->c_entry_count);
 	}
 	spin_unlock(&mb_cache_spinlock);
-	list_for_each_entry_safe(entry, tmp, &free_list, e_lru_list) {
-		__mb_cache_entry_forget(entry, gfp_mask);
-	}
+
 	return vfs_pressure_ratio(count);
 }
 
+static struct shrinker mb_cache_shrinker = {
+	.count_objects = mb_cache_shrink_count,
+	.scan_objects = mb_cache_shrink_scan,
+	.seeks = DEFAULT_SEEKS,
+};
 
 /*
  * mb_cache_create()  create a new cache
fs/nfs/dir.c +15 -3
index 54d7c47..b355905 100644
@@ -1958,17 +1958,20 @@ static void nfs_access_free_list(struct list_head *head)
 	}
 }
 
-int nfs_access_cache_shrinker(struct shrinker *shrink,
-			      struct shrink_control *sc)
+long
+nfs_access_cache_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	LIST_HEAD(head);
 	struct nfs_inode *nfsi, *next;
 	struct nfs_access_entry *cache;
 	int nr_to_scan = sc->nr_to_scan;
 	gfp_t gfp_mask = sc->gfp_mask;
+	long freed = 0;
 
 	if ((gfp_mask & GFP_KERNEL) != GFP_KERNEL)
-		return (nr_to_scan == 0) ? 0 : -1;
+		return -1;
 
 	spin_lock(&nfs_access_lru_lock);
 	list_for_each_entry_safe(nfsi, next, &nfs_access_lru_list, access_cache_inode_lru) {
@@ -1984,6 +1987,7 @@ int nfs_access_cache_shrinker(struct shrinker *shrink,
 				struct nfs_access_entry, lru);
 		list_move(&cache->lru, &head);
 		rb_erase(&cache->rb_node, &nfsi->access_cache);
+		freed++;
 		if (!list_empty(&nfsi->access_cache_entry_lru))
 			list_move_tail(&nfsi->access_cache_inode_lru,
 					&nfs_access_lru_list);
@@ -1998,6 +2002,14 @@ remove_lru_entry:
 	}
 	spin_unlock(&nfs_access_lru_lock);
 	nfs_access_free_list(&head);
+	return freed;
+}
+
+long
+nfs_access_cache_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
+{
 	return vfs_pressure_ratio(atomic_long_read(&nfs_access_nr_entries));
 }
 
fs/nfs/internal.h +3 -1
index 91e59a3..9651e20 100644
@@ -269,7 +269,9 @@ extern struct nfs_client *nfs_init_client(struct nfs_client *clp,
 			   const char *ip_addr, rpc_authflavor_t authflavour);
 
 /* dir.c */
-extern int nfs_access_cache_shrinker(struct shrinker *shrink,
+extern long nfs_access_cache_count(struct shrinker *shrink,
+					struct shrink_control *sc);
+extern long nfs_access_cache_scan(struct shrinker *shrink,
 					struct shrink_control *sc);
 struct dentry *nfs_lookup(struct inode *, struct dentry *, unsigned int);
 int nfs_create(struct inode *, struct dentry *, umode_t, bool);
fs/nfs/super.c +2 -1
index a366107..2fed70f 100644
@@ -359,7 +359,8 @@ static void unregister_nfs4_fs(void)
 #endif
 
 static struct shrinker acl_shrinker = {
-	.shrink		= nfs_access_cache_shrinker,
+	.count_objects	= nfs_access_cache_count,
+	.scan_objects	= nfs_access_cache_scan,
 	.seeks		= DEFAULT_SEEKS,
 };
 
fs/nfsd/nfscache.c +22 -9
index e76244e..5564c38 100644
@@ -59,11 +59,14 @@ static unsigned int		longest_chain_cachesize;
 
 static int	nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *vec);
 static void	cache_cleaner_func(struct work_struct *unused);
-static int 	nfsd_reply_cache_shrink(struct shrinker *shrink,
-					struct shrink_control *sc);
+static long	nfsd_reply_cache_count(struct shrinker *shrink,
+				       struct shrink_control *sc);
+static long	nfsd_reply_cache_scan(struct shrinker *shrink,
+				      struct shrink_control *sc);
 
 static struct shrinker nfsd_reply_cache_shrinker = {
-	.shrink	= nfsd_reply_cache_shrink,
+	.scan_objects = nfsd_reply_cache_scan,
+	.count_objects = nfsd_reply_cache_count,
 	.seeks	= 1,
 };
 
@@ -232,16 +235,18 @@ nfsd_cache_entry_expired(struct svc_cacherep *rp)
  * Walk the LRU list and prune off entries that are older than RC_EXPIRE.
  * Also prune the oldest ones when the total exceeds the max number of entries.
  */
-static void
+static long
 prune_cache_entries(void)
 {
 	struct svc_cacherep *rp, *tmp;
+	long freed = 0;
 
 	list_for_each_entry_safe(rp, tmp, &lru_head, c_lru) {
 		if (!nfsd_cache_entry_expired(rp) &&
 		    num_drc_entries <= max_drc_entries)
 			break;
 		nfsd_reply_cache_free_locked(rp);
+		freed++;
 	}
 
 	/*
@@ -254,6 +259,7 @@ prune_cache_entries(void)
 		cancel_delayed_work(&cache_cleaner);
 	else
 		mod_delayed_work(system_wq, &cache_cleaner, RC_EXPIRE);
+	return freed;
 }
 
 static void
@@ -264,20 +270,27 @@ cache_cleaner_func(struct work_struct *unused)
 	spin_unlock(&cache_lock);
 }
 
-static int
-nfsd_reply_cache_shrink(struct shrinker *shrink, struct shrink_control *sc)
+static long
+nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)
 {
-	unsigned int num;
+	long num;
 
 	spin_lock(&cache_lock);
-	if (sc->nr_to_scan)
-		prune_cache_entries();
 	num = num_drc_entries;
 	spin_unlock(&cache_lock);
 
 	return num;
 }
 
+static long
+nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long freed;
+	spin_lock(&cache_lock);
+	freed = prune_cache_entries();
+	spin_unlock(&cache_lock);
+	return freed;
+}
 /*
  * Walk an xdr_buf and get a CRC for at most the first RC_CSUMLEN bytes
  */
fs/quota/dquot.c +16 -18
index 762b09c..fd6b762 100644
@@ -687,44 +687,42 @@ int dquot_quota_sync(struct super_block *sb, int type)
 }
 EXPORT_SYMBOL(dquot_quota_sync);
 
-/* Free unused dquots from cache */
-static void prune_dqcache(int count)
+static long
+dqcache_shrink_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	struct list_head *head;
 	struct dquot *dquot;
+	long freed = 0;
 
 	head = free_dquots.prev;
-	while (head != &free_dquots && count) {
+	while (head != &free_dquots && sc->nr_to_scan) {
 		dquot = list_entry(head, struct dquot, dq_free);
 		remove_dquot_hash(dquot);
 		remove_free_dquot(dquot);
 		remove_inuse(dquot);
 		do_destroy_dquot(dquot);
-		count--;
+		sc->nr_to_scan--;
+		freed++;
 		head = free_dquots.prev;
 	}
+	return freed;
 }
 
-/*
- * This is called from kswapd when we think we need some
- * more memory
- */
-static int shrink_dqcache_memory(struct shrinker *shrink,
-				 struct shrink_control *sc)
-{
-	int nr = sc->nr_to_scan;
+static long
+dqcache_shrink_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 
-	if (nr) {
-		spin_lock(&dq_list_lock);
-		prune_dqcache(nr);
-		spin_unlock(&dq_list_lock);
-	}
+{
 	return vfs_pressure_ratio(
 	percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS]));
 }
 
 static struct shrinker dqcache_shrinker = {
-	.shrink = shrink_dqcache_memory,
+	.count_objects = dqcache_shrink_count,
+	.scan_objects = dqcache_shrink_scan,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/ubifs/shrinker.c +12 -8
index 9e1d056..669d8c0 100644
@@ -277,19 +277,23 @@ static int kick_a_thread(void)
 	return 0;
 }
 
-int ubifs_shrinker(struct shrinker *shrink, struct shrink_control *sc)
+long ubifs_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	long clean_zn_cnt = atomic_long_read(&ubifs_clean_zn_cnt);
+
+	/*
+	 * Due to the way UBIFS updates the clean znode counter it may
+	 * temporarily be negative.
+	 */
+	return clean_zn_cnt >= 0 ? clean_zn_cnt : 1;
+}
+
+long ubifs_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	int nr = sc->nr_to_scan;
 	int freed, contention = 0;
 	long clean_zn_cnt = atomic_long_read(&ubifs_clean_zn_cnt);
 
-	if (nr == 0)
-		/*
-		 * Due to the way UBIFS updates the clean znode counter it may
-		 * temporarily be negative.
-		 */
-		return clean_zn_cnt >= 0 ? clean_zn_cnt : 1;
-
 	if (!clean_zn_cnt) {
 		/*
 		 * No clean znodes, nothing to reap. All we can do in this case
fs/ubifs/super.c +2 -1
index f21acf0..ff357e0 100644
@@ -49,7 +49,8 @@ struct kmem_cache *ubifs_inode_slab;
 
 /* UBIFS TNC shrinker description */
 static struct shrinker ubifs_shrinker_info = {
-	.shrink = ubifs_shrinker,
+	.scan_objects = ubifs_shrink_scan,
+	.count_objects = ubifs_shrink_count,
 	.seeks = DEFAULT_SEEKS,
 };
 
fs/ubifs/ubifs.h +2 -1
index b2babce..bcdafcc 100644
@@ -1624,7 +1624,8 @@ int ubifs_tnc_start_commit(struct ubifs_info *c, struct ubifs_zbranch *zroot);
 int ubifs_tnc_end_commit(struct ubifs_info *c);
 
 /* shrinker.c */
-int ubifs_shrinker(struct shrinker *shrink, struct shrink_control *sc);
+long ubifs_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+long ubifs_shrink_count(struct shrinker *shrink, struct shrink_control *sc);
 
 /* commit.c */
 int ubifs_bg_thread(void *info);
Signature · 1 line
1.8.1.4
Signature · 8 lines
Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#40 lore raw

PATCH v6 16/31 drivers: convert shrinkers to new count/scan API

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert the driver shrinkers to the new API. Most changes are compile tested only because I either don't have the hardware or it's staging stuff.

FWIW, the md and android code is pretty good, but the rest of it makes me want to claw my eyes out. The amount of broken code I just encountered is mind boggling. I've added comments explaining what is broken, but I fear that some of the code would be best dealt with by being dragged behind the bike shed, burying in mud up to it's neck and then run over repeatedly with a blunt lawn mower.

Special mention goes to the zcache/zcache2 drivers. They can't co-exist in the build at the same time, they are under different menu options in menuconfig, they only show up when you've got the right set of mm subsystem options configured and so even compile testing is an exercise in pulling teeth. And that doesn't even take into account the horrible, broken code...

[ glommer: fixes for i915, android lowmem, zcache, bcache ]
CC Daniel Vetter <[email protected]>
CC Kent Overstreet <[email protected]>
CC Arve Hjønnevåg <[email protected]>
CC John Stultz <[email protected]>
CC David Rientjes <[email protected]>
CC Jerome Glisse <[email protected]>
CC Thomas Hellstrom <[email protected]>
---
 drivers/gpu/drm/i915/i915_dma.c           |  4 +-
 drivers/gpu/drm/i915/i915_gem.c           | 67 ++++++++++++++++++++++---------
 drivers/gpu/drm/ttm/ttm_page_alloc.c      | 48 ++++++++++++++--------
 drivers/gpu/drm/ttm/ttm_page_alloc_dma.c  | 55 ++++++++++++++++---------
 drivers/md/bcache/btree.c                 | 43 ++++++++++----------
 drivers/md/bcache/sysfs.c                 |  2 +-
 drivers/md/dm-bufio.c                     | 65 +++++++++++++++++++-----------
 drivers/staging/android/ashmem.c          | 45 ++++++++++++++-------
 drivers/staging/android/lowmemorykiller.c | 40 ++++++++++--------
 drivers/staging/zcache/zcache-main.c      | 29 ++++++++-----
 10 files changed, 256 insertions(+), 142 deletions(-)
drivers/gpu/drm/i915/i915_dma.c +2 -2
index 3b315ba..cfe3373 100644
@@ -1669,7 +1669,7 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 	return 0;
 
 out_gem_unload:
-	if (dev_priv->mm.inactive_shrinker.shrink)
+	if (dev_priv->mm.inactive_shrinker.scan_objects)
 		unregister_shrinker(&dev_priv->mm.inactive_shrinker);
 
 	if (dev->pdev->msi_enabled)
@@ -1705,7 +1705,7 @@ int i915_driver_unload(struct drm_device *dev)
 
 	i915_teardown_sysfs(dev);
 
-	if (dev_priv->mm.inactive_shrinker.shrink)
+	if (dev_priv->mm.inactive_shrinker.scan_objects)
 		unregister_shrinker(&dev_priv->mm.inactive_shrinker);
 
 	mutex_lock(&dev->struct_mutex);
drivers/gpu/drm/i915/i915_gem.c +49 -18

116 lines not on this page · raw message

drivers/gpu/drm/ttm/ttm_page_alloc.c +32 -16

74 lines not on this page · raw message

drivers/gpu/drm/ttm/ttm_page_alloc_dma.c +36 -19

91 lines not on this page · raw message

drivers/md/bcache/btree.c +22 -21

88 lines not on this page · raw message

drivers/md/bcache/sysfs.c +1 -1
index 4d9cca4..fa8d048 100644
@@ -535,7 +535,7 @@ STORE(__bch_cache_set)
 		struct shrink_control sc;
 		sc.gfp_mask = GFP_KERNEL;
 		sc.nr_to_scan = strtoul_or_return(buf);
-		c->shrink.shrink(&c->shrink, &sc);
+		c->shrink.scan_objects(&c->shrink, &sc);
 	}
 
 	sysfs_strtoul(congested_read_threshold_us,
drivers/md/dm-bufio.c +42 -23

118 lines not on this page · raw message

drivers/staging/android/ashmem.c +30 -15

88 lines not on this page · raw message

drivers/staging/android/lowmemorykiller.c +24 -16
index fe74494..d23bfea 100644
@@ -66,7 +66,15 @@ static unsigned long lowmem_deathpending_timeout;
 			pr_info(x);			\
 	} while (0)
 
-static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
+static long lowmem_count(struct shrinker *s, struct shrink_control *sc)
+{
+	return global_page_state(NR_ACTIVE_ANON) +
+		global_page_state(NR_ACTIVE_FILE) +
+		global_page_state(NR_INACTIVE_ANON) +
+		global_page_state(NR_INACTIVE_FILE);
+}
+
+static long lowmem_scan(struct shrinker *s, struct shrink_control *sc)
 {
 	struct task_struct *tsk;
 	struct task_struct *selected = NULL;
@@ -92,19 +100,17 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 			break;
 		}
 	}
-	if (sc->nr_to_scan > 0)
-		lowmem_print(3, "lowmem_shrink %lu, %x, ofree %d %d, ma %hd\n",
-				sc->nr_to_scan, sc->gfp_mask, other_free,
-				other_file, min_score_adj);
-	rem = global_page_state(NR_ACTIVE_ANON) +
-		global_page_state(NR_ACTIVE_FILE) +
-		global_page_state(NR_INACTIVE_ANON) +
-		global_page_state(NR_INACTIVE_FILE);
-	if (sc->nr_to_scan <= 0 || min_score_adj == OOM_SCORE_ADJ_MAX + 1) {
-		lowmem_print(5, "lowmem_shrink %lu, %x, return %d\n",
-			     sc->nr_to_scan, sc->gfp_mask, rem);
-		return rem;
+
+	lowmem_print(3, "lowmem_scan %lu, %x, ofree %d %d, ma %hd\n",
+			sc->nr_to_scan, sc->gfp_mask, other_free,
+			other_file, min_score_adj);
+
+	if (min_score_adj == OOM_SCORE_ADJ_MAX + 1) {
+		lowmem_print(5, "lowmem_scan %lu, %x, return 0\n",
+			     sc->nr_to_scan, sc->gfp_mask);
+		return 0;
 	}
+
 	selected_oom_score_adj = min_score_adj;
 
 	rcu_read_lock();
@@ -154,16 +160,18 @@ static int lowmem_shrink(struct shrinker *s, struct shrink_control *sc)
 		lowmem_deathpending_timeout = jiffies + HZ;
 		send_sig(SIGKILL, selected, 0);
 		set_tsk_thread_flag(selected, TIF_MEMDIE);
-		rem -= selected_tasksize;
+		rem += selected_tasksize;
 	}
-	lowmem_print(4, "lowmem_shrink %lu, %x, return %d\n",
+
+	lowmem_print(4, "lowmem_scan %lu, %x, return %d\n",
 		     sc->nr_to_scan, sc->gfp_mask, rem);
 	rcu_read_unlock();
 	return rem;
 }
 
 static struct shrinker lowmem_shrinker = {
-	.shrink = lowmem_shrink,
+	.scan_objects = lowmem_scan,
+	.count_objects = lowmem_count,
 	.seeks = DEFAULT_SEEKS * 16
 };
 
drivers/staging/zcache/zcache-main.c +18 -11
index 522cb8e..bbfcd4f 100644
@@ -1140,23 +1140,19 @@ static bool zcache_freeze;
  * pageframes in use.  FIXME POLICY: Probably the writeback should only occur
  * if the eviction doesn't free enough pages.
  */
-static int shrink_zcache_memory(struct shrinker *shrink,
-				struct shrink_control *sc)
+static long scan_zcache_memory(struct shrinker *shrink,
+			       struct shrink_control *sc)
 {
 	static bool in_progress;
-	int ret = -1;
-	int nr = sc->nr_to_scan;
 	int nr_evict = 0;
 	int nr_writeback = 0;
 	struct page *page;
 	int  file_pageframes_inuse, anon_pageframes_inuse;
-
-	if (nr <= 0)
-		goto skip_evict;
+	long freed = 0;
 
 	/* don't allow more than one eviction thread at a time */
 	if (in_progress)
-		goto skip_evict;
+		return 0;
 
 	in_progress = true;
 
@@ -1176,6 +1172,7 @@ static int shrink_zcache_memory(struct shrinker *shrink,
 		if (page == NULL)
 			break;
 		zcache_free_page(page);
+		freed++;
 	}
 
 	zcache_last_active_anon_pageframes =
@@ -1192,13 +1189,22 @@ static int shrink_zcache_memory(struct shrinker *shrink,
 #ifdef CONFIG_ZCACHE_WRITEBACK
 		int writeback_ret;
 		writeback_ret = zcache_frontswap_writeback();
-		if (writeback_ret == -ENOMEM)
+		if (writeback_ret != -ENOMEM)
+			freed++;
+		else
 #endif
 			break;
 	}
 	in_progress = false;
 
-skip_evict:
+	return freed;
+}
+
+static long count_zcache_memory(struct shrinker *shrink,
+				struct shrink_control *sc)
+{
+	int ret = -1;
+
 	/* resample: has changed, but maybe not all the way yet */
 	zcache_last_active_file_pageframes =
 		global_page_state(NR_LRU_BASE + LRU_ACTIVE_FILE);
@@ -1212,7 +1218,8 @@ skip_evict:
 }
 
 static struct shrinker zcache_shrinker = {
-	.shrink = shrink_zcache_memory,
+	.scan_objects = scan_zcache_memory,
+	.count_objects = count_zcache_memory,
 	.seeks = DEFAULT_SEEKS,
 };
 
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#41 lore raw

PATCH v6 17/31 i915: bail out earlier when shrinker cannot acquire mutex

In reply to Glauber Costa

The main shrinker driver will keep trying for a while to free objects if the returned value from the shrink scan procedure is 0. That means "no objects now", but a retry could very well succeed.

A negative value has a different meaning. It means it is impossible to shrink, and we would better bail out soon. We find this behavior more appropriate for the case where the lock cannot be taken. Specially given the hammer behavior of the i915: if another thread is already shrinking, we are likely not to be able to shrink anything anyway when we finally acquire the mutex.

CC Dave Chinner <[email protected]>
CC Mel Gorman <[email protected]>
CC Daniel Vetter <[email protected]>
CC Kent Overstreet <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +2 -2
index 6b17122..52b3ac1 100644
@@ -4448,10 +4448,10 @@ i915_gem_inactive_count(struct shrinker *shrinker, struct shrink_control *sc)
 
 	if (!mutex_trylock(&dev->struct_mutex)) {
 		if (!mutex_is_locked_by(&dev->struct_mutex, current))
-			return 0;
+			return -1;
 
 		if (dev_priv->mm.shrinker_no_lock_stealing)
-			return 0;
+			return -1;
 
 		unlock = false;
 	}
Signature · 1 line
1.8.1.4
#42 lore raw

PATCH v6 18/31 shrinker: convert remaining shrinkers to count/scan API

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

Convert the remaining couple of random shrinkers in the tree to the new API.

CC Marcelo Tosatti <[email protected]>
CC Gleb Natapov <[email protected]>
CC Chuck Lever <[email protected]>
CC J. Bruce Fields <[email protected]>
CC Trond Myklebust <[email protected]>
---
 arch/x86/kvm/mmu.c | 28 +++++++++++++++++++++-------
 net/sunrpc/auth.c  | 45 +++++++++++++++++++++++++++++++--------------
 2 files changed, 52 insertions(+), 21 deletions(-)
arch/x86/kvm/mmu.c +21 -7
index 004cc87..19b4edf 100644
@@ -4212,13 +4212,14 @@ restart:
 	spin_unlock(&kvm->mmu_lock);
 }
 
-static int mmu_shrink(struct shrinker *shrink, struct shrink_control *sc)
+static long
+mmu_shrink_scan(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
 {
 	struct kvm *kvm;
 	int nr_to_scan = sc->nr_to_scan;
-
-	if (nr_to_scan == 0)
-		goto out;
+	long freed = 0;
 
 	raw_spin_lock(&kvm_lock);
 
@@ -4246,24 +4247,37 @@ static int mmu_shrink(struct shrinker *shrink, struct shrink_control *sc)
 		idx = srcu_read_lock(&kvm->srcu);
 		spin_lock(&kvm->mmu_lock);
 
-		prepare_zap_oldest_mmu_page(kvm, &invalid_list);
+		freed += prepare_zap_oldest_mmu_page(kvm, &invalid_list);
 		kvm_mmu_commit_zap_page(kvm, &invalid_list);
 
 		spin_unlock(&kvm->mmu_lock);
 		srcu_read_unlock(&kvm->srcu, idx);
 
+		/*
+		 * unfair on small ones
+		 * per-vm shrinkers cry out
+		 * sadness comes quickly
+		 */
 		list_move_tail(&kvm->vm_list, &vm_list);
 		break;
 	}
 
 	raw_spin_unlock(&kvm_lock);
+	return freed;
 
-out:
+}
+
+static long
+mmu_shrink_count(
+	struct shrinker		*shrink,
+	struct shrink_control	*sc)
+{
 	return percpu_counter_read_positive(&kvm_total_used_mmu_pages);
 }
 
 static struct shrinker mmu_shrinker = {
-	.shrink = mmu_shrink,
+	.count_objects = mmu_shrink_count,
+	.scan_objects = mmu_shrink_scan,
 	.seeks = DEFAULT_SEEKS * 10,
 };
 
net/sunrpc/auth.c +31 -14

89 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#43 lore raw

PATCH v6 19/31 hugepage: convert huge zero page shrinker to new shrinker API

In reply to Glauber Costa

It consists of:

* returning long instead of int
* separating count from scan
* returning the number of freed entities in scan

---
 mm/huge_memory.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
mm/huge_memory.c +11 -6
index c5f2fa1..3a0eda8 100644
@@ -211,24 +211,29 @@ static void put_huge_zero_page(void)
 	BUG_ON(atomic_dec_and_test(&huge_zero_refcount));
 }
 
-static int shrink_huge_zero_page(struct shrinker *shrink,
-		struct shrink_control *sc)
+static long shrink_huge_zero_page_count(struct shrinker *shrink,
+					struct shrink_control *sc)
 {
-	if (!sc->nr_to_scan)
-		/* we can free zero page only if last reference remains */
-		return atomic_read(&huge_zero_refcount) == 1 ? HPAGE_PMD_NR : 0;
+	/* we can free zero page only if last reference remains */
+	return atomic_read(&huge_zero_refcount) == 1 ? HPAGE_PMD_NR : 0;
+}
 
+static long shrink_huge_zero_page_scan(struct shrinker *shrink,
+				       struct shrink_control *sc)
+{
 	if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) {
 		struct page *zero_page = xchg(&huge_zero_page, NULL);
 		BUG_ON(zero_page == NULL);
 		__free_page(zero_page);
+		return HPAGE_PMD_NR;
 	}
 
 	return 0;
 }
 
 static struct shrinker huge_zero_page_shrinker = {
-	.shrink = shrink_huge_zero_page,
+	.count_objects = shrink_huge_zero_page_count,
+	.scan_objects = shrink_huge_zero_page_scan,
 	.seeks = DEFAULT_SEEKS,
 };
 
Signature · 1 line
1.8.1.4
#44 lore raw

PATCH v6 20/31 shrinker: Kill old ->shrink API.

In reply to Glauber Costa

From: Dave Chinner <[email protected]>

There are no more users of this API, so kill it dead, dead, dead and quietly bury the corpse in a shallow, unmarked grave in a dark forest deep in the hills...

[ glommer: added flowers to the grave ]
---
 include/linux/shrinker.h      | 15 +++++----------
 include/trace/events/vmscan.h |  4 ++--
 mm/vmscan.c                   | 38 ++++++++------------------------------
 3 files changed, 15 insertions(+), 42 deletions(-)
include/linux/shrinker.h +5 -10
index 98be3ab..00a3e57 100644
@@ -7,14 +7,15 @@
  *
  * The 'gfpmask' refers to the allocation we are currently trying to
  * fulfil.
- *
- * Note that 'shrink' will be passed nr_to_scan == 0 when the VM is
- * querying the cache size, so a fastpath for that case is appropriate.
  */
 struct shrink_control {
 	gfp_t gfp_mask;
 
-	/* How many slab objects shrinker() should scan and try to reclaim */
+	/*
+	 * How many objects scan_objects should scan and try to reclaim.
+	 * This is reset before every call, so it is safe for callees
+	 * to modify.
+	 */
 	long nr_to_scan;
 
 	/* shrink from these nodes */
@@ -24,11 +25,6 @@ struct shrink_control {
 /*
  * A callback you can register to apply pressure to ageable caches.
  *
- * @shrink() should look through the least-recently-used 'nr_to_scan' entries
- * and attempt to free them up.  It should return the number of objects which
- * remain in the cache.  If it returns -1, it means it cannot do any scanning at
- * this time (eg. there is a risk of deadlock).
- *
  * @count_objects should return the number of freeable items in the cache. If
  * there are no objects to free or the number of freeable items cannot be
  * determined, it should return 0. No deadlock checks should be done during the
@@ -44,7 +40,6 @@ struct shrink_control {
  * @scan_objects will be made from the current reclaim context.
  */
 struct shrinker {
-	int (*shrink)(struct shrinker *, struct shrink_control *sc);
 	long (*count_objects)(struct shrinker *, struct shrink_control *sc);
 	long (*scan_objects)(struct shrinker *, struct shrink_control *sc);
 
include/trace/events/vmscan.h +2 -2
index 63cfccc..132a985 100644
@@ -202,7 +202,7 @@ TRACE_EVENT(mm_shrink_slab_start,
 
 	TP_fast_assign(
 		__entry->shr = shr;
-		__entry->shrink = shr->shrink;
+		__entry->shrink = shr->scan_objects;
 		__entry->nr_objects_to_shrink = nr_objects_to_shrink;
 		__entry->gfp_flags = sc->gfp_mask;
 		__entry->pgs_scanned = pgs_scanned;
@@ -241,7 +241,7 @@ TRACE_EVENT(mm_shrink_slab_end,
 
 	TP_fast_assign(
 		__entry->shr = shr;
-		__entry->shrink = shr->shrink;
+		__entry->shrink = shr->scan_objects;
 		__entry->unused_scan = unused_scan_cnt;
 		__entry->new_scan = new_scan_cnt;
 		__entry->retval = shrinker_retval;
mm/vmscan.c +8 -30
index 35a6a9b..64f66f6 100644
@@ -177,14 +177,6 @@ void unregister_shrinker(struct shrinker *shrinker)
 }
 EXPORT_SYMBOL(unregister_shrinker);
 
-static inline int do_shrinker_shrink(struct shrinker *shrinker,
-				     struct shrink_control *sc,
-				     unsigned long nr_to_scan)
-{
-	sc->nr_to_scan = nr_to_scan;
-	return (*shrinker->shrink)(shrinker, sc);
-}
-
 #define SHRINK_BATCH 128
 /*
  * Call the shrink functions to age shrinkable caches
@@ -230,11 +222,8 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 		long batch_size = shrinker->batch ? shrinker->batch
 						  : SHRINK_BATCH;
 
-		if (shrinker->scan_objects) {
-			max_pass = shrinker->count_objects(shrinker, shrinkctl);
-			WARN_ON(max_pass < 0);
-		} else
-			max_pass = do_shrinker_shrink(shrinker, shrinkctl, 0);
+		max_pass = shrinker->count_objects(shrinker, shrinkctl);
+		WARN_ON(max_pass < 0);
 		if (max_pass <= 0)
 			continue;
 
@@ -253,7 +242,7 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 		if (total_scan < 0) {
 			printk(KERN_ERR
 			"shrink_slab: %pF negative objects to delete nr=%ld\n",
-			       shrinker->shrink, total_scan);
+			       shrinker->scan_objects, total_scan);
 			total_scan = max_pass;
 		}
 
@@ -287,23 +276,12 @@ unsigned long shrink_slab(struct shrink_control *shrinkctl,
 		while (total_scan >= batch_size) {
 			long ret;
 
-			if (shrinker->scan_objects) {
-				shrinkctl->nr_to_scan = batch_size;
-				ret = shrinker->scan_objects(shrinker, shrinkctl);
+			shrinkctl->nr_to_scan = batch_size;
+			ret = shrinker->scan_objects(shrinker, shrinkctl);
 
-				if (ret == -1)
-					break;
-				freed += ret;
-			} else {
-				int nr_before;
-				nr_before = do_shrinker_shrink(shrinker, shrinkctl, 0);
-				ret = do_shrinker_shrink(shrinker, shrinkctl,
-								batch_size);
-				if (ret == -1)
-					break;
-				if (ret < nr_before)
-					freed += nr_before - ret;
-			}
+			if (ret == -1)
+				break;
+			freed += ret;
 
 			count_vm_events(SLABS_SCANNED, batch_size);
 			total_scan -= batch_size;
Signature · 1 line
1.8.1.4
#45 lore raw

PATCH v6 21/31 vmscan: also shrink slab in memcg pressure

In reply to Glauber Costa

Without the surrounding infrastructure, this patch is a bit of a hammer:
it will basically shrink objects from all memcgs under memcg pressure. At least, however, we will keep the scan limited to the shrinkers marked as per-memcg.

Future patches will implement the in-shrinker logic to filter objects based on its memcg association.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/memcontrol.h | 17 +++++++++++++++++
 include/linux/shrinker.h   |  4 ++++
 mm/memcontrol.c            | 16 +++++++++++++++-
 mm/vmscan.c                | 46 +++++++++++++++++++++++++++++++++++++++++++---
 4 files changed, 79 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +17 -0
index 7b4d9d7..489c6d7 100644
@@ -200,6 +200,9 @@ void mem_cgroup_split_huge_fixup(struct page *head);
 bool mem_cgroup_bad_page_check(struct page *page);
 void mem_cgroup_print_bad_page(struct page *page);
 #endif
+
+unsigned long
+memcg_zone_reclaimable_pages(struct mem_cgroup *memcg, struct zone *zone);
 #else /* CONFIG_MEMCG */
 struct mem_cgroup;
 
@@ -378,6 +381,12 @@ static inline void mem_cgroup_replace_page_cache(struct page *oldpage,
 				struct page *newpage)
 {
 }
+
+static inline unsigned long
+memcg_zone_reclaimable_pages(struct mem_cgroup *memcg, struct zone *zone)
+{
+	return 0;
+}
 #endif /* CONFIG_MEMCG */
 
 #if !defined(CONFIG_MEMCG) || !defined(CONFIG_DEBUG_VM)
@@ -430,6 +439,8 @@ static inline bool memcg_kmem_enabled(void)
 	return static_key_false(&memcg_kmem_enabled_key);
 }
 
+bool memcg_kmem_is_active(struct mem_cgroup *memcg);
+
 /*
  * In general, we'll do everything in our power to not incur in any overhead
  * for non-memcg users for the kmem functions. Not even a function call, if we
@@ -563,6 +574,12 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp)
 	return __memcg_kmem_get_cache(cachep, gfp);
 }
 #else
+
+static inline bool memcg_kmem_is_active(struct mem_cgroup *memcg)
+{
+	return false;
+}
+
 #define for_each_memcg_cache_index(_idx)	\
 	for (; NULL; )
 
include/linux/shrinker.h +4 -0
index 00a3e57..3b08869 100644
@@ -20,6 +20,9 @@ struct shrink_control {
 
 	/* shrink from these nodes */
 	nodemask_t nodes_to_scan;
+
+	/* reclaim from this memcg only (if not NULL) */
+	struct mem_cgroup *target_mem_cgroup;
 };
 
 /*
@@ -45,6 +48,7 @@ struct shrinker {
 
 	int seeks;	/* seeks to recreate an obj */
 	long batch;	/* reclaim batch size, 0 = default */
+	bool memcg_shrinker; /* memcg-aware shrinker */
 
 	/* These are for internal use */
 	struct list_head list;
mm/memcontrol.c +15 -1
index e34da3c..980bd2d 100644
@@ -465,7 +465,7 @@ static inline void memcg_kmem_set_active(struct mem_cgroup *memcg)
 	set_bit(KMEM_ACCOUNTED_ACTIVE, &memcg->kmem_account_flags);
 }
 
-static bool memcg_kmem_is_active(struct mem_cgroup *memcg)
+bool memcg_kmem_is_active(struct mem_cgroup *memcg)
 {
 	return test_bit(KMEM_ACCOUNTED_ACTIVE, &memcg->kmem_account_flags);
 }
@@ -1044,6 +1044,20 @@ mem_cgroup_zone_nr_lru_pages(struct mem_cgroup *memcg, int nid, int zid,
 	return ret;
 }
 
+unsigned long
+memcg_zone_reclaimable_pages(struct mem_cgroup *memcg, struct zone *zone)
+{
+	int nid = zone_to_nid(zone);
+	int zid = zone_idx(zone);
+	unsigned long val;
+
+	val = mem_cgroup_zone_nr_lru_pages(memcg, nid, zid, LRU_ALL_FILE);
+	if (do_swap_account)
+		val += mem_cgroup_zone_nr_lru_pages(memcg, nid, zid,
+						    LRU_ALL_ANON);
+	return val;
+}
+
 static unsigned long
 mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg,
 			int nid, unsigned int lru_mask)
mm/vmscan.c +43 -3

82 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#46 lore raw

PATCH v6 22/31 memcg,list_lru: duplicate LRUs upon kmemcg creation

In reply to Glauber Costa

When a new memcg is created, we need to open up room for its descriptors in all of the list_lrus that are marked per-memcg. The process is quite similar to the one we are using for the kmem caches: we initialize the new structures in an array indexed by kmemcg_id, and grow the array if needed. Key data like the size of the array will be shared between the kmem cache code and the list_lru code (they basically describe the same thing)

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/list_lru.h   |  48 +++++++++++++-
 include/linux/memcontrol.h |  12 ++++
 lib/list_lru.c             | 102 +++++++++++++++++++++++++++---
 mm/memcontrol.c            | 151 +++++++++++++++++++++++++++++++++++++++++++--
 mm/slab_common.c           |   1 -
 5 files changed, 297 insertions(+), 17 deletions(-)
include/linux/list_lru.h +47 -1
index f5e3c15..cd67f5c 100644
@@ -24,6 +24,23 @@ struct list_lru_node {
 	long			nr_items;
 } ____cacheline_aligned_in_smp;
 
+/*
+ * This is supposed to be M x N matrix, where M is kmem-limited memcg, and N is
+ * the number of nodes. Both dimensions are likely to be very small, but are
+ * potentially very big. Therefore we will allocate or grow them dynamically.
+ *
+ * The size of M will increase as new memcgs appear and can be 0 if no memcgs
+ * are being used. This is done in mm/memcontrol.c in a way quite similar than
+ * the way we use for the slab cache management.
+ *
+ * The size o N can't be determined at compile time, but won't increase once we
+ * determine it. It is nr_node_ids, the firmware-provided maximum number of
+ * nodes in a system.
+ */
+struct list_lru_array {
+	struct list_lru_node node[1];
+};
+
 struct list_lru {
 	/*
 	 * Because we use a fixed-size array, this struct can be very big if
@@ -37,9 +54,38 @@ struct list_lru {
 	 */
 	struct list_lru_node	node[MAX_NUMNODES];
 	nodemask_t		active_nodes;
+#ifdef CONFIG_MEMCG_KMEM
+	/* All memcg-aware LRUs will be chained in the lrus list */
+	struct list_head	lrus;
+	/* M x N matrix as described above */
+	struct list_lru_array	**memcg_lrus;
+#endif
 };
 
-int list_lru_init(struct list_lru *lru);
+struct mem_cgroup;
+#ifdef CONFIG_MEMCG_KMEM
+struct list_lru_array *lru_alloc_array(void);
+int memcg_update_all_lrus(unsigned long num);
+void list_lru_destroy(struct list_lru *lru);
+void list_lru_destroy_memcg(struct mem_cgroup *memcg);
+int __memcg_init_lru(struct list_lru *lru);
+#else
+static inline void list_lru_destroy(struct list_lru *lru)
+{
+}
+#endif
+
+int __list_lru_init(struct list_lru *lru, bool memcg_enabled);
+static inline int list_lru_init(struct list_lru *lru)
+{
+	return __list_lru_init(lru, false);
+}
+
+static inline int list_lru_init_memcg(struct list_lru *lru)
+{
+	return __list_lru_init(lru, true);
+}
+
 int list_lru_add(struct list_lru *lru, struct list_head *item);
 int list_lru_del(struct list_lru *lru, struct list_head *item);
 unsigned long
include/linux/memcontrol.h +12 -0
index 489c6d7..3442eb9 100644
@@ -23,6 +23,7 @@
 #include <linux/vm_event_item.h>
 #include <linux/hardirq.h>
 #include <linux/jump_label.h>
+#include <linux/list_lru.h>
 
 struct mem_cgroup;
 struct page_cgroup;
@@ -470,6 +471,12 @@ void memcg_update_array_size(int num_groups);
 struct kmem_cache *
 __memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp);
 
+int memcg_new_lru(struct list_lru *lru);
+int memcg_init_lru(struct list_lru *lru);
+
+int memcg_kmem_update_lru_size(struct list_lru *lru, int num_groups,
+			       bool new_lru);
+
 void mem_cgroup_destroy_cache(struct kmem_cache *cachep);
 void kmem_cache_destroy_memcg_children(struct kmem_cache *s);
 
@@ -633,6 +640,11 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp)
 static inline void kmem_cache_destroy_memcg_children(struct kmem_cache *s)
 {
 }
+
+static inline int memcg_init_lru(struct list_lru *lru)
+{
+	return 0;
+}
 #endif /* CONFIG_MEMCG_KMEM */
 #endif /* _LINUX_MEMCONTROL_H */
 
lib/list_lru.c +93 -9

123 lines not on this page · raw message

mm/memcontrol.c +145 -6

175 lines not on this page · raw message

mm/slab_common.c +0 -1
index ff3218a..b729c53 100644
@@ -102,7 +102,6 @@ int memcg_update_all_caches(int num_memcgs)
 			goto out;
 	}
 
-	memcg_update_array_size(num_memcgs);
 out:
 	mutex_unlock(&slab_mutex);
 	return ret;
Signature · 1 line
1.8.1.4
#47 lore raw

PATCH v6 23/31 lru: add an element to a memcg list

In reply to Glauber Costa

With the infrastructure we now have, we can add an element to a memcg LRU list instead of the global list. The memcg lists are still per-node.

Technically, we will never trigger per-node shrinking in the memcg is short of memory. Therefore an alternative to this would be to add the element to *both* a single-node memcg array and a per-node global array.

There are two main reasons for this design choice:

1) adding an extra list_head to each of the objects would waste 16-bytes per object, always remembering that we are talking about 1 dentry + 1 inode in the common case. This means a close to 10 % increase in the dentry size, and a lower yet significant increase in the inode size. In terms of total memory, this design pays 32-byte per-superblock-per-node (size of struct list_lru_node), which means that in any scenario where we have more than 10 dentries + inodes, we would already be paying more memory in the two-list-heads approach than we will here with 1 node x 10 superblocks. The turning point of course depends on the workload, but I hope the figures above would convince you that the memory footprint is in my side in any workload that matters.

2) The main drawback of this, namely, that we loose global LRU order, is not really seen by me as a disadvantage: if we are using memcg to isolate the workloads, global pressure should try to balance the amount reclaimed from all memcgs the same way the shrinkers will already naturally balance the amount reclaimed from each superblock. (This patchset needs some love in this regard, btw).

To help us easily tracking down which nodes have and which nodes doesn't have elements in the list, we will count on an auxiliary node bitmap in the global level.

[ v2: move memcg_kmem_lru_of_page to list_lru.c and then unpublish the
auxiliary functions it uses ]

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/list_lru.h   |  11 +++++
 include/linux/memcontrol.h |   8 ++++
 lib/list_lru.c             | 104 +++++++++++++++++++++++++++++++++++++++++----
 mm/memcontrol.c            |  24 ++++++++++-
 4 files changed, 137 insertions(+), 10 deletions(-)
include/linux/list_lru.h +11 -0
index cd67f5c..f6756fb 100644
@@ -53,12 +53,23 @@ struct list_lru {
 	 * structure, we may very well fail.
 	 */
 	struct list_lru_node	node[MAX_NUMNODES];
+	atomic_long_t		node_totals[MAX_NUMNODES];
 	nodemask_t		active_nodes;
 #ifdef CONFIG_MEMCG_KMEM
 	/* All memcg-aware LRUs will be chained in the lrus list */
 	struct list_head	lrus;
 	/* M x N matrix as described above */
 	struct list_lru_array	**memcg_lrus;
+	/*
+	 * The memcg_lrus is RCU protected, so we need to keep the previous
+	 * array around when we update it. But we can only do that after
+	 * synchronize_rcu(). A typical system has many LRUs, which means
+	 * that if we call synchronize_rcu after each LRU update, this
+	 * will become very expensive. We add this pointer here, and then
+	 * after all LRUs are update, we call synchronize_rcu() once, and
+	 * free all the old_arrays.
+	 */
+	void *old_array;
 #endif
 };
 
include/linux/memcontrol.h +8 -0
index 3442eb9..50f199f 100644
@@ -24,6 +24,7 @@
 #include <linux/hardirq.h>
 #include <linux/jump_label.h>
 #include <linux/list_lru.h>
+#include <linux/mm.h>
 
 struct mem_cgroup;
 struct page_cgroup;
@@ -474,6 +475,8 @@ __memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp);
 int memcg_new_lru(struct list_lru *lru);
 int memcg_init_lru(struct list_lru *lru);
 
+struct mem_cgroup *mem_cgroup_from_kmem_page(struct page *page);
+
 int memcg_kmem_update_lru_size(struct list_lru *lru, int num_groups,
 			       bool new_lru);
 
@@ -645,6 +648,11 @@ static inline int memcg_init_lru(struct list_lru *lru)
 {
 	return 0;
 }
+
+static inline struct mem_cgroup *mem_cgroup_from_kmem_page(struct page *page)
+{
+	return NULL;
+}
 #endif /* CONFIG_MEMCG_KMEM */
 #endif /* _LINUX_MEMCONTROL_H */
 
lib/list_lru.c +95 -9

151 lines not on this page · raw message

mm/memcontrol.c +23 -1
index d6f4b5e..4bf6738 100644
@@ -3335,9 +3335,15 @@ int memcg_kmem_update_lru_size(struct list_lru *lru, int num_groups,
 		 * either follow the new array or the old one and they contain
 		 * exactly the same information. The new space in the end is
 		 * always empty anyway.
+		 *
+		 * We do have to make sure that no more users of the old
+		 * memcg_lrus array exist before we free, and this is achieved
+		 * by rcu. Since it would be too slow to synchronize RCU for
+		 * every LRU, we store the pointer and let the LRU code free
+		 * all of them when all LRUs are updated.
 		 */
 		if (lru->memcg_lrus)
-			kfree(old_array);
+			lru->old_array = old_array;
 	}
 
 	if (lru->memcg_lrus) {
@@ -3481,6 +3487,22 @@ static inline void memcg_resume_kmem_account(void)
 	current->memcg_kmem_skip_account--;
 }
 
+struct mem_cgroup *mem_cgroup_from_kmem_page(struct page *page)
+{
+	struct page_cgroup *pc;
+	struct mem_cgroup *memcg = NULL;
+
+	pc = lookup_page_cgroup(page);
+	if (!PageCgroupUsed(pc))
+		return NULL;
+
+	lock_page_cgroup(pc);
+	if (PageCgroupUsed(pc))
+		memcg = pc->mem_cgroup;
+	unlock_page_cgroup(pc);
+	return memcg;
+}
+
 static void kmem_cache_destroy_work_func(struct work_struct *w)
 {
 	struct kmem_cache *cachep;
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#48 lore raw

PATCH v6 24/31 list_lru: per-memcg walks

In reply to Glauber Costa

This patch extend the list_lru interfaces to allow for a memcg parameter. Because most of its users won't need it, instead of modifying the function signatures we create a new set of _memcg() functions and write the old API ontop of that.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/list_lru.h   |  26 ++++++++---
 include/linux/memcontrol.h |   2 +
 lib/list_lru.c             | 110 ++++++++++++++++++++++++++++++++++-----------
 3 files changed, 107 insertions(+), 31 deletions(-)
include/linux/list_lru.h +20 -6
index f6756fb..9e6bbb1 100644
@@ -99,23 +99,37 @@ static inline int list_lru_init_memcg(struct list_lru *lru)
 
 int list_lru_add(struct list_lru *lru, struct list_head *item);
 int list_lru_del(struct list_lru *lru, struct list_head *item);
-unsigned long
-list_lru_count_nodemask(struct list_lru *lru, nodemask_t *nodes_to_count);
+
+unsigned long list_lru_count_nodemask_memcg(struct list_lru *lru,
+			nodemask_t *nodes_to_count, struct mem_cgroup *memcg);
+
+static inline unsigned long
+list_lru_count_nodemask(struct list_lru *lru, nodemask_t *nodes_to_count)
+{
+	return list_lru_count_nodemask_memcg(lru, nodes_to_count, NULL);
+}
 
 static inline unsigned long list_lru_count(struct list_lru *lru)
 {
 	return list_lru_count_nodemask(lru, &lru->active_nodes);
 }
 
-
 typedef enum lru_status
 (*list_lru_walk_cb)(struct list_head *item, spinlock_t *lock, void *cb_arg);
 
 typedef void (*list_lru_dispose_cb)(struct list_head *dispose_list);
 
-unsigned long
-list_lru_walk_nodemask(struct list_lru *lru, list_lru_walk_cb isolate,
-	   void *cb_arg, unsigned long nr_to_walk, nodemask_t *nodes_to_walk);
+unsigned long list_lru_walk_nodemask_memcg(struct list_lru *lru,
+	list_lru_walk_cb isolate, void *cb_arg, unsigned long nr_to_walk,
+	nodemask_t *nodes_to_walk, struct mem_cgroup *memcg);
+
+static inline unsigned long list_lru_walk_nodemask(struct list_lru *lru,
+	list_lru_walk_cb isolate, void *cb_arg, unsigned long nr_to_walk,
+	nodemask_t *nodes_to_walk)
+{
+	return list_lru_walk_nodemask_memcg(lru, isolate, cb_arg, nr_to_walk,
+					    &lru->active_nodes, NULL);
+}
 
 static inline unsigned long
 list_lru_walk(struct list_lru *lru, list_lru_walk_cb isolate,
include/linux/memcontrol.h +2 -0
index 50f199f..3eeece8 100644
@@ -593,6 +593,8 @@ static inline bool memcg_kmem_is_active(struct mem_cgroup *memcg)
 #define for_each_memcg_cache_index(_idx)	\
 	for (; NULL; )
 
+#define memcg_limited_groups_array_size 0
+
 static inline bool memcg_kmem_enabled(void)
 {
 	return false;
lib/list_lru.c +85 -25

169 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#49 lore raw

PATCH v6 25/31 memcg: per-memcg kmem shrinking

In reply to Glauber Costa

If the kernel limit is smaller than the user limit, we will have situations in which our allocations fail but freeing user pages will buy us nothing. In those, we would like to call a specialized memcg reclaimer that only frees kernel memory and leave the user memory alone. Those are also expected to fail when we account memcg->kmem, instead of when we account memcg->res. Based on that, this patch implements a memcg-specific reclaimer, that only shrinks kernel objects, withouth touching user pages.

There might be situations in which there are plenty of objects to shrink, but we can't do it because the __GFP_FS flag is not set. Although they can happen with user pages, they are a lot more common with fs-metadata: this is the case with almost all inode allocation.

Those allocations are, however, capable of waiting. So we can just span a worker, let it finish its job and proceed with the allocation. As slow as it is, at this point we are already past any hopes anyway.

[ v2: moved congestion_wait call to vmscan.c ]
Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/swap.h |   2 +
 mm/memcontrol.c      | 180 ++++++++++++++++++++++++++++++++++++++++-----------
 mm/vmscan.c          |  44 ++++++++++++-
 3 files changed, 187 insertions(+), 39 deletions(-)
include/linux/swap.h +2 -0
index ca031f7..5a0ef45 100644
@@ -268,6 +268,8 @@ extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 extern int __isolate_lru_page(struct page *page, isolate_mode_t mode);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem,
 						  gfp_t gfp_mask, bool noswap);
+extern unsigned long try_to_free_mem_cgroup_kmem(struct mem_cgroup *mem,
+						 gfp_t gfp_mask);
 extern unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *mem,
 						gfp_t gfp_mask, bool noswap,
 						struct zone *zone,
mm/memcontrol.c +142 -38

251 lines not on this page · raw message

mm/vmscan.c +43 -1

50 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#50 lore raw

PATCH v6 26/31 memcg: scan cache objects hierarchically

In reply to Glauber Costa

When reaching shrink_slab, we should descent in children memcg searching for objects that could be shrunk. This is true even if the memcg does not have kmem limits on, since the kmem res_counter will also be billed against the user res_counter of the parent.

It is possible that we will free objects and not free any pages, that will just harm the child groups without helping the parent group at all. But at this point, we basically are prepared to pay the price.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 include/linux/memcontrol.h |   6 ++
 mm/memcontrol.c            |  13 +++
 mm/vmscan.c                | 206 ++++++++++++++++++++++++++-------------------
 3 files changed, 139 insertions(+), 86 deletions(-)
include/linux/memcontrol.h +6 -0
index 3eeece8..c8b1412 100644
@@ -441,6 +441,7 @@ static inline bool memcg_kmem_enabled(void)
 	return static_key_false(&memcg_kmem_enabled_key);
 }
 
+bool memcg_kmem_should_reclaim(struct mem_cgroup *memcg);
 bool memcg_kmem_is_active(struct mem_cgroup *memcg);
 
 /*
@@ -585,6 +586,11 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp)
 }
 #else
 
+static inline bool memcg_kmem_should_reclaim(struct mem_cgroup *memcg)
+{
+	return false;
+}
+
 static inline bool memcg_kmem_is_active(struct mem_cgroup *memcg)
 {
 	return false;
mm/memcontrol.c +13 -0
index c563550..b8980d1 100644
@@ -3043,6 +3043,19 @@ static void __mem_cgroup_commit_charge(struct mem_cgroup *memcg,
 }
 
 #ifdef CONFIG_MEMCG_KMEM
+bool memcg_kmem_should_reclaim(struct mem_cgroup *memcg)
+{
+	struct mem_cgroup *iter;
+
+	for_each_mem_cgroup_tree(iter, memcg) {
+		if (memcg_kmem_is_active(iter)) {
+			mem_cgroup_iter_break(memcg, iter);
+			return true;
+		}
+	}
+	return false;
+}
+
 static inline bool memcg_can_account_kmem(struct mem_cgroup *memcg)
 {
 	return !mem_cgroup_disabled() && !mem_cgroup_is_root(memcg) &&
mm/vmscan.c +120 -86

240 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#51 lore raw

PATCH v6 27/31 vmscan: take at least one pass with shrinkers

In reply to Glauber Costa

In very low free kernel memory situations, it may be the case that we have less objects to free than our initial batch size. If this is the case, it is better to shrink those, and open space for the new workload then to keep them and fail the new allocations.

In particular, we are concerned with the direct reclaim case for memcg. Although this same technique can be applied to other situations just as well, we will start conservative and apply it for that case, which is the one that matters the most.

[ v6: only do it per memcg ]
[ v5: differentiate no-scan case, don't do this for kswapd ]

CC Dave Chinner <[email protected]>
CC Carlos Maiolino <[email protected]>
CC "Theodore Ts'o" <[email protected]>
CC Al Viro <[email protected]>
---
 mm/vmscan.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)
mm/vmscan.c +18 -5
index 9c45e5c..6bac41e 100644
@@ -273,20 +273,33 @@ shrink_slab_one(struct shrinker *shrinker, struct shrink_control *shrinkctl,
 				nr_pages_scanned, lru_pages,
 				max_pass, delta, total_scan);
 
-	while (total_scan >= batch_size) {
+	do {
 		long ret;
+		unsigned long nr_to_scan = min(batch_size, total_scan);
+		struct mem_cgroup *memcg = shrinkctl->target_mem_cgroup;
+
+		/*
+		 * Differentiate between "few objects" and "no objects"
+		 * as returned by the count step.
+		 */
+		if (!total_scan)
+			break;
+
+		if ((total_scan < batch_size) &&
+		   !(memcg && memcg_kmem_is_active(memcg)))
+			break;
 
-		shrinkctl->nr_to_scan = batch_size;
+		shrinkctl->nr_to_scan = nr_to_scan;
 		ret = shrinker->scan_objects(shrinker, shrinkctl);
 		if (ret == -1)
 			break;
 		freed += ret;
 
-		count_vm_events(SLABS_SCANNED, batch_size);
-		total_scan -= batch_size;
+		count_vm_events(SLABS_SCANNED, nr_to_scan);
+		total_scan -= nr_to_scan;
 
 		cond_resched();
-	}
+	} while (total_scan >= batch_size);
 
 	/*
 	 * move the unused scan count back into the shrinker in a
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#52 lore raw

PATCH v6 28/31 super: targeted memcg reclaim

In reply to Glauber Costa

We now have all our dentries and inodes placed in memcg-specific LRU lists. All we have to do is restrict the reclaim to the said lists in case of memcg pressure.

That can't be done so easily for the fs_objects part of the equation, since this is heavily fs-specific. What we do is pass on the context, and let the filesystems decide if they ever chose or want to. At this time, we just don't shrink them in memcg pressure (none is supported), leaving that for global pressure only.

Marking the superblock shrinker and its LRUs as memcg-aware will guarantee that the shrinkers will get invoked during targetted reclaim.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 fs/dcache.c   |  6 +++---
 fs/inode.c    |  6 +++---
 fs/internal.h |  5 +++--
 fs/super.c    | 39 +++++++++++++++++++++++++++------------
 4 files changed, 36 insertions(+), 20 deletions(-)
fs/dcache.c +3 -3
index 20ab9c0..5eda937 100644
@@ -885,13 +885,13 @@ dentry_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * use.
  */
 long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
-		     nodemask_t *nodes_to_walk)
+		     nodemask_t *nodes_to_walk, struct mem_cgroup *memcg)
 {
 	LIST_HEAD(dispose);
 	long freed;
 
-	freed = list_lru_walk_nodemask(&sb->s_dentry_lru, dentry_lru_isolate,
-				       &dispose, nr_to_scan, nodes_to_walk);
+	freed = list_lru_walk_nodemask_memcg(&sb->s_dentry_lru,
+		dentry_lru_isolate, &dispose, nr_to_scan, nodes_to_walk, memcg);
 	shrink_dentry_list(&dispose);
 	return freed;
 }
fs/inode.c +3 -3
index 3cf4cb0..6565d88 100644
@@ -747,13 +747,13 @@ inode_lru_isolate(struct list_head *item, spinlock_t *lru_lock, void *arg)
  * then are freed outside inode_lock by dispose_list().
  */
 long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan,
-		     nodemask_t *nodes_to_walk)
+			nodemask_t *nodes_to_walk, struct mem_cgroup *memcg)
 {
 	LIST_HEAD(freeable);
 	long freed;
 
-	freed = list_lru_walk_nodemask(&sb->s_inode_lru, inode_lru_isolate,
-				       &freeable, nr_to_scan, nodes_to_walk);
+	freed = list_lru_walk_nodemask_memcg(&sb->s_inode_lru,
+		inode_lru_isolate, &freeable, nr_to_scan, nodes_to_walk, memcg);
 	dispose_list(&freeable);
 	return freed;
 }
fs/internal.h +3 -2
index 91900f2..068c7f7 100644
@@ -16,6 +16,7 @@ struct file_system_type;
 struct linux_binprm;
 struct path;
 struct mount;
+struct mem_cgroup;
 
 /*
  * block_dev.c
@@ -111,7 +112,7 @@ extern int open_check_o_direct(struct file *f);
  */
 extern spinlock_t inode_sb_list_lock;
 extern long prune_icache_sb(struct super_block *sb, unsigned long nr_to_scan,
-			    nodemask_t *nodes_to_scan);
+		    nodemask_t *nodes_to_scan, struct mem_cgroup *memcg);
 extern void inode_add_lru(struct inode *inode);
 
 /*
@@ -128,7 +129,7 @@ extern int invalidate_inodes(struct super_block *, bool);
  */
 extern struct dentry *__d_alloc(struct super_block *, const struct qstr *);
 extern long prune_dcache_sb(struct super_block *sb, unsigned long nr_to_scan,
-			    nodemask_t *nodes_to_scan);
+		    nodemask_t *nodes_to_scan, struct mem_cgroup *memcg);
 
 /*
  * read_write.c
fs/super.c +27 -12
index 5c7b879..e92ebcb 100644
@@ -34,6 +34,7 @@
 #include <linux/cleancache.h>
 #include <linux/fsnotify.h>
 #include <linux/lockdep.h>
+#include <linux/memcontrol.h>
 #include "internal.h"
 
 
@@ -56,6 +57,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = {
 static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct super_block *sb;
+	struct mem_cgroup *memcg = sc->target_mem_cgroup;
 	long	fs_objects = 0;
 	long	total_objects;
 	long	freed = 0;
@@ -74,11 +76,13 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 	if (!grab_super_passive(sb))
 		return -1;
 
-	if (sb->s_op && sb->s_op->nr_cached_objects)
+	if (sb->s_op && sb->s_op->nr_cached_objects && !memcg)
 		fs_objects = sb->s_op->nr_cached_objects(sb, &sc->nodes_to_scan);
 
-	inodes = list_lru_count_nodemask(&sb->s_inode_lru, &sc->nodes_to_scan);
-	dentries = list_lru_count_nodemask(&sb->s_dentry_lru, &sc->nodes_to_scan);
+	inodes = list_lru_count_nodemask_memcg(&sb->s_inode_lru,
+					 &sc->nodes_to_scan, memcg);
+	dentries = list_lru_count_nodemask_memcg(&sb->s_dentry_lru,
+					   &sc->nodes_to_scan, memcg);
 	total_objects = dentries + inodes + fs_objects + 1;
 
 	/* proportion the scan between the caches */
@@ -89,8 +93,8 @@ static long super_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 	 * prune the dcache first as the icache is pinned by it, then
 	 * prune the icache, followed by the filesystem specific caches
 	 */
-	freed = prune_dcache_sb(sb, dentries, &sc->nodes_to_scan);
-	freed += prune_icache_sb(sb, inodes, &sc->nodes_to_scan);
+	freed = prune_dcache_sb(sb, dentries, &sc->nodes_to_scan, memcg);
+	freed += prune_icache_sb(sb, inodes, &sc->nodes_to_scan, memcg);
 
 	if (fs_objects) {
 		fs_objects = mult_frac(sc->nr_to_scan, fs_objects,
@@ -107,20 +111,26 @@ static long super_cache_count(struct shrinker *shrink, struct shrink_control *sc
 {
 	struct super_block *sb;
 	long	total_objects = 0;
+	struct mem_cgroup *memcg = sc->target_mem_cgroup;
 
 	sb = container_of(shrink, struct super_block, s_shrink);
 
 	if (!grab_super_passive(sb))
 		return -1;
 
-	if (sb->s_op && sb->s_op->nr_cached_objects)
+	/*
+	 * Ideally we would pass memcg to nr_cached_objects, and
+	 * let the underlying filesystem decide. Most likely the
+	 * path will be if (!memcg) return;, but even then.
+	 */
+	if (sb->s_op && sb->s_op->nr_cached_objects && !memcg)
 		total_objects = sb->s_op->nr_cached_objects(sb,
 						 &sc->nodes_to_scan);
 
-	total_objects += list_lru_count_nodemask(&sb->s_dentry_lru,
-						 &sc->nodes_to_scan);
-	total_objects += list_lru_count_nodemask(&sb->s_inode_lru,
-						 &sc->nodes_to_scan);
+	total_objects += list_lru_count_nodemask_memcg(&sb->s_dentry_lru,
+					 &sc->nodes_to_scan, memcg);
+	total_objects += list_lru_count_nodemask_memcg(&sb->s_inode_lru,
+					 &sc->nodes_to_scan, memcg);
 
 	total_objects = vfs_pressure_ratio(total_objects);
 	drop_super(sb);
@@ -199,8 +209,10 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)
 		INIT_HLIST_NODE(&s->s_instances);
 		INIT_HLIST_BL_HEAD(&s->s_anon);
 		INIT_LIST_HEAD(&s->s_inodes);
-		list_lru_init(&s->s_dentry_lru);
-		list_lru_init(&s->s_inode_lru);
+
+		list_lru_init_memcg(&s->s_dentry_lru);
+		list_lru_init_memcg(&s->s_inode_lru);
+
 		INIT_LIST_HEAD(&s->s_mounts);
 		init_rwsem(&s->s_umount);
 		lockdep_set_class(&s->s_umount, &type->s_umount_key);
@@ -236,6 +248,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags)
 		s->s_shrink.scan_objects = super_cache_scan;
 		s->s_shrink.count_objects = super_cache_count;
 		s->s_shrink.batch = 1024;
+		s->s_shrink.memcg_shrinker = true;
 	}
 out:
 	return s;
@@ -318,6 +331,8 @@ void deactivate_locked_super(struct super_block *s)
 
 		/* caches are now gone, we can safely kill the shrinker now */
 		unregister_shrinker(&s->s_shrink);
+		list_lru_destroy(&s->s_dentry_lru);
+		list_lru_destroy(&s->s_inode_lru);
 		put_filesystem(fs);
 		put_super(s);
 	} else {
Signature · 1 line
1.8.1.4
#53 lore raw

PATCH v6 29/31 memcg: move initialization to memcg creation

In reply to Glauber Costa

Those structures are only used for memcgs that are effectively using kmemcg. However, in a later patch I intend to use scan that list inconditionally (list empty meaning no kmem caches present), which simplifies the code a lot.

So move the initialization to early kmem creation.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memcontrol.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/memcontrol.c +2 -2
index b8980d1..6f6a330 100644
@@ -3323,9 +3323,7 @@ int memcg_update_cache_sizes(struct mem_cgroup *memcg)
 
 	memcg_update_array_size(num + 1);
 
-	INIT_LIST_HEAD(&memcg->memcg_slab_caches);
 	INIT_WORK(&memcg->kmemcg_shrink_work, kmemcg_shrink_work_fn);
-	mutex_init(&memcg->slab_caches_mutex);
 
 	return 0;
 out:
@@ -6318,6 +6316,8 @@ static int memcg_init_kmem(struct mem_cgroup *memcg, struct cgroup_subsys *ss)
 {
 	int ret;
 
+	INIT_LIST_HEAD(&memcg->memcg_slab_caches);
+	mutex_init(&memcg->slab_caches_mutex);
 	memcg->kmemcg_id = -1;
 	ret = memcg_propagate_kmem(memcg);
 	if (ret)
Signature · 1 line
1.8.1.4
#54 lore raw

PATCH v6 30/31 vmpressure: in-kernel notifications

In reply to Glauber Costa

From: Glauber Costa <[email protected]>

During the past weeks, it became clear to us that the shrinker interface we have right now works very well for some particular types of users, but not that well for others. The later are usually people interested in one-shot notifications, that were forced to adapt themselves to the count+scan behavior of shrinkers. To do so, they had no choice than to greatly abuse the shrinker interface producing little monsters all over.

During LSF/MM, one of the proposals that popped out during our session was to reuse Anton Voronstsov's vmpressure for this. They are designed for userspace consumption, but also provide a well-stablished, cgroup-aware entry point for notifications.

This patch extends that to also support in-kernel users. Events that should be generated for in-kernel consumption will be marked as such, and for those, we will call a registered function instead of triggering an eventfd notification.

Please note that due to my lack of understanding of each shrinker user, I will stay away from converting the actual users, you are all welcome to do so.

Cc Dave Chinner <[email protected]>
Cc John Stultz <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Johannes Weiner <[email protected]>
---
 include/linux/vmpressure.h |  6 ++++++
 mm/vmpressure.c            | 52 +++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 55 insertions(+), 3 deletions(-)
include/linux/vmpressure.h +6 -0
index 76be077..3131e72 100644
@@ -19,6 +19,9 @@ struct vmpressure {
 	/* Have to grab the lock on events traversal or modifications. */
 	struct mutex events_lock;
 
+	/* False if only kernel users want to be notified, true otherwise. */
+	bool notify_userspace;
+
 	struct work_struct work;
 };
 
@@ -36,6 +39,9 @@ extern struct vmpressure *css_to_vmpressure(struct cgroup_subsys_state *css);
 extern int vmpressure_register_event(struct cgroup *cg, struct cftype *cft,
 				     struct eventfd_ctx *eventfd,
 				     const char *args);
+
+extern int vmpressure_register_kernel_event(struct cgroup *cg,
+					    void (*fn)(void));
 extern void vmpressure_unregister_event(struct cgroup *cg, struct cftype *cft,
 					struct eventfd_ctx *eventfd);
 #else
mm/vmpressure.c +49 -3

90 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#55 lore raw

PATCH v6 31/31 memcg: reap dead memcgs upon global memory pressure.

In reply to Glauber Costa

When we delete kmem-enabled memcgs, they can still be zombieing around for a while. The reason is that the objects may still be alive, and we won't be able to delete them at destruction time.

The only entry point for that, though, are the shrinkers. The shrinker interface, however, is not exactly tailored to our needs. It could be a little bit better by using the API Dave Chinner proposed, but it is still not ideal since we aren't really a count-and-scan event, but more a one-off flush-all-you-can event that would have to abuse that somehow.

Cc Dave Chinner <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kamezawa Hiroyuki <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memcontrol.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 6 deletions(-)
mm/memcontrol.c +46 -6
index 6f6a330..7006f03 100644
@@ -400,7 +400,6 @@ static size_t memcg_size(void)
 		nr_node_ids * sizeof(struct mem_cgroup_per_node);
 }
 
-#ifdef CONFIG_MEMCG_DEBUG_ASYNC_DESTROY
 static LIST_HEAD(dangling_memcgs);
 static DEFINE_MUTEX(dangling_memcgs_mutex);
 
@@ -409,11 +408,14 @@ static inline void memcg_dangling_free(struct mem_cgroup *memcg)
 	mutex_lock(&dangling_memcgs_mutex);
 	list_del(&memcg->dead);
 	mutex_unlock(&dangling_memcgs_mutex);
+#ifdef CONFIG_MEMCG_DEBUG_ASYNC_DESTROY
 	free_pages((unsigned long)memcg->memcg_name, 0);
+#endif
 }
 
 static inline void memcg_dangling_add(struct mem_cgroup *memcg)
 {
+#ifdef CONFIG_MEMCG_DEBUG_ASYNC_DESTROY
 	/*
 	 * cgroup.c will do page-sized allocations most of the time,
 	 * so we'll just follow the pattern. Also, __get_free_pages
@@ -439,15 +441,12 @@ static inline void memcg_dangling_add(struct mem_cgroup *memcg)
 	}
 
 add_list:
+#endif
 	INIT_LIST_HEAD(&memcg->dead);
 	mutex_lock(&dangling_memcgs_mutex);
 	list_add(&memcg->dead, &dangling_memcgs);
 	mutex_unlock(&dangling_memcgs_mutex);
 }
-#else
-static inline void memcg_dangling_free(struct mem_cgroup *memcg) {}
-static inline void memcg_dangling_add(struct mem_cgroup *memcg) {}
-#endif
 
 static DEFINE_MUTEX(set_limit_mutex);
 
@@ -6312,6 +6311,41 @@ static int mem_cgroup_oom_control_write(struct cgroup *cgrp,
 }
 
 #ifdef CONFIG_MEMCG_KMEM
+static void memcg_vmpressure_shrink_dead(void)
+{
+	struct memcg_cache_params *params, *tmp;
+	struct kmem_cache *cachep;
+	struct mem_cgroup *memcg;
+
+	mutex_lock(&dangling_memcgs_mutex);
+	list_for_each_entry(memcg, &dangling_memcgs, dead) {
+		mutex_lock(&memcg->slab_caches_mutex);
+		/* The element may go away as an indirect result of shrink */
+		list_for_each_entry_safe(params, tmp,
+					 &memcg->memcg_slab_caches, list) {
+			cachep = memcg_params_to_cache(params);
+			/*
+			 * the cpu_hotplug lock is taken in kmem_cache_create
+			 * outside the slab_caches_mutex manipulation. It will
+			 * be taken by kmem_cache_shrink to flush the cache.
+			 * So we need to drop the lock. It is all right because
+			 * the lock only protects elements moving in and out the
+			 * list.
+			 */
+			mutex_unlock(&memcg->slab_caches_mutex);
+			kmem_cache_shrink(cachep);
+			mutex_lock(&memcg->slab_caches_mutex);
+		}
+		mutex_unlock(&memcg->slab_caches_mutex);
+	}
+	mutex_unlock(&dangling_memcgs_mutex);
+}
+
+static void memcg_register_kmem_events(struct cgroup *cont)
+{
+	vmpressure_register_kernel_event(cont, memcg_vmpressure_shrink_dead);
+}
+
 static int memcg_init_kmem(struct mem_cgroup *memcg, struct cgroup_subsys *ss)
 {
 	int ret;
@@ -6347,6 +6381,10 @@ static void kmem_cgroup_destroy(struct mem_cgroup *memcg)
 	}
 }
 #else
+static inline void memcg_register_kmem_events(struct cgroup *cont)
+{
+}
+
 static int memcg_init_kmem(struct mem_cgroup *memcg, struct cgroup_subsys *ss)
 {
 	return 0;
@@ -6732,8 +6770,10 @@ mem_cgroup_css_online(struct cgroup *cont)
 	struct mem_cgroup *memcg, *parent;
 	int error = 0;
 
-	if (!cont->parent)
+	if (!cont->parent) {
+		memcg_register_kmem_events(cont);
 		return 0;
+	}
 
 	mutex_lock(&memcg_create_mutex);
 	memcg = mem_cgroup_from_cont(cont);
Signature · 1 line
1.8.1.4
maintainer
#56 lore raw
In reply to Glauber Costa
Quoted 10 lines · Glauber Costa

Initial notes:
==============

Mel, Dave, this is the last round of fixes I have for the series. The fixes are few, and I was mostly interested in getting this out based on an up2date tree so Dave can verify it. This should apply fine ontop of Friday's linux-next. Alternatively, please grab from branch "kmemcg-lru-shrinker" at:

git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git

Quoted 7 lines

Main changes from *v5:
* Rebased to linux-next, and fix the conflicts with the dcache.
* Make sure LRU_RETRY only retry once
* Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function)
* Fix i915 return code when mutex cannot be acquired.
* Only scan less-than-batch objects in memcg scenarios

Ok, this is behaving a *lot* better than v5 in terms of initial balance and sustained behaviour under pure inode/dentry press workloads. The previous version was all over the place, not to mention unstable and prone to unrealted lockups in the block layer.

However, I'm not sure that the LRUness of reclaim is working correctly at this point. When I switch from a write only workload to a read-only workload (i.e. fsmark finishes and find starts), I see this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1923037 1807201 93% 1.12K 68729 28 2199328K xfs_inode
1914624 490812 25% 0.22K 53184 36 425472K xfs_ili

Note the xfs_ili slab capacity - there's half a million objects still in the cache, and they are only present on *dirty* inodes. Now, the read-only workload is iterating through a cold-cache lookup workload of 50 million inodes - at roughly 150,000/s. It's a touch-once workload, so shoul dbe turning the cache over completely every 10 seconds. However, in the time it's taken for me to explain this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1954493 1764661 90% 1.12K 69831 28 2234592K xfs_inode
1643868 281962 17% 0.22K 45663 36 365304K xfs_ili

Only 200k xfs_ili's have been freed. So the rate of reclaim of them is roughly 5k/s. Given the read-only nature of this workload, they should be gone from the cache in a few seconds. Another indication of problems here is the level of internal fragmentation of the xfs_ili slab. They should cycle out of the cache in LRU manner, just like inodes - the modify workload is a "touch once" workload as well, so there should be no internal fragmentation of the slab cache.

The stats I have of cache residency during the read-only part of the workload looks really bad. No steady state is reached, while on 3.9 a perfect steady state is reached within seconds and maintained until the workload changes. Part way through the read-only workload, this happened:

[  562.673080] sh (5007): dropped kernel caches: 3
[  629.617303] lowmemorykiller: send sigkill to 3953 (winbindd), adj 0, size 195
[  629.625499] lowmemorykiller: send sigkill to 3439 (pmcd), adj 0, size 158

And when the read-only workload finishes it's walk, I then start another "touch once" workload that removes all the files. that triggered:

[ 1002.183604] lowmemorykiller: send sigkill to 5827 (winbindd), adj 0, size 246
[ 1002.187822] lowmemorykiller: send sigkill to 3904 (winbindd), adj 0, size 134

Yeah, that stupid broken low memory killer is now kicking in, killing random processes - last run it killed two of the rm processes doing work, this time it killed winbindd and the PCP collection daemon that I use for remote stats monitoring.

So, yeah, there's still some broken stuff in this patchset that needs fixing. The script that I'm running to trigger these problems is pretty basic - it's the same workload I've been using for the past 3 years for measuring metadata performance of filesystems:

$ cat ./fsmark-50-test-xfs.sh 
#!/bin/bash

sudo umount /mnt/scratch > /dev/null 2>&1
sudo mkfs.xfs -f $@ -l size=131072b,sunit=8 /dev/vdc sudo mount -o nobarrier,logbsize=256k /dev/vdc /mnt/scratch sudo chmod 777 /mnt/scratch
cd /home/dave/src/fs_mark-3.3/

time ./fs_mark  -D  10000  -S0  -n  100000  -s  0  -L  63 \
        -d  /mnt/scratch/0  -d  /mnt/scratch/1 \
        -d  /mnt/scratch/2  -d  /mnt/scratch/3 \
        -d  /mnt/scratch/4  -d  /mnt/scratch/5 \
        -d  /mnt/scratch/6  -d  /mnt/scratch/7 \
        | tee >(stats --trim-outliers | tail -1 1>&2)

sync
sleep 30
sync

echo walking files
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time (

        for d in /mnt/scratch/[0-9]* ; do

                for i in $d/*; do
                        (
                                echo $i
                                find $i -ctime 1 > /dev/null
                        ) > /dev/null 2>&1
                done &
        done
        wait

)

echo removing files
for f in /mnt/scratch/* ; do time rm -rf $f & done wait
$

It's running on an 8p, 4GB RAM, 4-node fake numa virtual machine with a 100TB sparse image file being used for the test filesystem.

I'll spend some time over the next few days trying to work out what is causing these issues....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer
#57 lore raw
Quoted 28 lines · Dave Chinner
Quoted 8 lines · Glauber Costa

Initial notes: ==============

Mel, Dave, this is the last round of fixes I have for the series. The fixes are few, and I was mostly interested in getting this out based on an up2date tree so Dave can verify it. This should apply fine ontop of Friday's linux-next. Alternatively, please grab from branch "kmemcg-lru-shrinker" at:

Quoted 8 lines

git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git

Main changes from *v5: * Rebased to linux-next, and fix the conflicts with the dcache. * Make sure LRU_RETRY only retry once * Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function) * Fix i915 return code when mutex cannot be acquired. * Only scan less-than-batch objects in memcg scenarios

Ok, this is behaving a *lot* better than v5 in terms of initial balance and sustained behaviour under pure inode/dentry press workloads. The previous version was all over the place, not to mention unstable and prone to unrealted lockups in the block layer.

However, I'm not sure that the LRUness of reclaim is working correctly at this point. When I switch from a write only workload to a read-only workload (i.e. fsmark finishes and find starts), I see this:

....

Quoted 5 lines

So, yeah, there's still some broken stuff in this patchset that needs fixing. The script that I'm running to trigger these problems is pretty basic - it's the same workload I've been using for the past 3 years for measuring metadata performance of filesystems:

And unmounting an XFS filesystem after running this workload is hanging from time to time due to a reference counting problem on a buffer....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#58 lore raw
In reply to Dave Chinner
Quoted 19 lines · Dave Chinner
Quoted 10 lines · Glauber Costa

Initial notes:
==============

Mel, Dave, this is the last round of fixes I have for the series. The fixes are few, and I was mostly interested in getting this out based on an up2date tree so Dave can verify it. This should apply fine ontop of Friday's linux-next. Alternatively, please grab from branch "kmemcg-lru-shrinker" at:

git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git

Quoted 7 lines

Main changes from *v5:
* Rebased to linux-next, and fix the conflicts with the dcache.
* Make sure LRU_RETRY only retry once
* Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function)
* Fix i915 return code when mutex cannot be acquired.
* Only scan less-than-batch objects in memcg scenarios

Quoted 5 lines

Ok, this is behaving a *lot* better than v5 in terms of initial balance and sustained behaviour under pure inode/dentry press workloads. The previous version was all over the place, not to mention unstable and prone to unrealted lockups in the block layer.

Good to hear. About the problems you are seeing, if possible, I think it would beneficial to do what Mel did, and run a test workload without the upper half of the patches, IOW, excluding memcg. It should at least give us an indication about whether or not the problem lies in the way we are handling the LRU list, or in any memcg adaptation problem.

Quoted 11 lines

However, I'm not sure that the LRUness of reclaim is working correctly at this point. When I switch from a write only workload to a read-only workload (i.e. fsmark finishes and find starts), I see this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1923037 1807201 93% 1.12K 68729 28 2199328K xfs_inode
1914624 490812 25% 0.22K 53184 36 425472K xfs_ili

Note the xfs_ili slab capacity - there's half a million objects still in the cache, and they are only present on *dirty* inodes.

May a xfs-ignorant kernel hacker ask you what exactly xfs_ili is ?

Quoted 10 lines

Now, the read-only workload is iterating through a cold-cache lookup workload of 50 million inodes - at roughly 150,000/s. It's a touch-once workload, so shoul dbe turning the cache over completely every 10 seconds. However, in the time it's taken for me to explain this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1954493 1764661 90% 1.12K 69831 28 2234592K xfs_inode
1643868 281962 17% 0.22K 45663 36 365304K xfs_ili

Quoted 9 lines

Only 200k xfs_ili's have been freed. So the rate of reclaim of them is roughly 5k/s. Given the read-only nature of this workload, they should be gone from the cache in a few seconds. Another indication of problems here is the level of internal fragmentation of the xfs_ili slab. They should cycle out of the cache in LRU manner, just like inodes - the modify workload is a "touch once" workload as well, so there should be no internal fragmentation of the slab cache.

Initial testing I have done indicates - although it does not undoubtly prove - that the problem may be with dentries, not inodes. This is from attempted bisecting at the postmark benchmark, but as you said yourself, that kind of workload is not that bad any more, so this is unconclusive. In particular, the behavior of lru_walk mimics very well
the inode behavior, but for dentries, we have the following situation:

Before the patchset:
====================

relock:

while (!list_empty(&sb->s_dentry_lru)) {
        dentry = list_entry(sb->s_dentry_lru.prev,
                        struct dentry, d_lru);

        if (!spin_trylock(&dentry->d_lock)) {
                spin_unlock(&dcache_lru_lock);
                cpu_relax(); <== doesn't modify count
                goto relock;
        }

        if (dentry->d_flags & DCACHE_REFERENCED) {
                dentry->d_flags &= ~DCACHE_REFERENCED;
                list_move(&dentry->d_lru, &referenced);
                spin_unlock(&dentry->d_lock); <== doesn't modify count
        } else {
                list_move_tail(&dentry->d_lru, &tmp);
                dentry->d_flags |= DCACHE_SHRINK_LIST;
                spin_unlock(&dentry->d_lock);
                if (!--count)
                        break;
        }

The code we have in list_lru.c will decrement nr_to_walk in every situation except for LRU_RETRY.

There is also another problem I have just spotted while looking at this:

Differently from the experimental patch I sent Mel, the actual version of LRU_RETRY does not actually flip the flag, meaning the behavior is not what I envisioned.

You are seeing problems in the inode behavior, but since the dentry cache may pin them, I believe it is possible that the change in behavior in the dentry cache may drive the change in behavior in the inode cache, by keeping inodes that should be freed, pinned. (So far, just a theory)

Comments welcome.

Quoted 22 lines

The stats I have of cache residency during the read-only part of the workload looks really bad. No steady state is reached, while on 3.9 a perfect steady state is reached within seconds and maintained until the workload changes. Part way through the read-only workload, this happened:

[  562.673080] sh (5007): dropped kernel caches: 3
[  629.617303] lowmemorykiller: send sigkill to 3953 (winbindd), adj 0, size 195
[  629.625499] lowmemorykiller: send sigkill to 3439 (pmcd), adj 0, size 158

And when the read-only workload finishes it's walk, I then start another "touch once" workload that removes all the files. that triggered:

[ 1002.183604] lowmemorykiller: send sigkill to 5827 (winbindd), adj 0, size 246
[ 1002.187822] lowmemorykiller: send sigkill to 3904 (winbindd), adj 0, size 134

Yeah, that stupid broken low memory killer is now kicking in, killing random processes - last run it killed two of the rm processes doing work, this time it killed winbindd and the PCP collection daemon that I use for remote stats monitoring.

Quoted 5 lines

So, yeah, there's still some broken stuff in this patchset that needs fixing. The script that I'm running to trigger these problems is pretty basic - it's the same workload I've been using for the past 3 years for measuring metadata performance of filesystems:

I will try to run them myself, and I appreciate your diligence as well.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#59 lore raw
Quoted 37 lines · Glauber Costa
Quoted 19 lines · Dave Chinner
Quoted 10 lines · Glauber Costa

Initial notes:
==============

Mel, Dave, this is the last round of fixes I have for the series. The fixes are few, and I was mostly interested in getting this out based on an up2date tree so Dave can verify it. This should apply fine ontop of Friday's linux-next. Alternatively, please grab from branch "kmemcg-lru-shrinker" at:

git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg.git

Quoted 7 lines

Main changes from *v5:
* Rebased to linux-next, and fix the conflicts with the dcache.
* Make sure LRU_RETRY only retry once
* Prevent the bcache shrinker to scan the caches when disabled (by returning 0 in the count function)
* Fix i915 return code when mutex cannot be acquired.
* Only scan less-than-batch objects in memcg scenarios

Quoted 5 lines

Ok, this is behaving a *lot* better than v5 in terms of initial balance and sustained behaviour under pure inode/dentry press workloads. The previous version was all over the place, not to mention unstable and prone to unrealted lockups in the block layer.

Good to hear. About the problems you are seeing, if possible, I think it would beneficial to do what Mel did, and run a test workload without the upper half of the patches, IOW, excluding memcg. It should at least give us an indication about whether or not the problem lies in the way we are handling the LRU list, or in any memcg adaptation problem.

Quoted 5 lines

However, I'm not sure that the LRUness of reclaim is working correctly at this point. When I switch from a write only workload to a read-only workload (i.e. fsmark finishes and find starts), I see this:

Quoted 8 lines
Quoted 6 lines

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1923037 1807201 93% 1.12K 68729 28 2199328K xfs_inode
1914624 490812 25% 0.22K 53184 36 425472K xfs_ili

Note the xfs_ili slab capacity - there's half a million objects still in the cache, and they are only present on *dirty* inodes.

May a xfs-ignorant kernel hacker ask you what exactly xfs_ili is ?

struct xfs_inode_log_item.

It's the structure used to track the inode through the journalling subsystem, and it's only allocated when the inode is first modified. The embedded xfs_log_item is the abstraction used throughout the core transaction and journalling subsystems - every dirty metadata object in XFS has a xfs_log_item attached to it in some way.

Quoted 20 lines
Quoted 10 lines

Now, the read-only workload is iterating through a cold-cache lookup workload of 50 million inodes - at roughly 150,000/s. It's a touch-once workload, so shoul dbe turning the cache over completely every 10 seconds. However, in the time it's taken for me to explain this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1954493 1764661 90% 1.12K 69831 28 2234592K xfs_inode
1643868 281962 17% 0.22K 45663 36 365304K xfs_ili

Quoted 9 lines

Only 200k xfs_ili's have been freed. So the rate of reclaim of them is roughly 5k/s. Given the read-only nature of this workload, they should be gone from the cache in a few seconds. Another indication of problems here is the level of internal fragmentation of the xfs_ili slab. They should cycle out of the cache in LRU manner, just like inodes - the modify workload is a "touch once" workload as well, so there should be no internal fragmentation of the slab cache.

Quoted 2 lines

Initial testing I have done indicates - although it does not undoubtly prove - that the problem may be with dentries, not inodes

That tallies with the stats I'm seeing showing a significant difference in the balance of allocated vs "free" dentries. On a 3.9 kernel, the is little difference between them - dentries move quickly to the LRU and are considered free, while this patchset starts the same they quickly diverge, with the free count dropping well away from the allocated count.

....

Quoted 4 lines

You are seeing problems in the inode behavior, but since the dentry cache may pin them, I believe it is possible that the change in behavior in the dentry cache may drive the change in behavior in the inode cache, by keeping inodes that should be freed, pinned. (So far, just a theory)

Yup, that's the theory I'm working to given the inode used vs free count differences reflect the differences in the dentry behaviour...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
maintainer
#60 lore raw
Quoted 25 lines · Dave Chinner
Quoted 21 lines · Glauber Costa
Quoted 10 lines · Dave Chinner

Now, the read-only workload is iterating through a cold-cache lookup workload of 50 million inodes - at roughly 150,000/s. It's a touch-once workload, so shoul dbe turning the cache over completely every 10 seconds. However, in the time it's taken for me to explain this:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1954493 1764661 90% 1.12K 69831 28 2234592K xfs_inode
1643868 281962 17% 0.22K 45663 36 365304K xfs_ili

Quoted 9 lines

Only 200k xfs_ili's have been freed. So the rate of reclaim of them is roughly 5k/s. Given the read-only nature of this workload, they should be gone from the cache in a few seconds. Another indication of problems here is the level of internal fragmentation of the xfs_ili slab. They should cycle out of the cache in LRU manner, just like inodes - the modify workload is a "touch once" workload as well, so there should be no internal fragmentation of the slab cache.

Quoted 2 lines

Initial testing I have done indicates - although it does not undoubtly prove - that the problem may be with dentries, not inodes

Quoted 6 lines

That tallies with the stats I'm seeing showing a significant difference in the balance of allocated vs "free" dentries. On a 3.9 kernel, the is little difference between them - dentries move quickly to the LRU and are considered free, while this patchset starts the same they quickly diverge, with the free count dropping well away from the allocated count.

So, there's something early on going wrong in the patch set. This is from a tree at this patch in the series:

803d32a inode: convert inode lru list to generic lru list code.

Which is before the dentry cache is converted to the new LRU list code. So there's something wrong either in the underlying linux-next

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME

1894912 1610470  84%    0.06K  29608       64    118432K kmalloc-64
1894738 1660467  87%    1.12K  67696       28   2166272K xfs_inode
1892232 1633839  86%    0.22K  52562       36    420496K xfs_ili
1887962 1614100  85%    0.21K  51026       37    408208K dentry

562744 562191 99% 0.55K 20098 28 321568K radix_tree_node

And:

$ cat /proc/sys/fs/dentry-state 
1702143 96055   45      0       0       0
$

Which reflects this:

struct dentry_stat_t {
        int nr_dentry;
        int nr_unused;
        int age_limit;          /* age in seconds */
        int want_pages;         /* pages requested by system */
        int dummy[2];
};

Which basicaly says we have 1.7 million allocated dentrys, but only 100k dentries on the LRU lists. So there's something wrong either in the underlying linux-next tree, or the initial 3 dentry cache patches are now buggy.

<revert back to linux-next tree base>

<groan>

test-4 login: [ 71.106361] XFS (vdc): Mounting Filesystem

[   71.130097] XFS (vdc): Ending clean mount
[   91.980679] fs_mark (4394) used greatest stack depth: 3048 bytes left
[   92.286173] fs_mark (4396) used greatest stack depth: 3032 bytes left
[   92.340949] fs_mark (4397) used greatest stack depth: 3024 bytes left
[  120.162200] lowmemorykiller: send sigkill to 2948 (rsyslogd), adj 0, size 209
[  122.518167] fs_mark (4434) used greatest stack depth: 2952 bytes left
[  127.213331] lowmemorykiller: send sigkill to 3421 (pmcd), adj 0, size 202
[  165.402109] lowmemorykiller: send sigkill to 3302 (cron), adj 0, size 94
[  165.435809] lowmemorykiller: send sigkill to 1 (init), adj 0, size 87
[  169.003846] fs_mark (4484) used greatest stack depth: 2720 bytes left
[  189.093392] lowmemorykiller: send sigkill to 1 (init), adj 0, size 86
[  195.153252] lowmemorykiller: send sigkill to 1 (init), adj 0, size 80
[  209.016457] lowmemorykiller: send sigkill to 1 (init), adj 0, size 86
[  219.431805] lowmemorykiller: send sigkill to 1 (init), adj 0, size 86

So, the lowmemory killer is fucked up in the linux-next tree, not by this patchset. Before it killed pmcd, it looked like the dentry counters were running as per 3.9.0. Reboot, try again:

[   79.304611] lowmemorykiller: send sigkill to 4593 (fs_mark), adj 0, size 2121
[  131.334226] lowmemorykiller: send sigkill to 4647 (find), adj 0, size 7658
[  131.762285] lowmemorykiller: send sigkill to 4645 (find), adj 0, size 7658
[  131.858137] lowmemorykiller: send sigkill to 4653 (find), adj 0, size 7658
[  131.982366] lowmemorykiller: send sigkill to 4655 (find), adj 0, size 7658
[  132.455610] lowmemorykiller: send sigkill to 4657 (find), adj 0, size 7658
[  132.983835] lowmemorykiller: send sigkill to 4659 (find), adj 0, size 7658
[  133.136868] lowmemorykiller: send sigkill to 4661 (find), adj 0, size 7658
[  133.762004] lowmemorykiller: send sigkill to 4665 (find), adj 0, size 7658
[  139.666345] lowmemorykiller: send sigkill to 4685 (rm), adj 0, size 8195
[  142.964679] lowmemorykiller: send sigkill to 4691 (rm), adj 0, size 8195
[  154.573456] lowmemorykiller: send sigkill to 4686 (rm), adj 0, size 8293

Right, I'm turning that crap off.

Ok, that's more like what I expect:

$ cat /proc/sys/fs/dentry-state 
937104  929728  45      0       0       0
$ cat /proc/sys/fs/dentry-state 
1124254 1116881 45      0       0       0
$ cat /proc/sys/fs/dentry-state 
1256143 1248768 45      0       0       0
$ cat /proc/sys/fs/dentry-state 
761321  753937  45      0       0       0
$ cat /proc/sys/fs/dentry-state 
177308  169925  45      0       0       0
$ cat /proc/sys/fs/dentry-state 
614756  607371  45      0       0       0
$ cat /proc/sys/fs/dentry-state 
848316  840932  45      0       0       0

unused tracks allocated very closely.

So it's patch 4 that is broken:

dcache: remove dentries from LRU before putting on dispose list

I've found the problem. dentry_kill() returns the current dentry if it cannot lock the dentry->d_inode or the dentry->d_parent, and when that happens try_prune_one_dentry() silently fails to prune the dentry. But, at this point, we've already removed the dentry from both the LRU and the shrink list, and so it gets dropped on the floor.

patch 4 needs some work:

- fix the above leak shrink list leak
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readditional of dentry_lru_prune().

The reworked patch below does this.

Cheers,

Dave.

Signature · 50 lines
Dave Chinner
[email protected]

dcache: remove dentries from LRU before putting on dispose list

From: Dave Chinner <[email protected]>

One of the big problems with modifying the way the dcache shrinker
and LRU implementation works is that the LRU is abused in several
ways. One of these is shrink_dentry_list().

Basically, we can move a dentry off the LRU onto a different list
without doing any accounting changes, and then use dentry_lru_prune()
to remove it from what-ever list it is now on to do the LRU
accounting at that point.

This makes it -really hard- to change the LRU implementation. The
use of the per-sb LRU lock serialises movement of the dentries
between the different lists and the removal of them, and this is the
only reason that it works. If we want to break up the dentry LRU
lock and lists into, say, per-node lists, we remove the only
serialisation that allows this lru list/dispose list abuse to work.

To make this work effectively, the dispose list has to be isolated
from the LRU list - dentries have to be removed from the LRU
*before* being placed on the dispose list. This means that the LRU
accounting and isolation is completed before disposal is started,
and that means we can change the LRU implementation freely in
future.

This means that dentries *must* be marked with DCACHE_SHRINK_LIST
when they are placed on the dispose list so that we don't think that
parent dentries found in try_prune_one_dentry() are on the LRU when
the are actually on the dispose list. This would result in
accounting the dentry to the LRU a second time. Hence
dentry_lru_del() has to handle the DCACHE_SHRINK_LIST case
differently because the dentry isn't on the LRU list.

[ v2: don't decrement nr unused twice, spotted by Sha Zhengju ]
[ v7: (dchinner)
- shrink list leaks dentries when inode/parent can't be locked in
  dentry_kill().
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readdition of dentry_lru_prune(). ]

Signed-off-by: Dave Chinner <[email protected]>

---
 fs/dcache.c |   90 ++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 68 insertions(+), 22 deletions(-)
fs/dcache.c +68 -22
index 795c15d..edaf462 100644
@@ -315,7 +315,7 @@ static void dentry_unlink_inode(struct dentry * dentry)
 }
 
 /*
- * dentry_lru_(add|del|prune|move_tail) must be called with d_lock held.
+ * dentry_lru_(add|del|move_list) must be called with d_lock held.
  */
 static void dentry_lru_add(struct dentry *dentry)
 {
@@ -341,7 +341,8 @@ static void __dentry_lru_del(struct dentry *dentry)
  */
 static void dentry_lru_del(struct dentry *dentry)
 {
-	if (!list_empty(&dentry->d_lru)) {
+	if (!list_empty(&dentry->d_lru) &&
+	    !(dentry->d_flags & DCACHE_SHRINK_LIST)) {
 		spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 		__dentry_lru_del(dentry);
 		spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
@@ -350,13 +351,15 @@ static void dentry_lru_del(struct dentry *dentry)
 
 static void dentry_lru_move_list(struct dentry *dentry, struct list_head *list)
 {
+	BUG_ON(dentry->d_flags & DCACHE_SHRINK_LIST);
+
 	spin_lock(&dentry->d_sb->s_dentry_lru_lock);
 	if (list_empty(&dentry->d_lru)) {
 		list_add_tail(&dentry->d_lru, list);
-		dentry->d_sb->s_nr_dentry_unused++;
-		this_cpu_inc(nr_dentry_unused);
 	} else {
 		list_move_tail(&dentry->d_lru, list);
+		dentry->d_sb->s_nr_dentry_unused--;
+		this_cpu_dec(nr_dentry_unused);
 	}
 	spin_unlock(&dentry->d_sb->s_dentry_lru_lock);
 }
@@ -454,7 +457,8 @@ EXPORT_SYMBOL(d_drop);
  * If ref is non-zero, then decrement the refcount too.
  * Returns dentry requiring refcount drop, or NULL if we're done.
  */
-static inline struct dentry *dentry_kill(struct dentry *dentry, int ref)
+static inline struct dentry *
+dentry_kill(struct dentry *dentry, int ref, int unlock_on_failure)
 	__releases(dentry->d_lock)
 {
 	struct inode *inode;
@@ -463,8 +467,10 @@ static inline struct dentry *dentry_kill(struct dentry *dentry, int ref)
 	inode = dentry->d_inode;
 	if (inode && !spin_trylock(&inode->i_lock)) {
 relock:
-		spin_unlock(&dentry->d_lock);
-		cpu_relax();
+		if (unlock_on_failure) {
+			spin_unlock(&dentry->d_lock);
+			cpu_relax();
+		}
 		return dentry; /* try again with same dentry */
 	}
 	if (IS_ROOT(dentry))
@@ -551,7 +557,7 @@ repeat:
 	return;
 
 kill_it:
-	dentry = dentry_kill(dentry, 1);
+	dentry = dentry_kill(dentry, 1, 1);
 	if (dentry)
 		goto repeat;
 }
@@ -750,12 +756,12 @@ EXPORT_SYMBOL(d_prune_aliases);
  *
  * This may fail if locks cannot be acquired no problem, just try again.
  */
-static void try_prune_one_dentry(struct dentry *dentry)
+static struct dentry * try_prune_one_dentry(struct dentry *dentry)
 	__releases(dentry->d_lock)
 {
 	struct dentry *parent;
 
-	parent = dentry_kill(dentry, 0);
+	parent = dentry_kill(dentry, 0, 0);
 	/*
 	 * If dentry_kill returns NULL, we have nothing more to do.
 	 * if it returns the same dentry, trylocks failed. In either
@@ -767,9 +773,9 @@ static void try_prune_one_dentry(struct dentry *dentry)
 	 * fragmentation.
 	 */
 	if (!parent)
-		return;
+		return NULL;
 	if (parent == dentry)
-		return;
+		return dentry;
 
 	/* Prune ancestors. */
 	dentry = parent;
@@ -778,9 +784,9 @@ static void try_prune_one_dentry(struct dentry *dentry)
 		if (dentry->d_count > 1) {
 			dentry->d_count--;
 			spin_unlock(&dentry->d_lock);
-			return;
+			return NULL;
 		}
-		dentry = dentry_kill(dentry, 1);
+		dentry = dentry_kill(dentry, 1, 1);
 	}
 }
 
@@ -800,21 +806,31 @@ static void shrink_dentry_list(struct list_head *list)
 		}
 
 		/*
+		 * The dispose list is isolated and dentries are not accounted
+		 * to the LRU here, so we can simply remove it from the list
+		 * here regardless of whether it is referenced or not.
+		 */
+		list_del_init(&dentry->d_lru);
+		dentry->d_flags &= ~DCACHE_SHRINK_LIST;
+
+		/*
 		 * We found an inuse dentry which was not removed from
-		 * the LRU because of laziness during lookup.  Do not free
-		 * it - just keep it off the LRU list.
+		 * the LRU because of laziness during lookup. Do not free it.
 		 */
 		if (dentry->d_count) {
-			dentry_lru_del(dentry);
 			spin_unlock(&dentry->d_lock);
 			continue;
 		}
-
 		rcu_read_unlock();
 
-		try_prune_one_dentry(dentry);
+		dentry = try_prune_one_dentry(dentry);
 
 		rcu_read_lock();
+		if (dentry) {
+			dentry->d_flags |= DCACHE_SHRINK_LIST;
+			list_add(&dentry->d_lru, list);
+			spin_unlock(&dentry->d_lock);
+		}
 	}
 	rcu_read_unlock();
 }
@@ -855,8 +871,10 @@ relock:
 			list_move(&dentry->d_lru, &referenced);
 			spin_unlock(&dentry->d_lock);
 		} else {
-			list_move_tail(&dentry->d_lru, &tmp);
+			list_move(&dentry->d_lru, &tmp);
 			dentry->d_flags |= DCACHE_SHRINK_LIST;
+			this_cpu_dec(nr_dentry_unused);
+			sb->s_nr_dentry_unused--;
 			spin_unlock(&dentry->d_lock);
 			if (!--count)
 				break;
@@ -870,6 +888,27 @@ relock:
 	shrink_dentry_list(&tmp);
 }
 
+/*
+ * Mark all the dentries as on being the dispose list so we don't think they are
+ * still on the LRU if we try to kill them from ascending the parent chain in
+ * try_prune_one_dentry() rather than directly from the dispose list.
+ */
+static void
+shrink_dcache_list(
+	struct list_head *dispose)
+{
+	struct dentry *dentry;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(dentry, dispose, d_lru) {
+		spin_lock(&dentry->d_lock);
+		dentry->d_flags |= DCACHE_SHRINK_LIST;
+		spin_unlock(&dentry->d_lock);
+	}
+	rcu_read_unlock();
+	shrink_dentry_list(dispose);
+}
+
 /**
  * shrink_dcache_sb - shrink dcache for a superblock
  * @sb: superblock
@@ -883,9 +922,16 @@ void shrink_dcache_sb(struct super_block *sb)
 
 	spin_lock(&sb->s_dentry_lru_lock);
 	while (!list_empty(&sb->s_dentry_lru)) {
-		list_splice_init(&sb->s_dentry_lru, &tmp);
+		/*
+		 * account for removal here so we don't need to handle it later
+		 * even though the dentry is no longer on the lru list.
+		 */
 		spin_unlock(&sb->s_dentry_lru_lock);
-		shrink_dentry_list(&tmp);
+		list_splice_init(&sb->s_dentry_lru, &tmp);
+		this_cpu_sub(nr_dentry_unused, sb->s_nr_dentry_unused);
+		sb->s_nr_dentry_unused = 0;
+
+		shrink_dcache_list(&tmp);
 		spin_lock(&sb->s_dentry_lru_lock);
 	}
 	spin_unlock(&sb->s_dentry_lru_lock);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#61 lore raw
Quoted 1 line · Dave Chinner

On Tue, May 14, 2013 at 11:48:05AM +1000, Dave Chinner wrote:

...

Quoted 7 lines

patch 4 needs some work:

- fix the above leak shrink list leak
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readditional of dentry_lru_prune().

The reworked patch below does this.

And has a compile warning in it. And seeing as other patches need to be rebased as a result of this change, I'll post the new patches as responses to the patches that need to be modified....

Cheers,

Dave.

Signature · 8 lines
Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#62 lore raw
In reply to Dave Chinner
Quoted 7 lines · Dave Chinner

I've found the problem. dentry_kill() returns the current dentry if it cannot lock the dentry->d_inode or the dentry->d_parent, and when that happens try_prune_one_dentry() silently fails to prune the dentry. But, at this point, we've already removed the dentry from both the LRU and the shrink list, and so it gets dropped on the floor.

Great. I had already an idea that it had something to do with a dentry being removed from the LRU and not being put back, but I was looking at the wrong circumstance. oz, oz oi oi oi!

Quoted 5 lines

patch 4 needs some work:

- fix the above leak shrink list leak
- fix the scope of the sb locking inside shrink_dcache_sb()
- remove the readditional of dentry_lru_prune().

I readded this just because there are more work that needs to be done upon prune that is always the same. This is specially true in later patches, IIRC. I don't think dentry_lru_prune() has anything to do directly with the problem we are seeing now, and this is just a question of duplicated code vs not. But I am ultimately fine either way.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>