Skip to contentSkip to conversation

fs: Inode cache scalability

PATCH In review Dave Chinner 8 participants 111 messages Sep 29, 2010 – Oct 22, 2010

Subsystems (41)
lore
Share & tools
lore
#0 lore raw

cover fs: Inode cache scalability

This patch set is derived from Nick Piggin's VFS scalability tree. There doesn't appear to be any push to get that tree into shape for .37, so this is an attempt to start the process of finer grained review of the series for upstream inclusion. I'm hitting VFS lock contention problems with XFS on 8-16p machines now, so I need to get this stuff moving.

This patch set is just the basic inode_lock breakup patches plus a few more simple changes to the inode code. It stops short of introducing RCU inode freeing because those changes are not completely baked yet. It also stops short of changing the way inodes are tracked for writeback because I'd like not to spend my week after -rc1 is released fixing writeback again....

As a result, the full inode handling improvements of Nick's patch set are not realised with this short series. However, my own testing indicates that the amount of lock traffic and contention is down by an order of magnitude on an 8-way box for parallel inode create and unlink workloads, so there is still significant improvements in scalability from just this patch set.

I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.

Basically I'm trying to get this patchset ready for .37 (merge window is not really that far off now), and I'm aiming to have the rest of the inode changes (RCU freeing, writeback, etc) ready for .38. I may even look to some of the dcache changes for .38 depending on how much I can get tested and reviewed in that time frame.

Comments are welcome.

The current patchset is also available at the following location.

The following changes since commit b30a3f6257ed2105259b404d419b4964e363928c:

Linux 2.6.36-rc5 (2010-09-20 16:56:53 -0700)

are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git inode-scale

Eric Dumazet (2):
fs: inode per-cpu last_ino allocator
fs: Convert nr_inodes to a per-cpu counter

Nick Piggin (15):
kernel: add bl_list
fs: icache lock s_inodes list
fs: icache lock inode hash
fs: icache lock i_state
fs: icache lock i_count
fs: icache lock lru/writeback lists
fs: icache atomic inodes_stat
fs: icache protect inode state
fs: Make last_ino, iunique independent of inode_lock
fs: icache remove inode_lock
fs: Factor inode hash operations into functions
fs: Introduce per-bucket inode hash locks
fs: Implement lazy LRU updates for inodes.
fs: Inode counters do not need to be atomic.
fs: Clean up inode reference counting

 Documentation/filesystems/Locking        |    2 +-
 Documentation/filesystems/porting        |   10 +-
 Documentation/filesystems/vfs.txt        |    2 +-
 arch/powerpc/platforms/cell/spufs/file.c |    2 +-
 drivers/staging/pohmelfs/inode.c         |   14 +-
 fs/9p/vfs_inode.c                        |    4 +-
 fs/affs/inode.c                          |    2 +-
 fs/afs/dir.c                             |    4 +-
 fs/anon_inodes.c                         |    2 +-
 fs/bfs/dir.c                             |    2 +-
 fs/block_dev.c                           |    7 +-
 fs/btrfs/inode.c                         |   23 +-
 fs/buffer.c                              |    2 +-
 fs/ceph/mds_client.c                     |    2 +-
 fs/cifs/inode.c                          |    2 +-
 fs/coda/dir.c                            |    2 +-
 fs/drop_caches.c                         |   19 +-
 fs/exofs/inode.c                         |   10 +-
 fs/exofs/namei.c                         |    2 +-
 fs/ext2/namei.c                          |    2 +-
 fs/ext3/ialloc.c                         |    4 +-
 fs/ext3/namei.c                          |    2 +-
 fs/ext4/ialloc.c                         |    4 +-
 fs/ext4/namei.c                          |    2 +-
 fs/fs-writeback.c                        |  156 +++++---
 fs/gfs2/ops_inode.c                      |    2 +-
 fs/hfs/hfs_fs.h                          |    2 +-
 fs/hfs/inode.c                           |    2 +-
 fs/hfsplus/dir.c                         |    2 +-
 fs/hfsplus/hfsplus_fs.h                  |    2 +-
 fs/hfsplus/inode.c                       |    2 +-
 fs/hpfs/inode.c                          |    2 +-
 fs/inode.c                               |  603 ++++++++++++++++++++----------
 fs/jffs2/dir.c                           |    4 +-
 fs/jfs/jfs_txnmgr.c                      |    2 +-
 fs/jfs/namei.c                           |    2 +-
 fs/libfs.c                               |    2 +-
 fs/locks.c                               |    2 +-
 fs/logfs/dir.c                           |    2 +-
 fs/logfs/inode.c                         |    2 +-
 fs/logfs/readwrite.c                     |    6 +-
 fs/minix/namei.c                         |    2 +-
 fs/namei.c                               |    2 +-
 fs/nfs/dir.c                             |    2 +-
 fs/nfs/getroot.c                         |    4 +-
 fs/nfs/inode.c                           |    4 +-
 fs/nfs/nfs4state.c                       |    2 +-
 fs/nfs/write.c                           |    2 +-
 fs/nilfs2/gcdat.c                        |    1 +
 fs/nilfs2/gcinode.c                      |   22 +-
 fs/nilfs2/mdt.c                          |    2 +-
 fs/nilfs2/namei.c                        |    2 +-
 fs/nilfs2/segment.c                      |    2 +-
 fs/nilfs2/the_nilfs.h                    |    2 +-
 fs/notify/inode_mark.c                   |   46 ++-
 fs/notify/mark.c                         |    1 -
 fs/notify/vfsmount_mark.c                |    1 -
 fs/ntfs/inode.c                          |    4 +-
 fs/ntfs/super.c                          |    2 +-
 fs/ocfs2/inode.c                         |    2 +-
 fs/ocfs2/namei.c                         |    2 +-
 fs/quota/dquot.c                         |   36 +-
 fs/reiserfs/namei.c                      |    2 +-
 fs/reiserfs/stree.c                      |    2 +-
 fs/reiserfs/xattr.c                      |    2 +-
 fs/sysv/namei.c                          |    2 +-
 fs/ubifs/dir.c                           |    2 +-
 fs/ubifs/super.c                         |    2 +-
 fs/udf/namei.c                           |    2 +-
 fs/ufs/namei.c                           |    2 +-
 fs/xfs/linux-2.6/xfs_iops.c              |    2 +-
 fs/xfs/linux-2.6/xfs_trace.h             |    2 +-
 fs/xfs/xfs_inode.h                       |    4 +-
 include/linux/fs.h                       |   54 ++-
 include/linux/list_bl.h                  |  127 +++++++
 include/linux/rculist_bl.h               |  128 +++++++
 include/linux/writeback.h                |    4 +-
 ipc/mqueue.c                             |    2 +-
 kernel/futex.c                           |    2 +-
 kernel/sysctl.c                          |    4 +-
 mm/backing-dev.c                         |    8 +-
 mm/filemap.c                             |    6 +-
 mm/rmap.c                                |    6 +-
 mm/shmem.c                               |    6 +-
 net/socket.c                             |    2 +-
 85 files changed, 1001 insertions(+), 437 deletions(-)
 create mode 100644 include/linux/list_bl.h
 create mode 100644 include/linux/rculist_bl.h
#1 lore raw

PATCH 1/17 kernel: add bl_list

From: Nick Piggin <[email protected]>

Introduce a type of hlist that can support the use of the lowest bit in the hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes.

---
 include/linux/list_bl.h    |  127 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/rculist_bl.h |  128 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 255 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/list_bl.h
 create mode 100644 include/linux/rculist_bl.h
include/linux/list_bl.h new file +127 -0
new file mode 100644
index 0000000..cf8acfc
@@ -0,0 +1,127 @@
+#ifndef _LINUX_LIST_BL_H
+#define _LINUX_LIST_BL_H
+
+#include <linux/list.h>
+#include <linux/bit_spinlock.h>
+
+/*
+ * Special version of lists, where head of the list has a bit spinlock
+ * in the lowest bit. This is useful for scalable hash tables without
+ * increasing memory footprint overhead.
+ *
+ * For modification operations, the 0 bit of hlist_bl_head->first
+ * pointer must be set.
+ */
+
+#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK)
+#define LIST_BL_LOCKMASK	1UL
+#else
+#define LIST_BL_LOCKMASK	0UL
+#endif
+
+#ifdef CONFIG_DEBUG_LIST
+#define LIST_BL_BUG_ON(x) BUG_ON(x)
+#else
+#define LIST_BL_BUG_ON(x)
+#endif
+
+
+struct hlist_bl_head {
+	struct hlist_bl_node *first;
+};
+
+struct hlist_bl_node {
+	struct hlist_bl_node *next, **pprev;
+};
+#define INIT_HLIST_BL_HEAD(ptr) \
+	((ptr)->first = NULL)
+
+static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
+{
+	h->next = NULL;
+	h->pprev = NULL;
+}
+
+#define hlist_bl_entry(ptr, type, member) container_of(ptr,type,member)
+
+static inline int hlist_bl_unhashed(const struct hlist_bl_node *h)
+{
+	return !h->pprev;
+}
+
+static inline struct hlist_bl_node *hlist_bl_first(struct hlist_bl_head *h)
+{
+	return (struct hlist_bl_node *)
+		((unsigned long)h->first & ~LIST_BL_LOCKMASK);
+}
+
+static inline void hlist_bl_set_first(struct hlist_bl_head *h,
+					struct hlist_bl_node *n)
+{
+	LIST_BL_BUG_ON((unsigned long)n & LIST_BL_LOCKMASK);
+	LIST_BL_BUG_ON(!bit_spin_is_locked(0, (unsigned long *)&h->first));
+	h->first = (struct hlist_bl_node *)((unsigned long)n | LIST_BL_LOCKMASK);
+}
+
+static inline int hlist_bl_empty(const struct hlist_bl_head *h)
+{
+	return !((unsigned long)h->first & ~LIST_BL_LOCKMASK);
+}
+
+static inline void hlist_bl_add_head(struct hlist_bl_node *n,
+					struct hlist_bl_head *h)
+{
+	struct hlist_bl_node *first = hlist_bl_first(h);
+
+	n->next = first;
+	if (first)
+		first->pprev = &n->next;
+	n->pprev = &h->first;
+	hlist_bl_set_first(h, n);
+}
+
+static inline void __hlist_bl_del(struct hlist_bl_node *n)
+{
+	struct hlist_bl_node *next = n->next;
+	struct hlist_bl_node **pprev = n->pprev;
+
+	LIST_BL_BUG_ON((unsigned long)n & LIST_BL_LOCKMASK);
+
+	/* pprev may be `first`, so be careful not to lose the lock bit */
+	*pprev = (struct hlist_bl_node *)
+			((unsigned long)next |
+			 ((unsigned long)*pprev & LIST_BL_LOCKMASK));
+	if (next)
+		next->pprev = pprev;
+}
+
+static inline void hlist_bl_del(struct hlist_bl_node *n)
+{
+	__hlist_bl_del(n);
+	n->next = LIST_POISON1;
+	n->pprev = LIST_POISON2;
+}
+
+static inline void hlist_bl_del_init(struct hlist_bl_node *n)
+{
+	if (!hlist_bl_unhashed(n)) {
+		__hlist_bl_del(n);
+		INIT_HLIST_BL_NODE(n);
+	}
+}
+
+/**
+ * hlist_bl_for_each_entry	- iterate over list of given type
+ * @tpos:	the type * to use as a loop cursor.
+ * @pos:	the &struct hlist_node to use as a loop cursor.
+ * @head:	the head for your list.
+ * @member:	the name of the hlist_node within the struct.
+ *
+ */
+#define hlist_bl_for_each_entry(tpos, pos, head, member)		\
+	for (pos = hlist_bl_first(head);				\
+	     pos &&							\
+		({ tpos = hlist_bl_entry(pos, typeof(*tpos), member); 1;}); \
+	     pos = pos->next)
+
+#endif
include/linux/rculist_bl.h new file +128 -0

128 lines not on this page · raw message

Signature · 1 line
1.7.1
maintainer
#2 lore raw

Re: [PATCH 01/17] kernel: add bl_list

Quoted 9 lines · On Wed, 29 Sep 2010 22:18:33 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

Introduce a type of hlist that can support the use of the lowest bit in the hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes.

...

include/linux/list_bl.h line 43

Quoted 5 lines
include/linux/list_bl.h new file +5 -0
+static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
+{
+	h->next = NULL;
+	h->pprev = NULL;
+}

No need to shout.

include/linux/list_bl.h line 103

Quoted 9 lines

...

include/linux/list_bl.h new file +6 -0
+static inline void hlist_bl_del(struct hlist_bl_node *n)
+{
+	__hlist_bl_del(n);
+	n->next = LIST_POISON1;
+	n->pprev = LIST_POISON2;
+}

I'd suggest creating new poison values for hlist_bl's, leave LIST_POISON1 and LIST_POISON2 for list_head (and any other list variants which went and used them :()

Quoted 3 lines

...

#3 lore raw

Re: [PATCH 01/17] kernel: add bl_list

Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Wed, 29 Sep 2010 22:18:33 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

Introduce a type of hlist that can support the use of the lowest bit in the hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes.

...

Quoted 7 lines

include/linux/list_bl.h line 43

Quoted 5 lines
include/linux/list_bl.h new file +5 -0
+static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
+{
+	h->next = NULL;
+	h->pprev = NULL;
+}

No need to shout.

Just following the rest of the lists.

Quoted 12 lines

include/linux/list_bl.h line 103

Quoted 8 lines

...

include/linux/list_bl.h new file +6 -0
+static inline void hlist_bl_del(struct hlist_bl_node *n)
+{
+	__hlist_bl_del(n);
+	n->next = LIST_POISON1;
+	n->pprev = LIST_POISON2;
+}

I'd suggest creating new poison values for hlist_bl's, leave LIST_POISON1 and LIST_POISON2 for list_head (and any other list variants which went and used them :()

I guess they're used for lists, hlists, nulls lists. Would it really help much seeing as you have so many lists anyway? I guess we could do an incremental patch but I'll postpone it for now.

#4 lore raw

Re: [PATCH 01/17] kernel: add bl_list

Quoted 9 lines · Nick Piggin
Quoted 7 lines

include/linux/list_bl.h line 43

Quoted 5 lines
include/linux/list_bl.h new file +5 -0
+static inline void INIT_HLIST_BL_NODE(struct hlist_bl_node *h)
+{
+	h->next = NULL;
+	h->pprev = NULL;
+}

No need to shout.

Just following the rest of the lists.

Yes, just following hlist is logical. Anyway, I don't think this or the poisoning value really matter very much.

#6 lore raw

PATCH 2/17 fs: icache lock s_inodes list

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

To allow removal of the inode_lock, we first need to protect the superblock inode list with it's own lock instead of using the inode_lock for this purpose. Nest the new sb_inode_list_lock inside the inode_lock around the list operations it needs to protect.

---
 fs/drop_caches.c          |    4 ++++
 fs/fs-writeback.c         |    4 ++++
 fs/inode.c                |   19 +++++++++++++++++++
 fs/notify/inode_mark.c    |    2 ++
 fs/quota/dquot.c          |    6 ++++++
 include/linux/writeback.h |    1 +
 6 files changed, 36 insertions(+), 0 deletions(-)
fs/drop_caches.c +4 -0
index 2195c21..ab69ae7 100644
@@ -17,18 +17,22 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)
 	struct inode *inode, *toput_inode = NULL;
 
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
 			continue;
 		if (inode->i_mapping->nrpages == 0)
 			continue;
 		__iget(inode);
+		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 		invalidate_mapping_pages(inode->i_mapping, 0, -1);
 		iput(toput_inode);
 		toput_inode = inode;
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
 	}
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 	iput(toput_inode);
 }
fs/fs-writeback.c +4 -0
index 81e086d..9adc9d9 100644
@@ -1021,6 +1021,7 @@ static void wait_sb_inodes(struct super_block *sb)
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
 
 	/*
 	 * Data integrity sync. Must wait for all pages under writeback,
@@ -1038,6 +1039,7 @@ static void wait_sb_inodes(struct super_block *sb)
 		if (mapping->nrpages == 0)
 			continue;
 		__iget(inode);
+		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 		/*
 		 * We hold a reference to 'inode' so it couldn't have
@@ -1055,7 +1057,9 @@ static void wait_sb_inodes(struct super_block *sb)
 		cond_resched();
 
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
 	}
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 	iput(old_inode);
 }
fs/inode.c +19 -0
index 8646433..ca98254 100644
@@ -26,6 +26,15 @@
 #include <linux/posix_acl.h>
 
 /*
+ * Usage:
+ * sb_inode_list_lock protects:
+ *   s_inodes, i_sb_list
+ *
+ * Ordering:
+ * inode_lock
+ *   sb_inode_list_lock
+ */
+/*
  * This is needed for the following functions:
  *  - inode_has_buffers
  *  - invalidate_inode_buffers
@@ -83,6 +92,7 @@ static struct hlist_head *inode_hashtable __read_mostly;
  * the i_state of an inode while it is in use..
  */
 DEFINE_SPINLOCK(inode_lock);
+DEFINE_SPINLOCK(sb_inode_list_lock);
 
 /*
  * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -339,7 +349,9 @@ static void dispose_list(struct list_head *head)
 
 		spin_lock(&inode_lock);
 		hlist_del_init(&inode->i_hash);
+		spin_lock(&sb_inode_list_lock);
 		list_del_init(&inode->i_sb_list);
+		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 
 		wake_up_inode(inode);
@@ -371,6 +383,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 		 * shrink_icache_memory() away.
 		 */
 		cond_resched_lock(&inode_lock);
+		cond_resched_lock(&sb_inode_list_lock);
 
 		next = next->next;
 		if (tmp == head)
@@ -408,8 +421,10 @@ int invalidate_inodes(struct super_block *sb)
 
 	down_write(&iprune_sem);
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
 	fsnotify_unmount_inodes(&sb->s_inodes);
 	busy = invalidate_list(&sb->s_inodes, &throw_away);
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 
 	dispose_list(&throw_away);
@@ -597,7 +612,9 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
 {
 	inodes_stat.nr_inodes++;
 	list_add(&inode->i_list, &inode_in_use);
+	spin_lock(&sb_inode_list_lock);
 	list_add(&inode->i_sb_list, &sb->s_inodes);
+	spin_unlock(&sb_inode_list_lock);
 	if (head)
 		hlist_add_head(&inode->i_hash, head);
 }
@@ -1231,7 +1248,9 @@ static void iput_final(struct inode *inode)
 		hlist_del_init(&inode->i_hash);
 	}
 	list_del_init(&inode->i_list);
+	spin_lock(&sb_inode_list_lock);
 	list_del_init(&inode->i_sb_list);
+	spin_unlock(&sb_inode_list_lock);
 	WARN_ON(inode->i_state & I_NEW);
 	inode->i_state |= I_FREEING;
 	inodes_stat.nr_inodes--;
fs/notify/inode_mark.c +2 -0
index 33297c0..34b1585 100644
@@ -283,6 +283,7 @@ void fsnotify_unmount_inodes(struct list_head *list)
 		 * will be added since the umount has begun.  Finally,
 		 * iprune_mutex keeps shrink_icache_memory() away.
 		 */
+		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 
 		if (need_iput_tmp)
@@ -296,5 +297,6 @@ void fsnotify_unmount_inodes(struct list_head *list)
 		iput(inode);
 
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
 	}
 }
fs/quota/dquot.c +6 -0
index aad1316..2e3b913 100644
@@ -897,6 +897,7 @@ static void add_dquot_ref(struct super_block *sb, int type)
 #endif
 
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
 			continue;
@@ -910,6 +911,7 @@ static void add_dquot_ref(struct super_block *sb, int type)
 			continue;
 
 		__iget(inode);
+		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 
 		iput(old_inode);
@@ -921,7 +923,9 @@ static void add_dquot_ref(struct super_block *sb, int type)
 		 * keep the reference and iput it later. */
 		old_inode = inode;
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
 	}
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 	iput(old_inode);
 
@@ -1004,6 +1008,7 @@ static void remove_dquot_ref(struct super_block *sb, int type,
 	int reserved = 0;
 
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		/*
 		 *  We have to scan also I_NEW inodes because they can already
@@ -1017,6 +1022,7 @@ static void remove_dquot_ref(struct super_block *sb, int type,
 			remove_inode_dquot_ref(inode, type, tofree_head);
 		}
 	}
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 #ifdef CONFIG_QUOTA_DEBUG
 	if (reserved) {
include/linux/writeback.h +1 -0
index 72a5d64..9974edb 100644
@@ -10,6 +10,7 @@
 struct backing_dev_info;
 
 extern spinlock_t inode_lock;
+extern spinlock_t sb_inode_list_lock;
 extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
Signature · 1 line
1.7.1
#7 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 6 lines · Dave Chinner

From: Nick Piggin <[email protected]>

To allow removal of the inode_lock, we first need to protect the superblock inode list with it's own lock instead of using the inode_lock for this purpose. Nest the new sb_inode_list_lock inside the inode_lock around the list operations it needs to protect.

Is there any good reason not to make this lock per-superblock?

#8 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner

From: Nick Piggin <[email protected]>

To allow removal of the inode_lock, we first need to protect the superblock inode list with it's own lock instead of using the inode_lock for this purpose. Nest the new sb_inode_list_lock inside the inode_lock around the list operations it needs to protect.

Is there any good reason not to make this lock per-superblock?

Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what inode_lock used to.

If we did want to make it per-superblock, that would come at the last part of the series, where inode_lock is removed and steps are being taken to improve scalability and locking.

But I don't see why we want to make it per-superblock really anyway. We want to have scalability within a single superblock, so per CPU locks are needed. Once we have those, per-superblock doesn't really buy much.

#9 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 3 lines · Nick Piggin

Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what inode_lock used to.

As seen by Dave's respin making it per-sb was just as easy as making it global. And it really is the logical synchronization domain.

#10 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Nick Piggin

Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what inode_lock used to.

As seen by Dave's respin making it per-sb was just as easy as making it global. And it really is the logical synchronization domain.

If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.

And the entire idea of the first half of the inode series is that it starts simple and just uses globals to demonstrate the locking steps. It's obviously not supposed to be a "production" locking model so I prefer it to be like that.

#11 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 3 lines · Nick Piggin

If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.

Right now the patches split up the inode lock and do not add per-cpu magic. It's not any more work to move from per-sb lists to per-cpu locking if we eventually do it than moving from global to per-cpu.

I'm not entirely convinced moving s_inodes to a per-cpu list is a good idea. For now per-sb is just fine for disk filesystems as they have much more fs-wide cachelines they touch for inode creatation/deletion anyway, and for sockets/pipes a variant of your patch to not ever add them to s_inodes sounds like the better approach.

If we eventually hit the limit for disk filesystems I have some better ideas to solve this. One is to abuse whatever data sturcture we use for the inode hash also for iterating over all inodes - we only iterate over them in very few places, and none of them is a fast path.

#12 lore raw

Re: [PATCH 02/17] fs: icache lock s_inodes list

Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Nick Piggin

If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.

Right now the patches split up the inode lock and do not add per-cpu magic. It's not any more work to move from per-sb lists to per-cpu locking if we eventually do it than moving from global to per-cpu.

But it's more work to do per-sb lists than a global list, and as I'm going to a per-cpu locking anyway it's a strange transition to go from per-sb to per-cpu (rather than per-sb, per-cpu). In short, the fact that I build up the locking transformations starting with global locks is just not something that can be held against my patch set (unless you really disagree with the whole concept of how the series is structured).

Quoted 6 lines

I'm not entirely convinced moving s_inodes to a per-cpu list is a good idea. For now per-sb is just fine for disk filesystems as they have much more fs-wide cachelines they touch for inode creatation/deletion anyway, and for sockets/pipes a variant of your patch to not ever add them to s_inodes sounds like the better approach.

Traditional filesystems on slow spinning disk are not the main problem. It's very fast ssds and storage servers. XFS actually with its per-AG lock splitting can already have problems on small servers with not-incredibly-fast storage with per-sb scalability bottlenecks.

And if the VFS is not scalable, then the contention doesn't even get pushed into the filesystem so the fs developers never even _see_ the locking problems to fix them.

I'm telling you it will be increasingly a problem because cores and storage speeds continue to increase, and also people want to manage more storage with fewer filesystems. It's obvious that it will be a problem.

I've already got per cpu locking in vfsmounts and files lock, so it's not magic.

Quoted 4 lines

If we eventually hit the limit for disk filesystems I have some better ideas to solve this. One is to abuse whatever data sturcture we use for the inode hash also for iterating over all inodes - we only iterate over them in very few places, and none of them is a fast path.

Doing your handwaving about changing data types and better ideas is just not helpful. _If_ you do have some better ideas, and _if_ we change the data structure, _then_ it's trivial to change from percpu locking to your better idea. It just doesn't work as an argument to slow progress.

#13 lore raw

PATCH 3/17 fs: icache lock inode hash

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Currently the inode hash lists are protected by the inode_lock. To allow removal of the inode_lock, we need to protect the inode hash table lists with a new lock. Nest the new inode_hash_lock inside the inode_lock to protect the hash lists.

---
 fs/inode.c                |   33 ++++++++++++++++++++++++++++++++-
 include/linux/writeback.h |    1 +
 2 files changed, 33 insertions(+), 1 deletions(-)
fs/inode.c +32 -1
index ca98254..9d7ffb1 100644
@@ -29,10 +29,14 @@
  * Usage:
  * sb_inode_list_lock protects:
  *   s_inodes, i_sb_list
+ * inode_hash_lock protects:
+ *   inode hash table, i_hash
  *
  * Ordering:
  * inode_lock
  *   sb_inode_list_lock
+ * inode_lock
+ *   inode_hash_lock
  */
 /*
  * This is needed for the following functions:
@@ -93,6 +97,7 @@ static struct hlist_head *inode_hashtable __read_mostly;
  */
 DEFINE_SPINLOCK(inode_lock);
 DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);
 
 /*
  * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -348,7 +353,9 @@ static void dispose_list(struct list_head *head)
 		evict(inode);
 
 		spin_lock(&inode_lock);
+		spin_lock(&inode_hash_lock);
 		hlist_del_init(&inode->i_hash);
+		spin_unlock(&inode_hash_lock);
 		spin_lock(&sb_inode_list_lock);
 		list_del_init(&inode->i_sb_list);
 		spin_unlock(&sb_inode_list_lock);
@@ -557,17 +564,20 @@ static struct inode *find_inode(struct super_block *sb,
 	struct inode *inode = NULL;
 
 repeat:
+	spin_lock(&inode_hash_lock);
 	hlist_for_each_entry(inode, node, head, i_hash) {
 		if (inode->i_sb != sb)
 			continue;
 		if (!test(inode, data))
 			continue;
 		if (inode->i_state & (I_FREEING|I_WILL_FREE)) {
+			spin_unlock(&inode_hash_lock);
 			__wait_on_freeing_inode(inode);
 			goto repeat;
 		}
 		break;
 	}
+	spin_unlock(&inode_hash_lock);
 	return node ? inode : NULL;
 }
 
@@ -582,17 +592,20 @@ static struct inode *find_inode_fast(struct super_block *sb,
 	struct inode *inode = NULL;
 
 repeat:
+	spin_lock(&inode_hash_lock);
 	hlist_for_each_entry(inode, node, head, i_hash) {
 		if (inode->i_ino != ino)
 			continue;
 		if (inode->i_sb != sb)
 			continue;
 		if (inode->i_state & (I_FREEING|I_WILL_FREE)) {
+			spin_unlock(&inode_hash_lock);
 			__wait_on_freeing_inode(inode);
 			goto repeat;
 		}
 		break;
 	}
+	spin_unlock(&inode_hash_lock);
 	return node ? inode : NULL;
 }
 
@@ -615,8 +628,11 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
 	spin_lock(&sb_inode_list_lock);
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
-	if (head)
+	if (head) {
+		spin_lock(&inode_hash_lock);
 		hlist_add_head(&inode->i_hash, head);
+		spin_unlock(&inode_hash_lock);
+	}
 }
 
 /**
@@ -1094,7 +1110,9 @@ int insert_inode_locked(struct inode *inode)
 	while (1) {
 		struct hlist_node *node;
 		struct inode *old = NULL;
+
 		spin_lock(&inode_lock);
+		spin_lock(&inode_hash_lock);
 		hlist_for_each_entry(old, node, head, i_hash) {
 			if (old->i_ino != ino)
 				continue;
@@ -1106,9 +1124,11 @@ int insert_inode_locked(struct inode *inode)
 		}
 		if (likely(!node)) {
 			hlist_add_head(&inode->i_hash, head);
+			spin_unlock(&inode_hash_lock);
 			spin_unlock(&inode_lock);
 			return 0;
 		}
+		spin_unlock(&inode_hash_lock);
 		__iget(old);
 		spin_unlock(&inode_lock);
 		wait_on_inode(old);
@@ -1134,6 +1154,7 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,
 		struct inode *old = NULL;
 
 		spin_lock(&inode_lock);
+		spin_lock(&inode_hash_lock);
 		hlist_for_each_entry(old, node, head, i_hash) {
 			if (old->i_sb != sb)
 				continue;
@@ -1145,9 +1166,11 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,
 		}
 		if (likely(!node)) {
 			hlist_add_head(&inode->i_hash, head);
+			spin_unlock(&inode_hash_lock);
 			spin_unlock(&inode_lock);
 			return 0;
 		}
+		spin_unlock(&inode_hash_lock);
 		__iget(old);
 		spin_unlock(&inode_lock);
 		wait_on_inode(old);
@@ -1172,7 +1195,9 @@ void __insert_inode_hash(struct inode *inode, unsigned long hashval)
 {
 	struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
 	spin_lock(&inode_lock);
+	spin_lock(&inode_hash_lock);
 	hlist_add_head(&inode->i_hash, head);
+	spin_unlock(&inode_hash_lock);
 	spin_unlock(&inode_lock);
 }
 EXPORT_SYMBOL(__insert_inode_hash);
@@ -1186,7 +1211,9 @@ EXPORT_SYMBOL(__insert_inode_hash);
 void remove_inode_hash(struct inode *inode)
 {
 	spin_lock(&inode_lock);
+	spin_lock(&inode_hash_lock);
 	hlist_del_init(&inode->i_hash);
+	spin_unlock(&inode_hash_lock);
 	spin_unlock(&inode_lock);
 }
 EXPORT_SYMBOL(remove_inode_hash);
@@ -1245,7 +1272,9 @@ static void iput_final(struct inode *inode)
 		WARN_ON(inode->i_state & I_NEW);
 		inode->i_state &= ~I_WILL_FREE;
 		inodes_stat.nr_unused--;
+		spin_lock(&inode_hash_lock);
 		hlist_del_init(&inode->i_hash);
+		spin_unlock(&inode_hash_lock);
 	}
 	list_del_init(&inode->i_list);
 	spin_lock(&sb_inode_list_lock);
@@ -1257,7 +1286,9 @@ static void iput_final(struct inode *inode)
 	spin_unlock(&inode_lock);
 	evict(inode);
 	spin_lock(&inode_lock);
+	spin_lock(&inode_hash_lock);
 	hlist_del_init(&inode->i_hash);
+	spin_unlock(&inode_hash_lock);
 	spin_unlock(&inode_lock);
 	wake_up_inode(inode);
 	BUG_ON(inode->i_state != (I_FREEING | I_CLEAR));
include/linux/writeback.h +1 -0
index 9974edb..35d6e81 100644
@@ -11,6 +11,7 @@ struct backing_dev_info;
 
 extern spinlock_t inode_lock;
 extern spinlock_t sb_inode_list_lock;
+extern spinlock_t inode_hash_lock;
 extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
Signature · 1 line
1.7.1
maintainer
#14 lore raw

Re: [PATCH 03/17] fs: icache lock inode hash

Quoted 3 lines · On Wed, 29 Sep 2010 22:18:35 +1000 Dave Chinner <[email protected]> wrote:

DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);

I assume these all go away later on. If not, they'll probably all land in the same cacheline!

#15 lore raw

Re: [PATCH 03/17] fs: icache lock inode hash

Quoted 8 lines · Andrew Morton
Quoted 3 lines · On Wed, 29 Sep 2010 22:18:35 +1000 Dave Chinner <[email protected]> wrote:

DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);

I assume these all go away later on. If not, they'll probably all land in the same cacheline!

indeed, the hash lock goes away in this series, as does the inode_lock. I'll check the other new locks don't land in the same cacheline.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#16 lore raw

Re: [PATCH 03/17] fs: icache lock inode hash

In reply to Dave Chinner
Quoted 6 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Currently the inode hash lists are protected by the inode_lock. To allow removal of the inode_lock, we need to protect the inode hash table lists with a new lock. Nest the new inode_hash_lock inside the inode_lock to protect the hash lists.

There is no reason to make inode_hash_lock global, it's only used inside inode.c

#17 lore raw

Re: [PATCH 03/17] fs: icache lock inode hash

Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Currently the inode hash lists are protected by the inode_lock. To allow removal of the inode_lock, we need to protect the inode hash table lists with a new lock. Nest the new inode_hash_lock inside the inode_lock to protect the hash lists.

There is no reason to make inode_hash_lock global, it's only used inside inode.c

It was like that while hugetlbfs still used it. Fixed, thanks.

#18 lore raw

PATCH 4/17 fs: icache lock i_state

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

We currently protect the per-inode state flags with the inode_lock. Using a global lock to protect per-object state is overkill when we coul duse a per-inode lock to protect the state. Use the inode->i_lock for this, and wrap all the state changes and checks with the inode->i_lock.

---
 fs/drop_caches.c  |    9 ++++--
 fs/fs-writeback.c |   29 +++++++++++++++--
 fs/inode.c        |   86 +++++++++++++++++++++++++++++++++++++++++++++-------
 fs/nilfs2/gcdat.c |    1 +
 fs/quota/dquot.c  |   12 ++++---
 5 files changed, 113 insertions(+), 24 deletions(-)
fs/drop_caches.c +6 -3
index ab69ae7..45bdf88 100644
@@ -19,11 +19,14 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)
 	spin_lock(&inode_lock);
 	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
-			continue;
-		if (inode->i_mapping->nrpages == 0)
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)
+				|| inode->i_mapping->nrpages == 0) {
+			spin_unlock(&inode->i_lock);
 			continue;
+		}
 		__iget(inode);
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 		invalidate_mapping_pages(inode->i_mapping, 0, -1);
fs/fs-writeback.c +25 -4
index 9adc9d9..7bd1aef 100644
@@ -280,10 +280,12 @@ static void inode_wait_for_writeback(struct inode *inode)
 	wait_queue_head_t *wqh;
 
 	wqh = bit_waitqueue(&inode->i_state, __I_SYNC);
-	 while (inode->i_state & I_SYNC) {
+	while (inode->i_state & I_SYNC) {
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 		__wait_on_bit(wqh, &wq, inode_wait, TASK_UNINTERRUPTIBLE);
 		spin_lock(&inode_lock);
+		spin_lock(&inode->i_lock);
 	}
 }
 
@@ -337,6 +339,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 	/* Set I_SYNC, reset I_DIRTY_PAGES */
 	inode->i_state |= I_SYNC;
 	inode->i_state &= ~I_DIRTY_PAGES;
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 
 	ret = do_writepages(mapping, wbc);
@@ -358,8 +361,10 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 	 * write_inode()
 	 */
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	dirty = inode->i_state & I_DIRTY;
 	inode->i_state &= ~(I_DIRTY_SYNC | I_DIRTY_DATASYNC);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	/* Don't write the inode if only I_DIRTY_PAGES was set */
 	if (dirty & (I_DIRTY_SYNC | I_DIRTY_DATASYNC)) {
@@ -369,6 +374,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 	}
 
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & I_FREEING)) {
 		if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
@@ -479,7 +485,9 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 			return 0;
 		}
 
+		spin_lock(&inode->i_lock);
 		if (inode->i_state & (I_NEW | I_WILL_FREE)) {
+			spin_unlock(&inode->i_lock);
 			requeue_io(inode);
 			continue;
 		}
@@ -487,8 +495,10 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 		 * Was this inode dirtied after sync_sb_inodes was called?
 		 * This keeps sync from extra jobs and livelock.
 		 */
-		if (inode_dirtied_after(inode, wbc->wb_start))
+		if (inode_dirtied_after(inode, wbc->wb_start)) {
+			spin_unlock(&inode->i_lock);
 			return 1;
+		}
 
 		BUG_ON(inode->i_state & I_FREEING);
 		__iget(inode);
@@ -501,6 +511,7 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 			 */
 			redirty_tail(inode);
 		}
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 		iput(inode);
 		cond_resched();
@@ -936,6 +947,7 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 		block_dump___mark_inode_dirty(inode);
 
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	if ((inode->i_state & flags) != flags) {
 		const int was_dirty = inode->i_state & I_DIRTY;
 
@@ -986,6 +998,7 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 		}
 	}
 out:
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 
 	if (wakeup_bdi)
@@ -1033,12 +1046,16 @@ static void wait_sb_inodes(struct super_block *sb)
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		struct address_space *mapping;
 
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
-			continue;
 		mapping = inode->i_mapping;
 		if (mapping->nrpages == 0)
 			continue;
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}
 		__iget(inode);
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 		/*
@@ -1165,7 +1182,9 @@ int write_inode_now(struct inode *inode, int sync)
 
 	might_sleep();
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	ret = writeback_single_inode(inode, &wbc);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	if (sync)
 		inode_sync_wait(inode);
@@ -1189,7 +1208,9 @@ int sync_inode(struct inode *inode, struct writeback_control *wbc)
 	int ret;
 
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	ret = writeback_single_inode(inode, wbc);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	return ret;
 }
fs/inode.c +74 -12

312 lines not on this page · raw message

fs/nilfs2/gcdat.c +1 -0
index 84a45d1..c51f0e8 100644
@@ -27,6 +27,7 @@
 #include "page.h"
 #include "mdt.h"
 
+/* XXX: what protects i_state? */
 int nilfs_init_gcdat_inode(struct the_nilfs *nilfs)
 {
 	struct inode *dat = nilfs->ns_dat, *gcdat = nilfs->ns_gc_dat;
fs/quota/dquot.c +7 -5
index 2e3b913..15f66f1 100644
@@ -899,18 +899,20 @@ static void add_dquot_ref(struct super_block *sb, int type)
 	spin_lock(&inode_lock);
 	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
+		spin_lock(&inode->i_lock);
+		if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW) ||
+		    !atomic_read(&inode->i_writecount) ||
+		    !dqinit_needed(inode, type))) {
+			spin_unlock(&inode->i_lock);
 			continue;
+		}
 #ifdef CONFIG_QUOTA_DEBUG
 		if (unlikely(inode_get_rsv_space(inode) > 0))
 			reserved = 1;
 #endif
-		if (!atomic_read(&inode->i_writecount))
-			continue;
-		if (!dqinit_needed(inode, type))
-			continue;
 
 		__iget(inode);
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 		spin_unlock(&inode_lock);
 
Signature · 1 line
1.7.1
#19 lore raw

Re: [PATCH 04/17] fs: icache lock i_state

fs/drop_caches.c static void drop_pagecache_sb(struct super_block *sb, void *unused) · line 24

Quoted 3 lines
fs/drop_caches.c +3 -0
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)
+				|| inode->i_mapping->nrpages == 0) {

This is some pretty strange formatting.

		if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) ||
		    inode->i_mapping->nrpages == 0) {

would be more standard.

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1056

Quoted 13 lines
fs/fs-writeback.c +5 -2
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		struct address_space *mapping;
 
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
-			continue;
 		mapping = inode->i_mapping;
 		if (mapping->nrpages == 0)
 			continue;
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}

Can we access the mapping safely when the inode isn't actually fully setup? Even if we can I'd rather not introduce this change hidden inside an unrelated patch.

#20 lore raw

Re: [PATCH 04/17] fs: icache lock i_state

Quoted 15 lines · Christoph Hellwig

fs/drop_caches.c static void drop_pagecache_sb(struct super_block *sb, void *unused) · line 24

Quoted 3 lines
fs/drop_caches.c +3 -0
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)
+				|| inode->i_mapping->nrpages == 0) {

This is some pretty strange formatting.

		if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) ||
		    inode->i_mapping->nrpages == 0) {

would be more standard.

Quoted 3 lines
fs/fs-writeback.c +0 -0
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		struct address_space *mapping;
Quoted 14 lines

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1056

Quoted 10 lines
fs/fs-writeback.c +5 -2
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
-			continue;
 		mapping = inode->i_mapping;
 		if (mapping->nrpages == 0)
 			continue;
+		spin_lock(&inode->i_lock);
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) {
+			spin_unlock(&inode->i_lock);
+			continue;
+		}

Can we access the mapping safely when the inode isn't actually fully setup? Even if we can I'd rather not introduce this change hidden inside an unrelated patch.

Good point, fixed.

#21 lore raw

PATCH 5/17 fs: icache lock i_count

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

the inode reference count is currently an atomic variable so that it can be sampled/modified outside the inode_lock. However, the inode_lock is still needed to synchronise the final reference count and checks against the inode state.

With the inode state now protected by a per-inode lock, we can protect the inode reference count with the same per-inode lock and still check and modify the count and state atomically. By using the i_lock in this manner, we no longer need an atomic reference count field, so convert all the reference counting to be protected by the inode->i_lock and change i_count to a non-atomic variable.

---
 arch/powerpc/platforms/cell/spufs/file.c |    2 +-
 drivers/staging/pohmelfs/inode.c         |   14 +++++++----
 fs/9p/vfs_inode.c                        |    4 ++-
 fs/affs/inode.c                          |    4 ++-
 fs/afs/dir.c                             |    4 ++-
 fs/anon_inodes.c                         |    4 ++-
 fs/bfs/dir.c                             |    4 ++-
 fs/block_dev.c                           |   15 ++++++++++--
 fs/btrfs/inode.c                         |   23 +++++++++++++++----
 fs/ceph/mds_client.c                     |    2 +-
 fs/cifs/inode.c                          |    2 +-
 fs/coda/dir.c                            |    4 ++-
 fs/exofs/inode.c                         |   12 +++++++--
 fs/exofs/namei.c                         |    4 ++-
 fs/ext2/namei.c                          |    4 ++-
 fs/ext3/ialloc.c                         |    4 +-
 fs/ext3/namei.c                          |    4 ++-
 fs/ext4/ialloc.c                         |    4 +-
 fs/ext4/namei.c                          |    4 ++-
 fs/fs-writeback.c                        |    4 +-
 fs/gfs2/ops_inode.c                      |    4 ++-
 fs/hfsplus/dir.c                         |    4 ++-
 fs/hpfs/inode.c                          |    2 +-
 fs/inode.c                               |   36 ++++++++++++++++++++---------
 fs/jffs2/dir.c                           |    8 +++++-
 fs/jfs/jfs_txnmgr.c                      |    4 ++-
 fs/jfs/namei.c                           |    4 ++-
 fs/libfs.c                               |    4 ++-
 fs/locks.c                               |    2 +-
 fs/logfs/dir.c                           |    4 ++-
 fs/logfs/readwrite.c                     |    6 ++++-
 fs/minix/namei.c                         |    4 ++-
 fs/namei.c                               |    7 ++++-
 fs/nfs/dir.c                             |    4 ++-
 fs/nfs/getroot.c                         |    4 ++-
 fs/nfs/inode.c                           |    4 +-
 fs/nfs/nfs4state.c                       |    2 +-
 fs/nfs/write.c                           |    2 +-
 fs/nilfs2/mdt.c                          |    2 +-
 fs/nilfs2/namei.c                        |    4 ++-
 fs/notify/inode_mark.c                   |   21 ++++++++++------
 fs/ntfs/super.c                          |    4 ++-
 fs/ocfs2/namei.c                         |    4 ++-
 fs/reiserfs/namei.c                      |    4 ++-
 fs/reiserfs/stree.c                      |    2 +-
 fs/sysv/namei.c                          |    4 ++-
 fs/ubifs/dir.c                           |    4 ++-
 fs/ubifs/super.c                         |    2 +-
 fs/udf/namei.c                           |    4 ++-
 fs/ufs/namei.c                           |    4 ++-
 fs/xfs/linux-2.6/xfs_iops.c              |    4 ++-
 fs/xfs/linux-2.6/xfs_trace.h             |    2 +-
 fs/xfs/xfs_inode.h                       |    6 +++-
 include/linux/fs.h                       |    2 +-
 ipc/mqueue.c                             |    7 ++++-
 kernel/futex.c                           |    4 ++-
 mm/shmem.c                               |    4 ++-
 net/socket.c                             |    4 ++-
 58 files changed, 224 insertions(+), 95 deletions(-)
arch/powerpc/platforms/cell/spufs/file.c +1 -1
index 1a40da9..79238ba 100644
@@ -1549,7 +1549,7 @@ static int spufs_mfc_open(struct inode *inode, struct file *file)
 	if (ctx->owner != current->mm)
 		return -EINVAL;
 
-	if (atomic_read(&inode->i_count) != 1)
+	if (inode->i_count != 1)
 		return -EBUSY;
 
 	mutex_lock(&ctx->mapping_lock);
drivers/staging/pohmelfs/inode.c +9 -5
index 97dae29..2dc95dd 100644
@@ -1289,13 +1289,15 @@ static void pohmelfs_put_super(struct super_block *sb)
 		dprintk("%s: ino: %llu, pi: %p, inode: %p, count: %u.\n",
 				__func__, pi->ino, pi, inode, count);
 
-		if (atomic_read(&inode->i_count) != count) {
+		spin_lock(&inode->i_lock);
+		if (inode->i_count) != count) {
 			printk("%s: ino: %llu, pi: %p, inode: %p, count: %u, i_count: %d.\n",
 					__func__, pi->ino, pi, inode, count,
-					atomic_read(&inode->i_count));
-			count = atomic_read(&inode->i_count);
+					inode->i_count);
+			count = inode->i_count;
 			in_drop_list++;
 		}
+		spin_unlock(&inode->i_lock);
 
 		while (count--)
 			iput(&pi->vfs_inode);
@@ -1305,7 +1307,7 @@ static void pohmelfs_put_super(struct super_block *sb)
 		pi = POHMELFS_I(inode);
 
 		dprintk("%s: ino: %llu, pi: %p, inode: %p, i_count: %u.\n",
-				__func__, pi->ino, pi, inode, atomic_read(&inode->i_count));
+				__func__, pi->ino, pi, inode, inode->i_count);
 
 		/*
 		 * These are special inodes, they were created during
@@ -1313,7 +1315,9 @@ static void pohmelfs_put_super(struct super_block *sb)
 		 * so they live here with reference counter being 1 and prevent
 		 * umount from succeed since it believes that they are busy.
 		 */
-		count = atomic_read(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		count = inode->i_count;
+		spin_lock(&inode->i_lock);
 		if (count) {
 			list_del_init(&inode->i_sb_list);
 			while (count--)
fs/9p/vfs_inode.c +3 -1
index 9e670d5..5e1d774 100644
@@ -1791,7 +1791,9 @@ v9fs_vfs_link_dotl(struct dentry *old_dentry, struct inode *dir,
 		/* Caching disabled. No need to get upto date stat info.
 		 * This dentry will be released immediately. So, just i_count++
 		 */
-		atomic_inc(&old_dentry->d_inode->i_count);
+		spin_lock(&old_dentry->d_inode->i_lock);
+		old_dentry->d_inode->i_count++;
+		spin_unlock(&old_dentry->d_inode->i_lock);
 	}
 
 	dentry->d_op = old_dentry->d_op;
fs/affs/inode.c +3 -1
index 3a0fdec..cb9e773 100644
@@ -388,7 +388,9 @@ affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s3
 		affs_adjust_checksum(inode_bh, block - be32_to_cpu(chain));
 		mark_buffer_dirty_inode(inode_bh, inode);
 		inode->i_nlink = 2;
-		atomic_inc(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		inode->i_count++;
+		spin_unlock(&inode->i_lock);
 	}
 	affs_fix_checksum(sb, bh);
 	mark_buffer_dirty_inode(bh, inode);
fs/afs/dir.c +3 -1
index 0d38c09..4d8598c 100644
@@ -1045,7 +1045,9 @@ static int afs_link(struct dentry *from, struct inode *dir,
 	if (ret < 0)
 		goto link_error;
 
-	atomic_inc(&vnode->vfs_inode.i_count);
+	spin_lock(&vnode->vfs_inode.i_lock);
+	vnode->vfs_inode.i_count++;
+	spin_unlock(&vnode->vfs_inode.i_lock);
 	d_instantiate(dentry, &vnode->vfs_inode);
 	key_put(key);
 	_leave(" = 0");
fs/anon_inodes.c +3 -1
index e4b75d6..c50dc2a 100644
@@ -114,7 +114,9 @@ struct file *anon_inode_getfile(const char *name,
 	 * so we can avoid doing an igrab() and we can use an open-coded
 	 * atomic_inc().
 	 */
-	atomic_inc(&anon_inode_inode->i_count);
+	spin_lock(&anon_inode_inode->i_lock);
+	anon_inode_inode->i_count++;
+	spin_unlock(&anon_inode_inode->i_lock);
 
 	path.dentry->d_op = &anon_inodefs_dentry_operations;
 	d_instantiate(path.dentry, anon_inode_inode);
fs/bfs/dir.c +3 -1
index d967e05..d42fc72 100644
@@ -176,7 +176,9 @@ static int bfs_link(struct dentry *old, struct inode *dir,
 	inc_nlink(inode);
 	inode->i_ctime = CURRENT_TIME_SEC;
 	mark_inode_dirty(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	d_instantiate(new, inode);
 	mutex_unlock(&info->bfs_lock);
 	return 0;
fs/block_dev.c +12 -3
index 50e8c85..140451c 100644
@@ -550,7 +550,12 @@ EXPORT_SYMBOL(bdget);
  */
 struct block_device *bdgrab(struct block_device *bdev)
 {
-	atomic_inc(&bdev->bd_inode->i_count);
+	struct inode *inode = bdev->bd_inode;
+
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
+
 	return bdev;
 }
 
@@ -580,7 +585,9 @@ static struct block_device *bd_acquire(struct inode *inode)
 	spin_lock(&bdev_lock);
 	bdev = inode->i_bdev;
 	if (bdev) {
-		atomic_inc(&bdev->bd_inode->i_count);
+		spin_lock(&inode->i_lock);
+		bdev->bd_inode->i_count++;
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&bdev_lock);
 		return bdev;
 	}
@@ -596,7 +603,9 @@ static struct block_device *bd_acquire(struct inode *inode)
 			 * So, we can access it via ->i_mapping always
 			 * without igrab().
 			 */
-			atomic_inc(&bdev->bd_inode->i_count);
+			spin_lock(&inode->i_lock);
+			bdev->bd_inode->i_count++;
+			spin_unlock(&inode->i_lock);
 			inode->i_bdev = bdev;
 			inode->i_mapping = bdev->bd_inode->i_mapping;
 			list_add(&inode->i_devices, &bdev->bd_inodes);
fs/btrfs/inode.c +18 -5
index c038644..ffb8aec 100644
@@ -1964,8 +1964,13 @@ void btrfs_add_delayed_iput(struct inode *inode)
 	struct btrfs_fs_info *fs_info = BTRFS_I(inode)->root->fs_info;
 	struct delayed_iput *delayed;
 
-	if (atomic_add_unless(&inode->i_count, -1, 1))
+	spin_lock(&inode->i_lock);
+	if (inode->i_count > 1) {
+		inode->i_count--;
+		spin_unlock(&inode->i_lock);
 		return;
+	}
+	spin_unlock(&inode->i_lock);
 
 	delayed = kmalloc(sizeof(*delayed), GFP_NOFS | __GFP_NOFAIL);
 	delayed->inode = inode;
@@ -2718,11 +2723,17 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode *dir,
 		return ERR_PTR(-ENOSPC);
 
 	/* check if there is someone else holds reference */
-	if (S_ISDIR(inode->i_mode) && atomic_read(&inode->i_count) > 1)
+	spin_lock(&inode->i_lock);
+	if (S_ISDIR(inode->i_mode) && inode->i_count > 1) {
+		spin_unlock(&inode->i_lock);
 		return ERR_PTR(-ENOSPC);
+	}
 
-	if (atomic_read(&inode->i_count) > 2)
+	if (inode->i_count > 2) {
+		spin_unlock(&inode->i_lock);
 		return ERR_PTR(-ENOSPC);
+	}
+	spin_unlock(&inode->i_lock);
 
 	if (xchg(&root->fs_info->enospc_unlink, 1))
 		return ERR_PTR(-ENOSPC);
@@ -3939,7 +3950,7 @@ again:
 		inode = igrab(&entry->vfs_inode);
 		if (inode) {
 			spin_unlock(&root->inode_lock);
-			if (atomic_read(&inode->i_count) > 1)
+			if (inode->i_count > 1)
 				d_prune_aliases(inode);
 			/*
 			 * btrfs_drop_inode will have it removed from
@@ -4758,7 +4769,9 @@ static int btrfs_link(struct dentry *old_dentry, struct inode *dir,
 	}
 
 	btrfs_set_trans_block_group(trans, dir);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	err = btrfs_add_nondir(trans, dentry, inode, 1, index);
 
fs/ceph/mds_client.c +1 -1
index f091b13..cb4d673 100644
@@ -1102,7 +1102,7 @@ static int trim_caps_cb(struct inode *inode, struct ceph_cap *cap, void *arg)
 		spin_unlock(&inode->i_lock);
 		d_prune_aliases(inode);
 		dout("trim_caps_cb %p cap %p  pruned, count now %d\n",
-		     inode, cap, atomic_read(&inode->i_count));
+		     inode, cap, inode->i_count);
 		return 0;
 	}
 
fs/cifs/inode.c +1 -1
index 93f77d4..af0f050 100644
@@ -1639,7 +1639,7 @@ int cifs_revalidate_dentry(struct dentry *dentry)
 	}
 
 	cFYI(1, "Revalidate: %s inode 0x%p count %d dentry: 0x%p d_time %ld "
-		 "jiffies %ld", full_path, inode, inode->i_count.counter,
+		 "jiffies %ld", full_path, inode, inode->i_count,
 		 dentry, dentry->d_time, jiffies);
 
 	if (CIFS_SB(sb)->tcon->unix_ext)
fs/coda/dir.c +3 -1
index ccd98b0..2e52ad6 100644
@@ -303,7 +303,9 @@ static int coda_link(struct dentry *source_de, struct inode *dir_inode,
 	}
 
 	coda_dir_update_mtime(dir_inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	d_instantiate(de, inode);
 	inc_nlink(inode);
 
fs/exofs/inode.c +9 -3
index eb7368e..3e7f967 100644
@@ -1101,7 +1101,9 @@ static void create_done(struct exofs_io_state *ios, void *p)
 
 	set_obj_created(oi);
 
-	atomic_dec(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count--;
+	spin_unlock(&inode->i_lock);
 	wake_up(&oi->i_wq);
 }
 
@@ -1154,14 +1156,18 @@ struct inode *exofs_new_inode(struct inode *dir, int mode)
 	/* increment the refcount so that the inode will still be around when we
 	 * reach the callback
 	 */
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	ios->done = create_done;
 	ios->private = inode;
 	ios->cred = oi->i_cred;
 	ret = exofs_sbi_create(ios);
 	if (ret) {
-		atomic_dec(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		inode->i_count--;
+		spin_unlock(&inode->i_lock);
 		exofs_put_io_state(ios);
 		return ERR_PTR(ret);
 	}
fs/exofs/namei.c +3 -1
index b7dd0c2..506778a 100644
@@ -153,7 +153,9 @@ static int exofs_link(struct dentry *old_dentry, struct inode *dir,
 
 	inode->i_ctime = CURRENT_TIME;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	return exofs_add_nondir(dentry, inode);
 }
fs/ext2/namei.c +3 -1
index 71efb0e..a5b9a54 100644
@@ -206,7 +206,9 @@ static int ext2_link (struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	err = ext2_add_link(dentry, inode);
 	if (!err) {
fs/ext3/ialloc.c +2 -2
index 4ab72db..cdb398b 100644
@@ -100,9 +100,9 @@ void ext3_free_inode (handle_t *handle, struct inode * inode)
 	struct ext3_sb_info *sbi;
 	int fatal = 0, err;
 
-	if (atomic_read(&inode->i_count) > 1) {
+	if (inode->i_count > 1) {
 		printk ("ext3_free_inode: inode has count=%d\n",
-					atomic_read(&inode->i_count));
+					inode->i_count);
 		return;
 	}
 	if (inode->i_nlink) {
fs/ext3/namei.c +3 -1
index 2b35ddb..4e3b5ff 100644
@@ -2260,7 +2260,9 @@ retry:
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inc_nlink(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	err = ext3_add_entry(handle, dentry, inode);
 	if (!err) {
fs/ext4/ialloc.c +2 -2
index 45853e0..720b42d 100644
@@ -189,9 +189,9 @@ void ext4_free_inode(handle_t *handle, struct inode *inode)
 	struct ext4_sb_info *sbi;
 	int fatal = 0, err, count, cleared;
 
-	if (atomic_read(&inode->i_count) > 1) {
+	if (inode->i_count > 1) {
 		printk(KERN_ERR "ext4_free_inode: inode has count=%d\n",
-		       atomic_read(&inode->i_count));
+		       inode->i_count);
 		return;
 	}
 	if (inode->i_nlink) {
fs/ext4/namei.c +3 -1
index 314c0d3..4dbb5e5 100644
@@ -2312,7 +2312,9 @@ retry:
 
 	inode->i_ctime = ext4_current_time(inode);
 	ext4_inc_count(handle, inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	err = ext4_add_entry(handle, dentry, inode);
 	if (!err) {
fs/fs-writeback.c +2 -2
index 7bd1aef..2edaad7 100644
@@ -309,7 +309,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 	unsigned dirty;
 	int ret;
 
-	if (!atomic_read(&inode->i_count))
+	if (!inode->i_count)
 		WARN_ON(!(inode->i_state & (I_WILL_FREE|I_FREEING)));
 	else
 		WARN_ON(inode->i_state & I_WILL_FREE);
@@ -406,7 +406,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 			 * completion.
 			 */
 			redirty_tail(inode);
-		} else if (atomic_read(&inode->i_count)) {
+		} else if (inode->i_count) {
 			/*
 			 * The inode is clean, inuse
 			 */
fs/gfs2/ops_inode.c +3 -1
index 1009be2..49c38dc 100644
@@ -253,7 +253,9 @@ out_parent:
 	gfs2_holder_uninit(ghs);
 	gfs2_holder_uninit(ghs + 1);
 	if (!error) {
-		atomic_inc(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		inode->i_count++;
+		spin_unlock(&inode->i_lock);
 		d_instantiate(dentry, inode);
 		mark_inode_dirty(inode);
 	}
fs/hfsplus/dir.c +3 -1
index 764fd1b..55fa48d 100644
@@ -301,7 +301,9 @@ static int hfsplus_link(struct dentry *src_dentry, struct inode *dst_dir,
 
 	inc_nlink(inode);
 	hfsplus_instantiate(dst_dentry, inode, cnid);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	inode->i_ctime = CURRENT_TIME_SEC;
 	mark_inode_dirty(inode);
 	HFSPLUS_SB(sb).file_count++;
fs/hpfs/inode.c +1 -1
index 56f0da1..7f6b7ef 100644
@@ -183,7 +183,7 @@ void hpfs_write_inode(struct inode *i)
 	struct hpfs_inode_info *hpfs_inode = hpfs_i(i);
 	struct inode *parent;
 	if (i->i_ino == hpfs_sb(i->i_sb)->sb_root) return;
-	if (hpfs_inode->i_rddir_off && !atomic_read(&i->i_count)) {
+	if (hpfs_inode->i_rddir_off && !i->i_count) {
 		if (*hpfs_inode->i_rddir_off) printk("HPFS: write_inode: some position still there\n");
 		kfree(hpfs_inode->i_rddir_off);
 		hpfs_inode->i_rddir_off = NULL;
fs/inode.c +25 -11
index 906a4ad..2e8ab8e 100644
@@ -32,14 +32,13 @@
  * inode_hash_lock protects:
  *   inode hash table, i_hash
  * inode->i_lock protects:
- *   i_state
+ *   i_state, i_count
  *
  * Ordering:
  * inode_lock
  *   sb_inode_list_lock
  *     inode->i_lock
- * inode_lock
- *   inode_hash_lock
+ *       inode_hash_lock
  */
 /*
  * This is needed for the following functions:
@@ -150,7 +149,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	inode->i_sb = sb;
 	inode->i_blkbits = sb->s_blocksize_bits;
 	inode->i_flags = 0;
-	atomic_set(&inode->i_count, 1);
+	inode->i_count = 1;
 	inode->i_op = &empty_iops;
 	inode->i_fop = &empty_fops;
 	inode->i_nlink = 1;
@@ -301,7 +300,8 @@ void __iget(struct inode *inode)
 {
 	assert_spin_locked(&inode->i_lock);
 
-	if (atomic_inc_return(&inode->i_count) != 1)
+	inode->i_count++;
+	if (inode->i_count > 1)
 		return;
 
 	if (!(inode->i_state & (I_DIRTY|I_SYNC)))
@@ -407,7 +407,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 			continue;
 		}
 		invalidate_inode_buffers(inode);
-		if (!atomic_read(&inode->i_count)) {
+		if (!inode->i_count) {
 			list_move(&inode->i_list, dispose);
 			WARN_ON(inode->i_state & I_NEW);
 			inode->i_state |= I_FREEING;
@@ -457,7 +457,7 @@ static int can_unuse(struct inode *inode)
 		return 0;
 	if (inode_has_buffers(inode))
 		return 0;
-	if (atomic_read(&inode->i_count))
+	if (inode->i_count)
 		return 0;
 	if (inode->i_data.nrpages)
 		return 0;
@@ -495,7 +495,7 @@ static void prune_icache(int nr_to_scan)
 		inode = list_entry(inode_unused.prev, struct inode, i_list);
 
 		spin_lock(&inode->i_lock);
-		if (inode->i_state || atomic_read(&inode->i_count)) {
+		if (inode->i_state || inode->i_count) {
 			list_move(&inode->i_list, &inode_unused);
 			spin_unlock(&inode->i_lock);
 			continue;
@@ -1307,8 +1307,6 @@ static void iput_final(struct inode *inode)
 	else
 		drop = generic_drop_inode(inode);
 
-	spin_lock(&sb_inode_list_lock);
-	spin_lock(&inode->i_lock);
 	if (!drop) {
 		if (!(inode->i_state & (I_DIRTY|I_SYNC)))
 			list_move(&inode->i_list, &inode_unused);
@@ -1368,8 +1366,24 @@ void iput(struct inode *inode)
 	if (inode) {
 		BUG_ON(inode->i_state & I_CLEAR);
 
-		if (atomic_dec_and_lock(&inode->i_count, &inode_lock))
+retry:
+		spin_lock(&inode->i_lock);
+		if (inode->i_count == 1) {
+			if (!spin_trylock(&inode_lock)) {
+				spin_unlock(&inode->i_lock);
+				goto retry;
+			}
+			if (!spin_trylock(&sb_inode_list_lock)) {
+				spin_unlock(&inode_lock);
+				spin_unlock(&inode->i_lock);
+				goto retry;
+			}
+			inode->i_count--;
 			iput_final(inode);
+		} else {
+			inode->i_count--;
+			spin_unlock(&inode->i_lock);
+		}
 	}
 }
 EXPORT_SYMBOL(iput);
fs/jffs2/dir.c +6 -2
index ed78a3c..4d1bcfa 100644
@@ -289,7 +289,9 @@ static int jffs2_link (struct dentry *old_dentry, struct inode *dir_i, struct de
 		mutex_unlock(&f->sem);
 		d_instantiate(dentry, old_dentry->d_inode);
 		dir_i->i_mtime = dir_i->i_ctime = ITIME(now);
-		atomic_inc(&old_dentry->d_inode->i_count);
+		spin_lock(&old_dentry->d_inode->i_lock);
+		old_dentry->d_inode->i_count++;
+		spin_unlock(&old_dentry->d_inode->i_lock);
 	}
 	return ret;
 }
@@ -864,7 +866,9 @@ static int jffs2_rename (struct inode *old_dir_i, struct dentry *old_dentry,
 		printk(KERN_NOTICE "jffs2_rename(): Link succeeded, unlink failed (err %d). You now have a hard link\n", ret);
 		/* Might as well let the VFS know */
 		d_instantiate(new_dentry, old_dentry->d_inode);
-		atomic_inc(&old_dentry->d_inode->i_count);
+		spin_lock(&old_dentry->d_inode->i_lock);
+		old_dentry->d_inode->i_count++;
+		spin_unlock(&old_dentry->d_inode->i_lock);
 		new_dir_i->i_mtime = new_dir_i->i_ctime = ITIME(now);
 		return ret;
 	}
fs/jfs/jfs_txnmgr.c +3 -1
index d945ea7..820212f 100644
@@ -1279,7 +1279,9 @@ int txCommit(tid_t tid,		/* transaction identifier */
 	 * lazy commit thread finishes processing
 	 */
 	if (tblk->xflag & COMMIT_DELETE) {
-		atomic_inc(&tblk->u.ip->i_count);
+		spin_lock(&tblk->u.ip->i_lock);
+		tblk->u.ip->i_count++;
+		spin_unlock(&tblk->u.ip->i_lock);
 		/*
 		 * Avoid a rare deadlock
 		 *
fs/jfs/namei.c +3 -1
index a9cf8e8..3259008 100644
@@ -839,7 +839,9 @@ static int jfs_link(struct dentry *old_dentry,
 	ip->i_ctime = CURRENT_TIME;
 	dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	mark_inode_dirty(dir);
-	atomic_inc(&ip->i_count);
+	spin_lock(&ip->i_lock);
+	ip->i_count++;
+	spin_unlock(&ip->i_lock);
 
 	iplist[0] = ip;
 	iplist[1] = dir;
fs/libfs.c +3 -1
index 0a9da95..0fa4dbe 100644
@@ -255,7 +255,9 @@ int simple_link(struct dentry *old_dentry, struct inode *dir, struct dentry *den
 
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	inc_nlink(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	dget(dentry);
 	d_instantiate(dentry, inode);
 	return 0;
fs/locks.c +1 -1
index ab24d49..6d97e65 100644
@@ -1376,7 +1376,7 @@ int generic_setlease(struct file *filp, long arg, struct file_lock **flp)
 			goto out;
 		if ((arg == F_WRLCK)
 		    && ((atomic_read(&dentry->d_count) > 1)
-			|| (atomic_read(&inode->i_count) > 1)))
+			|| inode->i_count > 1))
 			goto out;
 	}
 
fs/logfs/dir.c +3 -1
index 9777eb5..90eb51f 100644
@@ -569,7 +569,9 @@ static int logfs_link(struct dentry *old_dentry, struct inode *dir,
 		return -EMLINK;
 
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	inode->i_nlink++;
 	mark_inode_dirty_sync(inode);
 
fs/logfs/readwrite.c +5 -1
index 6127baf..0d92191 100644
@@ -1002,8 +1002,12 @@ static int __logfs_is_valid_block(struct inode *inode, u64 bix, u64 ofs)
 {
 	struct logfs_inode *li = logfs_inode(inode);
 
-	if ((inode->i_nlink == 0) && atomic_read(&inode->i_count) == 1)
+	spin_lock(&inode->i_lock);
+	if ((inode->i_nlink == 0) && inode->i_count == 1) {
+		spin_unlock(&inode->i_lock);
 		return 0;
+	}
+	spin_unlock(&inode->i_lock);
 
 	if (bix < I0_BLOCKS)
 		return logfs_is_valid_direct(li, bix, ofs);
fs/minix/namei.c +3 -1
index f3f3578..a4a160f 100644
@@ -101,7 +101,9 @@ static int minix_link(struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	return add_nondir(dentry, inode);
 }
 
fs/namei.c +5 -2
index 24896e8..817d6bb 100644
@@ -2290,8 +2290,11 @@ static long do_unlinkat(int dfd, const char __user *pathname)
 		if (nd.last.name[nd.last.len])
 			goto slashes;
 		inode = dentry->d_inode;
-		if (inode)
-			atomic_inc(&inode->i_count);
+		if (inode) {
+			spin_lock(&inode->i_lock);
+			inode->i_count++;
+			spin_unlock(&inode->i_lock);
+		}
 		error = mnt_want_write(nd.path.mnt);
 		if (error)
 			goto exit2;
fs/nfs/dir.c +3 -1
index e257172..375b6b5 100644
@@ -1580,7 +1580,9 @@ nfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry)
 	d_drop(dentry);
 	error = NFS_PROTO(dir)->link(inode, dir, &dentry->d_name);
 	if (error == 0) {
-		atomic_inc(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		inode->i_count++;
+		spin_unlock(&inode->i_lock);
 		d_add(dentry, inode);
 	}
 	return error;
fs/nfs/getroot.c +3 -1
index a70e446..c6db37e 100644
@@ -55,7 +55,9 @@ static int nfs_superblock_set_dummy_root(struct super_block *sb, struct inode *i
 			return -ENOMEM;
 		}
 		/* Circumvent igrab(): we know the inode is not being freed */
-		atomic_inc(&inode->i_count);
+		spin_lock(&inode->i_lock);
+		inode->i_count++;
+		spin_unlock(&inode->i_lock);
 		/*
 		 * Ensure that this dentry is invisible to d_find_alias().
 		 * Otherwise, it may be spliced into the tree by
fs/nfs/inode.c +2 -2
index 7d2d6c7..dfb79e5 100644
@@ -384,7 +384,7 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr)
 	dprintk("NFS: nfs_fhget(%s/%Ld ct=%d)\n",
 		inode->i_sb->s_id,
 		(long long)NFS_FILEID(inode),
-		atomic_read(&inode->i_count));
+		inode->i_count);
 
 out:
 	return inode;
@@ -1190,7 +1190,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
 
 	dfprintk(VFS, "NFS: %s(%s/%ld ct=%d info=0x%x)\n",
 			__func__, inode->i_sb->s_id, inode->i_ino,
-			atomic_read(&inode->i_count), fattr->valid);
+			inode->i_count, fattr->valid);
 
 	if ((fattr->valid & NFS_ATTR_FATTR_FILEID) && nfsi->fileid != fattr->fileid)
 		goto out_fileid;
fs/nfs/nfs4state.c +1 -1
index 3e2f19b..d7fc5d0 100644
@@ -506,8 +506,8 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner)
 		state->owner = owner;
 		atomic_inc(&owner->so_count);
 		list_add(&state->inode_states, &nfsi->open_states);
-		state->inode = igrab(inode);
 		spin_unlock(&inode->i_lock);
+		state->inode = igrab(inode);
 		/* Note: The reclaim code dictates that we add stateless
 		 * and read-only stateids to the end of the list */
 		list_add_tail(&state->open_states, &owner->so_states);
fs/nfs/write.c +1 -1
index 874972d..129ebaa 100644
@@ -390,7 +390,7 @@ static int nfs_inode_add_request(struct inode *inode, struct nfs_page *req)
 	error = radix_tree_insert(&nfsi->nfs_page_tree, req->wb_index, req);
 	BUG_ON(error);
 	if (!nfsi->npages) {
-		igrab(inode);
+		__iget(inode);
 		if (nfs_have_delegation(inode, FMODE_WRITE))
 			nfsi->change_attr++;
 	}
fs/nilfs2/mdt.c +1 -1
index d01aff4..db0b75b 100644
@@ -480,7 +480,7 @@ nilfs_mdt_new_common(struct the_nilfs *nilfs, struct super_block *sb,
 		inode->i_sb = sb; /* sb may be NULL for some meta data files */
 		inode->i_blkbits = nilfs->ns_blocksize_bits;
 		inode->i_flags = 0;
-		atomic_set(&inode->i_count, 1);
+		inode->i_count = 1;
 		inode->i_nlink = 1;
 		inode->i_ino = ino;
 		inode->i_mode = S_IFREG;
fs/nilfs2/namei.c +3 -1
index ad6ed2c..9e287ea 100644
@@ -219,7 +219,9 @@ static int nilfs_link(struct dentry *old_dentry, struct inode *dir,
 
 	inode->i_ctime = CURRENT_TIME;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	err = nilfs_add_nondir(dentry, inode);
 	if (!err)
fs/notify/inode_mark.c +13 -8
index 34b1585..70f7e16 100644
@@ -257,24 +257,29 @@ void fsnotify_unmount_inodes(struct list_head *list)
 		 * evict all inodes with zero i_count from icache which is
 		 * unnecessarily violent and may in fact be illegal to do.
 		 */
-		if (!atomic_read(&inode->i_count))
+		if (!inode->i_count)
 			continue;
 
 		need_iput_tmp = need_iput;
 		need_iput = NULL;
 
 		/* In case fsnotify_inode_delete() drops a reference. */
-		if (inode != need_iput_tmp)
+		if (inode != need_iput_tmp) {
+			spin_lock(&inode->i_lock);
 			__iget(inode);
-		else
+			spin_unlock(&inode->i_lock);
+		} else
 			need_iput_tmp = NULL;
 
 		/* In case the dropping of a reference would nuke next_i. */
-		if ((&next_i->i_sb_list != list) &&
-		    atomic_read(&next_i->i_count) &&
-		    !(next_i->i_state & (I_FREEING | I_WILL_FREE))) {
-			__iget(next_i);
-			need_iput = next_i;
+		if (&next_i->i_sb_list != list) {
+			spin_lock(&next_i->i_lock);
+			if (next_i->i_count &&
+			    !(next_i->i_state & (I_FREEING | I_WILL_FREE))) {
+				__iget(next_i);
+				need_iput = next_i;
+			}
+			spin_unlock(&next_i->i_lock);
 		}
 
 		/*
fs/ntfs/super.c +3 -1
index 5128061..2e380ba 100644
@@ -2930,7 +2930,9 @@ static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent)
 	}
 	if ((sb->s_root = d_alloc_root(vol->root_ino))) {
 		/* We increment i_count simulating an ntfs_iget(). */
-		atomic_inc(&vol->root_ino->i_count);
+		spin_lock(&vol->root_ino->i_lock);
+		vol->root_ino->i_count++;
+		spin_unlock(&vol->root_ino->i_lock);
 		ntfs_debug("Exiting, status successful.");
 		/* Release the default upcase if it has no users. */
 		mutex_lock(&ntfs_lock);
fs/ocfs2/namei.c +3 -1
index a00dda2..9c46feb 100644
@@ -741,7 +741,9 @@ static int ocfs2_link(struct dentry *old_dentry,
 		goto out_commit;
 	}
 
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	dentry->d_op = &ocfs2_dentry_ops;
 	d_instantiate(dentry, inode);
 
fs/reiserfs/namei.c +3 -1
index ee78d4a..1efebb2 100644
@@ -1156,7 +1156,9 @@ static int reiserfs_link(struct dentry *old_dentry, struct inode *dir,
 	inode->i_ctime = CURRENT_TIME_SEC;
 	reiserfs_update_sd(&th, inode);
 
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	d_instantiate(dentry, inode);
 	retval = journal_end(&th, dir->i_sb, jbegin_count);
 	reiserfs_write_unlock(dir->i_sb);
fs/reiserfs/stree.c +1 -1
index 313d39d..b8d4d67 100644
@@ -1477,7 +1477,7 @@ static int maybe_indirect_to_direct(struct reiserfs_transaction_handle *th,
 	 ** reading in the last block.  The user will hit problems trying to
 	 ** read the file, but for now we just skip the indirect2direct
 	 */
-	if (atomic_read(&inode->i_count) > 1 ||
+	if (inode->i_count > 1 ||
 	    !tail_has_to_be_packed(inode) ||
 	    !page || (REISERFS_I(inode)->i_flags & i_nopack_mask)) {
 		/* leave tail in an unformatted node */
fs/sysv/namei.c +3 -1
index 33e047b..d63da9b 100644
@@ -126,7 +126,9 @@ static int sysv_link(struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	return add_nondir(dentry, inode);
 }
fs/ubifs/dir.c +3 -1
index 87ebcce..c204b5c 100644
@@ -550,7 +550,9 @@ static int ubifs_link(struct dentry *old_dentry, struct inode *dir,
 
 	lock_2_inodes(dir, inode);
 	inc_nlink(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	inode->i_ctime = ubifs_current_time(inode);
 	dir->i_size += sz_change;
 	dir_ui->ui_size = dir->i_size;
fs/ubifs/super.c +1 -1
index cd5900b..c3c79b7 100644
@@ -342,7 +342,7 @@ static void ubifs_evict_inode(struct inode *inode)
 		goto out;
 
 	dbg_gen("inode %lu, mode %#x", inode->i_ino, (int)inode->i_mode);
-	ubifs_assert(!atomic_read(&inode->i_count));
+	ubifs_assert(!inode->i_count);
 
 	truncate_inode_pages(&inode->i_data, 0);
 
fs/udf/namei.c +3 -1
index bf5fc67..d8b0dc8 100644
@@ -1101,7 +1101,9 @@ static int udf_link(struct dentry *old_dentry, struct inode *dir,
 	inc_nlink(inode);
 	inode->i_ctime = current_fs_time(inode->i_sb);
 	mark_inode_dirty(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	d_instantiate(dentry, inode);
 	unlock_kernel();
 
fs/ufs/namei.c +3 -1
index b056f02..8cbf920 100644
@@ -180,7 +180,9 @@ static int ufs_link (struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 
 	error = ufs_add_nondir(dentry, inode);
 	unlock_kernel();
fs/xfs/linux-2.6/xfs_iops.c +3 -1
index b1fc2a6..332cdf5 100644
@@ -352,7 +352,9 @@ xfs_vn_link(
 	if (unlikely(error))
 		return -error;
 
-	atomic_inc(&inode->i_count);
+	spin_lock(&inode->i_lock);
+	inode->i_count++;
+	spin_unlock(&inode->i_lock);
 	d_instantiate(dentry, inode);
 	return 0;
 }
fs/xfs/linux-2.6/xfs_trace.h +1 -1
index be5dffd..065db30 100644
@@ -599,7 +599,7 @@ DECLARE_EVENT_CLASS(xfs_iref_class,
 	TP_fast_assign(
 		__entry->dev = VFS_I(ip)->i_sb->s_dev;
 		__entry->ino = ip->i_ino;
-		__entry->count = atomic_read(&VFS_I(ip)->i_count);
+		__entry->count = VFS_I(ip)->i_count;
 		__entry->pincount = atomic_read(&ip->i_pincount);
 		__entry->caller_ip = caller_ip;
 	),
fs/xfs/xfs_inode.h +4 -2
index 0898c54..859628b 100644
@@ -481,8 +481,10 @@ void		xfs_mark_inode_dirty_sync(xfs_inode_t *);
 
 #define IHOLD(ip) \
 do { \
-	ASSERT(atomic_read(&VFS_I(ip)->i_count) > 0) ; \
-	atomic_inc(&(VFS_I(ip)->i_count)); \
+	spin_lock(&VFS_I(ip)->i_lock); \
+	ASSERT(VFS_I(ip)->i_count > 0) ; \
+	VFS_I(ip)->i_count++; \
+	spin_unlock(&VFS_I(ip)->i_lock); \
 	trace_xfs_ihold(ip, _THIS_IP_); \
 } while (0)
 
include/linux/fs.h +1 -1
index 76041b6..415c88e 100644
@@ -728,7 +728,7 @@ struct inode {
 	struct list_head	i_sb_list;
 	struct list_head	i_dentry;
 	unsigned long		i_ino;
-	atomic_t		i_count;
+	unsigned int		i_count;
 	unsigned int		i_nlink;
 	uid_t			i_uid;
 	gid_t			i_gid;
ipc/mqueue.c +5 -2
index c60e519..7fe9efb 100644
@@ -768,8 +768,11 @@ SYSCALL_DEFINE1(mq_unlink, const char __user *, u_name)
 	}
 
 	inode = dentry->d_inode;
-	if (inode)
-		atomic_inc(&inode->i_count);
+	if (inode) {
+		spin_lock(&inode->i_lock);
+		inode->i_count++;
+		spin_unlock(&inode->i_lock);
+	}
 	err = mnt_want_write(ipc_ns->mq_mnt);
 	if (err)
 		goto out_err;
kernel/futex.c +3 -1
index 6a3a5fa..e9dfa00 100644
@@ -168,7 +168,9 @@ static void get_futex_key_refs(union futex_key *key)
 
 	switch (key->both.offset & (FUT_OFF_INODE|FUT_OFF_MMSHARED)) {
 	case FUT_OFF_INODE:
-		atomic_inc(&key->shared.inode->i_count);
+		spin_lock(&key->shared.inode->i_lock);
+		key->shared.inode->i_count++;
+		spin_unlock(&key->shared.inode->i_lock);
 		break;
 	case FUT_OFF_MMSHARED:
 		atomic_inc(&key->private.mm->mm_count);
mm/shmem.c +3 -1
index 080b09a..56229bb 100644
@@ -1903,7 +1903,9 @@ static int shmem_link(struct dentry *old_dentry, struct inode *dir, struct dentr
 	dir->i_size += BOGO_DIRENT_SIZE;
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	inc_nlink(inode);
-	atomic_inc(&inode->i_count);	/* New dentry reference */
+	spin_lock(&inode->i_lock);
+	inode->i_count++;	/* New dentry reference */
+	spin_unlock(&inode->i_lock);
 	dget(dentry);		/* Extra pinning count for the created dentry */
 	d_instantiate(dentry, inode);
 out:
net/socket.c +3 -1
index 2270b94..5431af1 100644
@@ -377,7 +377,9 @@ static int sock_alloc_file(struct socket *sock, struct file **f, int flags)
 		  &socket_file_ops);
 	if (unlikely(!file)) {
 		/* drop dentry, keep inode */
-		atomic_inc(&path.dentry->d_inode->i_count);
+		spin_lock(&path.dentry->d_inode->i_lock);
+		path.dentry->d_inode->i_count++;
+		spin_unlock(&path.dentry->d_inode->i_lock);
 		path_put(&path);
 		put_unused_fd(fd);
 		return -ENFILE;
Signature · 1 line
1.7.1
maintainer
#22 lore raw

Re: [PATCH 05/17] fs: icache lock i_count

arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552

Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1 -1
-	if (atomic_read(&inode->i_count) != 1)
+	if (inode->i_count != 1)

This really should have been renamed to catch unconverted code.

Such code usually wouldn't compile anyway, but it will if it takes the address of i_count only (for example).

And maybe we should access this guy via accessor functions, dunno.

#23 lore raw

Re: [PATCH 05/17] fs: icache lock i_count

Quoted 9 lines · Andrew Morton

arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552

Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1 -1
-	if (atomic_read(&inode->i_count) != 1)
+	if (inode->i_count != 1)

This really should have been renamed to catch unconverted code.

Such code usually wouldn't compile anyway, but it will if it takes the address of i_count only (for example).

If people do whacky things they'll lose - there is a reason why C has a fairly strict type system after all. We've changed types of variables all the time and we didn't run into problems.

Quoted 1 line

And maybe we should access this guy via accessor functions, dunno.

Seems like complete overkill.

maintainer
#24 lore raw

Re: [PATCH 05/17] fs: icache lock i_count

Quoted 8 lines · On Fri, 1 Oct 2010 01:55:36 -0400 Christoph Hellwig <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552

Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1 -1
-	if (atomic_read(&inode->i_count) != 1)
+	if (inode->i_count != 1)

This really should have been renamed to catch unconverted code.

Quoted 6 lines
Quoted 2 lines

Such code usually wouldn't compile anyway, but it will if it takes the address of i_count only (for example).

If people do whacky things they'll lose - there is a reason why C has a fairly strict type system after all. We've changed types of variables all the time and we didn't run into problems.

No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.

Quoted 3 lines
Quoted 1 line

And maybe we should access this guy via accessor functions, dunno.

Seems like complete overkill.

Still wrong. We do this frequently and we do it in areas where we believe that the implementation might change in the future.

Had we done it with i_count from day one then this part of the patchset would be far simpler.

#25 lore raw

Re: [PATCH 05/17] fs: icache lock i_count

Quoted 4 lines · Andrew Morton

No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.

Sorry, if you run code with that obvious warnings you beg for trouble. If you really believe your advanced users arw too stupid to read compiler warnings enforcing -Werror is for sure better than obsfucating the code.

Quoted 5 lines

Still wrong. We do this frequently and we do it in areas where we believe that the implementation might change in the future.

Had we done it with i_count from day one then this part of the patchset would be far simpler.

I don't thin that's quite true. The big point of using i_lock is that we can hold it over accessing other things that also are protected by it. No accessor is going to help you with that. For plain opencoded increments we indeed need a helper as shown by Chris' aio speedup and the churn in here - but that's already added later in the series and I asked Dave to move it before this patch.

maintainer
#26 lore raw

Re: [PATCH 05/17] fs: icache lock i_count

Quoted 10 lines · On Fri, 1 Oct 2010 02:16:02 -0400 Christoph Hellwig <[email protected]> wrote:
Quoted 4 lines · Andrew Morton

No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.

Sorry, if you run code with that obvious warnings you beg for trouble. If you really believe your advanced users arw too stupid to read compiler warnings enforcing -Werror is for sure better than obsfucating the code.

Well, it has happened, fairly regularly. A common scenario is where someone has done a conversion in one tree and someone else has touched overlapping code in another tree and when the two meet in linux-next, splat. Renaming the field simply eliminates this.

Of course, the warnings don't get noticed because of the enormous warning storm which a kernel build produces (generally much worse on non-x86, btw).

Another reason for remaining a field is when we desire that it henceforth be accessed via accessor functions - renaming it will reliably break any unconverted code.

#27 lore raw

PATCH 6/17 fs: icache lock lru/writeback lists

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

---
 fs/fs-writeback.c         |   48 +++++++++++++++++++++++++++++++++++++++++++-
 fs/inode.c                |   44 +++++++++++++++++++++++++++++++++-------
 include/linux/writeback.h |    1 +
 mm/backing-dev.c          |    4 +++
 4 files changed, 87 insertions(+), 10 deletions(-)
fs/fs-writeback.c +46 -2
index 2edaad7..fb7b723 100644
@@ -161,6 +161,7 @@ static void redirty_tail(struct inode *inode)
 {
 	struct bdi_writeback *wb = &inode_to_bdi(inode)->wb;
 
+	assert_spin_locked(&wb_inode_list_lock);
 	if (!list_empty(&wb->b_dirty)) {
 		struct inode *tail;
 
@@ -178,6 +179,7 @@ static void requeue_io(struct inode *inode)
 {
 	struct bdi_writeback *wb = &inode_to_bdi(inode)->wb;
 
+	assert_spin_locked(&wb_inode_list_lock);
 	list_move(&inode->i_list, &wb->b_more_io);
 }
 
@@ -218,6 +220,7 @@ static void move_expired_inodes(struct list_head *delaying_queue,
 	struct inode *inode;
 	int do_sb_sort = 0;
 
+	assert_spin_locked(&wb_inode_list_lock);
 	while (!list_empty(delaying_queue)) {
 		inode = list_entry(delaying_queue->prev, struct inode, i_list);
 		if (older_than_this &&
@@ -281,11 +284,13 @@ static void inode_wait_for_writeback(struct inode *inode)
 
 	wqh = bit_waitqueue(&inode->i_state, __I_SYNC);
 	while (inode->i_state & I_SYNC) {
+		spin_unlock(&wb_inode_list_lock);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 		__wait_on_bit(wqh, &wq, inode_wait, TASK_UNINTERRUPTIBLE);
 		spin_lock(&inode_lock);
 		spin_lock(&inode->i_lock);
+		spin_lock(&wb_inode_list_lock);
 	}
 }
 
@@ -339,6 +344,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 	/* Set I_SYNC, reset I_DIRTY_PAGES */
 	inode->i_state |= I_SYNC;
 	inode->i_state &= ~I_DIRTY_PAGES;
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 
@@ -375,6 +381,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 
 	spin_lock(&inode_lock);
 	spin_lock(&inode->i_lock);
+	spin_lock(&wb_inode_list_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & I_FREEING)) {
 		if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
@@ -461,11 +468,18 @@ static bool pin_sb_for_writeback(struct super_block *sb)
 static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 		struct writeback_control *wbc, bool only_this_sb)
 {
+again:
 	while (!list_empty(&wb->b_io)) {
 		long pages_skipped;
 		struct inode *inode = list_entry(wb->b_io.prev,
 						 struct inode, i_list);
 
+		if (!spin_trylock(&inode->i_lock)) {
+			spin_unlock(&wb_inode_list_lock);
+			spin_lock(&wb_inode_list_lock);
+			goto again;
+		}
+
 		if (inode->i_sb != sb) {
 			if (only_this_sb) {
 				/*
@@ -474,9 +488,12 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 				 * to it back onto the dirty list.
 				 */
 				redirty_tail(inode);
+				spin_unlock(&inode->i_lock);
 				continue;
 			}
 
+			spin_unlock(&inode->i_lock);
+
 			/*
 			 * The inode belongs to a different superblock.
 			 * Bounce back to the caller to unpin this and
@@ -485,10 +502,9 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 			return 0;
 		}
 
-		spin_lock(&inode->i_lock);
 		if (inode->i_state & (I_NEW | I_WILL_FREE)) {
-			spin_unlock(&inode->i_lock);
 			requeue_io(inode);
+			spin_unlock(&inode->i_lock);
 			continue;
 		}
 		/*
@@ -511,11 +527,13 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,
 			 */
 			redirty_tail(inode);
 		}
+		spin_unlock(&wb_inode_list_lock);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
+		spin_lock(&wb_inode_list_lock);
 		if (wbc->nr_to_write <= 0) {
 			wbc->more_io = 1;
 			return 1;
@@ -535,6 +553,9 @@ void writeback_inodes_wb(struct bdi_writeback *wb,
 	if (!wbc->wb_start)
 		wbc->wb_start = jiffies; /* livelock avoidance */
 	spin_lock(&inode_lock);
+again:
+	spin_lock(&wb_inode_list_lock);
+
 	if (!wbc->for_kupdate || list_empty(&wb->b_io))
 		queue_io(wb, wbc->older_than_this);
 
@@ -544,7 +565,12 @@ void writeback_inodes_wb(struct bdi_writeback *wb,
 		struct super_block *sb = inode->i_sb;
 
 		if (!pin_sb_for_writeback(sb)) {
+			if (!spin_trylock(&inode->i_lock)) {
+				spin_unlock(&wb_inode_list_lock);
+				goto again;
+			}
 			requeue_io(inode);
+			spin_unlock(&inode->i_lock);
 			continue;
 		}
 		ret = writeback_sb_inodes(sb, wb, wbc, false);
@@ -553,6 +579,7 @@ void writeback_inodes_wb(struct bdi_writeback *wb,
 		if (ret)
 			break;
 	}
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode_lock);
 	/* Leave any unwritten inodes on b_io */
 }
@@ -563,9 +590,11 @@ static void __writeback_inodes_sb(struct super_block *sb,
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	spin_lock(&inode_lock);
+	spin_lock(&wb_inode_list_lock);
 	if (!wbc->for_kupdate || list_empty(&wb->b_io))
 		queue_io(wb, wbc->older_than_this);
 	writeback_sb_inodes(sb, wb, wbc, true);
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode_lock);
 }
 
@@ -676,13 +705,22 @@ static long wb_writeback(struct bdi_writeback *wb,
 		 * become available for writeback. Otherwise
 		 * we'll just busyloop.
 		 */
+retry:
 		spin_lock(&inode_lock);
+		spin_lock(&wb_inode_list_lock);
 		if (!list_empty(&wb->b_more_io))  {
 			inode = list_entry(wb->b_more_io.prev,
 						struct inode, i_list);
+			if (!spin_trylock(&inode->i_lock)) {
+				spin_unlock(&wb_inode_list_lock);
+				spin_unlock(&inode_lock);
+				goto retry;
+			}
 			trace_wbc_writeback_wait(&wbc, wb->bdi);
 			inode_wait_for_writeback(inode);
+			spin_unlock(&inode->i_lock);
 		}
+		spin_unlock(&wb_inode_list_lock);
 		spin_unlock(&inode_lock);
 	}
 
@@ -993,8 +1031,10 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 					wakeup_bdi = true;
 			}
 
+			spin_lock(&wb_inode_list_lock);
 			inode->dirtied_when = jiffies;
 			list_move(&inode->i_list, &bdi->wb.b_dirty);
+			spin_unlock(&wb_inode_list_lock);
 		}
 	}
 out:
@@ -1183,7 +1223,9 @@ int write_inode_now(struct inode *inode, int sync)
 	might_sleep();
 	spin_lock(&inode_lock);
 	spin_lock(&inode->i_lock);
+	spin_lock(&wb_inode_list_lock);
 	ret = writeback_single_inode(inode, &wbc);
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	if (sync)
@@ -1209,7 +1251,9 @@ int sync_inode(struct inode *inode, struct writeback_control *wbc)
 
 	spin_lock(&inode_lock);
 	spin_lock(&inode->i_lock);
+	spin_lock(&wb_inode_list_lock);
 	ret = writeback_single_inode(inode, wbc);
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	return ret;
fs/inode.c +36 -8

144 lines not on this page · raw message

include/linux/writeback.h +1 -0
index 35d6e81..8b9c24f 100644
@@ -11,6 +11,7 @@ struct backing_dev_info;
 
 extern spinlock_t inode_lock;
 extern spinlock_t sb_inode_list_lock;
+extern spinlock_t wb_inode_list_lock;
 extern spinlock_t inode_hash_lock;
 extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
mm/backing-dev.c +4 -0
index c2bf86f..b1e2987 100644
@@ -73,12 +73,14 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
 
 	nr_wb = nr_dirty = nr_io = nr_more_io = 0;
 	spin_lock(&inode_lock);
+	spin_lock(&wb_inode_list_lock);
 	list_for_each_entry(inode, &wb->b_dirty, i_list)
 		nr_dirty++;
 	list_for_each_entry(inode, &wb->b_io, i_list)
 		nr_io++;
 	list_for_each_entry(inode, &wb->b_more_io, i_list)
 		nr_more_io++;
+	spin_unlock(&wb_inode_list_lock);
 	spin_unlock(&inode_lock);
 
 	global_dirty_limits(&background_thresh, &dirty_thresh);
@@ -681,9 +683,11 @@ void bdi_destroy(struct backing_dev_info *bdi)
 		struct bdi_writeback *dst = &default_backing_dev_info.wb;
 
 		spin_lock(&inode_lock);
+		spin_lock(&wb_inode_list_lock);
 		list_splice(&bdi->wb.b_dirty, &dst->b_dirty);
 		list_splice(&bdi->wb.b_io, &dst->b_io);
 		list_splice(&bdi->wb.b_more_io, &dst->b_more_io);
+		spin_unlock(&wb_inode_list_lock);
 		spin_unlock(&inode_lock);
 	}
 
Signature · 1 line
1.7.1
maintainer
#28 lore raw

Re: [PATCH 06/17] fs: icache lock lru/writeback lists

Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.

I'd suggest that each such site have a comment explaining why we're resorting to this.

#29 lore raw

Re: [PATCH 06/17] fs: icache lock lru/writeback lists

Quoted 10 lines · Andrew Morton
Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.

Yes, they are, but I don't know exactly why it is so trylock happy. I'll try to dig out the reason for it and:

Quoted 2 lines

I'd suggest that each such site have a comment explaining why we're resorting to this.

At least get this far.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#30 lore raw

Re: [PATCH 06/17] fs: icache lock lru/writeback lists

In reply to Andrew Morton
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

Quoted 5 lines

All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.

I'd suggest that each such site have a comment explaining why we're resorting to this.

They're really a side effect of how I'm building up the locking in steps and then streamlining it in steps. Most of them disappear or get much improved as inode removal, rcu, etc greatly help with lock ordering.

The intermediate steps are not supposed to be so pretty, so much as an easily verifiable "ok, we have enough locking to cover what inode_lock used to be protecting".

#31 lore raw

Re: [PATCH 06/17] fs: icache lock lru/writeback lists

In reply to Dave Chinner
Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

As a band-aid to get rid of the inode_lock this might be fine, but I don't really like it. For one all the list are per-bdi_writeback, so the lock should be as well. Second the lock is held over far too long periods during writeback, which leads to a lot of whacky trylock operations and unlock and sleep cycles inside it. In practice we only need it in the places where we manipulate the lists.

Also it feels like it really should nest outside i_lock, not inside it, but I need to look more deeply to figure why that might not easily be possible.

But maybe all that is better left as a separate patch on top of the current queue.

#32 lore raw

Re: [PATCH 06/17] fs: icache lock lru/writeback lists

Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.

Quoted 6 lines

As a band-aid to get rid of the inode_lock this might be fine, but I don't really like it. For one all the list are per-bdi_writeback, so the lock should be as well. Second the lock is held over far too long periods during writeback, which leads to a lot of whacky trylock operations and unlock and sleep cycles inside it. In practice we only need it in the places where we manipulate the lists.

per-bdi writeback lock won't work with the patch set as it stands - it also protects the LRU which is a global list. I'll have to pull back another patch to split the LRU and IO lists to make this lock per-bdi.

Quoted 3 lines

Also it feels like it really should nest outside i_lock, not inside it, but I need to look more deeply to figure why that might not easily be possible.

Yeah, I'm trying to rework the patch series to not nest anything inside i_lock. The more I look at all that trylock stuff, the more my eyes bleed....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#33 lore raw

PATCH 7/17 fs: icache atomic inodes_stat

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

The inode use statistics are currently protected by the inode_lock. Before we can remove the inode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.

---
 fs/fs-writeback.c  |    6 ++++--
 fs/inode.c         |   26 ++++++++++++++------------
 include/linux/fs.h |   13 ++++++-------
 3 files changed, 24 insertions(+), 21 deletions(-)
fs/fs-writeback.c +4 -2
index fb7b723..f6c8975 100644
@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));
 
 	if (nr_pages) {
 		struct wb_writeback_work work = {
@@ -1144,7 +1145,8 @@ void writeback_inodes_sb(struct super_block *sb)
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	work.nr_pages = nr_dirty + nr_unstable +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));
 
 	bdi_queue_work(sb->s_bdi, &work);
 	wait_for_completion(&done);
fs/inode.c +14 -12
index e15620f..6d982e6 100644
@@ -122,7 +122,10 @@ static DECLARE_RWSEM(iprune_sem);
 /*
  * Statistics gathering..
  */
-struct inodes_stat_t inodes_stat;
+struct inodes_stat_t inodes_stat = {
+	.nr_inodes = ATOMIC_INIT(0),
+	.nr_unused = ATOMIC_INIT(0),
+};
 
 static struct kmem_cache *inode_cachep __read_mostly;
 
@@ -313,7 +316,7 @@ void __iget(struct inode *inode)
 		list_move(&inode->i_list, &inode_in_use);
 		spin_unlock(&wb_inode_list_lock);
 	}
-	inodes_stat.nr_unused--;
+	atomic_dec(&inodes_stat.nr_unused);
 }
 
 void end_writeback(struct inode *inode)
@@ -377,9 +380,7 @@ static void dispose_list(struct list_head *head)
 		destroy_inode(inode);
 		nr_disposed++;
 	}
-	spin_lock(&inode_lock);
-	inodes_stat.nr_inodes -= nr_disposed;
-	spin_unlock(&inode_lock);
+	atomic_sub(nr_disposed, &inodes_stat.nr_inodes);
 }
 
 /*
@@ -428,7 +429,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 		busy = 1;
 	}
 	/* only unused inodes may be cached with i_count zero */
-	inodes_stat.nr_unused -= count;
+	atomic_sub(count, &inodes_stat.nr_unused);
 	return busy;
 }
 
@@ -545,7 +546,7 @@ again2:
 		spin_unlock(&inode->i_lock);
 		nr_pruned++;
 	}
-	inodes_stat.nr_unused -= nr_pruned;
+	atomic_sub(nr_pruned, &inodes_stat.nr_unused);
 	if (current_is_kswapd())
 		__count_vm_events(KSWAPD_INODESTEAL, reap);
 	else
@@ -578,7 +579,8 @@ static int shrink_icache_memory(struct shrinker *shrink, int nr, gfp_t gfp_mask)
 			return -1;
 		prune_icache(nr);
 	}
-	return (inodes_stat.nr_unused / 100) * sysctl_vfs_cache_pressure;
+	return (atomic_read(&inodes_stat.nr_unused) / 100) *
+					sysctl_vfs_cache_pressure;
 }
 
 static struct shrinker icache_shrinker = {
@@ -671,7 +673,7 @@ static inline void
 __inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
 			struct inode *inode)
 {
-	inodes_stat.nr_inodes++;
+	atomic_inc(&inodes_stat.nr_inodes);
 	spin_lock(&sb_inode_list_lock);
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
@@ -1336,7 +1338,7 @@ static void iput_final(struct inode *inode)
 			list_move(&inode->i_list, &inode_unused);
 			spin_unlock(&wb_inode_list_lock);
 		}
-		inodes_stat.nr_unused++;
+		atomic_inc(&inodes_stat.nr_unused);
 		if (sb->s_flags & MS_ACTIVE) {
 			spin_unlock(&inode->i_lock);
 			spin_unlock(&sb_inode_list_lock);
@@ -1354,7 +1356,7 @@ static void iput_final(struct inode *inode)
 		spin_lock(&inode->i_lock);
 		WARN_ON(inode->i_state & I_NEW);
 		inode->i_state &= ~I_WILL_FREE;
-		inodes_stat.nr_unused--;
+		atomic_dec(&inodes_stat.nr_unused);
 		spin_lock(&inode_hash_lock);
 		hlist_del_init(&inode->i_hash);
 		spin_unlock(&inode_hash_lock);
@@ -1366,9 +1368,9 @@ static void iput_final(struct inode *inode)
 	spin_unlock(&sb_inode_list_lock);
 	WARN_ON(inode->i_state & I_NEW);
 	inode->i_state |= I_FREEING;
-	inodes_stat.nr_inodes--;
 	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
+	atomic_dec(&inodes_stat.nr_inodes);
 	evict(inode);
 	spin_lock(&inode_lock);
 	spin_lock(&inode_hash_lock);
include/linux/fs.h +6 -7
index 415c88e..d60f256 100644
@@ -39,13 +39,6 @@ struct files_stat_struct {
 	int max_files;		/* tunable */
 };
 
-struct inodes_stat_t {
-	int nr_inodes;
-	int nr_unused;
-	int dummy[5];		/* padding for sysctl ABI compatibility */
-};
-
-
 #define NR_FILE  8192	/* this can well be larger on a larger system */
 
 #define MAY_EXEC 1
@@ -416,6 +409,12 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset,
 			ssize_t bytes, void *private, int ret,
 			bool is_async);
 
+struct inodes_stat_t {
+	atomic_t nr_inodes;
+	atomic_t nr_unused;
+	int dummy[5];		/* padding for sysctl ABI compatibility */
+};
+
 /*
  * Attribute flags.  These should be or-ed together to figure out what
  * has been changed!
Signature · 1 line
1.7.1
maintainer
#34 lore raw

Re: [PATCH 07/17] fs: icache atomic inodes_stat

Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

The inode use statistics are currently protected by the inode_lock. Before we can remove the inode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.

typo

fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768

Quoted 12 lines

...

fs/fs-writeback.c +2 -1
@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));

race bug.

fs/fs-writeback.c void writeback_inodes_sb(struct super_block *sb) · line 1149

Quoted 9 lines
fs/fs-writeback.c +2 -1
 	if (nr_pages) {
 		struct wb_writeback_work work = {
@@ -1144,7 +1145,8 @@ void writeback_inodes_sb(struct super_block *sb)
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	work.nr_pages = nr_dirty + nr_unstable +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));

and another.

OK, they aren't serious ones. But known regressions shouldn't be snuck into the kernel unchangelogged and uncommented :(

Quoted 16 lines
fs/fs-writeback.c +0 -7
 	bdi_queue_work(sb->s_bdi, &work);
 	wait_for_completion(&done);
 
...
 
-struct inodes_stat_t {
-	int nr_inodes;
-	int nr_unused;
-	int dummy[5];		/* padding for sysctl ABI compatibility */
-};
-
-
 #define NR_FILE  8192	/* this can well be larger on a larger system */
 
 #define MAY_EXEC 1

fs/fs-writeback.c typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · line 416

Quoted 8 lines
fs/fs-writeback.c +5 -0
@@ -416,6 +409,12 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset,
 			ssize_t bytes, void *private, int ret,
 			bool is_async);
 
+struct inodes_stat_t {
+	atomic_t nr_inodes;
+	atomic_t nr_unused;
+	int dummy[5];		/* padding for sysctl ABI compatibility */
+};

OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an atomic_t has the same layout, alignment and padding as an int.

Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.

It shouldn't be snuck into the tree unchangelogged and uncommented.

(time passes)

OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.

#35 lore raw

Re: [PATCH 07/17] fs: icache atomic inodes_stat

Quoted 14 lines · Andrew Morton
Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

The inode use statistics are currently protected by the inode_lock. Before we can remove the inode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.

typo

Quoted 3 lines

...

Quoted 11 lines

fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768

Quoted 9 lines
fs/fs-writeback.c +2 -1
@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));

race bug.

What new race? The code has gone from using subtraction of unlocked counters to using subtraction of unlocked atomic counters. I can't see any new race condition in that transformation....

Quoted 13 lines

fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb)

Quoted 5 lines
fs/fs-writeback.c +5 -0
+struct inodes_stat_t {
+	atomic_t nr_inodes;
+	atomic_t nr_unused;
+	int dummy[5];		/* padding for sysctl ABI compatibility */
+};

OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an atomic_t has the same layout, alignment and padding as an int.

Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.

Quoted 6 lines

It shouldn't be snuck into the tree unchangelogged and uncommented.

(time passes)

OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.

That's the multiple changes to the counter infrastructure I described in the preliminary series description. clearly it needs to be cleaned up.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
maintainer
#36 lore raw

Re: [PATCH 07/17] fs: icache atomic inodes_stat

Quoted 12 lines · On Thu, 30 Sep 2010 16:20:57 +1000 Dave Chinner <[email protected]> wrote:
Quoted 11 lines

fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768

Quoted 9 lines
fs/fs-writeback.c +2 -1
@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
+			(atomic_read(&inodes_stat.nr_inodes) -
+			atomic_read(&inodes_stat.nr_unused));

race bug.

Quoted 3 lines

What new race? The code has gone from using subtraction of unlocked counters to using subtraction of unlocked atomic counters. I can't see any new race condition in that transformation....

Oh. I assumed that all the above used to happen under inode_lock.

Coz that's wot the changelog told me :)

#37 lore raw

Re: [PATCH 07/17] fs: icache atomic inodes_stat

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

The inode use statistics are currently protected by the inode_lock. Before we can remove the inode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.

typo

It's Dave, I swear :)

Quoted 13 lines

include/linux/fs.h typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · line 416

Quoted 5 lines
include/linux/fs.h +5 -0
+struct inodes_stat_t {
+	atomic_t nr_inodes;
+	atomic_t nr_unused;
+	int dummy[5];		/* padding for sysctl ABI compatibility */
+};

OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an atomic_t has the same layout, alignment and padding as an int.

Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.

Quoted 6 lines

It shouldn't be snuck into the tree unchangelogged and uncommented.

(time passes)

OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.

Yeah it is. I might end up folding the per-cpu stuff back over it and avoid the issue completely. Otherwise I'll add a comment.

#38 lore raw

PATCH 8/17 fs: icache protect inode state

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to protect the inode list operations with the inode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.

---
 fs/inode.c |   33 +++++++++++++++++++++++++++++----
 1 files changed, 29 insertions(+), 4 deletions(-)
fs/inode.c +29 -4
index 6d982e6..8fbc4d4 100644
@@ -34,7 +34,11 @@
  * wb_inode_list_lock protects:
  *   inode_in_use, inode_unused, b_io, b_more_io, b_dirty, i_list
  * inode->i_lock protects:
- *   i_state, i_count
+ *   i_state
+ *   i_count
+ *   i_hash
+ *   i_list
+ *   i_sb_list
  *
  * Ordering:
  * inode_lock
@@ -368,12 +372,14 @@ static void dispose_list(struct list_head *head)
 		evict(inode);
 
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
+		spin_lock(&inode->i_lock);
 		spin_lock(&inode_hash_lock);
 		hlist_del_init(&inode->i_hash);
 		spin_unlock(&inode_hash_lock);
-		spin_lock(&sb_inode_list_lock);
 		list_del_init(&inode->i_sb_list);
 		spin_unlock(&sb_inode_list_lock);
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 
 		wake_up_inode(inode);
@@ -674,7 +680,6 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
 			struct inode *inode)
 {
 	atomic_inc(&inodes_stat.nr_inodes);
-	spin_lock(&sb_inode_list_lock);
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
 	spin_lock(&wb_inode_list_lock);
@@ -704,7 +709,10 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)
 	struct hlist_head *head = inode_hashtable + hash(sb, inode->i_ino);
 
 	spin_lock(&inode_lock);
+	spin_lock(&sb_inode_list_lock);
+	spin_lock(&inode->i_lock);
 	__inode_add_to_lists(sb, head, inode);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 }
 EXPORT_SYMBOL_GPL(inode_add_to_lists);
@@ -736,9 +744,12 @@ struct inode *new_inode(struct super_block *sb)
 	inode = alloc_inode(sb);
 	if (inode) {
 		spin_lock(&inode_lock);
+		spin_lock(&sb_inode_list_lock);
+		spin_lock(&inode->i_lock);
 		inode->i_ino = ++last_ino;
 		inode->i_state = 0;
 		__inode_add_to_lists(sb, NULL, inode);
+		spin_unlock(&inode->i_lock);
 		spin_unlock(&inode_lock);
 	}
 	return inode;
@@ -802,11 +813,14 @@ static struct inode *get_new_inode(struct super_block *sb,
 		/* We released the lock, so.. */
 		old = find_inode(sb, head, test, data);
 		if (!old) {
+			spin_lock(&sb_inode_list_lock);
+			spin_lock(&inode->i_lock);
 			if (set(inode, data))
 				goto set_failed;
 
 			inode->i_state = I_NEW;
 			__inode_add_to_lists(sb, head, inode);
+			spin_unlock(&inode->i_lock);
 			spin_unlock(&inode_lock);
 
 			/* Return the locked inode with I_NEW set, the
@@ -831,6 +845,7 @@ static struct inode *get_new_inode(struct super_block *sb,
 
 set_failed:
 	spin_unlock(&inode->i_lock);
+	spin_unlock(&sb_inode_list_lock);
 	spin_unlock(&inode_lock);
 	destroy_inode(inode);
 	return NULL;
@@ -853,9 +868,12 @@ static struct inode *get_new_inode_fast(struct super_block *sb,
 		/* We released the lock, so.. */
 		old = find_inode_fast(sb, head, ino);
 		if (!old) {
+			spin_lock(&sb_inode_list_lock);
+			spin_lock(&inode->i_lock);
 			inode->i_ino = ino;
 			inode->i_state = I_NEW;
 			__inode_add_to_lists(sb, head, inode);
+			spin_unlock(&inode->i_lock);
 			spin_unlock(&inode_lock);
 
 			/* Return the locked inode with I_NEW set, the
@@ -1270,10 +1288,13 @@ EXPORT_SYMBOL(insert_inode_locked4);
 void __insert_inode_hash(struct inode *inode, unsigned long hashval)
 {
 	struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
+
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	spin_lock(&inode_hash_lock);
 	hlist_add_head(&inode->i_hash, head);
 	spin_unlock(&inode_hash_lock);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 }
 EXPORT_SYMBOL(__insert_inode_hash);
@@ -1287,9 +1308,11 @@ EXPORT_SYMBOL(__insert_inode_hash);
 void remove_inode_hash(struct inode *inode)
 {
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	spin_lock(&inode_hash_lock);
 	hlist_del_init(&inode->i_hash);
 	spin_unlock(&inode_hash_lock);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 }
 EXPORT_SYMBOL(remove_inode_hash);
@@ -1356,10 +1379,10 @@ static void iput_final(struct inode *inode)
 		spin_lock(&inode->i_lock);
 		WARN_ON(inode->i_state & I_NEW);
 		inode->i_state &= ~I_WILL_FREE;
-		atomic_dec(&inodes_stat.nr_unused);
 		spin_lock(&inode_hash_lock);
 		hlist_del_init(&inode->i_hash);
 		spin_unlock(&inode_hash_lock);
+		atomic_dec(&inodes_stat.nr_unused);
 	}
 	spin_lock(&wb_inode_list_lock);
 	list_del_init(&inode->i_list);
@@ -1373,9 +1396,11 @@ static void iput_final(struct inode *inode)
 	atomic_dec(&inodes_stat.nr_inodes);
 	evict(inode);
 	spin_lock(&inode_lock);
+	spin_lock(&inode->i_lock);
 	spin_lock(&inode_hash_lock);
 	hlist_del_init(&inode->i_hash);
 	spin_unlock(&inode_hash_lock);
+	spin_unlock(&inode->i_lock);
 	spin_unlock(&inode_lock);
 	wake_up_inode(inode);
 	BUG_ON(inode->i_state != (I_FREEING | I_CLEAR));
Signature · 1 line
1.7.1
#39 lore raw

Re: [PATCH 08/17] fs: icache protect inode state

Quoted 8 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to protect the inode list operations with the inode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.

The subject does not seem to match the patch description and content.

#40 lore raw

Re: [PATCH 08/17] fs: icache protect inode state

Quoted 11 lines · Christoph Hellwig
Quoted 8 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to protect the inode list operations with the inode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.

The subject does not seem to match the patch description and content.

It is adding i_lock around remaining places where an inode can be moved on or off icache data structures. As I've described, this is quite central to my locking design, isn't the changelog understandable?

#41 lore raw

PATCH 9/17 fs: Make last_ino, iunique independent of inode_lock

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to make the last_ino and iunique counters independent of the inode_lock. last_ino can be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside the inode_lock to provide the same protection that the inode_lock previously provided.

---
 fs/inode.c |   28 ++++++++++++++++++++++------
 1 files changed, 22 insertions(+), 6 deletions(-)
fs/inode.c +22 -6
index 8fbc4d4..c45b2db 100644
@@ -736,7 +736,7 @@ struct inode *new_inode(struct super_block *sb)
 	 * error if st_ino won't fit in target struct field. Use 32bit counter
 	 * here to attempt to avoid that.
 	 */
-	static unsigned int last_ino;
+	static atomic_t last_ino = ATOMIC_INIT(0);
 	struct inode *inode;
 
 	spin_lock_prefetch(&inode_lock);
@@ -746,7 +746,7 @@ struct inode *new_inode(struct super_block *sb)
 		spin_lock(&inode_lock);
 		spin_lock(&sb_inode_list_lock);
 		spin_lock(&inode->i_lock);
-		inode->i_ino = ++last_ino;
+		inode->i_ino = (unsigned int)atomic_inc_return(&last_ino);
 		inode->i_state = 0;
 		__inode_add_to_lists(sb, NULL, inode);
 		spin_unlock(&inode->i_lock);
@@ -897,6 +897,22 @@ static struct inode *get_new_inode_fast(struct super_block *sb,
 	return inode;
 }
 
+static int test_inode_iunique(struct super_block * sb, struct hlist_head *head, unsigned long ino)
+{
+	struct hlist_node *node;
+	struct inode * inode = NULL;
+
+	spin_lock(&inode_hash_lock);
+	hlist_for_each_entry(inode, node, head, i_hash) {
+		if (inode->i_ino == ino && inode->i_sb == sb) {
+			spin_unlock(&inode_hash_lock);
+			return 0;
+		}
+	}
+	spin_unlock(&inode_hash_lock);
+	return 1;
+}
+
 /**
  *	iunique - get a unique inode number
  *	@sb: superblock
@@ -918,20 +934,20 @@ ino_t iunique(struct super_block *sb, ino_t max_reserved)
 	 * error if st_ino won't fit in target struct field. Use 32bit counter
 	 * here to attempt to avoid that.
 	 */
+	static DEFINE_SPINLOCK(unique_lock);
 	static unsigned int counter;
-	struct inode *inode;
 	struct hlist_head *head;
 	ino_t res;
 
 	spin_lock(&inode_lock);
+	spin_lock(&unique_lock);
 	do {
 		if (counter <= max_reserved)
 			counter = max_reserved + 1;
 		res = counter++;
 		head = inode_hashtable + hash(sb, res);
-		inode = find_inode_fast(sb, head, res);
-		spin_unlock(&inode->i_lock);
-	} while (inode != NULL);
+	} while (!test_inode_iunique(sb, head, res));
+	spin_unlock(&unique_lock);
 	spin_unlock(&inode_lock);
 
 	return res;
Signature · 1 line
1.7.1
maintainer
#42 lore raw

Re: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock

Quoted 16 lines · On Wed, 29 Sep 2010 22:18:41 +1000 Dave Chinner <[email protected]> wrote:

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to make the last_ino and iunique counters independent of the inode_lock. last_ino can be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside the inode_lock to provide the same protection that the inode_lock previously provided.

...

fs/inode.c +5 -0
+static int test_inode_iunique(struct super_block * sb, struct hlist_head *head, unsigned long ino)
+{
+	struct hlist_node *node;
+	struct inode * inode = NULL;
+

fs/inode.c static struct inode *get_new_inode_fast(struct super_block *sb, · line 914

Quoted 10 lines
fs/inode.c +10 -0
+	spin_lock(&inode_hash_lock);
+	hlist_for_each_entry(inode, node, head, i_hash) {
+		if (inode->i_ino == ino && inode->i_sb == sb) {
+			spin_unlock(&inode_hash_lock);
+			return 0;
+		}
+	}
+	spin_unlock(&inode_hash_lock);
+	return 1;
+}

Please run all the patches through checkpatch.

Please document this function. Why does it exist? What does it do? How does it do it? Try to improve the code!

Quoted 3 lines

...

#43 lore raw

Re: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock

In reply to Dave Chinner
Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to make the last_ino and iunique counters independent of the inode_lock. last_ino can be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside the inode_lock to provide the same protection that the inode_lock previously provided.

Given that last_ino becomes a per-cpu construct only a few patches later I think there's no point to make it an atomic_t here - just reorder the per-cpu patch before the inode_lock removal.

#44 lore raw

Re: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock

Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before removing the inode_lock, we need to make the last_ino and iunique counters independent of the inode_lock. last_ino can be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside the inode_lock to provide the same protection that the inode_lock previously provided.

Quoted 3 lines

Given that last_ino becomes a per-cpu construct only a few patches later I think there's no point to make it an atomic_t here - just reorder the per-cpu patch before the inode_lock removal.

I wanted to avoid doing any of that until inode_lock is gone, but perhaps for this one it makes sense. At the very least, I'll merge the latter two patches into one, and perhaps this one too.

#45 lore raw

PATCH 10/17 fs: icache remove inode_lock

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

All the functionality that the inode_lock protected has now been wrapped up in new independent locks and/or functionality. Hence the inode_lock does not serve a purpose any longer and hence can now be removed.

---
 Documentation/filesystems/Locking |    2 +-
 Documentation/filesystems/porting |   10 ++++-
 Documentation/filesystems/vfs.txt |    2 +-
 fs/buffer.c                       |    2 +-
 fs/drop_caches.c                  |    4 --
 fs/fs-writeback.c                 |   47 ++++--------------
 fs/inode.c                        |   95 +++++++------------------------------
 fs/logfs/inode.c                  |    2 +-
 fs/notify/inode_mark.c            |   23 +++++----
 fs/notify/mark.c                  |    1 -
 fs/notify/vfsmount_mark.c         |    1 -
 fs/ntfs/inode.c                   |    4 +-
 fs/ocfs2/inode.c                  |    2 +-
 fs/quota/dquot.c                  |   16 ++----
 include/linux/fs.h                |    2 +-
 include/linux/writeback.h         |    1 -
 mm/backing-dev.c                  |    4 --
 mm/filemap.c                      |    6 +-
 mm/rmap.c                         |    6 +-
 19 files changed, 69 insertions(+), 161 deletions(-)
Documentation/filesystems/Locking +1 -1
index 2db4283..e92dad2 100644
@@ -114,7 +114,7 @@ alloc_inode:
 destroy_inode:
 dirty_inode:				(must not sleep)
 write_inode:
-drop_inode:				!!!inode_lock!!!
+drop_inode:				!!!i_lock, sb_inode_list_lock!!!
 evict_inode:
 put_super:		write
 write_super:		read
Documentation/filesystems/porting +9 -1
index b12c895..ab07213 100644
@@ -299,7 +299,7 @@ be used instead.  It gets called whenever the inode is evicted, whether it has
 remaining links or not.  Caller does *not* evict the pagecache or inode-associated
 metadata buffers; getting rid of those is responsibility of method, as it had
 been for ->delete_inode().
-	->drop_inode() returns int now; it's called on final iput() with inode_lock
+	->drop_inode() returns int now; it's called on final iput() with i_lock
 held and it returns true if filesystems wants the inode to be dropped.  As before,
 generic_drop_inode() is still the default and it's been updated appropriately.
 generic_delete_inode() is also alive and it consists simply of return 1.  Note that
@@ -318,3 +318,11 @@ if it's zero is not *and* *never* *had* *been* enough.  Final unlink() and iput(
 may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly
 free the on-disk inode, you may end up doing that while ->write_inode() is writing
 to it.
+
+
+[mandatory]
+	inode_lock is gone, replaced by fine grained locks. See fs/inode.c
+for details of what locks to replace inode_lock with in order to protect
+particular things. Most of the time, a filesystem only needs ->i_lock, which
+protects *all* the inode state and its membership on lists that was
+previously protected with inode_lock.
Documentation/filesystems/vfs.txt +1 -1
index ed7e5ef..405beb2 100644
@@ -246,7 +246,7 @@ or bottom half).
 	should be synchronous or not, not all filesystems check this flag.
 
   drop_inode: called when the last access to the inode is dropped,
-	with the inode_lock spinlock held.
+	with the i_lock and sb_inode_list_lock spinlock held.
 
 	This method should be either NULL (normal UNIX filesystem
 	semantics) or "generic_delete_inode" (for filesystems that do not
fs/buffer.c +1 -1
index 3e7dca2..66f7afd 100644
@@ -1145,7 +1145,7 @@ __getblk_slow(struct block_device *bdev, sector_t block, int size)
  * inode list.
  *
  * mark_buffer_dirty() is atomic.  It takes bh->b_page->mapping->private_lock,
- * mapping->tree_lock and the global inode_lock.
+ * and mapping->tree_lock.
  */
 void mark_buffer_dirty(struct buffer_head *bh)
 {
fs/drop_caches.c +0 -4
index 45bdf88..0884447 100644
@@ -16,7 +16,6 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)
 {
 	struct inode *inode, *toput_inode = NULL;
 
-	spin_lock(&inode_lock);
 	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		spin_lock(&inode->i_lock);
@@ -28,15 +27,12 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)
 		__iget(inode);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
-		spin_unlock(&inode_lock);
 		invalidate_mapping_pages(inode->i_mapping, 0, -1);
 		iput(toput_inode);
 		toput_inode = inode;
-		spin_lock(&inode_lock);
 		spin_lock(&sb_inode_list_lock);
 	}
 	spin_unlock(&sb_inode_list_lock);
-	spin_unlock(&inode_lock);
 	iput(toput_inode);
 }
 
fs/fs-writeback.c +11 -36

199 lines not on this page · raw message

fs/inode.c +18 -77

428 lines not on this page · raw message

fs/logfs/inode.c +1 -1
index d8c71ec..a67b607 100644
@@ -286,7 +286,7 @@ static int logfs_write_inode(struct inode *inode, struct writeback_control *wbc)
 	return ret;
 }
 
-/* called with inode_lock held */
+/* called with i_lock held */
 static int logfs_drop_inode(struct inode *inode)
 {
 	struct logfs_super *super = logfs_super(inode->i_sb);
fs/notify/inode_mark.c +12 -11
index 70f7e16..e51d065 100644
@@ -22,7 +22,7 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
-#include <linux/writeback.h> /* for inode_lock */
+#include <linux/writeback.h>
 
 #include <asm/atomic.h>
 
@@ -232,9 +232,8 @@ out:
  * fsnotify_unmount_inodes - an sb is unmounting.  handle any watched inodes.
  * @list: list of inodes being unmounted (sb->s_inodes)
  *
- * Called with inode_lock held, protecting the unmounting super block's list
- * of inodes, and with iprune_mutex held, keeping shrink_icache_memory() at bay.
- * We temporarily drop inode_lock, however, and CAN block.
+ * Called with iprune_mutex held, keeping shrink_icache_memory() at bay.
+ * sb_inode_list_lock to protect the super block's list of inodes.
  */
 void fsnotify_unmount_inodes(struct list_head *list)
 {
@@ -243,13 +242,16 @@ void fsnotify_unmount_inodes(struct list_head *list)
 	list_for_each_entry_safe(inode, next_i, list, i_sb_list) {
 		struct inode *need_iput_tmp;
 
+		spin_lock(&inode->i_lock);
 		/*
 		 * We cannot __iget() an inode in state I_FREEING,
 		 * I_WILL_FREE, or I_NEW which is fine because by that point
 		 * the inode cannot have any associated watches.
 		 */
-		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW))
+		if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) {
+			spin_unlock(&inode->i_lock);
 			continue;
+		}
 
 		/*
 		 * If i_count is zero, the inode cannot have any watches and
@@ -257,19 +259,20 @@ void fsnotify_unmount_inodes(struct list_head *list)
 		 * evict all inodes with zero i_count from icache which is
 		 * unnecessarily violent and may in fact be illegal to do.
 		 */
-		if (!inode->i_count)
+		if (!inode->i_count) {
+			spin_unlock(&inode->i_lock);
 			continue;
+		}
 
 		need_iput_tmp = need_iput;
 		need_iput = NULL;
 
 		/* In case fsnotify_inode_delete() drops a reference. */
 		if (inode != need_iput_tmp) {
-			spin_lock(&inode->i_lock);
 			__iget(inode);
-			spin_unlock(&inode->i_lock);
 		} else
 			need_iput_tmp = NULL;
+		spin_unlock(&inode->i_lock);
 
 		/* In case the dropping of a reference would nuke next_i. */
 		if (&next_i->i_sb_list != list) {
@@ -283,13 +286,12 @@ void fsnotify_unmount_inodes(struct list_head *list)
 		}
 
 		/*
-		 * We can safely drop inode_lock here because we hold
+		 * We can safely drop sb_inode_list_lock here because we hold
 		 * references on both inode and next_i.  Also no new inodes
 		 * will be added since the umount has begun.  Finally,
 		 * iprune_mutex keeps shrink_icache_memory() away.
 		 */
 		spin_unlock(&sb_inode_list_lock);
-		spin_unlock(&inode_lock);
 
 		if (need_iput_tmp)
 			iput(need_iput_tmp);
@@ -301,7 +303,6 @@ void fsnotify_unmount_inodes(struct list_head *list)
 
 		iput(inode);
 
-		spin_lock(&inode_lock);
 		spin_lock(&sb_inode_list_lock);
 	}
 }
fs/notify/mark.c +0 -1
index 325185e..50c0085 100644
@@ -91,7 +91,6 @@
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/srcu.h>
-#include <linux/writeback.h> /* for inode_lock */
 
 #include <asm/atomic.h>
 
fs/notify/vfsmount_mark.c +0 -1
index 56772b5..6f8eefe 100644
@@ -23,7 +23,6 @@
 #include <linux/mount.h>
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
-#include <linux/writeback.h> /* for inode_lock */
 
 #include <asm/atomic.h>
 
fs/ntfs/inode.c +2 -2
index 93622b1..7c530f3 100644
@@ -54,7 +54,7 @@
  *
  * Return 1 if the attributes match and 0 if not.
  *
- * NOTE: This function runs with the inode_lock spin lock held so it is not
+ * NOTE: This function runs with the i_lock spin lock held so it is not
  * allowed to sleep.
  */
 int ntfs_test_inode(struct inode *vi, ntfs_attr *na)
@@ -98,7 +98,7 @@ int ntfs_test_inode(struct inode *vi, ntfs_attr *na)
  *
  * Return 0 on success and -errno on error.
  *
- * NOTE: This function runs with the inode_lock spin lock held so it is not
+ * NOTE: This function runs with the i_lock spin lock held so it is not
  * allowed to sleep. (Hence the GFP_ATOMIC allocation.)
  */
 static int ntfs_init_locked_inode(struct inode *vi, ntfs_attr *na)
fs/ocfs2/inode.c +1 -1
index eece3e0..65c61e2 100644
@@ -1195,7 +1195,7 @@ void ocfs2_evict_inode(struct inode *inode)
 	ocfs2_clear_inode(inode);
 }
 
-/* Called under inode_lock, with no more references on the
+/* Called under i_lock, with no more references on the
  * struct inode, so it's safe here to check the flags field
  * and to manipulate i_nlink without any other locks. */
 int ocfs2_drop_inode(struct inode *inode)
fs/quota/dquot.c +5 -11
index 15f66f1..69bc754 100644
@@ -76,7 +76,7 @@
 #include <linux/buffer_head.h>
 #include <linux/capability.h>
 #include <linux/quotaops.h>
-#include <linux/writeback.h> /* for inode_lock, oddly enough.. */
+#include <linux/writeback.h>
 
 #include <asm/uaccess.h>
 
@@ -896,7 +896,6 @@ static void add_dquot_ref(struct super_block *sb, int type)
 	int reserved = 0;
 #endif
 
-	spin_lock(&inode_lock);
 	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		spin_lock(&inode->i_lock);
@@ -914,21 +913,18 @@ static void add_dquot_ref(struct super_block *sb, int type)
 		__iget(inode);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
-		spin_unlock(&inode_lock);
 
 		iput(old_inode);
 		__dquot_initialize(inode, type);
 		/* We hold a reference to 'inode' so it couldn't have been
-		 * removed from s_inodes list while we dropped the inode_lock.
-		 * We cannot iput the inode now as we can be holding the last
-		 * reference and we cannot iput it under inode_lock. So we
-		 * keep the reference and iput it later. */
+		 * removed from s_inodes list while we dropped the
+		 * sb_inode_list_lock.  We cannot iput the inode now as we can
+		 * be holding the last reference and we cannot iput it under
+		 * lock. So we keep the reference and iput it later. */
 		old_inode = inode;
-		spin_lock(&inode_lock);
 		spin_lock(&sb_inode_list_lock);
 	}
 	spin_unlock(&sb_inode_list_lock);
-	spin_unlock(&inode_lock);
 	iput(old_inode);
 
 #ifdef CONFIG_QUOTA_DEBUG
@@ -1009,7 +1005,6 @@ static void remove_dquot_ref(struct super_block *sb, int type,
 	struct inode *inode;
 	int reserved = 0;
 
-	spin_lock(&inode_lock);
 	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		/*
@@ -1025,7 +1020,6 @@ static void remove_dquot_ref(struct super_block *sb, int type,
 		}
 	}
 	spin_unlock(&sb_inode_list_lock);
-	spin_unlock(&inode_lock);
 #ifdef CONFIG_QUOTA_DEBUG
 	if (reserved) {
 		printk(KERN_WARNING "VFS (%s): Writes happened after quota"
include/linux/fs.h +1 -1
index d60f256..46a51b9 100644
@@ -1579,7 +1579,7 @@ struct super_operations {
 };
 
 /*
- * Inode state bits.  Protected by inode_lock.
+ * Inode state bits.  Protected by i_lock.
  *
  * Three bits determine the dirty state of the inode, I_DIRTY_SYNC,
  * I_DIRTY_DATASYNC and I_DIRTY_PAGES.
include/linux/writeback.h +0 -1
index 8b9c24f..d266f0d 100644
@@ -9,7 +9,6 @@
 
 struct backing_dev_info;
 
-extern spinlock_t inode_lock;
 extern spinlock_t sb_inode_list_lock;
 extern spinlock_t wb_inode_list_lock;
 extern spinlock_t inode_hash_lock;
mm/backing-dev.c +0 -4
index b1e2987..c874f7c 100644
@@ -72,7 +72,6 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
 	struct inode *inode;
 
 	nr_wb = nr_dirty = nr_io = nr_more_io = 0;
-	spin_lock(&inode_lock);
 	spin_lock(&wb_inode_list_lock);
 	list_for_each_entry(inode, &wb->b_dirty, i_list)
 		nr_dirty++;
@@ -81,7 +80,6 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
 	list_for_each_entry(inode, &wb->b_more_io, i_list)
 		nr_more_io++;
 	spin_unlock(&wb_inode_list_lock);
-	spin_unlock(&inode_lock);
 
 	global_dirty_limits(&background_thresh, &dirty_thresh);
 	bdi_thresh = bdi_dirty_limit(bdi, dirty_thresh);
@@ -682,13 +680,11 @@ void bdi_destroy(struct backing_dev_info *bdi)
 	if (bdi_has_dirty_io(bdi)) {
 		struct bdi_writeback *dst = &default_backing_dev_info.wb;
 
-		spin_lock(&inode_lock);
 		spin_lock(&wb_inode_list_lock);
 		list_splice(&bdi->wb.b_dirty, &dst->b_dirty);
 		list_splice(&bdi->wb.b_io, &dst->b_io);
 		list_splice(&bdi->wb.b_more_io, &dst->b_more_io);
 		spin_unlock(&wb_inode_list_lock);
-		spin_unlock(&inode_lock);
 	}
 
 	bdi_unregister(bdi);
mm/filemap.c +3 -3
index 3d4df44..ece6ef2 100644
@@ -80,7 +80,7 @@
  *  ->i_mutex
  *    ->i_alloc_sem             (various)
  *
- *  ->inode_lock
+ *  ->i_lock
  *    ->sb_lock			(fs/fs-writeback.c)
  *    ->mapping->tree_lock	(__sync_single_inode)
  *
@@ -98,8 +98,8 @@
  *    ->zone.lru_lock		(check_pte_range->isolate_lru_page)
  *    ->private_lock		(page_remove_rmap->set_page_dirty)
  *    ->tree_lock		(page_remove_rmap->set_page_dirty)
- *    ->inode_lock		(page_remove_rmap->set_page_dirty)
- *    ->inode_lock		(zap_pte_range->set_page_dirty)
+ *    ->i_lock			(page_remove_rmap->set_page_dirty)
+ *    ->i_lock			(zap_pte_range->set_page_dirty)
  *    ->private_lock		(zap_pte_range->__set_page_dirty_buffers)
  *
  *  ->task->proc_lock
mm/rmap.c +3 -3
index f6f0d2d..9aa1d3e 100644
@@ -31,11 +31,11 @@
  *             swap_lock (in swap_duplicate, swap_info_get)
  *               mmlist_lock (in mmput, drain_mmlist and others)
  *               mapping->private_lock (in __set_page_dirty_buffers)
- *               inode_lock (in set_page_dirty's __mark_inode_dirty)
- *                 sb_lock (within inode_lock in fs/fs-writeback.c)
+ *               i_lock (in set_page_dirty's __mark_inode_dirty)
+ *                 sb_lock (within i_lock in fs/fs-writeback.c)
  *                 mapping->tree_lock (widely used, in set_page_dirty,
  *                           in arch-dependent flush_dcache_mmap_lock,
- *                           within inode_lock in __sync_single_inode)
+ *                           within i_lock in __sync_single_inode)
  *
  * (code doesn't rely on that order so it could be switched around)
  * ->tasklist_lock
Signature · 1 line
1.7.1
#46 lore raw

PATCH 11/17 fs: Factor inode hash operations into functions

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the inode_hash_lock. Make it private by adding a function __remove_inode_hash that can be called by filesystems instead of open-coding their own inode hash removal operations.

---
 fs/inode.c                |   39 ++++++++++++++++++++++++---------------
 include/linux/fs.h        |    1 +
 include/linux/writeback.h |    1 -
 3 files changed, 25 insertions(+), 16 deletions(-)
fs/inode.c +24 -15
index 153f8d2..24141cc 100644
@@ -105,7 +105,7 @@ static struct hlist_head *inode_hashtable __read_mostly;
  */
 DEFINE_SPINLOCK(sb_inode_list_lock);
 DEFINE_SPINLOCK(wb_inode_list_lock);
-DEFINE_SPINLOCK(inode_hash_lock);
+static DEFINE_SPINLOCK(inode_hash_lock);
 
 /*
  * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -371,9 +371,7 @@ static void dispose_list(struct list_head *head)
 
 		spin_lock(&sb_inode_list_lock);
 		spin_lock(&inode->i_lock);
-		spin_lock(&inode_hash_lock);
-		hlist_del_init(&inode->i_hash);
-		spin_unlock(&inode_hash_lock);
+		__remove_inode_hash(inode);
 		list_del_init(&inode->i_sb_list);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
@@ -1274,6 +1272,20 @@ void __insert_inode_hash(struct inode *inode, unsigned long hashval)
 EXPORT_SYMBOL(__insert_inode_hash);
 
 /**
+ *	__remove_inode_hash - remove an inode from the hash
+ *	@inode: inode to unhash
+ *
+ *	Remove an inode from the superblock. inode->i_lock must be
+ *	held.
+ */
+void __remove_inode_hash(struct inode *inode)
+{
+	spin_lock(&inode_hash_lock);
+	hlist_del_init(&inode->i_hash);
+	spin_unlock(&inode_hash_lock);
+}
+
+/**
  *	remove_inode_hash - remove an inode from the hash
  *	@inode: inode to unhash
  *
@@ -1282,9 +1294,7 @@ EXPORT_SYMBOL(__insert_inode_hash);
 void remove_inode_hash(struct inode *inode)
 {
 	spin_lock(&inode->i_lock);
-	spin_lock(&inode_hash_lock);
-	hlist_del_init(&inode->i_hash);
-	spin_unlock(&inode_hash_lock);
+	__remove_inode_hash(inode);
 	spin_unlock(&inode->i_lock);
 }
 EXPORT_SYMBOL(remove_inode_hash);
@@ -1348,9 +1358,7 @@ static void iput_final(struct inode *inode)
 		spin_lock(&inode->i_lock);
 		WARN_ON(inode->i_state & I_NEW);
 		inode->i_state &= ~I_WILL_FREE;
-		spin_lock(&inode_hash_lock);
-		hlist_del_init(&inode->i_hash);
-		spin_unlock(&inode_hash_lock);
+		__remove_inode_hash(inode);
 		atomic_dec(&inodes_stat.nr_unused);
 	}
 	spin_lock(&wb_inode_list_lock);
@@ -1363,11 +1371,12 @@ static void iput_final(struct inode *inode)
 	spin_unlock(&inode->i_lock);
 	atomic_dec(&inodes_stat.nr_inodes);
 	evict(inode);
-	spin_lock(&inode->i_lock);
-	spin_lock(&inode_hash_lock);
-	hlist_del_init(&inode->i_hash);
-	spin_unlock(&inode_hash_lock);
-	spin_unlock(&inode->i_lock);
+
+	/*
+	 * i_lock is required to delete from hash because find_inode_fast
+	 * might find us but go to sleep before we run wake_up_inode.
+	 */
+	remove_inode_hash(inode);
 	wake_up_inode(inode);
 	BUG_ON(inode->i_state != (I_FREEING | I_CLEAR));
 	destroy_inode(inode);
include/linux/fs.h +1 -0
index 46a51b9..da0ebf1 100644
@@ -2186,6 +2186,7 @@ extern int should_remove_suid(struct dentry *);
 extern int file_remove_suid(struct file *);
 
 extern void __insert_inode_hash(struct inode *, unsigned long hashval);
+extern void __remove_inode_hash(struct inode *);
 extern void remove_inode_hash(struct inode *);
 static inline void insert_inode_hash(struct inode *inode) {
 	__insert_inode_hash(inode, inode->i_ino);
include/linux/writeback.h +0 -1
index d266f0d..0f6fe0c 100644
@@ -11,7 +11,6 @@ struct backing_dev_info;
 
 extern spinlock_t sb_inode_list_lock;
 extern spinlock_t wb_inode_list_lock;
-extern spinlock_t inode_hash_lock;
 extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
Signature · 1 line
1.7.1
#47 lore raw

Re: [PATCH 11/17] fs: Factor inode hash operations into functions

Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the inode_hash_lock. Make it private by adding a function __remove_inode_hash that can be called by filesystems instead of open-coding their own inode hash removal operations.

I like the factoring, but this changelog is misleading. At least in this series no new user of __remove_inode_hash appears, and I'm not sure where it would appear anyway. Just making the function global without actually exporting it is not helping external filesystems anyway. For now it can simply be made static.

#48 lore raw

Re: [PATCH 11/17] fs: Factor inode hash operations into functions

Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner

From: Nick Piggin <[email protected]>

Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the inode_hash_lock. Make it private by adding a function __remove_inode_hash that can be called by filesystems instead of open-coding their own inode hash removal operations.

Quoted 5 lines

I like the factoring, but this changelog is misleading. At least in this series no new user of __remove_inode_hash appears, and I'm not sure where it would appear anyway. Just making the function global without actually exporting it is not helping external filesystems anyway. For now it can simply be made static.

Yeah, hugetlbfs was using this a while back as I said, and I've missed refactoring it. Will do.

#49 lore raw

PATCH 12/17 fs: Introduce per-bucket inode hash locks

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Now the inode_hash_lock is private, we can change the hash locking to be more scalable. Convert the inode hash to use the new bit-locked hash list implementation that allows per-bucket locks to be used. This allows us to replace the global inode_hash_lock with finer grained locking without increasing the size of the hash table.

---
 fs/btrfs/inode.c        |    2 +-
 fs/fs-writeback.c       |    2 +-
 fs/hfs/hfs_fs.h         |    2 +-
 fs/hfs/inode.c          |    2 +-
 fs/hfsplus/hfsplus_fs.h |    2 +-
 fs/hfsplus/inode.c      |    2 +-
 fs/inode.c              |  198 ++++++++++++++++++++++++++---------------------
 fs/nilfs2/gcinode.c     |   22 +++---
 fs/nilfs2/segment.c     |    2 +-
 fs/nilfs2/the_nilfs.h   |    2 +-
 fs/reiserfs/xattr.c     |    2 +-
 include/linux/fs.h      |    3 +-
 mm/shmem.c              |    4 +-
 13 files changed, 135 insertions(+), 110 deletions(-)
fs/btrfs/inode.c +1 -1
index ffb8aec..7675f0c 100644
@@ -3860,7 +3860,7 @@ again:
 	p = &root->inode_tree.rb_node;
 	parent = NULL;
 
-	if (hlist_unhashed(&inode->i_hash))
+	if (hlist_bl_unhashed(&inode->i_hash))
 		return;
 
 	spin_lock(&root->inode_lock);
fs/fs-writeback.c +1 -1
index dc983ea..9b63fc7 100644
@@ -990,7 +990,7 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 		 * dirty list.  Add blockdev inodes as well.
 		 */
 		if (!S_ISBLK(inode->i_mode)) {
-			if (hlist_unhashed(&inode->i_hash))
+			if (hlist_bl_unhashed(&inode->i_hash))
 				goto out;
 		}
 		if (inode->i_state & I_FREEING)
fs/hfs/hfs_fs.h +1 -1
index 4f55651..24591be 100644
@@ -148,7 +148,7 @@ struct hfs_sb_info {
 
 	int fs_div;
 
-	struct hlist_head rsrc_inodes;
+	struct hlist_bl_head rsrc_inodes;
 };
 
 #define HFS_FLG_BITMAP_DIRTY	0
fs/hfs/inode.c +1 -1
index 397b7ad..7778298 100644
@@ -524,7 +524,7 @@ static struct dentry *hfs_file_lookup(struct inode *dir, struct dentry *dentry,
 	HFS_I(inode)->rsrc_inode = dir;
 	HFS_I(dir)->rsrc_inode = inode;
 	igrab(dir);
-	hlist_add_head(&inode->i_hash, &HFS_SB(dir->i_sb)->rsrc_inodes);
+	hlist_bl_add_head(&inode->i_hash, &HFS_SB(dir->i_sb)->rsrc_inodes);
 	mark_inode_dirty(inode);
 out:
 	d_add(dentry, inode);
fs/hfsplus/hfsplus_fs.h +1 -1
index dc856be..499f5a5 100644
@@ -144,7 +144,7 @@ struct hfsplus_sb_info {
 
 	unsigned long flags;
 
-	struct hlist_head rsrc_inodes;
+	struct hlist_bl_head rsrc_inodes;
 };
 
 #define HFSPLUS_SB_WRITEBACKUP	0x0001
fs/hfsplus/inode.c +1 -1
index c5a979d..b755cf0 100644
@@ -202,7 +202,7 @@ static struct dentry *hfsplus_file_lookup(struct inode *dir, struct dentry *dent
 	HFSPLUS_I(inode).rsrc_inode = dir;
 	HFSPLUS_I(dir).rsrc_inode = inode;
 	igrab(dir);
-	hlist_add_head(&inode->i_hash, &HFSPLUS_SB(sb).rsrc_inodes);
+	hlist_bl_add_head(&inode->i_hash, &HFSPLUS_SB(sb).rsrc_inodes);
 	mark_inode_dirty(inode);
 out:
 	d_add(dentry, inode);
fs/inode.c +110 -88

531 lines not on this page · raw message

fs/nilfs2/gcinode.c +12 -10
index bed3a78..ce7344e 100644
@@ -196,13 +196,13 @@ int nilfs_init_gccache(struct the_nilfs *nilfs)
 	INIT_LIST_HEAD(&nilfs->ns_gc_inodes);
 
 	nilfs->ns_gc_inodes_h =
-		kmalloc(sizeof(struct hlist_head) * NILFS_GCINODE_HASH_SIZE,
+		kmalloc(sizeof(struct hlist_bl_head) * NILFS_GCINODE_HASH_SIZE,
 			GFP_NOFS);
 	if (nilfs->ns_gc_inodes_h == NULL)
 		return -ENOMEM;
 
 	for (loop = 0; loop < NILFS_GCINODE_HASH_SIZE; loop++)
-		INIT_HLIST_HEAD(&nilfs->ns_gc_inodes_h[loop]);
+		INIT_HLIST_BL_HEAD(&nilfs->ns_gc_inodes_h[loop]);
 	return 0;
 }
 
@@ -254,18 +254,18 @@ static unsigned long ihash(ino_t ino, __u64 cno)
  */
 struct inode *nilfs_gc_iget(struct the_nilfs *nilfs, ino_t ino, __u64 cno)
 {
-	struct hlist_head *head = nilfs->ns_gc_inodes_h + ihash(ino, cno);
-	struct hlist_node *node;
+	struct hlist_bl_head *head = nilfs->ns_gc_inodes_h + ihash(ino, cno);
+	struct hlist_bl_node *node;
 	struct inode *inode;
 
-	hlist_for_each_entry(inode, node, head, i_hash) {
+	hlist_bl_for_each_entry(inode, node, head, i_hash) {
 		if (inode->i_ino == ino && NILFS_I(inode)->i_cno == cno)
 			return inode;
 	}
 
 	inode = alloc_gcinode(nilfs, ino, cno);
 	if (likely(inode)) {
-		hlist_add_head(&inode->i_hash, head);
+		hlist_bl_add_head(&inode->i_hash, head);
 		list_add(&NILFS_I(inode)->i_dirty, &nilfs->ns_gc_inodes);
 	}
 	return inode;
@@ -284,16 +284,18 @@ void nilfs_clear_gcinode(struct inode *inode)
  */
 void nilfs_remove_all_gcinode(struct the_nilfs *nilfs)
 {
-	struct hlist_head *head = nilfs->ns_gc_inodes_h;
-	struct hlist_node *node, *n;
+	struct hlist_bl_head *head = nilfs->ns_gc_inodes_h;
+	struct hlist_bl_node *node;
 	struct inode *inode;
 	int loop;
 
 	for (loop = 0; loop < NILFS_GCINODE_HASH_SIZE; loop++, head++) {
-		hlist_for_each_entry_safe(inode, node, n, head, i_hash) {
-			hlist_del_init(&inode->i_hash);
+restart:
+		hlist_bl_for_each_entry(inode, node, head, i_hash) {
+			hlist_bl_del_init(&inode->i_hash);
 			list_del_init(&NILFS_I(inode)->i_dirty);
 			nilfs_clear_gcinode(inode); /* might sleep */
+			goto restart;
 		}
 	}
 }
fs/nilfs2/segment.c +1 -1
index 9fd051a..038251c 100644
@@ -2452,7 +2452,7 @@ nilfs_remove_written_gcinodes(struct the_nilfs *nilfs, struct list_head *head)
 	list_for_each_entry_safe(ii, n, head, i_dirty) {
 		if (!test_bit(NILFS_I_UPDATED, &ii->i_state))
 			continue;
-		hlist_del_init(&ii->vfs_inode.i_hash);
+		hlist_bl_del_init(&ii->vfs_inode.i_hash);
 		list_del_init(&ii->i_dirty);
 		nilfs_clear_gcinode(&ii->vfs_inode);
 	}
fs/nilfs2/the_nilfs.h +1 -1
index f785a7b..1ab441a 100644
@@ -167,7 +167,7 @@ struct the_nilfs {
 
 	/* GC inode list and hash table head */
 	struct list_head	ns_gc_inodes;
-	struct hlist_head      *ns_gc_inodes_h;
+	struct hlist_bl_head      *ns_gc_inodes_h;
 
 	/* Disk layout information (static) */
 	unsigned int		ns_blocksize_bits;
fs/reiserfs/xattr.c +1 -1
index 8c4cf27..ea2f55c 100644
@@ -424,7 +424,7 @@ int reiserfs_prepare_write(struct file *f, struct page *page,
 static void update_ctime(struct inode *inode)
 {
 	struct timespec now = current_fs_time(inode->i_sb);
-	if (hlist_unhashed(&inode->i_hash) || !inode->i_nlink ||
+	if (hlist_bl_unhashed(&inode->i_hash) || !inode->i_nlink ||
 	    timespec_equal(&inode->i_ctime, &now))
 		return;
 
include/linux/fs.h +2 -1
index da0ebf1..f06be07 100644
@@ -376,6 +376,7 @@ struct files_stat_struct {
 #include <linux/capability.h>
 #include <linux/semaphore.h>
 #include <linux/fiemap.h>
+#include <linux/rculist_bl.h>
 
 #include <asm/atomic.h>
 #include <asm/byteorder.h>
@@ -722,7 +723,7 @@ struct posix_acl;
 #define ACL_NOT_CACHED ((void *)(-1))
 
 struct inode {
-	struct hlist_node	i_hash;
+	struct hlist_bl_node	i_hash;
 	struct list_head	i_list;		/* backing dev IO list */
 	struct list_head	i_sb_list;
 	struct list_head	i_dentry;
mm/shmem.c +2 -2
index 56229bb..b83b442 100644
@@ -2148,7 +2148,7 @@ static int shmem_encode_fh(struct dentry *dentry, __u32 *fh, int *len,
 	if (*len < 3)
 		return 255;
 
-	if (hlist_unhashed(&inode->i_hash)) {
+	if (hlist_bl_unhashed(&inode->i_hash)) {
 		/* Unfortunately insert_inode_hash is not idempotent,
 		 * so as we hash inodes here rather than at creation
 		 * time, we need a lock to ensure we only try
@@ -2156,7 +2156,7 @@ static int shmem_encode_fh(struct dentry *dentry, __u32 *fh, int *len,
 		 */
 		static DEFINE_SPINLOCK(lock);
 		spin_lock(&lock);
-		if (hlist_unhashed(&inode->i_hash))
+		if (hlist_bl_unhashed(&inode->i_hash))
 			__insert_inode_hash(inode,
 					    inode->i_ino + inode->i_generation);
 		spin_unlock(&lock);
Signature · 1 line
1.7.1
#50 lore raw

Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks

Instead of doing the lock overkill on a still fundamentally global data structure what about replacing this with something better. I know you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.

then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?

If it was up to me I'd dtop this (and the bl_list addition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.

#51 lore raw

Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks

Quoted 6 lines · Christoph Hellwig

Instead of doing the lock overkill on a still fundamentally global data structure what about replacing this with something better. I know you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.

The XFS inode cache design is tied tightly to the inode layout in XFS, so the tree-per-ag-per-mount parallelism design really does not work in a generic manner. Sure, we could probably make it a hashed-tree rather than hashed-link-list design, but that's a much more fundamental change than just splitting the locks up.

Quoted 2 lines

then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?

I've been running comparitive benchmarks on ext4 as well so that I also test all the generic paths.

Quoted 3 lines

If it was up to me I'd dtop this (and the bl_list addition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.

I think that it's going to take a lot of work to come up with something more generically optimal, so in the mean time I think this is a net win for filesystems that use the generic icache.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#52 lore raw

Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks

In reply to Christoph Hellwig
Quoted 2 lines · Christoph Hellwig

Instead of doing the lock overkill on a still fundamentally global data

How do you figure it is overkill? Actually the hash insertion/removal scales *really* well with per-bucket locks and it is a technique used and proven in other parts of the kernel like networking.

Having a global lock there is certainly a huge bottleneck when you start increasing system size, so I don't know why you keep arguing against this.

Quoted 1 line

structure what about replacing this with something better.

I won't be doing this until after the scalability work.

Quoted 10 lines

you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.

then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?

If it was up to me I'd dtop this (and the bl_list addition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.

I do care about scalability of icache code. Given how simple this is, and seeing as we're about to have the big locking rework, I much prefer just fixing all the global locks now (which need to be fixed anyway).

#53 lore raw

PATCH 13/17 fs: Implement lazy LRU updates for inodes.

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Convert the inode LRU to use lazy updates to reduce lock and cacheline traffic. We avoid moving inodes around in the LRU list during iget/iput operations so these frequent operations don't need to access the LRUs. Instead, we defer the refcount checks to reclaim-time and use a per-inode state flag, I_REFERENCED, to tell reclaim that iget has touched the inode in the past. This means that only reclaim should be touching the LRU with any frequency, hence significantly reducing lock acquisitions and the amount contention on LRU updates.

---
 fs/fs-writeback.c         |   20 +++--------
 fs/inode.c                |   81 +++++++++++++--------------------------------
 include/linux/fs.h        |   20 +++++++----
 include/linux/writeback.h |    1 -
 4 files changed, 42 insertions(+), 80 deletions(-)
fs/fs-writeback.c +6 -14
index 9b63fc7..432a4df 100644
@@ -408,15 +408,8 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
 			 * completion.
 			 */
 			redirty_tail(inode);
-		} else if (inode->i_count) {
-			/*
-			 * The inode is clean, inuse
-			 */
-			list_move(&inode->i_list, &inode_in_use);
 		} else {
-			/*
-			 * The inode is clean, unused
-			 */
+			/* The inode is clean */
 			list_move(&inode->i_list, &inode_unused);
 		}
 	}
@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)
 	 */
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
-	spin_lock(&sb_inode_list_lock);
-
 	/*
 	 * Data integrity sync. Must wait for all pages under writeback,
 	 * because there may have been pages dirtied before our sync
@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)
 	 * In which case, the inode may not be on the dirty list, but
 	 * we still have to wait for that writeout.
 	 */
+	spin_lock(&sb_inode_list_lock);
 	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
 		struct address_space *mapping;
 
@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)
 		spin_unlock(&sb_inode_list_lock);
 		/*
 		 * We hold a reference to 'inode' so it couldn't have been
-		 * removed from s_inodes list while we dropped the
-		 * sb_inode_list_lock.  We cannot iput the inode now as we can
-		 * be holding the last reference and we cannot iput it under
-		 * spinlock. So we keep the reference and iput it later.
+		 * removed from s_inodes list while we dropped the i_lock.  We
+		 * cannot iput the inode now as we can be holding the last
+		 * reference and we cannot iput it under spinlock. So we keep
+		 * the reference and iput it later.
 		 */
 		iput(old_inode);
 		old_inode = inode;
fs/inode.c +23 -58

164 lines not on this page · raw message

include/linux/fs.h +13 -7
index f06be07..096a5eb 100644
@@ -1627,16 +1627,17 @@ struct super_operations {
  *
  * Q: What is the difference between I_WILL_FREE and I_FREEING?
  */
-#define I_DIRTY_SYNC		1
-#define I_DIRTY_DATASYNC	2
-#define I_DIRTY_PAGES		4
+#define I_DIRTY_SYNC		0x01
+#define I_DIRTY_DATASYNC	0x02
+#define I_DIRTY_PAGES		0x04
 #define __I_NEW			3
 #define I_NEW			(1 << __I_NEW)
-#define I_WILL_FREE		16
-#define I_FREEING		32
-#define I_CLEAR			64
+#define I_WILL_FREE		0x10
+#define I_FREEING		0x20
+#define I_CLEAR			0x40
 #define __I_SYNC		7
 #define I_SYNC			(1 << __I_SYNC)
+#define I_REFERENCED		0x100
 
 #define I_DIRTY (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES)
 
@@ -2177,7 +2178,6 @@ extern int insert_inode_locked4(struct inode *, unsigned long, int (*test)(struc
 extern int insert_inode_locked(struct inode *);
 extern void unlock_new_inode(struct inode *);
 
-extern void __iget(struct inode * inode);
 extern void iget_failed(struct inode *);
 extern void end_writeback(struct inode *);
 extern void destroy_inode(struct inode *);
@@ -2392,6 +2392,12 @@ extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt);
 extern void save_mount_options(struct super_block *sb, char *options);
 extern void replace_mount_options(struct super_block *sb, char *options);
 
+static inline void __iget(struct inode *inode)
+{
+	assert_spin_locked(&inode->i_lock);
+	inode->i_count++;
+}
+
 static inline ino_t parent_ino(struct dentry *dentry)
 {
 	ino_t res;
include/linux/writeback.h +0 -1
index 0f6fe0c..cde6993 100644
@@ -11,7 +11,6 @@ struct backing_dev_info;
 
 extern spinlock_t sb_inode_list_lock;
 extern spinlock_t wb_inode_list_lock;
-extern struct list_head inode_in_use;
 extern struct list_head inode_unused;
 
 /*
Signature · 1 line
1.7.1
#54 lore raw

Re: [PATCH 13/17] fs: Implement lazy LRU updates for inodes.

Quoted 10 lines
fs/fs-writeback.c +0 -2
@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)
 	 */
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
-	spin_lock(&sb_inode_list_lock);
-
 	/*
 	 * Data integrity sync. Must wait for all pages under writeback,
 	 * because there may have been pages dirtied before our sync

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1061

Quoted 4 lines
fs/fs-writeback.c +1 -0
@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)
 	 * In which case, the inode may not be on the dirty list, but
 	 * we still have to wait for that writeout.
 	 */
+	spin_lock(&sb_inode_list_lock);

I think this should be folded back into the patch introducing sb_inode_list_lock.

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1081

Quoted 12 lines
fs/fs-writeback.c +4 -4
@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)
 		spin_unlock(&sb_inode_list_lock);
 		/*
 		 * We hold a reference to 'inode' so it couldn't have been
-		 * removed from s_inodes list while we dropped the
-		 * sb_inode_list_lock.  We cannot iput the inode now as we can
-		 * be holding the last reference and we cannot iput it under
-		 * spinlock. So we keep the reference and iput it later.
+		 * removed from s_inodes list while we dropped the i_lock.  We
+		 * cannot iput the inode now as we can be holding the last
+		 * reference and we cannot iput it under spinlock. So we keep
+		 * the reference and iput it later.

This also looks like a hunk that got in by accident and should be merged into an earlier patch.

fs/inode.c static int invalidate_list(struct list_head *head, struct list_head *dispose) · line 420

Quoted 11 lines
fs/inode.c +2 -1
@@ -431,11 +412,12 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 		invalidate_inode_buffers(inode);
 		if (!inode->i_count) {
 			spin_lock(&wb_inode_list_lock);
-			list_move(&inode->i_list, dispose);
+			list_del(&inode->i_list);
 			spin_unlock(&wb_inode_list_lock);
 			WARN_ON(inode->i_state & I_NEW);
 			inode->i_state |= I_FREEING;
 			spin_unlock(&inode->i_lock);
+			list_add(&inode->i_list, dispose);

Moving the list_add out of the lock looks fine, but I can't really see how it's related to the rest of the patch.

fs/inode.c again: · line 498

Quoted 7 lines
fs/inode.c +7 -0
+		if (inode->i_count || (inode->i_state & ~I_REFERENCED)) {
+			list_del_init(&inode->i_list);
+			spin_unlock(&inode->i_lock);
+			atomic_dec(&inodes_stat.nr_unused);
+			continue;
+		}
+		if (inode->i_state) {

Slightly confusing but okay given the only i_state that will get us here is I_REFERENCED. Do we really care about the additional cycle or two a dumb compiler might generate when writing

	if (inode->i_state & I_REFERENCED)

?

fs/inode.c again: · line 505

Quoted 2 lines
fs/inode.c +1 -0
 		if (inode_has_buffers(inode) || inode->i_data.nrpages) {
+			list_move(&inode->i_list, &inode_unused);

Why are we now moving the inode to the front of the list?

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 657

Quoted 10 lines
fs/inode.c +0 -3
@@ -687,9 +652,6 @@ __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,
 	atomic_inc(&inodes_stat.nr_inodes);
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
-	spin_lock(&wb_inode_list_lock);
-	list_add(&inode->i_list, &inode_in_use);
-	spin_unlock(&wb_inode_list_lock);
 	if (b) {
 		spin_lock_bucket(b);
 		hlist_bl_add_head(&inode->i_hash, &b->head);

At some point it might be worth to split this into

inode_add_to_sb_list

and

	__inode_add_to_hash

but that can be left for later.

#55 lore raw

Re: [PATCH 13/17] fs: Implement lazy LRU updates for inodes.

Quoted 18 lines · Christoph Hellwig
Quoted 10 lines
fs/fs-writeback.c +0 -2
@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)
 	 */
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
-	spin_lock(&sb_inode_list_lock);
-
 	/*
 	 * Data integrity sync. Must wait for all pages under writeback,
 	 * because there may have been pages dirtied before our sync

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1061

Quoted 4 lines
fs/fs-writeback.c +1 -0
@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)
 	 * In which case, the inode may not be on the dirty list, but
 	 * we still have to wait for that writeout.
 	 */
+	spin_lock(&sb_inode_list_lock);

I think this should be folded back into the patch introducing sb_inode_list_lock.

Quoted 15 lines

fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1081

Quoted 12 lines
fs/fs-writeback.c +4 -4
@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)
 		spin_unlock(&sb_inode_list_lock);
 		/*
 		 * We hold a reference to 'inode' so it couldn't have been
-		 * removed from s_inodes list while we dropped the
-		 * sb_inode_list_lock.  We cannot iput the inode now as we can
-		 * be holding the last reference and we cannot iput it under
-		 * spinlock. So we keep the reference and iput it later.
+		 * removed from s_inodes list while we dropped the i_lock.  We
+		 * cannot iput the inode now as we can be holding the last
+		 * reference and we cannot iput it under spinlock. So we keep
+		 * the reference and iput it later.

This also looks like a hunk that got in by accident and should be merged into an earlier patch.

These two actually came from a patch to do rcu locking (which Dave has changed a bit, but originally due to my fault), so I'll fix those, thanks.

Quoted 6 lines
Quoted 6 lines
fs/inode.c +1 -1
@@ -431,11 +412,12 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 		invalidate_inode_buffers(inode);
 		if (!inode->i_count) {
 			spin_lock(&wb_inode_list_lock);
-			list_move(&inode->i_list, dispose);
+			list_del(&inode->i_list);
Quoted 8 lines

fs/inode.c static int invalidate_list(struct list_head *head, struct list_head *dispose) · line 420

Quoted 5 lines
fs/inode.c +1 -0
 			spin_unlock(&wb_inode_list_lock);
 			WARN_ON(inode->i_state & I_NEW);
 			inode->i_state |= I_FREEING;
 			spin_unlock(&inode->i_lock);
+			list_add(&inode->i_list, dispose);

Moving the list_add out of the lock looks fine, but I can't really see how it's related to the rest of the patch.

Just helps shows that dispose isn't being protected by wb_inode_list_lock, I guess.

Quoted 13 lines

fs/inode.c again: · line 498

Quoted 8 lines
fs/inode.c +7 -0
+		if (inode->i_count || (inode->i_state & ~I_REFERENCED)) {
+			list_del_init(&inode->i_list);
+			spin_unlock(&inode->i_lock);
+			atomic_dec(&inodes_stat.nr_unused);
+			continue;
+		}
+		if (inode->i_state) {

Slightly confusing but okay given the only i_state that will get us here is I_REFERENCED. Do we really care about the additional cycle or two a dumb compiler might generate when writing

Quoted 1 line
	if (inode->i_state & I_REFERENCED)

Sure, why not.

Quoted 7 lines

?

fs/inode.c again: · line 505

Quoted 2 lines
fs/inode.c +1 -0
 		if (inode_has_buffers(inode) || inode->i_data.nrpages) {
+			list_move(&inode->i_list, &inode_unused);

Why are we now moving the inode to the front of the list?

It was always being moved to the front of the list, but with lazy LRU, iput_final doesn't move it for us, hence the list_move here.

Without this, it busy-spins and locks badly under heavy reclaim load when buffers or pagecache can't be invalidated.

Seeing as it wasn't obvious to you, I'll add a comment here.

I was thinking we should probably have a shortcut to go back to the tail of the LRU in case of invalidation success, but that's out of the scope of this patch and I never got around to testing such a change yet.

#56 lore raw

PATCH 14/17 fs: Inode counters do not need to be atomic.

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

atomics for counters do not scale on large machines, so convert them back to normal variables protected by spin locks. We can do this because the counters are associated with specific list operations that are protected by locks; nr_inodes can be protected by the sb_inode_list_lock, and nr_unused can be protected by the wb_inode_list_lock.

---
 fs/fs-writeback.c  |    6 ++----
 fs/inode.c         |   30 ++++++++++++------------------
 include/linux/fs.h |   12 ++++++------
 3 files changed, 20 insertions(+), 28 deletions(-)
fs/fs-writeback.c +2 -4
index 432a4df..8e390e8 100644
@@ -743,8 +743,7 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			(atomic_read(&inodes_stat.nr_inodes) -
-			atomic_read(&inodes_stat.nr_unused));
+			inodes_stat.nr_inodes - inodes_stat.nr_unused;
 
 	if (nr_pages) {
 		struct wb_writeback_work work = {
@@ -1116,8 +1115,7 @@ void writeback_inodes_sb(struct super_block *sb)
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
 	work.nr_pages = nr_dirty + nr_unstable +
-			(atomic_read(&inodes_stat.nr_inodes) -
-			atomic_read(&inodes_stat.nr_unused));
+			inodes_stat.nr_inodes - inodes_stat.nr_unused;
 
 	bdi_queue_work(sb->s_bdi, &work);
 	wait_for_completion(&done);
fs/inode.c +12 -18
index 50599d7..d279517 100644
@@ -139,8 +139,8 @@ static DECLARE_RWSEM(iprune_sem);
  * Statistics gathering..
  */
 struct inodes_stat_t inodes_stat = {
-	.nr_inodes = ATOMIC_INIT(0),
-	.nr_unused = ATOMIC_INIT(0),
+	.nr_inodes = 0,
+	.nr_unused = 0,
 };
 
 static struct kmem_cache *inode_cachep __read_mostly;
@@ -376,7 +376,6 @@ static void dispose_list(struct list_head *head)
 		destroy_inode(inode);
 		nr_disposed++;
 	}
-	atomic_sub(nr_disposed, &inodes_stat.nr_inodes);
 }
 
 /*
@@ -385,7 +384,7 @@ static void dispose_list(struct list_head *head)
 static int invalidate_list(struct list_head *head, struct list_head *dispose)
 {
 	struct list_head *next;
-	int busy = 0, count = 0;
+	int busy = 0;
 
 	next = head->next;
 	for (;;) {
@@ -413,19 +412,17 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)
 		if (!inode->i_count) {
 			spin_lock(&wb_inode_list_lock);
 			list_del(&inode->i_list);
+			inodes_stat.nr_unused--;
 			spin_unlock(&wb_inode_list_lock);
 			WARN_ON(inode->i_state & I_NEW);
 			inode->i_state |= I_FREEING;
 			spin_unlock(&inode->i_lock);
 			list_add(&inode->i_list, dispose);
-			count++;
 			continue;
 		}
 		spin_unlock(&inode->i_lock);
 		busy = 1;
 	}
-	/* only unused inodes may be cached with i_count zero */
-	atomic_sub(count, &inodes_stat.nr_unused);
 	return busy;
 }
 
@@ -471,7 +468,6 @@ EXPORT_SYMBOL(invalidate_inodes);
 static void prune_icache(int nr_to_scan)
 {
 	LIST_HEAD(freeable);
-	int nr_pruned = 0;
 	unsigned long reap = 0;
 
 	down_read(&iprune_sem);
@@ -492,7 +488,7 @@ again:
 		if (inode->i_count || (inode->i_state & ~I_REFERENCED)) {
 			list_del_init(&inode->i_list);
 			spin_unlock(&inode->i_lock);
-			atomic_dec(&inodes_stat.nr_unused);
+			inodes_stat.nr_unused--;
 			continue;
 		}
 		if (inode->i_state) {
@@ -518,9 +514,8 @@ again:
 		WARN_ON(inode->i_state & I_NEW);
 		inode->i_state |= I_FREEING;
 		spin_unlock(&inode->i_lock);
-		nr_pruned++;
+		inodes_stat.nr_unused--;
 	}
-	atomic_sub(nr_pruned, &inodes_stat.nr_unused);
 	if (current_is_kswapd())
 		__count_vm_events(KSWAPD_INODESTEAL, reap);
 	else
@@ -552,8 +547,7 @@ static int shrink_icache_memory(struct shrinker *shrink, int nr, gfp_t gfp_mask)
 			return -1;
 		prune_icache(nr);
 	}
-	return (atomic_read(&inodes_stat.nr_unused) / 100) *
-					sysctl_vfs_cache_pressure;
+	return (inodes_stat.nr_unused / 100) * sysctl_vfs_cache_pressure;
 }
 
 static struct shrinker icache_shrinker = {
@@ -649,7 +643,7 @@ static inline void
 __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,
 			struct inode *inode)
 {
-	atomic_inc(&inodes_stat.nr_inodes);
+	inodes_stat.nr_inodes++;
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
 	if (b) {
@@ -1325,9 +1319,9 @@ static void iput_final(struct inode *inode)
 		if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
 			spin_lock(&wb_inode_list_lock);
 			list_move(&inode->i_list, &inode_unused);
+			inodes_stat.nr_unused++;
 			spin_unlock(&wb_inode_list_lock);
 		}
-		atomic_inc(&inodes_stat.nr_unused);
 		if (sb->s_flags & MS_ACTIVE) {
 			spin_unlock(&inode->i_lock);
 			spin_unlock(&sb_inode_list_lock);
@@ -1347,16 +1341,16 @@ static void iput_final(struct inode *inode)
 	if (!list_empty(&inode->i_list)) {
 		spin_lock(&wb_inode_list_lock);
 		list_del_init(&inode->i_list);
-		spin_unlock(&wb_inode_list_lock);
 		if (!inode->i_state)
-			atomic_dec(&inodes_stat.nr_unused);
+			inodes_stat.nr_unused--;
+		spin_unlock(&wb_inode_list_lock);
 	}
 	list_del_init(&inode->i_sb_list);
+	inodes_stat.nr_inodes--;
 	spin_unlock(&sb_inode_list_lock);
 	WARN_ON(inode->i_state & I_NEW);
 	inode->i_state |= I_FREEING;
 	spin_unlock(&inode->i_lock);
-	atomic_dec(&inodes_stat.nr_inodes);
 	evict(inode);
 
 	/*
include/linux/fs.h +6 -6
index 096a5eb..3a43313 100644
@@ -32,6 +32,12 @@
 #define SEEK_END	2	/* seek relative to end of file */
 #define SEEK_MAX	SEEK_END
 
+struct inodes_stat_t {
+	int nr_inodes;
+	int nr_unused;
+	int dummy[5];		/* padding for sysctl ABI compatibility */
+};
+
 /* And dynamically-tunable limits and defaults: */
 struct files_stat_struct {
 	int nr_files;		/* read only */
@@ -410,12 +416,6 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset,
 			ssize_t bytes, void *private, int ret,
 			bool is_async);
 
-struct inodes_stat_t {
-	atomic_t nr_inodes;
-	atomic_t nr_unused;
-	int dummy[5];		/* padding for sysctl ABI compatibility */
-};
-
 /*
  * Attribute flags.  These should be or-ed together to figure out what
  * has been changed!
Signature · 1 line
1.7.1
#57 lore raw

PATCH 15/17 fs: inode per-cpu last_ino allocator

In reply to Dave Chinner

From: Eric Dumazet <[email protected]>

last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

[npiggin: some extra commenting and use of defines]

---
 fs/inode.c |   50 +++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 43 insertions(+), 7 deletions(-)
fs/inode.c +43 -7
index d279517..1388450 100644
@@ -653,6 +653,48 @@ __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,
 	}
 }
 
+#ifdef CONFIG_SMP
+#define LAST_INO_BATCH 1024
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+static atomic_t shared_last_ino;
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int *p = &get_cpu_var(last_ino);
+	unsigned int res = *p;
+
+	if (unlikely((res & (LAST_INO_BATCH-1)) == 0))
+		res = (unsigned int)atomic_add_return(LAST_INO_BATCH,
+				&shared_last_ino) - LAST_INO_BATCH;
+
+	*p = ++res;
+	put_cpu_var(last_ino);
+	return res;
+}
+#else
+static unsigned int last_ino_get(void)
+{
+	static unsigned int last_ino;
+
+	return ++last_ino;
+}
+#endif
+
 /**
  * inode_add_to_lists - add a new inode to relevant lists
  * @sb: superblock inode belongs to
@@ -690,19 +732,13 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);
  */
 struct inode *new_inode(struct super_block *sb)
 {
-	/*
-	 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
-	 * error if st_ino won't fit in target struct field. Use 32bit counter
-	 * here to attempt to avoid that.
-	 */
-	static atomic_t last_ino = ATOMIC_INIT(0);
 	struct inode *inode;
 
 	inode = alloc_inode(sb);
 	if (inode) {
 		spin_lock(&sb_inode_list_lock);
 		spin_lock(&inode->i_lock);
-		inode->i_ino = (unsigned int)atomic_inc_return(&last_ino);
+		inode->i_ino = last_ino_get();
 		inode->i_state = 0;
 		__inode_add_to_lists(sb, NULL, inode);
 		spin_unlock(&inode->i_lock);
Signature · 1 line
1.7.1
#58 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 6 lines · Dave Chinner

From: Eric Dumazet <[email protected]>

last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.

And the good thing is most users of new_inode couldn't care less about the fake i_ino assigned because they have a real inode number. So the first step is to move the i_ino assignment into a separate helper and only use it in those filesystems that need it. Second step is to figure out why some filesystems need iunique() and some are fine with the incrementing counter and then we should find a scalable way to generate an inode number - preferably just one and not to, but if that's not possible we need some documentation on why which one is needed.

#59 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner

From: Eric Dumazet <[email protected]>

last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.

Quoted 9 lines

And the good thing is most users of new_inode couldn't care less about the fake i_ino assigned because they have a real inode number. So the first step is to move the i_ino assignment into a separate helper and only use it in those filesystems that need it. Second step is to figure out why some filesystems need iunique() and some are fine with the incrementing counter and then we should find a scalable way to generate an inode number - preferably just one and not to, but if that's not possible we need some documentation on why which one is needed.

Sounds like a good plan, but I don't really have time right now to understand the iget routines of every single filesystem to determine which rely on the current new_inode() allocated inode number. I think that is best left for a later cleanup, seeing as the last_ino scalability problem is easily addressed....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#60 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 5 lines · Dave Chinner

Sounds like a good plan, but I don't really have time right now to understand the iget routines of every single filesystem to determine which rely on the current new_inode() allocated inode number. I think that is best left for a later cleanup, seeing as the last_ino scalability problem is easily addressed...

It's fairly easy to do it pessimisticly - all disk based filesystem don't need it. Anyway, I can do this ontop of your series later. It just seems a bit counter-intuitive to scale something we don't actually need it most cases.

maintainer
#61 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

In reply to Dave Chinner
Quoted 35 lines · On Wed, 29 Sep 2010 22:18:47 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

[npiggin: some extra commenting and use of defines]

...

fs/inode.c +18 -0
+#ifdef CONFIG_SMP
+#define LAST_INO_BATCH 1024
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 683

Quoted 10 lines
fs/inode.c +10 -0
+static atomic_t shared_last_ino;
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int *p = &get_cpu_var(last_ino);
+	unsigned int res = *p;
+
+	if (unlikely((res & (LAST_INO_BATCH-1)) == 0))
+		res = (unsigned int)atomic_add_return(LAST_INO_BATCH,
+				&shared_last_ino) - LAST_INO_BATCH;

May as well remove the "- LAST_INO_BATCH" there, I think. It'll skew the results a tad at startup, but why does that matter?

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695

Quoted 11 lines
fs/inode.c +11 -0
+	*p = ++res;
+	put_cpu_var(last_ino);
+	return res;
+}
+#else
+static unsigned int last_ino_get(void)
+{
+	static unsigned int last_ino;
+
+	return ++last_ino;
+}

This is racy with CONFIG_PREEMPT on some architectures, I suspect. I'd suggest conversion to atomic_t with, of course, an explanatory comment ;)

Quoted 1 line

+#endif

#62 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 4 lines · Andrew Morton
Quoted 2 lines · On Wed, 29 Sep 2010 22:18:47 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

Please note my new email address, thanks.

Quoted 38 lines
Quoted 38 lines

last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

[npiggin: some extra commenting and use of defines]

...

fs/inode.c +23 -0
+#ifdef CONFIG_SMP
+#define LAST_INO_BATCH 1024
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+static atomic_t shared_last_ino;
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int *p = &get_cpu_var(last_ino);
Quoted 8 lines

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 683

Quoted 5 lines
fs/inode.c +5 -0
+	unsigned int res = *p;
+
+	if (unlikely((res & (LAST_INO_BATCH-1)) == 0))
+		res = (unsigned int)atomic_add_return(LAST_INO_BATCH,
+				&shared_last_ino) - LAST_INO_BATCH;

May as well remove the "- LAST_INO_BATCH" there, I think. It'll skew the results a tad at startup, but why does that matter?

Because on x86, atomic_add_return(val, ptr) uses xadd() + val

So, using "atomic_add_return(val, ptr) - val" removes one instruction ;)

Quoted 16 lines

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695

Quoted 12 lines
fs/inode.c +11 -0
+	*p = ++res;
+	put_cpu_var(last_ino);
+	return res;
+}
+#else
+static unsigned int last_ino_get(void)
+{
+	static unsigned int last_ino;
+
+	return ++last_ino;
+}

This is racy with CONFIG_PREEMPT on some architectures, I suspect. I'd suggest conversion to atomic_t with, of course, an explanatory comment ;)

Thanks, I'll rework the patch !

I am pretty happy to see some interest on this patch serie, eventually :)

#63 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Andrew Morton

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695

Quoted 6 lines
fs/inode.c +6 -0
+static unsigned int last_ino_get(void)
+{
+	static unsigned int last_ino;
+
+	return ++last_ino;
+}

This is racy with CONFIG_PREEMPT on some architectures, I suspect. I'd suggest conversion to atomic_t with, of course, an explanatory comment ;)

In fact this code was OK when I submitted my original patch back in 2008, since it replaced fs/inode.c

	inode->i_ino = ++last_ino;

And this was protected by a surrounding spinlock
(spin_lock(&inode_lock); at that time)

Even after Nick patches, preemption is still disabled (by two spinlocks... spin_lock(&sb_inode_list_lock); / spin_lock(&inode->i_lock);)

So patch 15/17 seems good to me, I re-sign it as-is ;)

If it happens preemption is re-enabled later (with future patches), we might need to change last_ino_get() too.

Thanks !

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#64 lore raw

Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator

Quoted 18 lines · On Thu, 30 Sep 2010 09:53:09 +0200 Eric Dumazet <[email protected]> wrote:

Le jeudi 30 septembre 2010 __ 07:36 +0200, Eric Dumazet a __crit :

Quoted 13 lines

Le mercredi 29 septembre 2010 __ 21:53 -0700, Andrew Morton a __crit :

Quoted 10 lines

fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695

Quoted 6 lines
fs/inode.c +6 -0
+static unsigned int last_ino_get(void)
+{
+	static unsigned int last_ino;
+
+	return ++last_ino;
+}

This is racy with CONFIG_PREEMPT on some architectures, I suspect. I'd suggest conversion to atomic_t with, of course, an explanatory comment ;)

In fact this code was OK when I submitted my original patch back in 2008, since it replaced fs/inode.c

Quoted 8 lines
	inode->i_ino = ++last_ino;

And this was protected by a surrounding spinlock
(spin_lock(&inode_lock); at that time)

Even after Nick patches, preemption is still disabled (by two spinlocks... spin_lock(&sb_inode_list_lock); / spin_lock(&inode->i_lock);)

You know, if it took you and me this long to work that out then perhaps the code isn't quite as clear as we would like it to be, no?

I think you know what's coming next ;)

As a general rule, if a reviewer's comment doesn't result in a code change then it should result in a changelog fix or a code comment. Because if the code wasn't clear enough to the reviewer then it won't be clear enough to later readers.

Quoted 6 lines

So patch 15/17 seems good to me, I re-sign it as-is ;)

If it happens preemption is re-enabled later (with future patches), we might need to change last_ino_get() too.

Perhaps

	WARN_ON_ONCE(preemptible());

if we had a developer-only version of WARN_ON_ONCE, which we don't.

#65 lore raw

PATCH 1/1 fs: inode per-cpu last_ino allocator

Quoted 5 lines · Andrew Morton

Perhaps

	WARN_ON_ONCE(preemptible());

if we had a developer-only version of WARN_ON_ONCE, which we don't.

Or just use a regular PER_CPU variable, even on !SMP, and get preempt safe implementation.

What do you think of following patch, on top of current linux-2.6 tree ?

Thanks

[PATCH] fs: inode per-cpu last_ino allocator

new_inode() dirties a contended cache line to get increasing inode numbers.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

---
 fs/inode.c |   45 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 38 insertions(+), 7 deletions(-)
fs/inode.c +38 -7
index 8646433..122914e 100644
@@ -624,6 +624,43 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)
 }
 EXPORT_SYMBOL_GPL(inode_add_to_lists);
 
+#define LAST_INO_BATCH 1024
+
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+
+static noinline unsigned int last_ino_get(void)
+{
+	unsigned int *p = &get_cpu_var(last_ino);
+	unsigned int res = *p;
+
+#ifdef CONFIG_SMP
+	if (unlikely((res & (LAST_INO_BATCH - 1)) == 0)) {
+		static atomic_t shared_last_ino;
+		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);
+
+		res = next - LAST_INO_BATCH;
+	}
+#endif
+	*p = ++res;
+	put_cpu_var(last_ino);
+	return res;
+}
+
 /**
  *	new_inode 	- obtain an inode
  *	@sb: superblock
@@ -638,12 +675,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);
  */
 struct inode *new_inode(struct super_block *sb)
 {
-	/*
-	 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
-	 * error if st_ino won't fit in target struct field. Use 32bit counter
-	 * here to attempt to avoid that.
-	 */
-	static unsigned int last_ino;
 	struct inode *inode;
 
 	spin_lock_prefetch(&inode_lock);
@@ -652,7 +683,7 @@ struct inode *new_inode(struct super_block *sb)
 	if (inode) {
 		spin_lock(&inode_lock);
 		__inode_add_to_lists(sb, NULL, inode);
-		inode->i_ino = ++last_ino;
+		inode->i_ino = last_ino_get();
 		inode->i_state = 0;
 		spin_unlock(&inode_lock);
 	}
maintainer
#66 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 10 lines · On Thu, 30 Sep 2010 12:22:16 +0200 Eric Dumazet <[email protected]> wrote:

Le jeudi 30 septembre 2010 __ 01:14 -0700, Andrew Morton a __crit :

Quoted 5 lines

Perhaps

	WARN_ON_ONCE(preemptible());

if we had a developer-only version of WARN_ON_ONCE, which we don't.

Or just use a regular PER_CPU variable, even on !SMP, and get preempt safe implementation.

Good stuff.

Quoted 11 lines

What do you think of following patch, on top of current linux-2.6 tree ?

...

fs/inode.c +7 -0
+static noinline unsigned int last_ino_get(void)
+{
+	unsigned int *p = &get_cpu_var(last_ino);
+	unsigned int res = *p;
+
+#ifdef CONFIG_SMP
+	if (unlikely((res & (LAST_INO_BATCH - 1)) == 0)) {

fs/inode.c void inode_add_to_lists(struct super_block *sb, struct inode *inode) · line 662

Quoted 10 lines
fs/inode.c +10 -0
+		static atomic_t shared_last_ino;
+		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);
+
+		res = next - LAST_INO_BATCH;
+	}
+#endif
+	*p = ++res;
+	put_cpu_var(last_ino);
+	return res;
+}

Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.

Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.

#67 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 4 lines · Andrew Morton

Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.

Hmm, I see, please check this new patch, using the most modern stuff ;)

Quoted 2 lines

Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.

Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !

Here is the new generated code on i686 (with the noinline) :
pretty good ;)

c02e5930 <last_ino_get>:

c02e5930:	55                   	push   %ebp
c02e5931:	89 e5                	mov    %esp,%ebp
c02e5933:	64 a1 44 29 7d c0    	mov    %fs:0xc07d2944,%eax
c02e5939:	a9 ff 03 00 00       	test   $0x3ff,%eax
c02e593e:	74 09                	je     c02e5949 <last_ino_get+0x19>
c02e5940:	40                   	inc    %eax
c02e5941:	64 a3 44 29 7d c0    	mov    %eax,%fs:0xc07d2944
c02e5947:	c9                   	leave  
c02e5948:	c3                   	ret    
c02e5949:	b8 00 04 00 00       	mov    $0x400,%eax

c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880
c02e5956: eb e8 jmp c02e5940 <last_ino_get+0x10>

Thanks

[PATCH] fs: inode per-cpu last_ino allocator

new_inode() dirties a contended cache line to get increasing inode numbers.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

---
 fs/inode.c |   47 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 40 insertions(+), 7 deletions(-)
fs/inode.c +40 -7
index 8646433..5c233f0 100644
@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)
 }
 EXPORT_SYMBOL_GPL(inode_add_to_lists);
 
+#define LAST_INO_BATCH 1024
+
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int res;
+
+	get_cpu();
+	res = __this_cpu_read(last_ino);
+#ifdef CONFIG_SMP
+	if (unlikely((res & (LAST_INO_BATCH - 1)) == 0)) {
+		static atomic_t shared_last_ino;
+		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);
+
+		res = next - LAST_INO_BATCH;
+	}
+#endif
+	res++;
+	__this_cpu_write(last_ino, res);
+	put_cpu();
+	return res;
+}
+
 /**
  *	new_inode 	- obtain an inode
  *	@sb: superblock
@@ -638,12 +677,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);
  */
 struct inode *new_inode(struct super_block *sb)
 {
-	/*
-	 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
-	 * error if st_ino won't fit in target struct field. Use 32bit counter
-	 * here to attempt to avoid that.
-	 */
-	static unsigned int last_ino;
 	struct inode *inode;
 
 	spin_lock_prefetch(&inode_lock);
@@ -652,7 +685,7 @@ struct inode *new_inode(struct super_block *sb)
 	if (inode) {
 		spin_lock(&inode_lock);
 		__inode_add_to_lists(sb, NULL, inode);
-		inode->i_ino = ++last_ino;
+		inode->i_ino = last_ino_get();
 		inode->i_state = 0;
 		spin_unlock(&inode_lock);
 	}

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#68 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 29 lines · On Thu, 30 Sep 2010 19:28:05 +0200 Eric Dumazet <[email protected]> wrote:

Le jeudi 30 septembre 2010 __ 09:45 -0700, Andrew Morton a __crit :

Quoted 4 lines

Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.

Hmm, I see, please check this new patch, using the most modern stuff ;)

Quoted 2 lines

Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.

Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !

Here is the new generated code on i686 (with the noinline) :
pretty good ;)

c02e5930 <last_ino_get>:

c02e5930:	55                   	push   %ebp
c02e5931:	89 e5                	mov    %esp,%ebp
c02e5933:	64 a1 44 29 7d c0    	mov    %fs:0xc07d2944,%eax
c02e5939:	a9 ff 03 00 00       	test   $0x3ff,%eax
c02e593e:	74 09                	je     c02e5949 <last_ino_get+0x19>
c02e5940:	40                   	inc    %eax
c02e5941:	64 a3 44 29 7d c0    	mov    %eax,%fs:0xc07d2944
c02e5947:	c9                   	leave  
c02e5948:	c3                   	ret    
c02e5949:	b8 00 04 00 00       	mov    $0x400,%eax
Quoted 3 lines

c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880
c02e5956: eb e8 jmp c02e5940 <last_ino_get+0x10>

That uniprocessor, PREEMPT=n I guess.

Quoted 34 lines
fs/inode.c +28 -0
@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)
 }
 EXPORT_SYMBOL_GPL(inode_add_to_lists);
 
+#define LAST_INO_BATCH 1024
+
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int res;
+
+	get_cpu();
+	res = __this_cpu_read(last_ino);
+#ifdef CONFIG_SMP
+	if (unlikely((res & (LAST_INO_BATCH - 1)) == 0)) {
+		static atomic_t shared_last_ino;

fs/inode.c void inode_add_to_lists(struct super_block *sb, struct inode *inode) · line 664

Quoted 10 lines
fs/inode.c +10 -0
+		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);
+
+		res = next - LAST_INO_BATCH;
+	}
+#endif
+	res++;
+	__this_cpu_write(last_ino, res);
+	put_cpu();
+	return res;
+}

Looks good ;)

#69 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 35 lines · Andrew Morton
Quoted 29 lines · On Thu, 30 Sep 2010 19:28:05 +0200 Eric Dumazet <[email protected]> wrote:

Le jeudi 30 septembre 2010 __ 09:45 -0700, Andrew Morton a __crit :

Quoted 4 lines

Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.

Hmm, I see, please check this new patch, using the most modern stuff ;)

Quoted 2 lines

Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.

Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !

Here is the new generated code on i686 (with the noinline) :
pretty good ;)

c02e5930 <last_ino_get>:

c02e5930:	55                   	push   %ebp
c02e5931:	89 e5                	mov    %esp,%ebp
c02e5933:	64 a1 44 29 7d c0    	mov    %fs:0xc07d2944,%eax
c02e5939:	a9 ff 03 00 00       	test   $0x3ff,%eax
c02e593e:	74 09                	je     c02e5949 <last_ino_get+0x19>
c02e5940:	40                   	inc    %eax
c02e5941:	64 a3 44 29 7d c0    	mov    %eax,%fs:0xc07d2944
c02e5947:	c9                   	leave  
c02e5948:	c3                   	ret    
c02e5949:	b8 00 04 00 00       	mov    $0x400,%eax
Quoted 3 lines

c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880
c02e5956: eb e8 jmp c02e5940 <last_ino_get+0x10>

Quoted 2 lines

That uniprocessor, PREEMPT=n I guess.

It is SMP. PREEMPT=n

On UP, you would not have the %fs suffixes, and no "lock" before the xadd.

#70 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

In reply to Eric Dumazet

Eric,

what workload did this matter for? As mentioned elsewhere we don't actually need to set i_ino in new_inode in many cases. Normal disk/network filesystem already set their own i_ino anyway and don't need it at all. Various in-kernel filesystems don't need any valid i_ino and Nick's full series actually has a patch dealing with it, which only leaves various user-mountable syntetic filesystems that want to generate an inode number, and IMHO they're better off using iunique.

#71 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 11 lines · Christoph Hellwig

Eric,

what workload did this matter for? As mentioned elsewhere we don't actually need to set i_ino in new_inode in many cases. Normal disk/network filesystem already set their own i_ino anyway and don't need it at all. Various in-kernel filesystems don't need any valid i_ino and Nick's full series actually has a patch dealing with it, which only leaves various user-mountable syntetic filesystems that want to generate an inode number, and IMHO they're better off using iunique.

Dont focus on this small last_ino problem, it is only one problem out of a huge pile.

As you might know, I am mostly a netdev guy ;)

In 2008, I tried to scale workloads that need to setup / dismantles thousand of sockets per second. Then Nick took the challenge, because it was clear nobody was interested to review my stuff, or maybe I was not enough re-sending patches.

FS guys live in a closed world, it seems, they want to control every bit of patches.

for (i = 0; i < LIMIT; i++)

	close(socket(...));

My results were impressive :

(From 27.5 seconds to 1.62 s, on a 8 cpus machine)

http://linux.derkeiler.com/Mailing-Lists/Kernel/2008-12/msg04479.html
(fs: Scalability of sockets/pipes allocation/deallocation on SMP)

<quote>
Hi Andrew

Take v2 of this patch serie got no new feedback, maybe its time for mm inclusion for a while ?

In this third version I added last two patches, one intialy from Christoph
Lameter, and one to avoid dirtying mnt->mnt_count on hardwired fs.

Many thanks to Christoph and Paul for this SLAB_DESTROY_PER_RCU work done
on "struct file".

Thank you

Short summary : Nice speedups for allocation/deallocation of sockets/pipes
(From 27.5 seconds to 1.62 s, on a 8 cpus machine)

Long version :

To allocate a socket or a pipe we :

0) Do the usual file table manipulation (pretty scalable these days, but would be faster if 'struct file' were using SLAB_DESTROY_BY_RCU and avoid call_rcu() cache killer). This point is addressed by 6th patch.

1) allocate an inode with new_inode()
This function :
- locks inode_lock,
- dirties nr_inodes counter
- dirties inode_in_use list (for sockets/pipes, this is useless)
- dirties superblock s_inodes. - dirties last_ino counter All these are in different cache lines unfortunatly.

2) allocate a dentry
d_alloc() takes dcache_lock,
insert dentry on its parent list (dirtying sock_mnt->mnt_sb->s_root) dirties nr_dentry

3) d_instantiate() dentry (dcache_lock taken again)

4) init_file() -> atomic_inc() on sock_mnt->refcount

At close() time, we must undo the things. Its even more expensive because
of the _atomic_dec_and_lock() that stress a lot, and because of two cache
lines that are touched when an element is deleted from a list (previous and next items)

This is really bad, since sockets/pipes dont need to be visible in dcache
or an inode list per super block.

This patch series get rid of all but one contended cache lines for sockets, pipes and anonymous fd (signalfd, timerfd, ...)

socketallocbench is a very simple program (attached to this mail) that makes
a loop :

for (i = 0; i < 1000000; i++)
close(socket(AF_INET, SOCK_STREAM, 0));

Cost if one cpu runs the program :

real 1.561s
user 0.092s
sys 1.469s

Cost if 8 processes are launched on a 8 CPU machine (socketallocbench -n 8) :

real 27.496s <<<< !!!! >>>>
user 0.657s
sys 3m39.092s

Oprofile results (for the 8 process run, 3 times):

CPU: Core 2, speed 3000.03 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (Unhalted core cycles) count 100000
samples cum. samples % cum. % symbol name
3347352 3347352 28.0232 28.0232 _atomic_dec_and_lock 3301428 6648780 27.6388 55.6620 d_instantiate
2971130 9619910 24.8736 80.5355 d_alloc
241318 9861228 2.0203 82.5558 init_file
146190 10007418 1.2239 83.7797 __slab_free
144149 10151567 1.2068 84.9864 inotify_d_instantiate 143971 10295538 1.2053 86.1917 inet_create
137168 10432706 1.1483 87.3401 new_inode
117549 10550255 0.9841 88.3242 add_partial
110795 10661050 0.9275 89.2517 generic_drop_inode
107137 10768187 0.8969 90.1486 kmem_cache_alloc
94029 10862216 0.7872 90.9358 tcp_close
82837 10945053 0.6935 91.6293 dput
67486 11012539 0.5650 92.1943 dentry_iput
57751 11070290 0.4835 92.6778 iput
54327 11124617 0.4548 93.1326 tcp_v4_init_sock
49921 11174538 0.4179 93.5505 sysenter_past_esp
47616 11222154 0.3986 93.9491 kmem_cache_free
30792 11252946 0.2578 94.2069 clear_inode
27540 11280486 0.2306 94.4375 copy_from_user
26509 11306995 0.2219 94.6594 init_timer
26363 11333358 0.2207 94.8801 discard_slab
25284 11358642 0.2117 95.0918 __fput
22482 11381124 0.1882 95.2800 __percpu_counter_add
20369 11401493 0.1705 95.4505 sock_alloc
18501 11419994 0.1549 95.6054 inet_csk_destroy_sock
17923 11437917 0.1500 95.7555 sys_close

This patch serie avoids all contented cache lines and makes this "bench" pretty fast.

New cost if run on one cpu :

real 1.245s (instead of 1.561s)
user 0.074s
sys 1.161s

If run on 8 CPUS :

real 1.624s
user 0.580s
sys 12.296s

On oprofile, we finally can see network stuff coming at the front of expensive stuff. (with the exception of kmem_cache_[z]alloc(), because it has to clear 192 bytes of file structures, this takes half of the time)

CPU: Core 2, speed 3000.09 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100
000
samples cum. samples % cum. % symbol name
176586 176586 10.9376 10.9376 kmem_cache_alloc
169838 346424 10.5196 21.4572 tcp_close
105331 451755 6.5241 27.9813 tcp_v4_init_sock
105146 556901 6.5126 34.4939 tcp_v4_destroy_sock
83307 640208 5.1600 39.6539 sysenter_past_esp
80241 720449 4.9701 44.6239 inet_csk_destroy_sock
74263 794712 4.5998 49.2237 kmem_cache_free
56806 851518 3.5185 52.7422 __percpu_counter_add
48619 900137 3.0114 55.7536 copy_from_user
44803 944940 2.7751 58.5287 init_timer
28539 973479 1.7677 60.2964 d_alloc
27795 1001274 1.7216 62.0180 alloc_fd
26747 1028021 1.6567 63.6747 __fput
24312 1052333 1.5059 65.1805 sys_close
24205 1076538 1.4992 66.6798 inet_create
22409 1098947 1.3880 68.0677 alloc_inode
21359 1120306 1.3230 69.3907 release_sock
19865 1140171 1.2304 70.6211 fd_install
19472 1159643 1.2061 71.8272 lock_sock_nested
18956 1178599 1.1741 73.0013 sock_init_data
17301 1195900 1.0716 74.0729 drop_file_write_access
17113 1213013 1.0600 75.1329 inotify_d_instantiate
16384 1229397 1.0148 76.1477 dput
15173 1244570 0.9398 77.0875 local_bh_enable_ip
15017 1259587 0.9301 78.0176 local_bh_enable
13354 1272941 0.8271 78.8448 __sock_create
13139 1286080 0.8138 79.6586 inet_release
13062 1299142 0.8090 80.4676 sysenter_do_call
11935 1311077 0.7392 81.2069 iput_single

This patch serie contains 7 patches, against linux-2.6 tree, plus one patch in mm (fs: filp_cachep can be static in fs/file_table.c)

[PATCH 1/7] fs: Use a percpu_counter to track nr_dentry

Adding a percpu_counter nr_dentry avoids cache line ping pongs between cpus to maintain this metric, and dcache_lock is no more needed to protect dentry_stat.nr_dentry

We centralize nr_dentry updates at the right place :
- increments in d_alloc()
- decrements in d_free()

d_alloc() can avoid taking dcache_lock if parent is NULL

("socketallocbench -n 8" bench result : 27.5s to 25s)

[PATCH 2/7] fs: Use a percpu_counter to track nr_inodes

Avoids cache line ping pongs between cpus and prepare next patch, because updates of nr_inodes dont need inode_lock anymore.

("socketallocbench -n 8" bench result : no difference at this point)

[PATCH 3/7] fs: Introduce a per_cpu last_ino allocator

new_inode() dirties a contended cache line to get increasing inode numbers.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations.

This reduce contention on the shared last_ino, and give same spreading ino numbers than before.
(same wraparound after 232 allocations)

("socketallocbench -n 8" result : no difference)

[PATCH 4/7] fs: Introduce SINGLE dentries for pipes, socket, anon fd

Sockets, pipes and anonymous fds have interesting properties.

Like other files, they use a dentry and an inode.

But dentries for these kind of files are not hashed into dcache, since there is no way someone can lookup such a file in the vfs tree. (/proc/{pid}/fd/{number} uses a different mechanism)

Still, allocating and freeing such dentries are expensive processes, because we currently take dcache_lock inside d_alloc(), d_instantiate(), and dput(). This lock is very contended on SMP machines.

This patch defines a new DCACHE_SINGLE flag, to mark a dentry as a single one (for sockets, pipes, anonymous fd), and a new d_alloc_single(const struct qstr *name, struct inode *inode) method, called by the three subsystems.

Internally, dput() can take a fast path to dput_single() for SINGLE dentries. No more atomic_dec_and_lock()
for such dentries.

Differences betwen an SINGLE dentry and a normal one are :

1) SINGLE dentry has the DCACHE_SINGLE flag
2) SINGLE dentry's parent is itself (DCACHE_DISCONNECTED) This to avoid taking a reference on sb 'root' dentry, shared by too many dentries.
3) They are not hashed into global hash table (DCACHE_UNHASHED)
4) Their d_alias list is empty

(socket8 bench result : from 25s to 19.9s)

[PATCH 5/7] fs: new_inode_single() and iput_single()

Goal of this patch is to not touch inode_lock for socket/pipes/anonfd inodes allocation/freeing.

SINGLE dentries are attached to inodes that dont need to be linked in a list of inodes, being "inode_in_use" or "sb->s_inodes" As inode_lock was taken only to protect these lists, we avoid taking it as well.

Using iput_single() from dput_single() avoids taking inode_lock at freeing time.

This patch has a very noticeable effect, because we avoid dirtying of three contended cache lines in new_inode(), and five cache lines in iput()

("socketallocbench -n 8" result : from 19.9s to 3.01s)

[PATH 6/7] fs: struct file move from call_rcu() to SLAB_DESTROY_BY_RCU

From: Christoph Lameter <cl@xxxxxxxxxxxxxxxxxxxx>

Currently we schedule RCU frees for each file we free separately. That has
several drawbacks against the earlier file handling (in 2.6.5 f.e.), which
did not require RCU callbacks:

1. Excessive number of RCU callbacks can be generated causing long RCU queues that in turn cause long latencies. We hit SLUB page allocation more often than necessary.

2. The cache hot object is not preserved between free and realloc. A close
followed by another open is very fast with the RCUless approach because the last freed object is returned by the slab allocator that is still cache hot. RCU free means that the object is not immediately available again. The new object is cache cold and therefore open/close performance tests show a significant degradation with the RCU implementation.

One solution to this problem is to move the RCU freeing into the Slab allocator by specifying SLAB_DESTROY_BY_RCU as an option at slab creation
time. The slab allocator will do RCU frees only when it is necessary to dispose of slabs of objects (rare). So with that approach we can cut out the RCU overhead significantly.

However, the slab allocator may return the object for another use even before the RCU period has expired under SLAB_DESTROY_BY_RCU. This means there is the (unlikely) possibility that the object is going to be switched under us in sections protected by rcu_read_lock() and rcu_read_unlock(). So we need to verify that we have acquired the correct
object after establishing a stable object reference (incrementing the refcounter does that).

</quote>

http://linux.derkeiler.com/Mailing-Lists/Kernel/2008-12/msg04475.html
(1/7 fs: Use a percpu_counter to track nr_dentry)

http://patchwork.ozlabs.org/patch/13602/
(2/7 Use a percpu_counter to track nr_inodes)

http://patchwork.ozlabs.org/patch/13603/
(3/7 Introduce a per_cpu last_ino allocator)

http://patchwork.ozlabs.org/patch/13605/
(4/7 Introduce SINGLE dentries for pipes, socket, anon fd)

...

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#72 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

In reply to Eric Dumazet
Quoted 29 lines · Eric Dumazet
Quoted 4 lines · Andrew Morton

Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.

Hmm, I see, please check this new patch, using the most modern stuff ;)

Quoted 2 lines

Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.

Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !

Here is the new generated code on i686 (with the noinline) :
pretty good ;)

c02e5930 <last_ino_get>:

c02e5930:	55                   	push   %ebp
c02e5931:	89 e5                	mov    %esp,%ebp
c02e5933:	64 a1 44 29 7d c0    	mov    %fs:0xc07d2944,%eax
c02e5939:	a9 ff 03 00 00       	test   $0x3ff,%eax
c02e593e:	74 09                	je     c02e5949 <last_ino_get+0x19>
c02e5940:	40                   	inc    %eax
c02e5941:	64 a3 44 29 7d c0    	mov    %eax,%fs:0xc07d2944
c02e5947:	c9                   	leave  
c02e5948:	c3                   	ret    
c02e5949:	b8 00 04 00 00       	mov    $0x400,%eax
Quoted 7 lines

c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880
c02e5956: eb e8 jmp c02e5940 <last_ino_get+0x10>

Thanks

[PATCH] fs: inode per-cpu last_ino allocator

Thanks Eric, this looks good. You didn't seem to add a comment about preempt safety that Andrew wanted, but I'll add it.

Quoted 95 lines

new_inode() dirties a contended cache line to get increasing inode numbers.

Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).

---
 fs/inode.c |   47 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 40 insertions(+), 7 deletions(-)
fs/inode.c +40 -8
index 8646433..5c233f0 100644
@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)
 }
 EXPORT_SYMBOL_GPL(inode_add_to_lists);
 
+#define LAST_INO_BATCH 1024
+
+/*
+ * Each cpu owns a range of LAST_INO_BATCH numbers.
+ * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
+ * to renew the exhausted range.
+ *
+ * This does not significantly increase overflow rate because every CPU can
+ * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
+ * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
+ * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
+ * overflow rate by 2x, which does not seem too significant.
+ *
+ * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
+ * error if st_ino won't fit in target struct field. Use 32bit counter
+ * here to attempt to avoid that.
+ */
+static DEFINE_PER_CPU(unsigned int, last_ino);
+
+static unsigned int last_ino_get(void)
+{
+	unsigned int res;
+
+	get_cpu();
+	res = __this_cpu_read(last_ino);
+#ifdef CONFIG_SMP
+	if (unlikely((res & (LAST_INO_BATCH - 1)) == 0)) {
+		static atomic_t shared_last_ino;
+		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);
+
+		res = next - LAST_INO_BATCH;
+	}
+#endif
+	res++;
+	__this_cpu_write(last_ino, res);
+	put_cpu();
+	return res;
+}
+
 /**
  *	new_inode 	- obtain an inode
  *	@sb: superblock
@@ -638,12 +677,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);
  */
 struct inode *new_inode(struct super_block *sb)
 {
-	/*
-	 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
-	 * error if st_ino won't fit in target struct field. Use 32bit counter
-	 * here to attempt to avoid that.
-	 */
-	static unsigned int last_ino;
 	struct inode *inode;
 
 	spin_lock_prefetch(&inode_lock);
@@ -652,7 +685,7 @@ struct inode *new_inode(struct super_block *sb)
 	if (inode) {
 		spin_lock(&inode_lock);
 		__inode_add_to_lists(sb, NULL, inode);
-		inode->i_ino = ++last_ino;
+		inode->i_ino = last_ino_get();
 		inode->i_state = 0;
 		spin_unlock(&inode_lock);
 	}
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#73 lore raw

Re: [PATCH] fs: inode per-cpu last_ino allocator

Quoted 5 lines · Nick Piggin
Quoted 1 line · Eric Dumazet

[PATCH] fs: inode per-cpu last_ino allocator

Thanks Eric, this looks good. You didn't seem to add a comment about preempt safety that Andrew wanted, but I'll add it.

Oh I beg your pardon, you added preempt disable in the code. Much safer, thanks.

#74 lore raw

PATCH 16/17 fs: Convert nr_inodes to a per-cpu counter

In reply to Dave Chinner

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

---
 fs/fs-writeback.c  |   16 +++++++++++++---
 fs/inode.c         |   35 +++++++++++++++++++++++++++++++++--
 include/linux/fs.h |    5 ++++-
 kernel/sysctl.c    |    4 ++--
 4 files changed, 52 insertions(+), 8 deletions(-)
fs/fs-writeback.c +13 -3
index 8e390e8..348cc18 100644
@@ -728,6 +728,7 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 {
 	unsigned long expired;
 	long nr_pages;
+	int nr_dirty_inodes;
 
 	/*
 	 * When set to zero, disable periodic writeback
@@ -740,10 +741,15 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)
 	if (time_before(jiffies, expired))
 		return 0;
 
+	/* approximate dirty inodes */
+	nr_dirty_inodes = get_nr_inodes() - get_nr_inodes_unused();
+	if (nr_dirty_inodes < 0)
+		nr_dirty_inodes = 0;
+
 	wb->last_old_flush = jiffies;
 	nr_pages = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
-			inodes_stat.nr_inodes - inodes_stat.nr_unused;
+			nr_dirty_inodes;
 
 	if (nr_pages) {
 		struct wb_writeback_work work = {
@@ -1105,6 +1111,7 @@ void writeback_inodes_sb(struct super_block *sb)
 {
 	unsigned long nr_dirty = global_page_state(NR_FILE_DIRTY);
 	unsigned long nr_unstable = global_page_state(NR_UNSTABLE_NFS);
+	int nr_dirty_inodes;
 	DECLARE_COMPLETION_ONSTACK(done);
 	struct wb_writeback_work work = {
 		.sb		= sb,
@@ -1114,8 +1121,11 @@ void writeback_inodes_sb(struct super_block *sb)
 
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 
-	work.nr_pages = nr_dirty + nr_unstable +
-			inodes_stat.nr_inodes - inodes_stat.nr_unused;
+	nr_dirty_inodes = get_nr_inodes() - get_nr_inodes_unused();
+	if (nr_dirty_inodes < 0)
+		nr_dirty_inodes = 0;
+
+	work.nr_pages = nr_dirty + nr_unstable + nr_dirty_inodes;
 
 	bdi_queue_work(sb->s_bdi, &work);
 	wait_for_completion(&done);
fs/inode.c +33 -2
index 1388450..a91efab 100644
@@ -143,8 +143,38 @@ struct inodes_stat_t inodes_stat = {
 	.nr_unused = 0,
 };
 
+static DEFINE_PER_CPU(unsigned int, nr_inodes);
+
 static struct kmem_cache *inode_cachep __read_mostly;
 
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}
+
+int get_nr_inodes_unused(void)
+{
+	return inodes_stat.nr_unused;
+}
+
+/*
+ * Handle nr_dentry sysctl
+ */
+int proc_nr_inodes(ctl_table *table, int write,
+		   void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+#if defined(CONFIG_SYSCTL) && defined(CONFIG_PROC_FS)
+	inodes_stat.nr_inodes = get_nr_inodes();
+	return proc_dointvec(table, write, buffer, lenp, ppos);
+#else
+	return -ENOSYS;
+#endif
+}
+
 static void wake_up_inode(struct inode *inode)
 {
 	/*
@@ -232,6 +262,8 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	inode->i_fsnotify_mask = 0;
 #endif
 
+	this_cpu_inc(nr_inodes);
+
 	return 0;
 out:
 	return -ENOMEM;
@@ -272,6 +304,7 @@ void __destroy_inode(struct inode *inode)
 	if (inode->i_default_acl && inode->i_default_acl != ACL_NOT_CACHED)
 		posix_acl_release(inode->i_default_acl);
 #endif
+	this_cpu_dec(nr_inodes);
 }
 EXPORT_SYMBOL(__destroy_inode);
 
@@ -643,7 +676,6 @@ static inline void
 __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,
 			struct inode *inode)
 {
-	inodes_stat.nr_inodes++;
 	list_add(&inode->i_sb_list, &sb->s_inodes);
 	spin_unlock(&sb_inode_list_lock);
 	if (b) {
@@ -1382,7 +1414,6 @@ static void iput_final(struct inode *inode)
 		spin_unlock(&wb_inode_list_lock);
 	}
 	list_del_init(&inode->i_sb_list);
-	inodes_stat.nr_inodes--;
 	spin_unlock(&sb_inode_list_lock);
 	WARN_ON(inode->i_state & I_NEW);
 	inode->i_state |= I_FREEING;
include/linux/fs.h +4 -1
index 3a43313..d2ee5d0 100644
@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;
 extern int get_max_files(void);
 extern int sysctl_nr_open;
 extern struct inodes_stat_t inodes_stat;
+extern int get_nr_inodes(void);
+extern int get_nr_inodes_unused(void);
 extern int leases_enable, lease_break_time;
 
 struct buffer_head;
@@ -2477,7 +2479,8 @@ ssize_t simple_attr_write(struct file *file, const char __user *buf,
 struct ctl_table;
 int proc_nr_files(struct ctl_table *table, int write,
 		  void __user *buffer, size_t *lenp, loff_t *ppos);
-
+int proc_nr_inodes(struct ctl_table *table, int write,
+		   void __user *buffer, size_t *lenp, loff_t *ppos);
 int __init get_filesystem_list(char *buf);
 
 #define ACC_MODE(x) ("\004\002\006\006"[(x)&O_ACCMODE])
kernel/sysctl.c +2 -2
index f88552c..33d1733 100644
@@ -1340,14 +1340,14 @@ static struct ctl_table fs_table[] = {
 		.data		= &inodes_stat,
 		.maxlen		= 2*sizeof(int),
 		.mode		= 0444,
-		.proc_handler	= proc_dointvec,
+		.proc_handler	= proc_nr_inodes,
 	},
 	{
 		.procname	= "inode-state",
 		.data		= &inodes_stat,
 		.maxlen		= 7*sizeof(int),
 		.mode		= 0444,
-		.proc_handler	= proc_dointvec,
+		.proc_handler	= proc_nr_inodes,
 	},
 	{
 		.procname	= "file-nr",
Signature · 1 line
1.7.1
#75 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

fs/inode.c struct inodes_stat_t inodes_stat = { · line 176

Quoted 13 lines
fs/inode.c +13 -0
+/*
+ * Handle nr_dentry sysctl
+ */
+int proc_nr_inodes(ctl_table *table, int write,
+		   void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+#if defined(CONFIG_SYSCTL) && defined(CONFIG_PROC_FS)
+	inodes_stat.nr_inodes = get_nr_inodes();
+	return proc_dointvec(table, write, buffer, lenp, ppos);
+#else
+	return -ENOSYS;
+#endif
+}

Why would we even bother to define the handler if we don't have /proc/sys/ support?

maintainer
#76 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

In reply to Dave Chinner
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

...

fs/inode.c struct inodes_stat_t inodes_stat = { · line 157

Quoted 8 lines
fs/inode.c +8 -0
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}

This reimplements percpu_counter_sum_positive(), rather poorly

If one never intends to use the approximate percpu_counter_read() then one could initialise the counter with a really large batch value, for a very small performance gain.

fs/inode.c extern struct files_stat_struct files_stat; · line 411

Quoted 13 lines
fs/inode.c +6 -0
+int get_nr_inodes_unused(void)
+{
+	return inodes_stat.nr_unused;
+}
 
...
@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;
 extern int get_max_files(void);
 extern int sysctl_nr_open;
 extern struct inodes_stat_t inodes_stat;
+extern int get_nr_inodes(void);
+extern int get_nr_inodes_unused(void);

These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.

Quoted 3 lines

...

#77 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

...

Quoted 10 lines

fs/inode.c struct inodes_stat_t inodes_stat = { · line 157

Quoted 8 lines
fs/inode.c +8 -0
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}

This reimplements percpu_counter_sum_positive(), rather poorly

I thought so - it was on my list of things to check when redoing this patch. I'll fix that up appropritately.

Quoted 12 lines

If one never intends to use the approximate percpu_counter_read() then one could initialise the counter with a really large batch value, for a very small performance gain.

Quoted 7 lines
fs/inode.c +4 -0
+int get_nr_inodes_unused(void)
+{
+	return inodes_stat.nr_unused;
+}
 
...
Quoted 9 lines

include/linux/fs.h extern struct files_stat_struct files_stat; · line 411

Quoted 6 lines
include/linux/fs.h +2 -0
@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;
 extern int get_max_files(void);
 extern int sysctl_nr_open;
 extern struct inodes_stat_t inodes_stat;
+extern int get_nr_inodes(void);
+extern int get_nr_inodes_unused(void);

These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.

Will fix.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#78 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 16 lines · Dave Chinner
Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

...

Quoted 10 lines
Quoted 10 lines

fs/inode.c struct inodes_stat_t inodes_stat = { · line 157

Quoted 8 lines
fs/inode.c +8 -0
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}

This reimplements percpu_counter_sum_positive(), rather poorly

Why is it poorly?

Quoted 3 lines
Quoted 3 lines

If one never intends to use the approximate percpu_counter_read() then one could initialise the counter with a really large batch value, for a very small performance gain.

I did that to start with, and I was just looking to shave off cycles and icache size. this_cpu_inc on x86 on a local variable is really tiny and fast. percpu_counter does a function call which is large and clobbers memory and registers, several branches, several loads and stores, etc.

When it is a simple dumb statistics counter but with a critical fastpath, this_cpu_inc just seems to be so much better.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fs/inode.c +4 -0
+int get_nr_inodes_unused(void)
+{
+	return inodes_stat.nr_unused;
+}
 
...
Quoted 9 lines
Quoted 9 lines

include/linux/fs.h extern struct files_stat_struct files_stat; · line 411

Quoted 6 lines
include/linux/fs.h +2 -0
@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;
 extern int get_max_files(void);
 extern int sysctl_nr_open;
 extern struct inodes_stat_t inodes_stat;
+extern int get_nr_inodes(void);
+extern int get_nr_inodes_unused(void);

These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.

Any ideas? inodes_stat_nr_unused()?

#79 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 28 lines · Nick Piggin
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

...

Quoted 10 lines
Quoted 10 lines

fs/inode.c struct inodes_stat_t inodes_stat = { · line 157

Quoted 8 lines
fs/inode.c +8 -0
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}

This reimplements percpu_counter_sum_positive(), rather poorly

Quoted 1 line

Why is it poorly?

Nick

Some people believe percpu_counter object is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...

But if we do this loop only on rare events, like "cat /proc/sys/fs/inode-nr", then the percpu_counter() is more expensive, because percpu_add() _is_ more expensive :

- Its a function call and lot of instructions/cycles per call, while this_cpu_inc(nr_inodes) is a single instruction, using no register on x86.

- Its possibly accessing a shared spinlock and counter when the percpu counter reaches the batch limit.

To recap : nr_inodes is not a counter that needs to be estimated in real time, since we have not limit on number of inodes in the machine (limit is the memory allocator).

Unless someone can prove "cat /proc/sys/fs/inode-nr" must be performed thousand of times per second on their setup, the choice I made to scale nr_inodes is better over the 'obvious percpu_counter choice'

This choice was made to scale some counters in network stack some years ago, and this rocks.

Thanks

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#80 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 38 lines · On Sat, 16 Oct 2010 10:29:08 +0200 Eric Dumazet <[email protected]> wrote:

Le samedi 16 octobre 2010 __ 18:55 +1100, Nick Piggin a __crit :

Quoted 28 lines · Dave Chinner
Quoted 16 lines · Andrew Morton
Quoted 15 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
Quoted 14 lines

From: Eric Dumazet <[email protected]>

The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.

...

Quoted 10 lines
Quoted 10 lines

fs/inode.c struct inodes_stat_t inodes_stat = { · line 157

Quoted 8 lines
fs/inode.c +8 -0
+int get_nr_inodes(void)
+{
+	int i;
+	int sum = 0;
+	for_each_possible_cpu(i)
+		sum += per_cpu(nr_inodes, i);
+	return sum < 0 ? 0 : sum;
+}

This reimplements percpu_counter_sum_positive(), rather poorly

Quoted 1 line

Why is it poorly?

Nick

Some people believe percpu_counter object is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...

Quoted 7 lines

But if we do this loop only on rare events, like "cat /proc/sys/fs/inode-nr", then the percpu_counter() is more expensive, because percpu_add() _is_ more expensive :

- Its a function call and lot of instructions/cycles per call, while this_cpu_inc(nr_inodes) is a single instruction, using no register on x86.

You want an inlined percpu_counter_inc() then write one! Bonus points for writing this_cpu_add_return() and doing it without a preempt_disable(). It collapses to just a few instructions.

It's extremely poor form to say "oh X sucks so I'm going to implement my own" without first addressing why X allegedly sucks.

Quoted 2 lines

- Its possibly accessing a shared spinlock and counter when the percpu counter reaches the batch limit.

That's in the noise foor.

Quoted 10 lines

To recap : nr_inodes is not a counter that needs to be estimated in real time, since we have not limit on number of inodes in the machine (limit is the memory allocator).

Unless someone can prove "cat /proc/sys/fs/inode-nr" must be performed thousand of times per second on their setup, the choice I made to scale nr_inodes is better over the 'obvious percpu_counter choice'

This choice was made to scale some counters in network stack some years ago, and this rocks.

And we get open-coded reimplementations of the same damn thing all over the tree.

#81 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 10 lines · Andrew Morton
Quoted 9 lines · On Sat, 16 Oct 2010 10:29:08 +0200 Eric Dumazet <[email protected]> wrote:

Some people believe percpu_counter object is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...

But if we do this loop only on rare events, like "cat /proc/sys/fs/inode-nr", then the percpu_counter() is more expensive, because percpu_add() _is_ more expensive :

Quoted 8 lines
Quoted 3 lines

- Its a function call and lot of instructions/cycles per call, while this_cpu_inc(nr_inodes) is a single instruction, using no register on x86.

You want an inlined percpu_counter_inc() then write one! Bonus points for writing this_cpu_add_return() and doing it without a preempt_disable(). It collapses to just a few instructions.

A few instructions, but no guarantee of false sharing eviction.

Each time one cpu dirties the percpu_counter object, it slow down other cpus because they need to fetch the cache line again.

Btw, I believe my previous patch against include/linux/percpu_counter.h was lost. Are you sure I am the right guy to work on percpu_counter infra ? If yes I can implement your inlined idea.

Thanks

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#82 lore raw

PATCH 1/1 percpu_counter : add percpu_counter_add_fast()

Andrew

I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.

Thanks

[PATCH] percpu_counter : percpu_counter_add_fast()

The current way to change a percpu_counter is to call percpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)

When we dont need to maintain the approximate value of the percpu_counter (aka fbc->count), and dont need a "s64" wide counter but a regular "int" or "long" one, we can use this new function :
percpu_counter_add_fast()

This function is pretty fast :
- One instruction on x86 SMP, no register pressure.
- Is safe in preempt enabled contexts.
- No lock acquisition, less false sharing.

Users of this percpu_counter variant should not use percpu_counter_read() or percpu_counter_read_positive() anymore, only percpu_counter_sum{_positive}() variant.

Note: we could add later irqsafe variant, still one instruction on x86 SMP...

CC Christoph Lameter <[email protected]>
CC Nick Piggin <[email protected]>
CC Dave Chinner <[email protected]>
---
 include/linux/percpu_counter.h |   36 +++++++++++++++++++++++++++----
 lib/percpu_counter.c           |   12 +++++-----
 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/percpu_counter.h +32 -4
index 8a7d510..b9f4cc1 100644
@@ -3,7 +3,9 @@
 /*
  * A simple "approximate counter" for use in ext2 and ext3 superblocks.
  *
- * WARNING: these things are HUGE.  4 kbytes per counter on 32-way P4.
+ * WARNING: these things are big.  sizeof(long) bytes per possible cpu per counter.
+ * For a 64 cpus 64bit machine :
+ *	64*8 (512) bytes + sizeof(struct percpu_counter)
  */
 
 #include <linux/spinlock.h>
@@ -21,7 +23,7 @@ struct percpu_counter {
 #ifdef CONFIG_HOTPLUG_CPU
 	struct list_head list;	/* All percpu_counters are on a list */
 #endif
-	s32 __percpu *counters;
+	long __percpu *counters;
 };
 
 extern int percpu_counter_batch;
@@ -38,7 +40,7 @@ int __percpu_counter_init(struct percpu_counter *fbc, s64 amount,
 
 void percpu_counter_destroy(struct percpu_counter *fbc);
 void percpu_counter_set(struct percpu_counter *fbc, s64 amount);
-void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch);
+void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, long batch);
 s64 __percpu_counter_sum(struct percpu_counter *fbc);
 int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs);
 
@@ -47,6 +49,24 @@ static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount)
 	__percpu_counter_add(fbc, amount, percpu_counter_batch);
 }
 
+/**
+ * percpu_counter_add_fast - fast variant of percpu_counter_add
+ * @fbc: pointer to percpu_counter
+ * @amount: value to add to counter
+ *
+ * Add amount to a percpu_counter object, without approximate (fbc->count)
+ * estimation / correction.
+ * Notes :
+ * - This fast version is limited to "long" counters, not "s64".
+ * - It is preempt safe, but not IRQ safe (on UP)
+ * - Use of percpu_counter_read{_positive}() is discouraged.
+ * - fbc->count accumulates the counters from offlined cpus.
+ */
+static inline void percpu_counter_add_fast(struct percpu_counter *fbc, long amount)
+{
+	this_cpu_add(*fbc->counters, amount);
+}
+
 static inline s64 percpu_counter_sum_positive(struct percpu_counter *fbc)
 {
 	s64 ret = __percpu_counter_sum(fbc);
@@ -118,7 +138,15 @@ percpu_counter_add(struct percpu_counter *fbc, s64 amount)
 }
 
 static inline void
-__percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
+percpu_counter_add_fast(struct percpu_counter *fbc, long amount)
+{
+	preempt_disable();
+	fbc->count += amount;
+	preempt_enable();
+}
+
+static inline void
+__percpu_counter_add(struct percpu_counter *fbc, s64 amount, long batch)
 {
 	percpu_counter_add(fbc, amount);
 }
lib/percpu_counter.c +6 -6
index ec9048e..93d50a5 100644
@@ -18,7 +18,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount)
 
 	spin_lock(&fbc->lock);
 	for_each_possible_cpu(cpu) {
-		s32 *pcount = per_cpu_ptr(fbc->counters, cpu);
+		long *pcount = per_cpu_ptr(fbc->counters, cpu);
 		*pcount = 0;
 	}
 	fbc->count = amount;
@@ -26,10 +26,10 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount)
 }
 EXPORT_SYMBOL(percpu_counter_set);
 
-void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)
+void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, long batch)
 {
 	s64 count;
-	s32 *pcount;
+	long *pcount;
 	int cpu = get_cpu();
 
 	pcount = per_cpu_ptr(fbc->counters, cpu);
@@ -58,7 +58,7 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc)
 	spin_lock(&fbc->lock);
 	ret = fbc->count;
 	for_each_online_cpu(cpu) {
-		s32 *pcount = per_cpu_ptr(fbc->counters, cpu);
+		long *pcount = per_cpu_ptr(fbc->counters, cpu);
 		ret += *pcount;
 	}
 	spin_unlock(&fbc->lock);
@@ -72,7 +72,7 @@ int __percpu_counter_init(struct percpu_counter *fbc, s64 amount,
 	spin_lock_init(&fbc->lock);
 	lockdep_set_class(&fbc->lock, key);
 	fbc->count = amount;
-	fbc->counters = alloc_percpu(s32);
+	fbc->counters = alloc_percpu(long);
 	if (!fbc->counters)
 		return -ENOMEM;
 #ifdef CONFIG_HOTPLUG_CPU
@@ -123,7 +123,7 @@ static int __cpuinit percpu_counter_hotcpu_callback(struct notifier_block *nb,
 	cpu = (unsigned long)hcpu;
 	mutex_lock(&percpu_counters_lock);
 	list_for_each_entry(fbc, &percpu_counters, list) {
-		s32 *pcount;
+		long *pcount;
 		unsigned long flags;
 
 		spin_lock_irqsave(&fbc->lock, flags);
#83 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 2 lines · Eric Dumazet

I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.

The prior patch was accepted by Andrew.

Quoted 1 line

+ * - It is preempt safe, but not IRQ safe (on UP)

The IRQ safeness depends on the arch. this_cpu_add() in general only guarantees safety against preemption. It so happens that the x86 implementation is irq safe as well.

The IRQ safety for UP is therefore not an issue if you use this_cpu_add().

If you want to guarantee irqsafeness then use irqsafe_cpu_add() instead. It generates the same code on x86 for SMP but takes care of the UP issues.

include/linux/percpu_counter.h static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · line 68

Quoted 4 lines
include/linux/percpu_counter.h +4 -0
+static inline void percpu_counter_add_fast(struct percpu_counter *fbc, long amount)
+{
+	this_cpu_add(*fbc->counters, amount);
+}

What happens in case of counter overflow?

#84 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Eric Dumazet

I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.

The prior patch was accepted by Andrew.

Quoted 8 lines
Quoted 1 line

+ * - It is preempt safe, but not IRQ safe (on UP)

The IRQ safeness depends on the arch. this_cpu_add() in general only guarantees safety against preemption. It so happens that the x86 implementation is irq safe as well.

The IRQ safety for UP is therefore not an issue if you use this_cpu_add().

Nope, on UP, we dont use a per_cpu field, just a "s64 count".

struct percpu_counter {
        s64 count;
};
Quoted 10 lines

If you want to guarantee irqsafeness then use irqsafe_cpu_add() instead. It generates the same code on x86 for SMP but takes care of the UP issues.

include/linux/percpu_counter.h static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · line 68

Quoted 4 lines
include/linux/percpu_counter.h +4 -0
+static inline void percpu_counter_add_fast(struct percpu_counter *fbc, long amount)
+{
+	this_cpu_add(*fbc->counters, amount);
+}

What happens in case of counter overflow?

Nothing special, as I stated the usuable width of such counters would be restricted to a long, not an s64.

It should be enough to count "number of inodes, of sockets, ..."

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#85 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 5 lines · Eric Dumazet

Nope, on UP, we dont use a per_cpu field, just a "s64 count".

struct percpu_counter {
        s64 count;
};

Ok that is percpu counter specific then.

maintainer
#86 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

In reply to Eric Dumazet
Quoted 9 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:

The current way to change a percpu_counter is to call percpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)

When we dont need to maintain the approximate value of the percpu_counter (aka fbc->count), and dont need a "s64" wide counter but a regular "int" or "long" one, we can use this new function :
percpu_counter_add_fast()

Quoted 8 lines

This function is pretty fast :
- One instruction on x86 SMP, no register pressure.
- Is safe in preempt enabled contexts.
- No lock acquisition, less false sharing.

Users of this percpu_counter variant should not use percpu_counter_read() or percpu_counter_read_positive() anymore, only percpu_counter_sum{_positive}() variant.

That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using percpu_counter_add() variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of

{
	val = this_cpu_add_return(*fbc->counters, amount);
	if (unlikely(abs(val) > fbc->batch))
		out_of_line_stuff();
}	

I suppose what you're proposing here is useful, although the name isn't a good one. It's a different way of using the existing data structure. I'd suggest that a better name is something like percpu_counter_add_local()?

#87 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 7 lines · Andrew Morton

That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using percpu_counter_add() variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of

{
	val = this_cpu_add_return(*fbc->counters, amount);

this_cpu_add_return() is not in the kernel but could be realized using a variant offshoot of cmpxchg_local. I had something like that initially but omitted it since there was no use case.

maintainer
#88 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 7 lines · On Thu, 21 Oct 2010 18:10:13 -0500 (CDT) Christoph Lameter <[email protected]> wrote:
Quoted 5 lines · Andrew Morton

That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using percpu_counter_add() variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of

Quoted 6 lines
Quoted 2 lines
{
	val = this_cpu_add_return(*fbc->counters, amount);

this_cpu_add_return() is not in the kernel but could be realized using a variant offshoot of cmpxchg_local. I had something like that initially but omitted it since there was no use case.

this_cpu_add_return() isn't really needed in this application.

{
	this_cpu_add(*fbc->counters, amount);
	if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch))
		out_of_line_stuff();
}	

will work just fine.

maintainer
#89 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 9 lines · On Thu, 21 Oct 2010 17:45:36 -0700 Andrew Morton <[email protected]>…

this_cpu_add_return() isn't really needed in this application.

{
	this_cpu_add(*fbc->counters, amount);
	if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch))
		out_of_line_stuff();
}	

will work just fine.

Did that. Got alarmed at a few things.

The compiler cannot CSE the above code - it has to reload the percpu base each time. Doing it by hand:

{
	long *p;

	p = this_cpu_ptr(fbc->counters);
	*p += amount;
	if (unlikely(abs(*p) > fbc->batch))
		out_of_line_stuff();
}	

generates better code.

So this:

static __always_inline void percpu_counter_add_batch(struct percpu_counter *fbc,
s64 amount, long batch)

{
	long *pcounter;

	preempt_disable();
	pcounter = this_cpu_ptr(fbc->counters);
	*pcounter += amount;
	if (unlikely(abs(*pcounter) >= batch))
		percpu_counter_handle_overflow(fbc);
	preempt_enable();
}

when compiling this:

lib/proportions.c renamed from lib/proportions.c~b +5 -0
@@ -263,6 +263,11 @@ void __prop_inc_percpu(struct prop_descr
 	prop_put_global(pd, pg);
 }
 
+void foo(struct prop_local_percpu *pl)
+{
+	percpu_counter_add(&pl->events, 1);
+}
+
 /*
  * identical to __prop_inc_percpu, except that it limits this pl's fraction to
  * @frac/PROP_FRAC_BASE by ignoring events when this limit has been exceeded.

comes down to

.globl foo
.type foo, @function
foo:

	pushq	%rbp	#
	movslq	percpu_counter_batch(%rip),%rcx	# percpu_counter_batch, batch
	movq	96(%rdi), %rdx	# <variable>.counters, tcp_ptr__
	movq	%rsp, %rbp	#,
#APP
	add %gs:this_cpu_off, %rdx	# this_cpu_off, tcp_ptr__
#NO_APP
	movq	(%rdx), %rax	#* tcp_ptr__, D.11817
	incq	%rax	# D.11817
	movq	%rax, (%rdx)	# D.11817,* tcp_ptr__
	cqto
	xorq	%rdx, %rax	# tmp67, D.11817
	subq	%rdx, %rax	# tmp67, D.11817
	cmpq	%rcx, %rax	# batch, D.11817
	jl	.L33	#,
	call	percpu_counter_handle_overflow	#

.L33:
leave
ret

But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the percpu_counter.h functions into lib/proportions.s. That's rather a worry.

lib/proportions.o got rather larger as a result of inlining things and it's not obvious that it's all a net benefit.

#90 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 87 lines · Andrew Morton
Quoted 9 lines · On Thu, 21 Oct 2010 17:45:36 -0700 Andrew Morton <[email protected]>…

this_cpu_add_return() isn't really needed in this application.

{
	this_cpu_add(*fbc->counters, amount);
	if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch))
		out_of_line_stuff();
}	

will work just fine.

Did that. Got alarmed at a few things.

The compiler cannot CSE the above code - it has to reload the percpu base each time. Doing it by hand:

{
	long *p;

	p = this_cpu_ptr(fbc->counters);
	*p += amount;
	if (unlikely(abs(*p) > fbc->batch))
		out_of_line_stuff();
}	

generates better code.

So this:

static __always_inline void percpu_counter_add_batch(struct percpu_counter *fbc,
s64 amount, long batch)

{
	long *pcounter;

	preempt_disable();
	pcounter = this_cpu_ptr(fbc->counters);
	*pcounter += amount;
	if (unlikely(abs(*pcounter) >= batch))
		percpu_counter_handle_overflow(fbc);
	preempt_enable();
}

when compiling this:

lib/proportions.c renamed from lib/proportions.c~b +5 -0
@@ -263,6 +263,11 @@ void __prop_inc_percpu(struct prop_descr
 	prop_put_global(pd, pg);
 }
 
+void foo(struct prop_local_percpu *pl)
+{
+	percpu_counter_add(&pl->events, 1);
+}
+
 /*
  * identical to __prop_inc_percpu, except that it limits this pl's fraction to
  * @frac/PROP_FRAC_BASE by ignoring events when this limit has been exceeded.

comes down to

.globl foo
.type foo, @function
foo:

	pushq	%rbp	#
	movslq	percpu_counter_batch(%rip),%rcx	# percpu_counter_batch, batch
	movq	96(%rdi), %rdx	# <variable>.counters, tcp_ptr__
	movq	%rsp, %rbp	#,
#APP
	add %gs:this_cpu_off, %rdx	# this_cpu_off, tcp_ptr__
#NO_APP
	movq	(%rdx), %rax	#* tcp_ptr__, D.11817
	incq	%rax	# D.11817
	movq	%rax, (%rdx)	# D.11817,* tcp_ptr__
	cqto
	xorq	%rdx, %rax	# tmp67, D.11817
	subq	%rdx, %rax	# tmp67, D.11817
	cmpq	%rcx, %rax	# batch, D.11817
	jl	.L33	#,
	call	percpu_counter_handle_overflow	#

.L33:
leave
ret

Quoted 4 lines

But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the percpu_counter.h functions into lib/proportions.s. That's rather a worry.

You you have the "ignore inlining" and "compile for size" turned on? They often suck.

maintainer
#91 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 6 lines · On Fri, 22 Oct 2010 12:58:45 +1100 Nick Piggin <[email protected]> wrote:
Quoted 4 lines

But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the percpu_counter.h functions into lib/proportions.s. That's rather a worry.

You you have the "ignore inlining"

# CONFIG_OPTIMIZE_INLINING is not set

Quoted 1 line

and "compile for size" turned on?

CONFIG_CC_OPTIMIZE_FOR_SIZE=y

Quoted 1 line

They often suck.

Everything sucks. Are they any use?

With

# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_OPTIMIZE_INLINING=y

my kernel/built-in.o text went from 563638 bytes to 659852. That's rather a lot.

I haven't looked at this stuff in years. Has anyone dug into it?

#92 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton

this_cpu_add_return() isn't really needed in this application.

{
	this_cpu_add(*fbc->counters, amount);
	if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch))
		out_of_line_stuff();
}	

will work just fine.

Hmm, you cannot do this on 32bit machines because "amount" is 64bit wide.

Switching counters to s64 is not an option (makes summation racy, and memory use bigger)

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#93 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

In reply to Eric Dumazet
Quoted 3 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:

The current way to change a percpu_counter is to call percpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)

This is incorrect. With my compiler it's 25 instructions except in the very rare case where a batch overflow occurs.

And more than half of that is function call entry/exit overhead.

#94 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:

The current way to change a percpu_counter is to call percpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)

This is incorrect. With my compiler it's 25 instructions except in the very rare case where a batch overflow occurs.

Hmm

Quoted 2 lines

And more than half of that is function call entry/exit overhead.

gcc version 4.5.1

count : 5 instructions to call function

c10cfbb5:	a1 00 8f 53 c1       	mov    0xc1538f00,%eax
c10cfbba:	31 c9                	xor    %ecx,%ecx
c10cfbbc:	89 04 24             	mov    %eax,(%esp)
c10cfbbf:	ba 01 00 00 00       	mov    $0x1,%edx
c10cfbc4:	b8 c0 5b 50 c1       	mov    $0xc1505bc0,%eax
c10cfbc9:	e8 a2 64 0b 00       	call   c1186070 <__percpu_counter_add>

Then 39 instructions in hot path (no lock taken)

So its more than 40 as I stated

c1186070 <__percpu_counter_add>:

c1186070:	55                   	push   %ebp
c1186071:	89 e5                	mov    %esp,%ebp
c1186073:	83 ec 1c             	sub    $0x1c,%esp
c1186076:	89 5d f4             	mov    %ebx,-0xc(%ebp)
c1186079:	89 75 f8             	mov    %esi,-0x8(%ebp)
c118607c:	89 7d fc             	mov    %edi,-0x4(%ebp)
c118607f:	89 c3                	mov    %eax,%ebx
c1186081:	8b 73 20             	mov    0x20(%ebx),%esi
c1186084:	64 a1 6c 10 59 c1    	mov    %fs:0xc159106c,%eax
c118608a:	8b 3c 85 a0 7d 53 c1 	mov    -0x3eac8260(,%eax,4),%edi
c1186091:	01 fe                	add    %edi,%esi
c1186093:	89 75 e8             	mov    %esi,-0x18(%ebp)
c1186096:	8b 06                	mov    (%esi),%eax
c1186098:	8b 75 08             	mov    0x8(%ebp),%esi
c118609b:	89 c7                	mov    %eax,%edi
c118609d:	89 45 ec             	mov    %eax,-0x14(%ebp)
c11860a0:	c1 ff 1f             	sar    $0x1f,%edi
c11860a3:	01 55 ec             	add    %edx,-0x14(%ebp)
c11860a6:	89 7d f0             	mov    %edi,-0x10(%ebp)
c11860a9:	89 f7                	mov    %esi,%edi
c11860ab:	11 4d f0             	adc    %ecx,-0x10(%ebp)
c11860ae:	c1 ff 1f             	sar    $0x1f,%edi
c11860b1:	39 7d f0             	cmp    %edi,-0x10(%ebp)
c11860b4:	7f 3a                	jg     c11860f0 <__percpu_counter_add+0x80>
c11860b6:	7d 68                	jge    c1186120 <__percpu_counter_add+0xb0>
c11860b8:	8b 4d 08             	mov    0x8(%ebp),%ecx
c11860bb:	f7 d9                	neg    %ecx
c11860bd:	89 ca                	mov    %ecx,%edx
c11860bf:	c1 fa 1f             	sar    $0x1f,%edx
c11860c2:	39 55 f0             	cmp    %edx,-0x10(%ebp)
c11860c5:	7e 19                	jle    c11860e0 <__percpu_counter_add+0x70>
c11860c7:	8b 7d ec             	mov    -0x14(%ebp),%edi
c11860ca:	8b 75 e8             	mov    -0x18(%ebp),%esi
c11860cd:	89 3e                	mov    %edi,(%esi)
c11860cf:	8b 5d f4             	mov    -0xc(%ebp),%ebx
c11860d2:	8b 75 f8             	mov    -0x8(%ebp),%esi
c11860d5:	8b 7d fc             	mov    -0x4(%ebp),%edi
c11860d8:	c9                   	leave  
c11860d9:	c3                   	ret    
c11860da:	8d b6 00 00 00 00    	lea    0x0(%esi),%esi
c11860e0:	7c 0e                	jl     c11860f0 <__percpu_counter_add+0x80>
c11860e2:	39 4d ec             	cmp    %ecx,-0x14(%ebp)
c11860e5:	77 e0                	ja     c11860c7 <__percpu_counter_add+0x57>
c11860e7:	89 f6                	mov    %esi,%esi
c11860e9:	8d bc 27 00 00 00 00 	lea    0x0(%edi,%eiz,1),%edi
c11860f0:	89 d8                	mov    %ebx,%eax
c11860f2:	e8 e9 41 1d 00       	call   c135a2e0 <_raw_spin_lock>
c11860f7:	8b 45 ec             	mov    -0x14(%ebp),%eax
c11860fa:	8b 55 f0             	mov    -0x10(%ebp),%edx
c11860fd:	01 43 10             	add    %eax,0x10(%ebx)
c1186100:	89 d8                	mov    %ebx,%eax
c1186102:	11 53 14             	adc    %edx,0x14(%ebx)
c1186105:	8b 55 e8             	mov    -0x18(%ebp),%edx
c1186108:	c7 02 00 00 00 00    	movl   $0x0,(%edx)
c118610e:	e8 6d 41 1d 00       	call   c135a280 <_raw_spin_unlock>
c1186113:	8b 5d f4             	mov    -0xc(%ebp),%ebx
c1186116:	8b 75 f8             	mov    -0x8(%ebp),%esi
c1186119:	8b 7d fc             	mov    -0x4(%ebp),%edi
c118611c:	c9                   	leave  
c118611d:	c3                   	ret    
c118611e:	66 90                	xchg   %ax,%ax
c1186120:	8b 7d 08             	mov    0x8(%ebp),%edi
c1186123:	39 7d ec             	cmp    %edi,-0x14(%ebp)
c1186126:	73 c8                	jae    c11860f0 <__percpu_counter_add+0x80>
c1186128:	8b 4d 08             	mov    0x8(%ebp),%ecx
c118612b:	f7 d9                	neg    %ecx
c118612d:	89 ca                	mov    %ecx,%edx
c118612f:	c1 fa 1f             	sar    $0x1f,%edx
c1186132:	39 55 f0             	cmp    %edx,-0x10(%ebp)
c1186135:	7f 90                	jg     c11860c7 <__percpu_counter_add+0x57>
c1186137:	eb a7                	jmp    c11860e0 <__percpu_counter_add+0x70>
maintainer
#95 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 1 line · On Fri, 22 Oct 2010 00:58:14 +0200 Eric Dumazet <[email protected]> wrote:

Then 39 instructions in hot path (no lock taken)

Wow, your compiler sucks.

.globl __percpu_counter_add
.type __percpu_counter_add, @function
__percpu_counter_add:

	pushq	%rbp	#
	movq	%rsp, %rbp	#,
	pushq	%r13	#
	movq	%rdi, %r13	# fbc, fbc
	pushq	%r12	#
	pushq	%rbx	#
	pushq	%r10	#
	movq	96(%rdi), %rbx	# <variable>.counters, pcount
#APP
	movl %gs:cpu_number,%eax	# cpu_number, pfo_ret__
#NO_APP
	cltq
	addq	__per_cpu_offset(,%rax,8), %rbx	# __per_cpu_offset, pcount
	movslq	(%rbx),%rax	#* pcount, tmp70
	leaq	(%rax,%rsi), %r12	#, count
	movslq	%edx,%rax	# batch, batch
	cmpq	%rax, %r12	# batch, count
	jge	.L43	#,
	negl	%edx	# batch
	movslq	%edx,%rax	# batch, tmp74
	cmpq	%rax, %r12	# tmp74, count
	jg	.L45	#,
.L43:
	movq	%r13, %rdi	# fbc, D.14396
	call	_raw_spin_lock	#
	addq	%r12, 72(%r13)	# count, <variable>.count
	movq	%r13, %rdi	# fbc, D.14396
	movl	$0, (%rbx)	#,* pcount
	call	_raw_spin_unlock	#
	jmp	.L47	#
.L45:
	movl	%r12d, (%rbx)	# count,* pcount
.L47:
	popq	%r9	#
	popq	%rbx	#
	popq	%r12	#
	popq	%r13	#
	leave
	ret
#96 lore raw

Re: [PATCH] percpu_counter : add percpu_counter_add_fast()

Quoted 7 lines · Andrew Morton
Quoted 1 line · On Fri, 22 Oct 2010 00:58:14 +0200 Eric Dumazet <[email protected]> wrote:

Then 39 instructions in hot path (no lock taken)

Wow, your compiler sucks.

Actually my machines are mostly 32bit. Is it a problem ?

Compiler is OK, really.

percpu_counter() is misnamed, it should be named percpu_counter64() or something like that :-(

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#97 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

In reply to Eric Dumazet

(working in time-reversed oreder again :()

Quoted 3 lines · On Sat, 16 Oct 2010 11:31:15 +0200 Eric Dumazet <[email protected]> wrote:

Btw, I believe my previous patch against include/linux/percpu_counter.h was lost. Are you sure I am the right guy to work on percpu_counter infra ? If yes I can implement your inlined idea.

I don't know what "previous patch" you're referring to.

#98 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

Quoted 3 lines · Andrew Morton

(working in time-reversed oreder again :()

I don't know what "previous patch" you're referring to.

Nothing exciting, it was to update the comment that was slightly wrong about size of percpu_counter...

(percpu_counter: change inaccurate comment)

https://patchwork.kernel.org/patch/238441/

#99 lore raw

Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter

In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

include/linux/fs.h extern struct files_stat_struct files_stat; · line 411

Quoted 2 lines
include/linux/fs.h +2 -0
+extern int get_nr_inodes(void);
+extern int get_nr_inodes_unused(void);

These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.

We don't really need to export these anyway. We have two callers for each of them, and both are in the form of:

	/* approximate dirty inodes */
	nr_dirty_inodes = get_nr_inodes() - get_nr_inodes_unused();
	if (nr_dirty_inodes < 0)
		nr_dirty_inodes = 0;

which means we should just have a properly documented get_nr_dirty_inodes() helper, the rest can stay private to inode.c

#100 lore raw

PATCH 17/17 fs: Clean up inode reference counting

In reply to Dave Chinner

From: Nick Piggin <[email protected]>

Lots of filesystem code open codes the act of getting a reference to an inode. Factor the open coded inode lock, increment, unlock into a function iget(). Then rename __iget to iget_ilock so that nothing is directly incrementing the inode reference count for trivial operations.

---
 fs/affs/inode.c             |    4 +---
 fs/anon_inodes.c            |    4 +---
 fs/bfs/dir.c                |    4 +---
 fs/block_dev.c              |   14 +++-----------
 fs/btrfs/inode.c            |    4 +---
 fs/coda/dir.c               |    4 +---
 fs/drop_caches.c            |    2 +-
 fs/exofs/inode.c            |    4 +---
 fs/exofs/namei.c            |    4 +---
 fs/ext2/namei.c             |    4 +---
 fs/ext3/namei.c             |    4 +---
 fs/ext4/namei.c             |    4 +---
 fs/fs-writeback.c           |    6 +++---
 fs/gfs2/ops_inode.c         |    4 +---
 fs/hfsplus/dir.c            |    4 +---
 fs/inode.c                  |   18 +++++++++---------
 fs/jffs2/dir.c              |    8 ++------
 fs/jfs/jfs_txnmgr.c         |    4 +---
 fs/jfs/namei.c              |    4 +---
 fs/libfs.c                  |    4 +---
 fs/logfs/dir.c              |    4 +---
 fs/minix/namei.c            |    4 +---
 fs/namei.c                  |    7 ++-----
 fs/nfs/dir.c                |    4 +---
 fs/nfs/getroot.c            |    6 ++----
 fs/nfs/write.c              |    2 +-
 fs/nilfs2/namei.c           |    4 +---
 fs/notify/inode_mark.c      |    8 ++++----
 fs/ntfs/super.c             |    4 +---
 fs/ocfs2/namei.c            |    4 +---
 fs/quota/dquot.c            |    2 +-
 fs/reiserfs/namei.c         |    4 +---
 fs/sysv/namei.c             |    4 +---
 fs/ubifs/dir.c              |    4 +---
 fs/udf/namei.c              |    4 +---
 fs/ufs/namei.c              |    4 +---
 fs/xfs/linux-2.6/xfs_iops.c |    4 +---
 fs/xfs/xfs_inode.h          |    4 +---
 include/linux/fs.h          |   10 +++++++++-
 ipc/mqueue.c                |    7 ++-----
 kernel/futex.c              |    4 +---
 mm/shmem.c                  |    4 +---
 net/socket.c                |    4 +---
 43 files changed, 70 insertions(+), 144 deletions(-)
fs/affs/inode.c +1 -3
index cb9e773..f892be2 100644
@@ -388,9 +388,7 @@ affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s3
 		affs_adjust_checksum(inode_bh, block - be32_to_cpu(chain));
 		mark_buffer_dirty_inode(inode_bh, inode);
 		inode->i_nlink = 2;
-		spin_lock(&inode->i_lock);
-		inode->i_count++;
-		spin_unlock(&inode->i_lock);
+		iget(inode);
 	}
 	affs_fix_checksum(sb, bh);
 	mark_buffer_dirty_inode(bh, inode);
fs/anon_inodes.c +1 -3
index c50dc2a..517286c 100644
@@ -114,9 +114,7 @@ struct file *anon_inode_getfile(const char *name,
 	 * so we can avoid doing an igrab() and we can use an open-coded
 	 * atomic_inc().
 	 */
-	spin_lock(&anon_inode_inode->i_lock);
-	anon_inode_inode->i_count++;
-	spin_unlock(&anon_inode_inode->i_lock);
+	iget(anon_inode_inode);
 
 	path.dentry->d_op = &anon_inodefs_dentry_operations;
 	d_instantiate(path.dentry, anon_inode_inode);
fs/bfs/dir.c +1 -3
index d42fc72..0a42a91 100644
@@ -176,9 +176,7 @@ static int bfs_link(struct dentry *old, struct inode *dir,
 	inc_nlink(inode);
 	inode->i_ctime = CURRENT_TIME_SEC;
 	mark_inode_dirty(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	d_instantiate(new, inode);
 	mutex_unlock(&info->bfs_lock);
 	return 0;
fs/block_dev.c +3 -11
index 140451c..276a641 100644
@@ -550,11 +550,7 @@ EXPORT_SYMBOL(bdget);
  */
 struct block_device *bdgrab(struct block_device *bdev)
 {
-	struct inode *inode = bdev->bd_inode;
-
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(bdev->bd_inode);
 
 	return bdev;
 }
@@ -585,9 +581,7 @@ static struct block_device *bd_acquire(struct inode *inode)
 	spin_lock(&bdev_lock);
 	bdev = inode->i_bdev;
 	if (bdev) {
-		spin_lock(&inode->i_lock);
-		bdev->bd_inode->i_count++;
-		spin_unlock(&inode->i_lock);
+		bdgrab(bdev);
 		spin_unlock(&bdev_lock);
 		return bdev;
 	}
@@ -603,9 +597,7 @@ static struct block_device *bd_acquire(struct inode *inode)
 			 * So, we can access it via ->i_mapping always
 			 * without igrab().
 			 */
-			spin_lock(&inode->i_lock);
-			bdev->bd_inode->i_count++;
-			spin_unlock(&inode->i_lock);
+			iget(bdev->bd_inode);
 			inode->i_bdev = bdev;
 			inode->i_mapping = bdev->bd_inode->i_mapping;
 			list_add(&inode->i_devices, &bdev->bd_inodes);
fs/btrfs/inode.c +1 -3
index 7675f0c..6664ddf 100644
@@ -4769,9 +4769,7 @@ static int btrfs_link(struct dentry *old_dentry, struct inode *dir,
 	}
 
 	btrfs_set_trans_block_group(trans, dir);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	err = btrfs_add_nondir(trans, dentry, inode, 1, index);
 
fs/coda/dir.c +1 -3
index 2e52ad6..f97bc3b 100644
@@ -303,9 +303,7 @@ static int coda_link(struct dentry *source_de, struct inode *dir_inode,
 	}
 
 	coda_dir_update_mtime(dir_inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	d_instantiate(de, inode);
 	inc_nlink(inode);
 
fs/drop_caches.c +1 -1
index 0884447..af0b6da 100644
@@ -24,7 +24,7 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)
 			spin_unlock(&inode->i_lock);
 			continue;
 		}
-		__iget(inode);
+		iget_ilock(inode);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 		invalidate_mapping_pages(inode->i_mapping, 0, -1);
fs/exofs/inode.c +1 -3
index 3e7f967..fd0b9f2 100644
@@ -1156,9 +1156,7 @@ struct inode *exofs_new_inode(struct inode *dir, int mode)
 	/* increment the refcount so that the inode will still be around when we
 	 * reach the callback
 	 */
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	ios->done = create_done;
 	ios->private = inode;
fs/exofs/namei.c +1 -3
index 506778a..9bdc11f 100644
@@ -153,9 +153,7 @@ static int exofs_link(struct dentry *old_dentry, struct inode *dir,
 
 	inode->i_ctime = CURRENT_TIME;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	return exofs_add_nondir(dentry, inode);
 }
fs/ext2/namei.c +1 -3
index a5b9a54..499326b 100644
@@ -206,9 +206,7 @@ static int ext2_link (struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	err = ext2_add_link(dentry, inode);
 	if (!err) {
fs/ext3/namei.c +1 -3
index 4e3b5ff..3742e5a 100644
@@ -2260,9 +2260,7 @@ retry:
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inc_nlink(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	err = ext3_add_entry(handle, dentry, inode);
 	if (!err) {
fs/ext4/namei.c +1 -3
index 4dbb5e5..9752c00 100644
@@ -2312,9 +2312,7 @@ retry:
 
 	inode->i_ctime = ext4_current_time(inode);
 	ext4_inc_count(handle, inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	err = ext4_add_entry(handle, dentry, inode);
 	if (!err) {
fs/fs-writeback.c +3 -3
index 348cc18..9c10fdc 100644
@@ -294,7 +294,7 @@ static void inode_wait_for_writeback(struct inode *inode)
 
 /*
  * Write out an inode's dirty pages. Either the caller has ref on the inode
- * (either via __iget or via syscall against an fd) or the inode has
+ * (either via iget or via syscall against an fd) or the inode has
  * I_WILL_FREE set (via generic_forget_inode)
  *
  * If `wait' is set, wait on the writeout.
@@ -505,7 +505,7 @@ again:
 		}
 
 		BUG_ON(inode->i_state & I_FREEING);
-		__iget(inode);
+		iget_ilock(inode);
 		pages_skipped = wbc->pages_skipped;
 		writeback_single_inode(inode, wbc);
 		if (wbc->pages_skipped != pages_skipped) {
@@ -1075,7 +1075,7 @@ static void wait_sb_inodes(struct super_block *sb)
 			spin_unlock(&inode->i_lock);
 			continue;
 		}
-		__iget(inode);
+		iget_ilock(inode);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 		/*
fs/gfs2/ops_inode.c +1 -3
index 49c38dc..4f1719a 100644
@@ -253,9 +253,7 @@ out_parent:
 	gfs2_holder_uninit(ghs);
 	gfs2_holder_uninit(ghs + 1);
 	if (!error) {
-		spin_lock(&inode->i_lock);
-		inode->i_count++;
-		spin_unlock(&inode->i_lock);
+		iget(inode);
 		d_instantiate(dentry, inode);
 		mark_inode_dirty(inode);
 	}
fs/hfsplus/dir.c +1 -3
index 55fa48d..f163fab 100644
@@ -301,9 +301,7 @@ static int hfsplus_link(struct dentry *src_dentry, struct inode *dst_dir,
 
 	inc_nlink(inode);
 	hfsplus_instantiate(dst_dentry, inode, cnid);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	inode->i_ctime = CURRENT_TIME_SEC;
 	mark_inode_dirty(inode);
 	HFSPLUS_SB(sb).file_count++;
fs/inode.c +9 -9
index a91efab..57eb850 100644
@@ -533,7 +533,7 @@ again:
 		if (inode_has_buffers(inode) || inode->i_data.nrpages) {
 			list_move(&inode->i_list, &inode_unused);
 			spin_unlock(&wb_inode_list_lock);
-			__iget(inode);
+			iget_ilock(inode);
 			spin_unlock(&inode->i_lock);
 
 			if (remove_inode_buffers(inode))
@@ -591,7 +591,7 @@ static struct shrinker icache_shrinker = {
 static void __wait_on_freeing_inode(struct inode *inode);
 /*
  * Called with the inode lock held.
- * NOTE: we are not increasing the inode-refcount, you must call __iget()
+ * NOTE: we are not increasing the inode-refcount, you must call iget_ilock()
  * by hand after calling find_inode now! This simplifies iunique and won't
  * add any additional branch in the common code.
  */
@@ -855,7 +855,7 @@ static struct inode *get_new_inode(struct super_block *sb,
 		 * us. Use the old inode instead of the one we just
 		 * allocated.
 		 */
-		__iget(old);
+		iget_ilock(old);
 		spin_unlock(&old->i_lock);
 		destroy_inode(inode);
 		inode = old;
@@ -904,7 +904,7 @@ static struct inode *get_new_inode_fast(struct super_block *sb,
 		 * us. Use the old inode instead of the one we just
 		 * allocated.
 		 */
-		__iget(old);
+		iget_ilock(old);
 		spin_unlock(&old->i_lock);
 		destroy_inode(inode);
 		inode = old;
@@ -975,7 +975,7 @@ struct inode *igrab(struct inode *inode)
 
 	spin_lock(&inode->i_lock);
 	if (!(inode->i_state & (I_FREEING|I_WILL_FREE))) {
-		__iget(inode);
+		iget_ilock(inode);
 	} else {
 		/*
 		 * Handle the case where s_op->clear_inode is not been
@@ -1018,7 +1018,7 @@ static struct inode *ifind(struct super_block *sb,
 
 	inode = find_inode(sb, b, test, data);
 	if (inode) {
-		__iget(inode);
+		iget_ilock(inode);
 		spin_unlock(&inode->i_lock);
 		if (likely(wait))
 			wait_on_inode(inode);
@@ -1050,7 +1050,7 @@ static struct inode *ifind_fast(struct super_block *sb,
 
 	inode = find_inode_fast(sb, b, ino);
 	if (inode) {
-		__iget(inode);
+		iget_ilock(inode);
 		spin_unlock(&inode->i_lock);
 		wait_on_inode(inode);
 		return inode;
@@ -1239,7 +1239,7 @@ repeat:
 			return 0;
 		}
 		spin_unlock_bucket(b);
-		__iget(old);
+		iget_ilock(old);
 		spin_unlock(&old->i_lock);
 		wait_on_inode(old);
 		if (unlikely(!hlist_bl_unhashed(&old->i_hash))) {
@@ -1284,7 +1284,7 @@ repeat:
 			return 0;
 		}
 		spin_unlock_bucket(b);
-		__iget(old);
+		iget_ilock(old);
 		spin_unlock(&old->i_lock);
 		wait_on_inode(old);
 		if (unlikely(!hlist_bl_unhashed(&old->i_hash))) {
fs/jffs2/dir.c +2 -6
index 4d1bcfa..85f523b 100644
@@ -289,9 +289,7 @@ static int jffs2_link (struct dentry *old_dentry, struct inode *dir_i, struct de
 		mutex_unlock(&f->sem);
 		d_instantiate(dentry, old_dentry->d_inode);
 		dir_i->i_mtime = dir_i->i_ctime = ITIME(now);
-		spin_lock(&old_dentry->d_inode->i_lock);
-		old_dentry->d_inode->i_count++;
-		spin_unlock(&old_dentry->d_inode->i_lock);
+		iget(old_dentry->d_inode);
 	}
 	return ret;
 }
@@ -866,9 +864,7 @@ static int jffs2_rename (struct inode *old_dir_i, struct dentry *old_dentry,
 		printk(KERN_NOTICE "jffs2_rename(): Link succeeded, unlink failed (err %d). You now have a hard link\n", ret);
 		/* Might as well let the VFS know */
 		d_instantiate(new_dentry, old_dentry->d_inode);
-		spin_lock(&old_dentry->d_inode->i_lock);
-		old_dentry->d_inode->i_count++;
-		spin_unlock(&old_dentry->d_inode->i_lock);
+		iget(old_dentry->d_inode);
 		new_dir_i->i_mtime = new_dir_i->i_ctime = ITIME(now);
 		return ret;
 	}
fs/jfs/jfs_txnmgr.c +1 -3
index 820212f..d5764b7 100644
@@ -1279,9 +1279,7 @@ int txCommit(tid_t tid,		/* transaction identifier */
 	 * lazy commit thread finishes processing
 	 */
 	if (tblk->xflag & COMMIT_DELETE) {
-		spin_lock(&tblk->u.ip->i_lock);
-		tblk->u.ip->i_count++;
-		spin_unlock(&tblk->u.ip->i_lock);
+		iget(tblk->u.ip);
 		/*
 		 * Avoid a rare deadlock
 		 *
fs/jfs/namei.c +1 -3
index 3259008..7b1c8ea 100644
@@ -839,9 +839,7 @@ static int jfs_link(struct dentry *old_dentry,
 	ip->i_ctime = CURRENT_TIME;
 	dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	mark_inode_dirty(dir);
-	spin_lock(&ip->i_lock);
-	ip->i_count++;
-	spin_unlock(&ip->i_lock);
+	iget(ip);
 
 	iplist[0] = ip;
 	iplist[1] = dir;
fs/libfs.c +1 -3
index 0fa4dbe..1269ddb 100644
@@ -255,9 +255,7 @@ int simple_link(struct dentry *old_dentry, struct inode *dir, struct dentry *den
 
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	inc_nlink(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	dget(dentry);
 	d_instantiate(dentry, inode);
 	return 0;
fs/logfs/dir.c +1 -3
index 90eb51f..4a09080 100644
@@ -569,9 +569,7 @@ static int logfs_link(struct dentry *old_dentry, struct inode *dir,
 		return -EMLINK;
 
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	inode->i_nlink++;
 	mark_inode_dirty_sync(inode);
 
fs/minix/namei.c +1 -3
index a4a160f..0d5a830 100644
@@ -101,9 +101,7 @@ static int minix_link(struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	return add_nondir(dentry, inode);
 }
 
fs/namei.c +2 -5
index 817d6bb..e905661 100644
@@ -2290,11 +2290,8 @@ static long do_unlinkat(int dfd, const char __user *pathname)
 		if (nd.last.name[nd.last.len])
 			goto slashes;
 		inode = dentry->d_inode;
-		if (inode) {
-			spin_lock(&inode->i_lock);
-			inode->i_count++;
-			spin_unlock(&inode->i_lock);
-		}
+		if (inode)
+			iget(inode);
 		error = mnt_want_write(nd.path.mnt);
 		if (error)
 			goto exit2;
fs/nfs/dir.c +1 -3
index 375b6b5..c1435a9 100644
@@ -1580,9 +1580,7 @@ nfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry)
 	d_drop(dentry);
 	error = NFS_PROTO(dir)->link(inode, dir, &dentry->d_name);
 	if (error == 0) {
-		spin_lock(&inode->i_lock);
-		inode->i_count++;
-		spin_unlock(&inode->i_lock);
+		iget(inode);
 		d_add(dentry, inode);
 	}
 	return error;
fs/nfs/getroot.c +2 -4
index c6db37e..89e4ab8 100644
@@ -54,10 +54,8 @@ static int nfs_superblock_set_dummy_root(struct super_block *sb, struct inode *i
 			iput(inode);
 			return -ENOMEM;
 		}
-		/* Circumvent igrab(): we know the inode is not being freed */
-		spin_lock(&inode->i_lock);
-		inode->i_count++;
-		spin_unlock(&inode->i_lock);
+		/* We know the inode is not being freed */
+		iget(inode);
 		/*
 		 * Ensure that this dentry is invisible to d_find_alias().
 		 * Otherwise, it may be spliced into the tree by
fs/nfs/write.c +1 -1
index 129ebaa..346a9db 100644
@@ -390,7 +390,7 @@ static int nfs_inode_add_request(struct inode *inode, struct nfs_page *req)
 	error = radix_tree_insert(&nfsi->nfs_page_tree, req->wb_index, req);
 	BUG_ON(error);
 	if (!nfsi->npages) {
-		__iget(inode);
+		iget_ilock(inode);
 		if (nfs_have_delegation(inode, FMODE_WRITE))
 			nfsi->change_attr++;
 	}
fs/nilfs2/namei.c +1 -3
index 9e287ea..99dedf3 100644
@@ -219,9 +219,7 @@ static int nilfs_link(struct dentry *old_dentry, struct inode *dir,
 
 	inode->i_ctime = CURRENT_TIME;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	err = nilfs_add_nondir(dentry, inode);
 	if (!err)
fs/notify/inode_mark.c +4 -4
index e51d065..ecc8c18 100644
@@ -244,7 +244,7 @@ void fsnotify_unmount_inodes(struct list_head *list)
 
 		spin_lock(&inode->i_lock);
 		/*
-		 * We cannot __iget() an inode in state I_FREEING,
+		 * We cannot iget() an inode in state I_FREEING,
 		 * I_WILL_FREE, or I_NEW which is fine because by that point
 		 * the inode cannot have any associated watches.
 		 */
@@ -255,7 +255,7 @@ void fsnotify_unmount_inodes(struct list_head *list)
 
 		/*
 		 * If i_count is zero, the inode cannot have any watches and
-		 * doing an __iget/iput with MS_ACTIVE clear would actually
+		 * doing an iget/iput with MS_ACTIVE clear would actually
 		 * evict all inodes with zero i_count from icache which is
 		 * unnecessarily violent and may in fact be illegal to do.
 		 */
@@ -269,7 +269,7 @@ void fsnotify_unmount_inodes(struct list_head *list)
 
 		/* In case fsnotify_inode_delete() drops a reference. */
 		if (inode != need_iput_tmp) {
-			__iget(inode);
+			iget_ilock(inode);
 		} else
 			need_iput_tmp = NULL;
 		spin_unlock(&inode->i_lock);
@@ -279,7 +279,7 @@ void fsnotify_unmount_inodes(struct list_head *list)
 			spin_lock(&next_i->i_lock);
 			if (next_i->i_count &&
 			    !(next_i->i_state & (I_FREEING | I_WILL_FREE))) {
-				__iget(next_i);
+				iget_ilock(next_i);
 				need_iput = next_i;
 			}
 			spin_unlock(&next_i->i_lock);
fs/ntfs/super.c +1 -3
index 2e380ba..a0e45f3 100644
@@ -2930,9 +2930,7 @@ static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent)
 	}
 	if ((sb->s_root = d_alloc_root(vol->root_ino))) {
 		/* We increment i_count simulating an ntfs_iget(). */
-		spin_lock(&vol->root_ino->i_lock);
-		vol->root_ino->i_count++;
-		spin_unlock(&vol->root_ino->i_lock);
+		iget(vol->root_ino);
 		ntfs_debug("Exiting, status successful.");
 		/* Release the default upcase if it has no users. */
 		mutex_lock(&ntfs_lock);
fs/ocfs2/namei.c +1 -3
index 9c46feb..d2d972b 100644
@@ -741,9 +741,7 @@ static int ocfs2_link(struct dentry *old_dentry,
 		goto out_commit;
 	}
 
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	dentry->d_op = &ocfs2_dentry_ops;
 	d_instantiate(dentry, inode);
 
fs/quota/dquot.c +1 -1
index 69bc754..697bd6e 100644
@@ -910,7 +910,7 @@ static void add_dquot_ref(struct super_block *sb, int type)
 			reserved = 1;
 #endif
 
-		__iget(inode);
+		iget_ilock(inode);
 		spin_unlock(&inode->i_lock);
 		spin_unlock(&sb_inode_list_lock);
 
fs/reiserfs/namei.c +1 -3
index 1efebb2..1de28a0 100644
@@ -1156,9 +1156,7 @@ static int reiserfs_link(struct dentry *old_dentry, struct inode *dir,
 	inode->i_ctime = CURRENT_TIME_SEC;
 	reiserfs_update_sd(&th, inode);
 
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	d_instantiate(dentry, inode);
 	retval = journal_end(&th, dir->i_sb, jbegin_count);
 	reiserfs_write_unlock(dir->i_sb);
fs/sysv/namei.c +1 -3
index d63da9b..1620948 100644
@@ -126,9 +126,7 @@ static int sysv_link(struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	return add_nondir(dentry, inode);
 }
fs/ubifs/dir.c +1 -3
index c204b5c..7ccb249 100644
@@ -550,9 +550,7 @@ static int ubifs_link(struct dentry *old_dentry, struct inode *dir,
 
 	lock_2_inodes(dir, inode);
 	inc_nlink(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	inode->i_ctime = ubifs_current_time(inode);
 	dir->i_size += sz_change;
 	dir_ui->ui_size = dir->i_size;
fs/udf/namei.c +1 -3
index d8b0dc8..acd070b 100644
@@ -1101,9 +1101,7 @@ static int udf_link(struct dentry *old_dentry, struct inode *dir,
 	inc_nlink(inode);
 	inode->i_ctime = current_fs_time(inode->i_sb);
 	mark_inode_dirty(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	d_instantiate(dentry, inode);
 	unlock_kernel();
 
fs/ufs/namei.c +1 -3
index 8cbf920..f6e9232 100644
@@ -180,9 +180,7 @@ static int ufs_link (struct dentry * old_dentry, struct inode * dir,
 
 	inode->i_ctime = CURRENT_TIME_SEC;
 	inode_inc_link_count(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 
 	error = ufs_add_nondir(dentry, inode);
 	unlock_kernel();
fs/xfs/linux-2.6/xfs_iops.c +1 -3
index 332cdf5..8e9652f 100644
@@ -352,9 +352,7 @@ xfs_vn_link(
 	if (unlikely(error))
 		return -error;
 
-	spin_lock(&inode->i_lock);
-	inode->i_count++;
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	d_instantiate(dentry, inode);
 	return 0;
 }
fs/xfs/xfs_inode.h +1 -3
index 859628b..820a791 100644
@@ -481,10 +481,8 @@ void		xfs_mark_inode_dirty_sync(xfs_inode_t *);
 
 #define IHOLD(ip) \
 do { \
-	spin_lock(&VFS_I(ip)->i_lock); \
 	ASSERT(VFS_I(ip)->i_count > 0) ; \
-	VFS_I(ip)->i_count++; \
-	spin_unlock(&VFS_I(ip)->i_lock); \
+	iget(VFS_I(ip)); \
 	trace_xfs_ihold(ip, _THIS_IP_); \
 } while (0)
 
include/linux/fs.h +9 -1
index d2ee5d0..b07847c 100644
@@ -2394,12 +2394,20 @@ extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt);
 extern void save_mount_options(struct super_block *sb, char *options);
 extern void replace_mount_options(struct super_block *sb, char *options);
 
-static inline void __iget(struct inode *inode)
+static inline void iget_ilock(struct inode *inode)
 {
 	assert_spin_locked(&inode->i_lock);
+	BUG_ON(inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE));
 	inode->i_count++;
 }
 
+static inline void iget(struct inode *inode)
+{
+	spin_lock(&inode->i_lock);
+	iget_ilock(inode);
+	spin_unlock(&inode->i_lock);
+}
+
 static inline ino_t parent_ino(struct dentry *dentry)
 {
 	ino_t res;
ipc/mqueue.c +2 -5
index 7fe9efb..625dcfa 100644
@@ -768,11 +768,8 @@ SYSCALL_DEFINE1(mq_unlink, const char __user *, u_name)
 	}
 
 	inode = dentry->d_inode;
-	if (inode) {
-		spin_lock(&inode->i_lock);
-		inode->i_count++;
-		spin_unlock(&inode->i_lock);
-	}
+	if (inode)
+		iget(inode);
 	err = mnt_want_write(ipc_ns->mq_mnt);
 	if (err)
 		goto out_err;
kernel/futex.c +1 -3
index e9dfa00..b587a8b 100644
@@ -168,9 +168,7 @@ static void get_futex_key_refs(union futex_key *key)
 
 	switch (key->both.offset & (FUT_OFF_INODE|FUT_OFF_MMSHARED)) {
 	case FUT_OFF_INODE:
-		spin_lock(&key->shared.inode->i_lock);
-		key->shared.inode->i_count++;
-		spin_unlock(&key->shared.inode->i_lock);
+		iget(key->shared.inode);
 		break;
 	case FUT_OFF_MMSHARED:
 		atomic_inc(&key->private.mm->mm_count);
mm/shmem.c +1 -3
index b83b442..b24943a 100644
@@ -1903,9 +1903,7 @@ static int shmem_link(struct dentry *old_dentry, struct inode *dir, struct dentr
 	dir->i_size += BOGO_DIRENT_SIZE;
 	inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
 	inc_nlink(inode);
-	spin_lock(&inode->i_lock);
-	inode->i_count++;	/* New dentry reference */
-	spin_unlock(&inode->i_lock);
+	iget(inode);
 	dget(dentry);		/* Extra pinning count for the created dentry */
 	d_instantiate(dentry, inode);
 out:
net/socket.c +1 -3
index 5431af1..c5afc82 100644
@@ -377,9 +377,7 @@ static int sock_alloc_file(struct socket *sock, struct file **f, int flags)
 		  &socket_file_ops);
 	if (unlikely(!file)) {
 		/* drop dentry, keep inode */
-		spin_lock(&path.dentry->d_inode->i_lock);
-		path.dentry->d_inode->i_count++;
-		spin_unlock(&path.dentry->d_inode->i_lock);
+		iget(path.dentry->d_inode);
 		path_put(&path);
 		put_unused_fd(fd);
 		return -ENFILE;
Signature · 1 line
1.7.1
#101 lore raw

Re: [PATCH 17/17] fs: Clean up inode reference counting

Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have iget_locked. I think iref/iref_locked would be good enough names for the helpers.

One other things is that a lot of the current igrab users could use this helper instead - there's very few places where we look at inodes that don't already have a permanent reference when we try to acquire another one.

#102 lore raw

Re: [PATCH 17/17] fs: Clean up inode reference counting

Quoted 4 lines · Christoph Hellwig

Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have iget_locked.

It's a bit of a mess. We also have __iget. I don't care much about the name though so I'll see what you've chosen in future posts.

I'll see how it looks to move earlier in the series, shouldn't be a big problem.

#103 lore raw

Re: [PATCH 17/17] fs: Clean up inode reference counting

Quoted 11 lines · Nick Piggin
Quoted 4 lines · Christoph Hellwig

Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have iget_locked.

It's a bit of a mess. We also have __iget. I don't care much about the name though so I'll see what you've chosen in future posts.

I'll see how it looks to move earlier in the series, shouldn't be a big problem.

Dave has sorted out the naming mess quite nicely in his repost.

#104 lore raw

Re: [PATCH 17/17] fs: Clean up inode reference counting

Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Nick Piggin
Quoted 4 lines · Christoph Hellwig

Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have iget_locked.

Quoted 7 lines
Quoted 5 lines

It's a bit of a mess. We also have __iget. I don't care much about the name though so I'll see what you've chosen in future posts.

I'll see how it looks to move earlier in the series, shouldn't be a big problem.

Dave has sorted out the naming mess quite nicely in his repost.

Pity we can't use get/put but as I said, no big deal. I'll pick up those names.

maintainer
#105 lore raw

Re: [PATCH 17/17] fs: Clean up inode reference counting

In reply to Dave Chinner
Quoted 4 lines · On Wed, 29 Sep 2010 22:18:49 +1000 Dave Chinner <[email protected]> wrote:
include/linux/fs.h +1 -1
-static inline void __iget(struct inode *inode)
+static inline void iget_ilock(struct inode *inode)
 {
 	assert_spin_locked(&inode->i_lock);

include/linux/fs.h extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt); · line 2409

Quoted 10 lines
include/linux/fs.h +7 -0
+	BUG_ON(inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE));
 	inode->i_count++;
 }
 
+static inline void iget(struct inode *inode)
+{
+	spin_lock(&inode->i_lock);
+	iget_ilock(inode);
+	spin_unlock(&inode->i_lock);
+}

I suspect that for typical configs, these functions will generate amazing amounts of code.

Please measure this. We may decide to uninline both.

#106 lore raw
In reply to Dave Chinner
Quoted 6 lines · Dave Chinner

I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.

There's two obvious ordering issues: first the inode counters that you mentioned. I think this is esialy fixed by simply dropping both batches messing with it - we should have the new locks protecting it in places once inode_lock is dropped. The other one is the clean up inode reference counting patch, which sounds like it should be earlier in the series so that we have the helpers in place before touching all places that opencode an inode reference count increment.

Thanks forpicking up the work on this while Nick is travelling and some more comments on the individual patches will follow.

#107 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner

I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.

Quoted 7 lines

There's two obvious ordering issues: first the inode counters that you mentioned. I think this is esialy fixed by simply dropping both batches messing with it - we should have the new locks protecting it in places once inode_lock is dropped. The other one is the clean up inode reference counting patch, which sounds like it should be earlier in the series so that we have the helpers in place before touching all places that opencode an inode reference count increment.

Yeah, I thought you'd want that. ;) I'll reorder the iget helper patch to be the first in the series which should reduce churn quite a bit, and then convert both the nr_inode and nr_unused counters to be per-cpu before any of the other modifications and so they can be ignored completely in later patches.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#108 lore raw
In reply to Dave Chinner

Btw, another relatively easy patch from Nick's series is:

	http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git;a=commitdiff;h=950bbff18b19663abc2d260e352efd51fd79dbf7

although I think the second hunk could be done better by simply not calling __inode_add_to_lists but just factoring out a hunk to add the inode only to the i_sb_list which is all that's left after the patches in this series. And of course both hunks should be separate patches.

#109 lore raw
In reply to Dave Chinner

We have serious problems with 34.6 in a machine with ~11TiB xfs, with a lot of simultaneous IO, particularly hundreds of rm and a sync afterwards. Maybe they're related to these issues.

The machine is a file server (almost all via http/apache) and has several thousand connections all the time. It behaves quite well for at most 4 days; from then on kswapd's start appearing on the display of top consuming ever increasing percentages of cpu. This is no problem, the machine has 16 nearly idle cores. However, after about 5-7 days there's an abrupt transition: in about 30s the load goes to several thousand, apache shows up consuming all possible cpu and downloads nearly stop. I have to reboot the machine to get service back. It manages to unmount the filesystems and reboot properly.

Stopping/restarting apache restores the situation but only for a short while; after about 2-3h the problem reappears. That's why I have to reboot.

With 35.6 the behaviour seems to have changed: now often CONFIG_DETECT_HUNG_TASK produces this kind of call trace in the log:

[<ffffffff81098578>] ? igrab+0x10/0x30 [<ffffffff811160fe>] ? xfs_sync_inode_valid+0x4c/0x76 [<ffffffff81116241>] ? xfs_sync_inode_data+0x1b/0xa8 [<ffffffff811163e0>] ? xfs_inode_ag_walk+0x96/0xe4 [<ffffffff811163dd>] ? xfs_inode_ag_walk+0x93/0xe4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff81116495>] ? xfs_inode_ag_iterator+0x67/0xc4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff81116712>] ? xfs_sync_data+0x22/0x42 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff8111678b>] ? xfs_quiesce_data+0x2b/0x94 [<ffffffff81113f03>] ? xfs_fs_sync_fs+0x2d/0xd7 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff810a48c4>] ? __sync_filesystem+0x62/0x7b [<ffffffff8108993e>] ? iterate_supers+0x60/0x9d [<ffffffff810a493a>] ? sys_sync+0x3f/0x53 [<ffffffff81001dab>] ? system_call_fastpath+0x16/0x1b

It doesn't seem to cause service disruption (at least the flux graphs don't show drops). I didn't see it happen while I was watching so it may be that service degrades for short intervals. Uptime with 35.6 is only 3d8h so it's still not sure that the breakdown of 34.6 is gone but kswapd's cpu usages are very small, less than with 34.6 for a similar uptime. There are only 2 filesystems, and the big one has 256 AGs. They're not mounted with delaylog.

#110 lore raw
Quoted 40 lines · Carlos Carvalho

We have serious problems with 34.6 in a machine with ~11TiB xfs, with a lot of simultaneous IO, particularly hundreds of rm and a sync afterwards. Maybe they're related to these issues.

The machine is a file server (almost all via http/apache) and has several thousand connections all the time. It behaves quite well for at most 4 days; from then on kswapd's start appearing on the display of top consuming ever increasing percentages of cpu. This is no problem, the machine has 16 nearly idle cores. However, after about 5-7 days there's an abrupt transition: in about 30s the load goes to several thousand, apache shows up consuming all possible cpu and downloads nearly stop. I have to reboot the machine to get service back. It manages to unmount the filesystems and reboot properly.

Stopping/restarting apache restores the situation but only for a short while; after about 2-3h the problem reappears. That's why I have to reboot.

With 35.6 the behaviour seems to have changed: now often CONFIG_DETECT_HUNG_TASK produces this kind of call trace in the log:

[<ffffffff81098578>] ? igrab+0x10/0x30 [<ffffffff811160fe>] ? xfs_sync_inode_valid+0x4c/0x76 [<ffffffff81116241>] ? xfs_sync_inode_data+0x1b/0xa8 [<ffffffff811163e0>] ? xfs_inode_ag_walk+0x96/0xe4 [<ffffffff811163dd>] ? xfs_inode_ag_walk+0x93/0xe4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff81116495>] ? xfs_inode_ag_iterator+0x67/0xc4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff81116712>] ? xfs_sync_data+0x22/0x42 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff8111678b>] ? xfs_quiesce_data+0x2b/0x94 [<ffffffff81113f03>] ? xfs_fs_sync_fs+0x2d/0xd7 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff810a48c4>] ? __sync_filesystem+0x62/0x7b [<ffffffff8108993e>] ? iterate_supers+0x60/0x9d [<ffffffff810a493a>] ? sys_sync+0x3f/0x53 [<ffffffff81001dab>] ? system_call_fastpath+0x16/0x1b

Quoted 7 lines

It doesn't seem to cause service disruption (at least the flux graphs don't show drops). I didn't see it happen while I was watching so it may be that service degrades for short intervals. Uptime with 35.6 is only 3d8h so it's still not sure that the breakdown of 34.6 is gone but kswapd's cpu usages are very small, less than with 34.6 for a similar uptime. There are only 2 filesystems, and the big one has 256 AGs. They're not mounted with delaylog.

Apply this:

http://www.oss.sgi.com/archives/xfs/2010-10/msg00000.html

And in future, can you please report bugs in a new thread to the appropriate lists ([email protected]), not as a reply to a completely unrelated development thread....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]