Skip to contentSkip to conversation

vfsmount scaling and other bits

PATCH In review [email protected] 7 participants 22 messages Oct 15, 2009 – Oct 19, 2009

Subsystems (8)
lore
Share & tools
lore
#1 lore raw

PATCH 1/6 fs: invalidate sb->s_bdev on remount,ro

In reply to [email protected]

Fixes a problem reported by "Jorge Boncompte [DTI2]" <[email protected]> who is seeing corruption trying to snapshot a minix filesystem image. Some filesystems modify their metadata via a path other than the bdev buffer cache (eg. they may use a private linear mapping for their metadata, or implement directories in pagecache, etc). Also, file data modifications usually go to the bdev via their own mappings.

These updates are not coherent with buffercache IO (eg. via /dev/bdev) and never have been. However there could be a reasonable expectation that after a mount -oremount,ro operation then the buffercache should subsequently be coherent with previous filesystem modifications.

So invalidate the bdev mappings on a remount,ro operation to provide a coherency point.

The problem was exposed when we switched the old rd to brd because old rd didn't really function like a normal block device and updates to rd via mappings other than the buffercache would still end up going into its buffercache. But the same problem has always affected other "normal" block devices, including loop.

---
 fs/super.c |   16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
fs/super.c +13 -3
Index: linux-2.6/fs/super.c
===================================================================
@@ -568,7 +568,7 @@ out:
 int do_remount_sb(struct super_block *sb, int flags, void *data, int force)
 {
 	int retval;
-	int remount_rw;
+	int remount_rw, remount_ro;
 
 	if (sb->s_frozen != SB_UNFROZEN)
 		return -EBUSY;
@@ -583,9 +583,12 @@ int do_remount_sb(struct super_block *sb
 	shrink_dcache_sb(sb);
 	sync_filesystem(sb);
 
+	remount_ro = (flags & MS_RDONLY) && !(sb->s_flags & MS_RDONLY);
+	remount_rw = !(flags & MS_RDONLY) && (sb->s_flags & MS_RDONLY);
+
 	/* If we are remounting RDONLY and current sb is read/write,
 	   make sure there are no rw files opened */
-	if ((flags & MS_RDONLY) && !(sb->s_flags & MS_RDONLY)) {
+	if (remount_ro) {
 		if (force)
 			mark_files_ro(sb);
 		else if (!fs_may_remount_ro(sb))
@@ -594,7 +597,6 @@ int do_remount_sb(struct super_block *sb
 		if (retval < 0 && retval != -ENOSYS)
 			return -EBUSY;
 	}
-	remount_rw = !(flags & MS_RDONLY) && (sb->s_flags & MS_RDONLY);
 
 	if (sb->s_op->remount_fs) {
 		retval = sb->s_op->remount_fs(sb, &flags, data);
@@ -604,6 +606,14 @@ int do_remount_sb(struct super_block *sb
 	sb->s_flags = (sb->s_flags & ~MS_RMT_MASK) | (flags & MS_RMT_MASK);
 	if (remount_rw)
 		vfs_dq_quota_on_remount(sb);
+	 /* Some filesystems modify their metadata via some other path
+	    than the bdev buffer cache (eg. use a private mapping, or
+	    directories in pagecache, etc). Also file data modifications
+	    go via their own mappings. So If we try to mount readonly
+	    then copy the filesystem from bdev, we could get stale data,
+	    so invalidate it to give a best effort at coherency. */
+	if (remount_ro && sb->s_bdev)
+		invalidate_bdev(sb->s_bdev);
 	return 0;
 }
 
#2 lore raw

PATCH 2/6 fs: no games with DCACHE_UNHASHED

In reply to [email protected]

(this is in -mm)

Filesystems outside the regular namespace do not have to clear DCACHE_UNHASHED in order to have a working /proc/$pid/fd/XXX. Nothing in proc prevents the fd link from being used if its dentry is not in the hash.

Also, it does not get put into the dcache hash if DCACHE_UNHASHED is clear; that depends on the filesystem calling d_add or d_rehash.

So delete the misleading comments and needless code.

---
 fs/anon_inodes.c |   16 ----------------
 fs/pipe.c        |   18 ------------------
 net/socket.c     |   19 -------------------
 3 files changed, 53 deletions(-)
fs/pipe.c +0 -18
Index: linux-2.6/fs/pipe.c
===================================================================
@@ -887,17 +887,6 @@ void free_pipe_info(struct inode *inode)
 }
 
 static struct vfsmount *pipe_mnt __read_mostly;
-static int pipefs_delete_dentry(struct dentry *dentry)
-{
-	/*
-	 * At creation time, we pretended this dentry was hashed
-	 * (by clearing DCACHE_UNHASHED bit in d_flags)
-	 * At delete time, we restore the truth : not hashed.
-	 * (so that dput() can proceed correctly)
-	 */
-	dentry->d_flags |= DCACHE_UNHASHED;
-	return 0;
-}
 
 /*
  * pipefs_dname() is called from d_path().
@@ -909,7 +898,6 @@ static char *pipefs_dname(struct dentry
 }
 
 static const struct dentry_operations pipefs_dentry_operations = {
-	.d_delete	= pipefs_delete_dentry,
 	.d_dname	= pipefs_dname,
 };
 
@@ -969,12 +957,6 @@ struct file *create_write_pipe(int flags
 		goto err_inode;
 
 	dentry->d_op = &pipefs_dentry_operations;
-	/*
-	 * We dont want to publish this dentry into global dentry hash table.
-	 * We pretend dentry is already hashed, by unsetting DCACHE_UNHASHED
-	 * This permits a working /proc/$pid/fd/XXX on pipes
-	 */
-	dentry->d_flags &= ~DCACHE_UNHASHED;
 	d_instantiate(dentry, inode);
 
 	err = -ENFILE;
net/socket.c +0 -19
Index: linux-2.6/net/socket.c
===================================================================
@@ -306,18 +306,6 @@ static struct file_system_type sock_fs_t
 	.kill_sb =	kill_anon_super,
 };
 
-static int sockfs_delete_dentry(struct dentry *dentry)
-{
-	/*
-	 * At creation time, we pretended this dentry was hashed
-	 * (by clearing DCACHE_UNHASHED bit in d_flags)
-	 * At delete time, we restore the truth : not hashed.
-	 * (so that dput() can proceed correctly)
-	 */
-	dentry->d_flags |= DCACHE_UNHASHED;
-	return 0;
-}
-
 /*
  * sockfs_dname() is called from d_path().
  */
@@ -328,7 +316,6 @@ static char *sockfs_dname(struct dentry
 }
 
 static const struct dentry_operations sockfs_dentry_operations = {
-	.d_delete = sockfs_delete_dentry,
 	.d_dname  = sockfs_dname,
 };
 
@@ -377,12 +364,6 @@ static int sock_attach_fd(struct socket
 		return -ENOMEM;
 
 	dentry->d_op = &sockfs_dentry_operations;
-	/*
-	 * We dont want to push this dentry into global dentry hash table.
-	 * We pretend dentry is already hashed, by unsetting DCACHE_UNHASHED
-	 * This permits a working /proc/$pid/fd/XXX on sockets
-	 */
-	dentry->d_flags &= ~DCACHE_UNHASHED;
 	d_instantiate(dentry, SOCK_INODE(sock));
 
 	sock->file = file;
fs/anon_inodes.c +0 -16
Index: linux-2.6/fs/anon_inodes.c
===================================================================
@@ -35,24 +35,11 @@ static int anon_inodefs_get_sb(struct fi
 			     mnt);
 }
 
-static int anon_inodefs_delete_dentry(struct dentry *dentry)
-{
-	/*
-	 * We faked vfs to believe the dentry was hashed when we created it.
-	 * Now we restore the flag so that dput() will work correctly.
-	 */
-	dentry->d_flags |= DCACHE_UNHASHED;
-	return 1;
-}
-
 static struct file_system_type anon_inode_fs_type = {
 	.name		= "anon_inodefs",
 	.get_sb		= anon_inodefs_get_sb,
 	.kill_sb	= kill_anon_super,
 };
-static const struct dentry_operations anon_inodefs_dentry_operations = {
-	.d_delete	= anon_inodefs_delete_dentry,
-};
 
 /*
  * nop .set_page_dirty method so that people can use .page_mkwrite on
@@ -117,9 +104,6 @@ struct file *anon_inode_getfile(const ch
 	 */
 	atomic_inc(&anon_inode_inode->i_count);
 
-	dentry->d_op = &anon_inodefs_dentry_operations;
-	/* Do not publish this dentry inside the global dentry hash table */
-	dentry->d_flags &= ~DCACHE_UNHASHED;
 	d_instantiate(dentry, anon_inode_inode);
 
 	error = -ENFILE;
#0 lore raw

cover vfsmount scaling and other bits

Hi Al,

Finally got around to some numbers and a few comments with vfsmount scaling. Patch looks a bit nicer now, and numbers aren't too bad I think.

I also broke the brlock stuff into its own header... it's a bit ugly but I couldn't see a nice way to do the per-CPU stuff without adding indirection and memory allocation of dynamic per-CPU allocations. Anyone like to yell at me for this, or suggest a better approach?

I also put some other reasonable looking bits and pieces in there which have filtered to the bottom of my vfs stack, as there are some clashes otherwise. If you want me to stop sending them to you, could you merge them or tell me why I'm an idiot, please?

Thanks,
Nick

#10 lore raw

PATCH 3/6 fs: dcache remove d_mounted

Rather than keep a d_mounted count in the dentry (which is only used to speculatively take a look in the mount hash table if it is non-zero), set a dentry flag instead. The flag can be cleared by checking the hash table to see if there are any mounts left. It is not time critical because it is performed at detach time.

This saves 4 bytes on 32-bit, nothing on 64-bit but it does provide a hole I might use later (and some configs have larger than 32-bit spinlocks which might make use of the hole).

Autofs4 conversion and changelog by Ian Kent <[email protected]>:
In autofs4, when expring direct (or offset) mounts we need to ensure that we block user path walks into the autofs mount, which is covered by another mount. To do this we clear the mounted status so that follows stop before walking into the mount and are essentially blocked until the expire is completed. The automount daemon still finds the correct dentry for the umount due to the follow mount logic in fs/autofs4/root.c:autofs4_follow_link(), which is set as an inode operation for direct and offset mounts only and is called following the lookup that stopped at the covered mount.

At the end of the expire the covering mount probably has gone away so the mounted status need not be restored. But we need to check this and only restore the mounted status if the expire failed.

---
 fs/autofs4/expire.c    |   13 +++++++++++--
 fs/dcache.c            |    1 -
 fs/namespace.c         |   17 ++++++++++++++---
 include/linux/dcache.h |   42 +++++++++++++++++++++---------------------
 4 files changed, 46 insertions(+), 27 deletions(-)
fs/dcache.c +0 -1
Index: linux-2.6/fs/dcache.c
===================================================================
@@ -947,7 +947,6 @@ struct dentry *d_alloc(struct dentry * p
 	dentry->d_sb = NULL;
 	dentry->d_op = NULL;
 	dentry->d_fsdata = NULL;
-	dentry->d_mounted = 0;
 	INIT_HLIST_NODE(&dentry->d_hash);
 	INIT_LIST_HEAD(&dentry->d_lru);
 	INIT_LIST_HEAD(&dentry->d_subdirs);
fs/namespace.c +14 -3
Index: linux-2.6/fs/namespace.c
===================================================================
@@ -467,6 +467,15 @@ static void __touch_mnt_namespace(struct
 	}
 }
 
+static void dentry_reset_mounted(struct vfsmount *mnt, struct dentry *dentry)
+{
+	if (!__lookup_mnt(mnt, dentry, 0)) {
+		spin_lock(&dentry->d_lock);
+		dentry->d_flags &= ~DCACHE_MOUNTED;
+		spin_unlock(&dentry->d_lock);
+	}
+}
+
 static void detach_mnt(struct vfsmount *mnt, struct path *old_path)
 {
 	old_path->dentry = mnt->mnt_mountpoint;
@@ -475,7 +484,7 @@ static void detach_mnt(struct vfsmount *
 	mnt->mnt_mountpoint = mnt->mnt_root;
 	list_del_init(&mnt->mnt_child);
 	list_del_init(&mnt->mnt_hash);
-	old_path->dentry->d_mounted--;
+	dentry_reset_mounted(old_path->mnt, old_path->dentry);
 }
 
 void mnt_set_mountpoint(struct vfsmount *mnt, struct dentry *dentry,
@@ -483,7 +492,9 @@ void mnt_set_mountpoint(struct vfsmount
 {
 	child_mnt->mnt_parent = mntget(mnt);
 	child_mnt->mnt_mountpoint = dget(dentry);
-	dentry->d_mounted++;
+	spin_lock(&dentry->d_lock);
+	dentry->d_flags |= DCACHE_MOUNTED;
+	spin_unlock(&dentry->d_lock);
 }
 
 static void attach_mnt(struct vfsmount *mnt, struct path *path)
@@ -1015,7 +1026,7 @@ void umount_tree(struct vfsmount *mnt, i
 		list_del_init(&p->mnt_child);
 		if (p->mnt_parent != p) {
 			p->mnt_parent->mnt_ghosts++;
-			p->mnt_mountpoint->d_mounted--;
+			dentry_reset_mounted(p->mnt_parent, p->mnt_mountpoint);
 		}
 		change_mnt_propagation(p, MS_PRIVATE);
 	}
include/linux/dcache.h +21 -21

72 lines not on this page · raw message

fs/autofs4/expire.c +11 -2
Index: linux-2.6/fs/autofs4/expire.c
===================================================================
@@ -276,7 +276,9 @@ struct dentry *autofs4_expire_direct(str
 		struct autofs_info *ino = autofs4_dentry_ino(root);
 		if (d_mountpoint(root)) {
 			ino->flags |= AUTOFS_INF_MOUNTPOINT;
-			root->d_mounted--;
+			spin_lock(&root->d_lock);
+			root->d_flags &= ~DCACHE_MOUNTED;
+			spin_unlock(&root->d_lock);
 		}
 		ino->flags |= AUTOFS_INF_EXPIRING;
 		init_completion(&ino->expire_complete);
@@ -499,7 +501,14 @@ int autofs4_do_expire_multi(struct super
 
 		spin_lock(&sbi->fs_lock);
 		if (ino->flags & AUTOFS_INF_MOUNTPOINT) {
-			sb->s_root->d_mounted++;
+			spin_lock(&sb->s_root->d_lock);
+			/*
+			 * If we haven't been expired away, then reset
+			 * mounted status.
+			 */
+			if (mnt->mnt_parent != mnt)
+				sb->s_root->d_flags |= DCACHE_MOUNTED;
+			spin_unlock(&sb->s_root->d_lock);
 			ino->flags &= ~AUTOFS_INF_MOUNTPOINT;
 		}
 		ino->flags &= ~AUTOFS_INF_EXPIRING;
#21 lore raw

PATCH 6/6 fs: scale mntget/mntput

In reply to [email protected]

Improve scalability of mntget/mntput by using per-cpu counters protected by the reader side of the brlock vfsmount_lock. MNT_MOUNTED in mnt_flags keeps track of whether the vfsmount is actually attached to the tree so we can shortcut the now-expensive refcount check in mntput (just decrement the count because we know there must be at least one ref left).

No extra atomics in the common case because atomic mnt refcount is now replaced with per-CPU spinlock. Code will be bigger and more complex however. With the previous per-cpu locking patch, mount lookups and common case refcounting are now per-cpu and should be ideally scalable. path lookups (and hence path_get/path_put) within the same vfsmount should now be more scalable, however this will often be hidden by dcache_lock on final dput, and d_lock on common path elements (eg. cwd or root dentry).

---
 fs/libfs.c            |    1 
 fs/namespace.c        |  151 +++++++++++++++++++++++++++++++++++++++++++++-----
 fs/pnode.c            |    4 -
 include/linux/mount.h |   34 ++++-------
 4 files changed, 152 insertions(+), 38 deletions(-)
fs/namespace.c +136 -15
Index: linux-2.6/fs/namespace.c
===================================================================
@@ -138,6 +138,61 @@ void mnt_release_group_id(struct vfsmoun
 	mnt->mnt_group_id = 0;
 }
 
+/*
+ * vfsmount lock must be held for read
+ */
+static inline void add_mnt_count(struct vfsmount *mnt, int n)
+{
+#ifdef CONFIG_SMP
+	(*per_cpu_ptr(mnt->mnt_count, smp_processor_id())) += n;
+#else
+	mnt->mnt_count += n;
+#endif
+}
+
+/*
+ * vfsmount lock must be held for read
+ */
+static inline void inc_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+	(*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))++;
+#else
+	mnt->mnt_count++;
+#endif
+}
+
+/*
+ * vfsmount lock must be held for read
+ */
+static inline void dec_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+	(*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))--;
+#else
+	mnt->mnt_count--;
+#endif
+}
+
+/*
+ * vfsmount lock must be held for write
+ */
+unsigned int count_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+	unsigned int count = 0;
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		count += *per_cpu_ptr(mnt->mnt_count, cpu);
+	}
+
+	return count;
+#else
+	return mnt->mnt_count;
+#endif
+}
+
 struct vfsmount *alloc_vfsmnt(const char *name)
 {
 	struct vfsmount *mnt = kmem_cache_zalloc(mnt_cache, GFP_KERNEL);
@@ -154,7 +209,13 @@ struct vfsmount *alloc_vfsmnt(const char
 				goto out_free_id;
 		}
 
-		atomic_set(&mnt->mnt_count, 1);
+#ifdef CONFIG_SMP
+		mnt->mnt_count = alloc_percpu(int);
+		if (!mnt->mnt_count)
+			goto out_free_devname;
+#else
+		mnt->mnt_count = 0;
+#endif
 		INIT_LIST_HEAD(&mnt->mnt_hash);
 		INIT_LIST_HEAD(&mnt->mnt_child);
 		INIT_LIST_HEAD(&mnt->mnt_mounts);
@@ -166,14 +227,19 @@ struct vfsmount *alloc_vfsmnt(const char
 #ifdef CONFIG_SMP
 		mnt->mnt_writers = alloc_percpu(int);
 		if (!mnt->mnt_writers)
-			goto out_free_devname;
+			goto out_free_mntcount;
 #else
 		mnt->mnt_writers = 0;
 #endif
+		preempt_disable();
+		inc_mnt_count(mnt);
+		preempt_enable();
 	}
 	return mnt;
 
 #ifdef CONFIG_SMP
+out_free_mntcount:
+	free_percpu(mnt->mnt_count);
 out_free_devname:
 	kfree(mnt->mnt_devname);
 #endif
@@ -512,6 +578,8 @@ static void detach_mnt(struct vfsmount *
 	list_del_init(&mnt->mnt_child);
 	list_del_init(&mnt->mnt_hash);
 	dentry_reset_mounted(old_path->mnt, old_path->dentry);
+	WARN_ON(!(mnt->mnt_flags & MNT_MOUNTED));
+	mnt->mnt_flags &= ~MNT_MOUNTED;
 }
 
 /*
@@ -536,6 +604,8 @@ static void attach_mnt(struct vfsmount *
 	list_add_tail(&mnt->mnt_hash, mount_hashtable +
 			hash(path->mnt, path->dentry));
 	list_add_tail(&mnt->mnt_child, &path->mnt->mnt_mounts);
+	WARN_ON(mnt->mnt_flags & MNT_MOUNTED);
+	mnt->mnt_flags |= MNT_MOUNTED;
 }
 
 /*
@@ -558,6 +628,8 @@ static void commit_tree(struct vfsmount
 	list_add_tail(&mnt->mnt_hash, mount_hashtable +
 				hash(parent, mnt->mnt_mountpoint));
 	list_add_tail(&mnt->mnt_child, &parent->mnt_mounts);
+	WARN_ON(mnt->mnt_flags & MNT_MOUNTED);
+	mnt->mnt_flags |= MNT_MOUNTED;
 	touch_mnt_namespace(n);
 }
 
@@ -652,6 +724,9 @@ static inline void __mntput(struct vfsmo
 	/*
 	 * atomic_dec_and_lock() used to deal with ->mnt_count decrements
 	 * provides barriers, so count_mnt_writers() below is safe.  AV
+	 * XXX: hmm, we no longer have an atomic_dec_and_lock, so load of
+	 * mnt_writers may be moved up into the vfsmount lock critical
+	 * section? Do we need an smp_mb()?
 	 */
 	WARN_ON(count_mnt_writers(mnt));
 	dput(mnt->mnt_root);
@@ -661,44 +736,79 @@ static inline void __mntput(struct vfsmo
 
 void mntput_no_expire(struct vfsmount *mnt)
 {
-repeat:
-	if (!vfsmount_atomic_dec_and_wlock(&mnt->mnt_count))
+	if (likely(mnt->mnt_flags & MNT_MOUNTED)) {
+		vfsmount_rlock();
+		if (unlikely(!mnt->mnt_flags & MNT_MOUNTED)) {
+			vfsmount_runlock();
+			goto repeat;
+		}
+		dec_mnt_count(mnt);
+		vfsmount_runlock();
+
 		return;
+	}
 
+repeat:
+	vfsmount_wlock();
+	BUG_ON(mnt->mnt_flags & MNT_MOUNTED);
+	dec_mnt_count(mnt);
+	if (count_mnt_count(mnt)) {
+		vfsmount_wunlock();
+		return;
+	}
 	if (likely(!mnt->mnt_pinned)) {
 		vfsmount_wunlock();
 		__mntput(mnt);
 		return;
 	}
-	atomic_add(mnt->mnt_pinned + 1, &mnt->mnt_count);
+	add_mnt_count(mnt, mnt->mnt_pinned + 1);
 	mnt->mnt_pinned = 0;
 	vfsmount_wunlock();
 	acct_auto_close_mnt(mnt);
 	security_sb_umount_close(mnt);
 	goto repeat;
 }
-
 EXPORT_SYMBOL(mntput_no_expire);
 
+void mntput(struct vfsmount *mnt)
+{
+	if (mnt) {
+		/* avoid cacheline pingpong, hope gcc doesn't get "smart" */
+		if (unlikely(mnt->mnt_expiry_mark))
+			mnt->mnt_expiry_mark = 0;
+		mntput_no_expire(mnt);
+	}
+}
+EXPORT_SYMBOL(mntput);
+
+struct vfsmount *mntget(struct vfsmount *mnt)
+{
+	if (mnt) {
+		preempt_disable();
+		inc_mnt_count(mnt);
+		preempt_enable();
+	}
+	return mnt;
+}
+EXPORT_SYMBOL(mntget);
+
 void mnt_pin(struct vfsmount *mnt)
 {
 	vfsmount_wlock();
 	mnt->mnt_pinned++;
 	vfsmount_wunlock();
 }
-
 EXPORT_SYMBOL(mnt_pin);
 
 void mnt_unpin(struct vfsmount *mnt)
 {
 	vfsmount_wlock();
 	if (mnt->mnt_pinned) {
-		atomic_inc(&mnt->mnt_count);
+		inc_mnt_count(mnt);
 		mnt->mnt_pinned--;
 	}
 	vfsmount_wunlock();
 }
-
 EXPORT_SYMBOL(mnt_unpin);
 
 static inline void mangle(struct seq_file *m, const char *s)
@@ -979,12 +1089,13 @@ int may_umount_tree(struct vfsmount *mnt
 	int minimum_refs = 0;
 	struct vfsmount *p;
 
-	vfsmount_rlock();
+	/* write lock needed for count_mnt_count */
+	vfsmount_wlock();
 	for (p = mnt; p; p = next_mnt(p, mnt)) {
-		actual_refs += atomic_read(&p->mnt_count);
+		actual_refs += count_mnt_count(p);
 		minimum_refs += 2;
 	}
-	vfsmount_runlock();
+	vfsmount_wunlock();
 
 	if (actual_refs > minimum_refs)
 		return 0;
@@ -1011,10 +1122,10 @@ int may_umount(struct vfsmount *mnt)
 {
 	int ret = 1;
 
-	vfsmount_rlock();
+	vfsmount_wlock();
 	if (propagate_mount_busy(mnt, 2))
 		ret = 0;
-	vfsmount_runlock();
+	vfsmount_wunlock();
 
 	return ret;
 }
@@ -1065,6 +1176,8 @@ void umount_tree(struct vfsmount *mnt, i
 		__touch_mnt_namespace(p->mnt_ns);
 		p->mnt_ns = NULL;
 		list_del_init(&p->mnt_child);
+		WARN_ON(!(p->mnt_flags & MNT_MOUNTED));
+		p->mnt_flags &= ~MNT_MOUNTED;
 		if (p->mnt_parent != p) {
 			p->mnt_parent->mnt_ghosts++;
 			dentry_reset_mounted(p->mnt_parent, p->mnt_mountpoint);
@@ -1096,8 +1209,16 @@ static int do_umount(struct vfsmount *mn
 		    flags & (MNT_FORCE | MNT_DETACH))
 			return -EINVAL;
 
-		if (atomic_read(&mnt->mnt_count) != 2)
+		/*
+		 * probably don't strictly need the lock here if we examined
+		 * all race cases, but it's a slowpath.
+		 */
+		vfsmount_wlock();
+		if (count_mnt_count(mnt) != 2) {
+			vfsmount_wlock();
 			return -EBUSY;
+		}
+		vfsmount_wunlock();
 
 		if (!xchg(&mnt->mnt_expiry_mark, 1))
 			return -EAGAIN;
include/linux/mount.h +13 -21
Index: linux-2.6/include/linux/mount.h
===================================================================
@@ -32,11 +32,13 @@ struct mnt_namespace;
 
 #define MNT_SHRINKABLE	0x100
 #define MNT_WRITE_HOLD	0x200
+#define MNT_MOUNTED	0x400
 
 #define MNT_SHARED	0x1000	/* if the vfsmount is a shared mount */
 #define MNT_UNBINDABLE	0x2000	/* if the vfsmount is a unbindable mount */
 #define MNT_PNODE_MASK	0x3000	/* propagation flag mask */
 
+
 struct vfsmount {
 	struct list_head mnt_hash;
 	struct vfsmount *mnt_parent;	/* fs we are mounted on */
@@ -57,12 +59,6 @@ struct vfsmount {
 	struct mnt_namespace *mnt_ns;	/* containing namespace */
 	int mnt_id;			/* mount identifier */
 	int mnt_group_id;		/* peer group identifier */
-	/*
-	 * We put mnt_count & mnt_expiry_mark at the end of struct vfsmount
-	 * to let these frequently modified fields in a separate cache line
-	 * (so that reads of mnt_flags wont ping-pong on SMP machines)
-	 */
-	atomic_t mnt_count;
 	int mnt_expiry_mark;		/* true if marked for expiry */
 	int mnt_pinned;
 	int mnt_ghosts;
@@ -71,6 +67,11 @@ struct vfsmount {
 #else
 	int mnt_writers;
 #endif
+#ifdef CONFIG_SMP
+	int *mnt_count;
+#else
+	int mnt_count;
+#endif
 };
 
 static inline int *get_mnt_writers_ptr(struct vfsmount *mnt)
@@ -82,34 +83,25 @@ static inline int *get_mnt_writers_ptr(s
 #endif
 }
 
-static inline struct vfsmount *mntget(struct vfsmount *mnt)
-{
-	if (mnt)
-		atomic_inc(&mnt->mnt_count);
-	return mnt;
-}
-
 struct file; /* forward dec */
 
 DECLARE_BRLOCK(vfsmount);
 
+extern unsigned int count_mnt_count(struct vfsmount *mnt);
+
 extern int mnt_want_write(struct vfsmount *mnt);
 extern int mnt_want_write_file(struct file *file);
 extern int mnt_clone_write(struct vfsmount *mnt);
 extern void mnt_drop_write(struct vfsmount *mnt);
+
 extern void mntput_no_expire(struct vfsmount *mnt);
+extern struct vfsmount *mntget(struct vfsmount *mnt);
+extern void mntput(struct vfsmount *mnt);
+
 extern void mnt_pin(struct vfsmount *mnt);
 extern void mnt_unpin(struct vfsmount *mnt);
 extern int __mnt_is_readonly(struct vfsmount *mnt);
 
-static inline void mntput(struct vfsmount *mnt)
-{
-	if (mnt) {
-		mnt->mnt_expiry_mark = 0;
-		mntput_no_expire(mnt);
-	}
-}
-
 extern struct vfsmount *do_kern_mount(const char *fstype, int flags,
 				      const char *name, void *data);
 
fs/pnode.c +2 -2
Index: linux-2.6/fs/pnode.c
===================================================================
@@ -282,7 +282,7 @@ out:
  */
 static inline int do_refcount_check(struct vfsmount *mnt, int count)
 {
-	int mycount = atomic_read(&mnt->mnt_count) - mnt->mnt_ghosts;
+	int mycount = count_mnt_count(mnt) - mnt->mnt_ghosts;
 	return (mycount > count);
 }
 
@@ -294,7 +294,7 @@ static inline int do_refcount_check(stru
  * Check if any of these mounts that **do not have submounts**
  * have more references than 'refcnt'. If so return busy.
  *
- * vfsmount lock must be held for read or write
+ * vfsmount lock must be held for write
  */
 int propagate_mount_busy(struct vfsmount *mnt, int refcnt)
 {
fs/libfs.c +1 -0
Index: linux-2.6/fs/libfs.c
===================================================================
@@ -244,6 +244,7 @@ int get_sb_pseudo(struct file_system_typ
 	d_instantiate(dentry, root);
 	s->s_root = dentry;
 	s->s_flags |= MS_ACTIVE;
+	mnt->mnt_flags |= MNT_MOUNTED;
 	simple_set_mnt(mnt, s);
 	return 0;
 
#20 lore raw

PATCH 5/6 fs: brlock vfsmount_lock

In reply to [email protected]

Use a brlock for the vfsmount lock. It must be taken for write whenever modifying the mount hash or associated fields, and may be taken for read when performing mount hash lookups.

On a 64 core, 64 socket, 32 node Altix system (so a decent amount of latency for remote nodes), a simple dumb mount microbenchmark (mount --bind mnt mnt2 ; umount mnt2 loop 1000 times), before this patch it took 6.8s, afterwards took 7.1s, or about 5% increase in time.

Number of atomics should remain the same for fastpath rlock cases, though code will be slightly larger due to per-cpu access. Scalability will probably not be incredibly improved in common cases yet, due to other locks getting in the way. However independent path lookups over mountpoints should be one case where scalability is improved.

---
 fs/dcache.c                |    4 -
 fs/namei.c                 |   13 +--
 fs/namespace.c             |  178 ++++++++++++++++++++++++++++-----------------
 fs/pnode.c                 |   11 ++
 fs/proc/base.c             |    4 -
 include/linux/mount.h      |    4 -
 kernel/audit_tree.c        |    6 -
 security/tomoyo/realpath.c |    4 -
 8 files changed, 141 insertions(+), 83 deletions(-)
fs/dcache.c +2 -2
Index: linux-2.6/fs/dcache.c
===================================================================
@@ -1908,7 +1908,7 @@ char *__d_path(const struct path *path,
 	char *end = buffer + buflen;
 	char *retval;
 
-	spin_lock(&vfsmount_lock);
+	vfsmount_rlock();
 	prepend(&end, &buflen, "\0", 1);
 	if (d_unlinked(dentry) &&
 		(prepend(&end, &buflen, " (deleted)", 10) != 0))
@@ -1944,7 +1944,7 @@ char *__d_path(const struct path *path,
 	}
 
 out:
-	spin_unlock(&vfsmount_lock);
+	vfsmount_runlock();
 	return retval;
 
 global_root:
fs/namei.c +7 -6
Index: linux-2.6/fs/namei.c
===================================================================
@@ -685,15 +685,16 @@ int follow_up(struct path *path)
 {
 	struct vfsmount *parent;
 	struct dentry *mountpoint;
-	spin_lock(&vfsmount_lock);
+
+	vfsmount_rlock();
 	parent = path->mnt->mnt_parent;
 	if (parent == path->mnt) {
-		spin_unlock(&vfsmount_lock);
+		vfsmount_runlock();
 		return 0;
 	}
 	mntget(parent);
 	mountpoint = dget(path->mnt->mnt_mountpoint);
-	spin_unlock(&vfsmount_lock);
+	vfsmount_runlock();
 	dput(path->dentry);
 	path->dentry = mountpoint;
 	mntput(path->mnt);
@@ -772,15 +773,15 @@ static __always_inline void follow_dotdo
 			break;
 		}
 		spin_unlock(&dcache_lock);
-		spin_lock(&vfsmount_lock);
+		vfsmount_rlock();
 		parent = nd->path.mnt->mnt_parent;
 		if (parent == nd->path.mnt) {
-			spin_unlock(&vfsmount_lock);
+			vfsmount_runlock();
 			break;
 		}
 		mntget(parent);
 		nd->path.dentry = dget(nd->path.mnt->mnt_mountpoint);
-		spin_unlock(&vfsmount_lock);
+		vfsmount_runlock();
 		dput(old);
 		mntput(nd->path.mnt);
 		nd->path.mnt = parent;
fs/namespace.c +113 -65

483 lines not on this page · raw message

fs/pnode.c +9 -2
Index: linux-2.6/fs/pnode.c
===================================================================
@@ -126,6 +126,9 @@ static int do_make_slave(struct vfsmount
 	return 0;
 }
 
+/*
+ * vfsmount lock must be held for write
+ */
 void change_mnt_propagation(struct vfsmount *mnt, int type)
 {
 	if (type == MS_SHARED) {
@@ -264,12 +267,12 @@ int propagate_mnt(struct vfsmount *dest_
 		prev_src_mnt  = child;
 	}
 out:
-	spin_lock(&vfsmount_lock);
+	vfsmount_wlock();
 	while (!list_empty(&tmp_list)) {
 		child = list_first_entry(&tmp_list, struct vfsmount, mnt_hash);
 		umount_tree(child, 0, &umount_list);
 	}
-	spin_unlock(&vfsmount_lock);
+	vfsmount_wunlock();
 	release_mounts(&umount_list);
 	return ret;
 }
@@ -290,6 +293,8 @@ static inline int do_refcount_check(stru
  * other mounts its parent propagates to.
  * Check if any of these mounts that **do not have submounts**
  * have more references than 'refcnt'. If so return busy.
+ *
+ * vfsmount lock must be held for read or write
  */
 int propagate_mount_busy(struct vfsmount *mnt, int refcnt)
 {
@@ -347,6 +352,8 @@ static void __propagate_umount(struct vf
  * collect all mounts that receive propagation from the mount in @list,
  * and return these additional mounts in the same list.
  * @list: the list of mounts to be unmounted.
+ *
+ * vfsmount lock must be held for write
  */
 int propagate_umount(struct list_head *list)
 {
fs/proc/base.c +2 -2
Index: linux-2.6/fs/proc/base.c
===================================================================
@@ -652,12 +652,12 @@ static unsigned mounts_poll(struct file
 
 	poll_wait(file, &ns->poll, wait);
 
-	spin_lock(&vfsmount_lock);
+	vfsmount_rlock();
 	if (p->event != ns->event) {
 		p->event = ns->event;
 		res |= POLLERR | POLLPRI;
 	}
-	spin_unlock(&vfsmount_lock);
+	vfsmount_runlock();
 
 	return res;
 }
include/linux/mount.h +3 -1
Index: linux-2.6/include/linux/mount.h
===================================================================
@@ -13,6 +13,7 @@
 #include <linux/list.h>
 #include <linux/nodemask.h>
 #include <linux/spinlock.h>
+#include <linux/brlock.h>
 #include <asm/atomic.h>
 
 struct super_block;
@@ -90,6 +91,8 @@ static inline struct vfsmount *mntget(st
 
 struct file; /* forward dec */
 
+DECLARE_BRLOCK(vfsmount);
+
 extern int mnt_want_write(struct vfsmount *mnt);
 extern int mnt_want_write_file(struct file *file);
 extern int mnt_clone_write(struct vfsmount *mnt);
@@ -123,7 +126,6 @@ extern int do_add_mount(struct vfsmount
 
 extern void mark_mounts_for_expiry(struct list_head *mounts);
 
-extern spinlock_t vfsmount_lock;
 extern dev_t name_to_dev_t(char *name);
 
 #endif /* _LINUX_MOUNT_H */
kernel/audit_tree.c +3 -3
Index: linux-2.6/kernel/audit_tree.c
===================================================================
@@ -758,15 +758,15 @@ int audit_tag_tree(char *old, char *new)
 			continue;
 		}
 
-		spin_lock(&vfsmount_lock);
+		vfsmount_rlock();
 		if (!is_under(mnt, dentry, &path)) {
-			spin_unlock(&vfsmount_lock);
+			vfsmount_runlock();
 			path_put(&path);
 			put_tree(tree);
 			mutex_lock(&audit_filter_mutex);
 			continue;
 		}
-		spin_unlock(&vfsmount_lock);
+		vfsmount_runlock();
 		path_put(&path);
 
 		list_for_each_entry(p, &list, mnt_list) {
security/tomoyo/realpath.c +2 -2
Index: linux-2.6/security/tomoyo/realpath.c
===================================================================
@@ -96,12 +96,12 @@ int tomoyo_realpath_from_path2(struct pa
 		root = current->fs->root;
 		path_get(&root);
 		read_unlock(&current->fs->lock);
-		spin_lock(&vfsmount_lock);
+		vfsmount_rlock();
 		if (root.mnt && root.mnt->mnt_ns)
 			ns_root.mnt = mntget(root.mnt->mnt_ns->root);
 		if (ns_root.mnt)
 			ns_root.dentry = dget(ns_root.mnt->mnt_root);
-		spin_unlock(&vfsmount_lock);
+		vfsmount_runlock();
 		spin_lock(&dcache_lock);
 		tmp = ns_root;
 		sp = __d_path(path, &tmp, newname, newname_len);
#12 lore raw

PATCH 4/6 brlock: introduce special brlocks

In reply to [email protected]

This patch introduces special brlocks, these can only be used as global locks, and use some preprocessor trickery to allow us to retain a more optimal per-cpu lock implementation. We don't bother working around lockdep yet.

---
 include/linux/brlock.h |  112 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
include/linux/brlock.h new file +112 -0
Index: linux-2.6/include/linux/brlock.h
===================================================================
@@ -0,0 +1,112 @@
+/*
+ * Specialised big-reader spinlock. Can only be declared as global variables
+ * to avoid overhead and keep things simple (and we don't want to start using
+ * these inside dynamically allocated structures).
+ *
+ * Copyright 2009, Nick Piggin, Novell Inc.
+ */
+#ifndef __LINUX_BRLOCK_H
+#define __LINUX_BRLOCK_H
+
+#include <linux/spinlock.h>
+#include <linux/percpu.h>
+#include <asm/atomic.h>
+
+#if defined(CONFIG_SMP) && !defined(CONFIG_LOCKDEP)
+#define DECLARE_BRLOCK(name)						\
+ DECLARE_PER_CPU(spinlock_t, name##_lock);				\
+ static inline void name##_lock_init(void) {				\
+	int i;								\
+	for_each_possible_cpu(i) {					\
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_lock_init(lock);					\
+	}								\
+ }									\
+ static inline void name##_rlock(void) {				\
+	spinlock_t *lock;						\
+	lock = &get_cpu_var(name##_lock);				\
+	spin_lock(lock);						\
+ }									\
+ static inline void name##_runlock(void) {				\
+	spinlock_t *lock;						\
+	lock = &__get_cpu_var(name##_lock);				\
+	spin_unlock(lock);						\
+	put_cpu_var(name##_lock);					\
+ }									\
+ extern void name##_wlock(void);					\
+ extern void name##_wunlock(void);					\
+ static inline int name##_atomic_dec_and_rlock(atomic_t *a) {		\
+	int ret;							\
+	spinlock_t *lock;						\
+	lock = &get_cpu_var(name##_lock);				\
+	ret = atomic_dec_and_lock(a, lock);				\
+	if (!ret)							\
+		put_cpu_var(name##_lock);				\
+	return ret;							\
+ }									\
+ extern int name##_atomic_dec_and_wlock__failed(atomic_t *a);		\
+ static inline int name##_atomic_dec_and_wlock(atomic_t *a) {		\
+	if (atomic_add_unless(a, -1, 1))				\
+		return 0;						\
+	return name##_atomic_dec_and_wlock__failed(a);			\
+ }
+
+#define DEFINE_BRLOCK(name)						\
+ DEFINE_PER_CPU(spinlock_t, name##_lock);				\
+ void name##_wlock(void) {						\
+	int i;								\
+	for_each_online_cpu(i) {					\
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_lock(lock);					\
+	}								\
+ }									\
+ void name##_wunlock(void) {						\
+	int i;								\
+	for_each_online_cpu(i) {					\
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_unlock(lock);					\
+	}								\
+ }									\
+ int name##_atomic_dec_and_wlock__failed(atomic_t *a) {			\
+	name##_wlock();							\
+	if (!atomic_dec_and_test(a)) {					\
+		name##_wunlock();					\
+		return 0;						\
+	}								\
+	return 1;							\
+ }
+
+#else
+
+#define DECLARE_BRLOCK(name)						\
+ spinlock_t name##_lock;						\
+ static inline void name##_lock_init(void) {				\
+	spin_lock_init(&name##_lock);					\
+ }									\
+ static inline void name##_rlock(void) {				\
+	spin_lock(&name##_lock);					\
+ }									\
+ static inline void name##_runlock(void) {				\
+	spin_unlock(&name##_lock);					\
+ }									\
+ static inline void name##_wlock(void) {				\
+	spin_lock(&name##_lock);					\
+ }									\
+ static inline void name##_wunlock(void) {				\
+	spin_unlock(&name##_lock);					\
+ }									\
+ static inline int name##_atomic_dec_and_rlock(atomic_t *a) {		\
+	return atomic_dec_and_lock(a, &name##_lock);			\
+ }									\
+ static inline int name##_atomic_dec_and_wlock(atomic_t *a) {		\
+	return atomic_dec_and_lock(a, &name##_lock);			\
+ }
+
+#define DEFINE_BRLOCK(name)						\
+ spinlock_t name##_lock
+#endif
+
+#endif
maintainer Acked-by on v1 2/6
#3 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

In reply to [email protected]

From: [email protected]
Date: Thu, 15 Oct 2009 15:40:28 +1100

Quoted 6 lines

(this is in -mm)

Filesystems outside the regular namespace do not have to clear DCACHE_UNHASHED in order to have a working /proc/$pid/fd/XXX. Nothing in proc prevents the fd link from being used if its dentry is not in the hash.

Quoted 7 lines

Also, it does not get put into the dcache hash if DCACHE_UNHASHED is clear; that depends on the filesystem calling d_add or d_rehash.

So delete the misleading comments and needless code.

For networking bits:

#13 lore raw

RFC PATCH 1/1 brlock: "fast" brlocks

In reply to [email protected]

[Not for merge. Stop reading if you're not interested in locking minutiae.]

OK, this is untested but I think the theory is right. Basically it is taking the idea from Dave M's cool brlock optimisation stuff with one further optimisation in that the read locker does not check the spinlock but rather we keep another wlocked variable together inthe same cacheline per CPU, so the read locker only has to touch one cacheline rather than 2.

This actually will reduce the number of atomics by 2 per path lookup, however we have an smp_mb() there now which is really nasty on some architectures (like ia64 and ppc64), and not that nice on x86 either. We can probably do something interesting on ia64 and ppc64 so that we take advantage of the fact rlocked and wlocked are in the same cacheline so cache coherency (rather than memory consistency) should always provide a strict ordering there. We still do need an acquire barrier -- but it is a much nicer lwsync or st.acq on ppc and ia64.

But: is the avoidance of the atomic RMW a big win? On x86 cores I've tested IIRC mfence is about as costly as a locked instruction which includes the mfence...

So long story short: it might be a small win but it is going to be very arch specific and will require arch specific code to do the barriers and things. The generic spinlock brlock isn't bad at all, so I'll just post this as a curiosity for the time being.

---
 include/linux/brlock.h |   97 +++++++++++++++++++++++++++++++------------------
 1 file changed, 63 insertions(+), 34 deletions(-)
include/linux/brlock.h +63 -34
Index: linux-2.6/include/linux/brlock.h
===================================================================
@@ -13,37 +13,45 @@
 #include <asm/atomic.h>
 
 #if defined(CONFIG_SMP) && !defined(CONFIG_LOCKDEP)
+
+struct brlock_node {
+	unsigned int rlocked;
+	unsigned int wlocked;
+};
+
 #define DECLARE_BRLOCK(name)						\
- DECLARE_PER_CPU(spinlock_t, name##_lock);				\
- static inline void name##_lock_init(void) {				\
-	int i;								\
-	for_each_possible_cpu(i) {					\
-		spinlock_t *lock;					\
-		lock = &per_cpu(name##_lock, i);			\
-		spin_lock_init(lock);					\
-	}								\
- }									\
+ DECLARE_PER_CPU_ALIGNED(struct brlock_node, name##_lock);		\
  static inline void name##_rlock(void) {				\
-	spinlock_t *lock;						\
-	lock = &get_cpu_var(name##_lock);				\
-	spin_lock(lock);						\
+	struct brlock_node *n;						\
+	n = &get_cpu_var(name##_lock);					\
+again:									\
+	n->rlocked++;							\
+	smp_mb(); /* ensure rlocked store before wlocked load */	\
+	if (unlikely(n->wlocked)) {					\
+		n->rlocked--;						\
+		while (n->wlocked)					\
+			cpu_relax();					\
+		goto again;						\
+	}								\
+	/* smp_mb() above also prevents crit stores leaking up */	\
  }									\
  static inline void name##_runlock(void) {				\
-	spinlock_t *lock;						\
-	lock = &__get_cpu_var(name##_lock);				\
-	spin_unlock(lock);						\
+	struct brlock_node *n;						\
+	smp_wmb(); /* prevent crit stores leaking out (XXX: should really be a release barrier because some architectures might leak crit loads out past the store, but I want to test x86 performance here) */				\
+	n = &__get_cpu_var(name##_lock);				\
+	n->rlocked--;							\
 	put_cpu_var(name##_lock);					\
  }									\
  extern void name##_wlock(void);					\
  extern void name##_wunlock(void);					\
  static inline int name##_atomic_dec_and_rlock(atomic_t *a) {		\
-	int ret;							\
-	spinlock_t *lock;						\
-	lock = &get_cpu_var(name##_lock);				\
-	ret = atomic_dec_and_lock(a, lock);				\
-	if (!ret)							\
-		put_cpu_var(name##_lock);				\
-	return ret;							\
+	if (likely(atomic_add_unless(a, -1, 1)))			\
+		return 0;						\
+	name##_rlock();							\
+	if (atomic_dec_and_test(a))					\
+		return 1;						\
+	name##_runlock();						\
+	return 0;							\
  }									\
  extern int name##_atomic_dec_and_wlock__failed(atomic_t *a);		\
  static inline int name##_atomic_dec_and_wlock(atomic_t *a) {		\
@@ -53,30 +61,51 @@
  }
 
 #define DEFINE_BRLOCK(name)						\
- DEFINE_PER_CPU(spinlock_t, name##_lock);				\
+ DEFINE_PER_CPU_ALIGNED(struct brlock_node, name##_lock);		\
+ static DEFINE_SPINLOCK(name##_wlock_lock);				\
+ static void name##_lock_init(void) {					\
+	int i;								\
+	for_each_possible_cpu(i) {					\
+		struct brlock_node *n;					\
+		n = &per_cpu(name##_lock, i);				\
+		n->rlocked = 0;						\
+		n->wlocked = 0;						\
+	}								\
+	spin_lock_init(&name##_wlock_lock);				\
+ }									\
  void name##_wlock(void) {						\
 	int i;								\
+	spin_lock(&name##_wlock_lock);					\
 	for_each_online_cpu(i) {					\
-		spinlock_t *lock;					\
-		lock = &per_cpu(name##_lock, i);			\
-		spin_lock(lock);					\
+		struct brlock_node *n;					\
+		n = &per_cpu(name##_lock, i);				\
+		n->wlocked = 1;						\
 	}								\
+	smp_mb(); /* ensure wlocked store before rlocked load */	\
+	for_each_online_cpu(i) {					\
+		struct brlock_node *n;					\
+		n = &per_cpu(name##_lock, i);				\
+		while (n->rlocked)					\
+			cpu_relax();					\
+	}								\
+	smp_mb(); /* prevent crit ops leaking above rlocked check */	\
  }									\
  void name##_wunlock(void) {						\
 	int i;								\
+	smp_mb(); /* prevent crit ops leaking past wlocked = 0 */	\
 	for_each_online_cpu(i) {					\
-		spinlock_t *lock;					\
-		lock = &per_cpu(name##_lock, i);			\
-		spin_unlock(lock);					\
+		struct brlock_node *n;					\
+		n = &per_cpu(name##_lock, i);				\
+		n->wlocked = 0;						\
 	}								\
+	spin_unlock(&name##_wlock_lock);				\
  }									\
  int name##_atomic_dec_and_wlock__failed(atomic_t *a) {			\
 	name##_wlock();							\
-	if (!atomic_dec_and_test(a)) {					\
-		name##_wunlock();					\
-		return 0;						\
-	}								\
-	return 1;							\
+	if (atomic_dec_and_test(a))					\
+		return 1;						\
+	name##_wunlock();						\
+	return 0;							\
  }
 
 #else
Acked-by David S. Miller Acked-by on v1 2/6
#4 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

In reply to [email protected]
Quoted 11 lines · [email protected] a écrit :

(this is in -mm)

Filesystems outside the regular namespace do not have to clear DCACHE_UNHASHED in order to have a working /proc/$pid/fd/XXX. Nothing in proc prevents the fd link from being used if its dentry is not in the hash.

Also, it does not get put into the dcache hash if DCACHE_UNHASHED is clear; that depends on the filesystem calling d_add or d_rehash.

So delete the misleading comments and needless code.

This was added in commit 304e61e6fbadec586dfe002b535f169a04248e49

[PATCH] net: don't insert socket dentries into dentry_hashtable

We currently insert socket dentries into the global dentry hashtable. This is suboptimal because there is currently no way these entries can be used for a lookup(). (/proc/xxx/fd/xxx uses a different mechanism). Inserting them in dentry hashtable slows dcache lookups.

To let __dpath() still work correctly (ie not adding a " (deleted)") after dentry name, we do :

- Right after d_alloc(), pretend they are hashed by clearing the DCACHE_UNHASHED bit.

- Call d_instantiate() instead of d_add() : dentry is not inserted in hash table.

__dpath() & friends work as intended during dentry lifetime.

- At dismantle time, once dput() must clear the dentry, setting again DCACHE_UNHASHED bit inside the custom d_delete() function provided by socket code, so that dput() can just kill_it.

Back in 2006, we had to perform this hack in order to not leak '(deleted)' in __d_path()

if (!IS_ROOT(dentry) && d_unhashed(dentry) &&

	(prepend(&end, &buflen, " (deleted)", 10) != 0)) 
		goto Elong;

In current kernel this part became :

if (d_unlinked(dentry) &&

	(prepend(&end, &buflen, " (deleted)", 10) != 0))
		goto Elong;

So your cleanup seems good, thanks !

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

Quoted 53 lines · Eric Dumazet
Quoted 11 lines · [email protected] a écrit :

(this is in -mm)

Filesystems outside the regular namespace do not have to clear DCACHE_UNHASHED in order to have a working /proc/$pid/fd/XXX. Nothing in proc prevents the fd link from being used if its dentry is not in the hash.

Also, it does not get put into the dcache hash if DCACHE_UNHASHED is clear; that depends on the filesystem calling d_add or d_rehash.

So delete the misleading comments and needless code.

This was added in commit 304e61e6fbadec586dfe002b535f169a04248e49

[PATCH] net: don't insert socket dentries into dentry_hashtable

We currently insert socket dentries into the global dentry hashtable. This is suboptimal because there is currently no way these entries can be used for a lookup(). (/proc/xxx/fd/xxx uses a different mechanism). Inserting them in dentry hashtable slows dcache lookups.

To let __dpath() still work correctly (ie not adding a " (deleted)") after dentry name, we do :

- Right after d_alloc(), pretend they are hashed by clearing the DCACHE_UNHASHED bit.

- Call d_instantiate() instead of d_add() : dentry is not inserted in hash table.

__dpath() & friends work as intended during dentry lifetime.

- At dismantle time, once dput() must clear the dentry, setting again DCACHE_UNHASHED bit inside the custom d_delete() function provided by socket code, so that dput() can just kill_it.

Back in 2006, we had to perform this hack in order to not leak '(deleted)' in __d_path()

if (!IS_ROOT(dentry) && d_unhashed(dentry) &&

	(prepend(&end, &buflen, " (deleted)", 10) != 0)) 
		goto Elong;

In current kernel this part became :

Quoted 8 lines

if (d_unlinked(dentry) &&

	(prepend(&end, &buflen, " (deleted)", 10) != 0))
		goto Elong;

So your cleanup seems good, thanks !

Ahh, hmm d_unlinked() is exactly the same code. OK, so I think this shows why I'm an idiot. I didn't think about the __d_path output. Though the comments are still misleading, in my defence (and my test programs did not use any anoninode fds)...

Now both sockets and pipes define a d_dname so they are OK, but anon_inodes does not. I think they should probably be made to just provide a d_dname anyway so we can have the familiar format of "pseudofs:[ino]" rather than "[pseudofs]" that we have now.

That should make this patch work for anon_inodes.c as well.

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

Quoted 6 lines · Nick Piggin

Now both sockets and pipes define a d_dname so they are OK, but anon_inodes does not. I think they should probably be made to just provide a d_dname anyway so we can have the familiar format of "pseudofs:[ino]" rather than "[pseudofs]" that we have now.

That should make this patch work for anon_inodes.c as well.

So what if we were to do this first? Are there any other reasons I've missed? (btw. this changes the format the link if that is a problem. Probably if we're going to do this change, we should change things like [timerfd] to just timerfd while we're there to improve consistency further.
--
Pipe and socket pseudo filesystems report their file descriptor link paths as pipe:[ino] and socket:[ino]. anon_inodefs allows subsystems to specify a name, but it does not report an associated inode number.

Implement this with anon_inodefs_dname in the same way pipefs and sockfs are.

---
 fs/anon_inodes.c |   10 ++++++++++
 1 file changed, 10 insertions(+)
fs/anon_inodes.c +10 -0
Index: linux-2.6/fs/anon_inodes.c
===================================================================
@@ -45,6 +45,15 @@ static int anon_inodefs_delete_dentry(st
 	return 1;
 }
 
+/*
+ * anon_inodefs_dname() is called from d_path().
+ */
+static char *anon_inodefs_dname(struct dentry *dentry, char *buffer, int buflen)
+{
+	return dynamic_dname(dentry, buffer, buflen, "%s:[%lu]",
+				dentry->d_name.name, dentry->d_inode->i_ino);
+}
+
 static struct file_system_type anon_inode_fs_type = {
 	.name		= "anon_inodefs",
 	.get_sb		= anon_inodefs_get_sb,
@@ -52,6 +61,7 @@ static struct file_system_type anon_inod
 };
 static const struct dentry_operations anon_inodefs_dentry_operations = {
 	.d_delete	= anon_inodefs_delete_dentry,
+	.d_dname	= anon_inodefs_dname,
 };
 
 /*
Acked-by
#7 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

Quoted 38 lines · Nick Piggin

So what if we were to do this first? Are there any other reasons I've missed? (btw. this changes the format the link if that is a problem. Probably if we're going to do this change, we should change things like [timerfd] to just timerfd while we're there to improve consistency further.
--
Pipe and socket pseudo filesystems report their file descriptor link paths as pipe:[ino] and socket:[ino]. anon_inodefs allows subsystems to specify a name, but it does not report an associated inode number.

Implement this with anon_inodefs_dname in the same way pipefs and sockfs are.

---
 fs/anon_inodes.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

Index: linux-2.6/fs/anon_inodes.c
===================================================================

fs/anon_inodes.c +9 -0
@@ -45,6 +45,15 @@ static int anon_inodefs_delete_dentry(st
 	return 1;
 }
 
+/*
+ * anon_inodefs_dname() is called from d_path().
+ */
+static char *anon_inodefs_dname(struct dentry *dentry, char *buffer, int buflen)
+{
+	return dynamic_dname(dentry, buffer, buflen, "%s:[%lu]",
+				dentry->d_name.name, dentry->d_inode->i_ino);
+}
+
 static struct file_system_type anon_inode_fs_type = {
 	.name		= "anon_inodefs",
 	.get_sb		= anon_inodefs_get_sb,

fs/anon_inodes.c static struct file_system_type anon_inod · line 67

Quoted 7 lines
fs/anon_inodes.c +1 -0
@@ -52,6 +61,7 @@ static struct file_system_type anon_inod
 };
 static const struct dentry_operations anon_inodefs_dentry_operations = {
 	.d_delete	= anon_inodefs_delete_dentry,
+	.d_dname	= anon_inodefs_dname,
 };
 
 /*

Yes, that would be fine, thanks Nick.

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by on v1 3/6
#11 lore raw

Re: [patch 3/6] fs: dcache remove d_mounted

In reply to [email protected]
Quoted 20 lines · On Thu, Oct 15, 2009 at 03:40:29PM +1100, [email protected] wrote:

Rather than keep a d_mounted count in the dentry (which is only used to speculatively take a look in the mount hash table if it is non-zero), set a dentry flag instead. The flag can be cleared by checking the hash table to see if there are any mounts left. It is not time critical because it is performed at detach time.

This saves 4 bytes on 32-bit, nothing on 64-bit but it does provide a hole I might use later (and some configs have larger than 32-bit spinlocks which might make use of the hole).

Autofs4 conversion and changelog by Ian Kent <[email protected]>:
In autofs4, when expring direct (or offset) mounts we need to ensure that we block user path walks into the autofs mount, which is covered by another mount. To do this we clear the mounted status so that follows stop before walking into the mount and are essentially blocked until the expire is completed. The automount daemon still finds the correct dentry for the umount due to the follow mount logic in fs/autofs4/root.c:autofs4_follow_link(), which is set as an inode operation for direct and offset mounts only and is called following the lookup that stopped at the covered mount.

Quoted 5 lines

At the end of the expire the covering mount probably has gone away so the mounted status need not be restored. But we need to check this and only restore the mounted status if the expire failed.

For the autofs4 bits.

#14 lore raw

Re: [rfc][patch 4a/6] brlock: "fast" brlocks

In reply to Nick Piggin
Quoted 17 lines · Nick Piggin

[Not for merge. Stop reading if you're not interested in locking minutiae.]

OK, this is untested but I think the theory is right. Basically it is taking the idea from Dave M's cool brlock optimisation stuff with one further optimisation in that the read locker does not check the spinlock but rather we keep another wlocked variable together inthe same cacheline per CPU, so the read locker only has to touch one cacheline rather than 2.

This actually will reduce the number of atomics by 2 per path lookup, however we have an smp_mb() there now which is really nasty on some architectures (like ia64 and ppc64), and not that nice on x86 either. We can probably do something interesting on ia64 and ppc64 so that we take advantage of the fact rlocked and wlocked are in the same cacheline so cache coherency (rather than memory consistency) should always provide a strict ordering there. We still do need an acquire barrier -- but it is a much nicer lwsync or st.acq on ppc and ia64.

Quoted 9 lines

But: is the avoidance of the atomic RMW a big win? On x86 cores I've tested IIRC mfence is about as costly as a locked instruction which includes the mfence...

So long story short: it might be a small win but it is going to be very arch specific and will require arch specific code to do the barriers and things. The generic spinlock brlock isn't bad at all, so I'll just post this as a curiosity for the time being.

fwiw, I rather like this implementation better, and adding lockdep annotations to this one shouldn't be hard.

#15 lore raw

Re: [rfc][patch 4a/6] brlock: "fast" brlocks

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Nick Piggin

[Not for merge. Stop reading if you're not interested in locking minutiae.]

OK, this is untested but I think the theory is right. Basically it is taking the idea from Dave M's cool brlock optimisation stuff with one further optimisation in that the read locker does not check the spinlock but rather we keep another wlocked variable together inthe same cacheline per CPU, so the read locker only has to touch one cacheline rather than 2.

This actually will reduce the number of atomics by 2 per path lookup, however we have an smp_mb() there now which is really nasty on some architectures (like ia64 and ppc64), and not that nice on x86 either. We can probably do something interesting on ia64 and ppc64 so that we take advantage of the fact rlocked and wlocked are in the same cacheline so cache coherency (rather than memory consistency) should always provide a strict ordering there. We still do need an acquire barrier -- but it is a much nicer lwsync or st.acq on ppc and ia64.

But: is the avoidance of the atomic RMW a big win? On x86 cores I've tested IIRC mfence is about as costly as a locked instruction which includes the mfence...

Quoted 8 lines
Quoted 5 lines

So long story short: it might be a small win but it is going to be very arch specific and will require arch specific code to do the barriers and things. The generic spinlock brlock isn't bad at all, so I'll just post this as a curiosity for the time being.

fwiw, I rather like this implementation better, and adding lockdep annotations to this one shouldn't be hard.

OK, although there is nothing preventing us from using raw spinlocks and a new lockdep object to annotate the other one, right?

The problem with this one is that firstly it is not suitable for a generic implementation (see XXX, we technically need smp_mb in the unlock rather than smp_wmb -- I don't know if any actual CPUs do this but several architectures do allow for stores to pass loads)

So we _really_ need to do proper acquire and release barriers both here in the unlock and in the lock as well for it to be competitive with the spinlocks.

smp_mb is a nasty hammer especially for release barrier because it prevents loads from being started early until the unlock store is visible. For acquire barrier it theoretically is not so bad, but in practice like on powerpc they do it with an instruction that has to apparently go out to the interconnect.

#8 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

In reply to Nick Piggin
Quoted 8 lines · Nick Piggin
Quoted 6 lines · Nick Piggin

Now both sockets and pipes define a d_dname so they are OK, but anon_inodes does not. I think they should probably be made to just provide a d_dname anyway so we can have the familiar format of "pseudofs:[ino]" rather than "[pseudofs]" that we have now.

That should make this patch work for anon_inodes.c as well.

Quoted 9 lines

So what if we were to do this first? Are there any other reasons I've missed? (btw. this changes the format the link if that is a problem. Probably if we're going to do this change, we should change things like [timerfd] to just timerfd while we're there to improve consistency further.
--
Pipe and socket pseudo filesystems report their file descriptor link paths as pipe:[ino] and socket:[ino]. anon_inodefs allows subsystems to specify a name, but it does not report an associated inode number.

I think that's because the inode number is always 0 -- there's just one anonymous inode.

Quoted 39 lines

Implement this with anon_inodefs_dname in the same way pipefs and sockfs are.

---
 fs/anon_inodes.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

Index: linux-2.6/fs/anon_inodes.c
===================================================================

fs/anon_inodes.c +10 -1
@@ -45,6 +45,15 @@ static int anon_inodefs_delete_dentry(st
 	return 1;
 }
 
+/*
+ * anon_inodefs_dname() is called from d_path().
+ */
+static char *anon_inodefs_dname(struct dentry *dentry, char *buffer, int buflen)
+{
+	return dynamic_dname(dentry, buffer, buflen, "%s:[%lu]",
+				dentry->d_name.name, dentry->d_inode->i_ino);
+}
+
 static struct file_system_type anon_inode_fs_type = {
 	.name		= "anon_inodefs",
 	.get_sb		= anon_inodefs_get_sb,
@@ -52,6 +61,7 @@ static struct file_system_type anon_inod
 };
 static const struct dentry_operations anon_inodefs_dentry_operations = {
 	.d_delete	= anon_inodefs_delete_dentry,
+	.d_dname	= anon_inodefs_dname,
 };
 
 /*
--

To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#9 lore raw

Re: [patch 2/6] fs: no games with DCACHE_UNHASHED

Quoted 19 lines · Matthew Wilcox
Quoted 8 lines · Nick Piggin
Quoted 6 lines · Nick Piggin

Now both sockets and pipes define a d_dname so they are OK, but anon_inodes does not. I think they should probably be made to just provide a d_dname anyway so we can have the familiar format of "pseudofs:[ino]" rather than "[pseudofs]" that we have now.

That should make this patch work for anon_inodes.c as well.

Quoted 9 lines

So what if we were to do this first? Are there any other reasons I've missed? (btw. this changes the format the link if that is a problem. Probably if we're going to do this change, we should change things like [timerfd] to just timerfd while we're there to improve consistency further.
--
Pipe and socket pseudo filesystems report their file descriptor link paths as pipe:[ino] and socket:[ino]. anon_inodefs allows subsystems to specify a name, but it does not report an associated inode number.

Quoted 2 lines

I think that's because the inode number is always 0 -- there's just one anonymous inode.

OIC. Hmm, well we could give them a unique number I guess. But anyway for now, let's just make anon_inodefs_dname just print dentry->d_name.name.

Quoted 46 lines
Quoted 40 lines

Implement this with anon_inodefs_dname in the same way pipefs and sockfs are.

---
 fs/anon_inodes.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

Index: linux-2.6/fs/anon_inodes.c
===================================================================

fs/anon_inodes.c +10 -1
@@ -45,6 +45,15 @@ static int anon_inodefs_delete_dentry(st
 	return 1;
 }
 
+/*
+ * anon_inodefs_dname() is called from d_path().
+ */
+static char *anon_inodefs_dname(struct dentry *dentry, char *buffer, int buflen)
+{
+	return dynamic_dname(dentry, buffer, buflen, "%s:[%lu]",
+				dentry->d_name.name, dentry->d_inode->i_ino);
+}
+
 static struct file_system_type anon_inode_fs_type = {
 	.name		= "anon_inodefs",
 	.get_sb		= anon_inodefs_get_sb,
@@ -52,6 +61,7 @@ static struct file_system_type anon_inod
 };
 static const struct dentry_operations anon_inodefs_dentry_operations = {
 	.d_delete	= anon_inodefs_delete_dentry,
+	.d_dname	= anon_inodefs_dname,
 };
 
 /*
--

To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#16 lore raw

Re: [patch 4/6] brlock: introduce special brlocks

In reply to [email protected]
Quoted 1 line · On Thu, 15 Oct 2009 15:40:30 +1100 [email protected] wrote:

+#define DECLARE_BRLOCK(name) \

This:

Quoted 4 lines

+ DECLARE_PER_CPU(spinlock_t, name##_lock); \
+ static inline void name##_lock_init(void) { \

include/linux/brlock.h new file +2 -0
+	int i;								\
+	for_each_possible_cpu(i) {					\

include/linux/brlock.h line 30

Quoted 10 lines
include/linux/brlock.h new file +10 -0
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_lock_init(lock);					\
+	}								\
+ }									\
+ static inline void name##_rlock(void) {				\
+	spinlock_t *lock;						\
+	lock = &get_cpu_var(name##_lock);				\
+	spin_lock(lock);						\
+ }									\

generates a definition, not a declaration. Hence DEFINE_BRLOCK.

</petpeeve #29>

#17 lore raw

Re: [patch 4/6] brlock: introduce special brlocks

Quoted 23 lines · Andrew Morton
Quoted 1 line · On Thu, 15 Oct 2009 15:40:30 +1100 [email protected] wrote:

+#define DECLARE_BRLOCK(name) \

This:

Quoted 4 lines

+ DECLARE_PER_CPU(spinlock_t, name##_lock); \
+ static inline void name##_lock_init(void) { \

include/linux/brlock.h new file +2 -0
+	int i;								\
+	for_each_possible_cpu(i) {					\

include/linux/brlock.h line 30

Quoted 10 lines
include/linux/brlock.h new file +10 -0
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_lock_init(lock);					\
+	}								\
+ }									\
+ static inline void name##_rlock(void) {				\
+	spinlock_t *lock;						\
+	lock = &get_cpu_var(name##_lock);				\
+	spin_lock(lock);						\
+ }									\

generates a definition, not a declaration. Hence DEFINE_BRLOCK.

Quoted 1 line

</petpeeve #29>

Well yes, but being a static inline, then I don't know of a better way. Probably just better not to pretend we are expanding a simple declaration here, and name it something differently? (BRLOCK_HEADER(blah))?

#18 lore raw

Re: [patch 4/6] brlock: introduce special brlocks

Quoted 24 lines · On Mon, 19 Oct 2009 11:49:09 +0200 Nick Piggin <[email protected]> wrote:
Quoted 23 lines · Andrew Morton
Quoted 1 line · On Thu, 15 Oct 2009 15:40:30 +1100 [email protected] wrote:

+#define DECLARE_BRLOCK(name) \

This:

Quoted 4 lines

+ DECLARE_PER_CPU(spinlock_t, name##_lock); \
+ static inline void name##_lock_init(void) { \

include/linux/brlock.h new file +2 -0
+	int i;								\
+	for_each_possible_cpu(i) {					\

include/linux/brlock.h line 30

Quoted 10 lines
include/linux/brlock.h new file +10 -0
+		spinlock_t *lock;					\
+		lock = &per_cpu(name##_lock, i);			\
+		spin_lock_init(lock);					\
+	}								\
+ }									\
+ static inline void name##_rlock(void) {				\
+	spinlock_t *lock;						\
+	lock = &get_cpu_var(name##_lock);				\
+	spin_lock(lock);						\
+ }									\

generates a definition, not a declaration. Hence DEFINE_BRLOCK.

Quoted 5 lines
Quoted 1 line

</petpeeve #29>

Well yes, but being a static inline, then I don't know of a better way. Probably just better not to pretend we are expanding a simple declaration here, and name it something differently? (BRLOCK_HEADER(blah))?

DEFINE_BRLOCK(blah)

#19 lore raw

Re: [patch 4/6] brlock: introduce special brlocks

Quoted 10 lines · Andrew Morton
Quoted 7 lines · On Mon, 19 Oct 2009 11:49:09 +0200 Nick Piggin <[email protected]> wrote:
Quoted 3 lines

generates a definition, not a declaration. Hence DEFINE_BRLOCK.

</petpeeve #29>

Well yes, but being a static inline, then I don't know of a better way. Probably just better not to pretend we are expanding a simple declaration here, and name it something differently? (BRLOCK_HEADER(blah))?

DEFINE_BRLOCK(blah)

Well I use DEFINE_BRLOCK for the .c file definitions which include non-static non-inline functions, so you can't put it in a .h. So AFAIKS you need both. Athough DECLARE_BRLOCK is not strictly for declarations because of those static inline functions so I agree the name is not ideal.