fs: Inode cache scalability
Subsystems (41)
cover fs: Inode cache scalability
This patch set is derived from Nick Piggin's VFS scalability tree. There doesn't appear to be any push to get that tree into shape for .37, so this is an attempt to start the process of finer grained review of the series for upstream inclusion. I'm hitting VFS lock contention problems with XFS on 8-16p machines now, so I need to get this stuff moving.
This patch set is just the basic inode_lock breakup patches plus a few more simple changes to the inode code. It stops short of introducing RCU inode freeing because those changes are not completely baked yet. It also stops short of changing the way inodes are tracked for writeback because I'd like not to spend my week after -rc1 is released fixing writeback again....
As a result, the full inode handling improvements of Nick's patch set are not realised with this short series. However, my own testing indicates that the amount of lock traffic and contention is down by an order of magnitude on an 8-way box for parallel inode create and unlink workloads, so there is still significant improvements in scalability from just this patch set.
I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.
Basically I'm trying to get this patchset ready for .37 (merge window is not really that far off now), and I'm aiming to have the rest of the inode changes (RCU freeing, writeback, etc) ready for .38. I may even look to some of the dcache changes for .38 depending on how much I can get tested and reviewed in that time frame.
Comments are welcome.
The current patchset is also available at the following location.
The following changes since commit b30a3f6257ed2105259b404d419b4964e363928c:
Linux 2.6.36-rc5 (2010-09-20 16:56:53 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git inode-scale
Eric Dumazet (2):
fs: inode per-cpu last_ino allocator
fs: Convert nr_inodes to a per-cpu counter
Nick Piggin (15):
kernel: add bl_list
fs: icache lock s_inodes list
fs: icache lock inode hash
fs: icache lock i_state
fs: icache lock i_count
fs: icache lock lru/writeback lists
fs: icache atomic inodes_stat
fs: icache protect inode state
fs: Make last_ino, iunique independent of inode_lock
fs: icache remove inode_lock
fs: Factor inode hash operations into functions
fs: Introduce per-bucket inode hash locks
fs: Implement lazy LRU updates for inodes.
fs: Inode counters do not need to be atomic.
fs: Clean up inode reference counting
Documentation/filesystems/Locking | 2 +- Documentation/filesystems/porting | 10 +- Documentation/filesystems/vfs.txt | 2 +- arch/powerpc/platforms/cell/spufs/file.c | 2 +- drivers/staging/pohmelfs/inode.c | 14 +- fs/9p/vfs_inode.c | 4 +- fs/affs/inode.c | 2 +- fs/afs/dir.c | 4 +- fs/anon_inodes.c | 2 +- fs/bfs/dir.c | 2 +- fs/block_dev.c | 7 +- fs/btrfs/inode.c | 23 +- fs/buffer.c | 2 +- fs/ceph/mds_client.c | 2 +- fs/cifs/inode.c | 2 +- fs/coda/dir.c | 2 +- fs/drop_caches.c | 19 +- fs/exofs/inode.c | 10 +- fs/exofs/namei.c | 2 +- fs/ext2/namei.c | 2 +- fs/ext3/ialloc.c | 4 +- fs/ext3/namei.c | 2 +- fs/ext4/ialloc.c | 4 +- fs/ext4/namei.c | 2 +- fs/fs-writeback.c | 156 +++++--- fs/gfs2/ops_inode.c | 2 +- fs/hfs/hfs_fs.h | 2 +- fs/hfs/inode.c | 2 +- fs/hfsplus/dir.c | 2 +- fs/hfsplus/hfsplus_fs.h | 2 +- fs/hfsplus/inode.c | 2 +- fs/hpfs/inode.c | 2 +- fs/inode.c | 603 ++++++++++++++++++++---------- fs/jffs2/dir.c | 4 +- fs/jfs/jfs_txnmgr.c | 2 +- fs/jfs/namei.c | 2 +- fs/libfs.c | 2 +- fs/locks.c | 2 +- fs/logfs/dir.c | 2 +- fs/logfs/inode.c | 2 +- fs/logfs/readwrite.c | 6 +- fs/minix/namei.c | 2 +- fs/namei.c | 2 +- fs/nfs/dir.c | 2 +- fs/nfs/getroot.c | 4 +- fs/nfs/inode.c | 4 +- fs/nfs/nfs4state.c | 2 +- fs/nfs/write.c | 2 +- fs/nilfs2/gcdat.c | 1 + fs/nilfs2/gcinode.c | 22 +- fs/nilfs2/mdt.c | 2 +- fs/nilfs2/namei.c | 2 +- fs/nilfs2/segment.c | 2 +- fs/nilfs2/the_nilfs.h | 2 +- fs/notify/inode_mark.c | 46 ++- fs/notify/mark.c | 1 - fs/notify/vfsmount_mark.c | 1 - fs/ntfs/inode.c | 4 +- fs/ntfs/super.c | 2 +- fs/ocfs2/inode.c | 2 +- fs/ocfs2/namei.c | 2 +- fs/quota/dquot.c | 36 +- fs/reiserfs/namei.c | 2 +- fs/reiserfs/stree.c | 2 +- fs/reiserfs/xattr.c | 2 +- fs/sysv/namei.c | 2 +- fs/ubifs/dir.c | 2 +- fs/ubifs/super.c | 2 +- fs/udf/namei.c | 2 +- fs/ufs/namei.c | 2 +- fs/xfs/linux-2.6/xfs_iops.c | 2 +- fs/xfs/linux-2.6/xfs_trace.h | 2 +- fs/xfs/xfs_inode.h | 4 +- include/linux/fs.h | 54 ++- include/linux/list_bl.h | 127 +++++++ include/linux/rculist_bl.h | 128 +++++++ include/linux/writeback.h | 4 +- ipc/mqueue.c | 2 +- kernel/futex.c | 2 +- kernel/sysctl.c | 4 +- mm/backing-dev.c | 8 +- mm/filemap.c | 6 +- mm/rmap.c | 6 +- mm/shmem.c | 6 +- net/socket.c | 2 +- 85 files changed, 1001 insertions(+), 437 deletions(-) create mode 100644 include/linux/list_bl.h create mode 100644 include/linux/rculist_bl.h
PATCH 1/17 kernel: add bl_list
From: Nick Piggin <[email protected]>
Introduce a type of hlist that can support the use of the lowest bit in the hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes.
--- include/linux/list_bl.h | 127 +++++++++++++++++++++++++++++++++++++++++++ include/linux/rculist_bl.h | 128 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 255 insertions(+), 0 deletions(-) create mode 100644 include/linux/list_bl.h create mode 100644 include/linux/rculist_bl.h
include/linux/list_bl.h new file +127 -0
@@ -0,0 +1,127 @@ +#ifndef _LINUX_LIST_BL_H +#define _LINUX_LIST_BL_H + +#include <linux/list.h> +#include <linux/bit_spinlock.h> + +/* +* Special version of lists, where head of the list has a bit spinlock +* in the lowest bit. This is useful for scalable hash tables without +* increasing memory footprint overhead. +* +* For modification operations, the 0 bit of hlist_bl_head->first +* pointer must be set. +*/ + +#if defined(CONFIG_SMP ) || defined(CONFIG_DEBUG_SPINLOCK ) +#define LIST_BL_LOCKMASK 1UL +#else +#define LIST_BL_LOCKMASK 0UL +#endif + +#ifdef CONFIG_DEBUG_LIST +#define LIST_BL_BUG_ON (x) BUG_ON(x) +#else +#define LIST_BL_BUG_ON (x) +#endif + + +struct hlist_bl_head { +struct hlist_bl_node *first; +}; + +struct hlist_bl_node { +struct hlist_bl_node *next, **pprev; +}; +#define INIT_HLIST_BL_HEAD (ptr) \ + ((ptr)->first = NULL) + +static inline void INIT_HLIST_BL_NODE (struct hlist_bl_node *h) +{ + h->next =NULL ; + h->pprev =NULL ; +} + +#define hlist_bl_entry (ptr, type, member) container_of(ptr,type,member) + +static inline int hlist_bl_unhashed (const struct hlist_bl_node *h) +{ +return !h->pprev; +} + +static inline struct hlist_bl_node *hlist_bl_first (struct hlist_bl_head *h) +{ +return (struct hlist_bl_node *) + ((unsigned long )h->first & ~LIST_BL_LOCKMASK ); +} + +static inline void hlist_bl_set_first (struct hlist_bl_head *h, +struct hlist_bl_node *n) +{ +LIST_BL_BUG_ON ((unsigned long )n &LIST_BL_LOCKMASK ); +LIST_BL_BUG_ON (!bit_spin_is_locked (0 , (unsigned long *)&h->first)); + h->first = (struct hlist_bl_node *)((unsigned long )n |LIST_BL_LOCKMASK ); +} + +static inline int hlist_bl_empty (const struct hlist_bl_head *h) +{ +return !((unsigned long )h->first & ~LIST_BL_LOCKMASK ); +} + +static inline void hlist_bl_add_head (struct hlist_bl_node *n, +struct hlist_bl_head *h) +{ +struct hlist_bl_node *first =hlist_bl_first (h); + + n->next = first; +if (first) + first->pprev = &n->next; + n->pprev = &h->first; +hlist_bl_set_first (h, n); +} + +static inline void __hlist_bl_del (struct hlist_bl_node *n) +{ +struct hlist_bl_node *next = n->next; +struct hlist_bl_node **pprev = n->pprev; + +LIST_BL_BUG_ON ((unsigned long )n &LIST_BL_LOCKMASK ); + +/* pprev may be `first`, so be careful not to lose the lock bit */ + *pprev = (struct hlist_bl_node *) + ((unsigned long )next | + ((unsigned long )*pprev &LIST_BL_LOCKMASK )); +if (next) + next->pprev = pprev; +} + +static inline void hlist_bl_del (struct hlist_bl_node *n) +{ +__hlist_bl_del (n); + n->next =LIST_POISON1 ; + n->pprev =LIST_POISON2 ; +} + +static inline void hlist_bl_del_init (struct hlist_bl_node *n) +{ +if (!hlist_bl_unhashed (n)) { +__hlist_bl_del (n); +INIT_HLIST_BL_NODE (n); + } +} + +/** +* hlist_bl_for_each_entry - iterate over list of given type +* @tpos: the type * to use as a loop cursor. +* @pos: the &struct hlist_node to use as a loop cursor. +* @head: the head for your list. +* @member: the name of the hlist_node within the struct. +* +*/ +#define hlist_bl_for_each_entry (tpos, pos, head, member) \ + for (pos = hlist_bl_first(head); \ + pos && \ + ({ tpos = hlist_bl_entry(pos, typeof(*tpos), member); 1;}); \ + pos = pos->next) + +#endif
include/linux/rculist_bl.h new file +128 -0
128 lines not on this page · raw message
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 01/17] kernel: add bl_list
Quoted 9 lines · On Wed, 29 Sep 2010 22:18:33 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
Introduce a type of hlist that can support the use of the lowest bit in the
hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes....
include/linux/list_bl.h line 43
Quoted 5 lines
include/linux/list_bl.h new file +5
-0+static inline void INIT_HLIST_BL_NODE (struct hlist_bl_node *h) +{ + h->next =NULL ; + h->pprev =NULL ; +}
No need to shout.
include/linux/list_bl.h line 103
Quoted 9 lines
...
include/linux/list_bl.h new file +6
-0+static inline void hlist_bl_del (struct hlist_bl_node *n) +{ +__hlist_bl_del (n); + n->next =LIST_POISON1 ; + n->pprev =LIST_POISON2 ; +}
I'd suggest creating new poison values for hlist_bl's, leave LIST_POISON1 and LIST_POISON2 for list_head (and any other list variants which went and used them :()
Quoted 3 lines
...
Re: [PATCH 01/17] kernel: add bl_list
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Wed, 29 Sep 2010 22:18:33 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
Introduce a type of hlist that can support the use of the lowest bit in the
hlist_head. This will be subsequently used to implement per-bucket bit spinlock for inode hashes....
Quoted 7 lines
include/linux/list_bl.h line 43
Quoted 5 lines
include/linux/list_bl.h new file +5
-0+static inline void INIT_HLIST_BL_NODE (struct hlist_bl_node *h) +{ + h->next =NULL ; + h->pprev =NULL ; +}No need to shout.
Just following the rest of the lists.
Quoted 12 lines
include/linux/list_bl.h line 103
Quoted 8 lines
...
include/linux/list_bl.h new file +6
-0+static inline void hlist_bl_del (struct hlist_bl_node *n) +{ +__hlist_bl_del (n); + n->next =LIST_POISON1 ; + n->pprev =LIST_POISON2 ; +}I'd suggest creating new poison values for hlist_bl's, leave
LIST_POISON1andLIST_POISON2forlist_head(and any other list variants which went and used them :()
I guess they're used for lists, hlists, nulls lists. Would it really help much seeing as you have so many lists anyway? I guess we could do an incremental patch but I'll postpone it for now.
Re: [PATCH 01/17] kernel: add bl_list
Quoted 9 lines · Nick Piggin
Quoted 7 lines
include/linux/list_bl.h line 43
Quoted 5 lines
include/linux/list_bl.h new file +5
-0+static inline void INIT_HLIST_BL_NODE (struct hlist_bl_node *h) +{ + h->next =NULL ; + h->pprev =NULL ; +}No need to shout.
Just following the rest of the lists.
Yes, just following hlist is logical. Anyway, I don't think this or the poisoning value really matter very much.
Re: [PATCH 01/17] kernel: add bl_list
In reply to Dave ChinnerI don't think rculist_bl.h is actually used in this series, so there is no point adding it.
PATCH 2/17 fs: icache lock s_inodes list
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
To allow removal of the inode_lock, we first need to protect the superblock inode list with it's own lock instead of using the inode_lock for this purpose. Nest the new sb_inode_list_lock inside the inode_lock around the list operations it needs to protect.
--- fs/drop_caches.c | 4 ++++ fs/fs-writeback.c | 4 ++++ fs/inode.c | 19 +++++++++++++++++++ fs/notify/inode_mark.c | 2 ++ fs/quota/dquot.c | 6 ++++++ include/linux/writeback.h | 1 + 6 files changed, 36 insertions(+), 0 deletions(-)
fs/drop_caches.c +4 -0
@@ -17,18 +17,22 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)struct inode *inode, *toput_inode =NULL ;spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ))continue ;if (inode->i_mapping->nrpages ==0 )continue ;__iget (inode); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);invalidate_mapping_pages (inode->i_mapping,0 ,-1 );iput (toput_inode); toput_inode = inode;spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); } +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);iput (toput_inode); }
fs/fs-writeback.c +4 -0
@@ -1021,6 +1021,7 @@ static void wait_sb_inodes(struct super_block *sb)WARN_ON (!rwsem_is_locked (&sb->s_umount));spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock);/* * Data integrity sync. Must wait for all pages under writeback,
@@ -1038,6 +1039,7 @@ static void wait_sb_inodes(struct super_block *sb)if (mapping->nrpages ==0 )continue ;__iget (inode); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);/* * We hold a reference to 'inode' so it couldn't have
@@ -1055,7 +1057,9 @@ static void wait_sb_inodes(struct super_block *sb)cond_resched ();spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); } +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);iput (old_inode); }
fs/inode.c +19 -0
@@ -26,6 +26,15 @@#include <linux/posix_acl.h> /* +* Usage: +* sb_inode_list_lock protects: +* s_inodes, i_sb_list +* +* Ordering: +* inode_lock +* sb_inode_list_lock +*/ +/* * This is needed for the following functions: * - inode_has_buffers * - invalidate_inode_buffers
@@ -83,6 +92,7 @@ static struct hlist_head *inode_hashtable __read_mostly;* the i_state of an inode while it is in use.. */ DEFINE_SPINLOCK (inode_lock); +DEFINE_SPINLOCK (sb_inode_list_lock);/* * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -339,7 +349,9 @@ static void dispose_list(struct list_head *head)spin_lock (&inode_lock);hlist_del_init (&inode->i_hash); +spin_lock (&sb_inode_list_lock);list_del_init (&inode->i_sb_list); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);wake_up_inode (inode);
@@ -371,6 +383,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)* shrink_icache_memory() away. */ cond_resched_lock (&inode_lock); +cond_resched_lock (&sb_inode_list_lock); next = next->next;if (tmp == head)
@@ -408,8 +421,10 @@ int invalidate_inodes(struct super_block *sb)down_write (&iprune_sem);spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock);fsnotify_unmount_inodes (&sb->s_inodes); busy =invalidate_list (&sb->s_inodes, &throw_away); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);dispose_list (&throw_away);
@@ -597,7 +612,9 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head, { inodes_stat.nr_inodes++;list_add (&inode->i_list, &inode_in_use); +spin_lock (&sb_inode_list_lock);list_add (&inode->i_sb_list, &sb->s_inodes); +spin_unlock (&sb_inode_list_lock);if (head)hlist_add_head (&inode->i_hash, head); }
@@ -1231,7 +1248,9 @@ static void iput_final(struct inode *inode)hlist_del_init (&inode->i_hash); }list_del_init (&inode->i_list); +spin_lock (&sb_inode_list_lock);list_del_init (&inode->i_sb_list); +spin_unlock (&sb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ; inodes_stat.nr_inodes--;
fs/notify/inode_mark.c +2 -0
@@ -283,6 +283,7 @@ void fsnotify_unmount_inodes(struct list_head *list)* will be added since the umount has begun. Finally, * iprune_mutex keeps shrink_icache_memory() away. */ +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);if (need_iput_tmp)
@@ -296,5 +297,6 @@ void fsnotify_unmount_inodes(struct list_head *list)iput (inode);spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); } }
fs/quota/dquot.c +6 -0
@@ -897,6 +897,7 @@ static void add_dquot_ref(struct super_block *sb, int type)#endif spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ))continue ;
@@ -910,6 +911,7 @@ static void add_dquot_ref(struct super_block *sb, int type)continue ;__iget (inode); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);iput (old_inode);
@@ -921,7 +923,9 @@ static void add_dquot_ref(struct super_block *sb, int type)* keep the reference and iput it later. */ old_inode = inode;spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); } +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);iput (old_inode);
@@ -1004,6 +1008,7 @@ static void remove_dquot_ref(struct super_block *sb, int type,int reserved =0 ;spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {/* * We have to scan also I_NEW inodes because they can already
@@ -1017,6 +1022,7 @@ static void remove_dquot_ref(struct super_block *sb, int type,remove_inode_dquot_ref (inode, type, tofree_head); } } +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);#ifdef CONFIG_QUOTA_DEBUG if (reserved) {
include/linux/writeback.h +1 -0
@@ -10,6 +10,7 @@struct backing_dev_info ;extern spinlock_t inode_lock; +extern spinlock_t sb_inode_list_lock;extern struct list_head inode_in_use;extern struct list_head inode_unused;
Signature · 1 line
1.7.1
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 6 lines · Dave Chinner
From: Nick Piggin <[email protected]>
To allow removal of the
inode_lock, we first need to protect the superblock inode list with it's own lock instead of using theinode_lockfor this purpose. Nest the newsb_inode_list_lockinside theinode_lockaround the list operations it needs to protect.
Is there any good reason not to make this lock per-superblock?
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner
From: Nick Piggin <[email protected]>
To allow removal of the
inode_lock, we first need to protect the superblock inode list with it's own lock instead of using theinode_lockfor this purpose. Nest the newsb_inode_list_lockinside theinode_lockaround the list operations it needs to protect.Is there any good reason not to make this lock per-superblock?
Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what inode_lock used to.
If we did want to make it per-superblock, that would come at the last part of the series, where inode_lock is removed and steps are being taken to improve scalability and locking.
But I don't see why we want to make it per-superblock really anyway. We want to have scalability within a single superblock, so per CPU locks are needed. Once we have those, per-superblock doesn't really buy much.
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 3 lines · Nick Piggin
Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what
inode_lockused to.
As seen by Dave's respin making it per-sb was just as easy as making it global. And it really is the logical synchronization domain.
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Nick Piggin
Because in the first part of the inode lock series, it is breaking locks in obvious small steps as possible, by adding global locks protecting bits of what
inode_lockused to.As seen by Dave's respin making it per-sb was just as easy as making it global. And it really is the logical synchronization domain.
If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.
And the entire idea of the first half of the inode series is that it starts simple and just uses globals to demonstrate the locking steps. It's obviously not supposed to be a "production" locking model so I prefer it to be like that.
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 3 lines · Nick Piggin
If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.
Right now the patches split up the inode lock and do not add per-cpu magic. It's not any more work to move from per-sb lists to per-cpu locking if we eventually do it than moving from global to per-cpu.
I'm not entirely convinced moving s_inodes to a per-cpu list is a good idea. For now per-sb is just fine for disk filesystems as they have much more fs-wide cachelines they touch for inode creatation/deletion anyway, and for sockets/pipes a variant of your patch to not ever add them to s_inodes sounds like the better approach.
If we eventually hit the limit for disk filesystems I have some better ideas to solve this. One is to abuse whatever data sturcture we use for the inode hash also for iterating over all inodes - we only iterate over them in very few places, and none of them is a fast path.
Re: [PATCH 02/17] fs: icache lock s_inodes list
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Nick Piggin
If you want it to be scalable within a single sb, it needs to be per cpu. If it is per-cpu it does not need to be per-sb as well which just adds bloat.
Right now the patches split up the inode lock and do not add per-cpu magic. It's not any more work to move from per-sb lists to per-cpu locking if we eventually do it than moving from global to per-cpu.
But it's more work to do per-sb lists than a global list, and as I'm going to a per-cpu locking anyway it's a strange transition to go from per-sb to per-cpu (rather than per-sb, per-cpu). In short, the fact that I build up the locking transformations starting with global locks is just not something that can be held against my patch set (unless you really disagree with the whole concept of how the series is structured).
Quoted 6 lines
I'm not entirely convinced moving
s_inodesto a per-cpu list is a good idea. For now per-sb is just fine for disk filesystems as they have much more fs-wide cachelines they touch for inode creatation/deletion anyway, and for sockets/pipes a variant of your patch to not ever add them tos_inodessounds like the better approach.
Traditional filesystems on slow spinning disk are not the main problem. It's very fast ssds and storage servers. XFS actually with its per-AG lock splitting can already have problems on small servers with not-incredibly-fast storage with per-sb scalability bottlenecks.
And if the VFS is not scalable, then the contention doesn't even get pushed into the filesystem so the fs developers never even _see_ the locking problems to fix them.
I'm telling you it will be increasingly a problem because cores and storage speeds continue to increase, and also people want to manage more storage with fewer filesystems. It's obvious that it will be a problem.
I've already got per cpu locking in vfsmounts and files lock, so it's not magic.
Quoted 4 lines
If we eventually hit the limit for disk filesystems I have some better ideas to solve this. One is to abuse whatever data sturcture we use for the inode hash also for iterating over all inodes - we only iterate over them in very few places, and none of them is a fast path.
Doing your handwaving about changing data types and better ideas is just not helpful. _If_ you do have some better ideas, and _if_ we change the data structure, _then_ it's trivial to change from percpu locking to your better idea. It just doesn't work as an argument to slow progress.
PATCH 3/17 fs: icache lock inode hash
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Currently the inode hash lists are protected by the inode_lock. To allow removal of the inode_lock, we need to protect the inode hash table lists with a new lock. Nest the new inode_hash_lock inside the inode_lock to protect the hash lists.
--- fs/inode.c | 33 ++++++++++++++++++++++++++++++++- include/linux/writeback.h | 1 + 2 files changed, 33 insertions(+), 1 deletions(-)
fs/inode.c +32 -1
@@ -29,10 +29,14 @@* Usage: * sb_inode_list_lock protects: * s_inodes, i_sb_list +* inode_hash_lock protects: +* inode hash table, i_hash * * Ordering: * inode_lock * sb_inode_list_lock +* inode_lock +* inode_hash_lock */ /* * This is needed for the following functions:
@@ -93,6 +97,7 @@ static struct hlist_head *inode_hashtable __read_mostly;*/ DEFINE_SPINLOCK (inode_lock);DEFINE_SPINLOCK (sb_inode_list_lock); +DEFINE_SPINLOCK (inode_hash_lock);/* * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -348,7 +353,9 @@ static void dispose_list(struct list_head *head)evict (inode);spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash); +spin_unlock (&inode_hash_lock);spin_lock (&sb_inode_list_lock);list_del_init (&inode->i_sb_list);spin_unlock (&sb_inode_list_lock);
@@ -557,17 +564,20 @@ static struct inode *find_inode(struct super_block *sb,struct inode *inode =NULL ; repeat: +spin_lock (&inode_hash_lock);hlist_for_each_entry (inode, node, head, i_hash) {if (inode->i_sb != sb)continue ;if (!test (inode, data))continue ;if (inode->i_state & (I_FREEING |I_WILL_FREE )) { +spin_unlock (&inode_hash_lock);__wait_on_freeing_inode (inode); goto repeat; }break ; } +spin_unlock (&inode_hash_lock);return node ? inode :NULL ; }
@@ -582,17 +592,20 @@ static struct inode *find_inode_fast(struct super_block *sb,struct inode *inode =NULL ; repeat: +spin_lock (&inode_hash_lock);hlist_for_each_entry (inode, node, head, i_hash) {if (inode->i_ino != ino)continue ;if (inode->i_sb != sb)continue ;if (inode->i_state & (I_FREEING |I_WILL_FREE )) { +spin_unlock (&inode_hash_lock);__wait_on_freeing_inode (inode); goto repeat; }break ; } +spin_unlock (&inode_hash_lock);return node ? inode :NULL ; }
@@ -615,8 +628,11 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head,spin_lock (&sb_inode_list_lock);list_add (&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock); -if (head) +if (head) { +spin_lock (&inode_hash_lock);hlist_add_head (&inode->i_hash, head); +spin_unlock (&inode_hash_lock); + } }/**
@@ -1094,7 +1110,9 @@ int insert_inode_locked(struct inode *inode)while (1 ) {struct hlist_node *node;struct inode *old =NULL ; +spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_for_each_entry (old, node, head, i_hash) {if (old->i_ino != ino)continue ;
@@ -1106,9 +1124,11 @@ int insert_inode_locked(struct inode *inode) }if (likely (!node)) {hlist_add_head (&inode->i_hash, head); +spin_unlock (&inode_hash_lock);spin_unlock (&inode_lock);return 0 ; } +spin_unlock (&inode_hash_lock);__iget (old);spin_unlock (&inode_lock);wait_on_inode (old);
@@ -1134,6 +1154,7 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval,struct inode *old =NULL ;spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_for_each_entry (old, node, head, i_hash) {if (old->i_sb != sb)continue ;
@@ -1145,9 +1166,11 @@ int insert_inode_locked4(struct inode *inode, unsigned long hashval, }if (likely (!node)) {hlist_add_head (&inode->i_hash, head); +spin_unlock (&inode_hash_lock);spin_unlock (&inode_lock);return 0 ; } +spin_unlock (&inode_hash_lock);__iget (old);spin_unlock (&inode_lock);wait_on_inode (old);
@@ -1172,7 +1195,9 @@ void __insert_inode_hash(struct inode *inode, unsigned long hashval) {struct hlist_head *head = inode_hashtable +hash (inode->i_sb, hashval);spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_add_head (&inode->i_hash, head); +spin_unlock (&inode_hash_lock);spin_unlock (&inode_lock); }EXPORT_SYMBOL (__insert_inode_hash);
@@ -1186,7 +1211,9 @@ EXPORT_SYMBOL(__insert_inode_hash);void remove_inode_hash (struct inode *inode) {spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash); +spin_unlock (&inode_hash_lock);spin_unlock (&inode_lock); }EXPORT_SYMBOL (remove_inode_hash);
@@ -1245,7 +1272,9 @@ static void iput_final(struct inode *inode)WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_WILL_FREE ; inodes_stat.nr_unused--; +spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash); +spin_unlock (&inode_hash_lock); }list_del_init (&inode->i_list);spin_lock (&sb_inode_list_lock);
@@ -1257,7 +1286,9 @@ static void iput_final(struct inode *inode)spin_unlock (&inode_lock);evict (inode);spin_lock (&inode_lock); +spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash); +spin_unlock (&inode_hash_lock);spin_unlock (&inode_lock);wake_up_inode (inode);BUG_ON (inode->i_state != (I_FREEING |I_CLEAR ));
include/linux/writeback.h +1 -0
@@ -11,6 +11,7 @@ struct backing_dev_info;extern spinlock_t inode_lock;extern spinlock_t sb_inode_list_lock; +extern spinlock_t inode_hash_lock;extern struct list_head inode_in_use;extern struct list_head inode_unused;
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 03/17] fs: icache lock inode hash
Quoted 3 lines · On Wed, 29 Sep 2010 22:18:35 +1000 Dave Chinner <[email protected]> wrote:
DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);
I assume these all go away later on. If not, they'll probably all land in the same cacheline!
Re: [PATCH 03/17] fs: icache lock inode hash
Quoted 8 lines · Andrew Morton
Quoted 3 lines · On Wed, 29 Sep 2010 22:18:35 +1000 Dave Chinner <[email protected]> wrote:
DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);I assume these all go away later on. If not, they'll probably all land in the same cacheline!
indeed, the hash lock goes away in this series, as does the inode_lock. I'll check the other new locks don't land in the same cacheline.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 03/17] fs: icache lock inode hash
In reply to Dave ChinnerQuoted 6 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Currently the inode hash lists are protected by the
inode_lock. To allow removal of theinode_lock, we need to protect the inode hash table lists with a new lock. Nest the newinode_hash_lockinside theinode_lockto protect the hash lists.
There is no reason to make inode_hash_lock global, it's only used inside inode.c
Re: [PATCH 03/17] fs: icache lock inode hash
Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Currently the inode hash lists are protected by the
inode_lock. To allow removal of theinode_lock, we need to protect the inode hash table lists with a new lock. Nest the newinode_hash_lockinside theinode_lockto protect the hash lists.There is no reason to make
inode_hash_lockglobal, it's only used insideinode.c
It was like that while hugetlbfs still used it. Fixed, thanks.
PATCH 4/17 fs: icache lock i_state
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
We currently protect the per-inode state flags with the inode_lock. Using a global lock to protect per-object state is overkill when we coul duse a per-inode lock to protect the state. Use the inode->i_lock for this, and wrap all the state changes and checks with the inode->i_lock.
--- fs/drop_caches.c | 9 ++++-- fs/fs-writeback.c | 29 +++++++++++++++-- fs/inode.c | 86 +++++++++++++++++++++++++++++++++++++++++++++------- fs/nilfs2/gcdat.c | 1 + fs/quota/dquot.c | 12 ++++--- 5 files changed, 113 insertions(+), 24 deletions(-)
fs/drop_caches.c +6 -3
@@ -19,11 +19,14 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) -continue ; -if (inode->i_mapping->nrpages ==0 ) +spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ) + || inode->i_mapping->nrpages ==0 ) { +spin_unlock (&inode->i_lock);continue ; + }__iget (inode); +spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);invalidate_mapping_pages (inode->i_mapping,0 ,-1 );
fs/fs-writeback.c +25 -4
@@ -280,10 +280,12 @@ static void inode_wait_for_writeback(struct inode *inode)wait_queue_head_t *wqh; wqh =bit_waitqueue (&inode->i_state, __I_SYNC); -while (inode->i_state &I_SYNC ) { +while (inode->i_state &I_SYNC ) { +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);__wait_on_bit (wqh, &wq, inode_wait,TASK_UNINTERRUPTIBLE );spin_lock (&inode_lock); +spin_lock (&inode->i_lock); } }
@@ -337,6 +339,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)/* Set I_SYNC, reset I_DIRTY_PAGES */ inode->i_state |=I_SYNC ; inode->i_state &= ~I_DIRTY_PAGES ; +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); ret =do_writepages (mapping, wbc);
@@ -358,8 +361,10 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)* write_inode() */ spin_lock (&inode_lock); +spin_lock (&inode->i_lock); dirty = inode->i_state &I_DIRTY ; inode->i_state &= ~(I_DIRTY_SYNC |I_DIRTY_DATASYNC ); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);/* Don't write the inode if only I_DIRTY_PAGES was set */ if (dirty & (I_DIRTY_SYNC |I_DIRTY_DATASYNC )) {
@@ -369,6 +374,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc) }spin_lock (&inode_lock); +spin_lock (&inode->i_lock); inode->i_state &= ~I_SYNC ;if (!(inode->i_state &I_FREEING )) {if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {
@@ -479,7 +485,9 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,return 0 ; } +spin_lock (&inode->i_lock);if (inode->i_state & (I_NEW |I_WILL_FREE )) { +spin_unlock (&inode->i_lock);requeue_io (inode);continue ; }
@@ -487,8 +495,10 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,* Was this inode dirtied after sync_sb_inodes was called? * This keeps sync from extra jobs and livelock. */ -if (inode_dirtied_after (inode, wbc->wb_start)) +if (inode_dirtied_after (inode, wbc->wb_start)) { +spin_unlock (&inode->i_lock);return 1 ; + }BUG_ON (inode->i_state &I_FREEING );__iget (inode);
@@ -501,6 +511,7 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,*/ redirty_tail (inode); } +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);iput (inode);cond_resched ();
@@ -936,6 +947,7 @@ void __mark_inode_dirty(struct inode *inode, int flags)block_dump___mark_inode_dirty (inode);spin_lock (&inode_lock); +spin_lock (&inode->i_lock);if ((inode->i_state & flags) != flags) {const int was_dirty = inode->i_state &I_DIRTY ;
@@ -986,6 +998,7 @@ void __mark_inode_dirty(struct inode *inode, int flags) } } out: +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);if (wakeup_bdi)
@@ -1033,12 +1046,16 @@ static void wait_sb_inodes(struct super_block *sb)list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) -continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )continue ; +spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) { +spin_unlock (&inode->i_lock); +continue ; + }__iget (inode); +spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);/*
@@ -1165,7 +1182,9 @@ int write_inode_now(struct inode *inode, int sync)might_sleep ();spin_lock (&inode_lock); +spin_lock (&inode->i_lock); ret =writeback_single_inode (inode, &wbc); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);if (sync)inode_sync_wait (inode);
@@ -1189,7 +1208,9 @@ int sync_inode(struct inode *inode, struct writeback_control *wbc)int ret;spin_lock (&inode_lock); +spin_lock (&inode->i_lock); ret =writeback_single_inode (inode, wbc); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);return ret; }
fs/inode.c +74 -12
312 lines not on this page · raw message
fs/nilfs2/gcdat.c +1 -0
@@ -27,6 +27,7 @@#include "page.h" #include "mdt.h" +/* XXX: what protects i_state? */ int nilfs_init_gcdat_inode (struct the_nilfs *nilfs) {struct inode *dat = nilfs->ns_dat, *gcdat = nilfs->ns_gc_dat;
fs/quota/dquot.c +7 -5
@@ -899,18 +899,20 @@ static void add_dquot_ref(struct super_block *sb, int type)spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) { -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +spin_lock (&inode->i_lock); +if ((inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ) || + !atomic_read (&inode->i_writecount) || + !dqinit_needed (inode, type))) { +spin_unlock (&inode->i_lock);continue ; + }#ifdef CONFIG_QUOTA_DEBUG if (unlikely (inode_get_rsv_space (inode) >0 )) reserved =1 ;#endif -if (!atomic_read (&inode->i_writecount)) -continue ; -if (!dqinit_needed (inode, type)) -continue ;__iget (inode); +spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);
Signature · 1 line
1.7.1
Re: [PATCH 04/17] fs: icache lock i_state
fs/drop_caches.c static void drop_pagecache_sb(struct super_block *sb, void *unused) · line 24
Quoted 3 lines
fs/drop_caches.c +3
-0+spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ) + || inode->i_mapping->nrpages ==0 ) {
This is some pretty strange formatting.
if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) ||
inode->i_mapping->nrpages == 0) {would be more standard.
fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1056
Quoted 13 lines
fs/fs-writeback.c +5
-2list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping; -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) -continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )continue ; +spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) { +spin_unlock (&inode->i_lock); +continue ; + }
Can we access the mapping safely when the inode isn't actually fully setup? Even if we can I'd rather not introduce this change hidden inside an unrelated patch.
Re: [PATCH 04/17] fs: icache lock i_state
Quoted 15 lines · Christoph Hellwig
fs/drop_caches.c static void drop_pagecache_sb(struct super_block *sb, void *unused) · line 24
Quoted 3 lines
fs/drop_caches.c +3
-0+spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW ) + || inode->i_mapping->nrpages ==0 ) {This is some pretty strange formatting.
if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) || inode->i_mapping->nrpages == 0) {would be more standard.
Quoted 3 lines
fs/fs-writeback.c +0
-0list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping;
Quoted 14 lines
fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1056
Quoted 10 lines
fs/fs-writeback.c +5
-2-if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) -continue ; mapping = inode->i_mapping;if (mapping->nrpages ==0 )continue ; +spin_lock (&inode->i_lock); +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) { +spin_unlock (&inode->i_lock); +continue ; + }Can we access the mapping safely when the inode isn't actually fully setup? Even if we can I'd rather not introduce this change hidden inside an unrelated patch.
Good point, fixed.
PATCH 5/17 fs: icache lock i_count
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
the inode reference count is currently an atomic variable so that it can be sampled/modified outside the inode_lock. However, the inode_lock is still needed to synchronise the final reference count and checks against the inode state.
With the inode state now protected by a per-inode lock, we can protect the inode reference count with the same per-inode lock and still check and modify the count and state atomically. By using the i_lock in this manner, we no longer need an atomic reference count field, so convert all the reference counting to be protected by the inode->i_lock and change i_count to a non-atomic variable.
--- arch/powerpc/platforms/cell/spufs/file.c | 2 +- drivers/staging/pohmelfs/inode.c | 14 +++++++---- fs/9p/vfs_inode.c | 4 ++- fs/affs/inode.c | 4 ++- fs/afs/dir.c | 4 ++- fs/anon_inodes.c | 4 ++- fs/bfs/dir.c | 4 ++- fs/block_dev.c | 15 ++++++++++-- fs/btrfs/inode.c | 23 +++++++++++++++---- fs/ceph/mds_client.c | 2 +- fs/cifs/inode.c | 2 +- fs/coda/dir.c | 4 ++- fs/exofs/inode.c | 12 +++++++-- fs/exofs/namei.c | 4 ++- fs/ext2/namei.c | 4 ++- fs/ext3/ialloc.c | 4 +- fs/ext3/namei.c | 4 ++- fs/ext4/ialloc.c | 4 +- fs/ext4/namei.c | 4 ++- fs/fs-writeback.c | 4 +- fs/gfs2/ops_inode.c | 4 ++- fs/hfsplus/dir.c | 4 ++- fs/hpfs/inode.c | 2 +- fs/inode.c | 36 ++++++++++++++++++++--------- fs/jffs2/dir.c | 8 +++++- fs/jfs/jfs_txnmgr.c | 4 ++- fs/jfs/namei.c | 4 ++- fs/libfs.c | 4 ++- fs/locks.c | 2 +- fs/logfs/dir.c | 4 ++- fs/logfs/readwrite.c | 6 ++++- fs/minix/namei.c | 4 ++- fs/namei.c | 7 ++++- fs/nfs/dir.c | 4 ++- fs/nfs/getroot.c | 4 ++- fs/nfs/inode.c | 4 +- fs/nfs/nfs4state.c | 2 +- fs/nfs/write.c | 2 +- fs/nilfs2/mdt.c | 2 +- fs/nilfs2/namei.c | 4 ++- fs/notify/inode_mark.c | 21 ++++++++++------ fs/ntfs/super.c | 4 ++- fs/ocfs2/namei.c | 4 ++- fs/reiserfs/namei.c | 4 ++- fs/reiserfs/stree.c | 2 +- fs/sysv/namei.c | 4 ++- fs/ubifs/dir.c | 4 ++- fs/ubifs/super.c | 2 +- fs/udf/namei.c | 4 ++- fs/ufs/namei.c | 4 ++- fs/xfs/linux-2.6/xfs_iops.c | 4 ++- fs/xfs/linux-2.6/xfs_trace.h | 2 +- fs/xfs/xfs_inode.h | 6 +++- include/linux/fs.h | 2 +- ipc/mqueue.c | 7 ++++- kernel/futex.c | 4 ++- mm/shmem.c | 4 ++- net/socket.c | 4 ++- 58 files changed, 224 insertions(+), 95 deletions(-)
arch/powerpc/platforms/cell/spufs/file.c +1 -1
@@ -1549,7 +1549,7 @@ static int spufs_mfc_open(struct inode *inode, struct file *file)if (ctx->owner != current->mm)return -EINVAL ; -if (atomic_read (&inode->i_count) !=1 ) +if (inode->i_count !=1 )return -EBUSY ;mutex_lock (&ctx->mapping_lock);
drivers/staging/pohmelfs/inode.c +9 -5
@@ -1289,13 +1289,15 @@ static void pohmelfs_put_super(struct super_block *sb)dprintk ("%s: ino: %llu, pi: %p, inode: %p, count: %u.\n" , __func__, pi->ino, pi, inode, count); -if (atomic_read (&inode->i_count) != count) { +spin_lock (&inode->i_lock); + if (inode->i_count) != count) {printk ("%s: ino: %llu, pi: %p, inode: %p, count: %u, i_count: %d.\n" , __func__, pi->ino, pi, inode, count, -atomic_read (&inode->i_count)); - count =atomic_read (&inode->i_count); + inode->i_count); + count = inode->i_count; in_drop_list++; } +spin_unlock (&inode->i_lock);while (count--)iput (&pi->vfs_inode);
@@ -1305,7 +1307,7 @@ static void pohmelfs_put_super(struct super_block *sb) pi =POHMELFS_I (inode);dprintk ("%s: ino: %llu, pi: %p, inode: %p, i_count: %u.\n" , - __func__, pi->ino, pi, inode,atomic_read (&inode->i_count)); + __func__, pi->ino, pi, inode, inode->i_count);/* * These are special inodes, they were created during
@@ -1313,7 +1315,9 @@ static void pohmelfs_put_super(struct super_block *sb)* so they live here with reference counter being 1 and prevent * umount from succeed since it believes that they are busy. */ - count =atomic_read (&inode->i_count); +spin_lock (&inode->i_lock); + count = inode->i_count; +spin_lock (&inode->i_lock);if (count) {list_del_init (&inode->i_sb_list);while (count--)
fs/9p/vfs_inode.c +3 -1
@@ -1791,7 +1791,9 @@ v9fs_vfs_link_dotl(struct dentry *old_dentry, struct inode *dir,/* Caching disabled. No need to get upto date stat info. * This dentry will be released immediately. So, just i_count++ */ -atomic_inc (&old_dentry->d_inode->i_count); +spin_lock (&old_dentry->d_inode->i_lock); + old_dentry->d_inode->i_count++; +spin_unlock (&old_dentry->d_inode->i_lock); } dentry->d_op = old_dentry->d_op;
fs/affs/inode.c +3 -1
@@ -388,7 +388,9 @@ affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s3affs_adjust_checksum (inode_bh, block -be32_to_cpu (chain));mark_buffer_dirty_inode (inode_bh, inode); inode->i_nlink =2 ; -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); }affs_fix_checksum (sb, bh);mark_buffer_dirty_inode (bh, inode);
fs/afs/dir.c +3 -1
@@ -1045,7 +1045,9 @@ static int afs_link(struct dentry *from, struct inode *dir,if (ret <0 ) goto link_error; -atomic_inc (&vnode->vfs_inode.i_count); +spin_lock (&vnode->vfs_inode.i_lock); + vnode->vfs_inode.i_count++; +spin_unlock (&vnode->vfs_inode.i_lock);d_instantiate (dentry, &vnode->vfs_inode);key_put (key);_leave (" = 0" );
fs/anon_inodes.c +3 -1
@@ -114,7 +114,9 @@ struct file *anon_inode_getfile(const char *name,* so we can avoid doing an igrab() and we can use an open-coded * atomic_inc(). */ -atomic_inc (&anon_inode_inode->i_count); +spin_lock (&anon_inode_inode->i_lock); + anon_inode_inode->i_count++; +spin_unlock (&anon_inode_inode->i_lock); path.dentry->d_op = &anon_inodefs_dentry_operations;d_instantiate (path.dentry, anon_inode_inode);
fs/bfs/dir.c +3 -1
@@ -176,7 +176,9 @@ static int bfs_link(struct dentry *old, struct inode *dir,inc_nlink (inode); inode->i_ctime =CURRENT_TIME_SEC ;mark_inode_dirty (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (new, inode);mutex_unlock (&info->bfs_lock);return 0 ;
fs/block_dev.c +12 -3
@@ -550,7 +550,12 @@ EXPORT_SYMBOL(bdget);*/ struct block_device *bdgrab (struct block_device *bdev) { -atomic_inc (&bdev->bd_inode->i_count); +struct inode *inode = bdev->bd_inode; + +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); +return bdev; }
@@ -580,7 +585,9 @@ static struct block_device *bd_acquire(struct inode *inode)spin_lock (&bdev_lock); bdev = inode->i_bdev;if (bdev) { -atomic_inc (&bdev->bd_inode->i_count); +spin_lock (&inode->i_lock); + bdev->bd_inode->i_count++; +spin_unlock (&inode->i_lock);spin_unlock (&bdev_lock);return bdev; }
@@ -596,7 +603,9 @@ static struct block_device *bd_acquire(struct inode *inode)* So, we can access it via ->i_mapping always * without igrab(). */ -atomic_inc (&bdev->bd_inode->i_count); +spin_lock (&inode->i_lock); + bdev->bd_inode->i_count++; +spin_unlock (&inode->i_lock); inode->i_bdev = bdev; inode->i_mapping = bdev->bd_inode->i_mapping;list_add (&inode->i_devices, &bdev->bd_inodes);
fs/btrfs/inode.c +18 -5
@@ -1964,8 +1964,13 @@ void btrfs_add_delayed_iput(struct inode *inode)struct btrfs_fs_info *fs_info =BTRFS_I (inode)->root->fs_info;struct delayed_iput *delayed; -if (atomic_add_unless (&inode->i_count,-1 ,1 )) +spin_lock (&inode->i_lock); +if (inode->i_count >1 ) { + inode->i_count--; +spin_unlock (&inode->i_lock);return ; + } +spin_unlock (&inode->i_lock); delayed =kmalloc (sizeof (*delayed),GFP_NOFS | __GFP_NOFAIL); delayed->inode = inode;
@@ -2718,11 +2723,17 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode *dir,return ERR_PTR (-ENOSPC );/* check if there is someone else holds reference */ -if (S_ISDIR (inode->i_mode) &&atomic_read (&inode->i_count) >1 ) +spin_lock (&inode->i_lock); +if (S_ISDIR (inode->i_mode) && inode->i_count >1 ) { +spin_unlock (&inode->i_lock);return ERR_PTR (-ENOSPC ); + } -if (atomic_read (&inode->i_count) >2 ) +if (inode->i_count >2 ) { +spin_unlock (&inode->i_lock);return ERR_PTR (-ENOSPC ); + } +spin_unlock (&inode->i_lock);if (xchg (&root->fs_info->enospc_unlink,1 ))return ERR_PTR (-ENOSPC );
@@ -3939,7 +3950,7 @@ again: inode =igrab (&entry->vfs_inode);if (inode) {spin_unlock (&root->inode_lock); -if (atomic_read (&inode->i_count) >1 ) +if (inode->i_count >1 )d_prune_aliases (inode);/* * btrfs_drop_inode will have it removed from
@@ -4758,7 +4769,9 @@ static int btrfs_link(struct dentry *old_dentry, struct inode *dir, }btrfs_set_trans_block_group (trans, dir); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); err =btrfs_add_nondir (trans, dentry, inode,1 , index);
fs/ceph/mds_client.c +1 -1
@@ -1102,7 +1102,7 @@ static int trim_caps_cb(struct inode *inode, struct ceph_cap *cap, void *arg)spin_unlock (&inode->i_lock);d_prune_aliases (inode);dout ("trim_caps_cb %p cap %p pruned, count now %d\n" , - inode, cap,atomic_read (&inode->i_count)); + inode, cap, inode->i_count);return 0 ; }
fs/cifs/inode.c +1 -1
@@ -1639,7 +1639,7 @@ int cifs_revalidate_dentry(struct dentry *dentry) }cFYI (1 ,"Revalidate: %s inode 0x%p count %d dentry: 0x%p d_time %ld " -"jiffies %ld" , full_path, inode, inode->i_count.counter, +"jiffies %ld" , full_path, inode, inode->i_count, dentry, dentry->d_time, jiffies);if (CIFS_SB (sb)->tcon->unix_ext)
fs/coda/dir.c +3 -1
@@ -303,7 +303,9 @@ static int coda_link(struct dentry *source_de, struct inode *dir_inode, }coda_dir_update_mtime (dir_inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (de, inode);inc_nlink (inode);
fs/exofs/inode.c +9 -3
@@ -1101,7 +1101,9 @@ static void create_done(struct exofs_io_state *ios, void *p)set_obj_created (oi); -atomic_dec (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count--; +spin_unlock (&inode->i_lock);wake_up (&oi->i_wq); }
@@ -1154,14 +1156,18 @@ struct inode *exofs_new_inode(struct inode *dir, int mode)/* increment the refcount so that the inode will still be around when we * reach the callback */ -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); ios->done = create_done; ios->private = inode; ios->cred = oi->i_cred; ret =exofs_sbi_create (ios);if (ret) { -atomic_dec (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count--; +spin_unlock (&inode->i_lock);exofs_put_io_state (ios);return ERR_PTR (ret); }
fs/exofs/namei.c +3 -1
@@ -153,7 +153,9 @@ static int exofs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);return exofs_add_nondir (dentry, inode); }
fs/ext2/namei.c +3 -1
@@ -206,7 +206,9 @@ static int ext2_link (struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); err =ext2_add_link (dentry, inode);if (!err) {
fs/ext3/ialloc.c +2 -2
@@ -100,9 +100,9 @@ void ext3_free_inode (handle_t *handle, struct inode * inode)struct ext3_sb_info *sbi;int fatal =0 , err; -if (atomic_read (&inode->i_count) >1 ) { +if (inode->i_count >1 ) {printk ("ext3_free_inode: inode has count=%d\n" , -atomic_read (&inode->i_count)); + inode->i_count);return ; }if (inode->i_nlink) {
fs/ext3/namei.c +3 -1
@@ -2260,7 +2260,9 @@ retry: inode->i_ctime =CURRENT_TIME_SEC ;inc_nlink (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); err =ext3_add_entry (handle, dentry, inode);if (!err) {
fs/ext4/ialloc.c +2 -2
@@ -189,9 +189,9 @@ void ext4_free_inode(handle_t *handle, struct inode *inode)struct ext4_sb_info *sbi;int fatal =0 , err, count, cleared; -if (atomic_read (&inode->i_count) >1 ) { +if (inode->i_count >1 ) {printk (KERN_ERR "ext4_free_inode: inode has count=%d\n" , -atomic_read (&inode->i_count)); + inode->i_count);return ; }if (inode->i_nlink) {
fs/ext4/namei.c +3 -1
@@ -2312,7 +2312,9 @@ retry: inode->i_ctime =ext4_current_time (inode);ext4_inc_count (handle, inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); err =ext4_add_entry (handle, dentry, inode);if (!err) {
fs/fs-writeback.c +2 -2
@@ -309,7 +309,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)unsigned dirty;int ret; -if (!atomic_read (&inode->i_count)) +if (!inode->i_count)WARN_ON (!(inode->i_state & (I_WILL_FREE |I_FREEING )));else WARN_ON (inode->i_state &I_WILL_FREE );
@@ -406,7 +406,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)* completion. */ redirty_tail (inode); - }else if (atomic_read (&inode->i_count)) { + }else if (inode->i_count) {/* * The inode is clean, inuse */
fs/gfs2/ops_inode.c +3 -1
@@ -253,7 +253,9 @@ out_parent:gfs2_holder_uninit (ghs);gfs2_holder_uninit (ghs +1 );if (!error) { -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (dentry, inode);mark_inode_dirty (inode); }
fs/hfsplus/dir.c +3 -1
@@ -301,7 +301,9 @@ static int hfsplus_link(struct dentry *src_dentry, struct inode *dst_dir,inc_nlink (inode);hfsplus_instantiate (dst_dentry, inode, cnid); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); inode->i_ctime =CURRENT_TIME_SEC ;mark_inode_dirty (inode);HFSPLUS_SB (sb).file_count++;
fs/hpfs/inode.c +1 -1
@@ -183,7 +183,7 @@ void hpfs_write_inode(struct inode *i)struct hpfs_inode_info *hpfs_inode =hpfs_i (i);struct inode *parent;if (i->i_ino ==hpfs_sb (i->i_sb)->sb_root)return ; -if (hpfs_inode->i_rddir_off && !atomic_read (&i->i_count)) { +if (hpfs_inode->i_rddir_off && !i->i_count) {if (*hpfs_inode->i_rddir_off)printk ("HPFS: write_inode: some position still there\n" );kfree (hpfs_inode->i_rddir_off); hpfs_inode->i_rddir_off =NULL ;
fs/inode.c +25 -11
@@ -32,14 +32,13 @@* inode_hash_lock protects: * inode hash table, i_hash * inode->i_lock protects: -* i_state +* i_state, i_count * * Ordering: * inode_lock * sb_inode_list_lock * inode->i_lock -* inode_lock -* inode_hash_lock +* inode_hash_lock */ /* * This is needed for the following functions:
@@ -150,7 +149,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) inode->i_sb = sb; inode->i_blkbits = sb->s_blocksize_bits; inode->i_flags =0 ; -atomic_set (&inode->i_count,1 ); + inode->i_count =1 ; inode->i_op = &empty_iops; inode->i_fop = &empty_fops; inode->i_nlink =1 ;
@@ -301,7 +300,8 @@ void __iget(struct inode *inode) {assert_spin_locked (&inode->i_lock); -if (atomic_inc_return (&inode->i_count) !=1 ) + inode->i_count++; +if (inode->i_count >1 )return ;if (!(inode->i_state & (I_DIRTY |I_SYNC )))
@@ -407,7 +407,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)continue ; }invalidate_inode_buffers (inode); -if (!atomic_read (&inode->i_count)) { +if (!inode->i_count) {list_move (&inode->i_list, dispose);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;
@@ -457,7 +457,7 @@ static int can_unuse(struct inode *inode)return 0 ;if (inode_has_buffers (inode))return 0 ; -if (atomic_read (&inode->i_count)) +if (inode->i_count)return 0 ;if (inode->i_data.nrpages)return 0 ;
@@ -495,7 +495,7 @@ static void prune_icache(int nr_to_scan) inode = list_entry(inode_unused.prev, struct inode, i_list);spin_lock (&inode->i_lock); -if (inode->i_state ||atomic_read (&inode->i_count)) { +if (inode->i_state || inode->i_count) {list_move (&inode->i_list, &inode_unused);spin_unlock (&inode->i_lock);continue ;
@@ -1307,8 +1307,6 @@ static void iput_final(struct inode *inode)else drop =generic_drop_inode (inode); -spin_lock (&sb_inode_list_lock); -spin_lock (&inode->i_lock);if (!drop) {if (!(inode->i_state & (I_DIRTY |I_SYNC )))list_move (&inode->i_list, &inode_unused);
@@ -1368,8 +1366,24 @@ void iput(struct inode *inode)if (inode) {BUG_ON (inode->i_state &I_CLEAR ); -if (atomic_dec_and_lock (&inode->i_count, &inode_lock)) +retry: +spin_lock (&inode->i_lock); +if (inode->i_count ==1 ) { +if (!spin_trylock (&inode_lock)) { +spin_unlock (&inode->i_lock); + goto retry; + } +if (!spin_trylock (&sb_inode_list_lock)) { +spin_unlock (&inode_lock); +spin_unlock (&inode->i_lock); + goto retry; + } + inode->i_count--;iput_final (inode); + }else { + inode->i_count--; +spin_unlock (&inode->i_lock); + } } }EXPORT_SYMBOL (iput);
fs/jffs2/dir.c +6 -2
@@ -289,7 +289,9 @@ static int jffs2_link (struct dentry *old_dentry, struct inode *dir_i, struct demutex_unlock (&f->sem);d_instantiate (dentry, old_dentry->d_inode); dir_i->i_mtime = dir_i->i_ctime =ITIME (now); -atomic_inc (&old_dentry->d_inode->i_count); +spin_lock (&old_dentry->d_inode->i_lock); + old_dentry->d_inode->i_count++; +spin_unlock (&old_dentry->d_inode->i_lock); }return ret; }
@@ -864,7 +866,9 @@ static int jffs2_rename (struct inode *old_dir_i, struct dentry *old_dentry,printk (KERN_NOTICE "jffs2_rename(): Link succeeded, unlink failed (err %d). You now have a hard link\n" , ret);/* Might as well let the VFS know */ d_instantiate (new_dentry, old_dentry->d_inode); -atomic_inc (&old_dentry->d_inode->i_count); +spin_lock (&old_dentry->d_inode->i_lock); + old_dentry->d_inode->i_count++; +spin_unlock (&old_dentry->d_inode->i_lock); new_dir_i->i_mtime = new_dir_i->i_ctime =ITIME (now);return ret; }
fs/jfs/jfs_txnmgr.c +3 -1
@@ -1279,7 +1279,9 @@ int txCommit(tid_t tid, /* transaction identifier */* lazy commit thread finishes processing */ if (tblk->xflag &COMMIT_DELETE ) { -atomic_inc (&tblk->u.ip->i_count); +spin_lock (&tblk->u.ip->i_lock); + tblk->u.ip->i_count++; +spin_unlock (&tblk->u.ip->i_lock);/* * Avoid a rare deadlock *
fs/jfs/namei.c +3 -1
@@ -839,7 +839,9 @@ static int jfs_link(struct dentry *old_dentry, ip->i_ctime =CURRENT_TIME ; dir->i_ctime = dir->i_mtime =CURRENT_TIME ;mark_inode_dirty (dir); -atomic_inc (&ip->i_count); +spin_lock (&ip->i_lock); + ip->i_count++; +spin_unlock (&ip->i_lock); iplist[0 ] = ip; iplist[1 ] = dir;
fs/libfs.c +3 -1
@@ -255,7 +255,9 @@ int simple_link(struct dentry *old_dentry, struct inode *dir, struct dentry *den inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ;inc_nlink (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);dget (dentry);d_instantiate (dentry, inode);return 0 ;
fs/locks.c +1 -1
@@ -1376,7 +1376,7 @@ int generic_setlease(struct file *filp, long arg, struct file_lock **flp) goto out;if ((arg ==F_WRLCK ) && ((atomic_read (&dentry->d_count) >1 ) - || (atomic_read (&inode->i_count) >1 ))) + || inode->i_count >1 )) goto out; }
fs/logfs/dir.c +3 -1
@@ -569,7 +569,9 @@ static int logfs_link(struct dentry *old_dentry, struct inode *dir,return -EMLINK ; inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ; -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); inode->i_nlink++;mark_inode_dirty_sync (inode);
fs/logfs/readwrite.c +5 -1
@@ -1002,8 +1002,12 @@ static int __logfs_is_valid_block(struct inode *inode, u64 bix, u64 ofs) {struct logfs_inode *li =logfs_inode (inode); -if ((inode->i_nlink ==0 ) &&atomic_read (&inode->i_count) ==1 ) +spin_lock (&inode->i_lock); +if ((inode->i_nlink ==0 ) && inode->i_count ==1 ) { +spin_unlock (&inode->i_lock);return 0 ; + } +spin_unlock (&inode->i_lock);if (bix <I0_BLOCKS )return logfs_is_valid_direct (li, bix, ofs);
fs/minix/namei.c +3 -1
@@ -101,7 +101,9 @@ static int minix_link(struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);return add_nondir (dentry, inode); }
fs/namei.c +5 -2
@@ -2290,8 +2290,11 @@ static long do_unlinkat(int dfd, const char __user *pathname)if (nd.last.name[nd.last.len]) goto slashes; inode = dentry->d_inode; -if (inode) -atomic_inc (&inode->i_count); +if (inode) { +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); + } error =mnt_want_write (nd.path.mnt);if (error) goto exit2;
fs/nfs/dir.c +3 -1
@@ -1580,7 +1580,9 @@ nfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry)d_drop (dentry); error =NFS_PROTO (dir)->link (inode, dir, &dentry->d_name);if (error ==0 ) { -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_add (dentry, inode); }return error;
fs/nfs/getroot.c +3 -1
@@ -55,7 +55,9 @@ static int nfs_superblock_set_dummy_root(struct super_block *sb, struct inode *ireturn -ENOMEM ; }/* Circumvent igrab(): we know the inode is not being freed */ -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);/* * Ensure that this dentry is invisible to d_find_alias(). * Otherwise, it may be spliced into the tree by
fs/nfs/inode.c +2 -2
@@ -384,7 +384,7 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr)dprintk ("NFS: nfs_fhget(%s/%Ld ct=%d)\n" , inode->i_sb->s_id, (long long )NFS_FILEID (inode), -atomic_read (&inode->i_count)); + inode->i_count); out:return inode;
@@ -1190,7 +1190,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)dfprintk (VFS ,"NFS: %s(%s/%ld ct=%d info=0x%x)\n" , __func__, inode->i_sb->s_id, inode->i_ino, -atomic_read (&inode->i_count), fattr->valid); + inode->i_count, fattr->valid);if ((fattr->valid &NFS_ATTR_FATTR_FILEID ) && nfsi->fileid != fattr->fileid) goto out_fileid;
fs/nfs/nfs4state.c +1 -1
@@ -506,8 +506,8 @@ nfs4_get_open_state(struct inode *inode, struct nfs4_state_owner *owner) state->owner = owner;atomic_inc (&owner->so_count);list_add (&state->inode_states, &nfsi->open_states); - state->inode =igrab (inode);spin_unlock (&inode->i_lock); + state->inode =igrab (inode);/* Note: The reclaim code dictates that we add stateless * and read-only stateids to the end of the list */ list_add_tail (&state->open_states, &owner->so_states);
fs/nfs/write.c +1 -1
@@ -390,7 +390,7 @@ static int nfs_inode_add_request(struct inode *inode, struct nfs_page *req) error =radix_tree_insert (&nfsi->nfs_page_tree, req->wb_index, req);BUG_ON (error);if (!nfsi->npages) { -igrab (inode); +__iget (inode);if (nfs_have_delegation (inode,FMODE_WRITE )) nfsi->change_attr++; }
fs/nilfs2/mdt.c +1 -1
@@ -480,7 +480,7 @@ nilfs_mdt_new_common(struct the_nilfs *nilfs, struct super_block *sb, inode->i_sb = sb;/* sb may be NULL for some meta data files */ inode->i_blkbits = nilfs->ns_blocksize_bits; inode->i_flags =0 ; -atomic_set (&inode->i_count,1 ); + inode->i_count =1 ; inode->i_nlink =1 ; inode->i_ino = ino; inode->i_mode =S_IFREG ;
fs/nilfs2/namei.c +3 -1
@@ -219,7 +219,9 @@ static int nilfs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); err =nilfs_add_nondir (dentry, inode);if (!err)
fs/notify/inode_mark.c +13 -8
@@ -257,24 +257,29 @@ void fsnotify_unmount_inodes(struct list_head *list)* evict all inodes with zero i_count from icache which is * unnecessarily violent and may in fact be illegal to do. */ -if (!atomic_read (&inode->i_count)) +if (!inode->i_count)continue ; need_iput_tmp = need_iput; need_iput =NULL ;/* In case fsnotify_inode_delete() drops a reference. */ -if (inode != need_iput_tmp) +if (inode != need_iput_tmp) { +spin_lock (&inode->i_lock);__iget (inode); -else +spin_unlock (&inode->i_lock); + }else need_iput_tmp =NULL ;/* In case the dropping of a reference would nuke next_i. */ -if ((&next_i->i_sb_list != list) && -atomic_read (&next_i->i_count) && - !(next_i->i_state & (I_FREEING |I_WILL_FREE ))) { -__iget (next_i); - need_iput = next_i; +if (&next_i->i_sb_list != list) { +spin_lock (&next_i->i_lock); +if (next_i->i_count && + !(next_i->i_state & (I_FREEING |I_WILL_FREE ))) { +__iget (next_i); + need_iput = next_i; + } +spin_unlock (&next_i->i_lock); }/*
fs/ntfs/super.c +3 -1
@@ -2930,7 +2930,9 @@ static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent) }if ((sb->s_root =d_alloc_root (vol->root_ino))) {/* We increment i_count simulating an ntfs_iget(). */ -atomic_inc (&vol->root_ino->i_count); +spin_lock (&vol->root_ino->i_lock); + vol->root_ino->i_count++; +spin_unlock (&vol->root_ino->i_lock);ntfs_debug ("Exiting, status successful." );/* Release the default upcase if it has no users. */ mutex_lock (&ntfs_lock);
fs/ocfs2/namei.c +3 -1
@@ -741,7 +741,9 @@ static int ocfs2_link(struct dentry *old_dentry, goto out_commit; } -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); dentry->d_op = &ocfs2_dentry_ops;d_instantiate (dentry, inode);
fs/reiserfs/namei.c +3 -1
@@ -1156,7 +1156,9 @@ static int reiserfs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME_SEC ;reiserfs_update_sd (&th, inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (dentry, inode); retval =journal_end (&th, dir->i_sb, jbegin_count);reiserfs_write_unlock (dir->i_sb);
fs/reiserfs/stree.c +1 -1
@@ -1477,7 +1477,7 @@ static int maybe_indirect_to_direct(struct reiserfs_transaction_handle *th,** reading in the last block. The user will hit problems trying to ** read the file, but for now we just skip the indirect2direct */ -if (atomic_read (&inode->i_count) >1 || +if (inode->i_count >1 || !tail_has_to_be_packed (inode) || !page || (REISERFS_I (inode)->i_flags & i_nopack_mask)) {/* leave tail in an unformatted node */
fs/sysv/namei.c +3 -1
@@ -126,7 +126,9 @@ static int sysv_link(struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);return add_nondir (dentry, inode); }
fs/ubifs/dir.c +3 -1
@@ -550,7 +550,9 @@ static int ubifs_link(struct dentry *old_dentry, struct inode *dir,lock_2_inodes (dir, inode);inc_nlink (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); inode->i_ctime =ubifs_current_time (inode); dir->i_size += sz_change; dir_ui->ui_size = dir->i_size;
fs/ubifs/super.c +1 -1
@@ -342,7 +342,7 @@ static void ubifs_evict_inode(struct inode *inode) goto out;dbg_gen ("inode %lu, mode %#x" , inode->i_ino, (int )inode->i_mode); -ubifs_assert (!atomic_read (&inode->i_count)); +ubifs_assert (!inode->i_count);truncate_inode_pages (&inode->i_data,0 );
fs/udf/namei.c +3 -1
@@ -1101,7 +1101,9 @@ static int udf_link(struct dentry *old_dentry, struct inode *dir,inc_nlink (inode); inode->i_ctime =current_fs_time (inode->i_sb);mark_inode_dirty (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (dentry, inode);unlock_kernel ();
fs/ufs/namei.c +3 -1
@@ -180,7 +180,9 @@ static int ufs_link (struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); error =ufs_add_nondir (dentry, inode);unlock_kernel ();
fs/xfs/linux-2.6/xfs_iops.c +3 -1
@@ -352,7 +352,9 @@ xfs_vn_link(if (unlikely (error))return -error; -atomic_inc (&inode->i_count); +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock);d_instantiate (dentry, inode);return 0 ; }
fs/xfs/linux-2.6/xfs_trace.h +1 -1
@@ -599,7 +599,7 @@ DECLARE_EVENT_CLASS(xfs_iref_class,TP_fast_assign ( __entry->dev = VFS_I(ip)->i_sb->s_dev; __entry->ino = ip->i_ino; - __entry->count =atomic_read (&VFS_I (ip)->i_count); + __entry->count =VFS_I (ip)->i_count; __entry->pincount =atomic_read (&ip->i_pincount); __entry->caller_ip = caller_ip; ),
fs/xfs/xfs_inode.h +4 -2
@@ -481,8 +481,10 @@ void xfs_mark_inode_dirty_sync(xfs_inode_t *);#define IHOLD (ip) \ do { \ - ASSERT(atomic_read(&VFS_I(ip)->i_count) > 0) ; \ - atomic_inc(&(VFS_I(ip)->i_count)); \ + spin_lock(&VFS_I(ip)->i_lock); \ + ASSERT(VFS_I(ip)->i_count > 0) ; \ + VFS_I(ip)->i_count++; \ + spin_unlock(&VFS_I(ip)->i_lock); \ trace_xfs_ihold(ip, _THIS_IP_); \ } while (0)
include/linux/fs.h +1 -1
@@ -728,7 +728,7 @@ struct inode {struct list_head i_sb_list;struct list_head i_dentry;unsigned long i_ino; -atomic_t i_count; +unsigned int i_count;unsigned int i_nlink;uid_t i_uid;gid_t i_gid;
ipc/mqueue.c +5 -2
@@ -768,8 +768,11 @@ SYSCALL_DEFINE1(mq_unlink, const char __user *, u_name) } inode = dentry->d_inode; -if (inode) -atomic_inc (&inode->i_count); +if (inode) { +spin_lock (&inode->i_lock); + inode->i_count++; +spin_unlock (&inode->i_lock); + } err =mnt_want_write (ipc_ns->mq_mnt);if (err) goto out_err;
kernel/futex.c +3 -1
@@ -168,7 +168,9 @@ static void get_futex_key_refs(union futex_key *key)switch (key->both.offset & (FUT_OFF_INODE |FUT_OFF_MMSHARED )) {case FUT_OFF_INODE : -atomic_inc (&key->shared.inode->i_count); +spin_lock (&key->shared.inode->i_lock); + key->shared.inode->i_count++; +spin_unlock (&key->shared.inode->i_lock);break ;case FUT_OFF_MMSHARED :atomic_inc (&key->private.mm->mm_count);
mm/shmem.c +3 -1
@@ -1903,7 +1903,9 @@ static int shmem_link(struct dentry *old_dentry, struct inode *dir, struct dentr dir->i_size +=BOGO_DIRENT_SIZE ; inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ;inc_nlink (inode); -atomic_inc (&inode->i_count);/* New dentry reference */ +spin_lock (&inode->i_lock); + inode->i_count++;/* New dentry reference */ +spin_unlock (&inode->i_lock);dget (dentry);/* Extra pinning count for the created dentry */ d_instantiate (dentry, inode); out:
net/socket.c +3 -1
@@ -377,7 +377,9 @@ static int sock_alloc_file(struct socket *sock, struct file **f, int flags) &socket_file_ops);if (unlikely (!file)) {/* drop dentry, keep inode */ -atomic_inc (&path.dentry->d_inode->i_count); +spin_lock (&path.dentry->d_inode->i_lock); + path.dentry->d_inode->i_count++; +spin_unlock (&path.dentry->d_inode->i_lock);path_put (&path);put_unused_fd (fd);return -ENFILE ;
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 05/17] fs: icache lock i_count
arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552
Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1
-1-if (atomic_read (&inode->i_count) !=1 ) +if (inode->i_count !=1 )
This really should have been renamed to catch unconverted code.
Such code usually wouldn't compile anyway, but it will if it takes the address of i_count only (for example).
And maybe we should access this guy via accessor functions, dunno.
Re: [PATCH 05/17] fs: icache lock i_count
Quoted 9 lines · Andrew Morton
arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552
Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1
-1-if (atomic_read (&inode->i_count) !=1 ) +if (inode->i_count !=1 )This really should have been renamed to catch unconverted code.
Such code usually wouldn't compile anyway, but it will if it takes the address of
i_countonly (for example).
If people do whacky things they'll lose - there is a reason why C has a fairly strict type system after all. We've changed types of variables all the time and we didn't run into problems.
Quoted 1 line
And maybe we should access this guy via accessor functions, dunno.
Seems like complete overkill.
Andrew Morton
maintainerRe: [PATCH 05/17] fs: icache lock i_count
Quoted 8 lines · On Fri, 1 Oct 2010 01:55:36 -0400 Christoph Hellwig <[email protected]> wrote:
Quoted 7 lines · Andrew Morton
arch/powerpc/platforms/cell/spufs/file.c static int spufs_mfc_open(struct inode *inode, struct file *file) · line 1552
Quoted 2 lines · On Wed, 29 Sep 2010 22:18:37 +1000 Dave Chinner <[email protected]> wrote:
arch/powerpc/platforms/cell/spufs/file.c +1
-1-if (atomic_read (&inode->i_count) !=1 ) +if (inode->i_count !=1 )This really should have been renamed to catch unconverted code.
Quoted 6 lines
Quoted 2 lines
Such code usually wouldn't compile anyway, but it will if it takes the address of
i_countonly (for example).If people do whacky things they'll lose - there is a reason why C has a fairly strict type system after all. We've changed types of variables all the time and we didn't run into problems.
No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.
Quoted 3 lines
Quoted 1 line
And maybe we should access this guy via accessor functions, dunno.
Seems like complete overkill.
Still wrong. We do this frequently and we do it in areas where we believe that the implementation might change in the future.
Had we done it with i_count from day one then this part of the patchset would be far simpler.
Re: [PATCH 05/17] fs: icache lock i_count
Quoted 4 lines · Andrew Morton
No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.
Sorry, if you run code with that obvious warnings you beg for trouble. If you really believe your advanced users arw too stupid to read compiler warnings enforcing -Werror is for sure better than obsfucating the code.
Quoted 5 lines
Still wrong. We do this frequently and we do it in areas where we believe that the implementation might change in the future.
Had we done it with
i_countfrom day one then this part of the patchset would be far simpler.
I don't thin that's quite true. The big point of using i_lock is that we can hold it over accessing other things that also are protected by it. No accessor is going to help you with that. For plain opencoded increments we indeed need a helper as shown by Chris' aio speedup and the churn in here - but that's already added later in the series and I asked Dave to move it before this patch.
Andrew Morton
maintainerRe: [PATCH 05/17] fs: icache lock i_count
Quoted 10 lines · On Fri, 1 Oct 2010 02:16:02 -0400 Christoph Hellwig <[email protected]> wrote:
Quoted 4 lines · Andrew Morton
No, we've run into problems *frequently*. A common case is where we convert a mutex to a spinlock or vice versa. If you don't rename the lock, the code still compiles (with warnings) and crashes horridly at runtime.
Sorry, if you run code with that obvious warnings you beg for trouble. If you really believe your advanced users arw too stupid to read compiler warnings enforcing -Werror is for sure better than obsfucating the code.
Well, it has happened, fairly regularly. A common scenario is where someone has done a conversion in one tree and someone else has touched overlapping code in another tree and when the two meet in linux-next, splat. Renaming the field simply eliminates this.
Of course, the warnings don't get noticed because of the enormous warning storm which a kernel build produces (generally much worse on non-x86, btw).
Another reason for remaining a field is when we desire that it henceforth be accessed via accessor functions - renaming it will reliably break any unconverted code.
PATCH 6/17 fs: icache lock lru/writeback lists
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
The inode moves between different lists protected by the inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name it wb_inode_list_lock and nest all the list manipulations in this lock inside the current inode_lock scope.
--- fs/fs-writeback.c | 48 +++++++++++++++++++++++++++++++++++++++++++- fs/inode.c | 44 +++++++++++++++++++++++++++++++++------- include/linux/writeback.h | 1 + mm/backing-dev.c | 4 +++ 4 files changed, 87 insertions(+), 10 deletions(-)
fs/fs-writeback.c +46 -2
@@ -161,6 +161,7 @@ static void redirty_tail(struct inode *inode) {struct bdi_writeback *wb = &inode_to_bdi (inode)->wb; +assert_spin_locked (&wb_inode_list_lock);if (!list_empty (&wb->b_dirty)) {struct inode *tail;
@@ -178,6 +179,7 @@ static void requeue_io(struct inode *inode) {struct bdi_writeback *wb = &inode_to_bdi (inode)->wb; +assert_spin_locked (&wb_inode_list_lock);list_move (&inode->i_list, &wb->b_more_io); }
@@ -218,6 +220,7 @@ static void move_expired_inodes(struct list_head *delaying_queue,struct inode *inode;int do_sb_sort =0 ; +assert_spin_locked (&wb_inode_list_lock);while (!list_empty (delaying_queue)) { inode = list_entry(delaying_queue->prev, struct inode, i_list); if (older_than_this &&
@@ -281,11 +284,13 @@ static void inode_wait_for_writeback(struct inode *inode) wqh =bit_waitqueue (&inode->i_state, __I_SYNC);while (inode->i_state &I_SYNC ) { +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);__wait_on_bit (wqh, &wq, inode_wait,TASK_UNINTERRUPTIBLE );spin_lock (&inode_lock);spin_lock (&inode->i_lock); +spin_lock (&wb_inode_list_lock); } }
@@ -339,6 +344,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)/* Set I_SYNC, reset I_DIRTY_PAGES */ inode->i_state |=I_SYNC ; inode->i_state &= ~I_DIRTY_PAGES ; +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);
@@ -375,6 +381,7 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)spin_lock (&inode_lock);spin_lock (&inode->i_lock); +spin_lock (&wb_inode_list_lock); inode->i_state &= ~I_SYNC ;if (!(inode->i_state &I_FREEING )) {if (mapping_tagged (mapping,PAGECACHE_TAG_DIRTY )) {
@@ -461,11 +468,18 @@ static bool pin_sb_for_writeback(struct super_block *sb)static int writeback_sb_inodes (struct super_block *sb,struct bdi_writeback *wb,struct writeback_control *wbc,bool only_this_sb) { +again:while (!list_empty (&wb->b_io)) {long pages_skipped;struct inode *inode =list_entry (wb->b_io.prev, struct inode, i_list); +if (!spin_trylock (&inode->i_lock)) { +spin_unlock (&wb_inode_list_lock); +spin_lock (&wb_inode_list_lock); + goto again; + } +if (inode->i_sb != sb) {if (only_this_sb) {/*
@@ -474,9 +488,12 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,* to it back onto the dirty list. */ redirty_tail (inode); +spin_unlock (&inode->i_lock);continue ; } +spin_unlock (&inode->i_lock); +/* * The inode belongs to a different superblock. * Bounce back to the caller to unpin this and
@@ -485,10 +502,9 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,return 0 ; } -spin_lock (&inode->i_lock);if (inode->i_state & (I_NEW |I_WILL_FREE )) { -spin_unlock (&inode->i_lock);requeue_io (inode); +spin_unlock (&inode->i_lock);continue ; }/*
@@ -511,11 +527,13 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,*/ redirty_tail (inode); } +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);iput (inode);cond_resched ();spin_lock (&inode_lock); +spin_lock (&wb_inode_list_lock);if (wbc->nr_to_write <=0 ) { wbc->more_io =1 ;return 1 ;
@@ -535,6 +553,9 @@ void writeback_inodes_wb(struct bdi_writeback *wb,if (!wbc->wb_start) wbc->wb_start = jiffies;/* livelock avoidance */ spin_lock (&inode_lock); +again: +spin_lock (&wb_inode_list_lock); +if (!wbc->for_kupdate ||list_empty (&wb->b_io))queue_io (wb, wbc->older_than_this);
@@ -544,7 +565,12 @@ void writeback_inodes_wb(struct bdi_writeback *wb,struct super_block *sb = inode->i_sb;if (!pin_sb_for_writeback (sb)) { +if (!spin_trylock (&inode->i_lock)) { +spin_unlock (&wb_inode_list_lock); + goto again; + }requeue_io (inode); +spin_unlock (&inode->i_lock);continue ; } ret =writeback_sb_inodes (sb, wb, wbc, false);
@@ -553,6 +579,7 @@ void writeback_inodes_wb(struct bdi_writeback *wb,if (ret)break ; } +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode_lock);/* Leave any unwritten inodes on b_io */ }
@@ -563,9 +590,11 @@ static void __writeback_inodes_sb(struct super_block *sb,WARN_ON (!rwsem_is_locked (&sb->s_umount));spin_lock (&inode_lock); +spin_lock (&wb_inode_list_lock);if (!wbc->for_kupdate ||list_empty (&wb->b_io))queue_io (wb, wbc->older_than_this);writeback_sb_inodes (sb, wb, wbc, true); +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode_lock); }
@@ -676,13 +705,22 @@ static long wb_writeback(struct bdi_writeback *wb,* become available for writeback. Otherwise * we'll just busyloop. */ +retry:spin_lock (&inode_lock); +spin_lock (&wb_inode_list_lock);if (!list_empty (&wb->b_more_io)) { inode =list_entry (wb->b_more_io.prev, struct inode, i_list); +if (!spin_trylock (&inode->i_lock)) { +spin_unlock (&wb_inode_list_lock); +spin_unlock (&inode_lock); + goto retry; + }trace_wbc_writeback_wait (&wbc, wb->bdi);inode_wait_for_writeback (inode); +spin_unlock (&inode->i_lock); } +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode_lock); }
@@ -993,8 +1031,10 @@ void __mark_inode_dirty(struct inode *inode, int flags) wakeup_bdi = true; } +spin_lock (&wb_inode_list_lock); inode->dirtied_when = jiffies;list_move (&inode->i_list, &bdi->wb.b_dirty); +spin_unlock (&wb_inode_list_lock); } } out:
@@ -1183,7 +1223,9 @@ int write_inode_now(struct inode *inode, int sync)might_sleep ();spin_lock (&inode_lock);spin_lock (&inode->i_lock); +spin_lock (&wb_inode_list_lock); ret =writeback_single_inode (inode, &wbc); +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);if (sync)
@@ -1209,7 +1251,9 @@ int sync_inode(struct inode *inode, struct writeback_control *wbc)spin_lock (&inode_lock);spin_lock (&inode->i_lock); +spin_lock (&wb_inode_list_lock); ret =writeback_single_inode (inode, wbc); +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);return ret;
fs/inode.c +36 -8
144 lines not on this page · raw message
include/linux/writeback.h +1 -0
@@ -11,6 +11,7 @@ struct backing_dev_info;extern spinlock_t inode_lock;extern spinlock_t sb_inode_list_lock; +extern spinlock_t wb_inode_list_lock;extern spinlock_t inode_hash_lock;extern struct list_head inode_in_use;extern struct list_head inode_unused;
mm/backing-dev.c +4 -0
@@ -73,12 +73,14 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v) nr_wb = nr_dirty = nr_io = nr_more_io =0 ;spin_lock (&inode_lock); +spin_lock (&wb_inode_list_lock);list_for_each_entry (inode, &wb->b_dirty, i_list) nr_dirty++;list_for_each_entry (inode, &wb->b_io, i_list) nr_io++;list_for_each_entry (inode, &wb->b_more_io, i_list) nr_more_io++; +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode_lock);global_dirty_limits (&background_thresh, &dirty_thresh);
@@ -681,9 +683,11 @@ void bdi_destroy(struct backing_dev_info *bdi)struct bdi_writeback *dst = &default_backing_dev_info.wb;spin_lock (&inode_lock); +spin_lock (&wb_inode_list_lock);list_splice (&bdi->wb.b_dirty, &dst->b_dirty);list_splice (&bdi->wb.b_io, &dst->b_io);list_splice (&bdi->wb.b_more_io, &dst->b_more_io); +spin_unlock (&wb_inode_list_lock);spin_unlock (&inode_lock); }
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 06/17] fs: icache lock lru/writeback lists
Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:
The inode moves between different lists protected by the
inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name itwb_inode_list_lockand nest all the list manipulations in this lock inside the currentinode_lockscope.
All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.
I'd suggest that each such site have a comment explaining why we're resorting to this.
Re: [PATCH 06/17] fs: icache lock lru/writeback lists
Quoted 10 lines · Andrew Morton
Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:
The inode moves between different lists protected by the
inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name itwb_inode_list_lockand nest all the list manipulations in this lock inside the currentinode_lockscope.All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.
Yes, they are, but I don't know exactly why it is so trylock happy. I'll try to dig out the reason for it and:
Quoted 2 lines
I'd suggest that each such site have a comment explaining why we're resorting to this.
At least get this far.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 06/17] fs: icache lock lru/writeback lists
In reply to Andrew MortonQuoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 29 Sep 2010 22:18:38 +1000 Dave Chinner <[email protected]> wrote:
The inode moves between different lists protected by the
inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name itwb_inode_list_lockand nest all the list manipulations in this lock inside the currentinode_lockscope.
Quoted 5 lines
All those spin_trylock()s are real ugly. They're unexplained in the changelog and unexplained in code comments.
I'd suggest that each such site have a comment explaining why we're resorting to this.
They're really a side effect of how I'm building up the locking in steps and then streamlining it in steps. Most of them disappear or get much improved as inode removal, rcu, etc greatly help with lock ordering.
The intermediate steps are not supposed to be so pretty, so much as an easily verifiable "ok, we have enough locking to cover what inode_lock used to be protecting".
Re: [PATCH 06/17] fs: icache lock lru/writeback lists
In reply to Dave ChinnerQuoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
The inode moves between different lists protected by the
inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name itwb_inode_list_lockand nest all the list manipulations in this lock inside the currentinode_lockscope.
As a band-aid to get rid of the inode_lock this might be fine, but I don't really like it. For one all the list are per-bdi_writeback, so the lock should be as well. Second the lock is held over far too long periods during writeback, which leads to a lot of whacky trylock operations and unlock and sleep cycles inside it. In practice we only need it in the places where we manipulate the lists.
Also it feels like it really should nest outside i_lock, not inside it, but I need to look more deeply to figure why that might not easily be possible.
But maybe all that is better left as a separate patch on top of the current queue.
Re: [PATCH 06/17] fs: icache lock lru/writeback lists
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
The inode moves between different lists protected by the
inode_lock. Introduce a new lock that protects all of the lists (dirty, unused, in use, etc) that the inode will move around as it changes state. As this is mostly a list for protecting the writeback lists, name itwb_inode_list_lockand nest all the list manipulations in this lock inside the currentinode_lockscope.
Quoted 6 lines
As a band-aid to get rid of the
inode_lockthis might be fine, but I don't really like it. For one all the list are per-bdi_writeback, so the lock should be as well. Second the lock is held over far too long periods during writeback, which leads to a lot of whacky trylock operations and unlock and sleep cycles inside it. In practice we only need it in the places where we manipulate the lists.
per-bdi writeback lock won't work with the patch set as it stands - it also protects the LRU which is a global list. I'll have to pull back another patch to split the LRU and IO lists to make this lock per-bdi.
Quoted 3 lines
Also it feels like it really should nest outside
i_lock, not inside it, but I need to look more deeply to figure why that might not easily be possible.
Yeah, I'm trying to rework the patch series to not nest anything inside i_lock. The more I look at all that trylock stuff, the more my eyes bleed....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 7/17 fs: icache atomic inodes_stat
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
The inode use statistics are currently protected by the inode_lock. Before we can remove the inode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.
--- fs/fs-writeback.c | 6 ++++-- fs/inode.c | 26 ++++++++++++++------------ include/linux/fs.h | 13 ++++++------- 3 files changed, 24 insertions(+), 21 deletions(-)
fs/fs-writeback.c +4 -2
@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));if (nr_pages) {struct wb_writeback_work work = {
@@ -1144,7 +1145,8 @@ void writeback_inodes_sb(struct super_block *sb)WARN_ON (!rwsem_is_locked (&sb->s_umount)); work.nr_pages = nr_dirty + nr_unstable + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));bdi_queue_work (sb->s_bdi, &work);wait_for_completion (&done);
fs/inode.c +14 -12
@@ -122,7 +122,10 @@ static DECLARE_RWSEM(iprune_sem);/* * Statistics gathering.. */ -struct inodes_stat_t inodes_stat; +struct inodes_stat_t inodes_stat = { + .nr_inodes =ATOMIC_INIT (0 ), + .nr_unused =ATOMIC_INIT (0 ), +};static struct kmem_cache *inode_cachep__read_mostly ;
@@ -313,7 +316,7 @@ void __iget(struct inode *inode)list_move (&inode->i_list, &inode_in_use);spin_unlock (&wb_inode_list_lock); } - inodes_stat.nr_unused--; +atomic_dec (&inodes_stat.nr_unused); }void end_writeback (struct inode *inode)
@@ -377,9 +380,7 @@ static void dispose_list(struct list_head *head)destroy_inode (inode); nr_disposed++; } -spin_lock (&inode_lock); - inodes_stat.nr_inodes -= nr_disposed; -spin_unlock (&inode_lock); +atomic_sub (nr_disposed, &inodes_stat.nr_inodes); }/*
@@ -428,7 +429,7 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose) busy =1 ; }/* only unused inodes may be cached with i_count zero */ - inodes_stat.nr_unused -= count; +atomic_sub (count, &inodes_stat.nr_unused);return busy; }
@@ -545,7 +546,7 @@ again2:spin_unlock (&inode->i_lock); nr_pruned++; } - inodes_stat.nr_unused -= nr_pruned; +atomic_sub (nr_pruned, &inodes_stat.nr_unused);if (current_is_kswapd ())__count_vm_events (KSWAPD_INODESTEAL , reap);else
@@ -578,7 +579,8 @@ static int shrink_icache_memory(struct shrinker *shrink, int nr, gfp_t gfp_mask)return -1 ;prune_icache (nr); } -return (inodes_stat.nr_unused /100 ) * sysctl_vfs_cache_pressure; +return (atomic_read (&inodes_stat.nr_unused) /100 ) * + sysctl_vfs_cache_pressure; }static struct shrinker icache_shrinker = {
@@ -671,7 +673,7 @@ static inline void __inode_add_to_lists(struct super_block *sb, struct hlist_head *head, struct inode *inode) { - inodes_stat.nr_inodes++; + atomic_inc(&inodes_stat.nr_inodes);spin_lock (&sb_inode_list_lock);list_add (&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock);
@@ -1336,7 +1338,7 @@ static void iput_final(struct inode *inode)list_move (&inode->i_list, &inode_unused);spin_unlock (&wb_inode_list_lock); } - inodes_stat.nr_unused++; +atomic_inc (&inodes_stat.nr_unused);if (sb->s_flags &MS_ACTIVE ) {spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);
@@ -1354,7 +1356,7 @@ static void iput_final(struct inode *inode)spin_lock (&inode->i_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_WILL_FREE ; - inodes_stat.nr_unused--; +atomic_dec (&inodes_stat.nr_unused);spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash);spin_unlock (&inode_hash_lock);
@@ -1366,9 +1368,9 @@ static void iput_final(struct inode *inode)spin_unlock (&sb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ; - inodes_stat.nr_inodes--;spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); +atomic_dec (&inodes_stat.nr_inodes);evict (inode);spin_lock (&inode_lock);spin_lock (&inode_hash_lock);
include/linux/fs.h +6 -7
@@ -39,13 +39,6 @@ struct files_stat_struct {int max_files;/* tunable */ }; -struct inodes_stat_t { -int nr_inodes; -int nr_unused; -int dummy[5 ];/* padding for sysctl ABI compatibility */ -}; - -#define NR_FILE 8192/* this can well be larger on a larger system */ #define MAY_EXEC 1
@@ -416,6 +409,12 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, ssize_t bytes, void *private, int ret, bool is_async); +struct inodes_stat_t { +atomic_t nr_inodes; +atomic_t nr_unused; +int dummy[5 ];/* padding for sysctl ABI compatibility */ +}; +/* * Attribute flags. These should be or-ed together to figure out what * has been changed!
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 07/17] fs: icache atomic inodes_stat
Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
The inode use statistics are currently protected by the
inode_lock. Before we can remove theinode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.
typo
fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768
Quoted 12 lines
...
fs/fs-writeback.c +2
-1@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));
race bug.
fs/fs-writeback.c void writeback_inodes_sb(struct super_block *sb) · line 1149
Quoted 9 lines
fs/fs-writeback.c +2
-1if (nr_pages) {struct wb_writeback_work work = {@@ -1144,7 +1145,8 @@ void writeback_inodes_sb(struct super_block *sb)WARN_ON (!rwsem_is_locked (&sb->s_umount)); work.nr_pages = nr_dirty + nr_unstable + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));
and another.
OK, they aren't serious ones. But known regressions shouldn't be snuck into the kernel unchangelogged and uncommented :(
Quoted 16 lines
fs/fs-writeback.c +0
-7bdi_queue_work (sb->s_bdi, &work);wait_for_completion (&done); -struct inodes_stat_t { -int nr_inodes; -int nr_unused; -int dummy[5 ];/* padding for sysctl ABI compatibility */ -}; - -#define NR_FILE 8192/* this can well be larger on a larger system */ #define MAY_EXEC 1
fs/fs-writeback.c typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · line 416
Quoted 8 lines
fs/fs-writeback.c +5
-0@@ -416,6 +409,12 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, ssize_t bytes, void *private, int ret, bool is_async); +struct inodes_stat_t { +atomic_t nr_inodes; +atomic_t nr_unused; +int dummy[5 ];/* padding for sysctl ABI compatibility */ +};
OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an atomic_t has the same layout, alignment and padding as an int.
Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.
It shouldn't be snuck into the tree unchangelogged and uncommented.
(time passes)
OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.
Re: [PATCH 07/17] fs: icache atomic inodes_stat
Quoted 14 lines · Andrew Morton
Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
The inode use statistics are currently protected by the
inode_lock. Before we can remove theinode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.typo
Quoted 3 lines
...
Quoted 11 lines
fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768
Quoted 9 lines
fs/fs-writeback.c +2
-1@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));race bug.
What new race? The code has gone from using subtraction of unlocked counters to using subtraction of unlocked atomic counters. I can't see any new race condition in that transformation....
Quoted 13 lines
fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb)
Quoted 5 lines
fs/fs-writeback.c +5
-0+struct inodes_stat_t { +atomic_t nr_inodes; +atomic_t nr_unused; +int dummy[5 ];/* padding for sysctl ABI compatibility */ +};OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an
atomic_thas the same layout, alignment and padding as an int.Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.
Quoted 6 lines
It shouldn't be snuck into the tree unchangelogged and uncommented.
(time passes)
OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.
That's the multiple changes to the counter infrastructure I described in the preliminary series description. clearly it needs to be cleaned up.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Andrew Morton
maintainerRe: [PATCH 07/17] fs: icache atomic inodes_stat
Quoted 12 lines · On Thu, 30 Sep 2010 16:20:57 +1000 Dave Chinner <[email protected]> wrote:
Quoted 11 lines
fs/fs-writeback.c static long wb_check_old_data_flush(struct bdi_writeback *wb) · line 768
Quoted 9 lines
fs/fs-writeback.c +2
-1@@ -764,7 +764,8 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - (inodes_stat.nr_inodes - inodes_stat.nr_unused); + (atomic_read (&inodes_stat.nr_inodes) - +atomic_read (&inodes_stat.nr_unused));race bug.
Quoted 3 lines
What new race? The code has gone from using subtraction of unlocked counters to using subtraction of unlocked atomic counters. I can't see any new race condition in that transformation....
Oh. I assumed that all the above used to happen under inode_lock.
Coz that's wot the changelog told me :)
Re: [PATCH 07/17] fs: icache atomic inodes_stat
In reply to Andrew MortonQuoted 9 lines · Andrew Morton
Quoted 6 lines · On Wed, 29 Sep 2010 22:18:39 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
The inode use statistics are currently protected by the
inode_lock. Before we can remove theinode_lock, we need to protect these counters against races. Do this by converting them to atomic counters so they ar enot dependent on any lock at all.typo
It's Dave, I swear :)
Quoted 13 lines
include/linux/fs.h typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · line 416
Quoted 5 lines
include/linux/fs.h +5
-0+struct inodes_stat_t { +atomic_t nr_inodes; +atomic_t nr_unused; +int dummy[5 ];/* padding for sysctl ABI compatibility */ +};OK, that's a hack. The first two "ints" are copied out to userspace. This change assumes that sizeof(atomic_t)=4 and that an
atomic_thas the same layout, alignment and padding as an int.Probably that's true in current kernels and with current architectures but it's a hack and it's presumptive.
Quoted 6 lines
It shouldn't be snuck into the tree unchangelogged and uncommented.
(time passes)
OK, I see that all of this gets reverted later on. Please update the changelog so the next reviewer doesn't get fooled.
Yeah it is. I might end up folding the per-cpu stuff back over it and avoid the issue completely. Otherwise I'll add a comment.
PATCH 8/17 fs: icache protect inode state
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Before removing the inode_lock, we need to protect the inode list operations with the inode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.
--- fs/inode.c | 33 +++++++++++++++++++++++++++++---- 1 files changed, 29 insertions(+), 4 deletions(-)
fs/inode.c +29 -4
@@ -34,7 +34,11 @@* wb_inode_list_lock protects: * inode_in_use, inode_unused, b_io, b_more_io, b_dirty, i_list * inode->i_lock protects: -* i_state, i_count +* i_state +* i_count +* i_hash +* i_list +* i_sb_list * * Ordering: * inode_lock
@@ -368,12 +372,14 @@ static void dispose_list(struct list_head *head)evict (inode);spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); +spin_lock (&inode->i_lock);spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash);spin_unlock (&inode_hash_lock); -spin_lock (&sb_inode_list_lock);list_del_init (&inode->i_sb_list);spin_unlock (&sb_inode_list_lock); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);wake_up_inode (inode);
@@ -674,7 +680,6 @@ __inode_add_to_lists(struct super_block *sb, struct hlist_head *head, struct inode *inode) {atomic_inc (&inodes_stat.nr_inodes); -spin_lock (&sb_inode_list_lock);list_add (&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock);spin_lock (&wb_inode_list_lock);
@@ -704,7 +709,10 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode)struct hlist_head *head = inode_hashtable +hash (sb, inode->i_ino);spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); +spin_lock (&inode->i_lock);__inode_add_to_lists (sb, head, inode); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); }EXPORT_SYMBOL_GPL (inode_add_to_lists);
@@ -736,9 +744,12 @@ struct inode *new_inode(struct super_block *sb) inode =alloc_inode (sb);if (inode) {spin_lock (&inode_lock); +spin_lock (&sb_inode_list_lock); +spin_lock (&inode->i_lock); inode->i_ino = ++last_ino; inode->i_state =0 ;__inode_add_to_lists (sb,NULL , inode); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); }return inode;
@@ -802,11 +813,14 @@ static struct inode *get_new_inode(struct super_block *sb,/* We released the lock, so.. */ old =find_inode (sb, head, test, data);if (!old) { +spin_lock (&sb_inode_list_lock); +spin_lock (&inode->i_lock);if (set (inode, data)) goto set_failed; inode->i_state =I_NEW ;__inode_add_to_lists (sb, head, inode); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);/* Return the locked inode with I_NEW set, the
@@ -831,6 +845,7 @@ static struct inode *get_new_inode(struct super_block *sb, set_failed:spin_unlock (&inode->i_lock); +spin_unlock (&sb_inode_list_lock);spin_unlock (&inode_lock);destroy_inode (inode);return NULL ;
@@ -853,9 +868,12 @@ static struct inode *get_new_inode_fast(struct super_block *sb,/* We released the lock, so.. */ old =find_inode_fast (sb, head, ino);if (!old) { +spin_lock (&sb_inode_list_lock); +spin_lock (&inode->i_lock); inode->i_ino = ino; inode->i_state =I_NEW ;__inode_add_to_lists (sb, head, inode); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);/* Return the locked inode with I_NEW set, the
@@ -1270,10 +1288,13 @@ EXPORT_SYMBOL(insert_inode_locked4);void __insert_inode_hash (struct inode *inode,unsigned long hashval) {struct hlist_head *head = inode_hashtable +hash (inode->i_sb, hashval); +spin_lock (&inode_lock); +spin_lock (&inode->i_lock);spin_lock (&inode_hash_lock);hlist_add_head (&inode->i_hash, head);spin_unlock (&inode_hash_lock); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); }EXPORT_SYMBOL (__insert_inode_hash);
@@ -1287,9 +1308,11 @@ EXPORT_SYMBOL(__insert_inode_hash);void remove_inode_hash (struct inode *inode) {spin_lock (&inode_lock); +spin_lock (&inode->i_lock);spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash);spin_unlock (&inode_hash_lock); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock); }EXPORT_SYMBOL (remove_inode_hash);
@@ -1356,10 +1379,10 @@ static void iput_final(struct inode *inode)spin_lock (&inode->i_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_WILL_FREE ; -atomic_dec (&inodes_stat.nr_unused);spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash);spin_unlock (&inode_hash_lock); +atomic_dec (&inodes_stat.nr_unused); }spin_lock (&wb_inode_list_lock);list_del_init (&inode->i_list);
@@ -1373,9 +1396,11 @@ static void iput_final(struct inode *inode)atomic_dec (&inodes_stat.nr_inodes);evict (inode);spin_lock (&inode_lock); +spin_lock (&inode->i_lock);spin_lock (&inode_hash_lock);hlist_del_init (&inode->i_hash);spin_unlock (&inode_hash_lock); +spin_unlock (&inode->i_lock);spin_unlock (&inode_lock);wake_up_inode (inode);BUG_ON (inode->i_state != (I_FREEING |I_CLEAR ));
Signature · 1 line
1.7.1
Re: [PATCH 08/17] fs: icache protect inode state
Quoted 8 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before removing the
inode_lock, we need to protect the inode list operations with theinode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.
The subject does not seem to match the patch description and content.
Re: [PATCH 08/17] fs: icache protect inode state
Quoted 11 lines · Christoph Hellwig
Quoted 8 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before removing the
inode_lock, we need to protect the inode list operations with theinode->i_lock. This ensures that all inode state changes are serialised regardless of the fact that the lists they are moving around might be protected by different locks. Hence we can safely protect an inode in transit from one list to another without needing to hold all the list locks at the same time.The subject does not seem to match the patch description and content.
It is adding i_lock around remaining places where an inode can be moved on or off icache data structures. As I've described, this is quite central to my locking design, isn't the changelog understandable?
PATCH 9/17 fs: Make last_ino, iunique independent of inode_lock
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Before removing the inode_lock, we need to make the last_ino and iunique counters independent of the inode_lock. last_ino can be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside the inode_lock to provide the same protection that the inode_lock previously provided.
--- fs/inode.c | 28 ++++++++++++++++++++++------ 1 files changed, 22 insertions(+), 6 deletions(-)
fs/inode.c +22 -6
@@ -736,7 +736,7 @@ struct inode *new_inode(struct super_block *sb)* error if st_ino won't fit in target struct field. Use 32bit counter * here to attempt to avoid that. */ -static unsigned int last_ino; +static atomic_t last_ino =ATOMIC_INIT (0 );struct inode *inode;spin_lock_prefetch (&inode_lock);
@@ -746,7 +746,7 @@ struct inode *new_inode(struct super_block *sb)spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);spin_lock (&inode->i_lock); - inode->i_ino = ++last_ino; + inode->i_ino = (unsigned int )atomic_inc_return (&last_ino); inode->i_state =0 ;__inode_add_to_lists (sb,NULL , inode);spin_unlock (&inode->i_lock);
@@ -897,6 +897,22 @@ static struct inode *get_new_inode_fast(struct super_block *sb,return inode; } +static int test_inode_iunique (struct super_block * sb,struct hlist_head *head,unsigned long ino) +{ +struct hlist_node *node; +struct inode * inode =NULL ; + +spin_lock (&inode_hash_lock); +hlist_for_each_entry (inode, node, head, i_hash) { +if (inode->i_ino == ino && inode->i_sb == sb) { +spin_unlock (&inode_hash_lock); +return 0 ; + } + } +spin_unlock (&inode_hash_lock); +return 1 ; +} +/** * iunique - get a unique inode number * @sb: superblock
@@ -918,20 +934,20 @@ ino_t iunique(struct super_block *sb, ino_t max_reserved)* error if st_ino won't fit in target struct field. Use 32bit counter * here to attempt to avoid that. */ +static DEFINE_SPINLOCK (unique_lock);static unsigned int counter; -struct inode *inode;struct hlist_head *head;ino_t res;spin_lock (&inode_lock); +spin_lock (&unique_lock);do {if (counter <= max_reserved) counter = max_reserved +1 ; res = counter++; head = inode_hashtable +hash (sb, res); - inode =find_inode_fast (sb, head, res); -spin_unlock (&inode->i_lock); - }while (inode !=NULL ); + }while (!test_inode_iunique (sb, head, res)); +spin_unlock (&unique_lock);spin_unlock (&inode_lock);return res;
Signature · 1 line
1.7.1
Andrew Morton
maintainerRe: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock
Quoted 16 lines · On Wed, 29 Sep 2010 22:18:41 +1000 Dave Chinner <[email protected]> wrote:
From: Nick Piggin <[email protected]>
Before removing the
inode_lock, we need to make thelast_inoand iunique counters independent of theinode_lock.last_inocan be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside theinode_lockto provide the same protection that theinode_lockpreviously provided....
fs/inode.c +5
-0+static int test_inode_iunique (struct super_block * sb,struct hlist_head *head,unsigned long ino) +{ +struct hlist_node *node; +struct inode * inode =NULL ; +
fs/inode.c static struct inode *get_new_inode_fast(struct super_block *sb, · line 914
Quoted 10 lines
fs/inode.c +10
-0+spin_lock (&inode_hash_lock); +hlist_for_each_entry (inode, node, head, i_hash) { +if (inode->i_ino == ino && inode->i_sb == sb) { +spin_unlock (&inode_hash_lock); +return 0 ; + } + } +spin_unlock (&inode_hash_lock); +return 1 ; +}
Please run all the patches through checkpatch.
Please document this function. Why does it exist? What does it do? How does it do it? Try to improve the code!
Quoted 3 lines
...
Re: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock
In reply to Dave ChinnerQuoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before removing the
inode_lock, we need to make thelast_inoand iunique counters independent of theinode_lock.last_inocan be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside theinode_lockto provide the same protection that theinode_lockpreviously provided.
Given that last_ino becomes a per-cpu construct only a few patches later I think there's no point to make it an atomic_t here - just reorder the per-cpu patch before the inode_lock removal.
Re: [PATCH 09/17] fs: Make last_ino, iunique independent of inode_lock
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before removing the
inode_lock, we need to make thelast_inoand iunique counters independent of theinode_lock.last_inocan be trivially converted to an atomic variable, while the iunique counter needs a new lock nested inside theinode_lockto provide the same protection that theinode_lockpreviously provided.
Quoted 3 lines
Given that
last_inobecomes a per-cpu construct only a few patches later I think there's no point to make it anatomic_there - just reorder the per-cpu patch before theinode_lockremoval.
I wanted to avoid doing any of that until inode_lock is gone, but perhaps for this one it makes sense. At the very least, I'll merge the latter two patches into one, and perhaps this one too.
PATCH 10/17 fs: icache remove inode_lock
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
All the functionality that the inode_lock protected has now been wrapped up in new independent locks and/or functionality. Hence the inode_lock does not serve a purpose any longer and hence can now be removed.
--- Documentation/filesystems/Locking | 2 +- Documentation/filesystems/porting | 10 ++++- Documentation/filesystems/vfs.txt | 2 +- fs/buffer.c | 2 +- fs/drop_caches.c | 4 -- fs/fs-writeback.c | 47 ++++-------------- fs/inode.c | 95 +++++++------------------------------ fs/logfs/inode.c | 2 +- fs/notify/inode_mark.c | 23 +++++---- fs/notify/mark.c | 1 - fs/notify/vfsmount_mark.c | 1 - fs/ntfs/inode.c | 4 +- fs/ocfs2/inode.c | 2 +- fs/quota/dquot.c | 16 ++---- include/linux/fs.h | 2 +- include/linux/writeback.h | 1 - mm/backing-dev.c | 4 -- mm/filemap.c | 6 +- mm/rmap.c | 6 +- 19 files changed, 69 insertions(+), 161 deletions(-)
Documentation/filesystems/Locking +1 -1
@@ -114,7 +114,7 @@ alloc_inode: destroy_inode: dirty_inode: (must not sleep) write_inode: -drop_inode: !!!inode_lock!!! +drop_inode: !!!i_lock, sb_inode_list_lock!!! evict_inode: put_super: write write_super: read
Documentation/filesystems/porting +9 -1
@@ -299,7 +299,7 @@ be used instead. It gets called whenever the inode is evicted, whether it has remaining links or not. Caller does *not* evict the pagecache or inode-associated metadata buffers; getting rid of those is responsibility of method, as it had been for ->delete_inode(). - ->drop_inode() returns int now; it's called on final iput() with inode_lock + ->drop_inode() returns int now; it's called on final iput() with i_lock held and it returns true if filesystems wants the inode to be dropped. As before, generic_drop_inode() is still the default and it's been updated appropriately. generic_delete_inode() is also alive and it consists simply of return 1. Note that
@@ -318,3 +318,11 @@ if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput( may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly free the on-disk inode, you may end up doing that while ->write_inode() is writing to it. + + +[mandatory] + inode_lock is gone, replaced by fine grained locks. See fs/inode.c +for details of what locks to replace inode_lock with in order to protect +particular things. Most of the time, a filesystem only needs ->i_lock, which +protects *all* the inode state and its membership on lists that was +previously protected with inode_lock.
Documentation/filesystems/vfs.txt +1 -1
@@ -246,7 +246,7 @@ or bottom half). should be synchronous or not, not all filesystems check this flag. drop_inode: called when the last access to the inode is dropped, - with the inode_lock spinlock held. + with the i_lock and sb_inode_list_lock spinlock held. This method should be either NULL (normal UNIX filesystem semantics) or "generic_delete_inode" (for filesystems that do not
fs/buffer.c +1 -1
@@ -1145,7 +1145,7 @@ __getblk_slow(struct block_device *bdev, sector_t block, int size)* inode list. * * mark_buffer_dirty() is atomic. It takes bh->b_page->mapping->private_lock, -* mapping->tree_lock and the global inode_lock. +* and mapping->tree_lock. */ void mark_buffer_dirty (struct buffer_head *bh) {
fs/drop_caches.c +0 -4
@@ -16,7 +16,6 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused) {struct inode *inode, *toput_inode =NULL ; -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {spin_lock (&inode->i_lock);
@@ -28,15 +27,12 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)__iget (inode);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);invalidate_mapping_pages (inode->i_mapping,0 ,-1 );iput (toput_inode); toput_inode = inode; -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock); }spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);iput (toput_inode); }
fs/fs-writeback.c +11 -36
199 lines not on this page · raw message
fs/inode.c +18 -77
428 lines not on this page · raw message
fs/logfs/inode.c +1 -1
@@ -286,7 +286,7 @@ static int logfs_write_inode(struct inode *inode, struct writeback_control *wbc)return ret; } -/* called with inode_lock held */ +/* called with i_lock held */ static int logfs_drop_inode (struct inode *inode) {struct logfs_super *super =logfs_super (inode->i_sb);
fs/notify/inode_mark.c +12 -11
@@ -22,7 +22,7 @@#include <linux/module.h> #include <linux/mutex.h> #include <linux/spinlock.h> -#include <linux/writeback.h> /* for inode_lock */ +#include <linux/writeback.h> #include <asm/atomic.h>
@@ -232,9 +232,8 @@ out:* fsnotify_unmount_inodes - an sb is unmounting. handle any watched inodes. * @list: list of inodes being unmounted (sb->s_inodes) * -* Called with inode_lock held, protecting the unmounting super block's list -* of inodes, and with iprune_mutex held, keeping shrink_icache_memory() at bay. -* We temporarily drop inode_lock, however, and CAN block. +* Called with iprune_mutex held, keeping shrink_icache_memory() at bay. +* sb_inode_list_lock to protect the super block's list of inodes. */ void fsnotify_unmount_inodes (struct list_head *list) {
@@ -243,13 +242,16 @@ void fsnotify_unmount_inodes(struct list_head *list)list_for_each_entry_safe (inode, next_i, list, i_sb_list) {struct inode *need_iput_tmp; +spin_lock (&inode->i_lock);/* * We cannot __iget() an inode in state I_FREEING, * I_WILL_FREE, or I_NEW which is fine because by that point * the inode cannot have any associated watches. */ -if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) +if (inode->i_state & (I_FREEING |I_WILL_FREE |I_NEW )) { +spin_unlock (&inode->i_lock);continue ; + }/* * If i_count is zero, the inode cannot have any watches and
@@ -257,19 +259,20 @@ void fsnotify_unmount_inodes(struct list_head *list)* evict all inodes with zero i_count from icache which is * unnecessarily violent and may in fact be illegal to do. */ -if (!inode->i_count) +if (!inode->i_count) { +spin_unlock (&inode->i_lock);continue ; + } need_iput_tmp = need_iput; need_iput =NULL ;/* In case fsnotify_inode_delete() drops a reference. */ if (inode != need_iput_tmp) { -spin_lock (&inode->i_lock);__iget (inode); -spin_unlock (&inode->i_lock); }else need_iput_tmp =NULL ; +spin_unlock (&inode->i_lock);/* In case the dropping of a reference would nuke next_i. */ if (&next_i->i_sb_list != list) {
@@ -283,13 +286,12 @@ void fsnotify_unmount_inodes(struct list_head *list) }/* -* We can safely drop inode_lock here because we hold +* We can safely drop sb_inode_list_lock here because we hold * references on both inode and next_i. Also no new inodes * will be added since the umount has begun. Finally, * iprune_mutex keeps shrink_icache_memory() away. */ spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);if (need_iput_tmp)iput (need_iput_tmp);
@@ -301,7 +303,6 @@ void fsnotify_unmount_inodes(struct list_head *list)iput (inode); -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock); } }
fs/notify/mark.c +0 -1
@@ -91,7 +91,6 @@#include <linux/slab.h> #include <linux/spinlock.h> #include <linux/srcu.h> -#include <linux/writeback.h> /* for inode_lock */ #include <asm/atomic.h>
fs/notify/vfsmount_mark.c +0 -1
@@ -23,7 +23,6 @@#include <linux/mount.h> #include <linux/mutex.h> #include <linux/spinlock.h> -#include <linux/writeback.h> /* for inode_lock */ #include <asm/atomic.h>
fs/ntfs/inode.c +2 -2
@@ -54,7 +54,7 @@* * Return 1 if the attributes match and 0 if not. * -* NOTE: This function runs with the inode_lock spin lock held so it is not +* NOTE: This function runs with the i_lock spin lock held so it is not * allowed to sleep. */ int ntfs_test_inode (struct inode *vi,ntfs_attr *na)
@@ -98,7 +98,7 @@ int ntfs_test_inode(struct inode *vi, ntfs_attr *na)* * Return 0 on success and -errno on error. * -* NOTE: This function runs with the inode_lock spin lock held so it is not +* NOTE: This function runs with the i_lock spin lock held so it is not * allowed to sleep. (Hence the GFP_ATOMIC allocation.) */ static int ntfs_init_locked_inode (struct inode *vi,ntfs_attr *na)
fs/ocfs2/inode.c +1 -1
@@ -1195,7 +1195,7 @@ void ocfs2_evict_inode(struct inode *inode)ocfs2_clear_inode (inode); } -/* Called under inode_lock, with no more references on the +/* Called under i_lock, with no more references on the * struct inode, so it's safe here to check the flags field * and to manipulate i_nlink without any other locks. */ int ocfs2_drop_inode (struct inode *inode)
fs/quota/dquot.c +5 -11
@@ -76,7 +76,7 @@#include <linux/buffer_head.h> #include <linux/capability.h> #include <linux/quotaops.h> -#include <linux/writeback.h> /* for inode_lock, oddly enough.. */ +#include <linux/writeback.h> #include <asm/uaccess.h>
@@ -896,7 +896,6 @@ static void add_dquot_ref(struct super_block *sb, int type)int reserved =0 ;#endif -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {spin_lock (&inode->i_lock);
@@ -914,21 +913,18 @@ static void add_dquot_ref(struct super_block *sb, int type)__iget (inode);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);iput (old_inode);__dquot_initialize (inode, type);/* We hold a reference to 'inode' so it couldn't have been -* removed from s_inodes list while we dropped the inode_lock. -* We cannot iput the inode now as we can be holding the last -* reference and we cannot iput it under inode_lock. So we -* keep the reference and iput it later. */ +* removed from s_inodes list while we dropped the +* sb_inode_list_lock. We cannot iput the inode now as we can +* be holding the last reference and we cannot iput it under +* lock. So we keep the reference and iput it later. */ old_inode = inode; -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock); }spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);iput (old_inode);#ifdef CONFIG_QUOTA_DEBUG
@@ -1009,7 +1005,6 @@ static void remove_dquot_ref(struct super_block *sb, int type,struct inode *inode;int reserved =0 ; -spin_lock (&inode_lock);spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {/*
@@ -1025,7 +1020,6 @@ static void remove_dquot_ref(struct super_block *sb, int type, } }spin_unlock (&sb_inode_list_lock); -spin_unlock (&inode_lock);#ifdef CONFIG_QUOTA_DEBUG if (reserved) { printk(KERN_WARNING "VFS (%s): Writes happened after quota"
include/linux/fs.h +1 -1
@@ -1579,7 +1579,7 @@ struct super_operations { };/* -* Inode state bits. Protected by inode_lock. +* Inode state bits. Protected by i_lock. * * Three bits determine the dirty state of the inode, I_DIRTY_SYNC, * I_DIRTY_DATASYNC and I_DIRTY_PAGES.
include/linux/writeback.h +0 -1
@@ -9,7 +9,6 @@struct backing_dev_info ; -extern spinlock_t inode_lock;extern spinlock_t sb_inode_list_lock;extern spinlock_t wb_inode_list_lock;extern spinlock_t inode_hash_lock;
mm/backing-dev.c +0 -4
@@ -72,7 +72,6 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)struct inode *inode; nr_wb = nr_dirty = nr_io = nr_more_io =0 ; -spin_lock (&inode_lock);spin_lock (&wb_inode_list_lock);list_for_each_entry (inode, &wb->b_dirty, i_list) nr_dirty++;
@@ -81,7 +80,6 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)list_for_each_entry (inode, &wb->b_more_io, i_list) nr_more_io++;spin_unlock (&wb_inode_list_lock); -spin_unlock (&inode_lock);global_dirty_limits (&background_thresh, &dirty_thresh); bdi_thresh =bdi_dirty_limit (bdi, dirty_thresh);
@@ -682,13 +680,11 @@ void bdi_destroy(struct backing_dev_info *bdi)if (bdi_has_dirty_io (bdi)) {struct bdi_writeback *dst = &default_backing_dev_info.wb; -spin_lock (&inode_lock);spin_lock (&wb_inode_list_lock);list_splice (&bdi->wb.b_dirty, &dst->b_dirty);list_splice (&bdi->wb.b_io, &dst->b_io);list_splice (&bdi->wb.b_more_io, &dst->b_more_io);spin_unlock (&wb_inode_list_lock); -spin_unlock (&inode_lock); }bdi_unregister (bdi);
mm/filemap.c +3 -3
@@ -80,7 +80,7 @@* ->i_mutex * ->i_alloc_sem (various) * -* ->inode_lock +* ->i_lock * ->sb_lock (fs/fs-writeback.c) * ->mapping->tree_lock (__sync_single_inode) *
@@ -98,8 +98,8 @@* ->zone.lru_lock (check_pte_range->isolate_lru_page) * ->private_lock (page_remove_rmap->set_page_dirty) * ->tree_lock (page_remove_rmap->set_page_dirty) -* ->inode_lock (page_remove_rmap->set_page_dirty) -* ->inode_lock (zap_pte_range->set_page_dirty) +* ->i_lock (page_remove_rmap->set_page_dirty) +* ->i_lock (zap_pte_range->set_page_dirty) * ->private_lock (zap_pte_range->__set_page_dirty_buffers) * * ->task->proc_lock
mm/rmap.c +3 -3
@@ -31,11 +31,11 @@* swap_lock (in swap_duplicate, swap_info_get) * mmlist_lock (in mmput, drain_mmlist and others) * mapping->private_lock (in __set_page_dirty_buffers) -* inode_lock (in set_page_dirty's __mark_inode_dirty) -* sb_lock (within inode_lock in fs/fs-writeback.c) +* i_lock (in set_page_dirty's __mark_inode_dirty) +* sb_lock (within i_lock in fs/fs-writeback.c) * mapping->tree_lock (widely used, in set_page_dirty, * in arch-dependent flush_dcache_mmap_lock, -* within inode_lock in __sync_single_inode) +* within i_lock in __sync_single_inode) * * (code doesn't rely on that order so it could be switched around) * ->tasklist_lock
Signature · 1 line
1.7.1
PATCH 11/17 fs: Factor inode hash operations into functions
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the inode_hash_lock. Make it private by adding a function __remove_inode_hash that can be called by filesystems instead of open-coding their own inode hash removal operations.
--- fs/inode.c | 39 ++++++++++++++++++++++++--------------- include/linux/fs.h | 1 + include/linux/writeback.h | 1 - 3 files changed, 25 insertions(+), 16 deletions(-)
fs/inode.c +24 -15
@@ -105,7 +105,7 @@ static struct hlist_head *inode_hashtable __read_mostly;*/ DEFINE_SPINLOCK (sb_inode_list_lock);DEFINE_SPINLOCK (wb_inode_list_lock); -DEFINE_SPINLOCK (inode_hash_lock); +static DEFINE_SPINLOCK (inode_hash_lock);/* * iprune_sem provides exclusion between the kswapd or try_to_free_pages
@@ -371,9 +371,7 @@ static void dispose_list(struct list_head *head)spin_lock (&sb_inode_list_lock);spin_lock (&inode->i_lock); -spin_lock (&inode_hash_lock); -hlist_del_init (&inode->i_hash); -spin_unlock (&inode_hash_lock); +__remove_inode_hash (inode);list_del_init (&inode->i_sb_list);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);
@@ -1274,6 +1272,20 @@ void __insert_inode_hash(struct inode *inode, unsigned long hashval)EXPORT_SYMBOL (__insert_inode_hash);/** +* __remove_inode_hash - remove an inode from the hash +* @inode: inode to unhash +* +* Remove an inode from the superblock. inode->i_lock must be +* held. +*/ +void __remove_inode_hash (struct inode *inode) +{ +spin_lock (&inode_hash_lock); +hlist_del_init (&inode->i_hash); +spin_unlock (&inode_hash_lock); +} + +/** * remove_inode_hash - remove an inode from the hash * @inode: inode to unhash *
@@ -1282,9 +1294,7 @@ EXPORT_SYMBOL(__insert_inode_hash);void remove_inode_hash (struct inode *inode) {spin_lock (&inode->i_lock); -spin_lock (&inode_hash_lock); -hlist_del_init (&inode->i_hash); -spin_unlock (&inode_hash_lock); +__remove_inode_hash (inode);spin_unlock (&inode->i_lock); }EXPORT_SYMBOL (remove_inode_hash);
@@ -1348,9 +1358,7 @@ static void iput_final(struct inode *inode)spin_lock (&inode->i_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state &= ~I_WILL_FREE ; -spin_lock (&inode_hash_lock); -hlist_del_init (&inode->i_hash); -spin_unlock (&inode_hash_lock); +__remove_inode_hash (inode);atomic_dec (&inodes_stat.nr_unused); }spin_lock (&wb_inode_list_lock);
@@ -1363,11 +1371,12 @@ static void iput_final(struct inode *inode)spin_unlock (&inode->i_lock);atomic_dec (&inodes_stat.nr_inodes);evict (inode); -spin_lock (&inode->i_lock); -spin_lock (&inode_hash_lock); -hlist_del_init (&inode->i_hash); -spin_unlock (&inode_hash_lock); -spin_unlock (&inode->i_lock); + +/* +* i_lock is required to delete from hash because find_inode_fast +* might find us but go to sleep before we run wake_up_inode. +*/ +remove_inode_hash (inode);wake_up_inode (inode);BUG_ON (inode->i_state != (I_FREEING |I_CLEAR ));destroy_inode (inode);
include/linux/fs.h +1 -0
@@ -2186,6 +2186,7 @@ extern int should_remove_suid(struct dentry *);extern int file_remove_suid (struct file *);extern void __insert_inode_hash (struct inode *,unsigned long hashval); +extern void __remove_inode_hash (struct inode *);extern void remove_inode_hash (struct inode *);static inline void insert_inode_hash (struct inode *inode) {__insert_inode_hash (inode, inode->i_ino);
include/linux/writeback.h +0 -1
@@ -11,7 +11,6 @@ struct backing_dev_info;extern spinlock_t sb_inode_list_lock;extern spinlock_t wb_inode_list_lock; -extern spinlock_t inode_hash_lock;extern struct list_head inode_in_use;extern struct list_head inode_unused;
Signature · 1 line
1.7.1
Re: [PATCH 11/17] fs: Factor inode hash operations into functions
Quoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the
inode_hash_lock. Make it private by adding a function__remove_inode_hashthat can be called by filesystems instead of open-coding their own inode hash removal operations.
I like the factoring, but this changelog is misleading. At least in this series no new user of __remove_inode_hash appears, and I'm not sure where it would appear anyway. Just making the function global without actually exporting it is not helping external filesystems anyway. For now it can simply be made static.
Re: [PATCH 11/17] fs: Factor inode hash operations into functions
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Dave Chinner
From: Nick Piggin <[email protected]>
Before we can replace the inode hash locking with a more scalable mechanism, we need to remove external users of the
inode_hash_lock. Make it private by adding a function__remove_inode_hashthat can be called by filesystems instead of open-coding their own inode hash removal operations.
Quoted 5 lines
I like the factoring, but this changelog is misleading. At least in this series no new user of
__remove_inode_hashappears, and I'm not sure where it would appear anyway. Just making the function global without actually exporting it is not helping external filesystems anyway. For now it can simply be made static.
Yeah, hugetlbfs was using this a while back as I said, and I've missed refactoring it. Will do.
PATCH 12/17 fs: Introduce per-bucket inode hash locks
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Now the inode_hash_lock is private, we can change the hash locking to be more scalable. Convert the inode hash to use the new bit-locked hash list implementation that allows per-bucket locks to be used. This allows us to replace the global inode_hash_lock with finer grained locking without increasing the size of the hash table.
--- fs/btrfs/inode.c | 2 +- fs/fs-writeback.c | 2 +- fs/hfs/hfs_fs.h | 2 +- fs/hfs/inode.c | 2 +- fs/hfsplus/hfsplus_fs.h | 2 +- fs/hfsplus/inode.c | 2 +- fs/inode.c | 198 ++++++++++++++++++++++++++--------------------- fs/nilfs2/gcinode.c | 22 +++--- fs/nilfs2/segment.c | 2 +- fs/nilfs2/the_nilfs.h | 2 +- fs/reiserfs/xattr.c | 2 +- include/linux/fs.h | 3 +- mm/shmem.c | 4 +- 13 files changed, 135 insertions(+), 110 deletions(-)
fs/btrfs/inode.c +1 -1
@@ -3860,7 +3860,7 @@ again: p = &root->inode_tree.rb_node; parent =NULL ; -if (hlist_unhashed (&inode->i_hash)) +if (hlist_bl_unhashed (&inode->i_hash))return ;spin_lock (&root->inode_lock);
fs/fs-writeback.c +1 -1
@@ -990,7 +990,7 @@ void __mark_inode_dirty(struct inode *inode, int flags)* dirty list. Add blockdev inodes as well. */ if (!S_ISBLK (inode->i_mode)) { -if (hlist_unhashed (&inode->i_hash)) +if (hlist_bl_unhashed (&inode->i_hash)) goto out; }if (inode->i_state &I_FREEING )
fs/hfs/hfs_fs.h +1 -1
@@ -148,7 +148,7 @@ struct hfs_sb_info {int fs_div; -struct hlist_head rsrc_inodes; +struct hlist_bl_head rsrc_inodes; };#define HFS_FLG_BITMAP_DIRTY 0
fs/hfs/inode.c +1 -1
@@ -524,7 +524,7 @@ static struct dentry *hfs_file_lookup(struct inode *dir, struct dentry *dentry,HFS_I (inode)->rsrc_inode = dir;HFS_I (dir)->rsrc_inode = inode;igrab (dir); -hlist_add_head (&inode->i_hash, &HFS_SB (dir->i_sb)->rsrc_inodes); +hlist_bl_add_head (&inode->i_hash, &HFS_SB (dir->i_sb)->rsrc_inodes);mark_inode_dirty (inode); out:d_add (dentry, inode);
fs/hfsplus/hfsplus_fs.h +1 -1
@@ -144,7 +144,7 @@ struct hfsplus_sb_info {unsigned long flags; -struct hlist_head rsrc_inodes; +struct hlist_bl_head rsrc_inodes; };#define HFSPLUS_SB_WRITEBACKUP 0x0001
fs/hfsplus/inode.c +1 -1
@@ -202,7 +202,7 @@ static struct dentry *hfsplus_file_lookup(struct inode *dir, struct dentry *dentHFSPLUS_I (inode).rsrc_inode = dir;HFSPLUS_I (dir).rsrc_inode = inode;igrab (dir); -hlist_add_head (&inode->i_hash, &HFSPLUS_SB (sb).rsrc_inodes); +hlist_bl_add_head (&inode->i_hash, &HFSPLUS_SB (sb).rsrc_inodes);mark_inode_dirty (inode); out:d_add (dentry, inode);
fs/inode.c +110 -88
531 lines not on this page · raw message
fs/nilfs2/gcinode.c +12 -10
@@ -196,13 +196,13 @@ int nilfs_init_gccache(struct the_nilfs *nilfs)INIT_LIST_HEAD (&nilfs->ns_gc_inodes); nilfs->ns_gc_inodes_h = -kmalloc (sizeof (struct hlist_head ) *NILFS_GCINODE_HASH_SIZE , +kmalloc (sizeof (struct hlist_bl_head ) *NILFS_GCINODE_HASH_SIZE ,GFP_NOFS );if (nilfs->ns_gc_inodes_h ==NULL )return -ENOMEM ;for (loop =0 ; loop <NILFS_GCINODE_HASH_SIZE ; loop++) -INIT_HLIST_HEAD (&nilfs->ns_gc_inodes_h[loop]); +INIT_HLIST_BL_HEAD (&nilfs->ns_gc_inodes_h[loop]);return 0 ; }
@@ -254,18 +254,18 @@ static unsigned long ihash(ino_t ino, __u64 cno)*/ struct inode *nilfs_gc_iget (struct the_nilfs *nilfs,ino_t ino,__u64 cno) { -struct hlist_head *head = nilfs->ns_gc_inodes_h +ihash (ino, cno); -struct hlist_node *node; +struct hlist_bl_head *head = nilfs->ns_gc_inodes_h +ihash (ino, cno); +struct hlist_bl_node *node;struct inode *inode; -hlist_for_each_entry (inode, node, head, i_hash) { +hlist_bl_for_each_entry (inode, node, head, i_hash) {if (inode->i_ino == ino &&NILFS_I (inode)->i_cno == cno)return inode; } inode =alloc_gcinode (nilfs, ino, cno);if (likely (inode)) { -hlist_add_head (&inode->i_hash, head); +hlist_bl_add_head (&inode->i_hash, head);list_add (&NILFS_I (inode)->i_dirty, &nilfs->ns_gc_inodes); }return inode;
@@ -284,16 +284,18 @@ void nilfs_clear_gcinode(struct inode *inode)*/ void nilfs_remove_all_gcinode (struct the_nilfs *nilfs) { -struct hlist_head *head = nilfs->ns_gc_inodes_h; -struct hlist_node *node, *n; +struct hlist_bl_head *head = nilfs->ns_gc_inodes_h; +struct hlist_bl_node *node;struct inode *inode;int loop;for (loop =0 ; loop <NILFS_GCINODE_HASH_SIZE ; loop++, head++) { -hlist_for_each_entry_safe (inode, node, n, head, i_hash) { -hlist_del_init (&inode->i_hash); +restart: +hlist_bl_for_each_entry (inode, node, head, i_hash) { +hlist_bl_del_init (&inode->i_hash);list_del_init (&NILFS_I (inode)->i_dirty);nilfs_clear_gcinode (inode);/* might sleep */ + goto restart; } } }
fs/nilfs2/segment.c +1 -1
@@ -2452,7 +2452,7 @@ nilfs_remove_written_gcinodes(struct the_nilfs *nilfs, struct list_head *head)list_for_each_entry_safe (ii, n, head, i_dirty) {if (!test_bit (NILFS_I_UPDATED , &ii->i_state))continue ; -hlist_del_init (&ii->vfs_inode.i_hash); +hlist_bl_del_init (&ii->vfs_inode.i_hash);list_del_init (&ii->i_dirty);nilfs_clear_gcinode (&ii->vfs_inode); }
fs/nilfs2/the_nilfs.h +1 -1
@@ -167,7 +167,7 @@ struct the_nilfs {/* GC inode list and hash table head */ struct list_head ns_gc_inodes; -struct hlist_head *ns_gc_inodes_h; +struct hlist_bl_head *ns_gc_inodes_h;/* Disk layout information (static) */ unsigned int ns_blocksize_bits;
fs/reiserfs/xattr.c +1 -1
@@ -424,7 +424,7 @@ int reiserfs_prepare_write(struct file *f, struct page *page,static void update_ctime (struct inode *inode) {struct timespec now =current_fs_time (inode->i_sb); -if (hlist_unhashed (&inode->i_hash) || !inode->i_nlink || +if (hlist_bl_unhashed (&inode->i_hash) || !inode->i_nlink ||timespec_equal (&inode->i_ctime, &now))return ;
include/linux/fs.h +2 -1
@@ -376,6 +376,7 @@ struct files_stat_struct {#include <linux/capability.h> #include <linux/semaphore.h> #include <linux/fiemap.h> +#include <linux/rculist_bl.h> #include <asm/atomic.h> #include <asm/byteorder.h>
@@ -722,7 +723,7 @@ struct posix_acl;#define ACL_NOT_CACHED ((void *)(-1))struct inode { -struct hlist_node i_hash; +struct hlist_bl_node i_hash;struct list_head i_list;/* backing dev IO list */ struct list_head i_sb_list;struct list_head i_dentry;
mm/shmem.c +2 -2
@@ -2148,7 +2148,7 @@ static int shmem_encode_fh(struct dentry *dentry, __u32 *fh, int *len,if (*len <3 )return 255 ; -if (hlist_unhashed (&inode->i_hash)) { +if (hlist_bl_unhashed (&inode->i_hash)) {/* Unfortunately insert_inode_hash is not idempotent, * so as we hash inodes here rather than at creation * time, we need a lock to ensure we only try
@@ -2156,7 +2156,7 @@ static int shmem_encode_fh(struct dentry *dentry, __u32 *fh, int *len,*/ static DEFINE_SPINLOCK (lock);spin_lock (&lock); -if (hlist_unhashed (&inode->i_hash)) +if (hlist_bl_unhashed (&inode->i_hash))__insert_inode_hash (inode, inode->i_ino + inode->i_generation);spin_unlock (&lock);
Signature · 1 line
1.7.1
Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks
Instead of doing the lock overkill on a still fundamentally global data structure what about replacing this with something better. I know you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.
then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?
If it was up to me I'd dtop this (and the bl_list addition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.
Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks
Quoted 6 lines · Christoph Hellwig
Instead of doing the lock overkill on a still fundamentally global data structure what about replacing this with something better. I know you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.
The XFS inode cache design is tied tightly to the inode layout in XFS, so the tree-per-ag-per-mount parallelism design really does not work in a generic manner. Sure, we could probably make it a hashed-tree rather than hashed-link-list design, but that's a much more fundamental change than just splitting the locks up.
Quoted 2 lines
then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?
I've been running comparitive benchmarks on ext4 as well so that I also test all the generic paths.
Quoted 3 lines
If it was up to me I'd dtop this (and the
bl_listaddition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.
I think that it's going to take a lot of work to come up with something more generically optimal, so in the mean time I think this is a net win for filesystems that use the generic icache.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 12/17] fs: Introduce per-bucket inode hash locks
In reply to Christoph HellwigQuoted 2 lines · Christoph Hellwig
Instead of doing the lock overkill on a still fundamentally global data
How do you figure it is overkill? Actually the hash insertion/removal scales *really* well with per-bucket locks and it is a technique used and proven in other parts of the kernel like networking.
Having a global lock there is certainly a huge bottleneck when you start increasing system size, so I don't know why you keep arguing against this.
Quoted 1 line
structure what about replacing this with something better.
I won't be doing this until after the scalability work.
Quoted 10 lines
you've already done this with the XFS icache, and while the per-AG concept obviously can't be generic at least some of the lessons could be applied.
then again how much testing did this get anyway given that you benchmark ran mostly XFS which doesn't hit this at all?
If it was up to me I'd dtop this (and the
bl_listaddition) from the series for now and wait for people who care about the scalability of the generic icache code to come up with a better data structure.
I do care about scalability of icache code. Given how simple this is, and seeing as we're about to have the big locking rework, I much prefer just fixing all the global locks now (which need to be fixed anyway).
PATCH 13/17 fs: Implement lazy LRU updates for inodes.
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Convert the inode LRU to use lazy updates to reduce lock and cacheline traffic. We avoid moving inodes around in the LRU list during iget/iput operations so these frequent operations don't need to access the LRUs. Instead, we defer the refcount checks to reclaim-time and use a per-inode state flag, I_REFERENCED, to tell reclaim that iget has touched the inode in the past. This means that only reclaim should be touching the LRU with any frequency, hence significantly reducing lock acquisitions and the amount contention on LRU updates.
--- fs/fs-writeback.c | 20 +++-------- fs/inode.c | 81 +++++++++++++-------------------------------- include/linux/fs.h | 20 +++++++---- include/linux/writeback.h | 1 - 4 files changed, 42 insertions(+), 80 deletions(-)
fs/fs-writeback.c +6 -14
@@ -408,15 +408,8 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)* completion. */ redirty_tail (inode); - }else if (inode->i_count) { -/* -* The inode is clean, inuse -*/ -list_move (&inode->i_list, &inode_in_use); }else { -/* -* The inode is clean, unused -*/ +/* The inode is clean */ list_move (&inode->i_list, &inode_unused); } }
@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)*/ WARN_ON (!rwsem_is_locked (&sb->s_umount)); -spin_lock (&sb_inode_list_lock); -/* * Data integrity sync. Must wait for all pages under writeback, * because there may have been pages dirtied before our sync
@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)* In which case, the inode may not be on the dirty list, but * we still have to wait for that writeout. */ +spin_lock (&sb_inode_list_lock);list_for_each_entry (inode, &sb->s_inodes, i_sb_list) {struct address_space *mapping;
@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)spin_unlock (&sb_inode_list_lock);/* * We hold a reference to 'inode' so it couldn't have been -* removed from s_inodes list while we dropped the -* sb_inode_list_lock. We cannot iput the inode now as we can -* be holding the last reference and we cannot iput it under -* spinlock. So we keep the reference and iput it later. +* removed from s_inodes list while we dropped the i_lock. We +* cannot iput the inode now as we can be holding the last +* reference and we cannot iput it under spinlock. So we keep +* the reference and iput it later. */ iput (old_inode); old_inode = inode;
fs/inode.c +23 -58
164 lines not on this page · raw message
include/linux/fs.h +13 -7
@@ -1627,16 +1627,17 @@ struct super_operations {* * Q: What is the difference between I_WILL_FREE and I_FREEING? */ -#define I_DIRTY_SYNC 1 -#define I_DIRTY_DATASYNC 2 -#define I_DIRTY_PAGES 4 +#define I_DIRTY_SYNC 0x01 +#define I_DIRTY_DATASYNC 0x02 +#define I_DIRTY_PAGES 0x04#define __I_NEW 3#define I_NEW (1 << __I_NEW) -#define I_WILL_FREE 16 -#define I_FREEING 32 -#define I_CLEAR 64 +#define I_WILL_FREE 0x10 +#define I_FREEING 0x20 +#define I_CLEAR 0x40#define __I_SYNC 7#define I_SYNC (1 << __I_SYNC) +#define I_REFERENCED 0x100#define I_DIRTY (I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES)
@@ -2177,7 +2178,6 @@ extern int insert_inode_locked4(struct inode *, unsigned long, int (*test)(strucextern int insert_inode_locked (struct inode *);extern void unlock_new_inode (struct inode *); -extern void __iget (struct inode * inode);extern void iget_failed (struct inode *);extern void end_writeback (struct inode *);extern void destroy_inode (struct inode *);
@@ -2392,6 +2392,12 @@ extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt);extern void save_mount_options (struct super_block *sb,char *options);extern void replace_mount_options (struct super_block *sb,char *options); +static inline void __iget (struct inode *inode) +{ +assert_spin_locked (&inode->i_lock); + inode->i_count++; +} +static inline ino_t parent_ino (struct dentry *dentry) {ino_t res;
include/linux/writeback.h +0 -1
@@ -11,7 +11,6 @@ struct backing_dev_info;extern spinlock_t sb_inode_list_lock;extern spinlock_t wb_inode_list_lock; -extern struct list_head inode_in_use;extern struct list_head inode_unused;/*
Signature · 1 line
1.7.1
Re: [PATCH 13/17] fs: Implement lazy LRU updates for inodes.
Quoted 10 lines
fs/fs-writeback.c +0
-2@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)*/ WARN_ON (!rwsem_is_locked (&sb->s_umount)); -spin_lock (&sb_inode_list_lock); -/* * Data integrity sync. Must wait for all pages under writeback, * because there may have been pages dirtied before our sync
fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1061
Quoted 4 lines
fs/fs-writeback.c +1
-0@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)* In which case, the inode may not be on the dirty list, but * we still have to wait for that writeout. */ +spin_lock (&sb_inode_list_lock);
I think this should be folded back into the patch introducing sb_inode_list_lock.
fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1081
Quoted 12 lines
fs/fs-writeback.c +4
-4@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)spin_unlock (&sb_inode_list_lock);/* * We hold a reference to 'inode' so it couldn't have been -* removed from s_inodes list while we dropped the -* sb_inode_list_lock. We cannot iput the inode now as we can -* be holding the last reference and we cannot iput it under -* spinlock. So we keep the reference and iput it later. +* removed from s_inodes list while we dropped the i_lock. We +* cannot iput the inode now as we can be holding the last +* reference and we cannot iput it under spinlock. So we keep +* the reference and iput it later.
This also looks like a hunk that got in by accident and should be merged into an earlier patch.
fs/inode.c static int invalidate_list(struct list_head *head, struct list_head *dispose) · line 420
Quoted 11 lines
fs/inode.c +2
-1@@ -431,11 +412,12 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)invalidate_inode_buffers (inode);if (!inode->i_count) {spin_lock (&wb_inode_list_lock); -list_move (&inode->i_list, dispose); +list_del (&inode->i_list);spin_unlock (&wb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;spin_unlock (&inode->i_lock); +list_add (&inode->i_list, dispose);
Moving the list_add out of the lock looks fine, but I can't really see how it's related to the rest of the patch.
fs/inode.c again: · line 498
Quoted 7 lines
fs/inode.c +7
-0+if (inode->i_count || (inode->i_state & ~I_REFERENCED )) { +list_del_init (&inode->i_list); +spin_unlock (&inode->i_lock); +atomic_dec (&inodes_stat.nr_unused); +continue ; + } +if (inode->i_state) {
Slightly confusing but okay given the only i_state that will get us here is I_REFERENCED. Do we really care about the additional cycle or two a dumb compiler might generate when writing
if (inode->i_state & I_REFERENCED)
?
fs/inode.c again: · line 505
Quoted 2 lines
fs/inode.c +1
-0if (inode_has_buffers (inode) || inode->i_data.nrpages) { +list_move (&inode->i_list, &inode_unused);
Why are we now moving the inode to the front of the list?
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 657
Quoted 10 lines
fs/inode.c +0
-3@@ -687,9 +652,6 @@ __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,atomic_inc (&inodes_stat.nr_inodes);list_add (&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock); -spin_lock (&wb_inode_list_lock); -list_add (&inode->i_list, &inode_in_use); -spin_unlock (&wb_inode_list_lock);if (b) {spin_lock_bucket (b);hlist_bl_add_head (&inode->i_hash, &b->head);
At some point it might be worth to split this into
inode_add_to_sb_list
and
__inode_add_to_hash
but that can be left for later.
Re: [PATCH 13/17] fs: Implement lazy LRU updates for inodes.
Quoted 18 lines · Christoph Hellwig
Quoted 10 lines
fs/fs-writeback.c +0
-2@@ -1058,8 +1051,6 @@ static void wait_sb_inodes(struct super_block *sb)*/ WARN_ON (!rwsem_is_locked (&sb->s_umount)); -spin_lock (&sb_inode_list_lock); -/* * Data integrity sync. Must wait for all pages under writeback, * because there may have been pages dirtied before our sync fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1061
Quoted 4 lines
fs/fs-writeback.c +1
-0@@ -1067,6 +1058,7 @@ static void wait_sb_inodes(struct super_block *sb)* In which case, the inode may not be on the dirty list, but * we still have to wait for that writeout. */ +spin_lock (&sb_inode_list_lock);I think this should be folded back into the patch introducing
sb_inode_list_lock.
Quoted 15 lines
fs/fs-writeback.c static void wait_sb_inodes(struct super_block *sb) · line 1081
Quoted 12 lines
fs/fs-writeback.c +4
-4@@ -1083,10 +1075,10 @@ static void wait_sb_inodes(struct super_block *sb)spin_unlock (&sb_inode_list_lock);/* * We hold a reference to 'inode' so it couldn't have been -* removed from s_inodes list while we dropped the -* sb_inode_list_lock. We cannot iput the inode now as we can -* be holding the last reference and we cannot iput it under -* spinlock. So we keep the reference and iput it later. +* removed from s_inodes list while we dropped the i_lock. We +* cannot iput the inode now as we can be holding the last +* reference and we cannot iput it under spinlock. So we keep +* the reference and iput it later. This also looks like a hunk that got in by accident and should be merged into an earlier patch.
These two actually came from a patch to do rcu locking (which Dave has changed a bit, but originally due to my fault), so I'll fix those, thanks.
Quoted 6 lines
Quoted 6 lines
fs/inode.c +1
-1@@ -431,11 +412,12 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)invalidate_inode_buffers (inode);if (!inode->i_count) {spin_lock (&wb_inode_list_lock); -list_move (&inode->i_list, dispose); +list_del (&inode->i_list);
Quoted 8 lines
fs/inode.c static int invalidate_list(struct list_head *head, struct list_head *dispose) · line 420
Quoted 5 lines
fs/inode.c +1
-0spin_unlock (&wb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;spin_unlock (&inode->i_lock); +list_add (&inode->i_list, dispose);Moving the
list_addout of the lock looks fine, but I can't really see how it's related to the rest of the patch.
Just helps shows that dispose isn't being protected by wb_inode_list_lock, I guess.
Quoted 13 lines
fs/inode.c again: · line 498
Quoted 8 lines
fs/inode.c +7
-0+if (inode->i_count || (inode->i_state & ~I_REFERENCED )) { +list_del_init (&inode->i_list); +spin_unlock (&inode->i_lock); +atomic_dec (&inodes_stat.nr_unused); +continue ; + } +if (inode->i_state) {Slightly confusing but okay given the only
i_statethat will get us here isI_REFERENCED. Do we really care about the additional cycle or two a dumb compiler might generate when writing
Quoted 1 line
if (inode->i_state & I_REFERENCED)
Sure, why not.
Quoted 7 lines
?
fs/inode.c again: · line 505
Quoted 2 lines
fs/inode.c +1
-0if (inode_has_buffers (inode) || inode->i_data.nrpages) { +list_move (&inode->i_list, &inode_unused);Why are we now moving the inode to the front of the list?
It was always being moved to the front of the list, but with lazy LRU, iput_final doesn't move it for us, hence the list_move here.
Without this, it busy-spins and locks badly under heavy reclaim load when buffers or pagecache can't be invalidated.
Seeing as it wasn't obvious to you, I'll add a comment here.
I was thinking we should probably have a shortcut to go back to the tail of the LRU in case of invalidation success, but that's out of the scope of this patch and I never got around to testing such a change yet.
PATCH 14/17 fs: Inode counters do not need to be atomic.
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
atomics for counters do not scale on large machines, so convert them back to normal variables protected by spin locks. We can do this because the counters are associated with specific list operations that are protected by locks; nr_inodes can be protected by the sb_inode_list_lock, and nr_unused can be protected by the wb_inode_list_lock.
--- fs/fs-writeback.c | 6 ++---- fs/inode.c | 30 ++++++++++++------------------ include/linux/fs.h | 12 ++++++------ 3 files changed, 20 insertions(+), 28 deletions(-)
fs/fs-writeback.c +2 -4
@@ -743,8 +743,7 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - (atomic_read (&inodes_stat.nr_inodes) - -atomic_read (&inodes_stat.nr_unused)); + inodes_stat.nr_inodes - inodes_stat.nr_unused;if (nr_pages) {struct wb_writeback_work work = {
@@ -1116,8 +1115,7 @@ void writeback_inodes_sb(struct super_block *sb)WARN_ON (!rwsem_is_locked (&sb->s_umount)); work.nr_pages = nr_dirty + nr_unstable + - (atomic_read (&inodes_stat.nr_inodes) - -atomic_read (&inodes_stat.nr_unused)); + inodes_stat.nr_inodes - inodes_stat.nr_unused;bdi_queue_work (sb->s_bdi, &work);wait_for_completion (&done);
fs/inode.c +12 -18
@@ -139,8 +139,8 @@ static DECLARE_RWSEM(iprune_sem);* Statistics gathering.. */ struct inodes_stat_t inodes_stat = { - .nr_inodes =ATOMIC_INIT (0 ), - .nr_unused =ATOMIC_INIT (0 ), + .nr_inodes =0 , + .nr_unused =0 , };static struct kmem_cache *inode_cachep__read_mostly ;
@@ -376,7 +376,6 @@ static void dispose_list(struct list_head *head)destroy_inode (inode); nr_disposed++; } -atomic_sub (nr_disposed, &inodes_stat.nr_inodes); }/*
@@ -385,7 +384,7 @@ static void dispose_list(struct list_head *head)static int invalidate_list (struct list_head *head,struct list_head *dispose) {struct list_head *next; -int busy =0 , count =0 ; +int busy =0 ; next = head->next;for (;;) {
@@ -413,19 +412,17 @@ static int invalidate_list(struct list_head *head, struct list_head *dispose)if (!inode->i_count) {spin_lock (&wb_inode_list_lock);list_del (&inode->i_list); + inodes_stat.nr_unused--;spin_unlock (&wb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;spin_unlock (&inode->i_lock);list_add (&inode->i_list, dispose); - count++;continue ; }spin_unlock (&inode->i_lock); busy =1 ; } -/* only unused inodes may be cached with i_count zero */ -atomic_sub (count, &inodes_stat.nr_unused);return busy; }
@@ -471,7 +468,6 @@ EXPORT_SYMBOL(invalidate_inodes);static void prune_icache (int nr_to_scan) {LIST_HEAD (freeable); -int nr_pruned =0 ;unsigned long reap =0 ;down_read (&iprune_sem);
@@ -492,7 +488,7 @@ again:if (inode->i_count || (inode->i_state & ~I_REFERENCED )) {list_del_init (&inode->i_list);spin_unlock (&inode->i_lock); -atomic_dec (&inodes_stat.nr_unused); + inodes_stat.nr_unused--;continue ; }if (inode->i_state) {
@@ -518,9 +514,8 @@ again:WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;spin_unlock (&inode->i_lock); - nr_pruned++; + inodes_stat.nr_unused--; } -atomic_sub (nr_pruned, &inodes_stat.nr_unused);if (current_is_kswapd ())__count_vm_events (KSWAPD_INODESTEAL , reap);else
@@ -552,8 +547,7 @@ static int shrink_icache_memory(struct shrinker *shrink, int nr, gfp_t gfp_mask)return -1 ;prune_icache (nr); } -return (atomic_read (&inodes_stat.nr_unused) /100 ) * - sysctl_vfs_cache_pressure; +return (inodes_stat.nr_unused /100 ) * sysctl_vfs_cache_pressure; }static struct shrinker icache_shrinker = {
@@ -649,7 +643,7 @@ static inline void __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, struct inode *inode) { - atomic_inc(&inodes_stat.nr_inodes); + inodes_stat.nr_inodes++;list_add (&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock);if (b) {
@@ -1325,9 +1319,9 @@ static void iput_final(struct inode *inode)if (!(inode->i_state & (I_DIRTY |I_SYNC ))) {spin_lock (&wb_inode_list_lock);list_move (&inode->i_list, &inode_unused); + inodes_stat.nr_unused++;spin_unlock (&wb_inode_list_lock); } -atomic_inc (&inodes_stat.nr_unused);if (sb->s_flags &MS_ACTIVE ) {spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);
@@ -1347,16 +1341,16 @@ static void iput_final(struct inode *inode)if (!list_empty (&inode->i_list)) {spin_lock (&wb_inode_list_lock);list_del_init (&inode->i_list); -spin_unlock (&wb_inode_list_lock);if (!inode->i_state) -atomic_dec (&inodes_stat.nr_unused); + inodes_stat.nr_unused--; +spin_unlock (&wb_inode_list_lock); }list_del_init (&inode->i_sb_list); + inodes_stat.nr_inodes--;spin_unlock (&sb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;spin_unlock (&inode->i_lock); -atomic_dec (&inodes_stat.nr_inodes);evict (inode);/*
include/linux/fs.h +6 -6
@@ -32,6 +32,12 @@#define SEEK_END 2/* seek relative to end of file */ #define SEEK_MAX SEEK_END +struct inodes_stat_t { +int nr_inodes; +int nr_unused; +int dummy[5 ];/* padding for sysctl ABI compatibility */ +}; +/* And dynamically-tunable limits and defaults: */ struct files_stat_struct {int nr_files;/* read only */
@@ -410,12 +416,6 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, ssize_t bytes, void *private, int ret, bool is_async); -struct inodes_stat_t { -atomic_t nr_inodes; -atomic_t nr_unused; -int dummy[5 ];/* padding for sysctl ABI compatibility */ -}; -/* * Attribute flags. These should be or-ed together to figure out what * has been changed!
Signature · 1 line
1.7.1
PATCH 15/17 fs: inode per-cpu last_ino allocator
In reply to Dave ChinnerFrom: Eric Dumazet <[email protected]>
last_ino was converted to an atomic variable to allow the inode_lock to go away. However, contended atomics do not scale on large machines, and new_inode() triggers excessive contention in such situations.
Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).
[npiggin: some extra commenting and use of defines]
--- fs/inode.c | 50 +++++++++++++++++++++++++++++++++++++++++++------- 1 files changed, 43 insertions(+), 7 deletions(-)
fs/inode.c +43 -7
@@ -653,6 +653,48 @@ __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, } } +#ifdef CONFIG_SMP +#define LAST_INO_BATCH 1024 +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); +static atomic_t shared_last_ino; + +static unsigned int last_ino_get (void ) +{ +unsigned int *p = &get_cpu_var (last_ino); +unsigned int res = *p; + +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) + res = (unsigned int )atomic_add_return (LAST_INO_BATCH , + &shared_last_ino) -LAST_INO_BATCH ; + + *p = ++res; +put_cpu_var (last_ino); +return res; +} +#else +static unsigned int last_ino_get (void ) +{ +static unsigned int last_ino; + +return ++last_ino; +} +#endif +/** * inode_add_to_lists - add a new inode to relevant lists * @sb: superblock inode belongs to
@@ -690,19 +732,13 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);*/ struct inode *new_inode (struct super_block *sb) { -/* -* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW -* error if st_ino won't fit in target struct field. Use 32bit counter -* here to attempt to avoid that. -*/ -static atomic_t last_ino =ATOMIC_INIT (0 );struct inode *inode; inode =alloc_inode (sb);if (inode) {spin_lock (&sb_inode_list_lock);spin_lock (&inode->i_lock); - inode->i_ino = (unsigned int )atomic_inc_return (&last_ino); + inode->i_ino =last_ino_get (); inode->i_state =0 ;__inode_add_to_lists (sb,NULL , inode);spin_unlock (&inode->i_lock);
Signature · 1 line
1.7.1
Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 6 lines · Dave Chinner
From: Eric Dumazet <[email protected]>
last_inowas converted to an atomic variable to allow theinode_lockto go away. However, contended atomics do not scale on large machines, andnew_inode()triggers excessive contention in such situations.
And the good thing is most users of new_inode couldn't care less about the fake i_ino assigned because they have a real inode number. So the first step is to move the i_ino assignment into a separate helper and only use it in those filesystems that need it. Second step is to figure out why some filesystems need iunique() and some are fine with the incrementing counter and then we should find a scalable way to generate an inode number - preferably just one and not to, but if that's not possible we need some documentation on why which one is needed.
Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner
From: Eric Dumazet <[email protected]>
last_inowas converted to an atomic variable to allow theinode_lockto go away. However, contended atomics do not scale on large machines, andnew_inode()triggers excessive contention in such situations.
Quoted 9 lines
And the good thing is most users of
new_inodecouldn't care less about the fakei_inoassigned because they have a real inode number. So the first step is to move thei_inoassignment into a separate helper and only use it in those filesystems that need it. Second step is to figure out why some filesystems neediunique()and some are fine with the incrementing counter and then we should find a scalable way to generate an inode number - preferably just one and not to, but if that's not possible we need some documentation on why which one is needed.
Sounds like a good plan, but I don't really have time right now to understand the iget routines of every single filesystem to determine which rely on the current new_inode() allocated inode number. I think that is best left for a later cleanup, seeing as the last_ino scalability problem is easily addressed....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 5 lines · Dave Chinner
Sounds like a good plan, but I don't really have time right now to understand the iget routines of every single filesystem to determine which rely on the current
new_inode()allocated inode number. I think that is best left for a later cleanup, seeing as thelast_inoscalability problem is easily addressed...
It's fairly easy to do it pessimisticly - all disk based filesystem don't need it. Anyway, I can do this ontop of your series later. It just seems a bit counter-intuitive to scale something we don't actually need it most cases.
Andrew Morton
maintainerRe: [PATCH 15/17] fs: inode per-cpu last_ino allocator
In reply to Dave ChinnerQuoted 35 lines · On Wed, 29 Sep 2010 22:18:47 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
last_inowas converted to an atomic variable to allow theinode_lockto go away. However, contended atomics do not scale on large machines, andnew_inode()triggers excessive contention in such situations.Solve this problem by providing to each cpu a
per_cpuvariable, feeded by the sharedlast_ino, but once every 1024 allocations. This reduces contention on the sharedlast_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).[npiggin: some extra commenting and use of defines]
...
fs/inode.c +18
-0+#ifdef CONFIG_SMP +#define LAST_INO_BATCH 1024 +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino);
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 683
Quoted 10 lines
fs/inode.c +10
-0+static atomic_t shared_last_ino; + +static unsigned int last_ino_get (void ) +{ +unsigned int *p = &get_cpu_var (last_ino); +unsigned int res = *p; + +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) + res = (unsigned int )atomic_add_return (LAST_INO_BATCH , + &shared_last_ino) -LAST_INO_BATCH ;
May as well remove the "- LAST_INO_BATCH" there, I think. It'll skew the results a tad at startup, but why does that matter?
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695
Quoted 11 lines
fs/inode.c +11
-0+ *p = ++res; +put_cpu_var (last_ino); + return res; +} +#else +static unsigned int last_ino_get (void ) +{ +static unsigned int last_ino; + +return ++last_ino; +}
This is racy with CONFIG_PREEMPT on some architectures, I suspect. I'd suggest conversion to atomic_t with, of course, an explanatory comment ;)
Quoted 1 line
+#endif
Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 4 lines · Andrew Morton
Quoted 2 lines · On Wed, 29 Sep 2010 22:18:47 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
Please note my new email address, thanks.
Quoted 38 lines
Quoted 38 lines
last_inowas converted to an atomic variable to allow theinode_lockto go away. However, contended atomics do not scale on large machines, andnew_inode()triggers excessive contention in such situations.Solve this problem by providing to each cpu a
per_cpuvariable, feeded by the sharedlast_ino, but once every 1024 allocations. This reduces contention on the sharedlast_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).[npiggin: some extra commenting and use of defines]
...
fs/inode.c +23
-0+#ifdef CONFIG_SMP +#define LAST_INO_BATCH 1024 +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); +static atomic_t shared_last_ino; + +static unsigned int last_ino_get (void ) +{ +unsigned int *p = &get_cpu_var (last_ino);
Quoted 8 lines
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 683
Quoted 5 lines
fs/inode.c +5
-0+unsigned int res = *p; + +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) + res = (unsigned int )atomic_add_return (LAST_INO_BATCH , + &shared_last_ino) -LAST_INO_BATCH ;May as well remove the "-
LAST_INO_BATCH" there, I think. It'll skew the results a tad at startup, but why does that matter?
Because on x86, atomic_add_return(val, ptr) uses xadd() + val
So, using "atomic_add_return(val, ptr) - val" removes one instruction ;)
Quoted 16 lines
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695
Quoted 12 lines
fs/inode.c +11
-0+ *p = ++res; +put_cpu_var (last_ino); + return res; +} +#else +static unsigned int last_ino_get (void ) +{ +static unsigned int last_ino; + +return ++last_ino; +}This is racy with
CONFIG_PREEMPTon some architectures, I suspect. I'd suggest conversion toatomic_twith, of course, an explanatory comment ;)
Thanks, I'll rework the patch !
I am pretty happy to see some interest on this patch serie, eventually :)
Re: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Andrew Morton
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695
Quoted 6 lines
fs/inode.c +6
-0+static unsigned int last_ino_get (void ) +{ +static unsigned int last_ino; + +return ++last_ino; +}This is racy with
CONFIG_PREEMPTon some architectures, I suspect. I'd suggest conversion toatomic_twith, of course, an explanatory comment ;)
In fact this code was OK when I submitted my original patch back in 2008, since it replaced fs/inode.c
inode->i_ino = ++last_ino;
And this was protected by a surrounding spinlock
(spin_lock(&inode_lock); at that time)
Even after Nick patches, preemption is still disabled (by two spinlocks... spin_lock(&sb_inode_list_lock); / spin_lock(&inode->i_lock);)
So patch 15/17 seems good to me, I re-sign it as-is ;)
If it happens preemption is re-enabled later (with future patches), we might need to change last_ino_get() too.
Thanks !
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Andrew Morton
maintainerRe: [PATCH 15/17] fs: inode per-cpu last_ino allocator
Quoted 18 lines · On Thu, 30 Sep 2010 09:53:09 +0200 Eric Dumazet <[email protected]> wrote:
Le jeudi 30 septembre 2010 __ 07:36 +0200, Eric Dumazet a
__crit:Quoted 13 lines
Le mercredi 29 septembre 2010 __ 21:53 -0700, Andrew Morton a
__crit:Quoted 10 lines
fs/inode.c __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, · line 695
Quoted 6 lines
fs/inode.c +6
-0+static unsigned int last_ino_get (void ) +{ +static unsigned int last_ino; + +return ++last_ino; +}This is racy with
CONFIG_PREEMPTon some architectures, I suspect. I'd suggest conversion toatomic_twith, of course, an explanatory comment ;)In fact this code was OK when I submitted my original patch back in 2008, since it replaced
fs/inode.c
Quoted 8 lines
inode->i_ino = ++last_ino;And this was protected by a surrounding spinlock
(spin_lock(&inode_lock); at that time)Even after Nick patches, preemption is still disabled (by two spinlocks...
spin_lock(&sb_inode_list_lock); /spin_lock(&inode->i_lock);)
You know, if it took you and me this long to work that out then perhaps the code isn't quite as clear as we would like it to be, no?
I think you know what's coming next ;)
As a general rule, if a reviewer's comment doesn't result in a code change then it should result in a changelog fix or a code comment. Because if the code wasn't clear enough to the reviewer then it won't be clear enough to later readers.
Quoted 6 lines
So patch 15/17 seems good to me, I re-sign it as-is ;)
Signed-off-by Eric Dumazet <[email protected]>If it happens preemption is re-enabled later (with future patches), we might need to change
last_ino_get()too.
Perhaps
WARN_ON_ONCE(preemptible());
if we had a developer-only version of WARN_ON_ONCE, which we don't.
PATCH 1/1 fs: inode per-cpu last_ino allocator
Quoted 5 lines · Andrew Morton
Perhaps
WARN_ON_ONCE(preemptible());if we had a developer-only version of
WARN_ON_ONCE, which we don't.
Or just use a regular PER_CPU variable, even on !SMP, and get preempt safe implementation.
What do you think of following patch, on top of current linux-2.6 tree ?
Thanks
[PATCH] fs: inode per-cpu last_ino allocator
new_inode() dirties a contended cache line to get increasing inode numbers.
Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).
--- fs/inode.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
fs/inode.c +38 -7
@@ -624,6 +624,43 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode) }EXPORT_SYMBOL_GPL (inode_add_to_lists); +#define LAST_INO_BATCH 1024 + +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); + +static noinline unsigned int last_ino_get(void) +{ +unsigned int *p = &get_cpu_var (last_ino); +unsigned int res = *p; + +#ifdef CONFIG_SMP +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) { +static atomic_t shared_last_ino; +int next =atomic_add_return (LAST_INO_BATCH , &shared_last_ino); + + res = next -LAST_INO_BATCH ; + } +#endif + *p = ++res; +put_cpu_var (last_ino); +return res; +} +/** * new_inode - obtain an inode * @sb: superblock
@@ -638,12 +675,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);*/ struct inode *new_inode (struct super_block *sb) { -/* -* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW -* error if st_ino won't fit in target struct field. Use 32bit counter -* here to attempt to avoid that. -*/ -static unsigned int last_ino;struct inode *inode;spin_lock_prefetch (&inode_lock);
@@ -652,7 +683,7 @@ struct inode *new_inode(struct super_block *sb)if (inode) {spin_lock (&inode_lock);__inode_add_to_lists (sb,NULL , inode); - inode->i_ino = ++last_ino; + inode->i_ino =last_ino_get (); inode->i_state =0 ;spin_unlock (&inode_lock); }
Andrew Morton
maintainerRe: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 10 lines · On Thu, 30 Sep 2010 12:22:16 +0200 Eric Dumazet <[email protected]> wrote:
Le jeudi 30 septembre 2010 __ 01:14 -0700, Andrew Morton a
__crit:Quoted 5 lines
Perhaps
WARN_ON_ONCE(preemptible());if we had a developer-only version of
WARN_ON_ONCE, which we don't.Or just use a regular
PER_CPUvariable, even on !SMP, and get preempt safe implementation.
Good stuff.
Quoted 11 lines
What do you think of following patch, on top of current linux-2.6 tree ?
...
fs/inode.c +7
-0+static noinline unsigned int last_ino_get(void) +{ +unsigned int *p = &get_cpu_var (last_ino); +unsigned int res = *p; + +#ifdef CONFIG_SMP +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) {
fs/inode.c void inode_add_to_lists(struct super_block *sb, struct inode *inode) · line 662
Quoted 10 lines
fs/inode.c +10
-0+static atomic_t shared_last_ino; +int next =atomic_add_return (LAST_INO_BATCH , &shared_last_ino); + + res = next - LAST_INO_BATCH; + } +#endif + *p = ++res; +put_cpu_var (last_ino); +return res; +}
Could eliminate `p' I guess, but that would involve using __get_cpu_var() as an lval, which looks vile and might generate worse code.
Readers of this code won't know why last_ino_get() was marked noinline. It looks wrong, really.
Re: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 4 lines · Andrew Morton
Could eliminate `p' I guess, but that would involve using
__get_cpu_var()as an lval, which looks vile and might generate worse code.
Hmm, I see, please check this new patch, using the most modern stuff ;)
Quoted 2 lines
Readers of this code won't know why
last_ino_get()was marked noinline. It looks wrong, really.
Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !
Here is the new generated code on i686 (with the noinline) :
pretty good ;)
c02e5930 <last_ino_get>:
c02e5930: 55 push %ebp c02e5931: 89 e5 mov %esp,%ebp c02e5933: 64 a1 44 29 7d c0 mov %fs:0xc07d2944,%eax c02e5939: a9 ff 03 00 00 test $0x3ff,%eax c02e593e: 74 09 je c02e5949 <last_ino_get+0x19> c02e5940: 40 inc %eax c02e5941: 64 a3 44 29 7d c0 mov %eax,%fs:0xc07d2944 c02e5947: c9 leave c02e5948: c3 ret c02e5949: b8 00 04 00 00 mov $0x400,%eax
c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880c02e5956: eb e8 jmp c02e5940 <last_ino_get+0x10>
Thanks
[PATCH] fs: inode per-cpu last_ino allocator
new_inode() dirties a contended cache line to get increasing inode numbers.
Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations. This reduces contention on the shared last_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).
--- fs/inode.c | 47 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-)
fs/inode.c +40 -7
@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode) }EXPORT_SYMBOL_GPL (inode_add_to_lists); +#define LAST_INO_BATCH 1024 + +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); + +static unsigned int last_ino_get (void ) +{ +unsigned int res; + +get_cpu (); + res =__this_cpu_read (last_ino); +#ifdef CONFIG_SMP +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) { +static atomic_t shared_last_ino; +int next =atomic_add_return (LAST_INO_BATCH , &shared_last_ino); + + res = next -LAST_INO_BATCH ; + } +#endif + res++; +__this_cpu_write (last_ino, res); +put_cpu (); +return res; +} +/** * new_inode - obtain an inode * @sb: superblock
@@ -638,12 +677,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);*/ struct inode *new_inode (struct super_block *sb) { -/* -* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW -* error if st_ino won't fit in target struct field. Use 32bit counter -* here to attempt to avoid that. -*/ -static unsigned int last_ino;struct inode *inode;spin_lock_prefetch (&inode_lock);
@@ -652,7 +685,7 @@ struct inode *new_inode(struct super_block *sb)if (inode) {spin_lock (&inode_lock);__inode_add_to_lists (sb,NULL , inode); - inode->i_ino = ++last_ino; + inode->i_ino =last_ino_get (); inode->i_state =0 ;spin_unlock (&inode_lock); }
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Andrew Morton
maintainerRe: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 29 lines · On Thu, 30 Sep 2010 19:28:05 +0200 Eric Dumazet <[email protected]> wrote:
Le jeudi 30 septembre 2010 __ 09:45 -0700, Andrew Morton a
__crit:Quoted 4 lines
Could eliminate `p' I guess, but that would involve using
__get_cpu_var()as an lval, which looks vile and might generate worse code.Hmm, I see, please check this new patch, using the most modern stuff ;)
Quoted 2 lines
Readers of this code won't know why
last_ino_get()was marked noinline. It looks wrong, really.Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !
Here is the new generated code on i686 (with the noinline) :
pretty good ;)
c02e5930<last_ino_get>:c02e5930: 55 push %ebp c02e5931: 89 e5 mov %esp,%ebp c02e5933: 64 a1 44 29 7d c0 mov %fs:0xc07d2944,%eax c02e5939: a9 ff 03 00 00 test $0x3ff,%eax c02e593e: 74 09 je c02e5949 <last_ino_get+0x19> c02e5940: 40 inc %eax c02e5941: 64 a3 44 29 7d c0 mov %eax,%fs:0xc07d2944 c02e5947: c9 leave c02e5948: c3 ret c02e5949: b8 00 04 00 00 mov $0x400,%eax
Quoted 3 lines
c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880c02e5956: eb e8 jmpc02e5940<last_ino_get+0x10>
That uniprocessor, PREEMPT=n I guess.
Quoted 34 lines
fs/inode.c +28
-0@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode) }EXPORT_SYMBOL_GPL (inode_add_to_lists); +#define LAST_INO_BATCH 1024 + +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); + +static unsigned int last_ino_get (void ) +{ +unsigned int res; + +get_cpu (); + res =__this_cpu_read (last_ino); +#ifdef CONFIG_SMP +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) { +static atomic_t shared_last_ino;
fs/inode.c void inode_add_to_lists(struct super_block *sb, struct inode *inode) · line 664
Quoted 10 lines
fs/inode.c +10
-0+int next =atomic_add_return (LAST_INO_BATCH , &shared_last_ino); + + res = next - LAST_INO_BATCH; + } +#endif + res++; +__this_cpu_write (last_ino, res); +put_cpu (); +return res; +}
Looks good ;)
Re: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 35 lines · Andrew Morton
Quoted 29 lines · On Thu, 30 Sep 2010 19:28:05 +0200 Eric Dumazet <[email protected]> wrote:
Le jeudi 30 septembre 2010 __ 09:45 -0700, Andrew Morton a
__crit:Quoted 4 lines
Could eliminate `p' I guess, but that would involve using
__get_cpu_var()as an lval, which looks vile and might generate worse code.Hmm, I see, please check this new patch, using the most modern stuff ;)
Quoted 2 lines
Readers of this code won't know why
last_ino_get()was marked noinline. It looks wrong, really.Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !
Here is the new generated code on i686 (with the noinline) :
pretty good ;)
c02e5930<last_ino_get>:c02e5930: 55 push %ebp c02e5931: 89 e5 mov %esp,%ebp c02e5933: 64 a1 44 29 7d c0 mov %fs:0xc07d2944,%eax c02e5939: a9 ff 03 00 00 test $0x3ff,%eax c02e593e: 74 09 je c02e5949 <last_ino_get+0x19> c02e5940: 40 inc %eax c02e5941: 64 a3 44 29 7d c0 mov %eax,%fs:0xc07d2944 c02e5947: c9 leave c02e5948: c3 ret c02e5949: b8 00 04 00 00 mov $0x400,%eaxQuoted 3 lines
c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880c02e5956: eb e8 jmpc02e5940<last_ino_get+0x10>
Quoted 2 lines
That uniprocessor, PREEMPT=n I guess.
It is SMP. PREEMPT=n
On UP, you would not have the %fs suffixes, and no "lock" before the xadd.
Re: [PATCH] fs: inode per-cpu last_ino allocator
In reply to Eric DumazetEric,
what workload did this matter for? As mentioned elsewhere we don't actually need to set i_ino in new_inode in many cases. Normal disk/network filesystem already set their own i_ino anyway and don't need it at all. Various in-kernel filesystems don't need any valid i_ino and Nick's full series actually has a patch dealing with it, which only leaves various user-mountable syntetic filesystems that want to generate an inode number, and IMHO they're better off using iunique.
Re: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 11 lines · Christoph Hellwig
Eric,
what workload did this matter for? As mentioned elsewhere we don't actually need to set
i_inoinnew_inodein many cases. Normal disk/network filesystem already set their owni_inoanyway and don't need it at all. Various in-kernel filesystems don't need any validi_inoand Nick's full series actually has a patch dealing with it, which only leaves various user-mountable syntetic filesystems that want to generate an inode number, and IMHO they're better off using iunique.
Dont focus on this small last_ino problem, it is only one problem out of a huge pile.
As you might know, I am mostly a netdev guy ;)
In 2008, I tried to scale workloads that need to setup / dismantles thousand of sockets per second. Then Nick took the challenge, because it was clear nobody was interested to review my stuff, or maybe I was not enough re-sending patches.
FS guys live in a closed world, it seems, they want to control every bit of patches.
for (i = 0; i < LIMIT; i++)
close(socket(...));
My results were impressive :
(From 27.5 seconds to 1.62 s, on a 8 cpus machine)
http://linux.derkeiler.com/Mailing-Lists/Kernel/2008-12/msg04479.html
(fs: Scalability of sockets/pipes allocation/deallocation on SMP)
<quote>
Hi Andrew
Take v2 of this patch serie got no new feedback, maybe its time for mm inclusion for a while ?
In this third version I added last two patches, one intialy from Christoph
Lameter, and one to avoid dirtying mnt->mnt_count on hardwired fs.
Many thanks to Christoph and Paul for this SLAB_DESTROY_PER_RCU work done
on "struct file".
Thank you
Short summary : Nice speedups for allocation/deallocation of sockets/pipes
(From 27.5 seconds to 1.62 s, on a 8 cpus machine)
Long version :
To allocate a socket or a pipe we :
0) Do the usual file table manipulation (pretty scalable these days, but would be faster if 'struct file' were using SLAB_DESTROY_BY_RCU and avoid call_rcu() cache killer). This point is addressed by 6th patch.
1) allocate an inode with new_inode()
This function :
- locks inode_lock,
- dirties nr_inodes counter
- dirties inode_in_use list (for sockets/pipes, this is useless)
- dirties superblock s_inodes. - dirties last_ino counter All these are in different cache lines unfortunatly.
2) allocate a dentryd_alloc() takes dcache_lock,
insert dentry on its parent list (dirtying sock_mnt->mnt_sb->s_root) dirties nr_dentry
3) d_instantiate() dentry (dcache_lock taken again)
4) init_file() -> atomic_inc() on sock_mnt->refcount
At close() time, we must undo the things. Its even more expensive because
of the _atomic_dec_and_lock() that stress a lot, and because of two cache
lines that are touched when an element is deleted from a list (previous and next items)
This is really bad, since sockets/pipes dont need to be visible in dcache
or an inode list per super block.
This patch series get rid of all but one contended cache lines for sockets, pipes and anonymous fd (signalfd, timerfd, ...)
socketallocbench is a very simple program (attached to this mail) that makes
a loop :
for (i = 0; i < 1000000; i++)close(socket(AF_INET, SOCK_STREAM, 0));
Cost if one cpu runs the program :
real 1.561s
user 0.092s
sys 1.469s
Cost if 8 processes are launched on a 8 CPU machine (socketallocbench -n 8) :
real 27.496s <<<< !!!! >>>>
user 0.657s
sys 3m39.092s
Oprofile results (for the 8 process run, 3 times):
CPU: Core 2, speed 3000.03 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (Unhalted core cycles) count 100000
samples cum. samples % cum. % symbol name
3347352 3347352 28.0232 28.0232 _atomic_dec_and_lock 3301428 6648780 27.6388 55.6620 d_instantiate
2971130 9619910 24.8736 80.5355 d_alloc
241318 9861228 2.0203 82.5558 init_file
146190 10007418 1.2239 83.7797 __slab_free
144149 10151567 1.2068 84.9864 inotify_d_instantiate 143971 10295538 1.2053 86.1917 inet_create
137168 10432706 1.1483 87.3401 new_inode
117549 10550255 0.9841 88.3242 add_partial
110795 10661050 0.9275 89.2517 generic_drop_inode
107137 10768187 0.8969 90.1486 kmem_cache_alloc
94029 10862216 0.7872 90.9358 tcp_close
82837 10945053 0.6935 91.6293 dput
67486 11012539 0.5650 92.1943 dentry_iput
57751 11070290 0.4835 92.6778 iput
54327 11124617 0.4548 93.1326 tcp_v4_init_sock
49921 11174538 0.4179 93.5505 sysenter_past_esp
47616 11222154 0.3986 93.9491 kmem_cache_free
30792 11252946 0.2578 94.2069 clear_inode
27540 11280486 0.2306 94.4375 copy_from_user
26509 11306995 0.2219 94.6594 init_timer
26363 11333358 0.2207 94.8801 discard_slab
25284 11358642 0.2117 95.0918 __fput
22482 11381124 0.1882 95.2800 __percpu_counter_add
20369 11401493 0.1705 95.4505 sock_alloc
18501 11419994 0.1549 95.6054 inet_csk_destroy_sock
17923 11437917 0.1500 95.7555 sys_close
This patch serie avoids all contented cache lines and makes this "bench" pretty fast.
New cost if run on one cpu :
real 1.245s (instead of 1.561s)
user 0.074s
sys 1.161s
If run on 8 CPUS :
real 1.624s
user 0.580s
sys 12.296s
On oprofile, we finally can see network stuff coming at the front of expensive stuff. (with the exception of kmem_cache_[z]alloc(), because it has to clear 192 bytes of file structures, this takes half of the time)
CPU: Core 2, speed 3000.09 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100
000
samples cum. samples % cum. % symbol name
176586 176586 10.9376 10.9376 kmem_cache_alloc
169838 346424 10.5196 21.4572 tcp_close
105331 451755 6.5241 27.9813 tcp_v4_init_sock
105146 556901 6.5126 34.4939 tcp_v4_destroy_sock
83307 640208 5.1600 39.6539 sysenter_past_esp
80241 720449 4.9701 44.6239 inet_csk_destroy_sock
74263 794712 4.5998 49.2237 kmem_cache_free
56806 851518 3.5185 52.7422 __percpu_counter_add
48619 900137 3.0114 55.7536 copy_from_user
44803 944940 2.7751 58.5287 init_timer
28539 973479 1.7677 60.2964 d_alloc
27795 1001274 1.7216 62.0180 alloc_fd
26747 1028021 1.6567 63.6747 __fput
24312 1052333 1.5059 65.1805 sys_close
24205 1076538 1.4992 66.6798 inet_create
22409 1098947 1.3880 68.0677 alloc_inode
21359 1120306 1.3230 69.3907 release_sock
19865 1140171 1.2304 70.6211 fd_install
19472 1159643 1.2061 71.8272 lock_sock_nested
18956 1178599 1.1741 73.0013 sock_init_data
17301 1195900 1.0716 74.0729 drop_file_write_access
17113 1213013 1.0600 75.1329 inotify_d_instantiate
16384 1229397 1.0148 76.1477 dput
15173 1244570 0.9398 77.0875 local_bh_enable_ip
15017 1259587 0.9301 78.0176 local_bh_enable
13354 1272941 0.8271 78.8448 __sock_create
13139 1286080 0.8138 79.6586 inet_release
13062 1299142 0.8090 80.4676 sysenter_do_call
11935 1311077 0.7392 81.2069 iput_single
This patch serie contains 7 patches, against linux-2.6 tree, plus one patch in mm (fs: filp_cachep can be static in fs/file_table.c)
[PATCH 1/7] fs: Use a percpu_counter to track nr_dentry
Adding a percpu_counter nr_dentry avoids cache line ping pongs between cpus to maintain this metric, and dcache_lock is no more needed to protect dentry_stat.nr_dentry
We centralize nr_dentry updates at the right place :
- increments in d_alloc()
- decrements in d_free()
d_alloc() can avoid taking dcache_lock if parent is NULL
("socketallocbench -n 8" bench result : 27.5s to 25s)
[PATCH 2/7] fs: Use a percpu_counter to track nr_inodes
Avoids cache line ping pongs between cpus and prepare next patch, because updates of nr_inodes dont need inode_lock anymore.
("socketallocbench -n 8" bench result : no difference at this point)
[PATCH 3/7] fs: Introduce a per_cpu last_ino allocator
new_inode() dirties a contended cache line to get increasing inode numbers.
Solve this problem by providing to each cpu a per_cpu variable, feeded by the shared last_ino, but once every 1024 allocations.
This reduce contention on the shared last_ino, and give same spreading ino numbers than before.
(same wraparound after 232 allocations)
("socketallocbench -n 8" result : no difference)
[PATCH 4/7] fs: Introduce SINGLE dentries for pipes, socket, anon fd
Sockets, pipes and anonymous fds have interesting properties.
Like other files, they use a dentry and an inode.
But dentries for these kind of files are not hashed into dcache, since there is no way someone can lookup such a file in the vfs tree. (/proc/{pid}/fd/{number} uses a different mechanism)
Still, allocating and freeing such dentries are expensive processes, because we currently take dcache_lock inside d_alloc(), d_instantiate(), and dput(). This lock is very contended on SMP machines.
This patch defines a new DCACHE_SINGLE flag, to mark a dentry as a single one (for sockets, pipes, anonymous fd), and a new d_alloc_single(const struct qstr *name, struct inode *inode) method, called by the three subsystems.
Internally, dput() can take a fast path to dput_single() for SINGLE dentries. No more atomic_dec_and_lock()
for such dentries.
Differences betwen an SINGLE dentry and a normal one are :
1) SINGLE dentry has the DCACHE_SINGLE flag
2) SINGLE dentry's parent is itself (DCACHE_DISCONNECTED) This to avoid taking a reference on sb 'root' dentry, shared by too many dentries.
3) They are not hashed into global hash table (DCACHE_UNHASHED)
4) Their d_alias list is empty
(socket8 bench result : from 25s to 19.9s)
[PATCH 5/7] fs: new_inode_single() and iput_single()
Goal of this patch is to not touch inode_lock for socket/pipes/anonfd inodes allocation/freeing.
SINGLE dentries are attached to inodes that dont need to be linked in a list of inodes, being "inode_in_use" or "sb->s_inodes" As inode_lock was taken only to protect these lists, we avoid taking it as well.
Using iput_single() from dput_single() avoids taking inode_lock at freeing time.
This patch has a very noticeable effect, because we avoid dirtying of three contended cache lines in new_inode(), and five cache lines in iput()
("socketallocbench -n 8" result : from 19.9s to 3.01s)
[PATH 6/7] fs: struct file move from call_rcu() to SLAB_DESTROY_BY_RCU
From: Christoph Lameter <cl@xxxxxxxxxxxxxxxxxxxx>
Currently we schedule RCU frees for each file we free separately. That has
several drawbacks against the earlier file handling (in 2.6.5 f.e.), which
did not require RCU callbacks:
1. Excessive number of RCU callbacks can be generated causing long RCU queues that in turn cause long latencies. We hit SLUB page allocation more often than necessary.
2. The cache hot object is not preserved between free and realloc. A close
followed by another open is very fast with the RCUless approach because the last freed object is returned by the slab allocator that is still cache hot. RCU free means that the object is not immediately available again. The new object is cache cold and therefore open/close performance tests show a significant degradation with the RCU implementation.
One solution to this problem is to move the RCU freeing into the Slab allocator by specifying SLAB_DESTROY_BY_RCU as an option at slab creation
time. The slab allocator will do RCU frees only when it is necessary to dispose of slabs of objects (rare). So with that approach we can cut out the RCU overhead significantly.
However, the slab allocator may return the object for another use even before the RCU period has expired under SLAB_DESTROY_BY_RCU. This means there is the (unlikely) possibility that the object is going to be switched under us in sections protected by rcu_read_lock() and rcu_read_unlock(). So we need to verify that we have acquired the correct
object after establishing a stable object reference (incrementing the refcounter does that).
</quote>
http://linux.derkeiler.com/Mailing-Lists/Kernel/2008-12/msg04475.html
(1/7 fs: Use a percpu_counter to track nr_dentry)
http://patchwork.ozlabs.org/patch/13602/
(2/7 Use a percpu_counter to track nr_inodes)
http://patchwork.ozlabs.org/patch/13603/
(3/7 Introduce a per_cpu last_ino allocator)
http://patchwork.ozlabs.org/patch/13605/
(4/7 Introduce SINGLE dentries for pipes, socket, anon fd)
...
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] fs: inode per-cpu last_ino allocator
In reply to Eric DumazetQuoted 29 lines · Eric Dumazet
Quoted 4 lines · Andrew Morton
Could eliminate `p' I guess, but that would involve using
__get_cpu_var()as an lval, which looks vile and might generate worse code.Hmm, I see, please check this new patch, using the most modern stuff ;)
Quoted 2 lines
Readers of this code won't know why
last_ino_get()was marked noinline. It looks wrong, really.Oops sorry, this was a temporary hack of mine to ease disassembly analysis. Good catch !
Here is the new generated code on i686 (with the noinline) :
pretty good ;)
c02e5930<last_ino_get>:c02e5930: 55 push %ebp c02e5931: 89 e5 mov %esp,%ebp c02e5933: 64 a1 44 29 7d c0 mov %fs:0xc07d2944,%eax c02e5939: a9 ff 03 00 00 test $0x3ff,%eax c02e593e: 74 09 je c02e5949 <last_ino_get+0x19> c02e5940: 40 inc %eax c02e5941: 64 a3 44 29 7d c0 mov %eax,%fs:0xc07d2944 c02e5947: c9 leave c02e5948: c3 ret c02e5949: b8 00 04 00 00 mov $0x400,%eax
Quoted 7 lines
c02e594e: f0 0f c1 05 80 c8 92 c0 lock xadd %eax,0xc092c880c02e5956: eb e8 jmpc02e5940<last_ino_get+0x10>Thanks
[PATCH] fs: inode per-cpu
last_inoallocator
Thanks Eric, this looks good. You didn't seem to add a comment about preempt safety that Andrew wanted, but I'll add it.
Quoted 95 lines
new_inode()dirties a contended cache line to get increasing inode numbers.Solve this problem by providing to each cpu a
per_cpuvariable, feeded by the sharedlast_ino, but once every 1024 allocations. This reduces contention on the sharedlast_ino, and give same spreading ino numbers than before (i.e. same wraparound after 2^32 allocations).Signed-off-by Eric Dumazet <[email protected]>Signed-off-by Nick Piggin <[email protected]>Signed-off-by Dave Chinner <[email protected]>--- fs/inode.c | 47 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-)fs/inode.c +40
-8@@ -624,6 +624,45 @@ void inode_add_to_lists(struct super_block *sb, struct inode *inode) }EXPORT_SYMBOL_GPL (inode_add_to_lists); +#define LAST_INO_BATCH 1024 + +/* +* Each cpu owns a range of LAST_INO_BATCH numbers. +* 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations, +* to renew the exhausted range. +* +* This does not significantly increase overflow rate because every CPU can +* consume at most LAST_INO_BATCH-1 unused inode numbers. So there is +* NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the +* 2^32 range, and is a worst-case. Even a 50% wastage would only increase +* overflow rate by 2x, which does not seem too significant. +* +* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW +* error if st_ino won't fit in target struct field. Use 32bit counter +* here to attempt to avoid that. +*/ +static DEFINE_PER_CPU(unsigned int, last_ino); + +static unsigned int last_ino_get (void ) +{ +unsigned int res; + +get_cpu (); + res =__this_cpu_read (last_ino); +#ifdef CONFIG_SMP +if (unlikely ((res & (LAST_INO_BATCH -1 )) ==0 )) { +static atomic_t shared_last_ino; +int next =atomic_add_return (LAST_INO_BATCH , &shared_last_ino); + + res = next -LAST_INO_BATCH ; + } +#endif + res++; +__this_cpu_write (last_ino, res); +put_cpu (); +return res; +} +/** * new_inode - obtain an inode * @sb: superblock @@ -638,12 +677,6 @@ EXPORT_SYMBOL_GPL(inode_add_to_lists);*/ struct inode *new_inode (struct super_block *sb) { -/* -* On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW -* error if st_ino won't fit in target struct field. Use 32bit counter -* here to attempt to avoid that. -*/ -static unsigned int last_ino;struct inode *inode;spin_lock_prefetch (&inode_lock);@@ -652,7 +685,7 @@ struct inode *new_inode(struct super_block *sb)if (inode) {spin_lock (&inode_lock);__inode_add_to_lists (sb,NULL , inode); - inode->i_ino = ++last_ino; + inode->i_ino =last_ino_get (); inode->i_state =0 ;spin_unlock (&inode_lock); } --To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] fs: inode per-cpu last_ino allocator
Quoted 5 lines · Nick Piggin
Quoted 1 line · Eric Dumazet
[PATCH] fs: inode per-cpu
last_inoallocatorThanks Eric, this looks good. You didn't seem to add a comment about preempt safety that Andrew wanted, but I'll add it.
Oh I beg your pardon, you added preempt disable in the code. Much safer, thanks.
PATCH 16/17 fs: Convert nr_inodes to a per-cpu counter
In reply to Dave ChinnerFrom: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
--- fs/fs-writeback.c | 16 +++++++++++++--- fs/inode.c | 35 +++++++++++++++++++++++++++++++++-- include/linux/fs.h | 5 ++++- kernel/sysctl.c | 4 ++-- 4 files changed, 52 insertions(+), 8 deletions(-)
fs/fs-writeback.c +13 -3
@@ -728,6 +728,7 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb) {unsigned long expired;long nr_pages; +int nr_dirty_inodes;/* * When set to zero, disable periodic writeback
@@ -740,10 +741,15 @@ static long wb_check_old_data_flush(struct bdi_writeback *wb)if (time_before (jiffies, expired))return 0 ; +/* approximate dirty inodes */ + nr_dirty_inodes =get_nr_inodes () -get_nr_inodes_unused (); +if (nr_dirty_inodes <0 ) + nr_dirty_inodes =0 ; + wb->last_old_flush = jiffies; nr_pages =global_page_state (NR_FILE_DIRTY ) +global_page_state (NR_UNSTABLE_NFS ) + - inodes_stat.nr_inodes - inodes_stat.nr_unused; + nr_dirty_inodes;if (nr_pages) {struct wb_writeback_work work = {
@@ -1105,6 +1111,7 @@ void writeback_inodes_sb(struct super_block *sb) {unsigned long nr_dirty =global_page_state (NR_FILE_DIRTY );unsigned long nr_unstable =global_page_state (NR_UNSTABLE_NFS ); +int nr_dirty_inodes;DECLARE_COMPLETION_ONSTACK (done);struct wb_writeback_work work = { .sb = sb,
@@ -1114,8 +1121,11 @@ void writeback_inodes_sb(struct super_block *sb)WARN_ON (!rwsem_is_locked (&sb->s_umount)); - work.nr_pages = nr_dirty + nr_unstable + - inodes_stat.nr_inodes - inodes_stat.nr_unused; + nr_dirty_inodes =get_nr_inodes () -get_nr_inodes_unused (); +if (nr_dirty_inodes <0 ) + nr_dirty_inodes =0 ; + + work.nr_pages = nr_dirty + nr_unstable + nr_dirty_inodes;bdi_queue_work (sb->s_bdi, &work);wait_for_completion (&done);
fs/inode.c +33 -2
@@ -143,8 +143,38 @@ struct inodes_stat_t inodes_stat = { .nr_unused = 0, }; +static DEFINE_PER_CPU(unsigned int, nr_inodes); +static struct kmem_cache *inode_cachep__read_mostly ; +int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +} + +int get_nr_inodes_unused (void ) +{ +return inodes_stat.nr_unused; +} + +/* +* Handle nr_dentry sysctl +*/ +int proc_nr_inodes (ctl_table *table,int write, +void __user *buffer,size_t *lenp,loff_t *ppos) +{ +#if defined(CONFIG_SYSCTL ) && defined(CONFIG_PROC_FS ) + inodes_stat.nr_inodes =get_nr_inodes (); +return proc_dointvec (table, write, buffer, lenp, ppos); +#else +return -ENOSYS ; +#endif +} +static void wake_up_inode (struct inode *inode) {/*
@@ -232,6 +262,8 @@ int inode_init_always(struct super_block *sb, struct inode *inode) inode->i_fsnotify_mask =0 ;#endif +this_cpu_inc (nr_inodes); +return 0 ; out:return -ENOMEM ;
@@ -272,6 +304,7 @@ void __destroy_inode(struct inode *inode)if (inode->i_default_acl && inode->i_default_acl !=ACL_NOT_CACHED )posix_acl_release (inode->i_default_acl);#endif +this_cpu_dec (nr_inodes); }EXPORT_SYMBOL (__destroy_inode);
@@ -643,7 +676,6 @@ static inline void __inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b, struct inode *inode) { - inodes_stat.nr_inodes++; list_add(&inode->i_sb_list, &sb->s_inodes);spin_unlock (&sb_inode_list_lock);if (b) {
@@ -1382,7 +1414,6 @@ static void iput_final(struct inode *inode)spin_unlock (&wb_inode_list_lock); }list_del_init (&inode->i_sb_list); - inodes_stat.nr_inodes--;spin_unlock (&sb_inode_list_lock);WARN_ON (inode->i_state &I_NEW ); inode->i_state |=I_FREEING ;
include/linux/fs.h +4 -1
@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;extern int get_max_files (void );extern int sysctl_nr_open;extern struct inodes_stat_t inodes_stat; +extern int get_nr_inodes (void ); +extern int get_nr_inodes_unused (void );extern int leases_enable, lease_break_time;struct buffer_head ;
@@ -2477,7 +2479,8 @@ ssize_t simple_attr_write(struct file *file, const char __user *buf,struct ctl_table ;int proc_nr_files (struct ctl_table *table,int write,void __user *buffer,size_t *lenp,loff_t *ppos); - +int proc_nr_inodes (struct ctl_table *table,int write, +void __user *buffer,size_t *lenp,loff_t *ppos);int __init get_filesystem_list (char *buf);#define ACC_MODE (x) ("\004\002\006\006"[(x)&O_ACCMODE])
kernel/sysctl.c +2 -2
@@ -1340,14 +1340,14 @@ static struct ctl_table fs_table[] = { .data = &inodes_stat, .maxlen = 2*sizeof(int), .mode = 0444, - .proc_handler = proc_dointvec, + .proc_handler = proc_nr_inodes, }, { .procname = "inode-state", .data = &inodes_stat, .maxlen = 7*sizeof(int), .mode = 0444, - .proc_handler = proc_dointvec, + .proc_handler = proc_nr_inodes, }, { .procname = "file-nr",
Signature · 1 line
1.7.1
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
fs/inode.c struct inodes_stat_t inodes_stat = { · line 176
Quoted 13 lines
fs/inode.c +13
-0+/* +* Handle nr_dentry sysctl +*/ +int proc_nr_inodes (ctl_table *table,int write, +void __user *buffer,size_t *lenp,loff_t *ppos) +{ +#if defined(CONFIG_SYSCTL ) && defined(CONFIG_PROC_FS ) + inodes_stat.nr_inodes =get_nr_inodes (); +return proc_dointvec (table, write, buffer, lenp, ppos); +#else +return -ENOSYS ; +#endif +}
Why would we even bother to define the handler if we don't have /proc/sys/ support?
Andrew Morton
maintainerRe: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
In reply to Dave ChinnerQuoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
...
fs/inode.c struct inodes_stat_t inodes_stat = { · line 157
Quoted 8 lines
fs/inode.c +8
-0+int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +}
This reimplements percpu_counter_sum_positive(), rather poorly
If one never intends to use the approximate percpu_counter_read() then one could initialise the counter with a really large batch value, for a very small performance gain.
fs/inode.c extern struct files_stat_struct files_stat; · line 411
Quoted 13 lines
fs/inode.c +6
-0+int get_nr_inodes_unused (void ) +{ +return inodes_stat.nr_unused; +}@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;extern int get_max_files (void );extern int sysctl_nr_open;extern struct inodes_stat_t inodes_stat; +extern int get_nr_inodes (void ); +extern int get_nr_inodes_unused (void );
These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.
Quoted 3 lines
...
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
...
Quoted 10 lines
fs/inode.c struct inodes_stat_t inodes_stat = { · line 157
Quoted 8 lines
fs/inode.c +8
-0+int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +}This reimplements
percpu_counter_sum_positive(), rather poorly
I thought so - it was on my list of things to check when redoing this patch. I'll fix that up appropritately.
Quoted 12 lines
If one never intends to use the approximate
percpu_counter_read()then one could initialise the counter with a really large batch value, for a very small performance gain.Quoted 7 lines
fs/inode.c +4
-0+int get_nr_inodes_unused (void ) +{ +return inodes_stat.nr_unused; +}
Quoted 9 lines
include/linux/fs.h extern struct files_stat_struct files_stat; · line 411
Quoted 6 lines
include/linux/fs.h +2
-0@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;extern int get_max_files (void );extern int sysctl_nr_open;extern struct inodes_stat_t inodes_stat; +extern int get_nr_inodes (void ); +extern int get_nr_inodes_unused (void );These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.
Will fix.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
...
Quoted 10 lines
Quoted 10 lines
fs/inode.c struct inodes_stat_t inodes_stat = { · line 157
Quoted 8 lines
fs/inode.c +8
-0+int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +}This reimplements
percpu_counter_sum_positive(), rather poorly
Why is it poorly?
Quoted 3 lines
Quoted 3 lines
If one never intends to use the approximate
percpu_counter_read()then one could initialise the counter with a really large batch value, for a very small performance gain.
I did that to start with, and I was just looking to shave off cycles and icache size. this_cpu_inc on x86 on a local variable is really tiny and fast. percpu_counter does a function call which is large and clobbers memory and registers, several branches, several loads and stores, etc.
When it is a simple dumb statistics counter but with a critical fastpath, this_cpu_inc just seems to be so much better.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fs/inode.c +4
-0+int get_nr_inodes_unused (void ) +{ +return inodes_stat.nr_unused; +}
Quoted 9 lines
Quoted 9 lines
include/linux/fs.h extern struct files_stat_struct files_stat; · line 411
Quoted 6 lines
include/linux/fs.h +2
-0@@ -407,6 +407,8 @@ extern struct files_stat_struct files_stat;extern int get_max_files (void );extern int sysctl_nr_open;extern struct inodes_stat_t inodes_stat; +extern int get_nr_inodes (void ); +extern int get_nr_inodes_unused (void );These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.
Any ideas? inodes_stat_nr_unused()?
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 28 lines · Nick Piggin
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Andrew Morton
Quoted 13 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
...
Quoted 10 lines
Quoted 10 lines
fs/inode.c struct inodes_stat_t inodes_stat = { · line 157
Quoted 8 lines
fs/inode.c +8
-0+int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +}This reimplements
percpu_counter_sum_positive(), rather poorly
Quoted 1 line
Why is it poorly?
Nick
Some people believe percpu_counter object is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...
But if we do this loop only on rare events, like "cat /proc/sys/fs/inode-nr", then the percpu_counter() is more expensive, because percpu_add() _is_ more expensive :
- Its a function call and lot of instructions/cycles per call, while this_cpu_inc(nr_inodes) is a single instruction, using no register on x86.
- Its possibly accessing a shared spinlock and counter when the percpu counter reaches the batch limit.
To recap : nr_inodes is not a counter that needs to be estimated in real time, since we have not limit on number of inodes in the machine (limit is the memory allocator).
Unless someone can prove "cat /proc/sys/fs/inode-nr" must be performed thousand of times per second on their setup, the choice I made to scale nr_inodes is better over the 'obvious percpu_counter choice'
This choice was made to scale some counters in network stack some years ago, and this rocks.
Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Andrew Morton
maintainerRe: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 38 lines · On Sat, 16 Oct 2010 10:29:08 +0200 Eric Dumazet <[email protected]> wrote:
Le samedi 16 octobre 2010 __ 18:55 +1100, Nick Piggin a
__crit:Quoted 28 lines · Dave Chinner
Quoted 16 lines · Andrew Morton
Quoted 15 lines · On Wed, 29 Sep 2010 22:18:48 +1000 Dave Chinner <[email protected]> wrote:
Quoted 14 lines
From: Eric Dumazet <[email protected]>
The number of inodes allocated does not need to be tied to the addition or removal of an inode to/from a list. If we are not tied to a list lock, we could update the counters when inodes are initialised or destroyed, but to do that we need to convert the counters to be per-cpu (i.e. independent of a lock). This means that we have the freedom to change the list/locking implementation without needing to care about the counters.
...
Quoted 10 lines
Quoted 10 lines
fs/inode.c struct inodes_stat_t inodes_stat = { · line 157
Quoted 8 lines
fs/inode.c +8
-0+int get_nr_inodes (void ) +{ +int i; +int sum =0 ; +for_each_possible_cpu (i) + sum +=per_cpu (nr_inodes, i); +return sum <0 ?0 : sum; +}This reimplements
percpu_counter_sum_positive(), rather poorlyQuoted 1 line
Why is it poorly?
Nick
Some people believe
percpu_counterobject is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...
Quoted 7 lines
But if we do this loop only on rare events, like "cat
/proc/sys/fs/inode-nr", then thepercpu_counter()is more expensive, becausepercpu_add()_is_ more expensive :- Its a function call and lot of instructions/cycles per call, while
this_cpu_inc(nr_inodes)is a single instruction, using no register on x86.
You want an inlined percpu_counter_inc() then write one! Bonus points for writing this_cpu_add_return() and doing it without a preempt_disable(). It collapses to just a few instructions.
It's extremely poor form to say "oh X sucks so I'm going to implement my own" without first addressing why X allegedly sucks.
Quoted 2 lines
- Its possibly accessing a shared spinlock and counter when the percpu counter reaches the batch limit.
That's in the noise foor.
Quoted 10 lines
To recap :
nr_inodesis not a counter that needs to be estimated in real time, since we have not limit on number of inodes in the machine (limit is the memory allocator).Unless someone can prove "cat
/proc/sys/fs/inode-nr" must be performed thousand of times per second on their setup, the choice I made to scalenr_inodesis better over the 'obviouspercpu_counterchoice'This choice was made to scale some counters in network stack some years ago, and this rocks.
And we get open-coded reimplementations of the same damn thing all over the tree.
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 10 lines · Andrew Morton
Quoted 9 lines · On Sat, 16 Oct 2010 10:29:08 +0200 Eric Dumazet <[email protected]> wrote:
Some people believe
percpu_counterobject is the right answer to such distributed counters, because the loop is done on 'online' cpus instead of 'possible' cpus. "It must be better if number of possible cpus is 4096 and only one or two cpus are online"...But if we do this loop only on rare events, like "cat
/proc/sys/fs/inode-nr", then thepercpu_counter()is more expensive, becausepercpu_add()_is_ more expensive :
Quoted 8 lines
Quoted 3 lines
- Its a function call and lot of instructions/cycles per call, while
this_cpu_inc(nr_inodes)is a single instruction, using no register on x86.You want an inlined
percpu_counter_inc()then write one! Bonus points for writingthis_cpu_add_return()and doing it without apreempt_disable(). It collapses to just a few instructions.
A few instructions, but no guarantee of false sharing eviction.
Each time one cpu dirties the percpu_counter object, it slow down other cpus because they need to fetch the cache line again.
Btw, I believe my previous patch against include/linux/percpu_counter.h was lost. Are you sure I am the right guy to work on percpu_counter infra ? If yes I can implement your inlined idea.
Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 1/1 percpu_counter : add percpu_counter_add_fast()
Andrew
I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.
Thanks
[PATCH] percpu_counter : percpu_counter_add_fast()
The current way to change a percpu_counter is to call percpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)
When we dont need to maintain the approximate value of the percpu_counter (aka fbc->count), and dont need a "s64" wide counter but a regular "int" or "long" one, we can use this new function :percpu_counter_add_fast()
This function is pretty fast :
- One instruction on x86 SMP, no register pressure.
- Is safe in preempt enabled contexts.
- No lock acquisition, less false sharing.
Users of this percpu_counter variant should not use percpu_counter_read() or percpu_counter_read_positive() anymore, only percpu_counter_sum{_positive}() variant.
Note: we could add later irqsafe variant, still one instruction on x86 SMP...
--- include/linux/percpu_counter.h | 36 +++++++++++++++++++++++++++---- lib/percpu_counter.c | 12 +++++----- 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/percpu_counter.h +32 -4
@@ -3,7 +3,9 @@/* * A simple "approximate counter" for use in ext2 and ext3 superblocks. * -* WARNING: these things are HUGE. 4 kbytes per counter on 32-way P4. +* WARNING: these things are big. sizeof(long) bytes per possible cpu per counter. +* For a 64 cpus 64bit machine : +* 64*8 (512) bytes + sizeof(struct percpu_counter) */ #include <linux/spinlock.h>
@@ -21,7 +23,7 @@ struct percpu_counter {#ifdef CONFIG_HOTPLUG_CPU struct list_head list;/* All percpu_counters are on a list */ #endif -s32 __percpu *counters; +long __percpu *counters; };extern int percpu_counter_batch;
@@ -38,7 +40,7 @@ int __percpu_counter_init(struct percpu_counter *fbc, s64 amount,void percpu_counter_destroy (struct percpu_counter *fbc);void percpu_counter_set (struct percpu_counter *fbc,s64 amount); -void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch); +void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,long batch);s64 __percpu_counter_sum (struct percpu_counter *fbc);int percpu_counter_compare (struct percpu_counter *fbc,s64 rhs);
@@ -47,6 +49,24 @@ static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount)__percpu_counter_add (fbc, amount, percpu_counter_batch); } +/** +* percpu_counter_add_fast - fast variant of percpu_counter_add +* @fbc: pointer to percpu_counter +* @amount: value to add to counter +* +* Add amount to a percpu_counter object, without approximate (fbc->count) +* estimation / correction. +* Notes : +* - This fast version is limited to "long" counters, not "s64". +* - It is preempt safe, but not IRQ safe (on UP) +* - Use of percpu_counter_read{_positive}() is discouraged. +* - fbc->count accumulates the counters from offlined cpus. +*/ +static inline void percpu_counter_add_fast (struct percpu_counter *fbc,long amount) +{ +this_cpu_add (*fbc->counters, amount); +} +static inline s64 percpu_counter_sum_positive (struct percpu_counter *fbc) {s64 ret =__percpu_counter_sum (fbc);
@@ -118,7 +138,15 @@ percpu_counter_add(struct percpu_counter *fbc, s64 amount) }static inline void -__percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) +percpu_counter_add_fast (struct percpu_counter *fbc,long amount) +{ +preempt_disable (); + fbc->count += amount; +preempt_enable (); +} + +static inline void +__percpu_counter_add (struct percpu_counter *fbc,s64 amount,long batch) {percpu_counter_add (fbc, amount); }
lib/percpu_counter.c +6 -6
@@ -18,7 +18,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount)spin_lock (&fbc->lock);for_each_possible_cpu (cpu) { -s32 *pcount =per_cpu_ptr (fbc->counters, cpu); +long *pcount =per_cpu_ptr (fbc->counters, cpu); *pcount =0 ; } fbc->count = amount;
@@ -26,10 +26,10 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount) }EXPORT_SYMBOL (percpu_counter_set); -void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,s32 batch) +void __percpu_counter_add (struct percpu_counter *fbc,s64 amount,long batch) {s64 count; -s32 *pcount; +long *pcount;int cpu =get_cpu (); pcount =per_cpu_ptr (fbc->counters, cpu);
@@ -58,7 +58,7 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc)spin_lock (&fbc->lock); ret = fbc->count;for_each_online_cpu (cpu) { -s32 *pcount =per_cpu_ptr (fbc->counters, cpu); +long *pcount =per_cpu_ptr (fbc->counters, cpu); ret += *pcount; }spin_unlock (&fbc->lock);
@@ -72,7 +72,7 @@ int __percpu_counter_init(struct percpu_counter *fbc, s64 amount,spin_lock_init (&fbc->lock);lockdep_set_class (&fbc->lock, key); fbc->count = amount; - fbc->counters =alloc_percpu (s32); + fbc->counters =alloc_percpu (long);if (!fbc->counters)return -ENOMEM ;#ifdef CONFIG_HOTPLUG_CPU
@@ -123,7 +123,7 @@ static int __cpuinit percpu_counter_hotcpu_callback(struct notifier_block *nb, cpu = (unsigned long )hcpu;mutex_lock (&percpu_counters_lock);list_for_each_entry (fbc, &percpu_counters, list) { -s32 *pcount; +long *pcount;unsigned long flags;spin_lock_irqsave (&fbc->lock, flags);
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 2 lines · Eric Dumazet
I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.
The prior patch was accepted by Andrew.
Quoted 1 line
+ * - It is preempt safe, but not IRQ safe (on UP)
The IRQ safeness depends on the arch. this_cpu_add() in general only guarantees safety against preemption. It so happens that the x86 implementation is irq safe as well.
The IRQ safety for UP is therefore not an issue if you use this_cpu_add().
If you want to guarantee irqsafeness then use irqsafe_cpu_add() instead. It generates the same code on x86 for SMP but takes care of the UP issues.
include/linux/percpu_counter.h static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · line 68
Quoted 4 lines
include/linux/percpu_counter.h +4
-0+static inline void percpu_counter_add_fast (struct percpu_counter *fbc,long amount) +{ +this_cpu_add (*fbc->counters, amount); +}
What happens in case of counter overflow?
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Eric Dumazet
I based following patch against linux-2.6, I dont know if previous Christoph patch is in a git tree. I'll respin it eventually.
The prior patch was accepted by Andrew.
Quoted 8 lines
Quoted 1 line
+ * - It is preempt safe, but not IRQ safe (on UP)
The IRQ safeness depends on the arch.
this_cpu_add()in general only guarantees safety against preemption. It so happens that the x86 implementation is irq safe as well.The IRQ safety for UP is therefore not an issue if you use
this_cpu_add().
Nope, on UP, we dont use a per_cpu field, just a "s64 count".
struct percpu_counter {
s64 count;
};Quoted 10 lines
If you want to guarantee irqsafeness then use
irqsafe_cpu_add()instead. It generates the same code on x86 for SMP but takes care of the UP issues.include/linux/percpu_counter.h static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · line 68
Quoted 4 lines
include/linux/percpu_counter.h +4
-0+static inline void percpu_counter_add_fast (struct percpu_counter *fbc,long amount) +{ +this_cpu_add (*fbc->counters, amount); +}What happens in case of counter overflow?
Nothing special, as I stated the usuable width of such counters would be restricted to a long, not an s64.
It should be enough to count "number of inodes, of sockets, ..."
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 5 lines · Eric Dumazet
Nope, on UP, we dont use a
per_cpufield, just a "s64 count".struct percpu_counter { s64 count; };
Ok that is percpu counter specific then.
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
In reply to Eric DumazetQuoted 9 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:
The current way to change a
percpu_counteris to callpercpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)When we dont need to maintain the approximate value of the
percpu_counter(akafbc->count), and dont need a "s64" wide counter but a regular "int" or "long" one, we can use this new function :percpu_counter_add_fast()
Quoted 8 lines
This function is pretty fast :
- One instruction on x86 SMP, no register pressure.
- Is safe in preempt enabled contexts.
- No lock acquisition, less false sharing.Users of this
percpu_countervariant should not usepercpu_counter_read()orpercpu_counter_read_positive()anymore, only percpu_counter_sum{_positive}() variant.
That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using percpu_counter_add() variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of
{
val = this_cpu_add_return(*fbc->counters, amount);
if (unlikely(abs(val) > fbc->batch))
out_of_line_stuff();
} I suppose what you're proposing here is useful, although the name isn't a good one. It's a different way of using the existing data structure. I'd suggest that a better name is something like percpu_counter_add_local()?
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 7 lines · Andrew Morton
That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using
percpu_counter_add()variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of{ val = this_cpu_add_return(*fbc->counters, amount);
this_cpu_add_return() is not in the kernel but could be realized using a variant offshoot of cmpxchg_local. I had something like that initially but omitted it since there was no use case.
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 7 lines · On Thu, 21 Oct 2010 18:10:13 -0500 (CDT) Christoph Lameter <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
That isn't actually what I was suggesting. I was suggesting the use of an inlined, this_cpu_add()-using
percpu_counter_add()variant which still does the batched spilling into ->count. IOW, just speed up the current implementation along the lines of
Quoted 6 lines
Quoted 2 lines
{ val = this_cpu_add_return(*fbc->counters, amount);
this_cpu_add_return()is not in the kernel but could be realized using a variant offshoot ofcmpxchg_local. I had something like that initially but omitted it since there was no use case.
this_cpu_add_return() isn't really needed in this application.
{
this_cpu_add(*fbc->counters, amount);
if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch))
out_of_line_stuff();
} will work just fine.
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 9 lines · On Thu, 21 Oct 2010 17:45:36 -0700 Andrew Morton <[email protected]>…
this_cpu_add_return()isn't really needed in this application.{ this_cpu_add(*fbc->counters, amount); if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch)) out_of_line_stuff(); }will work just fine.
Did that. Got alarmed at a few things.
The compiler cannot CSE the above code - it has to reload the percpu base each time. Doing it by hand:
{
long *p;
p = this_cpu_ptr(fbc->counters);
*p += amount;
if (unlikely(abs(*p) > fbc->batch))
out_of_line_stuff();
} generates better code.
So this:
static __always_inline void percpu_counter_add_batch(struct percpu_counter *fbc,
s64 amount, long batch)
{
long *pcounter;
preempt_disable();
pcounter = this_cpu_ptr(fbc->counters);
*pcounter += amount;
if (unlikely(abs(*pcounter) >= batch))
percpu_counter_handle_overflow(fbc);
preempt_enable();
}when compiling this:
lib/proportions.c renamed from lib/proportions.c~b +5 -0
@@ -263,6 +263,11 @@ void __prop_inc_percpu(struct prop_descrprop_put_global (pd, pg); } +void foo (struct prop_local_percpu *pl) +{ +percpu_counter_add (&pl->events,1 ); +} +/* * identical to __prop_inc_percpu, except that it limits this pl's fraction to * @frac/PROP_FRAC_BASE by ignoring events when this limit has been exceeded.
comes down to
.globl foo
.type foo, @function
foo:
pushq %rbp # movslq percpu_counter_batch(%rip),%rcx # percpu_counter_batch, batch movq 96(%rdi), %rdx # <variable>.counters, tcp_ptr__ movq %rsp, %rbp #, #APP add %gs:this_cpu_off, %rdx # this_cpu_off, tcp_ptr__ #NO_APP movq (%rdx), %rax #* tcp_ptr__, D.11817 incq %rax # D.11817 movq %rax, (%rdx) # D.11817,* tcp_ptr__ cqto xorq %rdx, %rax # tmp67, D.11817 subq %rdx, %rax # tmp67, D.11817 cmpq %rcx, %rax # batch, D.11817 jl .L33 #, call percpu_counter_handle_overflow #
.L33:
leave
ret
But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the percpu_counter.h functions into lib/proportions.s. That's rather a worry.
lib/proportions.o got rather larger as a result of inlining things and it's not obvious that it's all a net benefit.
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 87 lines · Andrew Morton
Quoted 9 lines · On Thu, 21 Oct 2010 17:45:36 -0700 Andrew Morton <[email protected]>…
this_cpu_add_return()isn't really needed in this application.{ this_cpu_add(*fbc->counters, amount); if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch)) out_of_line_stuff(); }will work just fine.
Did that. Got alarmed at a few things.
The compiler cannot CSE the above code - it has to reload the percpu base each time. Doing it by hand:
{ long *p; p = this_cpu_ptr(fbc->counters); *p += amount; if (unlikely(abs(*p) > fbc->batch)) out_of_line_stuff(); }generates better code.
So this:
static
__always_inlinevoid percpu_counter_add_batch(structpercpu_counter*fbc,
s64 amount, long batch){ long *pcounter; preempt_disable(); pcounter = this_cpu_ptr(fbc->counters); *pcounter += amount; if (unlikely(abs(*pcounter) >= batch)) percpu_counter_handle_overflow(fbc); preempt_enable(); }when compiling this:
lib/proportions.c renamed from lib/proportions.c~b +5
-0@@ -263,6 +263,11 @@ void __prop_inc_percpu(struct prop_descrprop_put_global (pd, pg); } +void foo (struct prop_local_percpu *pl) +{ +percpu_counter_add (&pl->events,1 ); +} +/* * identical to __prop_inc_percpu, except that it limits this pl's fraction to * @frac/PROP_FRAC_BASE by ignoring events when this limit has been exceeded. comes down to
.globl foo
.type foo, @function
foo:pushq %rbp # movslq percpu_counter_batch(%rip),%rcx # percpu_counter_batch, batch movq 96(%rdi), %rdx # <variable>.counters, tcp_ptr__ movq %rsp, %rbp #, #APP add %gs:this_cpu_off, %rdx # this_cpu_off, tcp_ptr__ #NO_APP movq (%rdx), %rax #* tcp_ptr__, D.11817 incq %rax # D.11817 movq %rax, (%rdx) # D.11817,* tcp_ptr__ cqto xorq %rdx, %rax # tmp67, D.11817 subq %rdx, %rax # tmp67, D.11817 cmpq %rcx, %rax # batch, D.11817 jl .L33 #, call percpu_counter_handle_overflow #.L33:
leave
ret
Quoted 4 lines
But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the
percpu_counter.hfunctions intolib/proportions.s. That's rather a worry.
You you have the "ignore inlining" and "compile for size" turned on? They often suck.
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 6 lines · On Fri, 22 Oct 2010 12:58:45 +1100 Nick Piggin <[email protected]> wrote:
Quoted 4 lines
But what's really alarming is that the compiler (4.0.2) is cheerily ignoring the inline directives and was generating out-of-line versions of most of the
percpu_counter.hfunctions intolib/proportions.s. That's rather a worry.You you have the "ignore inlining"
# CONFIG_OPTIMIZE_INLINING is not set
Quoted 1 line
and "compile for size" turned on?
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
Quoted 1 line
They often suck.
Everything sucks. Are they any use?
With
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not setCONFIG_OPTIMIZE_INLINING=y
my kernel/built-in.o text went from 563638 bytes to 659852. That's rather a lot.
I haven't looked at this stuff in years. Has anyone dug into it?
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
In reply to Andrew MortonQuoted 9 lines · Andrew Morton
this_cpu_add_return()isn't really needed in this application.{ this_cpu_add(*fbc->counters, amount); if (unlikely(abs(this_cpu_read(*fbc->counters)) > fbc->batch)) out_of_line_stuff(); }will work just fine.
Hmm, you cannot do this on 32bit machines because "amount" is 64bit wide.
Switching counters to s64 is not an option (makes summation racy, and memory use bigger)
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
In reply to Eric DumazetQuoted 3 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:
The current way to change a
percpu_counteris to callpercpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)
This is incorrect. With my compiler it's 25 instructions except in the very rare case where a batch overflow occurs.
And more than half of that is function call entry/exit overhead.
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 10 lines · Andrew Morton
Quoted 3 lines · On Sat, 16 Oct 2010 16:19:14 +0200 Eric Dumazet <[email protected]> wrote:
The current way to change a
percpu_counteris to callpercpu_counter_add(), which is a bit expensive. (More than 40 instructions, possible false sharing, ...)This is incorrect. With my compiler it's 25 instructions except in the very rare case where a batch overflow occurs.
Hmm
Quoted 2 lines
And more than half of that is function call entry/exit overhead.
gcc version 4.5.1
count : 5 instructions to call function
c10cfbb5: a1 00 8f 53 c1 mov 0xc1538f00,%eax c10cfbba: 31 c9 xor %ecx,%ecx c10cfbbc: 89 04 24 mov %eax,(%esp) c10cfbbf: ba 01 00 00 00 mov $0x1,%edx c10cfbc4: b8 c0 5b 50 c1 mov $0xc1505bc0,%eax c10cfbc9: e8 a2 64 0b 00 call c1186070 <__percpu_counter_add>
Then 39 instructions in hot path (no lock taken)
So its more than 40 as I stated
c1186070 <__percpu_counter_add>:
c1186070: 55 push %ebp c1186071: 89 e5 mov %esp,%ebp c1186073: 83 ec 1c sub $0x1c,%esp c1186076: 89 5d f4 mov %ebx,-0xc(%ebp) c1186079: 89 75 f8 mov %esi,-0x8(%ebp) c118607c: 89 7d fc mov %edi,-0x4(%ebp) c118607f: 89 c3 mov %eax,%ebx c1186081: 8b 73 20 mov 0x20(%ebx),%esi c1186084: 64 a1 6c 10 59 c1 mov %fs:0xc159106c,%eax c118608a: 8b 3c 85 a0 7d 53 c1 mov -0x3eac8260(,%eax,4),%edi c1186091: 01 fe add %edi,%esi c1186093: 89 75 e8 mov %esi,-0x18(%ebp) c1186096: 8b 06 mov (%esi),%eax c1186098: 8b 75 08 mov 0x8(%ebp),%esi c118609b: 89 c7 mov %eax,%edi c118609d: 89 45 ec mov %eax,-0x14(%ebp) c11860a0: c1 ff 1f sar $0x1f,%edi c11860a3: 01 55 ec add %edx,-0x14(%ebp) c11860a6: 89 7d f0 mov %edi,-0x10(%ebp) c11860a9: 89 f7 mov %esi,%edi c11860ab: 11 4d f0 adc %ecx,-0x10(%ebp) c11860ae: c1 ff 1f sar $0x1f,%edi c11860b1: 39 7d f0 cmp %edi,-0x10(%ebp) c11860b4: 7f 3a jg c11860f0 <__percpu_counter_add+0x80> c11860b6: 7d 68 jge c1186120 <__percpu_counter_add+0xb0> c11860b8: 8b 4d 08 mov 0x8(%ebp),%ecx c11860bb: f7 d9 neg %ecx c11860bd: 89 ca mov %ecx,%edx c11860bf: c1 fa 1f sar $0x1f,%edx c11860c2: 39 55 f0 cmp %edx,-0x10(%ebp) c11860c5: 7e 19 jle c11860e0 <__percpu_counter_add+0x70> c11860c7: 8b 7d ec mov -0x14(%ebp),%edi c11860ca: 8b 75 e8 mov -0x18(%ebp),%esi c11860cd: 89 3e mov %edi,(%esi) c11860cf: 8b 5d f4 mov -0xc(%ebp),%ebx c11860d2: 8b 75 f8 mov -0x8(%ebp),%esi c11860d5: 8b 7d fc mov -0x4(%ebp),%edi c11860d8: c9 leave c11860d9: c3 ret c11860da: 8d b6 00 00 00 00 lea 0x0(%esi),%esi c11860e0: 7c 0e jl c11860f0 <__percpu_counter_add+0x80> c11860e2: 39 4d ec cmp %ecx,-0x14(%ebp) c11860e5: 77 e0 ja c11860c7 <__percpu_counter_add+0x57> c11860e7: 89 f6 mov %esi,%esi c11860e9: 8d bc 27 00 00 00 00 lea 0x0(%edi,%eiz,1),%edi c11860f0: 89 d8 mov %ebx,%eax c11860f2: e8 e9 41 1d 00 call c135a2e0 <_raw_spin_lock> c11860f7: 8b 45 ec mov -0x14(%ebp),%eax c11860fa: 8b 55 f0 mov -0x10(%ebp),%edx c11860fd: 01 43 10 add %eax,0x10(%ebx) c1186100: 89 d8 mov %ebx,%eax c1186102: 11 53 14 adc %edx,0x14(%ebx) c1186105: 8b 55 e8 mov -0x18(%ebp),%edx c1186108: c7 02 00 00 00 00 movl $0x0,(%edx) c118610e: e8 6d 41 1d 00 call c135a280 <_raw_spin_unlock> c1186113: 8b 5d f4 mov -0xc(%ebp),%ebx c1186116: 8b 75 f8 mov -0x8(%ebp),%esi c1186119: 8b 7d fc mov -0x4(%ebp),%edi c118611c: c9 leave c118611d: c3 ret c118611e: 66 90 xchg %ax,%ax c1186120: 8b 7d 08 mov 0x8(%ebp),%edi c1186123: 39 7d ec cmp %edi,-0x14(%ebp) c1186126: 73 c8 jae c11860f0 <__percpu_counter_add+0x80> c1186128: 8b 4d 08 mov 0x8(%ebp),%ecx c118612b: f7 d9 neg %ecx c118612d: 89 ca mov %ecx,%edx c118612f: c1 fa 1f sar $0x1f,%edx c1186132: 39 55 f0 cmp %edx,-0x10(%ebp) c1186135: 7f 90 jg c11860c7 <__percpu_counter_add+0x57> c1186137: eb a7 jmp c11860e0 <__percpu_counter_add+0x70>
Andrew Morton
maintainerRe: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 1 line · On Fri, 22 Oct 2010 00:58:14 +0200 Eric Dumazet <[email protected]> wrote:
Then 39 instructions in hot path (no lock taken)
Wow, your compiler sucks.
.globl __percpu_counter_add
.type __percpu_counter_add, @function__percpu_counter_add:
pushq %rbp # movq %rsp, %rbp #, pushq %r13 # movq %rdi, %r13 # fbc, fbc pushq %r12 # pushq %rbx # pushq %r10 # movq 96(%rdi), %rbx # <variable>.counters, pcount #APP movl %gs:cpu_number,%eax # cpu_number, pfo_ret__ #NO_APP cltq addq __per_cpu_offset(,%rax,8), %rbx # __per_cpu_offset, pcount movslq (%rbx),%rax #* pcount, tmp70 leaq (%rax,%rsi), %r12 #, count movslq %edx,%rax # batch, batch cmpq %rax, %r12 # batch, count jge .L43 #, negl %edx # batch movslq %edx,%rax # batch, tmp74 cmpq %rax, %r12 # tmp74, count jg .L45 #, .L43: movq %r13, %rdi # fbc, D.14396 call _raw_spin_lock # addq %r12, 72(%r13) # count, <variable>.count movq %r13, %rdi # fbc, D.14396 movl $0, (%rbx) #,* pcount call _raw_spin_unlock # jmp .L47 # .L45: movl %r12d, (%rbx) # count,* pcount .L47: popq %r9 # popq %rbx # popq %r12 # popq %r13 # leave ret
Re: [PATCH] percpu_counter : add percpu_counter_add_fast()
Quoted 7 lines · Andrew Morton
Quoted 1 line · On Fri, 22 Oct 2010 00:58:14 +0200 Eric Dumazet <[email protected]> wrote:
Then 39 instructions in hot path (no lock taken)
Wow, your compiler sucks.
Actually my machines are mostly 32bit. Is it a problem ?
Compiler is OK, really.
percpu_counter() is misnamed, it should be named percpu_counter64() or something like that :-(
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Andrew Morton
maintainerRe: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
In reply to Eric Dumazet(working in time-reversed oreder again :()
Quoted 3 lines · On Sat, 16 Oct 2010 11:31:15 +0200 Eric Dumazet <[email protected]> wrote:
Btw, I believe my previous patch against
include/linux/percpu_counter.hwas lost. Are you sure I am the right guy to work onpercpu_counterinfra ? If yes I can implement your inlined idea.
I don't know what "previous patch" you're referring to.
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
Quoted 3 lines · Andrew Morton
(working in time-reversed oreder again :()
I don't know what "previous patch" you're referring to.
Nothing exciting, it was to update the comment that was slightly wrong about size of percpu_counter...
(percpu_counter: change inaccurate comment)
Re: [PATCH 16/17] fs: Convert nr_inodes to a per-cpu counter
In reply to Andrew MortonQuoted 5 lines · Andrew Morton
include/linux/fs.h extern struct files_stat_struct files_stat; · line 411
Quoted 2 lines
include/linux/fs.h +2
-0+extern int get_nr_inodes (void ); +extern int get_nr_inodes_unused (void );These are pretty cruddy names. Unfotunately we don't really have a vfs or "inode" subsystem name to prefix them with.
We don't really need to export these anyway. We have two callers for each of them, and both are in the form of:
/* approximate dirty inodes */ nr_dirty_inodes = get_nr_inodes() - get_nr_inodes_unused(); if (nr_dirty_inodes < 0) nr_dirty_inodes = 0;
which means we should just have a properly documented get_nr_dirty_inodes() helper, the rest can stay private to inode.c
PATCH 17/17 fs: Clean up inode reference counting
In reply to Dave ChinnerFrom: Nick Piggin <[email protected]>
Lots of filesystem code open codes the act of getting a reference to an inode. Factor the open coded inode lock, increment, unlock into a function iget(). Then rename __iget to iget_ilock so that nothing is directly incrementing the inode reference count for trivial operations.
--- fs/affs/inode.c | 4 +--- fs/anon_inodes.c | 4 +--- fs/bfs/dir.c | 4 +--- fs/block_dev.c | 14 +++----------- fs/btrfs/inode.c | 4 +--- fs/coda/dir.c | 4 +--- fs/drop_caches.c | 2 +- fs/exofs/inode.c | 4 +--- fs/exofs/namei.c | 4 +--- fs/ext2/namei.c | 4 +--- fs/ext3/namei.c | 4 +--- fs/ext4/namei.c | 4 +--- fs/fs-writeback.c | 6 +++--- fs/gfs2/ops_inode.c | 4 +--- fs/hfsplus/dir.c | 4 +--- fs/inode.c | 18 +++++++++--------- fs/jffs2/dir.c | 8 ++------ fs/jfs/jfs_txnmgr.c | 4 +--- fs/jfs/namei.c | 4 +--- fs/libfs.c | 4 +--- fs/logfs/dir.c | 4 +--- fs/minix/namei.c | 4 +--- fs/namei.c | 7 ++----- fs/nfs/dir.c | 4 +--- fs/nfs/getroot.c | 6 ++---- fs/nfs/write.c | 2 +- fs/nilfs2/namei.c | 4 +--- fs/notify/inode_mark.c | 8 ++++---- fs/ntfs/super.c | 4 +--- fs/ocfs2/namei.c | 4 +--- fs/quota/dquot.c | 2 +- fs/reiserfs/namei.c | 4 +--- fs/sysv/namei.c | 4 +--- fs/ubifs/dir.c | 4 +--- fs/udf/namei.c | 4 +--- fs/ufs/namei.c | 4 +--- fs/xfs/linux-2.6/xfs_iops.c | 4 +--- fs/xfs/xfs_inode.h | 4 +--- include/linux/fs.h | 10 +++++++++- ipc/mqueue.c | 7 ++----- kernel/futex.c | 4 +--- mm/shmem.c | 4 +--- net/socket.c | 4 +--- 43 files changed, 70 insertions(+), 144 deletions(-)
fs/affs/inode.c +1 -3
@@ -388,9 +388,7 @@ affs_add_entry(struct inode *dir, struct inode *inode, struct dentry *dentry, s3affs_adjust_checksum (inode_bh, block -be32_to_cpu (chain));mark_buffer_dirty_inode (inode_bh, inode); inode->i_nlink =2 ; -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); }affs_fix_checksum (sb, bh);mark_buffer_dirty_inode (bh, inode);
fs/anon_inodes.c +1 -3
@@ -114,9 +114,7 @@ struct file *anon_inode_getfile(const char *name,* so we can avoid doing an igrab() and we can use an open-coded * atomic_inc(). */ -spin_lock (&anon_inode_inode->i_lock); - anon_inode_inode->i_count++; -spin_unlock (&anon_inode_inode->i_lock); +iget (anon_inode_inode); path.dentry->d_op = &anon_inodefs_dentry_operations;d_instantiate (path.dentry, anon_inode_inode);
fs/bfs/dir.c +1 -3
@@ -176,9 +176,7 @@ static int bfs_link(struct dentry *old, struct inode *dir,inc_nlink (inode); inode->i_ctime =CURRENT_TIME_SEC ;mark_inode_dirty (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (new, inode);mutex_unlock (&info->bfs_lock);return 0 ;
fs/block_dev.c +3 -11
@@ -550,11 +550,7 @@ EXPORT_SYMBOL(bdget);*/ struct block_device *bdgrab (struct block_device *bdev) { -struct inode *inode = bdev->bd_inode; - -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (bdev->bd_inode);return bdev; }
@@ -585,9 +581,7 @@ static struct block_device *bd_acquire(struct inode *inode)spin_lock (&bdev_lock); bdev = inode->i_bdev;if (bdev) { -spin_lock (&inode->i_lock); - bdev->bd_inode->i_count++; -spin_unlock (&inode->i_lock); +bdgrab (bdev);spin_unlock (&bdev_lock);return bdev; }
@@ -603,9 +597,7 @@ static struct block_device *bd_acquire(struct inode *inode)* So, we can access it via ->i_mapping always * without igrab(). */ -spin_lock (&inode->i_lock); - bdev->bd_inode->i_count++; -spin_unlock (&inode->i_lock); +iget (bdev->bd_inode); inode->i_bdev = bdev; inode->i_mapping = bdev->bd_inode->i_mapping;list_add (&inode->i_devices, &bdev->bd_inodes);
fs/btrfs/inode.c +1 -3
@@ -4769,9 +4769,7 @@ static int btrfs_link(struct dentry *old_dentry, struct inode *dir, }btrfs_set_trans_block_group (trans, dir); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); err =btrfs_add_nondir (trans, dentry, inode,1 , index);
fs/coda/dir.c +1 -3
@@ -303,9 +303,7 @@ static int coda_link(struct dentry *source_de, struct inode *dir_inode, }coda_dir_update_mtime (dir_inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (de, inode);inc_nlink (inode);
fs/drop_caches.c +1 -1
@@ -24,7 +24,7 @@ static void drop_pagecache_sb(struct super_block *sb, void *unused)spin_unlock (&inode->i_lock);continue ; } -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);invalidate_mapping_pages (inode->i_mapping,0 ,-1 );
fs/exofs/inode.c +1 -3
@@ -1156,9 +1156,7 @@ struct inode *exofs_new_inode(struct inode *dir, int mode)/* increment the refcount so that the inode will still be around when we * reach the callback */ -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); ios->done = create_done; ios->private = inode;
fs/exofs/namei.c +1 -3
@@ -153,9 +153,7 @@ static int exofs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);return exofs_add_nondir (dentry, inode); }
fs/ext2/namei.c +1 -3
@@ -206,9 +206,7 @@ static int ext2_link (struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); err =ext2_add_link (dentry, inode);if (!err) {
fs/ext3/namei.c +1 -3
@@ -2260,9 +2260,7 @@ retry: inode->i_ctime =CURRENT_TIME_SEC ;inc_nlink (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); err =ext3_add_entry (handle, dentry, inode);if (!err) {
fs/ext4/namei.c +1 -3
@@ -2312,9 +2312,7 @@ retry: inode->i_ctime =ext4_current_time (inode);ext4_inc_count (handle, inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); err =ext4_add_entry (handle, dentry, inode);if (!err) {
fs/fs-writeback.c +3 -3
@@ -294,7 +294,7 @@ static void inode_wait_for_writeback(struct inode *inode)/* * Write out an inode's dirty pages. Either the caller has ref on the inode -* (either via __iget or via syscall against an fd) or the inode has +* (either via iget or via syscall against an fd) or the inode has * I_WILL_FREE set (via generic_forget_inode) * * If `wait' is set, wait on the writeout.
@@ -505,7 +505,7 @@ again: }BUG_ON (inode->i_state &I_FREEING ); -__iget (inode); +iget_ilock (inode); pages_skipped = wbc->pages_skipped;writeback_single_inode (inode, wbc);if (wbc->pages_skipped != pages_skipped) {
@@ -1075,7 +1075,7 @@ static void wait_sb_inodes(struct super_block *sb)spin_unlock (&inode->i_lock);continue ; } -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);/*
fs/gfs2/ops_inode.c +1 -3
@@ -253,9 +253,7 @@ out_parent:gfs2_holder_uninit (ghs);gfs2_holder_uninit (ghs +1 );if (!error) { -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (dentry, inode);mark_inode_dirty (inode); }
fs/hfsplus/dir.c +1 -3
@@ -301,9 +301,7 @@ static int hfsplus_link(struct dentry *src_dentry, struct inode *dst_dir,inc_nlink (inode);hfsplus_instantiate (dst_dentry, inode, cnid); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); inode->i_ctime =CURRENT_TIME_SEC ;mark_inode_dirty (inode);HFSPLUS_SB (sb).file_count++;
fs/inode.c +9 -9
@@ -533,7 +533,7 @@ again:if (inode_has_buffers (inode) || inode->i_data.nrpages) {list_move (&inode->i_list, &inode_unused);spin_unlock (&wb_inode_list_lock); -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);if (remove_inode_buffers (inode))
@@ -591,7 +591,7 @@ static struct shrinker icache_shrinker = {static void __wait_on_freeing_inode (struct inode *inode);/* * Called with the inode lock held. -* NOTE: we are not increasing the inode-refcount, you must call __iget() +* NOTE: we are not increasing the inode-refcount, you must call iget_ilock() * by hand after calling find_inode now! This simplifies iunique and won't * add any additional branch in the common code. */
@@ -855,7 +855,7 @@ static struct inode *get_new_inode(struct super_block *sb,* us. Use the old inode instead of the one we just * allocated. */ -__iget (old); +iget_ilock (old);spin_unlock (&old->i_lock);destroy_inode (inode); inode = old;
@@ -904,7 +904,7 @@ static struct inode *get_new_inode_fast(struct super_block *sb,* us. Use the old inode instead of the one we just * allocated. */ -__iget (old); +iget_ilock (old);spin_unlock (&old->i_lock);destroy_inode (inode); inode = old;
@@ -975,7 +975,7 @@ struct inode *igrab(struct inode *inode)spin_lock (&inode->i_lock);if (!(inode->i_state & (I_FREEING |I_WILL_FREE ))) { -__iget (inode); +iget_ilock (inode); }else {/* * Handle the case where s_op->clear_inode is not been
@@ -1018,7 +1018,7 @@ static struct inode *ifind(struct super_block *sb, inode =find_inode (sb, b, test, data);if (inode) { -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);if (likely (wait))wait_on_inode (inode);
@@ -1050,7 +1050,7 @@ static struct inode *ifind_fast(struct super_block *sb, inode =find_inode_fast (sb, b, ino);if (inode) { -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);wait_on_inode (inode);return inode;
@@ -1239,7 +1239,7 @@ repeat:return 0 ; }spin_unlock_bucket (b); -__iget (old); +iget_ilock (old);spin_unlock (&old->i_lock);wait_on_inode (old);if (unlikely (!hlist_bl_unhashed (&old->i_hash))) {
@@ -1284,7 +1284,7 @@ repeat:return 0 ; }spin_unlock_bucket (b); -__iget (old); +iget_ilock (old);spin_unlock (&old->i_lock);wait_on_inode (old);if (unlikely (!hlist_bl_unhashed (&old->i_hash))) {
fs/jffs2/dir.c +2 -6
@@ -289,9 +289,7 @@ static int jffs2_link (struct dentry *old_dentry, struct inode *dir_i, struct demutex_unlock (&f->sem);d_instantiate (dentry, old_dentry->d_inode); dir_i->i_mtime = dir_i->i_ctime =ITIME (now); -spin_lock (&old_dentry->d_inode->i_lock); - old_dentry->d_inode->i_count++; -spin_unlock (&old_dentry->d_inode->i_lock); +iget (old_dentry->d_inode); }return ret; }
@@ -866,9 +864,7 @@ static int jffs2_rename (struct inode *old_dir_i, struct dentry *old_dentry,printk (KERN_NOTICE "jffs2_rename(): Link succeeded, unlink failed (err %d). You now have a hard link\n" , ret);/* Might as well let the VFS know */ d_instantiate (new_dentry, old_dentry->d_inode); -spin_lock (&old_dentry->d_inode->i_lock); - old_dentry->d_inode->i_count++; -spin_unlock (&old_dentry->d_inode->i_lock); +iget (old_dentry->d_inode); new_dir_i->i_mtime = new_dir_i->i_ctime =ITIME (now);return ret; }
fs/jfs/jfs_txnmgr.c +1 -3
@@ -1279,9 +1279,7 @@ int txCommit(tid_t tid, /* transaction identifier */* lazy commit thread finishes processing */ if (tblk->xflag &COMMIT_DELETE ) { -spin_lock (&tblk->u.ip->i_lock); - tblk->u.ip->i_count++; -spin_unlock (&tblk->u.ip->i_lock); +iget (tblk->u.ip);/* * Avoid a rare deadlock *
fs/jfs/namei.c +1 -3
@@ -839,9 +839,7 @@ static int jfs_link(struct dentry *old_dentry, ip->i_ctime =CURRENT_TIME ; dir->i_ctime = dir->i_mtime =CURRENT_TIME ;mark_inode_dirty (dir); -spin_lock (&ip->i_lock); - ip->i_count++; -spin_unlock (&ip->i_lock); +iget (ip); iplist[0 ] = ip; iplist[1 ] = dir;
fs/libfs.c +1 -3
@@ -255,9 +255,7 @@ int simple_link(struct dentry *old_dentry, struct inode *dir, struct dentry *den inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ;inc_nlink (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);dget (dentry);d_instantiate (dentry, inode);return 0 ;
fs/logfs/dir.c +1 -3
@@ -569,9 +569,7 @@ static int logfs_link(struct dentry *old_dentry, struct inode *dir,return -EMLINK ; inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ; -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); inode->i_nlink++;mark_inode_dirty_sync (inode);
fs/minix/namei.c +1 -3
@@ -101,9 +101,7 @@ static int minix_link(struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);return add_nondir (dentry, inode); }
fs/namei.c +2 -5
@@ -2290,11 +2290,8 @@ static long do_unlinkat(int dfd, const char __user *pathname)if (nd.last.name[nd.last.len]) goto slashes; inode = dentry->d_inode; -if (inode) { -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); - } +if (inode) +iget (inode); error =mnt_want_write (nd.path.mnt);if (error) goto exit2;
fs/nfs/dir.c +1 -3
@@ -1580,9 +1580,7 @@ nfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry)d_drop (dentry); error =NFS_PROTO (dir)->link (inode, dir, &dentry->d_name);if (error ==0 ) { -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_add (dentry, inode); }return error;
fs/nfs/getroot.c +2 -4
@@ -54,10 +54,8 @@ static int nfs_superblock_set_dummy_root(struct super_block *sb, struct inode *iiput (inode);return -ENOMEM ; } -/* Circumvent igrab(): we know the inode is not being freed */ -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +/* We know the inode is not being freed */ +iget (inode);/* * Ensure that this dentry is invisible to d_find_alias(). * Otherwise, it may be spliced into the tree by
fs/nfs/write.c +1 -1
@@ -390,7 +390,7 @@ static int nfs_inode_add_request(struct inode *inode, struct nfs_page *req) error =radix_tree_insert (&nfsi->nfs_page_tree, req->wb_index, req);BUG_ON (error);if (!nfsi->npages) { -__iget (inode); +iget_ilock (inode);if (nfs_have_delegation (inode,FMODE_WRITE )) nfsi->change_attr++; }
fs/nilfs2/namei.c +1 -3
@@ -219,9 +219,7 @@ static int nilfs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); err =nilfs_add_nondir (dentry, inode);if (!err)
fs/notify/inode_mark.c +4 -4
@@ -244,7 +244,7 @@ void fsnotify_unmount_inodes(struct list_head *list)spin_lock (&inode->i_lock);/* -* We cannot __iget() an inode in state I_FREEING, +* We cannot iget() an inode in state I_FREEING, * I_WILL_FREE, or I_NEW which is fine because by that point * the inode cannot have any associated watches. */
@@ -255,7 +255,7 @@ void fsnotify_unmount_inodes(struct list_head *list)/* * If i_count is zero, the inode cannot have any watches and -* doing an __iget/iput with MS_ACTIVE clear would actually +* doing an iget/iput with MS_ACTIVE clear would actually * evict all inodes with zero i_count from icache which is * unnecessarily violent and may in fact be illegal to do. */
@@ -269,7 +269,7 @@ void fsnotify_unmount_inodes(struct list_head *list)/* In case fsnotify_inode_delete() drops a reference. */ if (inode != need_iput_tmp) { -__iget (inode); +iget_ilock (inode); }else need_iput_tmp =NULL ;spin_unlock (&inode->i_lock);
@@ -279,7 +279,7 @@ void fsnotify_unmount_inodes(struct list_head *list)spin_lock (&next_i->i_lock);if (next_i->i_count && !(next_i->i_state & (I_FREEING |I_WILL_FREE ))) { -__iget (next_i); +iget_ilock (next_i); need_iput = next_i; }spin_unlock (&next_i->i_lock);
fs/ntfs/super.c +1 -3
@@ -2930,9 +2930,7 @@ static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent) }if ((sb->s_root =d_alloc_root (vol->root_ino))) {/* We increment i_count simulating an ntfs_iget(). */ -spin_lock (&vol->root_ino->i_lock); - vol->root_ino->i_count++; -spin_unlock (&vol->root_ino->i_lock); +iget (vol->root_ino);ntfs_debug ("Exiting, status successful." );/* Release the default upcase if it has no users. */ mutex_lock (&ntfs_lock);
fs/ocfs2/namei.c +1 -3
@@ -741,9 +741,7 @@ static int ocfs2_link(struct dentry *old_dentry, goto out_commit; } -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); dentry->d_op = &ocfs2_dentry_ops;d_instantiate (dentry, inode);
fs/quota/dquot.c +1 -1
@@ -910,7 +910,7 @@ static void add_dquot_ref(struct super_block *sb, int type) reserved =1 ;#endif -__iget (inode); +iget_ilock (inode);spin_unlock (&inode->i_lock);spin_unlock (&sb_inode_list_lock);
fs/reiserfs/namei.c +1 -3
@@ -1156,9 +1156,7 @@ static int reiserfs_link(struct dentry *old_dentry, struct inode *dir, inode->i_ctime =CURRENT_TIME_SEC ;reiserfs_update_sd (&th, inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (dentry, inode); retval =journal_end (&th, dir->i_sb, jbegin_count);reiserfs_write_unlock (dir->i_sb);
fs/sysv/namei.c +1 -3
@@ -126,9 +126,7 @@ static int sysv_link(struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);return add_nondir (dentry, inode); }
fs/ubifs/dir.c +1 -3
@@ -550,9 +550,7 @@ static int ubifs_link(struct dentry *old_dentry, struct inode *dir,lock_2_inodes (dir, inode);inc_nlink (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); inode->i_ctime =ubifs_current_time (inode); dir->i_size += sz_change; dir_ui->ui_size = dir->i_size;
fs/udf/namei.c +1 -3
@@ -1101,9 +1101,7 @@ static int udf_link(struct dentry *old_dentry, struct inode *dir,inc_nlink (inode); inode->i_ctime =current_fs_time (inode->i_sb);mark_inode_dirty (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (dentry, inode);unlock_kernel ();
fs/ufs/namei.c +1 -3
@@ -180,9 +180,7 @@ static int ufs_link (struct dentry * old_dentry, struct inode * dir, inode->i_ctime =CURRENT_TIME_SEC ;inode_inc_link_count (inode); -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode); error =ufs_add_nondir (dentry, inode);unlock_kernel ();
fs/xfs/linux-2.6/xfs_iops.c +1 -3
@@ -352,9 +352,7 @@ xfs_vn_link(if (unlikely (error))return -error; -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); +iget (inode);d_instantiate (dentry, inode);return 0 ; }
fs/xfs/xfs_inode.h +1 -3
@@ -481,10 +481,8 @@ void xfs_mark_inode_dirty_sync(xfs_inode_t *);#define IHOLD (ip) \ do { \ - spin_lock(&VFS_I(ip)->i_lock); \ ASSERT(VFS_I(ip)->i_count > 0) ; \ - VFS_I(ip)->i_count++; \ - spin_unlock(&VFS_I(ip)->i_lock); \ + iget(VFS_I(ip)); \ trace_xfs_ihold(ip, _THIS_IP_); \ } while (0)
include/linux/fs.h +9 -1
@@ -2394,12 +2394,20 @@ extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt);extern void save_mount_options (struct super_block *sb,char *options);extern void replace_mount_options (struct super_block *sb,char *options); -static inline void __iget (struct inode *inode) +static inline void iget_ilock (struct inode *inode) {assert_spin_locked (&inode->i_lock); +BUG_ON (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE )); inode->i_count++; } +static inline void iget (struct inode *inode) +{ +spin_lock (&inode->i_lock); +iget_ilock (inode); +spin_unlock (&inode->i_lock); +} +static inline ino_t parent_ino (struct dentry *dentry) {ino_t res;
ipc/mqueue.c +2 -5
@@ -768,11 +768,8 @@ SYSCALL_DEFINE1(mq_unlink, const char __user *, u_name) } inode = dentry->d_inode; -if (inode) { -spin_lock (&inode->i_lock); - inode->i_count++; -spin_unlock (&inode->i_lock); - } +if (inode) +iget (inode); err =mnt_want_write (ipc_ns->mq_mnt);if (err) goto out_err;
kernel/futex.c +1 -3
@@ -168,9 +168,7 @@ static void get_futex_key_refs(union futex_key *key)switch (key->both.offset & (FUT_OFF_INODE |FUT_OFF_MMSHARED )) {case FUT_OFF_INODE : -spin_lock (&key->shared.inode->i_lock); - key->shared.inode->i_count++; -spin_unlock (&key->shared.inode->i_lock); +iget (key->shared.inode);break ;case FUT_OFF_MMSHARED :atomic_inc (&key->private.mm->mm_count);
mm/shmem.c +1 -3
@@ -1903,9 +1903,7 @@ static int shmem_link(struct dentry *old_dentry, struct inode *dir, struct dentr dir->i_size +=BOGO_DIRENT_SIZE ; inode->i_ctime = dir->i_ctime = dir->i_mtime =CURRENT_TIME ;inc_nlink (inode); -spin_lock (&inode->i_lock); - inode->i_count++;/* New dentry reference */ -spin_unlock (&inode->i_lock); +iget (inode);dget (dentry);/* Extra pinning count for the created dentry */ d_instantiate (dentry, inode); out:
net/socket.c +1 -3
@@ -377,9 +377,7 @@ static int sock_alloc_file(struct socket *sock, struct file **f, int flags) &socket_file_ops);if (unlikely (!file)) {/* drop dentry, keep inode */ -spin_lock (&path.dentry->d_inode->i_lock); - path.dentry->d_inode->i_count++; -spin_unlock (&path.dentry->d_inode->i_lock); +iget (path.dentry->d_inode);path_put (&path);put_unused_fd (fd);return -ENFILE ;
Signature · 1 line
1.7.1
Re: [PATCH 17/17] fs: Clean up inode reference counting
Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have iget_locked. I think iref/iref_locked would be good enough names for the helpers.
One other things is that a lot of the current igrab users could use this helper instead - there's very few places where we look at inodes that don't already have a permanent reference when we try to acquire another one.
Re: [PATCH 17/17] fs: Clean up inode reference counting
Quoted 4 lines · Christoph Hellwig
Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have
iget_locked.
It's a bit of a mess. We also have __iget. I don't care much about the name though so I'll see what you've chosen in future posts.
I'll see how it looks to move earlier in the series, shouldn't be a big problem.
Re: [PATCH 17/17] fs: Clean up inode reference counting
Quoted 11 lines · Nick Piggin
Quoted 4 lines · Christoph Hellwig
Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have
iget_locked.It's a bit of a mess. We also have
__iget. I don't care much about the name though so I'll see what you've chosen in future posts.I'll see how it looks to move earlier in the series, shouldn't be a big problem.
Dave has sorted out the naming mess quite nicely in his repost.
Re: [PATCH 17/17] fs: Clean up inode reference counting
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Nick Piggin
Quoted 4 lines · Christoph Hellwig
Besides moving this much earlier in the series as mentioned before the most important thing is giving the helpers a different name as the iget name is already used for a different purpose, even if got rid of the original iget and only have
iget_locked.
Quoted 7 lines
Quoted 5 lines
It's a bit of a mess. We also have
__iget. I don't care much about the name though so I'll see what you've chosen in future posts.I'll see how it looks to move earlier in the series, shouldn't be a big problem.
Dave has sorted out the naming mess quite nicely in his repost.
Pity we can't use get/put but as I said, no big deal. I'll pick up those names.
Andrew Morton
maintainerRe: [PATCH 17/17] fs: Clean up inode reference counting
In reply to Dave ChinnerQuoted 4 lines · On Wed, 29 Sep 2010 22:18:49 +1000 Dave Chinner <[email protected]> wrote:
include/linux/fs.h +1
-1-static inline void __iget (struct inode *inode) +static inline void iget_ilock (struct inode *inode) {assert_spin_locked (&inode->i_lock);
include/linux/fs.h extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt); · line 2409
Quoted 10 lines
include/linux/fs.h +7
-0+BUG_ON (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE )); inode->i_count++; } +static inline void iget (struct inode *inode) +{ +spin_lock (&inode->i_lock); +iget_ilock (inode); +spin_unlock (&inode->i_lock); +}
I suspect that for typical configs, these functions will generate amazing amounts of code.
Please measure this. We may decide to uninline both.
Quoted 6 lines · Dave Chinner
I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.
There's two obvious ordering issues: first the inode counters that you mentioned. I think this is esialy fixed by simply dropping both batches messing with it - we should have the new locks protecting it in places once inode_lock is dropped. The other one is the clean up inode reference counting patch, which sounds like it should be earlier in the series so that we have the helpers in place before touching all places that opencode an inode reference count increment.
Thanks forpicking up the work on this while Nick is travelling and some more comments on the individual patches will follow.
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Dave Chinner
I've only ported the patches so far, without changing anything significant other than the comit descriptions. One thing that has stood out as I've done this is that the ordering of the patches is not ideal, and some things (like the inode counters) are modified multiple times through the patch set. I'm quite happy to reorder/rework the series to fix these problems if that is desired.
Quoted 7 lines
There's two obvious ordering issues: first the inode counters that you mentioned. I think this is esialy fixed by simply dropping both batches messing with it - we should have the new locks protecting it in places once
inode_lockis dropped. The other one is the clean up inode reference counting patch, which sounds like it should be earlier in the series so that we have the helpers in place before touching all places that opencode an inode reference count increment.
Yeah, I thought you'd want that. ;) I'll reorder the iget helper patch to be the first in the series which should reduce churn quite a bit, and then convert both the nr_inode and nr_unused counters to be per-cpu before any of the other modifications and so they can be ignored completely in later patches.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Btw, another relatively easy patch from Nick's series is:
http://git.kernel.org/?p=linux/kernel/git/npiggin/linux-npiggin.git;a=commitdiff;h=950bbff18b19663abc2d260e352efd51fd79dbf7
although I think the second hunk could be done better by simply not calling __inode_add_to_lists but just factoring out a hunk to add the inode only to the i_sb_list which is all that's left after the patches in this series. And of course both hunks should be separate patches.
We have serious problems with 34.6 in a machine with ~11TiB xfs, with a lot of simultaneous IO, particularly hundreds of rm and a sync afterwards. Maybe they're related to these issues.
The machine is a file server (almost all via http/apache) and has several thousand connections all the time. It behaves quite well for at most 4 days; from then on kswapd's start appearing on the display of top consuming ever increasing percentages of cpu. This is no problem, the machine has 16 nearly idle cores. However, after about 5-7 days there's an abrupt transition: in about 30s the load goes to several thousand, apache shows up consuming all possible cpu and downloads nearly stop. I have to reboot the machine to get service back. It manages to unmount the filesystems and reboot properly.
Stopping/restarting apache restores the situation but only for a short while; after about 2-3h the problem reappears. That's why I have to reboot.
With 35.6 the behaviour seems to have changed: now often CONFIG_DETECT_HUNG_TASK produces this kind of call trace in the log:
[<ffffffff81098578>] ? igrab+0x10/0x30 [<ffffffff811160fe>] ? xfs_sync_inode_valid+0x4c/0x76 [<ffffffff81116241>] ? xfs_sync_inode_data+0x1b/0xa8 [<ffffffff811163e0>] ? xfs_inode_ag_walk+0x96/0xe4 [<ffffffff811163dd>] ? xfs_inode_ag_walk+0x93/0xe4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff81116495>] ? xfs_inode_ag_iterator+0x67/0xc4 [<ffffffff81116226>] ? xfs_sync_inode_data+0x0/0xa8 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff81116712>] ? xfs_sync_data+0x22/0x42 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff8111678b>] ? xfs_quiesce_data+0x2b/0x94 [<ffffffff81113f03>] ? xfs_fs_sync_fs+0x2d/0xd7 [<ffffffff810a48dd>] ? sync_one_sb+0x0/0x1e [<ffffffff810a48c4>] ? __sync_filesystem+0x62/0x7b [<ffffffff8108993e>] ? iterate_supers+0x60/0x9d [<ffffffff810a493a>] ? sys_sync+0x3f/0x53 [<ffffffff81001dab>] ? system_call_fastpath+0x16/0x1b
It doesn't seem to cause service disruption (at least the flux graphs don't show drops). I didn't see it happen while I was watching so it may be that service degrades for short intervals. Uptime with 35.6 is only 3d8h so it's still not sure that the breakdown of 34.6 is gone but kswapd's cpu usages are very small, less than with 34.6 for a similar uptime. There are only 2 filesystems, and the big one has 256 AGs. They're not mounted with delaylog.
Quoted 40 lines · Carlos Carvalho
We have serious problems with 34.6 in a machine with ~11TiB xfs, with a lot of simultaneous IO, particularly hundreds of rm and a sync afterwards. Maybe they're related to these issues.
The machine is a file server (almost all via http/apache) and has several thousand connections all the time. It behaves quite well for at most 4 days; from then on kswapd's start appearing on the display of top consuming ever increasing percentages of cpu. This is no problem, the machine has 16 nearly idle cores. However, after about 5-7 days there's an abrupt transition: in about 30s the load goes to several thousand, apache shows up consuming all possible cpu and downloads nearly stop. I have to reboot the machine to get service back. It manages to unmount the filesystems and reboot properly.
Stopping/restarting apache restores the situation but only for a short while; after about 2-3h the problem reappears. That's why I have to reboot.
With 35.6 the behaviour seems to have changed: now often
CONFIG_DETECT_HUNG_TASKproduces this kind of call trace in the log:[<
ffffffff81098578>] ?igrab+0x10/0x30[<ffffffff811160fe>] ?xfs_sync_inode_valid+0x4c/0x76[<ffffffff81116241>] ?xfs_sync_inode_data+0x1b/0xa8[<ffffffff811163e0>] ?xfs_inode_ag_walk+0x96/0xe4[<ffffffff811163dd>] ?xfs_inode_ag_walk+0x93/0xe4[<ffffffff81116226>] ?xfs_sync_inode_data+0x0/0xa8[<ffffffff81116495>] ?xfs_inode_ag_iterator+0x67/0xc4[<ffffffff81116226>] ?xfs_sync_inode_data+0x0/0xa8[<ffffffff810a48dd>] ?sync_one_sb+0x0/0x1e[<ffffffff81116712>] ?xfs_sync_data+0x22/0x42[<ffffffff810a48dd>] ?sync_one_sb+0x0/0x1e[<ffffffff8111678b>] ?xfs_quiesce_data+0x2b/0x94[<ffffffff81113f03>] ?xfs_fs_sync_fs+0x2d/0xd7[<ffffffff810a48dd>] ?sync_one_sb+0x0/0x1e[<ffffffff810a48c4>] ?__sync_filesystem+0x62/0x7b[<ffffffff8108993e>] ?iterate_supers+0x60/0x9d[<ffffffff810a493a>] ?sys_sync+0x3f/0x53[<ffffffff81001dab>] ?system_call_fastpath+0x16/0x1b
Quoted 7 lines
It doesn't seem to cause service disruption (at least the flux graphs don't show drops). I didn't see it happen while I was watching so it may be that service degrades for short intervals. Uptime with 35.6 is only 3d8h so it's still not sure that the breakdown of 34.6 is gone but kswapd's cpu usages are very small, less than with 34.6 for a similar uptime. There are only 2 filesystems, and the big one has 256 AGs. They're not mounted with delaylog.
Apply this:
http://www.oss.sgi.com/archives/xfs/2010-10/msg00000.html
And in future, can you please report bugs in a new thread to the appropriate lists ([email protected]), not as a reply to a completely unrelated development thread....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]