fs: inode rcu freeing and hash walking V2
Subsystems (36)
Hi Al,
The following four patches implement the extra functionality you wanted on top of the inode lock breakup. They lift the i_lock up out of writeback_single_inode(), implements RCU freeing of inodes via SLAB_DESTROY_BY_RCU, converts inode hash lookup operations to use RCU list walks. Comments are welcome.
I'm undecided whether passing the bucket identifier along with the hash head is the best way to handle the hash-chain-being-jumped detection. It might be cleaner to just push the hash chain calculation all the way in.
Most of the changes in this version are as a result of Eric Dumazet's initial review.
Version 2:
- split the inode slabcache config changes into a separate patch.
- convert inode hash lists to use hlist_nulls variant to enable simple checking of whether RCU hash list traversals jumped chains.
- recheck i_ino and i_sb after gaining i_lock during RCU lookups to ensure we've got a hold of the correct inode.
- clear i_ino when destroying an inode to allow hash lookups to discover freed inodes without needing to lock the inode.
- moved inode->i_lock initialisation to slab constructor so that it can be safely used during RCU traversals
- introduced hlist_nulls_add_fake() and converted filesystems to use that
PATCH 1/4 fs: pull inode->i_lock up out of writeback_single_inode
From: Dave Chinner <[email protected]>
First thing we do in writeback_single_inode() is take the i_lock and the last thing we do is drop it. A caller already holds the i_lock, so pull the i_lock out of writeback_single_inode() to reduce the round trips on this lock during inode writeback.
--- fs/fs-writeback.c | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-)
fs/fs-writeback.c +8 -7
@@ -318,9 +318,9 @@ static void inode_wait_for_writeback(struct inode *inode) }/* -* Write out an inode's dirty pages. Called under inode_wb_list_lock. Either -* the caller has an active reference on the inode or the inode has I_WILL_FREE -* set. +* Write out an inode's dirty pages. Called under inode_wb_list_lock and +* inode->i_lock. Either the caller has an active reference on the inode or +* the inode has I_WILL_FREE set. * * If `wait' is set, wait on the writeout. *
@@ -335,7 +335,6 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)unsigned dirty;int ret; -spin_lock (&inode->i_lock);if (!atomic_read (&inode->i_count))WARN_ON (!(inode->i_state & (I_WILL_FREE |I_FREEING )));else
@@ -351,7 +350,6 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc)* completed a full scan of b_io. */ if (wbc->sync_mode !=WB_SYNC_ALL ) { -spin_unlock (&inode->i_lock);requeue_io (inode);return 0 ; }
@@ -442,7 +440,6 @@ writeback_single_inode(struct inode *inode, struct writeback_control *wbc) } }inode_sync_complete (inode); -spin_unlock (&inode->i_lock);return ret; }
@@ -530,7 +527,6 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb, }__iget (inode); -spin_unlock (&inode->i_lock); pages_skipped = wbc->pages_skipped;writeback_single_inode (inode, wbc);
@@ -541,6 +537,7 @@ static int writeback_sb_inodes(struct super_block *sb, struct bdi_writeback *wb,*/ redirty_tail (inode); } +spin_unlock (&inode->i_lock);spin_unlock (&inode_wb_list_lock);iput (inode);cond_resched ();
@@ -1212,7 +1209,9 @@ int write_inode_now(struct inode *inode, int sync)might_sleep ();spin_lock (&inode_wb_list_lock); +spin_lock (&inode->i_lock); ret =writeback_single_inode (inode, &wbc); +spin_unlock (&inode->i_lock);spin_unlock (&inode_wb_list_lock);if (sync)inode_sync_wait (inode);
@@ -1236,7 +1235,9 @@ int sync_inode(struct inode *inode, struct writeback_control *wbc)int ret;spin_lock (&inode_wb_list_lock); +spin_lock (&inode->i_lock); ret =writeback_single_inode (inode, wbc); +spin_unlock (&inode->i_lock);spin_unlock (&inode_wb_list_lock);return ret; }
Signature · 1 line
1.7.2.3
PATCH 2/4 fs: Convert inode slab creation to use consistent parameters
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The inode slab creation functions in different filesystem don't all set up the slab caches correctly. Add a new flag SLAB_INODES that sets all the necessary flags for an inode slab cache and convert all the callers to use it. THis makes it easy to change the behaviour of all the inode slab caches in one go.
--- Documentation/filesystems/porting | 5 +++++ fs/adfs/super.c | 3 +-- fs/affs/super.c | 3 +-- fs/afs/super.c | 7 +++---- fs/befs/linuxvfs.c | 3 +-- fs/bfs/inode.c | 3 +-- fs/btrfs/inode.c | 2 +- fs/ceph/super.c | 3 +-- fs/cifs/cifsfs.c | 3 +-- fs/coda/inode.c | 3 +-- fs/efs/super.c | 3 +-- fs/exofs/super.c | 3 +-- fs/ext2/super.c | 3 +-- fs/ext3/super.c | 3 +-- fs/ext4/super.c | 3 +-- fs/fat/inode.c | 3 +-- fs/freevxfs/vxfs_super.c | 2 +- fs/fuse/inode.c | 6 +++--- fs/gfs2/main.c | 3 +-- fs/hfs/super.c | 3 ++- fs/hfsplus/super.c | 3 ++- fs/hpfs/super.c | 3 +-- fs/hugetlbfs/inode.c | 2 +- fs/inode.c | 4 +--- fs/isofs/inode.c | 4 +--- fs/jffs2/super.c | 3 +-- fs/jfs/super.c | 3 +-- fs/logfs/inode.c | 2 +- fs/minix/inode.c | 3 +-- fs/ncpfs/inode.c | 3 +-- fs/nfs/inode.c | 3 +-- fs/nilfs2/super.c | 4 ++-- fs/ntfs/super.c | 3 ++- fs/ocfs2/dlmfs/dlmfs.c | 3 +-- fs/ocfs2/super.c | 4 +--- fs/openpromfs/inode.c | 4 +--- fs/proc/inode.c | 3 +-- fs/qnx4/inode.c | 3 +-- fs/reiserfs/super.c | 7 ++----- fs/romfs/super.c | 3 +-- fs/squashfs/super.c | 3 ++- fs/sysv/inode.c | 3 +-- fs/ubifs/super.c | 2 +- fs/udf/super.c | 3 +-- fs/ufs/super.c | 3 +-- fs/xfs/linux-2.6/kmem.h | 1 + fs/xfs/linux-2.6/xfs_super.c | 4 ++-- include/linux/slab.h | 6 ++++++ ipc/mqueue.c | 3 ++- mm/shmem.c | 2 +- net/socket.c | 9 +++------ net/sunrpc/rpc_pipe.c | 5 ++--- 52 files changed, 76 insertions(+), 102 deletions(-)
Documentation/filesystems/porting +5 -0
@@ -319,3 +319,8 @@ if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput( may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly free the on-disk inode, you may end up doing that while ->write_inode() is writing to it. + +[mandatory] + Inodes must be allocated via a slab cache created with the +SLAB_INODE_CACHE flag set. This sets all the necessary slab cache flags for +correct operation and control of the cache across the system.
fs/adfs/super.c +1 -2
@@ -256,8 +256,7 @@ static int init_inodecache(void) { adfs_inode_cachep =kmem_cache_create ("adfs_inode_cache" ,sizeof (struct adfs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (adfs_inode_cachep ==NULL )return -ENOMEM ;
fs/affs/super.c +1 -2
@@ -113,8 +113,7 @@ static int init_inodecache(void) { affs_inode_cachep =kmem_cache_create ("affs_inode_cache" ,sizeof (struct affs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (affs_inode_cachep ==NULL )return -ENOMEM ;
fs/afs/super.c +3 -4
@@ -87,10 +87,9 @@ int __init afs_fs_init(void) ret = -ENOMEM ; afs_inode_cachep =kmem_cache_create ("afs_inode_cache" , -sizeof (struct afs_vnode ), -0 , -SLAB_HWCACHE_ALIGN , - afs_i_init_once); +sizeof (struct afs_vnode ),0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , + afs_i_init_once);if (!afs_inode_cachep) {printk (KERN_NOTICE "kAFS: Failed to allocate inode cache\n" );return ret;
fs/befs/linuxvfs.c +1 -2
@@ -431,8 +431,7 @@ befs_init_inodecache(void) { befs_inode_cachep =kmem_cache_create ("befs_inode_cache" ,sizeof (struct befs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (befs_inode_cachep ==NULL ) { printk(KERN_ERR "befs_init_inodecache: "
fs/bfs/inode.c +1 -2
@@ -264,8 +264,7 @@ static int init_inodecache(void) { bfs_inode_cachep =kmem_cache_create ("bfs_inode_cache" ,sizeof (struct bfs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (bfs_inode_cachep ==NULL )return -ENOMEM ;
fs/btrfs/inode.c +1 -1
@@ -6369,7 +6369,7 @@ int btrfs_init_cachep(void) { btrfs_inode_cachep =kmem_cache_create ("btrfs_inode_cache" ,sizeof (struct btrfs_inode ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , init_once); +SLAB_INODE_CACHE , init_once);if (!btrfs_inode_cachep) goto fail;
fs/ceph/super.c +1 -2
@@ -521,8 +521,7 @@ static int __init init_caches(void) ceph_inode_cachep =kmem_cache_create ("ceph_inode_info" ,sizeof (struct ceph_inode_info ), __alignof__(struct ceph_inode_info ), - (SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ), - ceph_inode_init_once); +SLAB_INODE_CACHE , ceph_inode_init_once);if (ceph_inode_cachep ==NULL )return -ENOMEM ;
fs/cifs/cifsfs.c +1 -2
@@ -784,8 +784,7 @@ cifs_init_inodecache(void) { cifs_inode_cachep =kmem_cache_create ("cifs_inode_cache" ,sizeof (struct cifsInodeInfo ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , cifs_init_once);if (cifs_inode_cachep ==NULL )return -ENOMEM ;
fs/coda/inode.c +1 -2
@@ -72,8 +72,7 @@ int coda_init_inodecache(void) { coda_inode_cachep =kmem_cache_create ("coda_inode_cache" ,sizeof (struct coda_inode_info ), -0 ,SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , - init_once); +0 ,SLAB_INODE_CACHE , init_once);if (coda_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/efs/super.c +1 -2
@@ -81,8 +81,7 @@ static int init_inodecache(void) { efs_inode_cachep =kmem_cache_create ("efs_inode_cache" ,sizeof (struct efs_inode_info ), -0 ,SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , - init_once); +0 ,SLAB_INODE_CACHE , init_once);if (efs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/exofs/super.c +1 -2
@@ -175,8 +175,7 @@ static int init_inodecache(void) { exofs_inode_cachep =kmem_cache_create ("exofs_inode_cache" ,sizeof (struct exofs_i_info ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , - exofs_init_once); +SLAB_INODE_CACHE , exofs_init_once);if (exofs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/ext2/super.c +1 -2
@@ -182,8 +182,7 @@ static int init_inodecache(void) { ext2_inode_cachep =kmem_cache_create ("ext2_inode_cache" ,sizeof (struct ext2_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (ext2_inode_cachep ==NULL )return -ENOMEM ;
fs/ext3/super.c +1 -2
@@ -509,8 +509,7 @@ static int init_inodecache(void) { ext3_inode_cachep =kmem_cache_create ("ext3_inode_cache" ,sizeof (struct ext3_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (ext3_inode_cachep ==NULL )return -ENOMEM ;
fs/ext4/super.c +1 -2
@@ -862,8 +862,7 @@ static int init_inodecache(void) { ext4_inode_cachep =kmem_cache_create ("ext4_inode_cache" ,sizeof (struct ext4_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (ext4_inode_cachep ==NULL )return -ENOMEM ;
fs/fat/inode.c +1 -2
@@ -535,8 +535,7 @@ static int __init fat_init_inodecache(void) { fat_inode_cachep =kmem_cache_create ("fat_inode_cache" ,sizeof (struct msdos_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (fat_inode_cachep ==NULL )return -ENOMEM ;
fs/freevxfs/vxfs_super.c +1 -1
@@ -267,7 +267,7 @@ vxfs_init(void) vxfs_inode_cachep =kmem_cache_create ("vxfs_inode" ,sizeof (struct vxfs_inode_info ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ,NULL ); +SLAB_INODE_CACHE ,NULL );if (!vxfs_inode_cachep)return -ENOMEM ; rv =register_filesystem (&vxfs_fs_type);
fs/fuse/inode.c +3 -3
@@ -1138,9 +1138,9 @@ static int __init fuse_fs_init(void) goto out_unreg; fuse_inode_cachep =kmem_cache_create ("fuse_inode" , -sizeof (struct fuse_inode ), -0 ,SLAB_HWCACHE_ALIGN , - fuse_inode_init_once); +sizeof (struct fuse_inode ),0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , + fuse_inode_init_once); err = -ENOMEM ;if (!fuse_inode_cachep) goto out_unreg2;
fs/gfs2/main.c +1 -2
@@ -108,8 +108,7 @@ static int __init init_gfs2_fs(void) gfs2_inode_cachep =kmem_cache_create ("gfs2_inode" ,sizeof (struct gfs2_inode ), -0 ,SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD , +0 ,SLAB_INODE_CACHE , gfs2_init_inode_once);if (!gfs2_inode_cachep) goto fail;
fs/hfs/super.c +2 -1
@@ -467,7 +467,8 @@ static int __init init_hfs_fs(void)int err; hfs_inode_cachep =kmem_cache_create ("hfs_inode_cache" , -sizeof (struct hfs_inode_info ),0 ,SLAB_HWCACHE_ALIGN , +sizeof (struct hfs_inode_info ),0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , hfs_init_once);if (!hfs_inode_cachep)return -ENOMEM ;
fs/hfsplus/super.c +2 -1
@@ -521,7 +521,8 @@ static int __init init_hfsplus_fs(void)int err; hfsplus_inode_cachep =kmem_cache_create ("hfsplus_icache" , -HFSPLUS_INODE_SIZE ,0 ,SLAB_HWCACHE_ALIGN , +HFSPLUS_INODE_SIZE ,0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , hfsplus_init_once);if (!hfsplus_inode_cachep)return -ENOMEM ;
fs/hpfs/super.c +1 -2
@@ -195,8 +195,7 @@ static int init_inodecache(void) { hpfs_inode_cachep =kmem_cache_create ("hpfs_inode_cache" ,sizeof (struct hpfs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (hpfs_inode_cachep ==NULL )return -ENOMEM ;
fs/hugetlbfs/inode.c +1 -1
@@ -996,7 +996,7 @@ static int __init init_hugetlbfs_fs(void) hugetlbfs_inode_cachep =kmem_cache_create ("hugetlbfs_inode_cache" ,sizeof (struct hugetlbfs_inode_info ), -0 ,0 , init_once); +0 ,SLAB_INODE_CACHE , init_once);if (hugetlbfs_inode_cachep ==NULL ) goto out2;
fs/inode.c +1 -3
@@ -1697,9 +1697,7 @@ void __init inode_init(void)/* inode slab cache */ inode_cachep =kmem_cache_create ("inode_cache" ,sizeof (struct inode ), -0 , - (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE |SLAB_PANIC , init_once);register_shrinker (&icache_shrinker);percpu_counter_init (&nr_inodes,0 );
fs/isofs/inode.c +1 -3
@@ -81,9 +81,7 @@ static int init_inodecache(void) { isofs_inode_cachep =kmem_cache_create ("isofs_inode_cache" ,sizeof (struct iso_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), - init_once); +0 ,SLAB_INODE_CACHE , init_once);if (isofs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/jffs2/super.c +1 -2
@@ -259,8 +259,7 @@ static int __init init_jffs2_fs(void) jffs2_inode_cachep =kmem_cache_create ("jffs2_i" ,sizeof (struct jffs2_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , jffs2_i_init_once);if (!jffs2_inode_cachep) {printk (KERN_ERR "JFFS2 error: Failed to initialise inode cache\n" );
fs/jfs/super.c +1 -2
@@ -795,8 +795,7 @@ static int __init init_jfs_fs(void) jfs_inode_cachep =kmem_cache_create ("jfs_ip" ,sizeof (struct jfs_inode_info ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , - init_once); +SLAB_INODE_CACHE , init_once);if (jfs_inode_cachep ==NULL )return -ENOMEM ;
fs/logfs/inode.c +1 -1
@@ -385,7 +385,7 @@ const struct super_operations logfs_super_operations = {int logfs_init_inode_cache (void ) { logfs_inode_cache =kmem_cache_create ("logfs_inode_cache" , -sizeof (struct logfs_inode ),0 ,SLAB_RECLAIM_ACCOUNT , +sizeof (struct logfs_inode ),0 ,SLAB_INODE_CACHE , logfs_init_once);if (!logfs_inode_cache)return -ENOMEM ;
fs/minix/inode.c +1 -2
@@ -84,8 +84,7 @@ static int init_inodecache(void) { minix_inode_cachep =kmem_cache_create ("minix_inode_cache" ,sizeof (struct minix_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (minix_inode_cachep ==NULL )return -ENOMEM ;
fs/ncpfs/inode.c +1 -2
@@ -76,8 +76,7 @@ static int init_inodecache(void) { ncp_inode_cachep =kmem_cache_create ("ncp_inode_cache" ,sizeof (struct ncp_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (ncp_inode_cachep ==NULL )return -ENOMEM ;
fs/nfs/inode.c +1 -2
@@ -1474,8 +1474,7 @@ static int __init nfs_init_inodecache(void) { nfs_inode_cachep =kmem_cache_create ("nfs_inode_cache" ,sizeof (struct nfs_inode ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (nfs_inode_cachep ==NULL )return -ENOMEM ;
fs/nilfs2/super.c +2 -2
@@ -1287,8 +1287,8 @@ static void nilfs_destroy_cachep(void)static int __init nilfs_init_cachep (void ) { nilfs_inode_cachep =kmem_cache_create ("nilfs2_inode_cache" , -sizeof (struct nilfs_inode_info ),0 , -SLAB_RECLAIM_ACCOUNT , nilfs_inode_init_once); +sizeof (struct nilfs_inode_info ),0 ,SLAB_INODE_CACHE , + nilfs_inode_init_once);if (!nilfs_inode_cachep) goto fail;
fs/ntfs/super.c +2 -1
@@ -3136,9 +3136,10 @@ static int __init init_ntfs_fs(void) goto inode_err_out; } +/* ntfs_big_inode_cache is the inode cache used for VFS level inodes */ ntfs_big_inode_cache =kmem_cache_create (ntfs_big_inode_cache_name,sizeof (big_ntfs_inode),0 , -SLAB_HWCACHE_ALIGN |SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , ntfs_big_inode_init_once);if (!ntfs_big_inode_cache) { printk(KERN_CRIT "NTFS: Failed to create %s!\n",
fs/ocfs2/dlmfs/dlmfs.c +1 -2
@@ -669,8 +669,7 @@ static int __init init_dlmfs_fs(void) dlmfs_inode_cache =kmem_cache_create ("dlmfs_inode_cache" ,sizeof (struct dlmfs_inode_private ), -0 , (SLAB_HWCACHE_ALIGN |SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 , (SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE ), dlmfs_init_once);if (!dlmfs_inode_cache) { status = -ENOMEM ;
fs/ocfs2/super.c +1 -3
@@ -1791,9 +1791,7 @@ static int ocfs2_initialize_mem_caches(void) { ocfs2_inode_cachep =kmem_cache_create ("ocfs2_inode_cache" ,sizeof (struct ocfs2_inode_info ), -0 , - (SLAB_HWCACHE_ALIGN |SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 , (SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE ), ocfs2_inode_init_once); ocfs2_dquot_cachep = kmem_cache_create("ocfs2_dquot_cache", sizeof(struct ocfs2_dquot),
fs/openpromfs/inode.c +1 -3
@@ -441,9 +441,7 @@ static int __init init_openprom_fs(void) op_inode_cachep =kmem_cache_create ("op_inode_cache" ,sizeof (struct op_inode_info ), -0 , - (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , op_inode_init_once);if (!op_inode_cachep)return -ENOMEM ;
fs/proc/inode.c +1 -2
@@ -82,8 +82,7 @@ void __init proc_init_inodecache(void) { proc_inode_cachep =kmem_cache_create ("proc_inode_cache" ,sizeof (struct proc_inode ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD |SLAB_PANIC ), +0 ,SLAB_INODE_CACHE |SLAB_PANIC , init_once); }
fs/qnx4/inode.c +1 -2
@@ -441,8 +441,7 @@ static int init_inodecache(void) { qnx4_inode_cachep =kmem_cache_create ("qnx4_inode_cache" ,sizeof (struct qnx4_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (qnx4_inode_cachep ==NULL )return -ENOMEM ;
fs/reiserfs/super.c +2 -5
@@ -546,11 +546,8 @@ static void init_once(void *foo)static int init_inodecache (void ) { reiserfs_inode_cachep =kmem_cache_create ("reiser_inode_cache" , -sizeof (struct -reiserfs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), - init_once); +sizeof (struct reiserfs_inode_info ), +0 ,SLAB_INODE_CACHE , init_once);if (reiserfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/romfs/super.c +1 -2
@@ -618,8 +618,7 @@ static int __init init_romfs_fs(void) romfs_inode_cachep =kmem_cache_create ("romfs_i" ,sizeof (struct romfs_inode_info ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , - romfs_i_init_once); +SLAB_INODE_CACHE , romfs_i_init_once);if (!romfs_inode_cachep) { printk(KERN_ERR
fs/squashfs/super.c +2 -1
@@ -392,7 +392,8 @@ static int __init init_inodecache(void) { squashfs_inode_cachep =kmem_cache_create ("squashfs_inode_cache" ,sizeof (struct squashfs_inode_info ),0 , -SLAB_HWCACHE_ALIGN |SLAB_RECLAIM_ACCOUNT , init_once); +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , + init_once);return squashfs_inode_cachep ?0 : -ENOMEM ; }
fs/sysv/inode.c +1 -2
@@ -360,8 +360,7 @@ const struct super_operations sysv_sops = {int __init sysv_init_icache (void ) { sysv_inode_cachep =kmem_cache_create ("sysv_inode_cache" , -sizeof (struct sysv_inode_info ),0 , -SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD , +sizeof (struct sysv_inode_info ),0 ,SLAB_INODE_CACHE , init_once);if (!sysv_inode_cachep)return -ENOMEM ;
fs/ubifs/super.c +1 -1
@@ -2191,7 +2191,7 @@ static int __init ubifs_init(void) err = -ENOMEM ; ubifs_inode_slab =kmem_cache_create ("ubifs_inode_slab" ,sizeof (struct ubifs_inode ),0 , -SLAB_MEM_SPREAD |SLAB_RECLAIM_ACCOUNT , +SLAB_INODE_CACHE , &inode_slab_ctor);if (!ubifs_inode_slab) goto out_reg;
fs/udf/super.c +1 -2
@@ -156,8 +156,7 @@ static int init_inodecache(void) { udf_inode_cachep =kmem_cache_create ("udf_inode_cache" ,sizeof (struct udf_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (!udf_inode_cachep)return -ENOMEM ;
fs/ufs/super.c +1 -2
@@ -1428,8 +1428,7 @@ static int init_inodecache(void) { ufs_inode_cachep =kmem_cache_create ("ufs_inode_cache" ,sizeof (struct ufs_inode_info ), -0 , (SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +0 ,SLAB_INODE_CACHE , init_once);if (ufs_inode_cachep ==NULL )return -ENOMEM ;
fs/xfs/linux-2.6/kmem.h +1 -0
@@ -82,6 +82,7 @@ extern void *kmem_zalloc_greedy(size_t *, size_t, size_t);#define KM_ZONE_HWALIGN SLAB_HWCACHE_ALIGN#define KM_ZONE_RECLAIM SLAB_RECLAIM_ACCOUNT#define KM_ZONE_SPREAD SLAB_MEM_SPREAD +#define KM_ZONE_INODES SLAB_INODE_CACHE#define kmem_zone kmem_cache#define kmem_zone_t struct kmem_cache
fs/xfs/linux-2.6/xfs_super.c +2 -2
@@ -1718,8 +1718,8 @@ xfs_init_zones(void) xfs_inode_zone =kmem_zone_init_flags (sizeof (xfs_inode_t),"xfs_inode" , -KM_ZONE_HWALIGN |KM_ZONE_RECLAIM |KM_ZONE_SPREAD , - xfs_fs_inode_init_once); +KM_ZONE_HWALIGN |KM_ZONE_INODES , + xfs_fs_inode_init_once);if (!xfs_inode_zone) goto out_destroy_efi_zone;
include/linux/slab.h +6 -0
@@ -93,6 +93,12 @@ (unsigned long)ZERO_SIZE_PTR)/* +* Set the default flags necessary for inode caches manipulated by the VFS. +*/ +#define SLAB_INODE_CACHE (SLAB_MEM_SPREAD | \ + SLAB_RECLAIM_ACCOUNT) + +/* * struct kmem_cache related prototypes */ void __init kmem_cache_init (void );
ipc/mqueue.c +2 -1
@@ -1268,7 +1268,8 @@ static int __init init_mqueue_fs(void) mqueue_inode_cachep =kmem_cache_create ("mqueue_inode_cache" ,sizeof (struct mqueue_inode_info ),0 , -SLAB_HWCACHE_ALIGN , init_once); +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , + init_once);if (mqueue_inode_cachep ==NULL )return -ENOMEM ;
mm/shmem.c +1 -1
@@ -2435,7 +2435,7 @@ static int init_inodecache(void) { shmem_inode_cachep =kmem_cache_create ("shmem_inode_cache" ,sizeof (struct shmem_inode_info ), -0 ,SLAB_PANIC , init_once); +0 ,SLAB_INODE_CACHE |SLAB_PANIC , init_once);return 0 ; }
net/socket.c +3 -6
@@ -288,12 +288,9 @@ static void init_once(void *foo)static int init_inodecache (void ) { sock_inode_cachep =kmem_cache_create ("sock_inode_cache" , -sizeof (struct socket_alloc ), -0 , - (SLAB_HWCACHE_ALIGN | -SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), - init_once); +sizeof (struct socket_alloc ),0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , + init_once);if (sock_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
net/sunrpc/rpc_pipe.c +2 -3
@@ -1056,9 +1056,8 @@ int register_rpc_pipefs(void)int err; rpc_inode_cachep =kmem_cache_create ("rpc_inode_cache" , -sizeof (struct rpc_inode ), -0 , (SLAB_HWCACHE_ALIGN |SLAB_RECLAIM_ACCOUNT | -SLAB_MEM_SPREAD ), +sizeof (struct rpc_inode ),0 , +SLAB_HWCACHE_ALIGN |SLAB_INODE_CACHE , init_once);if (!rpc_inode_cachep)return -ENOMEM ;
Signature · 1 line
1.7.2.3
Re: [PATCH 2/4] fs: Convert inode slab creation to use consistent parameters
Quoted 6 lines · Dave Chinner
From: Dave Chinner <[email protected]>
The inode slab creation functions in different filesystem don't all set up the slab caches correctly. Add a new flag
SLAB_INODESthat sets all the necessary flags for an inode slab cache and convert all the callers to use it. THis makes it easy to change the behaviour of all the inode slab caches in one go.
Any chance you could rediff this one ontop of current mainline and resend it?
PATCH 3/4 fs: Use RCU freeing of inodes via SLAB_DESTROY_BY_RCU
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Change the inode caches to use RCU freed inodes via the SLAB_DESTROY_BY_RCU method. This is the first step in converting the inode hash lookups to use RCU methods.
--- include/linux/slab.h | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
include/linux/slab.h +1 -0
@@ -96,6 +96,7 @@* Set the default flags necessary for inode caches manipulated by the VFS. */ #define SLAB_INODE_CACHE (SLAB_MEM_SPREAD | \ + SLAB_DESTROY_BY_RCU | \ SLAB_RECLAIM_ACCOUNT)/*
Signature · 1 line
1.7.2.3
PATCH 4/4 fs: rcu protect inode hash lookups
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that inodes are using RCU freeing, we can walk the hash lists using RCU protection during lookups. Convert all the hash list operations to use RCU-based operators and drop the inode_hash_lock around pure lookup operations.
Because we are using SLAB_DESTROY_BY_RCU, we need to be careful about the lockless lookups and the way we initialise the struct inode during allocation. The inode->i_lock is the critical item we need to avoid touching when allocating an inode, so move the initialisation of this lock to the slab constructor so that we can reliably use the lock even after the inode has been freed or is in the process of being allocated or initialised.
We also need to change the order of checks during hash lookups to validate we have an active inode before we skip it. That means we need to revalidate the inode number and superblock after we have checked it for being freed or unhashed under the inode->i_lock.
Finally, during traversals an inode can be freed and reallocated and linked into a new hash chain, so an RCU safe traversal can jump to a different chain without indication. On a cache miss we need to validate that we ended the search on the same hash chain we started on. Do this by converting the hash lists to the hlist_null list type and checking the list null on walk termination.
--- fs/hfs/inode.c | 2 +- fs/hfsplus/inode.c | 2 +- fs/inode.c | 234 +++++++++++++++++++++++++++++++------------- fs/jfs/jfs_imap.c | 2 +- include/linux/fs.h | 5 +- include/linux/list_nulls.h | 7 ++ 6 files changed, 178 insertions(+), 74 deletions(-)
fs/hfs/inode.c +1 -1
@@ -524,7 +524,7 @@ static struct dentry *hfs_file_lookup(struct inode *dir, struct dentry *dentry,HFS_I (inode)->rsrc_inode = dir;HFS_I (dir)->rsrc_inode = inode;igrab (dir); -hlist_add_fake (&inode->i_hash); +hlist_nulls_add_fake (&inode->i_hash);mark_inode_dirty (inode); out:d_add (dentry, inode);
fs/hfsplus/inode.c +1 -1
@@ -211,7 +211,7 @@ static struct dentry *hfsplus_file_lookup(struct inode *dir, struct dentry *dent* appear hashed, but do not put on any lists. hlist_del() * will work fine and require no locking. */ -hlist_add_fake (&inode->i_hash); +hlist_nulls_add_fake (&inode->i_hash);mark_inode_dirty (inode); out:
fs/inode.c +165 -69
534 lines not on this page · raw message
fs/jfs/jfs_imap.c +1 -1
@@ -497,7 +497,7 @@ struct inode *diReadSpecial(struct super_block *sb, ino_t inum, int secondary)* appear hashed, but do not put on any lists. hlist_del() * will work fine and require no locking. */ -hlist_add_fake (&ip->i_hash); +hlist_nulls_add_fake (&ip->i_hash);return (ip); }
include/linux/fs.h +3 -2
@@ -384,6 +384,7 @@ struct inodes_stat_t {#include <linux/cache.h> #include <linux/kobject.h> #include <linux/list.h> +#include <linux/rculist_nulls.h> #include <linux/radix-tree.h> #include <linux/prio_tree.h> #include <linux/init.h>
@@ -732,7 +733,7 @@ struct posix_acl;#define ACL_NOT_CACHED ((void *)(-1))struct inode { -struct hlist_node i_hash; +struct hlist_nulls_node i_hash;struct list_head i_wb_list;/* backing dev IO list */ struct list_head i_lru;/* inode LRU list */ struct list_head i_sb_list;
@@ -803,7 +804,7 @@ struct inode {static inline int inode_unhashed (struct inode *inode) { -return hlist_unhashed (&inode->i_hash); +return hlist_nulls_unhashed (&inode->i_hash); }/*
include/linux/list_nulls.h +7 -0
@@ -83,6 +83,13 @@ static inline void hlist_nulls_del(struct hlist_nulls_node *n) n->pprev =LIST_POISON2 ; } +/* after that we'll appear to be on some hlist and hlist_nulls_del will work */ +static inline void hlist_nulls_add_fake (struct hlist_nulls_node *n) +{ + n->next = (struct hlist_nulls_node *)1UL ; + n->pprev = &n->next; +} +/** * hlist_nulls_for_each_entry - iterate over list of given type * @tpos: the type * to use as a loop cursor.
Signature · 1 line
1.7.2.3
Re: [PATCH 4/4] fs: rcu protect inode hash lookups
Quoted 27 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that inodes are using RCU freeing, we can walk the hash lists using RCU protection during lookups. Convert all the hash list operations to use RCU-based operators and drop the
inode_hash_lockaround pure lookup operations.Because we are using
SLAB_DESTROY_BY_RCU, we need to be careful about the lockless lookups and the way we initialise the struct inode during allocation. Theinode->i_lockis the critical item we need to avoid touching when allocating an inode, so move the initialisation of this lock to the slab constructor so that we can reliably use the lock even after the inode has been freed or is in the process of being allocated or initialised.We also need to change the order of checks during hash lookups to validate we have an active inode before we skip it. That means we need to revalidate the inode number and superblock after we have checked it for being freed or unhashed under the
inode->i_lock.Finally, during traversals an inode can be freed and reallocated and linked into a new hash chain, so an RCU safe traversal can jump to a different chain without indication. On a cache miss we need to validate that we ended the search on the same hash chain we started on. Do this by converting the hash lists to the
hlist_nulllist type and checking the list null on walk termination.
Quoted 2 lines
Signed-off-by Dave Chinner <[email protected]>---
Very nice stuff Dave !
I have one question
Quoted 5 lines
include/linux/list_nulls.h +0
-0
include/linux/list_nulls.h static inline void hlist_nulls_del(struct hlist_nulls_node *n) · line 92
Quoted 10 lines
include/linux/list_nulls.h +7
-0@@ -83,6 +83,13 @@ static inline void hlist_nulls_del(struct hlist_nulls_node *n) n->pprev =LIST_POISON2 ; } +/* after that we'll appear to be on some hlist and hlist_nulls_del will work */ +static inline void hlist_nulls_add_fake (struct hlist_nulls_node *n) +{ + n->next = (struct hlist_nulls_node *)1UL ; + n->pprev = &n->next; +} +
I am not sure why you need to change n->next here ?
This might restart lookups (making them doing a 'goto repeat', and possibly abort a lookup on slot 0 of hash chain, since this is a 'nulls' value for slot 0.
So at least I suggest adding a comment, and possibly use a different slot number (outside of slot ranges). If so, change documentation to restrict to 2^(BITS_PER_LONG-2) number of possible slots in hash table.
n->next = (struct hlist_nulls_node *)(1UL | (1UL << (BITS_PER_LONG-1));
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 4/4] fs: rcu protect inode hash lookups
In reply to Dave ChinnerReview
Quoted 2 lines
(quoted diff) +1
-1- hlist_add_fake(&inode->i_hash); + hlist_nulls_add_fake(&inode->i_hash);
Please add a preparatory inode_fake_hash/inode_mark_hashed or similar helper to isolate filesystems from the implementation details of the hash list.
fs/inode.c void __destroy_inode(struct inode *inode) · line 300
Quoted 6 lines
fs/inode.c +6
-0+/* +* reset the inode number so during RCU traversals we do not match this +* inode in any lookups until it is fully re-initialised again during +* allocation. +*/ + inode->i_ino =0 ;
There is no hard rule that i_ino is an invalid inode number. It can happen quite easily for inodes using the generic last_ino allocator, and I would not be surprised if there's some filesystems using it as part of the on disk layour either.
fs/inode.c static struct shrinker icache_shrinker = { · line 808
Quoted 6 lines
fs/inode.c +5
-0+rcu_read_unlock (); +if (locked) +spin_unlock (&inode_hash_lock);__wait_on_freeing_inode (inode); +if (locked) +spin_lock (&inode_hash_lock);
I can't say I like the locked argument, but I don't see an easy way around it. Can you at least keept the unlocking/relocking inside __wait_on_freeing_inode so that it's centralized in a single place for both find_inode pathes?
While at it moving __wait_on_freeing_inode to be above ifind would making changes in this area a lot easier to read, so maybe you can throw in a patch for that, too?
Quoted 10 lines
static struct inode *ifind(struct
super_block*sb,fs/inode.c +2
-3- struct hlist_head *head, int (*test)(struct inode *, void *), + struct hlist_nulls_head *head, int chain, + int (*test)(struct inode *, void *), void *data, const int wait) { struct inode *inode; -spin_lock (&inode_hash_lock); - inode =find_inode (sb, head, test, data);
fs/inode.c EXPORT_SYMBOL(igrab); · line 1209
Quoted 10 lines
fs/inode.c +1
-2+ inode =find_inode (sb, head, chain, test, data, false);if (inode) { -spin_unlock (&inode_hash_lock);if (likely (wait))wait_on_inode (inode);return inode; } -spin_unlock (&inode_hash_lock);return NULL ; }
This is starting to get a rather pointless helper. I'd suggest just killing ifind/ifind_fast and opencoding them in the caller, possibly as a preparatory patch.