Skip to contentSkip to conversation

cpuset memory spread slab cache filesys

PATCH In review Paul Jackson 3 participants 26 messages Feb 27, 2006 – Mar 2, 2006

Subsystems (20)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 cpuset memory spread slab cache filesys

From: Paul Jackson <[email protected]>

Mark file system inode and similar slab caches subject to SLAB_MEM_SPREAD memory spreading.

If a slab cache is marked SLAB_MEM_SPREAD, then anytime that a task that's in a cpuset with the 'memory_spread_slab' option enabled goes to allocate from such a slab cache, the allocations are spread evenly over all the memory nodes (task->mems_allowed) allowed to that task, instead of favoring allocation on the node local to the current cpu.

The following inode and similar caches are marked SLAB_MEM_SPREAD:

    file                               cache
    ====                               =====
    fs/adfs/super.c                    adfs_inode_cache
    fs/affs/super.c                    affs_inode_cache
    fs/befs/linuxvfs.c                 befs_inode_cache
    fs/bfs/inode.c                     bfs_inode_cache
    fs/block_dev.c                     bdev_cache
    fs/cifs/cifsfs.c                   cifs_inode_cache
    fs/coda/inode.c                    coda_inode_cache
    fs/dquot.c                         dquot
    fs/efs/super.c                     efs_inode_cache
    fs/ext2/super.c                    ext2_inode_cache
    fs/ext2/xattr.c (fs/mbcache.c)     ext2_xattr
    fs/ext3/super.c                    ext3_inode_cache
    fs/ext3/xattr.c (fs/mbcache.c)     ext3_xattr
    fs/fat/cache.c                     fat_cache
    fs/fat/inode.c                     fat_inode_cache
    fs/freevxfs/vxfs_super.c           vxfs_inode
    fs/hpfs/super.c                    hpfs_inode_cache
    fs/isofs/inode.c                   isofs_inode_cache
    fs/jffs/inode-v23.c                jffs_fm
    fs/jffs2/super.c                   jffs2_i
    fs/jfs/super.c                     jfs_ip
    fs/minix/inode.c                   minix_inode_cache
    fs/ncpfs/inode.c                   ncp_inode_cache
    fs/nfs/direct.c                    nfs_direct_cache
    fs/nfs/inode.c                     nfs_inode_cache
    fs/ntfs/super.c                    ntfs_big_inode_cache_name
    fs/ntfs/super.c                    ntfs_inode_cache
    fs/ocfs2/dlm/dlmfs.c               dlmfs_inode_cache
    fs/ocfs2/super.c                   ocfs2_inode_cache
    fs/proc/inode.c                    proc_inode_cache
    fs/qnx4/inode.c                    qnx4_inode_cache
    fs/reiserfs/super.c                reiser_inode_cache
    fs/romfs/inode.c                   romfs_inode_cache
    fs/smbfs/inode.c                   smb_inode_cache
    fs/sysv/inode.c                    sysv_inode_cache
    fs/udf/super.c                     udf_inode_cache
    fs/ufs/super.c                     ufs_inode_cache
    net/socket.c                       sock_inode_cache
    net/sunrpc/rpc_pipe.c              rpc_inode_cache

The choice of which slab caches to so mark was quite simple. I marked those already marked SLAB_RECLAIM_ACCOUNT, except for fs/xfs, dentry_cache, inode_cache, and buffer_head, which were marked in a previous patch. Even though SLAB_RECLAIM_ACCOUNT is for a different purpose, it marks the same potentially large file system i/o related slab caches as we need for memory spreading.

Given that the rule now becomes "wherever you would have used a SLAB_RECLAIM_ACCOUNT slab cache flag before (usually the inode cache), use the SLAB_MEM_SPREAD flag too", this should be easy enough to maintain. Future file system writers will just copy one of the existing file system slab cache setups and tend to get it right without thinking.

==> This patch violates line length constraints, pushing
many lines past 80 columns. The next patch wraps text to fit within 80 columns again. I split these two so that it would be easy to see the changes caused by adding the SLAB_MEM_SPREAD option separately from the formatting changes caused by rewrapping source lines.

---

Andrew - these two cpuset-memory-spread-slab-cache-hooks-filesys and *-spacing patches should fit nicely in your *-mm stack,
right after cpuset-memory-spread-slab-cache-hooks.patch. -pj

 fs/adfs/super.c          |    2 +-
 fs/affs/super.c          |    2 +-
 fs/befs/linuxvfs.c       |    2 +-
 fs/bfs/inode.c           |    2 +-
 fs/block_dev.c           |    2 +-
 fs/cifs/cifsfs.c         |    2 +-
 fs/coda/inode.c          |    2 +-
 fs/dquot.c               |    2 +-
 fs/efs/super.c           |    2 +-
 fs/ext2/super.c          |    2 +-
 fs/ext3/super.c          |    2 +-
 fs/fat/cache.c           |    2 +-
 fs/fat/inode.c           |    2 +-
 fs/freevxfs/vxfs_super.c |    2 +-
 fs/hpfs/super.c          |    2 +-
 fs/isofs/inode.c         |    2 +-
 fs/jffs/inode-v23.c      |    4 ++--
 fs/jffs2/super.c         |    2 +-
 fs/jfs/super.c           |    2 +-
 fs/mbcache.c             |    2 +-
 fs/minix/inode.c         |    2 +-
 fs/ncpfs/inode.c         |    2 +-
 fs/nfs/direct.c          |    2 +-
 fs/nfs/inode.c           |    2 +-
 fs/ntfs/super.c          |    4 ++--
 fs/ocfs2/dlm/dlmfs.c     |    2 +-
 fs/ocfs2/super.c         |    2 +-
 fs/proc/inode.c          |    2 +-
 fs/qnx4/inode.c          |    2 +-
 fs/reiserfs/super.c      |    2 +-
 fs/romfs/inode.c         |    2 +-
 fs/smbfs/inode.c         |    2 +-
 fs/sysv/inode.c          |    2 +-
 fs/udf/super.c           |    2 +-
 fs/ufs/super.c           |    2 +-
 net/socket.c             |    2 +-
 net/sunrpc/rpc_pipe.c    |    2 +-
 37 files changed, 39 insertions(+), 39 deletions(-)
fs/adfs/super.c +1 -1
@@ -241,7 +241,7 @@ static int init_inodecache(void)
 {
 	adfs_inode_cachep = kmem_cache_create("adfs_inode_cache",
 					     sizeof(struct adfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (adfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/affs/super.c +1 -1
@@ -98,7 +98,7 @@ static int init_inodecache(void)
 {
 	affs_inode_cachep = kmem_cache_create("affs_inode_cache",
 					     sizeof(struct affs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (affs_inode_cachep == NULL)
 		return -ENOMEM;
fs/befs/linuxvfs.c +1 -1
@@ -427,7 +427,7 @@ befs_init_inodecache(void)
 {
 	befs_inode_cachep = kmem_cache_create("befs_inode_cache",
 					      sizeof (struct befs_inode_info),
-					      0, SLAB_RECLAIM_ACCOUNT,
+					      0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					      init_once, NULL);
 	if (befs_inode_cachep == NULL) {
 		printk(KERN_ERR "befs_init_inodecache: "
fs/bfs/inode.c +1 -1
@@ -257,7 +257,7 @@ static int init_inodecache(void)
 {
 	bfs_inode_cachep = kmem_cache_create("bfs_inode_cache",
 					     sizeof(struct bfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (bfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/block_dev.c +1 -1
@@ -319,7 +319,7 @@ void __init bdev_cache_init(void)
 {
 	int err;
 	bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode),
-			0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_PANIC,
+			0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD|SLAB_PANIC,
 			init_once, NULL);
 	err = register_filesystem(&bd_type);
 	if (err)
fs/cifs/cifsfs.c +1 -1
@@ -692,7 +692,7 @@ cifs_init_inodecache(void)
 {
 	cifs_inode_cachep = kmem_cache_create("cifs_inode_cache",
 					      sizeof (struct cifsInodeInfo),
-					      0, SLAB_RECLAIM_ACCOUNT,
+					      0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					      cifs_init_once, NULL);
 	if (cifs_inode_cachep == NULL)
 		return -ENOMEM;
fs/coda/inode.c +1 -1
@@ -71,7 +71,7 @@ int coda_init_inodecache(void)
 {
 	coda_inode_cachep = kmem_cache_create("coda_inode_cache",
 				sizeof(struct coda_inode_info),
-				0, SLAB_RECLAIM_ACCOUNT,
+				0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				init_once, NULL);
 	if (coda_inode_cachep == NULL)
 		return -ENOMEM;
fs/dquot.c +1 -1
@@ -1817,7 +1817,7 @@ static int __init dquot_init(void)
 
 	dquot_cachep = kmem_cache_create("dquot", 
 			sizeof(struct dquot), sizeof(unsigned long) * 4,
-			SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_PANIC,
+			SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD|SLAB_PANIC,
 			NULL, NULL);
 
 	order = 0;
fs/efs/super.c +1 -1
@@ -81,7 +81,7 @@ static int init_inodecache(void)
 {
 	efs_inode_cachep = kmem_cache_create("efs_inode_cache",
 				sizeof(struct efs_inode_info),
-				0, SLAB_RECLAIM_ACCOUNT,
+				0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				init_once, NULL);
 	if (efs_inode_cachep == NULL)
 		return -ENOMEM;
fs/ext2/super.c +1 -1
@@ -175,7 +175,7 @@ static int init_inodecache(void)
 {
 	ext2_inode_cachep = kmem_cache_create("ext2_inode_cache",
 					     sizeof(struct ext2_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (ext2_inode_cachep == NULL)
 		return -ENOMEM;
fs/ext3/super.c +1 -1
@@ -481,7 +481,7 @@ static int init_inodecache(void)
 {
 	ext3_inode_cachep = kmem_cache_create("ext3_inode_cache",
 					     sizeof(struct ext3_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (ext3_inode_cachep == NULL)
 		return -ENOMEM;
fs/fat/cache.c +1 -1
@@ -49,7 +49,7 @@ int __init fat_cache_init(void)
 {
 	fat_cache_cachep = kmem_cache_create("fat_cache",
 				sizeof(struct fat_cache),
-				0, SLAB_RECLAIM_ACCOUNT,
+				0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				init_once, NULL);
 	if (fat_cache_cachep == NULL)
 		return -ENOMEM;
fs/fat/inode.c +1 -1
@@ -518,7 +518,7 @@ static int __init fat_init_inodecache(vo
 {
 	fat_inode_cachep = kmem_cache_create("fat_inode_cache",
 					     sizeof(struct msdos_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (fat_inode_cachep == NULL)
 		return -ENOMEM;
fs/freevxfs/vxfs_super.c +1 -1
@@ -260,7 +260,7 @@ vxfs_init(void)
 {
 	vxfs_inode_cachep = kmem_cache_create("vxfs_inode",
 			sizeof(struct vxfs_inode_info), 0, 
-			SLAB_RECLAIM_ACCOUNT, NULL, NULL);
+			SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, NULL, NULL);
 	if (vxfs_inode_cachep)
 		return register_filesystem(&vxfs_fs_type);
 	return -ENOMEM;
fs/hpfs/super.c +1 -1
@@ -191,7 +191,7 @@ static int init_inodecache(void)
 {
 	hpfs_inode_cachep = kmem_cache_create("hpfs_inode_cache",
 					     sizeof(struct hpfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (hpfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/isofs/inode.c +1 -1
@@ -87,7 +87,7 @@ static int init_inodecache(void)
 {
 	isofs_inode_cachep = kmem_cache_create("isofs_inode_cache",
 					     sizeof(struct iso_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (isofs_inode_cachep == NULL)
 		return -ENOMEM;
fs/jffs/inode-v23.c +2 -2
@@ -1812,14 +1812,14 @@ init_jffs_fs(void)
 	}
 #endif
 	fm_cache = kmem_cache_create("jffs_fm", sizeof(struct jffs_fm),
-				     0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT, 
+				     0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				     NULL, NULL);
 	if (!fm_cache) {
 		return -ENOMEM;
 	}
 
 	node_cache = kmem_cache_create("jffs_node",sizeof(struct jffs_node),
-				       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT, 
+				       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				       NULL, NULL);
 	if (!node_cache) {
 		kmem_cache_destroy(fm_cache);
fs/jffs2/super.c +1 -1
@@ -331,7 +331,7 @@ static int __init init_jffs2_fs(void)
 
 	jffs2_inode_cachep = kmem_cache_create("jffs2_i",
 					     sizeof(struct jffs2_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     jffs2_i_init_once, NULL);
 	if (!jffs2_inode_cachep) {
 		printk(KERN_ERR "JFFS2 error: Failed to initialise inode cache\n");
fs/jfs/super.c +1 -1
@@ -634,7 +634,7 @@ static int __init init_jfs_fs(void)
 
 	jfs_inode_cachep =
 	    kmem_cache_create("jfs_ip", sizeof(struct jfs_inode_info), 0, 
-			    SLAB_RECLAIM_ACCOUNT, init_once, NULL);
+			    SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, init_once, NULL);
 	if (jfs_inode_cachep == NULL)
 		return -ENOMEM;
 
fs/mbcache.c +1 -1
@@ -288,7 +288,7 @@ mb_cache_create(const char *name, struct
 			INIT_LIST_HEAD(&cache->c_indexes_hash[m][n]);
 	}
 	cache->c_entry_cache = kmem_cache_create(name, entry_size, 0,
-		SLAB_RECLAIM_ACCOUNT, NULL, NULL);
+		SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, NULL, NULL);
 	if (!cache->c_entry_cache)
 		goto fail;
 
fs/minix/inode.c +1 -1
@@ -80,7 +80,7 @@ static int init_inodecache(void)
 {
 	minix_inode_cachep = kmem_cache_create("minix_inode_cache",
 					     sizeof(struct minix_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (minix_inode_cachep == NULL)
 		return -ENOMEM;
fs/ncpfs/inode.c +1 -1
@@ -72,7 +72,7 @@ static int init_inodecache(void)
 {
 	ncp_inode_cachep = kmem_cache_create("ncp_inode_cache",
 					     sizeof(struct ncp_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (ncp_inode_cachep == NULL)
 		return -ENOMEM;
fs/nfs/direct.c +1 -1
@@ -771,7 +771,7 @@ int nfs_init_directcache(void)
 {
 	nfs_direct_cachep = kmem_cache_create("nfs_direct_cache",
 						sizeof(struct nfs_direct_req),
-						0, SLAB_RECLAIM_ACCOUNT,
+						0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 						NULL, NULL);
 	if (nfs_direct_cachep == NULL)
 		return -ENOMEM;
fs/nfs/inode.c +1 -1
@@ -2163,7 +2163,7 @@ static int nfs_init_inodecache(void)
 {
 	nfs_inode_cachep = kmem_cache_create("nfs_inode_cache",
 					     sizeof(struct nfs_inode),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (nfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/ntfs/super.c +2 -2
@@ -3084,7 +3084,7 @@ static int __init init_ntfs_fs(void)
 
 	ntfs_inode_cache = kmem_cache_create(ntfs_inode_cache_name,
 			sizeof(ntfs_inode), 0,
-			SLAB_RECLAIM_ACCOUNT, NULL, NULL);
+			SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, NULL, NULL);
 	if (!ntfs_inode_cache) {
 		printk(KERN_CRIT "NTFS: Failed to create %s!\n",
 				ntfs_inode_cache_name);
@@ -3093,7 +3093,7 @@ static int __init init_ntfs_fs(void)
 
 	ntfs_big_inode_cache = kmem_cache_create(ntfs_big_inode_cache_name,
 			sizeof(big_ntfs_inode), 0,
-			SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT,
+			SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 			ntfs_big_inode_init_once, NULL);
 	if (!ntfs_big_inode_cache) {
 		printk(KERN_CRIT "NTFS: Failed to create %s!\n",
fs/ocfs2/dlm/dlmfs.c +1 -1
@@ -596,7 +596,7 @@ static int __init init_dlmfs_fs(void)
 
 	dlmfs_inode_cache = kmem_cache_create("dlmfs_inode_cache",
 				sizeof(struct dlmfs_inode_private),
-				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT,
+				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				dlmfs_init_once, NULL);
 	if (!dlmfs_inode_cache)
 		return -ENOMEM;
fs/ocfs2/super.c +1 -1
@@ -951,7 +951,7 @@ static int ocfs2_initialize_mem_caches(v
 {
 	ocfs2_inode_cachep = kmem_cache_create("ocfs2_inode_cache",
 					       sizeof(struct ocfs2_inode_info),
-					       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT,
+					       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					       ocfs2_inode_init_once, NULL);
 	if (!ocfs2_inode_cachep)
 		return -ENOMEM;
fs/proc/inode.c +1 -1
@@ -121,7 +121,7 @@ int __init proc_init_inodecache(void)
 {
 	proc_inode_cachep = kmem_cache_create("proc_inode_cache",
 					     sizeof(struct proc_inode),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (proc_inode_cachep == NULL)
 		return -ENOMEM;
fs/qnx4/inode.c +1 -1
@@ -546,7 +546,7 @@ static int init_inodecache(void)
 {
 	qnx4_inode_cachep = kmem_cache_create("qnx4_inode_cache",
 					     sizeof(struct qnx4_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (qnx4_inode_cachep == NULL)
 		return -ENOMEM;
fs/reiserfs/super.c +1 -1
@@ -521,7 +521,7 @@ static int init_inodecache(void)
 	reiserfs_inode_cachep = kmem_cache_create("reiser_inode_cache",
 						  sizeof(struct
 							 reiserfs_inode_info),
-						  0, SLAB_RECLAIM_ACCOUNT,
+						  0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 						  init_once, NULL);
 	if (reiserfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/romfs/inode.c +1 -1
@@ -579,7 +579,7 @@ static int init_inodecache(void)
 {
 	romfs_inode_cachep = kmem_cache_create("romfs_inode_cache",
 					     sizeof(struct romfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (romfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/smbfs/inode.c +1 -1
@@ -80,7 +80,7 @@ static int init_inodecache(void)
 {
 	smb_inode_cachep = kmem_cache_create("smb_inode_cache",
 					     sizeof(struct smb_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (smb_inode_cachep == NULL)
 		return -ENOMEM;
fs/sysv/inode.c +1 -1
@@ -342,7 +342,7 @@ int __init sysv_init_icache(void)
 {
 	sysv_inode_cachep = kmem_cache_create("sysv_inode_cache",
 			sizeof(struct sysv_inode_info), 0,
-			SLAB_RECLAIM_ACCOUNT,
+			SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 			init_once, NULL);
 	if (!sysv_inode_cachep)
 		return -ENOMEM;
fs/udf/super.c +1 -1
@@ -140,7 +140,7 @@ static int init_inodecache(void)
 {
 	udf_inode_cachep = kmem_cache_create("udf_inode_cache",
 					     sizeof(struct udf_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (udf_inode_cachep == NULL)
 		return -ENOMEM;
fs/ufs/super.c +1 -1
@@ -1184,7 +1184,7 @@ static int init_inodecache(void)
 {
 	ufs_inode_cachep = kmem_cache_create("ufs_inode_cache",
 					     sizeof(struct ufs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT,
+					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 					     init_once, NULL);
 	if (ufs_inode_cachep == NULL)
 		return -ENOMEM;
net/socket.c +1 -1
@@ -319,7 +319,7 @@ static int init_inodecache(void)
 {
 	sock_inode_cachep = kmem_cache_create("sock_inode_cache",
 				sizeof(struct socket_alloc),
-				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT,
+				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
 				init_once, NULL);
 	if (sock_inode_cachep == NULL)
 		return -ENOMEM;
net/sunrpc/rpc_pipe.c +1 -1
@@ -850,7 +850,7 @@ int register_rpc_pipefs(void)
 {
 	rpc_inode_cachep = kmem_cache_create("rpc_inode_cache",
                                              sizeof(struct rpc_inode),
-                                             0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT,
+                                             0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
                                              init_once, NULL);
 	if (!rpc_inode_cachep)
 		return -ENOMEM;
Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#1 lore raw

PATCH 2/2 cpuset memory spread slab cache format

From: Paul Jackson <[email protected]>

Rewrap the overly long source code lines resulting from the previous patch's addition of the slab cache flag SLAB_MEM_SPREAD. This patch contains only formatting changes, and no function change.

---

 fs/adfs/super.c       |    3 ++-
 fs/affs/super.c       |    3 ++-
 fs/befs/linuxvfs.c    |    3 ++-
 fs/bfs/inode.c        |    3 ++-
 fs/block_dev.c        |    3 ++-
 fs/cifs/cifsfs.c      |    3 ++-
 fs/dquot.c            |    3 ++-
 fs/ext2/super.c       |    3 ++-
 fs/ext3/super.c       |    3 ++-
 fs/fat/inode.c        |    3 ++-
 fs/hpfs/super.c       |    3 ++-
 fs/isofs/inode.c      |    3 ++-
 fs/jffs/inode-v23.c   |   10 ++++++----
 fs/jffs2/super.c      |    3 ++-
 fs/jfs/super.c        |    3 ++-
 fs/minix/inode.c      |    3 ++-
 fs/ncpfs/inode.c      |    3 ++-
 fs/nfs/direct.c       |    3 ++-
 fs/nfs/inode.c        |    3 ++-
 fs/ocfs2/dlm/dlmfs.c  |    3 ++-
 fs/ocfs2/super.c      |    8 +++++---
 fs/proc/inode.c       |    3 ++-
 fs/qnx4/inode.c       |    3 ++-
 fs/reiserfs/super.c   |    3 ++-
 fs/romfs/inode.c      |    3 ++-
 fs/smbfs/inode.c      |    3 ++-
 fs/udf/super.c        |    3 ++-
 fs/ufs/super.c        |    3 ++-
 net/socket.c          |    3 ++-
 net/sunrpc/rpc_pipe.c |    7 ++++---
 30 files changed, 69 insertions(+), 37 deletions(-)
fs/adfs/super.c +2 -1
@@ -241,7 +241,8 @@ static int init_inodecache(void)
 {
 	adfs_inode_cachep = kmem_cache_create("adfs_inode_cache",
 					     sizeof(struct adfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (adfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/affs/super.c +2 -1
@@ -98,7 +98,8 @@ static int init_inodecache(void)
 {
 	affs_inode_cachep = kmem_cache_create("affs_inode_cache",
 					     sizeof(struct affs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (affs_inode_cachep == NULL)
 		return -ENOMEM;
fs/befs/linuxvfs.c +2 -1
@@ -427,7 +427,8 @@ befs_init_inodecache(void)
 {
 	befs_inode_cachep = kmem_cache_create("befs_inode_cache",
 					      sizeof (struct befs_inode_info),
-					      0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					      0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					      init_once, NULL);
 	if (befs_inode_cachep == NULL) {
 		printk(KERN_ERR "befs_init_inodecache: "
fs/bfs/inode.c +2 -1
@@ -257,7 +257,8 @@ static int init_inodecache(void)
 {
 	bfs_inode_cachep = kmem_cache_create("bfs_inode_cache",
 					     sizeof(struct bfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (bfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/block_dev.c +2 -1
@@ -319,7 +319,8 @@ void __init bdev_cache_init(void)
 {
 	int err;
 	bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode),
-			0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD|SLAB_PANIC,
+			0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+				SLAB_MEM_SPREAD|SLAB_PANIC),
 			init_once, NULL);
 	err = register_filesystem(&bd_type);
 	if (err)
fs/cifs/cifsfs.c +2 -1
@@ -692,7 +692,8 @@ cifs_init_inodecache(void)
 {
 	cifs_inode_cachep = kmem_cache_create("cifs_inode_cache",
 					      sizeof (struct cifsInodeInfo),
-					      0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					      0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					      cifs_init_once, NULL);
 	if (cifs_inode_cachep == NULL)
 		return -ENOMEM;
fs/dquot.c +2 -1
@@ -1817,7 +1817,8 @@ static int __init dquot_init(void)
 
 	dquot_cachep = kmem_cache_create("dquot", 
 			sizeof(struct dquot), sizeof(unsigned long) * 4,
-			SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD|SLAB_PANIC,
+			(SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+				SLAB_MEM_SPREAD|SLAB_PANIC),
 			NULL, NULL);
 
 	order = 0;
fs/ext2/super.c +2 -1
@@ -175,7 +175,8 @@ static int init_inodecache(void)
 {
 	ext2_inode_cachep = kmem_cache_create("ext2_inode_cache",
 					     sizeof(struct ext2_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (ext2_inode_cachep == NULL)
 		return -ENOMEM;
fs/ext3/super.c +2 -1
@@ -481,7 +481,8 @@ static int init_inodecache(void)
 {
 	ext3_inode_cachep = kmem_cache_create("ext3_inode_cache",
 					     sizeof(struct ext3_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (ext3_inode_cachep == NULL)
 		return -ENOMEM;
fs/fat/inode.c +2 -1
@@ -518,7 +518,8 @@ static int __init fat_init_inodecache(vo
 {
 	fat_inode_cachep = kmem_cache_create("fat_inode_cache",
 					     sizeof(struct msdos_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (fat_inode_cachep == NULL)
 		return -ENOMEM;
fs/hpfs/super.c +2 -1
@@ -191,7 +191,8 @@ static int init_inodecache(void)
 {
 	hpfs_inode_cachep = kmem_cache_create("hpfs_inode_cache",
 					     sizeof(struct hpfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (hpfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/isofs/inode.c +2 -1
@@ -87,7 +87,8 @@ static int init_inodecache(void)
 {
 	isofs_inode_cachep = kmem_cache_create("isofs_inode_cache",
 					     sizeof(struct iso_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (isofs_inode_cachep == NULL)
 		return -ENOMEM;
fs/jffs/inode-v23.c +6 -4
@@ -1812,15 +1812,17 @@ init_jffs_fs(void)
 	}
 #endif
 	fm_cache = kmem_cache_create("jffs_fm", sizeof(struct jffs_fm),
-				     0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
-				     NULL, NULL);
+		       0,
+		       SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+		       NULL, NULL);
 	if (!fm_cache) {
 		return -ENOMEM;
 	}
 
 	node_cache = kmem_cache_create("jffs_node",sizeof(struct jffs_node),
-				       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
-				       NULL, NULL);
+		       0,
+		       SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+		       NULL, NULL);
 	if (!node_cache) {
 		kmem_cache_destroy(fm_cache);
 		return -ENOMEM;
fs/jffs2/super.c +2 -1
@@ -331,7 +331,8 @@ static int __init init_jffs2_fs(void)
 
 	jffs2_inode_cachep = kmem_cache_create("jffs2_i",
 					     sizeof(struct jffs2_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     jffs2_i_init_once, NULL);
 	if (!jffs2_inode_cachep) {
 		printk(KERN_ERR "JFFS2 error: Failed to initialise inode cache\n");
fs/jfs/super.c +2 -1
@@ -634,7 +634,8 @@ static int __init init_jfs_fs(void)
 
 	jfs_inode_cachep =
 	    kmem_cache_create("jfs_ip", sizeof(struct jfs_inode_info), 0, 
-			    SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, init_once, NULL);
+			    SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+			    init_once, NULL);
 	if (jfs_inode_cachep == NULL)
 		return -ENOMEM;
 
fs/minix/inode.c +2 -1
@@ -80,7 +80,8 @@ static int init_inodecache(void)
 {
 	minix_inode_cachep = kmem_cache_create("minix_inode_cache",
 					     sizeof(struct minix_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (minix_inode_cachep == NULL)
 		return -ENOMEM;
fs/ncpfs/inode.c +2 -1
@@ -72,7 +72,8 @@ static int init_inodecache(void)
 {
 	ncp_inode_cachep = kmem_cache_create("ncp_inode_cache",
 					     sizeof(struct ncp_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (ncp_inode_cachep == NULL)
 		return -ENOMEM;
fs/nfs/direct.c +2 -1
@@ -771,7 +771,8 @@ int nfs_init_directcache(void)
 {
 	nfs_direct_cachep = kmem_cache_create("nfs_direct_cache",
 						sizeof(struct nfs_direct_req),
-						0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+						0, (SLAB_RECLAIM_ACCOUNT|
+							SLAB_MEM_SPREAD),
 						NULL, NULL);
 	if (nfs_direct_cachep == NULL)
 		return -ENOMEM;
fs/nfs/inode.c +2 -1
@@ -2163,7 +2163,8 @@ static int nfs_init_inodecache(void)
 {
 	nfs_inode_cachep = kmem_cache_create("nfs_inode_cache",
 					     sizeof(struct nfs_inode),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (nfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/ocfs2/dlm/dlmfs.c +2 -1
@@ -596,7 +596,8 @@ static int __init init_dlmfs_fs(void)
 
 	dlmfs_inode_cache = kmem_cache_create("dlmfs_inode_cache",
 				sizeof(struct dlmfs_inode_private),
-				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+				0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+					SLAB_MEM_SPREAD),
 				dlmfs_init_once, NULL);
 	if (!dlmfs_inode_cache)
 		return -ENOMEM;
fs/ocfs2/super.c +5 -3
@@ -950,9 +950,11 @@ static void ocfs2_inode_init_once(void *
 static int ocfs2_initialize_mem_caches(void)
 {
 	ocfs2_inode_cachep = kmem_cache_create("ocfs2_inode_cache",
-					       sizeof(struct ocfs2_inode_info),
-					       0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
-					       ocfs2_inode_init_once, NULL);
+				       sizeof(struct ocfs2_inode_info),
+				       0,
+				       (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
+				       ocfs2_inode_init_once, NULL);
 	if (!ocfs2_inode_cachep)
 		return -ENOMEM;
 
fs/proc/inode.c +2 -1
@@ -121,7 +121,8 @@ int __init proc_init_inodecache(void)
 {
 	proc_inode_cachep = kmem_cache_create("proc_inode_cache",
 					     sizeof(struct proc_inode),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (proc_inode_cachep == NULL)
 		return -ENOMEM;
fs/qnx4/inode.c +2 -1
@@ -546,7 +546,8 @@ static int init_inodecache(void)
 {
 	qnx4_inode_cachep = kmem_cache_create("qnx4_inode_cache",
 					     sizeof(struct qnx4_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (qnx4_inode_cachep == NULL)
 		return -ENOMEM;
fs/reiserfs/super.c +2 -1
@@ -521,7 +521,8 @@ static int init_inodecache(void)
 	reiserfs_inode_cachep = kmem_cache_create("reiser_inode_cache",
 						  sizeof(struct
 							 reiserfs_inode_info),
-						  0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+						  0, (SLAB_RECLAIM_ACCOUNT|
+							SLAB_MEM_SPREAD),
 						  init_once, NULL);
 	if (reiserfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/romfs/inode.c +2 -1
@@ -579,7 +579,8 @@ static int init_inodecache(void)
 {
 	romfs_inode_cachep = kmem_cache_create("romfs_inode_cache",
 					     sizeof(struct romfs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (romfs_inode_cachep == NULL)
 		return -ENOMEM;
fs/smbfs/inode.c +2 -1
@@ -80,7 +80,8 @@ static int init_inodecache(void)
 {
 	smb_inode_cachep = kmem_cache_create("smb_inode_cache",
 					     sizeof(struct smb_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (smb_inode_cachep == NULL)
 		return -ENOMEM;
fs/udf/super.c +2 -1
@@ -140,7 +140,8 @@ static int init_inodecache(void)
 {
 	udf_inode_cachep = kmem_cache_create("udf_inode_cache",
 					     sizeof(struct udf_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (udf_inode_cachep == NULL)
 		return -ENOMEM;
fs/ufs/super.c +2 -1
@@ -1184,7 +1184,8 @@ static int init_inodecache(void)
 {
 	ufs_inode_cachep = kmem_cache_create("ufs_inode_cache",
 					     sizeof(struct ufs_inode_info),
-					     0, SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+					     0, (SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
 					     init_once, NULL);
 	if (ufs_inode_cachep == NULL)
 		return -ENOMEM;
net/socket.c +2 -1
@@ -319,7 +319,8 @@ static int init_inodecache(void)
 {
 	sock_inode_cachep = kmem_cache_create("sock_inode_cache",
 				sizeof(struct socket_alloc),
-				0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
+				0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+					SLAB_MEM_SPREAD),
 				init_once, NULL);
 	if (sock_inode_cachep == NULL)
 		return -ENOMEM;
net/sunrpc/rpc_pipe.c +4 -3
@@ -849,9 +849,10 @@ init_once(void * foo, kmem_cache_t * cac
 int register_rpc_pipefs(void)
 {
 	rpc_inode_cachep = kmem_cache_create("rpc_inode_cache",
-                                             sizeof(struct rpc_inode),
-                                             0, SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD,
-                                             init_once, NULL);
+				sizeof(struct rpc_inode),
+				0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
+						SLAB_MEM_SPREAD),
+				init_once, NULL);
 	if (!rpc_inode_cachep)
 		return -ENOMEM;
 	register_filesystem(&rpc_pipe_fs_type);
Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#2 lore raw
In reply to Paul Jackson
Quoted 8 lines · Paul Jackson

From: Paul Jackson <[email protected]>

Mark file system inode and similar slab caches subject to SLAB_MEM_SPREAD memory spreading.

If a slab cache is marked SLAB_MEM_SPREAD, then anytime that a task that's in a cpuset with the 'memory_spread_slab' option enabled

Is there a way to use it without cpumemsets?

I would assume it's useful for smaller machines too, but they generally don't use cpumemsets.
-Andi

#3 lore raw
Quoted 1 line · Andi

Is there a way to use it without cpumemsets?

Not that I know of. So far as I can recall, the task->mems_allowed field (over which the spreading is done) is only manipulated by the cpuset code. So at least what I have here requires cpusets to have any useful affect, yes.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#4 lore raw
Quoted 7 lines · Paul Jackson
Quoted 1 line · Andi

Is there a way to use it without cpumemsets?

Not that I know of. So far as I can recall, the task->mems_allowed field (over which the spreading is done) is only manipulated by the cpuset code. So at least what I have here requires cpusets to have any useful affect, yes.

Well we could fall back to interleave on the node online map if some sort of flag is set.

On the other hand setting memory policy to MPOL_INTERLEAVE already provides that functionality.

#5 lore raw
Quoted 2 lines · Christoph Lameter

On the other hand setting memory policy to MPOL_INTERLEAVE already provides that functionality.

Yes, but not selective for these slabs caches. I think it would be useful if we could interleave inodes/dentries but still keep a local policy for normal program memory.

-Andi

#6 lore raw
Quoted 8 lines · Andi Kleen
Quoted 2 lines · Christoph Lameter

On the other hand setting memory policy to MPOL_INTERLEAVE already provides that functionality.

Yes, but not selective for these slabs caches. I think it would be useful if we could interleave inodes/dentries but still keep a local policy for normal program memory.

We could make the memory policy only apply if the SLAB_MEM_SPREAD option is set:

mm/slab.c +3 -1
Index: linux-2.6.16-rc4-mm2/mm/slab.c
===================================================================
@@ -2871,7 +2871,9 @@ static void *alternate_node_alloc(struct
 	if (in_interrupt())
 		return NULL;
 	nid_alloc = nid_here = numa_node_id();
-	if (cpuset_do_slab_mem_spread() && (cachep->flags & SLAB_MEM_SPREAD))
+	if (!cachep->flags & SLAB_MEM_SPREAD)
+		return NULL;
+	if (cpuset_do_slab_mem_spread())
 		nid_alloc = cpuset_mem_spread_node();
 	else if (current->mempolicy)
 		nid_alloc = slab_node(current->mempolicy);
#7 lore raw
Quoted 3 lines · Christoph Lameter

We could make the memory policy only apply if the SLAB_MEM_SPREAD option is set:

Which memory policy? The one of the process?

mm/slab.c static void *alternate_node_alloc(struct · line 2874

Quoted 10 lines

Index: linux-2.6.16-rc4-mm2/mm/slab.c
===================================================================

mm/slab.c +1 -1
@@ -2871,7 +2871,9 @@ static void *alternate_node_alloc(struct
 	if (in_interrupt())
 		return NULL;
 	nid_alloc = nid_here = numa_node_id();
-	if (cpuset_do_slab_mem_spread() && (cachep->flags & SLAB_MEM_SPREAD))
+	if (!cachep->flags & SLAB_MEM_SPREAD)

brackets missing I guess.

mm/slab.c static void *alternate_node_alloc(struct · line 2879

Quoted 6 lines
mm/slab.c +2 -0
+		return NULL;
+	if (cpuset_do_slab_mem_spread())
 		nid_alloc = cpuset_mem_spread_node();
 	else if (current->mempolicy)
 		nid_alloc = slab_node(current->mempolicy);

-Andi

#8 lore raw
Quoted 3 lines · Andi Kleen
Quoted 1 line · Christoph Lameter

We could make the memory policy only apply if the SLAB_MEM_SPREAD option

Which memory policy? The one of the process?

Yes.

#9 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Andi Kleen
Quoted 1 line · Christoph Lameter

We could make the memory policy only apply if the SLAB_MEM_SPREAD option

Which memory policy? The one of the process?

Yes.

I don't quite get your logic here. For me it would be logical to apply the memory policy from the process for anything _but_ slab caches that have SLAB_MEM_SPREAD set. For those interleaving should be always used.

Why are you proposing to do it the other way round?

-Andi

#10 lore raw
Quoted 3 lines · Andi Kleen

I don't quite get your logic here. For me it would be logical to apply the memory policy from the process for anything _but_ slab caches that have SLAB_MEM_SPREAD set. For those interleaving should be always used.

Interleaving is a special feature to be used only if we know that the objects are used in a system wide fashion. Interleave should never be the default option.

F.e. inode interleaving for a process that is running on one node and scanning files will reduce performance. This is the typical case.

On the other hand if files are used by multiple processes in a cpuset then interleaving may be beneficial.

Quoted 1 line

Why are you proposing to do it the other way round?

Because these are the types of objects that may benefit from interleaving. Other slabs need to always be allocated in a node local way.

#11 lore raw

Hmmm ... your thread with Andi confuses me ...

Oh well.

I take it that Andi is suggesting that there be the option to override the tasks mempolicy, in the particular case of these file i/o slab caches, with an interleave over the online nodes.

This option would be useful in the case that a system is not using cpusets, but still wants to spread out these particular (sometimes large) file i/o caches.

Questions for Andi:

1) Are you content to have such a interleave of these particular file i/o slabs triggered by a mm/mempolicy.c option? Or do you think we need some sort of task external API to invoke this policy?

If mempolicy API works, then I would think that someone, such as yourself or Christoph, could easily enough propose an extension to the mempolicy API that invoked such a policy. It could leverage some of the apparatus already provided by my current patchset here, such as the per-slab SLAB_MEMP_SPREAD flag settings, the task PF_* flag bits and the hook in ____cache_alloc() to call out to alternate_node_alloc().

If a system wide API (that can be externally imposed on some or all tasks from outside the task) is desirable, then I am left wondering why you don't use cpusets for this.

2) Do you recommend that the page (file buffer) cache also be interleavable, across all online nodes, if optionally requested, on systems not using cpusets?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#12 lore raw
Quoted 7 lines · Paul Jackson

Hmmm ... your thread with Andi confuses me ...

Oh well.

I take it that Andi is suggesting that there be the option to override the tasks mempolicy, in the particular case of these file i/o slab caches, with an interleave over the online nodes.

Yep exactly.

Quoted 3 lines

This option would be useful in the case that a system is not using cpusets, but still wants to spread out these particular (sometimes large) file i/o caches.

Yep.

Quoted 5 lines

Questions for Andi:

1) Are you content to have such a interleave of these particular file i/o slabs triggered by a mm/mempolicy.c option? Or do you think we need some sort of task external API to invoke this policy?

Task external. mempolicy.c has no good way to handle multiple policies like this. I was thinking of a simple sysctl

I guess I can cook up a patch once your code is merged.

Quoted 3 lines

2) Do you recommend that the page (file buffer) cache also be interleavable, across all online nodes, if optionally requested, on systems not using cpusets?

Yes, but as a separate option.

-Andi

#13 lore raw
Quoted 6 lines
Quoted 3 lines

1) Are you content to have such a interleave of these particular file i/o slabs triggered by a mm/mempolicy.c option? Or do you think we need some sort of task external API to invoke this policy?

Task external. mempolicy.c has no good way to handle multiple policies like this. I was thinking of a simple sysctl

No need to implement a sysctl for this. The current cpuset facility should provide just what you want, if I am understanding correctly.

It would be really easy. Run with a kernel that has cpusets configured in. One time at boot, enable memory spreading for these slabs:
test -d /dev/cpuset || mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
echo 1 > /dev/cpuset/memory_spread_slab # enable system wide

That's all you need to do to enable this system wide.

With this, tasks will be spreading these selected slab caches, independently of whatever mempolicy they have.

To disable this memory spreading system wide:
echo 0 > /dev/cpuset/memory_spread_slab # disable system wide

If you want to control which tasks have these slab spread, then it is just a few more lines once at boottime. The following lines make a second cpuset 'spread_tasks'. Tasks in this second cpuset will be spread; the other tasks in the root cpuset won't be spread.

One time at boot:
test -d /dev/cpuset || mkdir /dev/cpuset mount -t cpuset cpuset /dev/cpuset
mkdir /dev/cpuset/spread_tasks
cat /dev/cpuset/cpus > /dev/cpuset/spread_tasks/cpus cat /dev/cpuset/mems > /dev/cpuset/spread_tasks/mems echo 1 > /dev/cpuset/spread_tasks/memory_spread_slab

Then during operation, for each task $pid that is to be spread:
echo $pid > /dev/cpuset/spread_tasks/tasks # enable for $pid

or to disable that spreading for a pid:
echo $pid > /dev/cpuset/tasks # disable for $pid

These echo's can be done with open/write/close system calls if you prefer.

The first two echos above would correspond to a sysctl that applied system wide, enabling or disabling memory spreading on these slabs for all tasks. The last two echo's correspond directly to a sysctl that applies to a single specified pid.

Why do a new sysctl, when the existing open/write/close system calls can do the same thing?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#14 lore raw
Quoted 9 lines · Paul Jackson
Quoted 6 lines
Quoted 3 lines

1) Are you content to have such a interleave of these particular file i/o slabs triggered by a mm/mempolicy.c option? Or do you think we need some sort of task external API to invoke this policy?

Task external. mempolicy.c has no good way to handle multiple policies like this. I was thinking of a simple sysctl

No need to implement a sysctl for this. The current cpuset facility should provide just what you want, if I am understanding correctly.

The main reason i'm reluctant to use this is that the cpuset fast path overhead (e.g. in memory allocators etc.) is quite large and I wouldn't like to recommend people to enable all this overhead by default just to get more useful dcache/inode behaviour on small NUMA systems.

-Andi

#15 lore raw
Quoted 7 lines · Andi Kleen
Quoted 2 lines

No need to implement a sysctl for this. The current cpuset facility should provide just what you want, if I am understanding correctly.

The main reason i'm reluctant to use this is that the cpuset fast path overhead (e.g. in memory allocators etc.) is quite large and I wouldn't like to recommend people to enable all this overhead by default just to get more useful dcache/inode behaviour on small NUMA systems.

Is this a gut feeling or do you have some measurements to back that up? Paul worked hard on making all the overhead in critical paths as light as possible and from what I can see he did a very good job.

#16 lore raw
In reply to Andi Kleen
Quoted 2 lines · Andi

The main reason i'm reluctant to use this is that the cpuset fast path overhead (e.g. in memory allocators etc.) is quite large

I disagree.

I spent much time minimizing that overhead over the last few months, as a direct result of your recommendation to do so.

Especially in the case that all tasks are in the root cpuset (as in the scenario I just suggested for setting this memory spreading policy for all tasks), the overhead is practically zero. The key hook is an inline test done (usually) once per page allocation on an essentially read only global 'number_of_cpusets' that determines it is <= 1.

I disagree with your "quite large" characterization.

Please explain further.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#17 lore raw
Quoted 8 lines · Paul Jackson
Quoted 2 lines · Andi

The main reason i'm reluctant to use this is that the cpuset fast path overhead (e.g. in memory allocators etc.) is quite large

I disagree.

I spent much time minimizing that overhead over the last few months, as a direct result of your recommendation to do so.

IIRC my recommendation only optimized the case of nobody using cpuset if I remember correctly.

Using a single cpuset would already drop into the slow path, right?

Hmm, possibly it's better now, but I remember being shocked last time I looked at the code in detail ow much code it executed for a normal page allocation and how many cache lines it touched. This was some time ago admittedly.

Also on a different angle I would like to make the dcache/inode spreading basically default on x86-64 and I'm not sure I want to get into the business of explaining all the distributions how to set up cpusets and set up new file systems.
For that a single switch that can be just set by default is much more practical.

Quoted 4 lines

Especially in the case that all tasks are in the root cpuset (as in the scenario I just suggested for setting this memory spreading policy for all tasks), the overhead is practically zero.

Ok.

Quoted 5 lines

The key hook is an
inline test done (usually) once per page allocation on an essentially read only global 'number_of_cpusets' that determines it is <= 1.

I disagree with your "quite large" characterization.

Agreed perhaps it was somewhat exaggerated.

-Andi

#18 lore raw
Quoted 5 lines
Quoted 2 lines

I spent much time minimizing that overhead over the last few months, as a direct result of your recommendation to do so.

IIRC my recommendation only optimized the case of nobody using cpuset if I remember correctly.

As a result of your general concern with the performance impact of cpusets on the page allocation code path, I optimized each element of it, not just the one case covered by your specific recommendation.

Take a look.

Quoted 1 line

Using a single cpuset would already drop into the slow path, right?

No - having a single cpuset is the fastest path. All tasks are in that root cpuset in that case, and all nodes allowed.

Quoted 2 lines

I'm not sure I want to get into the business
of explaining all the distributions how to set up cpusets ..

Good grief - I already quoted the 3 lines of boottime init script it would take - this can't require that much explaining, and your new sysctl can't get by with much less:
test -d /dev/cpuset || mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
echo 1 > /dev/cpuset/memory_spread_slab # enable system wide

Quoted 1 line

and set up new file systems.

That's a Linux source issue that matters a single time in the history of each file system type supported by Linux. It is not a customer or even distro issue.

And even from the perspective of maintaining Linux, this should be on autopilot. Every file systems inode cache is marked, and if we do nothing, as more file system types are invented for Linux, they will predictably cut+paste the inode slab cache setup from an existing file system, and "just get it right."

Quoted 2 lines

For that a single switch that can be just set by default is much more practical.

Doing it via cpusets is also a single switch that is set by default.

It is just as practical; well more practical - it's already there.

===

Mind you, I don't have any profound objections to such a sysctl.

I just don't see that it serves any purpose, and I suspect that misunderstandings of the performance impact of cpusets are the primary source of motivation for such a sysctl.

I prefer to (1) set the record straight on cpusets, and (2) avoid adding additional kernel mechanisms that are redundant.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#19 lore raw
Quoted 10 lines · Paul Jackson
Quoted 5 lines
Quoted 2 lines

I spent much time minimizing that overhead over the last few months, as a direct result of your recommendation to do so.

IIRC my recommendation only optimized the case of nobody using cpuset if I remember correctly.

As a result of your general concern with the performance impact of cpusets on the page allocation code path, I optimized each element of it, not just the one case covered by your specific recommendation.

Thanks for doing that work then.

Quoted 5 lines
Quoted 2 lines

Using a single cpuset would already drop into the slow path, right?

No - having a single cpuset is the fastest path. All tasks are in that root cpuset in that case, and all nodes allowed.

Faster than no cpuset?

Quoted 7 lines
Quoted 3 lines

I'm not sure I want to get into the business
of explaining all the distributions how to set up cpusets ..

Good grief - I already quoted the 3 lines of boottime init script it would take - this can't require that much explaining, and your new sysctl can't get by with much less:

It would just be on by default - no user space configuration needed.

Quoted 5 lines

And even from the perspective of maintaining Linux, this should be on autopilot. Every file systems inode cache is marked, and if we do nothing, as more file system types are invented for Linux, they will predictably cut+paste the inode slab cache setup from an existing file system, and "just get it right."

If something is a good default it shouldn't need user space configuration at all imho. Only the "weird" cases should.

Quoted 3 lines

I just don't see that it serves any purpose, and I suspect that misunderstandings of the performance impact of cpusets are the primary source of motivation for such a sysctl.

No that was just one. The other was having good defaults even on lightweight kernels.

-Andi

#20 lore raw
Quoted 4 lines
Quoted 2 lines

No - having a single cpuset is the fastest path. All tasks are in that root cpuset in that case, and all nodes allowed.

Faster than no cpuset?

If CONFIG_CPUSET is enabled (which I thought was likely to become the norm for most distros -- though you would know better than I if this is likely) then:

There is no such case as "no cpuset" !!

The minimal, fastest, case is one root cpuset holding all tasks.

Quoted 2 lines

If something is a good default it shouldn't need user space configuration at all imho. Only the "weird" cases should.

So are you just saying we got the default backwards?

Well ... I left the default for memory spreading these inode slab caches as it was - not spread (preferring node local).

I did that because I did not have the awareness that this default should be changed for most systems. I tend to leave defaults as they are, unless I have good reason to change them.

But for the SGI systems I care about, I'd prefer the default to be spreading them.

If you think it would be better to change this default, now that the mechanism is in place to do support spreading these slabs, then I could certainly go along with that.

Then your systems would not have to do anything in user space, unless they wanted to disable spreading these slabs (which of course they could easily do using cpusets ;).

Should we change the default to enable this spreading?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#21 lore raw
Quoted 5 lines · Paul Jackson
Quoted 3 lines

If something is a good default it shouldn't need user space configuration at all imho. Only the "weird" cases should.

So are you just saying we got the default backwards?

Yes.

Quoted 2 lines

But for the SGI systems I care about, I'd prefer the default to be spreading them.

I think it's the best default for smaller systems too. I've had people complaining about node inbalances that were caused by one or two being filled up with d/icache. And the small latencies of accessing them don't matter very much.

Quoted 3 lines

If you think it would be better to change this default, now that the mechanism is in place to do support spreading these slabs, then I could certainly go along with that.

Yes that would make me happy.

Quoted 5 lines

Then your systems would not have to do anything in user space, unless they wanted to disable spreading these slabs (which of course they could easily do using cpusets ;).

Should we change the default to enable this spreading?

I would be in favour of it

-Andi

#22 lore raw
Quoted 4 lines · Andi Kleen

I think it's the best default for smaller systems too. I've had people complaining about node inbalances that were caused by one or two being filled up with d/icache. And the small latencies of accessing them don't matter very much.

But these are special situations. Placing memory on a distance node is not beneficial in the standard case of a single threaded process churning along opening and closing files etc.

Interleave is only beneficial for special applications that use a common pool of data and that implement no other means of locality control. At that point we sacrifice the performance benefit that comes with node locality in order not to overload a single node.

Kernels before 2.6.16 suffer from special overload situations that are due to not having the ability to reclaim the pagecache and the slab cache. This is going to change in SLES10.

Quoted 5 lines
Quoted 3 lines

If you think it would be better to change this default, now that the mechanism is in place to do support spreading these slabs, then I could certainly go along with that.

Yes that would make me happy.

It seems that we are trying to sacrifice the performance of many in order to accomodate a few special situations. Cpusets are ideal for those situations since they allow the localization of the interleaving to a slice of the machine. Processes on the rest of the box can still get node local memory and run at optimal performance.

Quoted 1 line

I would be in favour of it

Please run performance tests with single threaded processes if you do not believe me before doing any of this.

#23 lore raw

Well, as to what should be the default for this, enabling or not enabling memory spreading on these inode and similar such slab caches, I will have to leave that up to others more expert than I.

I am content to suggest that the default should reflect either the current default (no spreading, as such was not an option until now) or what is best for the majority of systems (whatever that is).

Probably best not to change the default, unless there is a clear concensus that it's wrong in the majority of cases.

Whatever systems don't like the default can change it easily enough, one way or the other.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#24 lore raw
In reply to Christoph Lameter
Quoted 4 lines · Christoph Lameter

Interleave is only beneficial for special applications that use a common pool of data and that implement no other means of locality control. At that point we sacrifice the performance benefit that comes with node locality in order not to overload a single node.

My rationale is the locality is only important for cache lines that
are very frequently accessed. By far the most frequently accessed
items is user space data, system calls are still relatively rare compare to that and system calls that touch files even less so

(often in my measurements 40% and more of all syscalls are gettimeofday actually)

Also we don't have very good balancing control on dcaches.

The problem is when one node runs updatedb or similar it will end up allocating a lot of these objects, and then later when a process ends up on that node it can have trouble allocating node local memory (and a user process missing local memory is typically much worse than a kernel object)

I guess your remote claim changes will help, but I'm not convinced they are the best solution.

[hmm, actually didn't we discuss this once at length anyways. Apparently I failed to convince you back then @:]

Quoted 3 lines

Kernels before 2.6.16 suffer from special overload situations that are due to not having the ability to reclaim the pagecache and the slab cache.

Reclaiming is slow. Better not dig into this hole in the first place.

Quoted 4 lines
Quoted 1 line

I would be in favour of it

Please run performance tests with single threaded processes if you do not believe me before doing any of this.

Sure. But the motivation is less the single thread performance anyways, but more the degradation under extreme loads.

-Andi

#25 lore raw
Quoted 2 lines · Andi Kleen

(often in my measurements 40% and more of all syscalls are gettimeofday actually)

Yes that is why we excessively optimized gettimeofday in asm on ia64...

Quoted 1 line

Also we don't have very good balancing control on dcaches.

Right but I hope we will get there if we can get the zoned vm statistics in and rework the VM a bit.

Quoted 5 lines
Quoted 2 lines

Please run performance tests with single threaded processes if you do not believe me before doing any of this.

Sure. But the motivation is less the single thread performance anyways, but more the degradation under extreme loads.

The extreme loads may benefit from interleave. But note that the performance gains in the NUMA slab allocator came from exploiting locality. The support of policies and other off node memory accesses in the SLAB allocator is an afterthought. Only node local accesses can be served from per cpu caches with a simple interrupt on / off. Off node accesses generated by policies etc will require locking and working with remote memory structures.

If these are indeed rare user space accesses that require slab elements then these performance issues do not matter and you may be able to spread without performance penalties. However, our experience with the slab allocator was that intensive workloads can be influenced by slab performance.