Slab defragmentation V4
Subsystems (10)
cover Slab defragmentation V4
V3->V4:
- Optimize scan for slabs that need defragmentation
- Add /sys/slab/*/defrag_ratio to allow setting defrag limits per slab.
- Add support for buffer heads.
- Describe how the cleanup after the daily updatedb can be improved by slab defragmentation.
- Rediff against 2.6.22-rc6-mm1 (+ slub patches in mm but there should be minimal overlap--if any--with those)
V2->V3
- Support directory reclaim
- Add infrastructure to trigger defragmentation after slab shrinking if we have slabs with a high degree of fragmentation.
V1->V2
- Clean up control flow using a state variable. Simplify API. Back to 2 functions that now take arrays of objects.
- Inode defrag support for a set of filesystems
- Fix up dentry defrag support to work on negative dentries by adding a new dentry flag that indicates that a dentry is not in the process of being freed or allocated.
Slab defragmentation is useful to increase the object density in slab caches. On reclaim the fragmentation ratios will be checked and if a the object density (ratio between maximum objects that could be stored in the allocated slabs and the actuall objects in use) in a defragmentable slab is less than a certain percentage (defaults to 30%) then the slabs with the lowest number of objects in them will be freed which increases the object density.
Currently supported are
1. dentry defrag
2. inode defrag (with a generic interface to allow easy setup of more filesystems than the currently supported ext2/3/4 reiserfs, XFS and proc)
3. buffer_heads
One typical mechanism that triggers slab defragmentation on my systems is the daily run of
updatedb
Updatedb scans all files on the system which causes a high inode and dentry use. After updatedb is complete we need to go back to the regular use patterns (typical on my machine: kernel compiles). Those need the memory now for different purposes. The inodes and dentries used for updatedb will gradually be aged by the dentry/inode reclaim algorithm which will free up the dentries and inode entries randomly through the slabs that were allocated. As a result the slabs will become sparsely populated. If they become empty then they can be freed but a lot of them will remain sparsely populated. That is where slab defrag comes in: It removes the slabs with just a few entries reclaiming more memory for other uses.
Currently slab reclaim writes messages like this to the syslog if slab defrag is occurring:
Test results (see appended scripts / user space code for more data)
(3 level tree with 10 entries at first level , 20 at the second and 30 files at the third level. Files at the lowest level were removed to create inode fragmentation)
%Ra is the allocation ratio (need to apply the slabinfo patch to get those numbers)
inode reclaim in reiserfs
Name Objects Objsize Space Slabs/Part/Cpu O/S O %Ra %Ef Flg
dentry 14660 200 3.0M 733/0/1 20 0 100 97 Da reiser_inode_cache 1596 640 4.1M 256/201/1 25 2 24 24 DCa
Status after defrag
Name Objects Objsize Space Slabs/Part/Cpu O/S O %Ra %Ef Flg
dentry 8849 200 1.8M 454/17/1 20 0 97 95 Da reiser_inode_cache 1381 640 1.0M 65/11/0 25 2 84 82 DCa
Slab defragmentation can be triggered in two ways:
1. Manually by running
slabinfo -s <slabs-to-shrink>
or manually by the kernel calling
kmem_cache_shrink(slab)
(Currently only ACPI is doing such a call to a slab that has no defragmentation support. In that case we simply do what SLAB does:
drop per cpu caches and sift through partial list for free slabs).
2. Automatically if defragmentable slabs reach a certain degree of fragmentation.
The point where slab defragmentation occurs is can be set at
/proc/sys/vm/slab_defrag_ratio
Slab fragmentation is measured by how much of the possible objects in a slab are in use. The default setting for slab_defrag_ratio is 30%. This means that slab fragmentation is going to be triggered if there are more than 3 free object slots for each allocated object.
Setting the slab_defrag_ratio higher will cause more defragmentation runs. If slab_defrag_ratio is set to 0 then no slab defragmentation occurs.
Slabs are checked for their fragmentation levels after the slabs have been shrunk by running shrinkers in vm/scan.c during memory reclaim. This means that slab defragmentation is only triggered if we are under memory pressure and if there is significant slab fragmentation.
Test script:
#!/bin/sh
echo 30 >/proc/sys/vm/slab_defrag_ratio
./gazfiles c 3 10 20 30
echo "Status before"
slabinfo -D
./gazfiles d 2
echo "Status after removing files"
slabinfo -D
slabinfo -s
echo "Status after defrag"
slabinfo -D
./gazfiles d 0
gazfiles.c :
/*
* Create a gazillion of files to be able to create slab fragmentation *
* (C) 2007 sgi, Christoph Lameter <[email protected]>
*
* Create a n layered hierachy of files of empty files
*
* gazfiles <action> <levels> <n1> <n2> ...
*
* gazfiles c[reate] 3 50 50 50
*
* gazfiles s[hrink] <levels>
*
* gazfiles r[andomkill] <nr to kill>
*/
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <string.h>
#include <unistd.h>
#include <stdarg.h>
#include <getopt.h>
#include <regex.h>
#include <errno.h>
#define MAXIMUM_LEVELS 10
int level;
int sizes[MAXIMUM_LEVELS];
void fatal(const char *x, ...)
{
va_list ap;
va_start(ap, x);
vfprintf(stderr, x, ap);
va_end(ap);
exit(1);
}int read_gaz(void)
{
FILE *f = fopen(".gazinfo", "r");
int rc = 0;
int i;
if (!f)
return 0;
if (!fscanf(f, "%d", &level))
goto out;
if (level >= MAXIMUM_LEVELS)
goto out;
for (i = 0; i < level; i++)
if (!fscanf(f, " %d", &sizes[i]))
goto out;
rc = 1;
out:
fclose(f);
return rc;
}void write_gaz(void)
{
FILE *f = fopen(".gazinfo","w");
int i;
fprintf(f, "%d",level);
for (i = 0; i < level; i++)
fprintf(f," %d", sizes[i]);
fprintf(f, "\n");
fclose(f);
}void cre(int l)
{
int i;
for (i = 0; i < sizes[l - 1]; i++) {
char name[20];
sprintf(name, "%03d", i);
if (l < level) {
mkdir(name, 0775);
chdir(name);
cre(l + 1);
chdir("..");
} else {
FILE *f;
f = fopen(name,"w");
fprintf(f, "Test");
fclose(f);
}
}
}void create(int l, char **sz)
{
int i;
level = l;
for (i = 0; i < level; i++)
sizes[i] = atoi(sz[i]);
if (mkdir("gazf", 0775))
fatal("Cannot create gazf here\n");
chdir("gazf");
write_gaz();
cre(1);
chdir("..");
}void shrink(int level)
{
if (chdir("gazf"))
fatal("No gazfiles in this directory");
read_gaz();
chdir("..");
}void scand(int l, void (*func)(int, int, char *, unsigned long),
unsigned long level)
{
DIR *dir;
struct dirent *de;
dir = opendir(".");
if (!dir)
fatal("Cannot open directory");
while ((de = readdir(dir))) {
struct stat s;
if (de->d_name[0] == '.')
continue;
/*
* Some idiot broke the glibc library or made it impossible
* to figure out how to make readdir work right
*/
stat(de->d_name, &s);
if (S_ISDIR(s.st_mode))
de->d_type = DT_DIR;
if (de->d_type == DT_DIR) {
if (chdir(de->d_name))
fatal("Cannot enter %s", de->d_name);
scand(l + 1, func, level);
chdir("..");
func(l, 1, de->d_name, level);
} else {
func(l, 0, de->d_name, level);
}
}
closedir(dir);
}void traverse(void (*func)(int, int, char *, unsigned long),
unsigned long level)
{
if (chdir("gazf"))
fatal("No gazfiles in this directory");
scand(1, func, level);
chdir("..");
}void randomkill(int nr)
{
if (chdir("gazf"))
fatal("No gazfiles in this directory");
read_gaz();
chdir("..");
}void del_func(int l, int dir, char *name, unsigned long level)
{
if (l <= level)
return;
if (dir) {
if (rmdir(name))
fatal("Cannot remove directory %s");
} else {
if (unlink(name))
fatal("Cannot unlink file %s");
}
}void delete(int l)
{
if (l == 0) {
system("rm -rf gazf");
return;
}
traverse(del_func, l);
}void usage(void)
{
printf("gazfiles: Tool to manage gazillions of files\n\n");
printf("gazfiles create <levels> <#l1> <#l2> ...\n");
printf("gazfiles delete <levels>\n");
printf("gazfiles shrink <levels>\n");
printf("gazfiles randomkill <nr>\n\n");
printf("(C) 2007 sgi, Christoph Lameter <[email protected]>\n");
exit(0);
}int main(int argc, char *argv[])
{
if (argc < 2)
usage();
switch (argv[1][0]) {
case 'c' :
create(atoi(argv[2]), argv + 3);
break;
case 's' :
if (argc != 3)
usage();
shrink(atoi(argv[2]));
break;
case 'r' :
if (argc != 3)
usage();
randomkill(atoi(argv[2]));
break;
case 'd':
if (argc != 3)
usage();
delete(atoi(argv[2]));
break;
default:
usage();
}
return 0;
}PATCH 1/12 Slab defragmentation: Add support for kmem_cache_ops
We use the parameter formerly used by the destructor to pass an optional pointer to a kmem_cache_ops structure to kmem_cache_create.
kmem_cache_ops is created as empty. Later patches populate kmem_cache_ops.
Create a KMEM_CACHE_OPS macro that allows the specification of a the kmem_cache_ops parameter.
Code to handle kmem_cache_ops is added to SLUB. SLAB and SLOB are updated to be able to accept a kmem_cache_ops structure but will ignore it.
--- include/linux/slab.h | 13 +++++++++---- include/linux/slub_def.h | 1 + mm/slab.c | 6 +++--- mm/slob.c | 2 +- mm/slub.c | 44 ++++++++++++++++++++++++++++++-------------- 5 files changed, 44 insertions(+), 22 deletions(-)
include/linux/slab.h +9 -4
@@ -51,10 +51,13 @@void __init kmem_cache_init (void );int slab_is_available (void ); +struct kmem_cache_ops { +}; +struct kmem_cache *kmem_cache_create (const char *,size_t ,size_t ,unsigned long ,void (*)(void *,struct kmem_cache *,unsigned long ), -void (*)(void *,struct kmem_cache *,unsigned long )); +const struct kmem_cache_ops *s);void kmem_cache_destroy (struct kmem_cache *);int kmem_cache_shrink (struct kmem_cache *);void kmem_cache_free (struct kmem_cache *,void *);
@@ -70,9 +73,11 @@ int kmem_ptr_validate(struct kmem_cache* f.e. add ____cacheline_aligned_in_smp to the struct declaration * then the objects will be properly aligned in SMP configurations. */ -#define KMEM_CACHE (__struct, __flags) kmem_cache_create(#__struct,\ - sizeof(struct __struct), __alignof__(struct __struct),\ - (__flags), NULL, NULL) +#define KMEM_CACHE_OPS (__struct, __flags, __ops) \ + kmem_cache_create(#__struct, sizeof(struct __struct), \ + __alignof__(struct __struct), (__flags), NULL, (__ops)) + +#define KMEM_CACHE (__struct, __flags) KMEM_CACHE_OPS(__struct, __flags, NULL)/* * The largest kmalloc size supported by the slab allocators is
mm/slub.c +30 -14
115 lines not on this page · raw message
include/linux/slub_def.h +1 -0
@@ -42,6 +42,7 @@ struct kmem_cache {int objects;/* Number of objects in slab */ int refcount;/* Refcount for slab cache destroy */ void (*ctor)(void *,struct kmem_cache *,unsigned long ); +const struct kmem_cache_ops *ops;int inuse;/* Offset to metadata */ int align;/* Alignment */ const char *name;/* Name (only for display!) */
mm/slab.c +3 -3
@@ -2102,7 +2102,7 @@ static int __init_refok setup_cpu_cache(* @align: The required alignment for the objects. * @flags: SLAB flags * @ctor: A constructor for the objects. -* @dtor: A destructor for the objects (not implemented anymore). +* @ops: A kmem_cache_ops structure (ignored). * * Returns a ptr to the cache on success, NULL on failure. * Cannot be called within a int, but can be interrupted.
@@ -2128,7 +2128,7 @@ struct kmem_cache * kmem_cache_create (const char *name, size_t size, size_t align, unsigned long flags, void (*ctor)(void*, struct kmem_cache *, unsigned long), - void (*dtor)(void*, struct kmem_cache *, unsigned long)) + const struct kmem_cache_ops *ops) { size_t left_over, slab_size, ralign;struct kmem_cache *cachep =NULL , *pc;
@@ -2137,7 +2137,7 @@ kmem_cache_create (const char *name, siz* Sanity checks... these are all serious usage bugs. */ if (!name ||in_interrupt () || (size <BYTES_PER_WORD ) || - size >KMALLOC_MAX_SIZE || dtor) { + size >KMALLOC_MAX_SIZE ) {printk (KERN_ERR "%s: Early error in slab %s\n" , __FUNCTION__, name);BUG ();
mm/slob.c +1 -1
@@ -493,7 +493,7 @@ struct kmem_cache {struct kmem_cache *kmem_cache_create (const char *name,size_t size,size_t align,unsigned long flags,void (*ctor)(void *,struct kmem_cache *,unsigned long ), -void (*dtor)(void *,struct kmem_cache *,unsigned long )) +const struct kmem_cache_ops *o) {struct kmem_cache *c;
PATCH 2/12 Slab defragmentation: Core piece
In reply to Christoph LameterV3->V4:
- Reduce impact of checking for slab caches in need of defragmentation by
A. Ordering the slab list. Defraggable slabs come first. Then
we can only scan the slab list until we hit the first unfragmentable slab.
B. Avoid determining the number of objects in partial slabs
when we can deduce from the number of partial slabs that the fragmentation ratio will not be below the boundary.
C. Allocate a scratch area once for multiple defragmentaable
slab caches. In order to do that we need to determine the maximum number of objects in defragmentable slabs in kmem_cache_create().
- The slab defragmentation ratio can now be setup for each slab cache separately.
- Rename the existing "defrag_ratio" in /sys/slab/* to remote_node_defrag_ratio to free up the defrag_ratio field for defragmentation.
Slab defragmentation occurs either
1. Unconditionally when kmem_cache_shrink is called on slab by the kernel calling kmem_cache_shrink or slabinfo triggering slab shrinking. This form performs defragmentation on all nodes of a NUMA system.
2. Conditionally when kmem_cache_defrag(<percentage>, <node>) is called.
The defragmentation is only performed if the fragmentation of the slab is higher then the specified percentage. Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab cache could hold.
kmem_cache_defrag takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. If a node number was specified then defragmentation is only performed on a specific node.
Slab defragmentation is a memory intensive operation that can be sped up in a NUMA system if mostly node local memory is accessed. That is the case if we just have reclaimed reclaim on a node.
For defragmentation SLUB first generates a sorted list of partial slabs. Sorting is performed according to the number of objects allocated. Thus the slabs with the least objects will be at the end.
We extract slabs off the tail of that list until we have either reached a mininum number of slabs or until we encounter a slab that has more than a quarter of its objects allocated. Then we attempt to remove the objects from each of the slabs taken.
In order for a slabcache to support defragmentation a couple of functions must be defined via kmem_cache_ops. These are
void *get(struct kmem_cache *s, int nr, void **objects)
Must obtain a reference to the listed objects. SLUB guarantees that the objects are still allocated. However, other threads may be blocked in slab_free attempting to free objects in the slab. These may succeed as soon as get() returns to the slab allocator. The function must be able to detect the situation and void the attempts to handle such objects (by for example voiding the corresponding entry in the objects array).
No slab operations may be performed in get(). Interrupts are disabled. What can be done is very limited. The slab lock for the page with the object is taken. Any attempt to perform a slab operation may lead to a deadlock.
get() returns a private pointer that is passed to kick. Should we be unable to obtain all references then that pointer may indicate to the kick() function that it should not attempt any object removal or move but simply remove the reference counts.
void kick(struct kmem_cache *, int nr, void **objects, void *get_result)
After SLUB has established references to the objects in a slab it will drop all locks and then use kick() to move objects out of the slab. The existence of the object is guaranteed by virtue of the earlier obtained references via get(). The callback may perform any slab operation since no locks are held at the time of call.
The callback should remove the object from the slab in some way. This may be accomplished by reclaiming the object and then running kmem_cache_free() or reallocating it and then running kmem_cache_free(). Reallocation is advantageous because the partial slabs were just sorted to have the partial slabs with the most objects first. Reallocation is likely to result in filling up a slab in addition to freeing up one slab so that it also can be removed from the partial list.
Kick() does not return a result. SLUB will check the number of remaining objects in the slab. If all objects were removed then we know that the operation was successful.
--- include/linux/slab.h | 32 +++ include/linux/slub_def.h | 12 + mm/slab.c | 5 mm/slob.c | 5 mm/slub.c | 415 ++++++++++++++++++++++++++++++++++++++--------- 5 files changed, 395 insertions(+), 74 deletions(-)
include/linux/slab.h +32 -0
@@ -51,7 +51,39 @@void __init kmem_cache_init (void );int slab_is_available (void ); +struct kmem_cache ; +struct kmem_cache_ops { +/* +* Called with slab lock held and interrupts disabled. +* No slab operation may be performed. +* +* Parameters passed are the number of objects to process +* and an array of pointers to objects for which we +* need references. +* +* Returns a pointer that is passed to the kick function. +* If all objects cannot be moved then the pointer may +* indicate that this wont work and then kick can simply +* remove the references that were already obtained. +* +* The array passed to get() is also passed to kick(). The +* function may remove objects by setting array elements to NULL. +*/ +void *(*get)(struct kmem_cache *,int nr,void **); + +/* +* Called with no locks held and interrupts enabled. +* Any operation may be performed in kick(). +* +* Parameters passed are the number of objects in the array, +* the array of pointers to the objects and the pointer +* returned by get(). +* +* Success is checked by examining the number of remaining +* objects in the slab. +*/ +void (*kick)(struct kmem_cache *,int nr,void **,void *private); }; struct kmem_cache *kmem_cache_create(const char *, size_t, size_t,
mm/slub.c +343 -72
511 lines not on this page · raw message
mm/slab.c +5 -0
@@ -2518,6 +2518,11 @@ int kmem_cache_shrink(struct kmem_cache }EXPORT_SYMBOL (kmem_cache_shrink); +int kmem_cache_defrag (int percent,int node) +{ +return 0 ; +} +/** * kmem_cache_destroy - delete a cache * @cachep: the cache to destroy
mm/slob.c +5 -0
@@ -591,6 +591,11 @@ int kmem_cache_shrink(struct kmem_cache }EXPORT_SYMBOL (kmem_cache_shrink); +int kmem_cache_defrag (int percentage,int node) +{ +return 0 ; +} +int kmem_ptr_validate (struct kmem_cache *a,const void *b) {return 0 ;
include/linux/slub_def.h +10 -2
@@ -50,9 +50,17 @@ struct kmem_cache {#ifdef CONFIG_SLUB_DEBUG struct kobject kobj;/* For sysfs */ #endif - +int defrag_ratio;/* +* objects/possible objects limit. If we have +* less that the specified percentage of +* objects allocated then defrag passes +* will start to occur during reclaim. +*/ #ifdef CONFIG_NUMA -int defrag_ratio; +int remote_node_defrag_ratio;/* +* Defragmentation through allocation from a +* remote node +*/ struct kmem_cache_node *node[MAX_NUMNODES ];#endif struct page *cpu_slab[NR_CPUS ];
PATCH 3/12 Slab defragmentation: Updates to slabinfo.c
In reply to Christoph Lameter-D lists caches that support defragmentation
-C lists caches that use a ctor.
Change field names for defrag_ratio and remote_node_defrag_ratio.
--- Documentation/vm/slabinfo.c | 52 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 8 deletions(-)
Documentation/vm/slabinfo.c +44 -8
@@ -30,6 +30,8 @@ struct slabinfo {int hwcache_align, object_size, objs_per_slab;int sanity_checks, slab_size, store_user, trace;int order, poison, reclaim_account, red_zone; +int defrag, ctor; +int defrag_ratio, remote_node_defrag_ratio;unsigned long partial, objects, slabs;int numa[MAX_NODES ];int numa_partial[MAX_NODES ];
@@ -56,6 +58,8 @@ int show_slab = 0;int skip_zero =1 ;int show_numa =0 ;int show_track =0 ; +int show_defrag =0 ; +int show_ctor =0 ;int show_first_alias =0 ;int validate =0 ;int shrink =0 ;
@@ -90,18 +94,20 @@ void fatal(const char *x, ...)void usage (void ) {printf ("slabinfo 5/7/2007. (c) 2007 sgi. [email protected]\n\n" -"slabinfo [-ahnpvtsz] [-d debugopts] [slab-regexp]\n" +"slabinfo [-aCDefhilnosSrtTvz1] [-d debugopts] [slab-regexp]\n" "-a|--aliases Show aliases\n" +"-C|--ctor Show slabs with ctors\n" "-d<options>|--debug=<options> Set/Clear Debug options\n" -"-e|--empty Show empty slabs\n" +"-D|--defrag Show defragmentable caches\n" +"-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" "-h|--help Show usage information\n" "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" "-n|--numa Show NUMA information\n" -"-o|--ops Show kmem_cache_ops\n" +"-o|--ops Show kmem_cache_ops\n" "-s|--shrink Shrink slabs\n" -"-r|--report Detailed report on single slabs\n" +"-r|--report Detailed report on single slabs\n" "-S|--Size Sort by size\n" "-t|--tracking Show alloc/free information\n" "-T|--Totals Show summary information\n"
@@ -281,7 +287,7 @@ int line = 0;void first_line (void ) {printf ("Name Objects Objsize Space " -"Slabs/Part/Cpu O/S O %%Fr %%Ef Flg\n" ); +"Slabs/Part/Cpu O/S O %%Ra %%Ef Flg\n" ); }/*
@@ -324,7 +330,7 @@ void slab_numa(struct slabinfo *s, int mreturn ;if (!line) { -printf ("\n%-21s:" , mode ?"NUMA nodes" :"Slab" ); +printf ("\n%-21s: Rto " , mode ?"NUMA nodes" :"Slab" );for (node =0 ; node <= highest_node; node++)printf (" %4d" , node);printf ("\n----------------------" );
@@ -333,6 +339,7 @@ void slab_numa(struct slabinfo *s, int mprintf ("\n" ); }printf ("%-21s " , mode ?"All slabs" : s->name); +printf ("%3d " , s->remote_node_defrag_ratio);for (node =0 ; node <= highest_node; node++) {char b[20 ];
@@ -406,6 +413,8 @@ void report(struct slabinfo *s)printf ("** Slabs are destroyed via RCU\n" );if (s->reclaim_account)printf ("** Reclaim accounting active\n" ); +if (s->defrag) +printf ("** Defragmentation at %d%%\n" , s->defrag_ratio);printf ("\nSizes (bytes) Slabs Debug Memory\n" );printf ("------------------------------------------------------------------------\n" );
@@ -452,6 +461,12 @@ void slabcache(struct slabinfo *s)if (show_empty && s->slabs)return ; +if (show_defrag && !s->defrag) +return ; + +if (show_ctor && !s->ctor) +return ; +store_size (size_str,slab_size (s));sprintf (dist_str,"%lu/%lu/%d" , s->slabs, s->partial, s->cpu_slabs);
@@ -462,6 +477,10 @@ void slabcache(struct slabinfo *s) *p++ ='*' ;if (s->cache_dma) *p++ ='d' ; +if (s->defrag) + *p++ ='D' ; +if (s->ctor) + *p++ ='C' ;if (s->hwcache_align) *p++ ='A' ;if (s->poison)
@@ -481,7 +500,7 @@ void slabcache(struct slabinfo *s)printf ("%-21s %8ld %7d %8s %14s %4d %1d %3ld %3ld %s\n" , s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial *100 ) / s->slabs :100 , + s->slabs ? (s->objects *100 ) / (s->slabs * s->objs_per_slab) :100 , s->slabs ? (s->objects * s->object_size *100 ) / (s->slabs * (page_size << s->order)) :100 , flags);
@@ -1071,7 +1090,16 @@ void read_slab_dir(void)decode_numa_list (slab->numa, t); slab->store_user =get_obj ("store_user" ); slab->trace =get_obj ("trace" ); + slab->defrag_ratio =get_obj ("defrag_ratio" ); + slab->remote_node_defrag_ratio = +get_obj ("remote_node_defrag_ratio" );chdir (".." ); +if (read_slab_obj (slab,"ops" )) { +if (strstr (buffer,"ctor :" )) + slab->ctor =1 ; +if (strstr (buffer,"kick :" )) + slab->defrag =1 ; + }if (slab->name[0 ] ==':' ) alias_targets++; slab++;
@@ -1121,7 +1149,9 @@ void output_slabs(void)struct option opts[] = { {"aliases" ,0 ,NULL ,'a' }, + {"ctor" ,0 ,NULL ,'C' }, {"debug" ,2 ,NULL ,'d' }, + {"defrag" ,0 ,NULL ,'D' }, {"empty" ,0 ,NULL ,'e' }, {"first-alias" ,0 ,NULL ,'f' }, {"help" ,0 ,NULL ,'h' },
@@ -1146,7 +1176,7 @@ int main(int argc, char *argv[]) page_size =getpagesize (); -while ((c =getopt_long (argc, argv,"ad::efhil1noprstvzTS" , +while ((c =getopt_long (argc, argv,"ad::efhil1noprstvzCDTS" , opts,NULL )) !=-1 )switch (c) {case '1' :
@@ -1196,6 +1226,12 @@ int main(int argc, char *argv[])case 'z' : skip_zero =0 ;break ; +case 'C' : + show_ctor =1 ; +break ; +case 'D' : + show_defrag =1 ; +break ;case 'T' : show_totals =1 ;break ;
PATCH 4/12 Slab defragmentation: Logic to trigger defragmentation from memory reclaim
In reply to Christoph LameterAt some point slab defragmentation needs to be triggered. The logical point for this is after slab shrinking was performed in vmscan.c. At that point the fragmentation ratio of a slab was increased by objects being freed. So we call kmem_cache_defrag from there.
slab_shrink() from vmscan.c is called in some contexts to do global shrinking of slabs and in others to do shrinking for a particular zone. Pass the zone to slab_shrink, so that slab_shrink can call kmem_cache_defrag() and restrict the defragmentation to the node that is under memory pressure.
--- fs/drop_caches.c | 2 +- include/linux/mm.h | 2 +- include/linux/slab.h | 1 + mm/vmscan.c | 27 ++++++++++++++++++++------- 4 files changed, 23 insertions(+), 9 deletions(-)
include/linux/slab.h +1 -0
@@ -96,6 +96,7 @@ void kmem_cache_free(struct kmem_cache *unsigned int kmem_cache_size (struct kmem_cache *);const char *kmem_cache_name (struct kmem_cache *);int kmem_ptr_validate (struct kmem_cache *cachep,const void *ptr); +int kmem_cache_defrag (int node);/* * Please use this macro to create slab caches. Simply specify the
mm/vmscan.c +20 -7
@@ -152,10 +152,18 @@ EXPORT_SYMBOL(unregister_shrinker);* are eligible for the caller's allocation attempt. It is used for balancing * slab reclaim versus page reclaim. * +* zone is the zone for which we are shrinking the slabs. If the intent +* is to do a global shrink then zone may be NULL. Specification of a +* zone is currently only used to limit slab defragmentation to a NUMA node. +* The performace of shrink_slab would be better (in particular under NUMA) +* if it could be targeted as a whole to the zone that is under memory +* pressure but the VFS infrastructure does not allow that at the present +* time. +* * Returns the number of slab objects which we shrunk. */ unsigned long shrink_slab (unsigned long scanned,gfp_t gfp_mask, -unsigned long lru_pages) +unsigned long lru_pages,struct zone *zone) {struct shrinker *shrinker;unsigned long ret =0 ;
@@ -218,6 +226,8 @@ unsigned long shrink_slab(unsigned long shrinker->nr += total_scan; }up_read (&shrinker_rwsem); +if (gfp_mask & __GFP_FS) +kmem_cache_defrag (zone ?zone_to_nid (zone) :-1 );return ret; }
@@ -1163,7 +1173,8 @@ unsigned long try_to_free_pages(struct zif (!priority)disable_swap_token (); nr_reclaimed +=shrink_zones (priority, zones, &sc); -shrink_slab (sc.nr_scanned, gfp_mask, lru_pages); +shrink_slab (sc.nr_scanned, gfp_mask, lru_pages, +NULL );if (reclaim_state) { nr_reclaimed += reclaim_state->reclaimed_slab; reclaim_state->reclaimed_slab =0 ;
@@ -1333,7 +1344,7 @@ loop_again: nr_reclaimed +=shrink_zone (priority, zone, &sc); reclaim_state->reclaimed_slab =0 ; nr_slab =shrink_slab (sc.nr_scanned,GFP_KERNEL , - lru_pages); + lru_pages, zone); nr_reclaimed += reclaim_state->reclaimed_slab; total_scanned += sc.nr_scanned;if (zone->all_unreclaimable)
@@ -1601,7 +1612,7 @@ unsigned long shrink_all_memory(unsigned/* If slab caches are huge, it's better to hit them first */ while (nr_slab >= lru_pages) { reclaim_state.reclaimed_slab =0 ; -shrink_slab (nr_pages, sc.gfp_mask, lru_pages); +shrink_slab (nr_pages, sc.gfp_mask, lru_pages,NULL );if (!reclaim_state.reclaimed_slab)break ;
@@ -1639,7 +1650,7 @@ unsigned long shrink_all_memory(unsigned reclaim_state.reclaimed_slab =0 ;shrink_slab (sc.nr_scanned, sc.gfp_mask, -count_lru_pages ()); +count_lru_pages (),NULL ); ret += reclaim_state.reclaimed_slab;if (ret >= nr_pages) goto out;
@@ -1656,7 +1667,8 @@ unsigned long shrink_all_memory(unsignedif (!ret) {do { reclaim_state.reclaimed_slab =0 ; -shrink_slab (nr_pages, sc.gfp_mask,count_lru_pages ()); +shrink_slab (nr_pages, sc.gfp_mask, +count_lru_pages (),NULL ); ret += reclaim_state.reclaimed_slab; }while (ret < nr_pages && reclaim_state.reclaimed_slab >0 ); }
@@ -1816,7 +1828,8 @@ static int __zone_reclaim(struct zone *z* Note that shrink_slab will free memory on all zones and may * take a long time. */ -while (shrink_slab (sc.nr_scanned, gfp_mask, order) && +while (shrink_slab (sc.nr_scanned, gfp_mask, order, + zone) &&zone_page_state (zone,NR_SLAB_RECLAIMABLE ) > slab_reclaimable - nr_pages) ;
fs/drop_caches.c +1 -1
@@ -52,7 +52,7 @@ void drop_slab(void)int nr_objects;do { - nr_objects =shrink_slab (1000 ,GFP_KERNEL ,1000 ); + nr_objects =shrink_slab (1000 ,GFP_KERNEL ,1000 ,NULL ); }while (nr_objects >10 ); }
include/linux/mm.h +1 -1
@@ -1248,7 +1248,7 @@ int in_gate_area_no_task(unsigned long aint drop_caches_sysctl_handler (struct ctl_table *,int ,struct file *,void __user *,size_t *,loff_t *);unsigned long shrink_slab (unsigned long scanned,gfp_t gfp_mask, -unsigned long lru_pages); +unsigned long lru_pages,struct zone *zone);extern void drop_pagecache_sb (struct super_block *);void drop_pagecache (void );void drop_slab (void );
PATCH 5/12 Slab defragmentation: Log information to the syslog to show defrag operations
In reply to Christoph LameterDump information into the syslog during defragmentation actions to show that something is occurring and what effect it has.
This is likely only useful in mm for testing and verification.
--- mm/slub.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/slub.c +13 -0
@@ -2599,6 +2599,8 @@ static unsigned long sort_partial_list(sreturn freed; } +#define NR_INUSE 40 +/* * Shrink the slab cache on a particular node of the cache */
@@ -2610,6 +2612,8 @@ static unsigned long __kmem_cache_shrinkLIST_HEAD (zaplist);int freed;int inuse; +int nr[NR_INUSE ] = {0 , }; +int i;spin_lock_irqsave (&n->list_lock, flags); freed =sort_partial_list (s, n, scratch);
@@ -2646,6 +2650,8 @@ static unsigned long __kmem_cache_shrink n->nr_partial--;SetSlabFrozen (page);slab_unlock (page); +if (inuse <NR_INUSE ) + nr[inuse]++; }spin_unlock_irqrestore (&n->list_lock, flags);
@@ -2659,6 +2665,13 @@ static unsigned long __kmem_cache_shrinkif (__kmem_cache_vacate (s, page, flags, scratch) ==0 ) freed++; } +printk (KERN_INFO "Slab %s: Defrag freed %d pages. PartSlab config=" , + s->name, freed << s->order); + +for (i =0 ; i <NR_INUSE ; i++) +if (nr[i]) +printk (" %d=%d" , i, nr[i]); +printk ("\n" );return freed; }
PATCH 6/12 Slab defragmentation: Support dentry defragmentation
In reply to Christoph Lameterget() uses the dcache lock and then works with dget_locked to obtain a reference to the dentry. An additional complication is that the dentry may be in process of being freed or it may just have been allocated. We add an additional flag to d_flags to be able to determined the status of an object.
kick() is called after get() has been used and after the slab has dropped all of its own locks. The dentry pruning for unused entries works in a straighforward way.
--- fs/dcache.c | 113 +++++++++++++++++++++++++++++++++++++++++++++---- include/linux/dcache.h | 5 ++ 2 files changed, 110 insertions(+), 8 deletions(-)
fs/dcache.c +105 -8
@@ -135,6 +135,7 @@ static struct dentry *d_kill(struct dentlist_del (&dentry->d_u.d_child); dentry_stat.nr_dentry--;/* For d_free, below */ + dentry->d_flags &= ~DCACHE_ENTRY_VALID ;/*drops the locks, at that point nobody can reach this dentry */ dentry_iput (dentry); parent = dentry->d_parent;
@@ -951,6 +952,7 @@ struct dentry *d_alloc(struct dentry * pif (parent)list_add (&dentry->d_u.d_child, &parent->d_subdirs); dentry_stat.nr_dentry++; + dentry->d_flags |=DCACHE_ENTRY_VALID ;spin_unlock (&dcache_lock);return dentry;
@@ -2108,18 +2110,113 @@ static void __init dcache_init_early(voiINIT_HLIST_HEAD (&dentry_hashtable[loop]); } +/* +* The slab is holding off frees. Thus we can safely examine +* the object without the danger of it vanishing from under us. +*/ +static void *get_dentries (struct kmem_cache *s,int nr,void **v) +{ +struct dentry *dentry; +int i; + +spin_lock (&dcache_lock); +for (i =0 ; i < nr; i++) { + dentry = v[i]; +/* +* if DCACHE_ENTRY_VALID is not set then the dentry +* may be already in the process of being freed. +*/ +if (!(dentry->d_flags &DCACHE_ENTRY_VALID )) + v[i] =NULL ; +else +dget_locked (dentry); + } +spin_unlock (&dcache_lock); +return 0 ; +} + +/* +* Slab has dropped all the locks. Get rid of the +* refcount we obtained earlier and also rid of the +* object. +*/ +static void kick_dentries (struct kmem_cache *s,int nr,void **v,void *private) +{ +struct dentry *dentry; +int abort =0 ; +int i; + +/* +* First invalidate the dentries without holding the dcache lock +*/ +for (i =0 ; i < nr; i++) { + dentry = v[i]; + +if (dentry) +d_invalidate (dentry); + } + +/* +* If we are the last one holding a reference then the dentries can +* be freed. We need the dcache_lock. +*/ +spin_lock (&dcache_lock); +for (i =0 ; i < nr; i++) { + dentry = v[i]; +if (!dentry) +continue ; + +if (abort) + goto put_dentry; + +spin_lock (&dentry->d_lock); +if (atomic_read (&dentry->d_count) >1 ) { +/* +* Reference count was increased. This means that we +* cannot free one object which makes it impossible +* to reclaim this slab. So it is best to abandon the +* freeing of further objects. +*/ + abort =1 ; +spin_unlock (&dentry->d_lock); +put_dentry: +spin_unlock (&dcache_lock); +dput (dentry); +spin_lock (&dcache_lock); +continue ; + } + +/* Remove from LRU */ +if (!list_empty (&dentry->d_lru)) { + dentry_stat.nr_unused--; +list_del_init (&dentry->d_lru); + } +/* Drop the entry */ +prune_one_dentry (dentry,1 ); + } +spin_unlock (&dcache_lock); + +/* +* dentries are freed using RCU so we need to wait until RCU +* operations arei complete +*/ +if (!abort) +synchronize_rcu (); +} + +static struct kmem_cache_ops dentry_kmem_cache_ops = { + .get = get_dentries, + .kick = kick_dentries, +}; +static void __init dcache_init (unsigned long mempages) {int loop; -/* -* A constructor could be added for stable state like the lists, -* but it is probably not worth it because of the cache nature -* of the dcache. -*/ - dentry_cache =KMEM_CACHE (dentry, -SLAB_RECLAIM_ACCOUNT |SLAB_PANIC |SLAB_MEM_SPREAD ); - + dentry_cache =KMEM_CACHE_OPS (dentry, +SLAB_RECLAIM_ACCOUNT |SLAB_PANIC |SLAB_MEM_SPREAD , + &dentry_kmem_cache_ops); +register_shrinker (&dcache_shrinker);/* Hash may have been set up in dcache_init_early */
include/linux/dcache.h +5 -0
@@ -177,6 +177,11 @@ d_iput: no no no yes#define DCACHE_INOTIFY_PARENT_WATCHED 0x0020/* Parent inode is watched */ +#define DCACHE_ENTRY_VALID 0x0040/* +* Entry is valid and not in the +* process of being created or +* destroyed. +*/ extern spinlock_t dcache_lock;/**
PATCH 7/12 Slab defragmentation: Support for buffer_head defrag
In reply to Christoph LameterLimited defragmentation support for buffer heads. Simply try to free the buffers in a sparsely populated slab page.
--- fs/buffer.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 2 deletions(-)
fs/buffer.c +65 -2
@@ -3078,12 +3078,75 @@ static int buffer_cpu_notify(struct notireturn NOTIFY_OK ; } +/* +* Get references on buffers. +* +* We obtain references on the page that uses the buffer. v[i] will point to +* the corresponding page after get_buffers() is through. +* +* We are safe from the underlying page being removed simply by doing +* a get_page_unless_zero. The buffer head removal may race at will. +* try_to_free_buffes will later take appropriate locks to remove the +* buffers if they are still there. +* +* TODO: Write out dirty buffers to increase the chance of kick_buffers +* to be successful. +*/ +static void *get_buffers (struct kmem_cache *s,int nr,void **v) +{ +struct page *page; +struct buffer_head *bh; +int i; + +for (i =0 ; i < nr; i++) { + bh = v[i]; + page = bh->b_page; +if (page &&PagePrivate (page) &&get_page_unless_zero (page)) + v[i] = page; +else + v[i] =NULL ; + } +return NULL ; +} + +/* +* Despite its name: kick_buffers operates on a list of pointers to +* page structs that was setup by get_buffer +*/ +static void kick_buffers (struct kmem_cache *s,int nr,void **v, +void *private) +{ +struct page *page; +int i; + +for (i =0 ; i < nr; i++) { + page = v[i]; + +if (!page) +continue ; + +if (!TestSetPageLocked (page)) { +if (PagePrivate (page)) +try_to_free_buffers (page); +unlock_page (page); + } +put_page (page); + } +} + +static struct kmem_cache_ops buffer_head_kmem_cache_ops = { + .get = get_buffers, + .kick = kick_buffers, +}; + +void __init buffer_init (void ) {int nrpages; - bh_cachep =KMEM_CACHE (buffer_head, -SLAB_RECLAIM_ACCOUNT |SLAB_PANIC |SLAB_MEM_SPREAD ); + bh_cachep =KMEM_CACHE_OPS (buffer_head, +SLAB_RECLAIM_ACCOUNT |SLAB_PANIC |SLAB_MEM_SPREAD , + &buffer_head_kmem_cache_ops);/* * Limit the bh occupancy to 10% of ZONE_NORMAL
PATCH 8/12 Slab defragmentation: Support generic defragmentation for inode slab caches
In reply to Christoph LameterThis implements the ability to remove inodes in a particular slab from the inode cache. In order to remove an inode we may have to write out the pages of an inode, the inode itself and remove the dentries referring to the node.
Provide generic functionality that can be used by filesystems that have their own inode caches to also tie into the defragmentation functions that are made available here.
--- fs/inode.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++- include/linux/fs.h | 5 ++ 2 files changed, 105 insertions(+), 1 deletion(-)
fs/inode.c +100 -1
@@ -1351,6 +1351,105 @@ static int __init set_ihash_entries(char }__setup ("ihash_entries=" , set_ihash_entries); +static void *get_inodes (struct kmem_cache *s,int nr,void **v) +{ +int i; + +spin_lock (&inode_lock); +for (i =0 ; i < nr; i++) { +struct inode *inode = v[i]; + +if (inode->i_state & (I_FREEING |I_CLEAR |I_WILL_FREE )) + v[i] =NULL ; +else +__iget (inode); + } +spin_unlock (&inode_lock); +return NULL ; +} + +/* +* Function for filesystems that embedd struct inode into their own +* structures. The offset is the offset of the struct inode in the fs inode. +*/ +void *fs_get_inodes (struct kmem_cache *s,int nr,void **v, +unsigned long offset) +{ +int i; + +for (i =0 ; i < nr; i++) + v[i] += offset; + +return get_inodes (s, nr, v); +} +EXPORT_SYMBOL (fs_get_inodes); + +void kick_inodes (struct kmem_cache *s,int nr,void **v,void *private) +{ +struct inode *inode; +int i; +int abort =0 ; +LIST_HEAD (freeable); +struct super_block *sb; + +for (i =0 ; i < nr; i++) { + inode = v[i]; +if (!inode) +continue ; + +if (inode_has_buffers (inode) || inode->i_data.nrpages) { +if (remove_inode_buffers (inode)) +invalidate_mapping_pages (&inode->i_data, +0 ,-1 ); + } + +/* Invalidate children and dentry */ +if (S_ISDIR (inode->i_mode)) { +struct dentry *d =d_find_alias (inode); + +if (d) { +d_invalidate (d); +dput (d); + } + } + +if (inode->i_state &I_DIRTY ) +write_inode_now (inode,1 ); + +d_prune_aliases (inode); + } + +mutex_lock (&iprune_mutex); +for (i =0 ; i < nr; i++) { + inode = v[i]; +if (!inode) +continue ; + + sb = inode->i_sb; +iput (inode); +if (abort || !(sb->s_flags &MS_ACTIVE )) +continue ; + +spin_lock (&inode_lock); + abort = !can_unuse (inode); + +if (!abort) { +list_move (&inode->i_list, &freeable); + inode->i_state |=I_FREEING ; + inodes_stat.nr_unused--; + } +spin_unlock (&inode_lock); + } +dispose_list (&freeable); +mutex_unlock (&iprune_mutex); +} +EXPORT_SYMBOL (kick_inodes); + +static struct kmem_cache_ops inode_kmem_cache_ops = { + .get = get_inodes, + .kick = kick_inodes +}; +/* * Initialize the waitqueues and inode hash table. */
@@ -1389,7 +1488,7 @@ void __init inode_init(unsigned long mem (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC |SLAB_MEM_SPREAD ), init_once, - NULL); + &inode_kmem_cache_ops);register_shrinker (&icache_shrinker);/* Hash may have been set up in inode_init_early */
include/linux/fs.h +5 -0
@@ -1769,6 +1769,11 @@ static inline void insert_inode_hash(str__insert_inode_hash (inode, inode->i_ino); } +/* Helper functions for inode defragmentation support in filesystems */ +extern void kick_inodes (struct kmem_cache *,int ,void **,void *); +extern void *fs_get_inodes (struct kmem_cache *,int nr,void **, +unsigned long offset); +extern struct file *get_empty_filp (void );extern void file_move (struct file *f,struct list_head *list);extern void file_kill (struct file *f);
PATCH 9/12 Slab defragmentation: Support defragmentation for extX filesystem inodes
In reply to Christoph LameterUse the generic API for inodes established earlier to support all extX filesystem.
--- fs/ext2/super.c | 16 ++++++++++++++-- fs/ext3/super.c | 14 +++++++++++++- fs/ext4/super.c | 14 +++++++++++++- 3 files changed, 40 insertions(+), 4 deletions(-)
fs/ext2/super.c +14 -2
@@ -168,14 +168,26 @@ static void init_once(void * foo, structmutex_init (&ei->truncate_mutex);inode_init_once (&ei->vfs_inode); } - + +static void *ext2_get_inodes (struct kmem_cache *s,int nr,void **v) +{ +return fs_get_inodes (s, nr, v, + offsetof(struct ext2_inode_info , vfs_inode)); +} + +static struct kmem_cache_ops ext2_kmem_cache_ops = { + .get = ext2_get_inodes, + .kick = kick_inodes +}; +static int init_inodecache (void ) { ext2_inode_cachep =kmem_cache_create ("ext2_inode_cache" ,sizeof (struct ext2_inode_info ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ), - init_once,NULL ); + init_once, + &ext2_kmem_cache_ops);if (ext2_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/ext3/super.c +13 -1
@@ -484,13 +484,25 @@ static void init_once(void * foo, structinode_init_once (&ei->vfs_inode); } +static void *ext3_get_inodes (struct kmem_cache *s,int nr,void **v) +{ +return fs_get_inodes (s, nr, v, + offsetof(struct ext3_inode_info , vfs_inode)); +} + +static struct kmem_cache_ops ext3_kmem_cache_ops = { + .get = ext3_get_inodes, + .kick = kick_inodes +}; +static int init_inodecache (void ) { ext3_inode_cachep =kmem_cache_create ("ext3_inode_cache" ,sizeof (struct ext3_inode_info ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ), - init_once,NULL ); + init_once, + &ext3_kmem_cache_ops);if (ext3_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
fs/ext4/super.c +13 -1
@@ -544,13 +544,25 @@ static void init_once(void * foo, structinode_init_once (&ei->vfs_inode); } +static void *ext4_get_inodes (struct kmem_cache *s,int nr,void **v) +{ +return fs_get_inodes (s, nr, v, + offsetof(struct ext4_inode_info , vfs_inode)); +} + +static struct kmem_cache_ops ext4_kmem_cache_ops = { + .get = ext4_get_inodes, + .kick = kick_inodes +}; +static int init_inodecache (void ) { ext4_inode_cachep =kmem_cache_create ("ext4_inode_cache" ,sizeof (struct ext4_inode_info ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ), - init_once,NULL ); + init_once, + &ext4_kmem_cache_ops);if (ext4_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
PATCH 10/12 Slab defragmentation: Support inode defragmentation for xfs
In reply to Christoph LameterAdd slab defrag support.
--- fs/xfs/linux-2.6/kmem.h | 5 +++-- fs/xfs/linux-2.6/xfs_buf.c | 2 +- fs/xfs/linux-2.6/xfs_super.c | 13 ++++++++++++- fs/xfs/xfs_vfsops.c | 6 +++--- 4 files changed, 19 insertions(+), 7 deletions(-)
fs/xfs/linux-2.6/kmem.h +3 -2
@@ -79,9 +79,10 @@ kmem_zone_init(int size, char *zone_namestatic inline kmem_zone_t *kmem_zone_init_flags (int size,char *zone_name,unsigned long flags, -void (*construct)(void *,kmem_zone_t *,unsigned long )) +void (*construct)(void *,kmem_zone_t *,unsigned long ), +const struct kmem_cache_ops *ops) { -return kmem_cache_create (zone_name, size,0 , flags, construct,NULL ); +return kmem_cache_create (zone_name, size,0 , flags, construct, ops); } static inline void
fs/xfs/linux-2.6/xfs_buf.c +1 -1
@@ -1834,7 +1834,7 @@ xfs_buf_init(void)#endif xfs_buf_zone =kmem_zone_init_flags (sizeof (xfs_buf_t),"xfs_buf" , -KM_ZONE_HWALIGN ,NULL ); +KM_ZONE_HWALIGN ,NULL ,NULL );if (!xfs_buf_zone) goto out_free_trace_buf;
fs/xfs/linux-2.6/xfs_super.c +12 -1
@@ -355,13 +355,24 @@ xfs_fs_inode_init_once(inode_init_once (vn_to_inode ((bhv_vnode_t *)vnode)); } +static void *xfs_get_inodes (struct kmem_cache *s,int nr,void **v) +{ +return fs_get_inodes (s, nr, v, offsetof(bhv_vnode_t , v_inode)); +}; + +static struct kmem_cache_ops xfs_kmem_cache_ops = { + .get = xfs_get_inodes, + .kick = kick_inodes +}; + STATIC intxfs_init_zones (void ) { xfs_vnode_zone =kmem_zone_init_flags (sizeof (bhv_vnode_t),"xfs_vnode" ,KM_ZONE_HWALIGN |KM_ZONE_RECLAIM |KM_ZONE_SPREAD , - xfs_fs_inode_init_once); + xfs_fs_inode_init_once, + &xfs_kmem_cache_ops);if (!xfs_vnode_zone) goto out;
fs/xfs/xfs_vfsops.c +3 -3
@@ -109,13 +109,13 @@ xfs_init(void) xfs_inode_zone =kmem_zone_init_flags (sizeof (xfs_inode_t),"xfs_inode" ,KM_ZONE_HWALIGN |KM_ZONE_RECLAIM | -KM_ZONE_SPREAD ,NULL ); +KM_ZONE_SPREAD ,NULL ,NULL ); xfs_ili_zone =kmem_zone_init_flags (sizeof (xfs_inode_log_item_t),"xfs_ili" , -KM_ZONE_SPREAD ,NULL ); +KM_ZONE_SPREAD ,NULL ,NULL ); xfs_chashlist_zone =kmem_zone_init_flags (sizeof (xfs_chashlist_t),"xfs_chashlist" , -KM_ZONE_SPREAD ,NULL ); +KM_ZONE_SPREAD ,NULL ,NULL );/* * Allocate global trace buffers.
PATCH 11/12 Slab defragmentation: Support reiserfs inode defragmentation
In reply to Christoph LameterAdd inode defrag support
--- fs/reiserfs/super.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
fs/reiserfs/super.c +13 -1
@@ -520,6 +520,17 @@ static void init_once(void *foo, struct#endif } +static void *reiserfs_get_inodes (struct kmem_cache *s,int nr,void **v) +{ +return fs_get_inodes (s, nr, v, + offsetof(struct reiserfs_inode_info , vfs_inode)); +} + +struct kmem_cache_ops reiserfs_kmem_cache_ops = { + .get = reiserfs_get_inodes, + .kick = kick_inodes +}; +static int init_inodecache (void ) { reiserfs_inode_cachep = kmem_cache_create("reiser_inode_cache",
@@ -527,7 +538,8 @@ static int init_inodecache(void) reiserfs_inode_info),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_MEM_SPREAD ), - init_once, NULL); + init_once, + &reiserfs_kmem_cache_ops);if (reiserfs_inode_cachep ==NULL )return -ENOMEM ;return 0 ;
PATCH 12/12 Slab defragmentation: kmem_cache_vacate for antifrag / memory compaction
In reply to Christoph Lameter[only for review, untested, waiting for Mel to get around to use this to improve memory compaction or antifragmentation]
Special function kmem_cache_vacate() to push out the objects in a
specified slab. In order to make that work we will have to handle
slab page allocations in such a way that we can determine if a slab is valid whenever we access it regardless of its time in life.
A valid slab that can be freed has PageSlab(page) and page->inuse > 0 set. So we need to make sure in allocate_slab that page->inuse is zero before PageSlab is set otherwise kmem_cache_vacate may operate on a slab that has not been properly setup yet.
--- include/linux/slab.h | 1 mm/slab.c | 9 ++++ mm/slob.c | 9 ++++ mm/slub.c | 105 +++++++++++++++++++++++++++++++++++++++++++++++---- 4 files changed, 117 insertions(+), 7 deletions(-)
include/linux/slab.h +1 -0
@@ -97,6 +97,7 @@ unsigned int kmem_cache_size(struct kmemconst char *kmem_cache_name (struct kmem_cache *);int kmem_ptr_validate (struct kmem_cache *cachep,const void *ptr);int kmem_cache_defrag (int node); +int kmem_cache_vacate (struct page *);/* * Please use this macro to create slab caches. Simply specify the
mm/slab.c +9 -0
@@ -2523,6 +2523,15 @@ int kmem_cache_defrag(int percent, int nreturn 0 ; } +/* +* SLAB does not support slab defragmentation +*/ +int kmem_cache_vacate (struct page *page) +{ +return 0 ; +} +EXPORT_SYMBOL (kmem_cache_vacate); +/** * kmem_cache_destroy - delete a cache * @cachep: the cache to destroy
mm/slob.c +9 -0
@@ -596,6 +596,15 @@ int kmem_cache_defrag(int percentage, inreturn 0 ; } +/* +* SLOB does not support slab defragmentation +*/ +int kmem_cache_vacate (struct page *page) +{ +return 0 ; +} +EXPORT_SYMBOL (kmem_cache_vacate); +int kmem_ptr_validate (struct kmem_cache *a,const void *b) {return 0 ;
mm/slub.c +98 -7
131 lines not on this page · raw message