strange behavior creating and deleting files
Subsystems (2)
Hi all,
there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.
running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:
files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )
and sometime in the morning 25000 MB free RAM, and my system running too slow
I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.
It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.
I am running linux 2.4.26 on an x86 platform (gcc 3.2.3 uclib 0.9.20)
Simple sh file:
i=0
while [ 1 ] do
echo "dont allocate more memory please" > $i
rm $i
let i=$i+1
done
Any clue???
Quoted 22 lines · Mpourtounis Dimitris
there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.
running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:
files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )
Quoted 8 lines
and sometime in the morning 25000 MB free RAM, and my system running too slow
I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.
It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.
I'll admit that I don't _fully_ understand it either. Another user noticed related behaviour a couple of months ago, below is the patch which I hope you find fixes yours too. An equivalent fix (to fs/libfs.c) went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix. If it works well for you, let me know and I'll send it in for 2.4.28...
A tmpfs user reported increasingly slow directory reads when repeatedly creating and unlinking in a mkstemp-like way. The negative dentries accumulate alarmingly (until memory pressure finally frees them), and are just a hindrance to any in-memory filesystem. shmem_lookup set d_op to arrange for negative dentries to be deleted immediately.
(But I failed to discover how it is that on-disk filesystems seem to keep their negative dentries within manageable bounds: this effect was gross with tmpfs or ramfs, but no problem at all with extN or reiser.)
mm/shmem.c +15 -1
@@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo }/* +* Retaining negative dentries for an in-memory filesystem just wastes +* memory and lookup time: arrange for them to be deleted immediately. +*/ +static int shmem_delete_dentry (struct dentry *dentry) +{ +return 1 ; +} + +/* * Lookup the data. This is trivial - if the dentry didn't already -* exist, we know it is negative. +* exist, we know it is negative. Set d_op to delete negative dentries. */ static struct dentry *shmem_lookup (struct inode *dir,struct dentry *dentry) { +static struct dentry_operations shmem_dentry_operations = { + .d_delete = shmem_delete_dentry, + }; +if (dentry->d_name.len >NAME_MAX )return ERR_PTR (-ENAMETOOLONG ); + dentry->d_op = &shmem_dentry_operations;d_add (dentry,NULL );return NULL ; }
Quoted 32 lines · Hugh Dickins
Quoted 22 lines · Mpourtounis Dimitris
there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.
running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:
files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )Quoted 8 lines
and sometime in the morning 25000 MB free RAM, and my system running too slow
I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.
It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.
Quoted 7 lines
I'll admit that I don't _fully_ understand it either. Another user noticed related behaviour a couple of months ago, below is the patch which I hope you find fixes yours too. An equivalent fix (to
fs/libfs.c) went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix. If it works well for you, let me know and I'll send it in for 2.4.28...
I seems to work fine. Having created-deleted about 50000 files there is no leak. Hope it continues that way.
Great job, thanks !
DB
Quoted 43 lines
A tmpfs user reported increasingly slow directory reads when repeatedly creating and unlinking in a mkstemp-like way. The negative dentries accumulate alarmingly (until memory pressure finally frees them), and are just a hindrance to any in-memory filesystem.
shmem_lookupsetd_opto arrange for negative dentries to be deleted immediately.(But I failed to discover how it is that on-disk filesystems seem to keep their negative dentries within manageable bounds: this effect was gross with tmpfs or ramfs, but no problem at all with extN or reiser.)
Signed-off-by Hugh Dickins <[email protected]>mm/shmem.c +15
-1@@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo }/* +* Retaining negative dentries for an in-memory filesystem just wastes +* memory and lookup time: arrange for them to be deleted immediately. +*/ +static int shmem_delete_dentry (struct dentry *dentry) +{ +return 1 ; +} + +/* * Lookup the data. This is trivial - if the dentry didn't already -* exist, we know it is negative. +* exist, we know it is negative. Set d_op to delete negative dentries. */ static struct dentry *shmem_lookup (struct inode *dir,struct dentry *dentry) { +static struct dentry_operations shmem_dentry_operations = { + .d_delete = shmem_delete_dentry, + }; +if (dentry->d_name.len >NAME_MAX )return ERR_PTR (-ENAMETOOLONG ); + dentry->d_op = &shmem_dentry_operations;d_add (dentry,NULL );return NULL ; }