Skip to contentSkip to conversation

strange behavior creating and deleting files

DISCUSSION Mpourtounis Dimitris 2 participants 3 messages Sep 24, 2004 – Sep 25, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.

running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:

files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )

and sometime in the morning 25000 MB free RAM, and my system running too slow

I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.

It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.

I am running linux 2.4.26 on an x86 platform (gcc 3.2.3 uclib 0.9.20)

Simple sh file:
i=0
while [ 1 ] do
echo "dont allocate more memory please" > $i
rm $i
let i=$i+1
done

Any clue???

#1 lore raw
Quoted 22 lines · Mpourtounis Dimitris

there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.

running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:

files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )

Quoted 8 lines

and sometime in the morning 25000 MB free RAM, and my system running too slow

I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.

It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.

I'll admit that I don't _fully_ understand it either. Another user noticed related behaviour a couple of months ago, below is the patch which I hope you find fixes yours too. An equivalent fix (to fs/libfs.c) went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix. If it works well for you, let me know and I'll send it in for 2.4.28...

A tmpfs user reported increasingly slow directory reads when repeatedly creating and unlinking in a mkstemp-like way. The negative dentries accumulate alarmingly (until memory pressure finally frees them), and are just a hindrance to any in-memory filesystem. shmem_lookup set d_op to arrange for negative dentries to be deleted immediately.

(But I failed to discover how it is that on-disk filesystems seem to keep their negative dentries within manageable bounds: this effect was gross with tmpfs or ramfs, but no problem at all with extN or reiser.)

mm/shmem.c +15 -1
@@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo
 }
 
 /*
+ * Retaining negative dentries for an in-memory filesystem just wastes
+ * memory and lookup time: arrange for them to be deleted immediately.
+ */
+static int shmem_delete_dentry(struct dentry *dentry)
+{
+	return 1;
+}
+
+/*
  * Lookup the data. This is trivial - if the dentry didn't already
- * exist, we know it is negative.
+ * exist, we know it is negative.  Set d_op to delete negative dentries.
  */
 static struct dentry *shmem_lookup(struct inode *dir, struct dentry *dentry)
 {
+	static struct dentry_operations shmem_dentry_operations = {
+		.d_delete = shmem_delete_dentry,
+	};
+
 	if (dentry->d_name.len > NAME_MAX)
 		return ERR_PTR(-ENAMETOOLONG);
+	dentry->d_op = &shmem_dentry_operations;
 	d_add(dentry, NULL);
 	return NULL;
 }
#2 lore raw
Quoted 32 lines · Hugh Dickins
Quoted 22 lines · Mpourtounis Dimitris

there seems to be a strange behaviour in the way my system creates and deletes files, as long as memory allocation is concerned.

running a simple script that continuously creates and deletes files on tmpfs filesystem, a got the following results:

files created free memory on system
------------- ---------------------
0 48180
+6000 47936
+6000 47372
+6000 47372
+6000 47936
+6000 47936
+6000 47936
+6000 47936 (seems stable)
+9000 46976 (what on earth?)
+30000 45084
+80000 45084 (again stable)
+70000 39156 (not again...:( )

Quoted 8 lines

and sometime in the morning 25000 MB free RAM, and my system running too slow

I am sure these are a lot a files and under normal conditions, there will never be made and deleted so many.

It is that misbehavior of being stable for a long time and then again allocating memory that concerns me.

Quoted 7 lines

I'll admit that I don't _fully_ understand it either. Another user noticed related behaviour a couple of months ago, below is the patch which I hope you find fixes yours too. An equivalent fix (to fs/libfs.c) went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix. If it works well for you, let me know and I'll send it in for 2.4.28...

I seems to work fine. Having created-deleted about 50000 files there is no leak. Hope it continues that way.
Great job, thanks !

DB

Quoted 43 lines

A tmpfs user reported increasingly slow directory reads when repeatedly creating and unlinking in a mkstemp-like way. The negative dentries accumulate alarmingly (until memory pressure finally frees them), and are just a hindrance to any in-memory filesystem. shmem_lookup set d_op to arrange for negative dentries to be deleted immediately.

(But I failed to discover how it is that on-disk filesystems seem to keep their negative dentries within manageable bounds: this effect was gross with tmpfs or ramfs, but no problem at all with extN or reiser.)

mm/shmem.c +15 -1
@@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo
 }
 
 /*
+ * Retaining negative dentries for an in-memory filesystem just wastes
+ * memory and lookup time: arrange for them to be deleted immediately.
+ */
+static int shmem_delete_dentry(struct dentry *dentry)
+{
+	return 1;
+}
+
+/*
  * Lookup the data. This is trivial - if the dentry didn't already
- * exist, we know it is negative.
+ * exist, we know it is negative.  Set d_op to delete negative dentries.
  */
 static struct dentry *shmem_lookup(struct inode *dir, struct dentry *dentry)
 {
+	static struct dentry_operations shmem_dentry_operations = {
+		.d_delete = shmem_delete_dentry,
+	};
+
 	if (dentry->d_name.len > NAME_MAX)
 		return ERR_PTR(-ENAMETOOLONG);
+	dentry->d_op = &shmem_dentry_operations;
 	d_add(dentry, NULL);
 	return NULL;
 }