Is there a way to turn file caching off ?

Hi,

I'm running on a machine with 2GB of memory and dual PIII 550MHZ. Just after boot with "nothing else running":
I run a program that almost like dd if=/dev/null of=/local/test count=10000 bs=1000000
Except that there is a thread for reading and a thread for writing. The program itself almost doesn't take any CPU.
What's going on is top showing:
First cache grows to the size of RAM (2GB) with transfer rate
slowing down as the cache grows.
Then the transfer rates drops a lot (2 to 3 time slower than the
drive capacity) and there is a very high CPU usage of system time (more than a CPU) used by bdflush and kswapd (and some others like kupdated).

Of course my real application doesn't go from /dev/zero to file but it still only does sequential access, and it seems that I pay a high price for the file caching when I'm not using it at all.

Is there a way to turn file caching off, or at least limit its size ?

Thanks,

Laurent Chavet

What benefit do you think you would get by limiting its size? All that
would do is ensure you hit the cache thrashing point sooner.

DS

A similar phenomenon happens when you simply copy a file - file A is read into the cache and file B is written to the cache, until the memory runs out. Then both start to flush at the same time, creating a horrible performance hit (especially if A and B are on the same disk :)

I don't know a way to fix this except having the kernel correctly identify the access pattern and optimize for it (i.e. if it recognizes that cache pages are flushed in order to make room for more pages from the same inode, then it's probably a suboptimal caching pattern and instead it should probably increase the readahead and flush bigger chunks of pages at the same time). I don't think anything can be done to the writing queue (except maybe make the kernel understand that seek-time is more expensive than transfer-time, so it does not schedule the read/writeing each odd page..)

I'm still using 2.4.0 though so maybe this behaviour has been fixed to the better in later kernels..

As a sidenote, try the same thing on an WinNT box and watch it die :) Like unpacking a 1 GB file on a machine with 128 MB ram.. after it has unpacked the first 100 MB's or so, performance drops to 1% or something..

-BW

in this example only file B needs uses IO when being flushed; A's buffers aren't dirty, so they're just 'forgotten'

bdflush and 2.4's equivalent are tunable; if all you do is this copying then optimise these parameters for this task.

currently all the kernel's heuristics are feed-back control loops. what you are asking for is a feed-forward system: a way for the application to tell kernel "I'm only reading this once, so after I'm done, throw it out straight away"
and "I'm only writing this data, so after I'm done, start writing it out and then forget it"

perhaps you could try mounting the destinatinon 'sync' with dd at least you could specify block size,
there is also raw devices (no cache) mmaped files with madvise, fdatasync()... lots of ways.

there is the elevator seek mechanism...

Isn't this due to the journaling filesystem? All that seeking sure slows things down...

Try this (my example I've 2GB of ram)

turn all your swap off

dd about 15% of the size of your RAM:
dd if=/dev/zero of=/local/test count=300 bs=1000000

Run this program with SIZE about 95% of your RAM:

#include <stdlib.h>
#include <unistd.h>
#include <assert.h>

#define SIZE (1900 * 1024 * 1024)
int main()
{
int i;
char *p = malloc(SIZE);
assert (p != NULL);
for (i = 0; i < SIZE; i++)
p[i] = 1;
printf ("done %p\n", p);

while (1)
{
sleep (60);
}
return 0;
}

Watch top: when this program needs the memory that kswapd keep in cache they go both at 100% cpu (on SMP) but still the size of the program only grows at about 100KB/s, why is kswapd releasing it so slowly and taking so much CPU ?

Laurent Chavet

Because kswapd still has to scan all the (unfreeable) memory of the big process to determine it isn't freeable.

At the moment kswapd is really stupid about these corner-case situations, please bear with us while we fix it ...

... and don't forget to fill in a bugzilla item on the Linux-MM bugzilla ;)

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

This is not the only badly performing case actually. When one of ones swap partitions gets close to full, kswapd basically sits endlessly in get_swap_page() due to all the broken linear scan algorithms. I've tried to fix some of this with the patch below.

This may not be the case Laurent is seeing but it is a problem that needs fixing.

@@ -43,8 +43,22 @@
 
 #define SWAP_CLUSTER_MAX 32
 
-#define SWAP_MAP_MAX	0x7fff
-#define SWAP_MAP_BAD	0x8000
+#define SWAPFILE_CLUSTER 256
+
+struct swap_cluster_struct {
+	struct list_head	list;
+	int			nr_free;	/* 0 --> SWAPFILE_CLUSTER */
+	unsigned int		start_offset;
+};
+
+#define SWAP_MAP_MAX	0x7fffffff
+#define SWAP_MAP_BAD	0x80000000
+
+struct swap_map_struct {
+	struct list_head	list;
+	unsigned int		offset;
+	unsigned int		count;
+};
 
 struct swap_info_struct {
 	unsigned int flags;

@@ -52,11 +66,15 @@
 	spinlock_t sdev_lock;
 	struct dentry * swap_file;
 	struct vfsmount *swap_vfsmnt;
-	unsigned short * swap_map;
-	unsigned int lowest_bit;
-	unsigned int highest_bit;
-	unsigned int cluster_next;
-	unsigned int cluster_nr;
+	struct swap_map_struct * swap_maps;
+	struct list_head swap_map_free_list;
+
+	struct swap_cluster_struct * swap_clusters;
+	struct list_head swap_cluster_free_list;
+
+	struct swap_cluster_struct * curr_cluster;
+	unsigned int cluster_offset_next;
+
 	int prio;			/* swap priority */
 	int pages;
 	unsigned long max;

This is hard to get right. Sure - your unpack/copy program read once and
writes once. But the stuff might be used shortly thereafter by another process. For example: I unpack a kernel tarball. tar knows it writes only once and might not need more than 5M to do this as efficient as possible with my disks. A lot of other cache could be saved, fewer things swapped out.
But then I compile it. Todays system ensures that lots of the source is in memory already. Limiting the caching to what tar needed however will force the source to be read from disk once during the compile - not what I want at all.

A program may know its own access pattern, but it don't usually know future access patterns. Well, backing up the entire fs could benefit from a something like this, you probably won't need the backup again soon. But this is hard to know in many other cases.

Helge Hafting

They why would you tell tar not to use cache? If you know what's happening next you need to tell the system (feed-forward), not have it try to read your mind. I'm assuming your modified tar would have an option switch to cause this behaviour, not be hard coded...

Yes, so a script that does the above wouldn't enable no cache mode for written files. The program doesn't know, but the encompasing script (or person at console) does.

tar --please-leave-this-in-cache-pretty-please ?

John Lenton ([email protected]) -- Random fortune:
Si los bugs te abruman, cierra tu Windows.