clearing filesystem cache for I/O benchmarks

How can I purge all of the kernel's filesystem caches, so I can trust that my I/O (read) requests I'm trying to benchmark bypass the kernel filesystem cache?

Unfortunately, I cannot:

1) reboot the system

2) re-mount the filesystem where the reads are occuring

So I propose that I am left with the following options:

3) Reading through a file sufficiently larger than the RAM installed on the system? e.g. read through a 10GB file on a machine with 8GB of RAM

4) Since I can create the files fresh every time, I would write() them out using O_DIRECT flag to open(), then the immediately following read of that file would be guaranteed to avoid pulling it from cache.

So, can someone evaluate whether how whether options 3 and 4 would work, or offer other suggestons? And I wouldn't object if the issue of clearing disk and controller cache entered into the discussion (I'm thinking #3 would do a better job at clearing disk/controller caches).

In case it is relevant, here are the two relevant kernel versions I'm using, both under the distribution "Red Hat Enterprise Linux AS release 3 (Taroon)":

    Linux xio11 2.6.6 #2 SMP Wed Jun 9 10:37:24 EDT 2004 i686 i686 i386 GNU/Linux
    
    Linux xio06 2.4.21-9.ELhugemem #1 SMP Tue Apr 27 13:52:32 EDT 2004 i686 i686 i386 GNU/Linux

Thank you,

Benjamin Rutt

does "ioctl(fd, BLKFLSBUF,0)" suffice?

--cw

Take a look at the code in hdparm tool that handles the '-f' option.

Basically calling ioctl(fd, BLKFLSBUF, o) where fd is a file descriptor opened on the block device on which your filesystem resides should be enough to clear the cache.

Regards,

Tim

Tim Wright <[email protected]>
Splhi

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

Some time ago I was looking for that, too, and found "cfree". Have a look at <http://gizmolabs.org/~andrew/andrewweb/project.php?pid=3&tab=0>

It's a small utility and kernel module for linux-2.4 written by Andrew de los Reyes. It allows to clear portions of the buffer cache (e.g. for a complete sub-directory). I haven't analyzed it so I can't say if it does things correctly, though.

HTH

- - andreas

- --

Andreas Haumer                     | mailto:[email protected]
*x Software + Systeme              | http://www.xss.co.at/
Karmarschgasse 51/2/20             | Tel: +43-1-6060114-0
A-1100 Vienna, Austria             | Fax: +43-1-6060114-71

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBA2s2xJmyeGcXPhERAuaTAKCaxNRjhbzf3G5uL1lsXYg41eF+jQCeP808 DNcut1YDptMCNsvAeXrt+d8=
=xPh9
-----END PGP SIGNATURE-----

Thanks, that looks pretty useful, at least to force the I/O to make it outside the kernel. I'm still getting cache hits for some read tests though, no doubt due to cache near the physical disks and/or controllers. Correct me if I'm wrong, but this ioctl doesn't appear to go out and tell disks to clear their caches.

I think I'll use the BLKFLSBUF in any case in my tests though, as it doesn't seem to take very long to execute. It can't hurt, and should complement the act of reading through a large dummy file, which should take care of the disk/controller caches.

Benjamin Rutt

This might be due to read ahead... how do you check the cache hits, what read patterns do you have?

Greetings
Bernd

eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Either delete the benchmark test files or, in 2.6, use fsync+posix_fadvise(POSIX_FADV_DONTNEED);

There must be cache hits since my poor old IDE disk from 1998 can only perform at around 13 MB/sec for sustained sequential reads. The performance I was getting was 150 MB/sec for sustained sequential reads, which led me to think it was cache hits for certain, since there is no way my disk can be that fast.

The test setup is simply this:

1) create a target file for benchmarking, say 32 Megabytes (my system RAM is 256MB, enough to cache all of that 32MB file)
2) run hdparm -f <device> to clear cache
3) read the target file from beginning to end into a user memory buffer (e.g. 32k in size), ignoring the read data (i.e. there are no data operations I make on the read data, this is a pure I/O test)

Because of #3, I'm not doing anything to the read data (I'm just overwriting it with the next read) so I wouldn't imagine there is much time from one read to the next to leverage any readahead.

Benjamin Rutt

I'm not sure I follow. If I delete the benchmark files, I'll only need to create them again later in order to do a read test, and I'll have the same problem then, of how to eliminate the just-written-data from cache. Unless you're suggesting I write using some special mode that won't enter the written data into cache? (e.g. O_DIRECT?)

Thanks for the reference, I wasn't aware of that one. We are running some 2.4 kernels in our storage cluster unfortunately so that won't be usable for us everywhere. I take it POSIX_FADV_DONTNEED is ignored under 2.4.

A related question...if no posix_fadvise() advice has been given, does reading sequentially every byte of an 8GB file on a machine with <= 8GB of RAM guarantee that any page cache data that existed on the machine prior to the start of the 8GB read is now gone?

Benjamin Rutt

(Please don't remove people from the email recipient list when doing kernel work.)

OK.

posix_fadvise() will return -ENOSYS under 2.4.

However... If you write any amount of data to a file with O_DIRECT, that will, as a side-effect, remove _all_ of that file's pagecache. In 2.4 as well as 2.6. So you could scrub the pagecache by reading the first 4k then writing it back with O_DIRECT.

However O_DIRECT is supported on very few filesystems in 2.4. ext2 and reiserfs have it.

XFS in 2.4 has O_DIRECT, I think, but I don't know if the invalidation side-effect works on XFS.

It's not guaranteed that this will work - if the pages which you're trying to evict were accessed multiple times then it may take more page replacement to reliably shoot them down. But writing a 2xmemory file and then deleting it will be a reasonably effective way of evicting most of the other pagecache.

when benchmarking, please be careful that you don't end up benchmarking umount/mount, or sync, or..... it can be remarkably hard to avoid such mistakes.....

I tend to try to use large enough filesets that small things like cache flush happenstance or bitmap loading overhead do not sway the benchmark.

Rebooting tends to work for resetting the OS thoroughly, though I would be curious to hear comments on whether one ought to power down the disk drive so that its cache flushes......;-)

Sorry, I'm reading via gmane and my newsreader doesn't make it straightforward to do so. But I'll do it manually for you.

Thanks, that does work for ext3, very well. It's obvious that it clears kernel page cache and not controller/disk cache.

OK thanks, I'll take on good faith that this is the best scheme in general. I was actually doing a somewhat different approach, reading through a 2x memory "dummy" file before accessing the real file, but based on your advice, I'll instead just create a 2x "dummy" file, fsync it, and then delete it.

Thanks for the tips,

Benjamin Rutt

I agree, I've made some blunders like that in the past. However for write tests, we are including fsync() time, once, at the end of a file write, since I feel it's unfair to trim that time. Not including fsync() time would only test the ability of the various parts of the I/O systems to do write buffering. It's easy to do lots of write buffering, if you buy enough memory. Forcing the disks to write is the only fair way to compare writes between I/O systems.

Sounds familiar...we cycle among file sizes at every power of 2 point from 8MB to 64GB. So by the time we access the 64GB file, all the previous accesses for 8M..32GB will probably have pushed all of the 64GB file out of cache.

With the mass storage environment I'm working in, you'd need to power down the whole storage cluster, then remove that batteries that back the controller cache...yes, clearing kernel cache is often just the beginning. :)

Benjamin Rutt

fsync performance gives you different performance. Better to write more stuff to flush the cache.

It isn't fair. fsync is a different code path, and may be less efficient. Or more, depending on the fs. reiser4 is currently not well optimized for fsync, maybe next year I will change that but not this week....

Benchmarking well is hard.....

Hans

I haven't been paying attention, and I don't know if anyone's already suggested this, but going on the title, have you considered running the same benchmark more than once and just throwing away the first result?

I'm trying to understand how that would work. Let's take an example of a 64GB file that I'm writing out from scratch. I start a timer before writing. With my fsync() way of testing, I expect to stop the timer the moment last byte has been written and fsync() has been called.

I gather you're saying that continuing writing past the 64GB mark, causing LRU expiration of the last bytes of the 64GB bytes from write buffers is a more fair way to test, versus just calling fsync() once at the end. I'm happy to write my benchmarks this way too, except I need to know two configuration values now:

1) when to stop the timer?
2) how much more to write past 64GB?

I think we agree that forcing the disks to write all of the data before the timer stops is a fair way to compare between filesystems. Otherwise we're "almost" measuring disk throughput, except for what has been write-buffered...a real gray area. But I think you're pointing out that the results could be different depending on whether the fsync() method or your "write past the intented amount" method for flushing is used. I'd be happy to run these benchmarks both ways, as long as I knew how. If you can help me answer my above questions, I'll run them both ways.

Thanks,

Benjamin Rutt

I was gathering from upthread comments that data blocks that are read more than once will be given a priority to be retained in cache. So I think reading the same data twice could lead to unwanted cache hits. And besides, some of our file sizes are quite small (e.g. 8MB) such that reading through them the second time would almost guarantee cache hits. I see your point, though, for reading through a 64GB file on a system with 8GB of RAM. If such a system would retain in cache anything except the last ~8GB, I'd be very surprised.

Based on comments from Andrew Morton, I'm going to take the following approach to clear cache for read tests:

1) figure out the available RAM on the test system
2) write out a throwaway file twice that big, and fsync() it
3) delete that file

I gather this is optimistically the best way, that would work for all filesystem types.

As far as clearing disk/controller cache, I have a plan of (after the above has been done) reading through a 2GB "dummy" file that I create once before running the test battery. The 2GB figure comes from the fact that we have controllers with a 1GB cache. Plus, there are around 36 disks on the backend, all raided together into one raid device. So each disk brings 8MB of cache that we have to worry about as well. If it is totally obvious that Andrew Morton's above recipe will clear the disk/controller caches as well, the please point that out, but it isn't obvious to me.

Benjamin Rutt

Probably the best you can do is write enough in the course of the test that fsync at the end (or data remaining in cache) is insignificant noise. Benchmarks that make fsync or the cache significant unintentionally are common and bad.

I assure you, our benchmarks will avoid these common pitfalls.

Benjamin Rutt

Yep, it does.

cheers.

Nathan