correct file size to use

Hi list

I was wondering:
When i start
fio --name=/srv/mail/test --size=10M --direct=1 --runtime=60 --time_based --group_reporting --refill_buffer --rw=read --numjobs=1

I get as result:
read : io=1177.4MB, bw=20093KB/s, iops=5023, runt= 60000msec

But when I do a
fio --name=/srv/mail/test --size=10G --direct=1 --runtime=60 --time_based --group_reporting --refill_buffer --rw=read --numjobs=1

The result is:
read : io=976692KB, bw=16278KB/s, iops=4069, runt= 60001msec (decrease of 20%, in some scenarios it's even worse with more than 30%)

So how does the file size influence the speed (iops, bandwith)? My understanding was that it is not important from which filesize we are reading from - important is the block size and if the filesize is not big enough, fio just restarts to read the file from the beginning. How can that be explained? And which sizes should be used to get reliable results?

Thanks
Josef

From fio/HOWTO:

time_based If set, fio will run for the duration of the runtime
specified even if the file(s) are completely read or written. It will simply loop over the same workload as many times as the runtime allows.

The same 10M file will be read many times(118), so most of times it is read from page cache instead of the file, and the result can't be accurate enough.

Thanks,

Ming Lei

Tested on my system and saw no difference in performance. Direct io will bypass the page cache so the only thing that comes to mind would be an on-device cache, though, I would think the difference in performance would be more dramatic if that were the case.

[sorry for the lack of quoting, OWA doesn't do it nicely.]

Which OS & file system? If the storage target is an NFS mount, you'll still have the server cache. Also, IIRC solaris doesn't honor direct=1 anyway. If zfs is involved.... there's no escaping it's optimizations :).

Please give us more info about the setup.

z!

On Mon, Mar 17, 2014 at 7:16 AM, Matthew Eaton

Looks I miss the '--direct=1' parameter, :-(

Another possible cause might be related with storage for the big file. If the file is stored in lot of fragments of the disk, accessing the whole file might become slower than just the first 10M, but not sure if there is so big difference(30%).

Thanks,

Ming Lei

Am 17.03.2014 03:55, schrieb Ming Lei:

will bypass the page cache so the only thing that comes to mind would be an on-device cache, though, I would think the difference in performance would be more dramatic if that were the case.

This could be possible - I'm not so sure how to check that...
My setup is the following:
RHEL6.5 as virtual machine via VMWare - the storage is a Dell Equallogic Storage connected via ISCSI to the host. These tests are not made via NFS. I just wanted to make some tests before taking it all in production.
Thank you all for your answers!
Josef

If you want raw upper maximum numbers try using direct=1 ioengine=libaio with an appropriate iodepth against the openiscsi block device. Bear in mind things like iodepth and whether coalescing can take place can make a huge difference...

Sitsofe | http://sucs.org/~sits/

Am 17.03.2014 21:34, schrieb Sitsofe Wheeler:

Please explain, what do you mean with appropriate iodepth? What would you suggest?

Something higher than 1 :). I use 512, since AFAIK no storage array out there will actually queue up that many operations.

z!