Testing write master and multiple read clients on NFS share

Hi,

in my company we have some sort of database that operates on bit lists read directly from the files on the hard disk. While this is ok if you have an SSD in your server it gets tricky to keep performance up when using an NFS share.

While I was trying to assess different solutions I stumbled upon fio and was happy to see that it can also run in client-server-mode. So I was trying to emulate the following scenario:

- 1 node that takes care of updating the bit lists in the db files, nothing else
- 5 nodes that only read from those files and respond to queries, hopefully leveraging the local filesystem or NFS client cache

I read through the HOWTO and tried to pick the options that should simulate the scenario as best as possible. This is what I came up with:

[global]
ioengine=sync
iodepth=1
size=100%
io_size=4g
filesize=4g
blocksize=64k
direct=0
numjobs=4
directory=/mnt/nfs
filename_format=index.$filenum
fadvise_hint=0
lockfile=readwrite
randrepeat=1
nrfiles=8
openfiles=4
runtime=120
ramp_time=10
blocksize=64k
file_service_type=random:8
overwrite=1
fsync_on_close=1
random_distribution=zipf:0.6
norandommap
file_append=0
rate=1m

[reader]
rw=randread

[writer]
rw=randrw
rwmixwrite=80

Is this a viable approach or is this scenario not really testable with fio?

Quoted 49 lines

Hi,

in my company we have some sort of database that operates on bit lists read directly from the files on the hard disk. While this is ok if you have an SSD in your server it gets tricky to keep performance up when using an NFS share.

While I was trying to assess different solutions I stumbled upon fio and was happy to see that it can also run in client-server-mode. So I was trying to emulate the following scenario:

- 1 node that takes care of updating the bit lists in the db files, nothing else
- 5 nodes that only read from those files and respond to queries, hopefully leveraging the local filesystem or NFS client cache

I read through the HOWTO and tried to pick the options that should simulate the scenario as best as possible. This is what I came up with:

[global]
ioengine=sync
iodepth=1
size=100%
io_size=4g
filesize=4g
blocksize=64k
direct=0
numjobs=4
directory=/mnt/nfs
filename_format=index.$filenum
fadvise_hint=0
lockfile=readwrite
randrepeat=1
nrfiles=8
openfiles=4
runtime=120
ramp_time=10
blocksize=64k
file_service_type=random:8
overwrite=1
fsync_on_close=1
random_distribution=zipf:0.6
norandommap
file_append=0
rate=1m

[reader]
rw=randread

Quoted 6 lines

[writer]
rw=randrw
rwmixwrite=80

Is this a viable approach or is this scenario not really testable with fio?

I could be wrong but won't this make 4 readers and 4 writers. If that's not what you wanted perhaps numjobs could be moved to the reader entry only?

Thanks for the hint. I also found another mistake in limiting the reads and writes where I only wanted to limit the writes and have the reads as fast as possible. But in general this is a viable approach to my problem?

Quoted 3 lines

Thanks for the hint. I also found another mistake in limiting the reads and writes where I only wanted to limit the writes and have the reads as fast as possible. But in general this is a viable approach to my problem?

I can't say for sure whether it models your workload (your example made use of options I had to look up!) but it does look like an interesting approach...

Yeah, well, I needed a process that writes to the same files the others are reading from to somehow simulate expiring local caches (and to see, if NFS handles that the right way). And yes, it took me quite some time to dig through most of the options to tailor the tests to my needs

But I'm also glad to hear, that it doesn't look completely wrong for a first try :-)

Am 20.05.2015 um 11:03 schrieb Sitsofe Wheeler:

Quoted 13 lines · Michael Weinrich

Quoted 6 lines

Thanks for the hint. I also found another mistake in limiting the reads and
writes where I only wanted to limit the writes and have the reads as fast as
possible. But in general this is a viable approach to my problem?

I can't say for sure whether it models your workload (your example made use of options I had to look up!) but it does look like an interesting approach...

Yeah, well, I needed a process that writes to the same files the others are reading from to somehow simulate expiring local caches (and to see, if NFS handles that the right way). And yes, it took me quite some time to dig through most of the options to tailor the tests to my needs

But I'm also glad to hear, that it doesn't look completely wrong for a first try :-)

Quoted 7 lines

Yeah, well, I needed a process that writes to the same files the others are reading from to somehow simulate expiring local caches (and to see, if NFS handles that the right way). And yes, it took me quite some time to dig through most of the options to tailor the tests to my needs

But I'm also glad to hear, that it doesn't look completely wrong for a first try :-)

I think it looks very reasonable. I'd like to include your job file in the examples/ directory, if that's OK with you.

In general, I'd love to have a richer repository of job files included (or available, at least). It's much easier for new users to find something roughly like what they want and then modify it, than it is to start from scratch.