Large numbers of devices...

Before I start diving into the code, has anybody else out there had problems with 'fio' not being able to scale well with large numbers of devices (files) being used? I have a system w/ 32-cpus, 256GB RAM, plus 11 dual-ported FC HBAs connected to 44 HP MSA1000 FC controllers. (the 44 MSAs are spread out 4 per FC HBA). I'd like to use 'fio' to gather & produce scaling results, but I seem to run into inconsistencies once I get above using 26 or 27 of the 44 MSAs. I have noticed similar things in the past, but it hasn't been so bothersome. I have a locally crafted tool (much more limited than 'fio') called 'aiod' that /is/ able to scale up past 35 or 36 of the MSAs doing what I /believe/ is something similar. [Once past 35 or 36 devices we run into system issues which reduce scaling opportunities.]

In any event, an example fio job-file can be found at:

http://free.linux.hp.com/~adb/2009-08-17/044_disk_1_parts.txt

The graph showing the noise in the graph for fio can be found at:

http://free.linux.hp.com/~adb/2009-08-17/fio.png

And the "better" graph w/ aiod can be found at:

http://free.linux.hp.com/~adb/2009-08-17/aiod.png

The test uses between 1 and 4 partitions per LUN exported by each MSA (each LUN is crafted from 4 physical devices striped together.) You'll see in the latter graph the continued scaling up through almost 37 devices, and much tighter results after that (even with the tail-off at the end above 40 devices.)

Anyways, if there is something I'm missing in the fio job-file to help it scale better let me know, otherwise I'll go through the aiod code to see if there were any applicable scaling improvements there than can be applied to fio...

Alan D. Brunelle
Hewlett-Packard

Alan, I haven't run with that many files, so it's indeed possible that there's an inherent limitiation in the file selection in fio. It's not my best code, that part... If you have the time and inclination to jump in there and find out what is going on, then that would be great!

Jens Axboe

OK, will do - I just wanted to check that the fio job-file looked OK.

Alan

It looks fine, in fact if that is your setup, then you are not using the multi file stuff. Your job files basically creates a process per device, which should work fine as-is.

You can try and play with the iodepth batching control, iodepth_batch and iodepth_batch_complete. They both default to 1, meaning that it'll submit and complete 1 command at the time.

Jens Axboe

Setting iodepth_batch_complete to 25 (having set iodepth to 128, as before) seems to have helped quite a bit - I just did the second half of the test (from about 22 MSAs to 44) and the line looks much smoother to about 36 or 37 devices and has very similar features as 'aiod' was exhibiting around 40 devices (drop off as expected).

With aiod we default to using about 20% of the "depth" for the min value (iodepth_batch_complete in fio), and that seems to work quite well. In any event, I think you're "not best code" is looking pretty good! :-)

The only other thing I saw in the code that was strange was your timeout for the io_getevents call - it looks to be set to 0.0 seconds when iodepth_batch_complete is set to 0 (default being 1), is this what you really want? In aiod we set it to 10,000,000 nanoseconds (10 milliseconds), and we use that even if there is a min.

The new tail graph can be found at :

http://free.linux.hp.com/~adb/2009-08-17/fio-bcom=25.png

It is a little bit more noisy after 38 or 39 disks than the aiod graph -

http://free.linux.hp.com/~adb/2009-08-17/aiod.png

but nothing earth-shattering. (And I know that we're overloading the hyperlinks at that state, so it's going to be a bumpy ride no matter what.)

Thanks!
Alan
PS. This was run on your loop-direct branch'ed OS, so the good news is that it's handling a lot of traffic OK. So far it's been running quite solid.