Large Sequential Reads Being Broken Up. Why?

I am running a program on my Linux box which is asking for 2M IO (reads and writes) with the file handle being opened with the O_DIRECT flag. However, the IO being put out on the wire is no larger than 512K. My target device is the SCSI block device (/dev/sdb in this case). What is preventing me from getting large IO through the SCSI block layer? How can I fix it?

The sg device can achieve the 2M IO size, so I know its at least possible. How can I improve the IO size for the SCSI block layer?

Details:

Dell 2850 server with dual Xeons, 1G RAM
OS: Linux racerx 2.6.11.4-21.10-smp #1 SMP Tue Nov 29 14:32:49 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux Emulex LP11000 Fibre Channel HBA using driver version 8.0.13 (changing the driver hasn't helped, so far) I set the lookahead value pretty large to improve read performance (hdparm -a)
The scheduler for this device is anticipatory.

Any ideas?

Thanks,
Martin Schlining

Martin,

I was just doing some informal performance testing from a RedHat linux box to two commercial disk arrays. We have
an Emulex 8000 card on the box.

I found that exceeding 256 blocks; i.e., 128K I/O, actually decreased performance. Not only was the elapsed time
of a 200 MB file longer using sg_dd with 512MB, it consumed considerably more system time as reported by the 'time' command.

When I investigated the configuration, I found that many distributions will limit you to 1M transfers by default.

Certainly, the gains to be made by changing from 512K to 2MB are marginal and will probably add to your maintenance
costs if you ever change anything in the I/O layer.

-- David Egolf

[email protected] wrote on 01/30/2006 08:20:16 AM:

Here's one thing that will help you....
in the lpfc driver (in rev 8.0.13 - the file is lpfc_fcp.c), in the scsi_driver_template structure - add the field:

      .max_sectors = 0xFFFF,

As of 2.6.10, the kernel started paying attention to this field, which the emulex driver, as of that time, didn't set. The result was the kernel dropped back to a default max_sectors of 1024 - which results in a 512k max. The lpfc driver was updated in rev 8.0.29 with this change.

Caveat is : Even with this change, you must be using O_DIRECT to get high bandwidth. Otherwise, the upper layers will segment the requests (if I remember right, we had a hard time making a "normal" config exceed 256k).

-- james s

Actually, please also remember that the maximum SG element list size is 128 in a normal kernel (depending on the driver ... some drivers set lower limits as well), so on a very fragmented 4k page machine, you're unlikely to get above 512k just because you run out of SG table entries (obviously on 16k page machines, this goes up to 2MB, and if you're lucky enough to have a fully functional IOMMU, this limitation won't affect you at all).

James