speed up SATA

The "lba48" feature in ATA allows for addressing of sectors > 137GB, and also allows for transfers of up to 64K sector, instead of the traditional 256 sectors in older ATA.

libata simply limited all transfers to a 200 sectors (just under the 256 sector limit). This was mainly being careful, and making sure I had a solution that worked everywhere. I also wanted to see how the iommu S/G stuff would shake out.

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

What will happen when a PATA disk lies behind a Marvel(ous) bridge, as in most SATA disks today?

Is large transfers mandatory in the LBA48 spec and is LBA48 really mandatory in SATA?

Quoted 2 lines

What will happen when a PATA disk lies behind a Marvel(ous) bridge, as in most SATA disks today?

Larger transfers work fine in PATA, too.

Quoted 2 lines

Is large transfers mandatory in the LBA48 spec and is LBA48 really mandatory in SATA?

Yes and no, in that order :) SATA doesn't mandate lba48, but it is highly unlikely that you will see SATA disk without lba48.

Regardless, libata supports what the drive supports. Older disks still work just fine.

Hi Jeff.

Quoted 4 lines

Quoted 2 lines

What will happen when a PATA disk lies behind a Marvel(ous) bridge, as in most SATA disks today?

Larger transfers work fine in PATA, too.

Correct me if I'm wrong but wasn't the max number of sectors in a transfer limited to 255 instead of 256 because there were buggy drives out there before ? I seem to recall something like that but I could be wrong.

Hi Jeff.

Quoted 16 lines

The "lba48" feature in ATA allows for addressing of sectors > 137GB, and also allows for transfers of up to 64K sector, instead of the traditional 256 sectors in older ATA.

libata simply limited all transfers to a 200 sectors (just under the 256 sector limit). This was mainly being careful, and making sure I had a solution that worked everywhere. I also wanted to see how the iommu S/G stuff would shake out.

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

What will happen when a PATA disk lies behind a Marvel(ous) bridge, as in most SATA disks today?

Is large transfers mandatory in the LBA48 spec and is LBA48 really mandatory in SATA?

And yes, I saw that the dmesg showed a Maxtor drive, but I'm uncertain if that disk of yours has a Marvel chip on or not, since newer Maxtors might (have) come out (already) without a Marvel chip, I just don't know.

Quoted 19 lines · Stefan Smietanowski

Hi Jeff.

Quoted 16 lines

The "lba48" feature in ATA allows for addressing of sectors > 137GB, and also allows for transfers of up to 64K sector, instead of the traditional 256 sectors in older ATA.

libata simply limited all transfers to a 200 sectors (just under the 256 sector limit). This was mainly being careful, and making sure I had a solution that worked everywhere. I also wanted to see how the iommu S/G stuff would shake out.

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

What about latency?

What about recently discussed PRD table "limit" of 256 entries?

AFAIR these are the reasons why IDE driver is currently limiting max request size to 1024K on LBA48 disks.

That's the main limitation on request size right now... libata limits S/G table entries to 128[1], so a perfectly aligned, fully merged

Quoted 2 lines · Jeff Garzik

That's the main limitation on request size right now... libata limits S/G table entries to 128[1], so a perfectly aligned, fully merged

    ...

[1] because even though the block layer properly splits on segment boundaries, pci_map_sg() may violate those boundaries (James B and others are working on fixing this). So... for right now the driver must check the s/g entry boundaries after DMA mapping, and split them (again) if necessary. IDE does this in ide_build_dmatable().

Quoted 5 lines · Jeff Garzik

Quoted 2 lines · Jeff Garzik

That's the main limitation on request size right now... libata limits S/G table entries to 128[1], so a perfectly aligned, fully merged

Quoted 7 lines

   ...

[1] because even though the block layer properly splits on segment boundaries, pci_map_sg() may violate those boundaries (James B and others are working on fixing this). So... for right now the driver must check the s/g entry boundaries after DMA mapping, and split them (again) if necessary. IDE does this in ide_build_dmatable().

You are right but small clarification is needed: code in ide_build_dmatable() predates segment boundary support in block layer (IDE never relied on it).

Quoted 1 line

What about latency?

merged transfer will top out at 8MB. You don't see that unless you're on a totally quiet machine with tons of free, contiguous pages. So in practice it winds up being much smaller, the more loaded the system gets (and pagecache gets fragmented).

Latency definitely changes for the default case, but remember that a lot of that is writeback, or streaming writes. Latency-sensitive applications already know how to send small or no-wait I/Os, because

The "lba48" feature in ATA allows for addressing of sectors > 137GB, and also allows for transfers of up to 64K sector, instead of the traditional 256 sectors in older ATA.

libata simply limited all transfers to a 200 sectors (just under the 256 sector limit). This was mainly being careful, and making sure I had a solution that worked everywhere. I also wanted to see how the iommu S/G stuff would shake out.

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

See my reply to Bart.

Also, it is not the driver's responsibility to do anything but export the hardware maximums.

It's up to the sysadmin to choose a disk scheduling policy they like, which implies that a _scheduler_, not each individual driver, should place policy limitations on max_sectors.

Quoted 10 lines · Jeff Garzik

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

See my reply to Bart.

Quoted 4 lines

Also, it is not the driver's responsibility to do anything but export the hardware maximums.

It's up to the sysadmin to choose a disk scheduling policy they like, which implies that a _scheduler_, not each individual driver, should place policy limitations on max_sectors.

Yeah I suppose you're right there. In practice it doesn't work that way though, does it?

I also wouldn't want to lock out any users who wanted to use SATA at full speed ;-)

Quoted 1 line · Jeff Garzik

I also wouldn't want to lock out any users who wanted to use SATA at full speed ;-)

And full speed requires 32MB requests?

Quoted 7 lines · Jens Axboe

Quoted 2 lines · Jeff Garzik

I also wouldn't want to lock out any users who wanted to use SATA at full speed ;-)

And full speed requires 32MB requests?

Full speed is the SATA driver supporting the hardware maximum. The

Quoted 5 lines · Jeff Garzik

Quoted 3 lines · Jens Axboe

On Sat, Mar 27 2004, Jeff Garzik wrote:

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

I also wouldn't want to lock out any users who wanted to use SATA at full speed ;-)

And full speed requires 32MB requests?

Full speed is the SATA driver supporting the hardware maximum. The

Come on Jeff, don't be such a slave to the hardware specifications. Just because it's possible to send down 32MB requests doesn't necessarily mean it's a super thing to do, nor that it automagically makes 'things go faster'. The claim is that back-to-back 1MB requests are every bit as fast as a 32MB request (especially if you have a small queue depth, in that case there truly should be zero benefit to doing the bigger ones). The cut-off point is likely even lower than 1MB, I'm just using that figure as a value that is 'pretty big' yet doesn't incur too large latencies just because of its size.

Quoted 17 lines · Jens Axboe

Quoted 5 lines · Jeff Garzik

Quoted 3 lines · Jens Axboe

On Sat, Mar 27 2004, Jeff Garzik wrote:

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

I also wouldn't want to lock out any users who wanted to use SATA at full speed ;-)

And full speed requires 32MB requests?

Full speed is the SATA driver supporting the hardware maximum. The

Quoted 9 lines

Come on Jeff, don't be such a slave to the hardware specifications. Just because it's possible to send down 32MB requests doesn't necessarily mean it's a super thing to do, nor that it automagically makes 'things go faster'. The claim is that back-to-back 1MB requests are every bit as fast as a 32MB request (especially if you have a small queue depth, in that case there truly should be zero benefit to doing the bigger ones). The cut-off point is likely even lower than 1MB, I'm just using that figure as a value that is 'pretty big' yet doesn't incur too large latencies just because of its size.

For me this is a policy issue.

I agree that huge requst hurt latency. I just disagree that the _driver_ should artificially lower its maximums to fit a guess about what the best request size should be.

If there needs to be an overall limit on per-size size, do it at the block layer. It's not scalable to hardcode that limit into every driver. That's not the driver's job. The driver just exports the hardware limits, nothing more.

A limit is fine. I support that. An artificial limit in the driver is not.

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.

I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

From that point of view, the generic block layer is exactly the right

Quoted 8 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.

The driver doesn't know how fast the drive is either.

Without timing the drive, interface, and for different request sizes, neither the block layer _nor_ the driver know a suitable size.

Quoted 3 lines

I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

But the driver doesn't have _any_ knowledge of what the I/O scheduler wants. 1MB requests may be a cut-off above which there is negligable

throughput gain for SATA, but those requests may be _far_ too large for a low-latency I/O scheduling requirement.

If we have a high-level latency scheduling constraint that userspace should be able to issue a read and get the result within 50ms, or that the average latency for reads should be <500ms, how does the SATA driver limiting requests to 1MB help? It depends on the attached drive.

The fundamental problem here is that neither the driver nor the block layer have all the information needed to select optimal or maximum request sizes. That can only be found by timing the device, perhaps every time a request is made, and adjusting the I/O scheduling and request splitting parameters according to that timing and high-level latency requirements.

Quoted 1 line

From that point of view, the generic block layer is exactly the right

place to determine those parameters, because the calculation is not device-specific.

Quoted 11 lines · Jamie Lokier

Quoted 8 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.

Quoted 4 lines

The driver doesn't know how fast the drive is either.

Without timing the drive, interface, and for different request sizes, neither the block layer _nor_ the driver know a suitable size.

The driver may not know exactly, but it does know a ball park figure. You know if you are driving floppy (sucky transfer and latency), hard drive, cdrom (decent transfer, sucky seeks), etc.

Quoted 3 lines

The driver may not know exactly, but it does know a ball park figure. You know if you are driving floppy (sucky transfer and latency), hard drive, cdrom (decent transfer, sucky seeks), etc.

Agreed. Really we have two types of information:
* the device's hard limit
* the default limit that should be applied to that class of devices

I would much rather do something like

blk_queue_set_class(q, CLASS_DISK)

and have that default per-class policy

	switch (class) {
	case CLASS_DISK:
	q->max_sectors = min(q->max_sectors, CLASS_DISK_MAX_SECTORS);
	...

than hardcode the limit in the driver. That's easy and quick. That's a minimal solution that gives me what I want -- don't hardcode generic limits in the driver -- while IMO giving you what you want, a sane limit in an easy way.

Right now we are hardcoding the same per-class limits into each floppy driver, each disk driver, etc. At the very least devices that act the same way should all be using the same tunable, whether it's a compile-time tunable (CLASS_xxx_MAX_SECTORS) or a runtime tunable.

Long term, the IO scheduler and the VM should really be figuring out the best request size, from zero to <hardware limit>.

Could be used to limit tcq depth, not just request sizes solving two problems at once. I already have a tiny bit of keeping this accounting to do proper unplugs (right now it just looks at missing requests from the pool, doesn't work on tcq).

Quoted 4 lines · Jens Axboe

Could be used to limit tcq depth, not just request sizes solving two problems at once. I already have a tiny bit of keeping this accounting to do proper unplugs (right now it just looks at missing requests from the pool, doesn't work on tcq).

Does it make sense to allow different numbers of outstanding TCQ-reads and TCQ-writes?

Quoted 2 lines

Does it make sense to allow different numbers of outstanding TCQ-reads and TCQ-writes?

Might not be a silly thing to experiment with, definitely something that should be tested (added to list...)

Quoted 6 lines · Jamie Lokier

Quoted 4 lines · Jens Axboe

Could be used to limit tcq depth, not just request sizes solving two problems at once. I already have a tiny bit of keeping this accounting to do proper unplugs (right now it just looks at missing requests from the pool, doesn't work on tcq).

Quoted 2 lines

Does it make sense to allow different numbers of outstanding TCQ-reads and TCQ-writes?

Might not be a silly thing to experiment with, definitely something that should be tested (added to list...)

Quoted 10 lines · Jens Axboe

Quoted 8 lines · Jamie Lokier

Quoted 4 lines · Jens Axboe

Could be used to limit tcq depth, not just request sizes solving two problems at once. I already have a tiny bit of keeping this accounting to do proper unplugs (right now it just looks at missing requests from the pool, doesn't work on tcq).

Does it make sense to allow different numbers of outstanding TCQ-reads and TCQ-writes?

Quoted 2 lines

Might not be a silly thing to experiment with, definitely something that should be tested (added to list...)

I wonder if the max read size could/should be correlated with the readahead size for such devices... it sounds like a related property at least.

Indeed, it would be best to keep the read-ahead window at least a multiple of the max read size. So you don't get the nasty effects of having a 128k read-ahead window, but device with 255 sector limit resulting in 124KB + 4KB read.

Quoted 4 lines · Jens Axboe

Indeed, it would be best to keep the read-ahead window at least a multiple of the max read size. So you don't get the nasty effects of having a 128k read-ahead window, but device with 255 sector limit resulting in 124KB + 4KB read.

Any work underway to implement this?

Quoted 2 lines

Might not be a silly thing to experiment with, definitely something that should be tested (added to list...)

I wonder if the max read size could/should be correlated with the readahead size for such devices... it sounds like a related property at least.

Yep it sure does, but try and find a drive attached to a SATA controller that cannot do 40MiB/sec (or something like that). Storage doesn't move _that_ fast, you can keep up.

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.
So you want to put this _policy_ in the block layer, instead of in the

driver. That's an even worse decision if your reasoning is policy. The only such limits I would want to put in, are those of the bio where simply is best to keep that small and contained within a single page to avoid higher order allocations to do io. Limits based on general sound principles, not something that caters to some particular piece of hardware. I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

Quoted 8 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.
So you want to put this _policy_ in the block layer, instead of in the

Quoted 8 lines

driver. That's an even worse decision if your reasoning is policy. The only such limits I would want to put in, are those of the bio where simply is best to keep that small and contained within a single page to avoid higher order allocations to do io. Limits based on general sound principles, not something that caters to some particular piece of hardware. I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

How about per-device policies and driver hints wrt. optimal io?

Quoted 8 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.
So you want to put this _policy_ in the block layer, instead of in the

Quoted 8 lines

driver. That's an even worse decision if your reasoning is policy. The only such limits I would want to put in, are those of the bio where simply is best to keep that small and contained within a single page to avoid higher order allocations to do io. Limits based on general sound principles, not something that caters to some particular piece of hardware. I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

How about per-device policies and driver hints wrt. optimal io?

Quoted 18 lines · William Lee Irwin III

Quoted 16 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.
So you want to put this _policy_ in the block layer, instead of in the driver. That's an even worse decision if your reasoning is policy. The only such limits I would want to put in, are those of the bio where simply is best to keep that small and contained within a single page to avoid higher order allocations to do io. Limits based on general sound principles, not something that caters to some particular piece of hardware. I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

Quoted 1 line

How about per-device policies and driver hints wrt. optimal io?

Yep, user-tunable per-device policies with sane driver defaults.

Quoted 21 lines · Bartlomiej Zolnierkiewicz

Quoted 20 lines · William Lee Irwin III

Quoted 18 lines · Jens Axboe

Sorry, but I cannot disagree more. You think an artificial limit at the block layer is better than one imposed at the driver end, which actually has a lot more of an understanding of what hardware it is driving? This makes zero sense to me. Take floppy.c for instance, I really don't want 1MB requests there, since that would take a minute to complete. And I might not want 1MB requests on my Super-ZXY storage, because that beast completes io easily at an iorate of 200MB/sec.
So you want to put this _policy_ in the block layer, instead of in the driver. That's an even worse decision if your reasoning is policy. The only such limits I would want to put in, are those of the bio where simply is best to keep that small and contained within a single page to avoid higher order allocations to do io. Limits based on general sound principles, not something that caters to some particular piece of hardware. I absolutely refuse to put a global block layer 'optimal io size' restriction in, since that is the ugliest of policies and without having _any_ knowledge of what the hardware can do.

Quoted 3 lines

Quoted 1 line

How about per-device policies and driver hints wrt. optimal io?

Yep, user-tunable per-device policies with sane driver defaults.

BTW, these are trivial to expose through sysfs as their as inside the queue already.

Making something user tunable is usually not the best idea, if you can deduct these things automagically instead. So whether this is the best idea, depends on which way you want to go.

Quoted 3 lines · Jens Axboe

Making something user tunable is usually not the best idea, if you can deduct these things automagically instead. So whether this is the best idea, depends on which way you want to go.

I think it's the best idea for now, long-term we are better with automagic.

I like generic tunables such as "laptop mode" or "low latency" or "high throughput". These sorts of tunables should affect the "automagic" calculations.

Quoted 1 line · Jeff Garzik

I like generic tunables such as "laptop mode" or "low latency" or "high throughput". These sorts of tunables should affect the "automagic" calculations.

Not sure. Things like "low latency" and "high throughput" may need other things, such as "seek latency" and "bandwidth" as _inputs_, not as outputs.

Such device parameters should have reasonable defaults, and use a userspace app which runs a quick seek latency and bandwidth test at mount-time, poking the results into sysfs.

Quoted 6 lines · Bartlomiej Zolnierkiewicz

Quoted 3 lines · Jens Axboe

Making something user tunable is usually not the best idea, if you can deduct these things automagically instead. So whether this is the best idea, depends on which way you want to go.

Quoted 1 line

I think it's the best idea for now, long-term we are better with automagic.

Mostly agreed:

Like I mentioned in the last message, the IO scheduler and the VM should

Quoted 12 lines · Jeff Garzik

Quoted 8 lines · Bartlomiej Zolnierkiewicz

Quoted 3 lines · Jens Axboe

Making something user tunable is usually not the best idea, if you can deduct these things automagically instead. So whether this is the best idea, depends on which way you want to go.

I think it's the best idea for now, long-term we are better with automagic.

Quoted 3 lines

Mostly agreed:

Like I mentioned in the last message, the IO scheduler and the VM should

this is not an I/O scheduler or VM issue.

the max size of a request is something that should be set internally to the blkdev layer (at a lower level than the I/O scheduler or the VM layer).

If one day things will change and the harddisk will require 32M large DMA transactions to keep up with the speed of the disk, the thing should be still solved during disk discovery inside the blkdev layer. The

"automagic" suggestions discussed by Jamie and Jens should be just benchmarks internal to the blkdev layer, trying to read contigously first with 1M then 2M then 4M etc.. until the speed difference goes below 1% or whatever similar "autotune" algorithm.

My point is there are two maximums:

1) the hardware limit
2) the limit that "makes sense", e.g. 512k or 1M for most

The driver should only care about #1, and should be "told" #2.

A very, very, very minimal implementation could be this:

include/linux/blkdev.h Sun Mar 28 22:44:15 2004 renamed from include/linux/blkdev.h Fri Mar 12 04:33:07 2004 +2 -0

@@ -607,6 +607,24 @@
 
 extern void drive_stat_acct(struct request *, int, int);
 
+#define BLK_DISK_MAX_SECTORS   2048
+#define BLK_FLOPPY_MAX_SECTORS 64
 
 

Hardcoding such a maximum in the driver is inflexible and IMO incorrect.

Once you change the API too, then you can set the hardwre limit in your driver and relay on the highlevel blkdev code to find the optimal runtime dma size for bulk I/O, but today it's your driver that is enforcing a runtime bulk dma size, not the maximim limit of the controller, and so you should code your patch accordingly.

Quoted 5 lines · Andrea Arcangeli

Once you change the API too, then you can set the hardwre limit in your driver and relay on the highlevel blkdev code to find the optimal runtime dma size for bulk I/O, but today it's your driver that is enforcing a runtime bulk dma size, not the maximim limit of the controller, and so you should code your patch accordingly.

This magic 512k or 1M limit has nothing to do with SATA. It's a magic number whose definition is "we don't want PATA or SATA or SCSI disks doing larger requests than this for latency, VM, and similar reasons."

That definition belongs outside the low-level SATA driver.

This 512k/1M value is sure to change over time, which involves needlessly plugging the same value into multiple places. And when such changes occurs, you must be careful not to exceed hardware- and errata-based limits -- of which there is no distinction in the code.

I think the length of this discussion alone clearly implies that the low-level driver should not be responsible for selecting this value, if nothing else ;-)

Here's a quickly done patch that attempts to adjust the value based on a previous range of completed requests. It changes ->max_sectors to be a hardware limit, adding ->optimal_sectors to be our max issued io target. It is split on READ and WRITE. The target is to keep request execution time under BLK_IORATE_TARGET, which is 50ms in this patch. read-ahead max window is kept within a single request in size.

Here's a quickly done patch that attempts to adjust the value based on a previous range of completed requests. It changes ->max_sectors to be a hardware limit, adding ->optimal_sectors to be our max issued io target. It is split on READ and WRITE. The target is to keep request execution time under BLK_IORATE_TARGET, which is 50ms in this patch. read-ahead max window is kept within a single request in size.

So this is pretty half-assed, but it gets the point across. Things that should be looked at (meaning - should be done, but I didn't want to waste time on them now):

- I added the hook to see how many in-drive queued requests you have, so there's the possibility to add tcq knowledge as well.

- The optimal_sectors calculation is just an average of previous maximum with current maximum. The scheme has a number of break down points, for instance writes starving reads will give you a bad read execution time, further limiting ->optimal_sectors[READ]

- The optimal_sectors calculation is just an average of previous maximum with current maximum. The scheme has a number of break down points, for instance writes starving reads will give you a bad read execution time, further limiting ->optimal_sectors[READ]

Jens Axboe wrote:

Quoted 4 lines

- The optimal_sectors calculation is just an average of previous maximum with current maximum. The scheme has a number of break down points, for instance writes starving reads will give you a bad read execution time, further limiting ->optimal_sectors[READ]

I did look through your code a bit, but one thing to be concerned with is that going only on max throughput might be fooled by cache hits on the drive.

If you are taking your samples over time, that shouldn't matter... if the system workload is such that you are hitting the drive cache the majority of the time, you're not being "fooled" by cache hits, the patch would be taking those cache hits into account.

If the system isn't hitting the drive cache the majority of the time, statistical sampling will automatically notice that too...

In 2.4 reaching 512k DMA units that helped a lot, but going past 512k didn't help in my measurements. 1M maybe these days is needed (as Jens suggested) but >1M still sounds overkill and I completely agree with Jens about that.

Quoted 4 lines

In 2.4 reaching 512k DMA units that helped a lot, but going past 512k didn't help in my measurements. 1M maybe these days is needed (as Jens suggested) but >1M still sounds overkill and I completely agree with Jens about that.

at least 1Mb... more than 1mb (I doubt 32mb is really necessarily useful) is nice for flushing contiguous journal data to disk. between 1-8mb in one io.

at least 1mb is good when you have to process massive amounts of data, just read huge chunks of files, we tend to do 1mb. anyway, as you said, at some point it's a bit overkill . I thinkg 1-8mb makes sense.

Quoted 4 lines

In 2.4 reaching 512k DMA units that helped a lot, but going past 512k didn't help in my measurements. 1M maybe these days is needed (as Jens suggested) but >1M still sounds overkill and I completely agree with Jens about that.

at least 1Mb... more than 1mb (I doubt 32mb is really necessarily useful) is nice for flushing contiguous journal data to disk. between 1-8mb in one io.

at least 1mb is good when you have to process massive amounts of data, just read huge chunks of files, we tend to do 1mb. anyway, as you said, at some point it's a bit overkill . I thinkg 1-8mb makes sense.

Quoted 4 lines

In 2.4 reaching 512k DMA units that helped a lot, but going past 512k didn't help in my measurements. 1M maybe these days is needed (as Jens suggested) but >1M still sounds overkill and I completely agree with Jens about that.

at least 1Mb... more than 1mb (I doubt 32mb is really necessarily useful) is nice for flushing contiguous journal data to disk. between 1-8mb in one io.

at least 1mb is good when you have to process massive amounts of data, just read huge chunks of files, we tend to do 1mb. anyway, as you said, at some point it's a bit overkill . I thinkg 1-8mb makes sense.

The point is that if you run read contigously from disk with a 1M or 32M request size, the wall time speed difference will be maybe 0.01% or so. Running 100 irqs per second or 3 irq per second doesn't make any measurable difference. Same goes for keeping the I/O pipeline full, 1M

is more than enough to go at the speed of the storage with minimal cpu overhead. we waste 900 irqs per second just in the timer irq and another 900 irqs per second per-cpu in the per-cpu local interrupts in smp.
In 2.4 reaching 512k DMA units that helped a lot, but going past 512k didn't help in my measurements. 1M maybe these days is needed (as Jens suggested) but >1M still sounds overkill and I completely agree with Jens about that.

is beneficial at all, and your bootup log showing 32M is all but exciting, I'd be a lot more excited to see 512k there.

the effect of really big requests will be the same as the effect of permitting _more_ requests.

With this kind of control we could have /etc/init.d/io-optimize that paused the startup for 10 seconds and tests every device|controller in fstab and optimizes according to the .conf file for latency or speed... Or a daemon that retrieves statistics and adjusts the policies every minute?

Also, everybody says "do it in userland". This is doing (some of) it in userland :)

It's up to the sysadmin to choose a disk scheduling policy they like, which implies that a _scheduler_, not each individual driver, should place policy limitations on max_sectors.

Quoted 1 line · Jeff Garzik

It's up to the sysadmin to choose a disk scheduling policy they like, which implies that a _scheduler_, not each individual driver, should place policy limitations on max_sectors.

<tangent>

The block layer / scheduler guys should also think about a general (not SCSI specific) way to tune TCQ tag depth. That's IMO another policy decision.

I'm about to add a raft of SATA-2 hardware, all of which are queued. The standard depth is 32, but one board supports a whopping depth of 256.

Given past discussion on the topic, you probably don't want to queue 256 requests at a time to hardware :) But the sysadmin should be allowed to, if they wish...

Aside, you should make 2 or 4 tags the default though: that still gives you the pipelining without sacrificing latency and usibility.

The only area (I think) where large queues outperform the IO scheduler are lots of parallel, scattered reads. I think this is because the drive can immediately return cached data even though it looks like a large seek to the IO scheduler. (This is from testing on my single, old SCSI disk though.)

TCQ-on-write for ATA disks is yummy because you don't really know what the heck the ATA disk is writing at the present time. By the time the Linux disk scheduler gets around to deciding it has a nicely merged and scheduled set of requests, it may be totally wrong for the disk's IO scheduler. TCQ gives the disk a lot more power when the disk integrates writes into its internal IO scheduling.

Quoted 1 line · Jeff Garzik

TCQ-on-write for ATA disks is yummy because you don't really know what the heck the ATA disk is writing at the present time. By the time the Linux disk scheduler gets around to deciding it has a nicely merged and scheduled set of requests, it may be totally wrong for the disk's IO scheduler. TCQ gives the disk a lot more power when the disk integrates writes into its internal IO scheduling.

Slightly beneficial for throughput, disastrous for latency.

It appears the only way we'll ever get this gross misdesign fixed is to add a latency test to winbench.

"IOPs" are what make a lot of storage peeps excited these days, so they are being pushed in a low-latency direction anyway.

Quoted 1 line · Jeff Garzik

"IOPs" are what make a lot of storage peeps excited these days, so they are being pushed in a low-latency direction anyway.

IOPS says nothing about latency, it's still 100% throughput oriented.

Quoted 6 lines · Jens Axboe

Quoted 2 lines · Jeff Garzik

"IOPs" are what make a lot of storage peeps excited these days, so they are being pushed in a low-latency direction anyway.

Quoted 1 line

IOPS says nothing about latency, it's still 100% throughput oriented.

In order to achieve a large IOPs number, one must sent a ton of small requests. Per-request latency as well as overall throughput is a factor.

TCQ-on-write for ATA disks is yummy because you don't really know what the heck the ATA disk is writing at the present time. By the time the Linux disk scheduler gets around to deciding it has a nicely merged and scheduled set of requests, it may be totally wrong for the disk's IO scheduler. TCQ gives the disk a lot more power when the disk integrates writes into its internal IO scheduling.

Quoted 1 line · Jeff Garzik

TCQ-on-write for ATA disks is yummy because you don't really know what the heck the ATA disk is writing at the present time. By the time the Linux disk scheduler gets around to deciding it has a nicely merged and scheduled set of requests, it may be totally wrong for the disk's IO scheduler. TCQ gives the disk a lot more power when the disk integrates writes into its internal IO scheduling.

Does TCQ-on-write allow you to do ordered write commits, as with barriers, but without needing full cache flushes, and still get good performance?

Quoted 2 lines

Does TCQ-on-write allow you to do ordered write commits, as with barriers, but without needing full cache flushes, and still get good performance?

Nope, TCQ is just a bunch of commands rather than one. There are no special barrier indicators you can pass down with a command.

This is what I mean: turn off write cacheing, and performance on PATA drops because of the serialisation and lost inter-command time.

With TCQ-on-write, you can turn off write cacheing and in theory performance doesn't have to drop, is that right?

Quoted 5 lines · Jamie Lokier

This is what I mean: turn off write cacheing, and performance on PATA drops because of the serialisation and lost inter-command time.

With TCQ-on-write, you can turn off write cacheing and in theory performance doesn't have to drop, is that right?

hmmm, that's a good point. I honestly don't know. Something to be tested, I suppose...

My premature guess would be that you would need a queue depth larger than 2 or 4 before performance starts to not-suck.

This is what I mean: turn off write cacheing, and performance on PATA drops because of the serialisation and lost inter-command time.

With TCQ-on-write, you can turn off write cacheing and in theory performance doesn't have to drop, is that right?

Quoted 2 lines

This is what I mean: turn off write cacheing, and performance on PATA drops because of the serialisation and lost inter-command time.

Since you have to write the sectors in order (well, you don't have to, but the drives all do this), you lose a rev between each write when you don't queue commands or have write cacheing.

Quoted 2 lines

With TCQ-on-write, you can turn off write cacheing and in theory performance doesn't have to drop, is that right?

Correct. I have proven this to my satisfaction.

However, cached writes (queued or unqueued), especially small ones, will have WAY higher ops/sec. ATA TCQ is limited to 32 choices for the next best operation, but an 8MB buffer doing 4K random-ops could potentially have ~2000 choices for the next thing to do. (assuming perfect cache granularity, etc, etc)

At 32 choices, the seek and rotate are still somewhat significant, though way better than unqueued. With 2000 things to choose from, the drive is never, ever idle. Seek, land just-in-time, read/write, rinse/repeat.

Quoted 6 lines · Jeremy Higdon

Quoted 2 lines

This is what I mean: turn off write cacheing, and performance on PATA drops because of the serialisation and lost inter-command time.

Since you have to write the sectors in order (well, you don't have to, but the drives all do this), you lose a rev between each write when you don't queue commands or have write cacheing.

I don't see how the driver can write the sectors out of order, if there is no TCQ (we're talking PATA) and every write must be committed before it's acknowledged (write cache disabled).

Quoted 4 lines

Quoted 2 lines

With TCQ-on-write, you can turn off write cacheing and in theory performance doesn't have to drop, is that right?

Correct. I have proven this to my satisfaction.

Are you refuting the following assertion by Eric D. Mudama's, based on your measurements? In other words, are ATA's 32 TCQ slots enough to eliminate the performance advantage of write cacheing?

Quoted 3 lines

Are you refuting the following assertion by Eric D. Mudama's, based on your measurements? In other words, are ATA's 32 TCQ slots enough to eliminate the performance advantage of write cacheing?

I must apologize. I had thought the context was SCSI, but now I see that it is linux-ide. So please disregard comments about command queueing.

If you have write cache disabled and no TCQ (latter is common, former may or may not be), you want to write as big a chunk as you can.

Quoted 10 lines · Jamie Lokier

Quoted 6 lines · Jeff Garzik

TCQ-on-write for ATA disks is yummy because you don't really know what the heck the ATA disk is writing at the present time. By the time the Linux disk scheduler gets around to deciding it has a nicely merged and scheduled set of requests, it may be totally wrong for the disk's IO scheduler. TCQ gives the disk a lot more power when the disk integrates writes into its internal IO scheduling.

Quoted 2 lines

Does TCQ-on-write allow you to do ordered write commits, as with barriers, but without needing full cache flushes, and still get good performance?

Nope, TCQ is just a bunch of commands rather than one. There are no special barrier indicators you can pass down with a command.

What would be nice (and I seem to recall that Andre also pushed for this) would be the FUA bit doubling as an ordered tag indicator when using TCQ. It's one of those things that keep ATA squarely outside of the big machine uses. That other OS had a differing opinion of what to do with that, so...

Quoted 5 lines · Jens Axboe

What would be nice (and I seem to recall that Andre also pushed for this) would be the FUA bit doubling as an ordered tag indicator when using TCQ. It's one of those things that keep ATA squarely outside of the big machine uses. That other OS had a differing opinion of what to do with that, so...

Preach on, brother Jens :)

I agree completely. Or, the ATA guys could use SCSI's ordered tags / linked commands.

Regardless, there's ATA dain bramage that needs fixing... Sigh.

Quoted 9 lines · Jeff Garzik

Quoted 5 lines · Jens Axboe

What would be nice (and I seem to recall that Andre also pushed for this) would be the FUA bit doubling as an ordered tag indicator when using TCQ. It's one of those things that keep ATA squarely outside of the big machine uses. That other OS had a differing opinion of what to do with that, so...

Preach on, brother Jens :)

I think we already lost this one, I'm afraid :-)

Quoted 3 lines

I agree completely. Or, the ATA guys could use SCSI's ordered tags / linked commands.

Regardless, there's ATA dain bramage that needs fixing... Sigh.

Indeed, and it really hurt that they passed up this oportunity last time, ATA TCQ would have kicked so much more ass... Maybe Eric can beat some sense into his colleagues.

Quoted 6 lines

Quoted 4 lines

I agree completely. Or, the ATA guys could use SCSI's ordered tags / linked commands.

Regardless, there's ATA dain bramage that needs fixing... Sigh.

Quoted 3 lines

Indeed, and it really hurt that they passed up this oportunity last time, ATA TCQ would have kicked so much more ass... Maybe Eric can beat some sense into his colleagues.

I bet if we can come up with a decent proposal, with technical rationale for the change... that could be presented to the right ATA people :) It's worth a shot.

Quoted 1 line · Jeff Garzik

I bet if we can come up with a decent proposal, with technical rationale for the change... that could be presented to the right ATA people :) It's worth a shot.

Count me in for that. The ATA people seem to have a preference for

adding a new set of commands for this type of there, where as I (originally, I did actually send in a proposal like this on the ml) wanted to just use one of the reserved bits in the tag field.

Over and above the barrier issue, the general problem of "OS doesn't know precisely what's on the platter" leads to a nasty edge case:

If an error occurs where the typical resolution is a bus or device reset, cached writes that have been acknowledged to the OS but not yet hit the media will likely be lost. This seems to be worse in SATA, where you have a new class of errors (SATA link up/down, etc.) that is also typically dealt with via reset.

Jeff

I'm about to add a raft of SATA-2 hardware, all of which are queued. The standard depth is 32, but one board supports a whopping depth of 256.

Hi Jeff.

Quoted 1 line

I'm about to add a raft of SATA-2 hardware, all of which are queued. The standard depth is 32, but one board supports a whopping depth of 256.

Speaking of which .. I just read an announcement that someone (of course the name eludes me) announced a DVD Burner that's SATA.

Found it:

http://www.plextor.com/english/news/press/712SA_pr.htm

a) Are there provisions in the SATA (1) SPEC for support of non-disk units?

b) if (strcmp(a, "no"))
Do you know anything about it, ie is it SATA1 or 2 or what?

c) Let's ponder one gets a unit like this - is it usable with libata yet?

d) if (strcmp(c, "no"))

     Will it? :)

SATA ATAPI looks and works just like PATA ATAPI, with one notable exception: S/ATAPI will include "asynchronous notification", a feature that allows you to eliminate the polling of the cdrom driver that normally occurs.

You can use ATAPI on SATA today, using a PATA->SATA bridge. In fact that's the only way I can test SATA ATAPI at all, right now.

I hope somebody sends me one of these Plextor devices for testing ;-)

Hi Jeff.

Quoted 4 lines

SATA ATAPI looks and works just like PATA ATAPI, with one notable exception: S/ATAPI will include "asynchronous notification", a feature that allows you to eliminate the polling of the cdrom driver that normally occurs.

You can use ATAPI on SATA today, using a PATA->SATA bridge. In fact that's the only way I can test SATA ATAPI at all, right now.

Quoted 1 line

I hope somebody sends me one of these Plextor devices for testing ;-)

Would that mean that one uses the same (sub)drivers as normal SCSI devices do ?

sd/sg/etc ...

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

See my reply to Bart.

Also, it is not the driver's responsibility to do anything but export the hardware maximums.

It's up to the sysadmin to choose a disk scheduling policy they like, which implies that a _scheduler_, not each individual driver, should place policy limitations on max_sectors.

Quoted 9 lines · Jeff Garzik

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

See my reply to Bart.

Also, it is not the driver's responsibility to do anything but export the hardware maximums.

The problem is that what the 'reasonable size' is depends highly on the hardware. The 32MB doesn't make any sense to me for SATA (or for anything else, for that matter). Back-to-back 1MB requests (this is the default I chose for PATA) should be practially identical for throughput, and loads better for optimizing latencies. You cannot do much with 32MB requests wrt latency...

So you could change ->max_sectors to be 'max allowable io, hardware limitation' and add ->optimal_sectors to be 'best sized io'. I don't see tha it buys you anything, since you'd be going for optimal_sectors all the time anyways.

Additionally, a single bio cannot currently be bigger than 256 pages (ie 1MB request on 4k page). This _could_ be increased of course, but not beyond making ->bio_io_vec be bigger than a page. It's already bigger than that on x86 in fact, if you use 64-bit dma_addr_t. For 32-bit dma_addr_t 256 entries fit a page perfectly. Merging can get you bigger requests of course.

In summary, there needs to be some extremely good numbers and arguments for changing any of this, so far I don't see anything except people drooling over sending 32MB requests.

Quoted 33 lines · Jens Axboe

Quoted 10 lines · Jeff Garzik

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

See my reply to Bart.

Also, it is not the driver's responsibility to do anything but export the hardware maximums.

The problem is that what the 'reasonable size' is depends highly on the hardware. The 32MB doesn't make any sense to me for SATA (or for anything else, for that matter). Back-to-back 1MB requests (this is the default I chose for PATA) should be practially identical for throughput, and loads better for optimizing latencies. You cannot do much with 32MB requests wrt latency...

So you could change ->max_sectors to be 'max allowable io, hardware limitation' and add ->optimal_sectors to be 'best sized io'. I don't see tha it buys you anything, since you'd be going for optimal_sectors all the time anyways.

Additionally, a single bio cannot currently be bigger than 256 pages (ie 1MB request on 4k page). This _could_ be increased of course, but not beyond making ->bio_io_vec be bigger than a page. It's already bigger than that on x86 in fact, if you use 64-bit dma_addr_t. For 32-bit dma_addr_t 256 entries fit a page perfectly. Merging can get you bigger requests of course.

Quoted 3 lines

In summary, there needs to be some extremely good numbers and arguments for changing any of this, so far I don't see anything except people drooling over sending 32MB requests.

I think you're way too stuck on the 32MB number. Other limitations already limit that further.

If and when the upper layers pass down such requests, my driver can handle that. For today, various external limitations and factors -- such as what you describe above -- mean the driver won't be getting anywhere near 32MB requests. And my driver can handle that just fine too.

So you could change ->max_sectors to be 'max allowable io, hardware limitation' and add ->optimal_sectors to be 'best sized io'. I don't see tha it buys you anything, since you'd be going for optimal_sectors all the time anyways.

Quoted 6 lines

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

Hi Jeff,
I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

Quoted 8 lines · Nick Piggin

Quoted 3 lines

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

Hi Jeff,
I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

As far as interactivity and throughput is concerned, the effect of really
big requests will be the same as the effect of permitting _more_ requests. Namely: more memory can be under readahead or writeback at any particular point in time.

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

Quoted 3 lines · Nick Piggin

I think 32MB is too much. You incur latency and lose scheduling grainularity. I bet returns start diminishing pretty quickly after 1MB or so.

32-MB requests are the best for raw throughput.

~15ms to land at your target location, then pure 50-60MB/sec for the .5 seconds it takes to finish the operation. (media limited at that point)

Sure, there's more latency, but I guess that is application dependant.

Quoted 7 lines · Eric D. Mudama

32-MB requests are the best for raw throughput.

~15ms to land at your target location, then pure 50-60MB/sec for the .5 seconds it takes to finish the operation. (media limited at that point)

Sure, there's more latency, but I guess that is application dependant.

What about a queue depth of 2, and writing that 32MB in 1MB requests?

Er, forgot about the queue depth of only 2...

Even in that case, you'll more than likely still get better throughput with a single 32-MB command... If you send a pair of queued commands down, and the 2nd one is chosen, there's no reason that the first one won't get starved until the very end of the request, which would have bad latency on that command.

Well strictly, you send them one after the other. So unless you have something similar to our anticipatory scheduler or plugging mechanism, the drive should attack the first one first, shouldn't it?

Quoted 4 lines · Nick Piggin

Well strictly, you send them one after the other. So unless you have something similar to our anticipatory scheduler or plugging mechanism, the drive should attack the first one first, shouldn't it?

If you send 32 commands to our disk at once (TCQ/NCQ) we send 'em all to our back-end disk engine as fast as possible.

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

Date: Mon, 29 Mar 2004 13:36:42 +0200
From: Pavel Machek <[email protected]>
To: Jeff Garzik <[email protected]>
Cc: [email protected], Linux Kernel <[email protected]>,
Andrew Morton <[email protected]>
Subject: Re: [PATCH] speed up SATA

Hi!

Quoted 7 lines

Things seem to be looking pretty good, so it's now time to turn on lba48-sized transfers. Most SATA disks will be lba48 anyway, even the ones smaller than 137GB, for this and other reasons.

With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family.

32MB is one second if everything goes okay... That will be horrible for latency, right? --
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Pavel, when you say that 32MB would be 1 second is that due to the limits of the SATA bus or are you baseing this on a particular drive?

if you are basing it on a current single SATA drive (7200rpm 8MB cache, etc) consider that if you are writing to a solid state drive or to an array that presents itself as a single SATA drive then the transaction can happen MUCH faster.

if this is a limit of the SATA interface bitrate, consider that over the last several years EVERY interface has seen it's bitrate climb significantly, frequently with little (if any) fundamentalchange to the drivers nessasary to run things.

32MB is one second if everything goes okay... That will be horrible for latency, right?

[...]
With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family. [...]

Quoted 6 lines · Jeff Garzik

[...]
With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family. [...]

I have experienced a noticeable improvement concerning the CPU usage and disk throughput with this patch.

Benchmark specs:

o read from only 1 disk (sda), or from 2 disks (sda+sdb), with 1 or 2 instances of "dd if=/dev/sd? of=/dev/null bs=100M".
o hardware: two Seagate 160GB SATA, on a Silicon Image 3114, on a 32-bit/33MHz PCI bus, 1GB RAM.
o software: kernel 2.6.5-rc2-bk6-libata2.

Benchmark datas:

without the speed-up-sata patch with the speed-up-sata patch
reading sda reading sda+sdb reading sda reading sda+sdb
bi 57000 92000 57000 97000

in       1900              2400              1600              1800
cs       1800              3300              1400              1700

sy 11% 20% 9% 16%
us 0% 0% 0% 0%

("bi, in, cs, sy, us" have been reported by vmstat(8))

When reading only from sda, the speed-up-sata patch makes the number of interrupts/s drop from 1900 to 1600 (CPU usage: 11% to 9%). The throughput does not improve because 57000 blocks/s is the physical limit of the hardisk.

When reading from both sda and sdb, the improvement is more visible: the number of interrupts/s goes from 2400 to 1800 (CPU usage: 20% to 16%). But in this case, the throughput improves from 92000 blocks/s to 97000 blocks/s. I think I am reaching the physical limit of the PCI bus (theoretically it would be 133 MB/s or 133000 blocks/s). When setting the PCI latency timer of the SiI3114 controller to 240 (was 64), I am able to reach 100000 blocks/s.

As other people were complaining that the 32MB max request size might be too high, I did give a try to 1MB (by replacing "65534" by "2046" in the patch). There is no visible differences between 32MB and 1MB.

On 30 Mar 2004, Jens Axboe wrote:

| On Tue, Mar 30 2004, Marc Bevand wrote:
| [...]
| > As other people were complaining that the 32MB max request size might be too
| > high, I did give a try to 1MB (by replacing "65534" by "2046" in the patch).
| > There is no visible differences between 32MB and 1MB.
| 
| As suspected. BTW, you want to use 2048 there, not 2046. The 64K-2
| (which could be 64K-1) is just due to ->max_sectors being an unsigned
| short currently.

Okay, I thought the 2 sectors were used to store some extra informations.

I also forgot to mention that there does not seem to have any *latency* differences between 32MB and 1MB. Strange...

Quoted 4 lines · Jeff Garzik

[...]
With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family. [...]

I have experienced a noticeable improvement concerning the CPU usage and disk throughput with this patch.

Benchmark specs:

o read from only 1 disk (sda), or from 2 disks (sda+sdb), with 1 or 2 instances of "dd if=/dev/sd? of=/dev/null bs=100M".
o hardware: two Seagate 160GB SATA, on a Silicon Image 3114, on a 32-bit/33MHz PCI bus, 1GB RAM.
o software: kernel 2.6.5-rc2-bk6-libata2.

As other people were complaining that the 32MB max request size might be too high, I did give a try to 1MB (by replacing "65534" by "2046" in the patch). There is no visible differences between 32MB and 1MB.

PS: Jeff: "pci_dma_mapping_error()", in libata-core.c from your latest 2.6-libata patch, is an unresolved symbol. I have had to comment it out to be able to compile the kernel.

I think I am reaching the physical limit of the PCI bus (theoretically it would be 133 MB/s or 133000 blocks/s). When setting the PCI latency timer of the SiI3114 controller to 240 (was 64), I am able to reach 100000 blocks/s.

[...]
With this simple patch, the max request size goes from 128K to 32MB... so you can imagine this will definitely help performance. Throughput goes up. Interrupts go down. Fun for the whole family. [...]