Direct I/O bio size regression
PATCH 1/1 Direct I/O bio size regression
The change introduced here in 2.6.15:
sets the request queue max_sector size unconditionally to 1024 sectors in blk_queue_max_sectors() even if the underlying hardware can support a larger number of sectors.
Hence when building direct I/O bios, we have the situation where:
- dio_new_bio() limits bio vector size artifically to 1024 sectors / page size because bio_get_nr_vecs() is used q->max_sectors to size the new bio; and
- dio_bio_add_page() limits the total bio size to 1024 sectors because bio_add_page() now uses q->max_sectors to limit the size of the bio.
Therefore, we can't build direct I/Os larger than 1024 sectors even if the hardware supports large I/Os. This is a regression as before this mod we were able to issue direct I/Os limited by either the maximum number of vectors in an bio or the hardware limits.
The patch below (against 2.6.16) allows direct I/O to build bios as large as the underlying hardware will allow.
--- bio.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
fs/bio.c +2 -2
@@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_devicerequest_queue_t *q =bdev_get_queue (bdev);int nr_pages; - nr_pages = ((q->max_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + nr_pages = ((q->max_hw_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (nr_pages > q->max_phys_segments) nr_pages = q->max_phys_segments;if (nr_pages > q->max_hw_segments)
@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -return __bio_add_page (q, bio, page, len, offset, q->max_sectors); +return __bio_add_page (q, bio, page, len, offset, q->max_hw_sectors); }struct bio_map_data {
Quoted 44 lines · David Chinner
The change introduced here in 2.6.15:
sets the request queue
max_sectorsize unconditionally to 1024 sectors inblk_queue_max_sectors()even if the underlying hardware can support a larger number of sectors.Hence when building direct I/O bios, we have the situation where:
-
dio_new_bio()limits bio vector size artifically to 1024 sectors / page size becausebio_get_nr_vecs()is usedq->max_sectorsto size the new bio; and
-dio_bio_add_page()limits the total bio size to 1024 sectors becausebio_add_page()now usesq->max_sectorsto limit the size of the bio.Therefore, we can't build direct I/Os larger than 1024 sectors even if the hardware supports large I/Os. This is a regression as before this mod we were able to issue direct I/Os limited by either the maximum number of vectors in an bio or the hardware limits.
The patch below (against 2.6.16) allows direct I/O to build bios as large as the underlying hardware will allow.
Signed-off-by Dave Chinner <[email protected]>--- bio.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)Index:
2.6.x-xfs-new/fs/bio.c
===================================================================fs/bio.c +1
-1@@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_devicerequest_queue_t *q =bdev_get_queue (bdev);int nr_pages; - nr_pages = ((q->max_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + nr_pages = ((q->max_hw_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (nr_pages > q->max_phys_segments) nr_pages = q->max_phys_segments;if (nr_pages > q->max_hw_segments)
fs/bio.c int bio_add_page(struct bio *bio, struct · line 452
Quoted 8 lines
fs/bio.c +1
-1@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -return __bio_add_page (q, bio, page, len, offset, q->max_sectors); +return __bio_add_page (q, bio, page, len, offset, q->max_hw_sectors); }struct bio_map_data {
Clearly correct, I'll make sure this gets merged right away.
Signature · 1 line
Jens Axboe
Quoted 21 lines · Jens Axboe
Quoted 21 lines
Index:
2.6.x-xfs-new/fs/bio.c
===================================================================fs/bio.c +2
-2@@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_devicerequest_queue_t *q =bdev_get_queue (bdev);int nr_pages; - nr_pages = ((q->max_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + nr_pages = ((q->max_hw_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (nr_pages > q->max_phys_segments) nr_pages = q->max_phys_segments;if (nr_pages > q->max_hw_segments)@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -return __bio_add_page (q, bio, page, len, offset, q->max_sectors); +return __bio_add_page (q, bio, page, len, offset, q->max_hw_sectors); }
Quoted 3 lines
fs/bio.c int bio_add_page(struct bio *bio, struct · line 452
Quoted 1 line
fs/bio.c +0
-0struct bio_map_data {Clearly correct, I'll make sure this gets merged right away.
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO. That's why we have a bio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.
The best approach is probably to tune max_sectors on the system itself. That's why it is exposed, after all.
Signature · 1 line
Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Jens Axboe
Quoted 21 lines
Index:
2.6.x-xfs-new/fs/bio.c
===================================================================fs/bio.c +2
-2@@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_devicerequest_queue_t *q =bdev_get_queue (bdev);int nr_pages; - nr_pages = ((q->max_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + nr_pages = ((q->max_hw_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (nr_pages > q->max_phys_segments) nr_pages = q->max_phys_segments;if (nr_pages > q->max_hw_segments)@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -return __bio_add_page (q, bio, page, len, offset, q->max_sectors); +return __bio_add_page (q, bio, page, len, offset, q->max_hw_sectors); }
Quoted 6 lines
Quoted 3 lines
fs/bio.c int bio_add_page(struct bio *bio, struct · line 452
Quoted 1 line
fs/bio.c +0
-0struct bio_map_data {Clearly correct, I'll make sure this gets merged right away.
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O.....
FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
Quoted 3 lines
That's why we
have abio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.
bio_add_pc_page() requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio from sg_io(). The comment for bio_add_pc_page() says for use with REQ_PC queues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?
Also, it seems to me that using bio_add_pc_page() in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.
Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.
You mean /sys/block/sd*/max_sector_kb?
Cheers,
Dave.
Signature · 3 lines
Dave Chinner R&D Software Enginner SGI Australian Software Group
Quoted 36 lines · David Chinner
Quoted 22 lines · Jens Axboe
Quoted 21 lines · Jens Axboe
Quoted 21 lines
Index:
2.6.x-xfs-new/fs/bio.c
===================================================================fs/bio.c +2
-2@@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_devicerequest_queue_t *q =bdev_get_queue (bdev);int nr_pages; - nr_pages = ((q->max_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + nr_pages = ((q->max_hw_sectors <<9 ) +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (nr_pages > q->max_phys_segments) nr_pages = q->max_phys_segments;if (nr_pages > q->max_hw_segments)@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -return __bio_add_page (q, bio, page, len, offset, q->max_sectors); +return __bio_add_page (q, bio, page, len, offset, q->max_hw_sectors); }Quoted 6 lines
Quoted 3 lines
fs/bio.c int bio_add_page(struct bio *bio, struct · line 452
Quoted 1 line
fs/bio.c +0
-0struct bio_map_data {Clearly correct, I'll make sure this gets merged right away.
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O.....
Quoted 5 lines
FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
The change was needed to safely split max_sectors into two sane parts:
- The soft value, ->max_sectors, that holds a sane default of maximum io size. The main issue we want to prevent is filling the queue with huge amounts of io, both from a pinning POV but also from user latency reasons.
- The hard value, ->max_hw_sectors. Previously, there was no real clear definition of what ->max_sectors was supposed to do. We couldn't increase it to fit the hardware limits of most hardware, because that would hurt us latency/memory wise.
Quoted 9 lines
Quoted 3 lines
That's why we
have abio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.
bio_add_pc_page()requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio fromsg_io(). The comment forbio_add_pc_page()says for use withREQ_PCqueues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?
It's used for any SG_IO path, so that is not at all restricted to ide-cd. It covers all block devices.
Quoted 4 lines
Also, it seems to me that using
bio_add_pc_page()in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.
I'm not suggesting you do that at all. You should not have to change your file system. See below.
Quoted 4 lines
Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.You mean
/sys/block/sd*/max_sector_kb?
Exactly. Your max_hw_sectors_kb should already be correct, if not then that is a driver issue that needs to be fixed. And that's not a new issue, it was always so. You can then increase max_sectors_kb to any value as long as it's less than max_hw_sectors_kb, and your filesystem will happily build you ios as large as you need (equiv to what your patch would have accomplished).
Signature · 1 line
Jens Axboe
Quoted 18 lines · Jens Axboe
Quoted 11 lines · David Chinner
Quoted 3 lines · Jens Axboe
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O.....
Quoted 5 lines
FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
Quoted 6 lines
The change was needed to safely split
max_sectorsinto two sane parts:- The soft value, ->max_sectors, that holds a sane default of maximum io size. The main issue we want to prevent is filling the queue with huge amounts of io, both from a pinning POV but also from user latency reasons.
Got any data that you can share with us?
Wrt latency, is the problem to do with large requests causing short term latency? I thought that latency minimisation is the job of the I/O scheduler, so if this is the case, doesn't this indicate a deficiency of the I/O scheduler? e.g. the I/o scheduler could split large requests to reduce latency, just like you merge adjacent requests to reduce the number of I/Os and keep overall latency low...
And as to the pinning problem - if you have a problem with too much memory in the I/O queues, then the I/O queues are too deep or they need to be throttled based on the amount of data in them as well as the number of queued requests. It's the method or configuration of the I/O scheduler being used to throttle requests that is deficient here, not the fact that a filesystem is building large I/Os.
It seems to me that you've crippled the block layer to solve very specific problems that most people don't see. I haven't seen pinning problems since the cfq request queue depth was reduced from 8192 to 128 and all the I/O latency problems I see are to do with multiple small I/Os being issued rather than a single large I/O....
Quoted 4 lines
- The hard value, ->max_hw_sectors. Previously, there was no real clear definition of what ->max_sectors was supposed to do. We couldn't increase it to fit the hardware limits of most hardware, because that would hurt us latency/memory wise.
But we did have max_sectors = max_hw_sectors and I can't say that I've seen any evidence that it hurt us latency/memory wise.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.You mean
/sys/block/sd*/max_sector_kb?Exactly.
Not happy. Now, instead of having a default config that works just fine, we've got to change the config of every block device on every boot on every machine we sell. This is a big deal when you're talking about machines with _thousands_ of block devices on them all needing to have their defaults changed.
BTW, can you point me to the discussion(s) that lead to this mod so I can catch up on this quickly? I can't find anything on lkml or linux-fsdevel about it....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner R&D Software Enginner SGI Australian Software Group
Quoted 28 lines · David Chinner
Quoted 18 lines · Jens Axboe
Quoted 11 lines · David Chinner
Quoted 3 lines · Jens Axboe
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O.....
Quoted 5 lines
FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
Quoted 6 lines
The change was needed to safely split
max_sectorsinto two sane parts:- The soft value, ->max_sectors, that holds a sane default of maximum io size. The main issue we want to prevent is filling the queue with huge amounts of io, both from a pinning POV but also from user latency reasons.
Got any data that you can share with us?
Quoted 7 lines
Wrt latency, is the problem to do with large requests causing short term latency? I thought that latency minimisation is the job of the I/O scheduler, so if this is the case, doesn't this indicate a deficiency of the I/O scheduler? e.g. the I/o scheduler could split large requests to reduce latency, just like you merge adjacent requests to reduce the number of I/Os and keep overall latency low...
What would be the point of allowing you to build these large ios only to split them up again? It's not only painfully inefficient, it's also tricky to do since it requires extra allocations and no good place to do it.
Quoted 7 lines
And as to the pinning problem - if you have a problem with too much memory in the I/O queues, then the I/O queues are too deep or they need to be throttled based on the amount of data in them as well as the number of queued requests. It's the method or configuration of the I/O scheduler being used to throttle requests that is deficient here, not the fact that a filesystem is building large I/Os.
Quoted 5 lines
It seems to me that you've crippled the block layer to solve very specific problems that most people don't see. I haven't seen pinning problems since the cfq request queue depth was reduced from 8192 to 128 and all the I/O latency problems I see are to do with multiple small I/Os being issued rather than a single large I/O....
I haven't crippled anything, in fact it's a lot more flexible now. I don't know why you are whining, you have the exact same possibilities to do large ios as you did before. Up max_sectors_kb.
8192 requests was nasty. And guess what, any recent ide or sata drive should have 32768 as max_sectors_kb value. Multiply that by 128 * 2 (nr_requests * 2) and you have 8 times as much memory pinned in the queue as 8192 requests did for IDE.
Quoted 7 lines
Quoted 4 lines
- The hard value, ->max_hw_sectors. Previously, there was no real clear definition of what ->max_sectors was supposed to do. We couldn't increase it to fit the hardware limits of most hardware, because that would hurt us latency/memory wise.
But we did have
max_sectors=max_hw_sectorsand I can't say that I've seen any evidence that it hurt us latency/memory wise.
Well good for you.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.You mean
/sys/block/sd*/max_sector_kb?
Quoted 7 lines
Quoted 1 line
Exactly.
Not happy. Now, instead of having a default config that works just fine, we've got to change the config of every block device on every boot on every machine we sell. This is a big deal when you're talking about machines with _thousands_ of block devices on them all needing to have their defaults changed.
Oh please, it's a simple operation. I doubt you put monkeys in front of the machines doing this manually.
Quoted 3 lines
BTW, can you point me to the discussion(s) that lead to this mod so I can catch up on this quickly? I can't find anything on lkml or linux-fsdevel about it....
See the postings from Mike Christie that led to the patches containing this. WRT the max_sectors/max_hw_sectors splitup, see discussions from the -RT people on long completion run times on large requests.
Signature · 1 line
Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 10 lines · David Chinner
Quoted 6 lines · Jens Axboe
The change was needed to safely split
max_sectorsinto two sane parts:- The soft value, ->max_sectors, that holds a sane default of maximum io size. The main issue we want to prevent is filling the queue with huge amounts of io, both from a pinning POV but also from user latency reasons.
Got any data that you can share with us?
Quoted 7 lines
Wrt latency, is the problem to do with large requests causing short term latency? I thought that latency minimisation is the job of the I/O scheduler, so if this is the case, doesn't this indicate a deficiency of the I/O scheduler? e.g. the I/o scheduler could split large requests to reduce latency, just like you merge adjacent requests to reduce the number of I/Os and keep overall latency low...
Quoted 2 lines
What would be the point of allowing you to build these large ios only to split them up again?
Filesystems have good reasons to issue large I/Os. Large I/Os for XFS mean less locking and filesystem structure traversal (i.e CPU usage) for a given load, we execute fewer and larger allocations, we get better parallelism and scalability, less transactions are required which means less log writes to the device, etc. There's a few second order filesystem effects that I can think of like this that result in reduced filesystem I/O load on the block device...
AFAICT, the latency issue you talk about is not a filesystem issue but a block layer issue, so my question is why you saw this as something the filesystems were doing wrong rather than a deficiency with the I/O scheduler. I'm not advocating you do what I said, I was illustrating a possible alternate approach to reducing latency in the block layer when the filesystem issues large I/Os.
FWIW, I'd much prefer to do 2 concurrent 2MB I/Os than issue 2x(4x512KB) I/Os and hope that we don't get 8 seeks instead of 2. Larger I/Os give far more consistent performance than the equivalent throughput in small I/Os. On loads where latency matters, consistent, predictable latency is preferable even if it means a higher baseline latency.
Quoted 3 lines
it's also
tricky to do since it requires extra allocations and no good place to do it.
Allocation fails - means OOM, latency is already shot to pieces and you can simply ship it to disk as it stands. And i'd think that the place to split it would be in the merge function. My guess is that dm and md would provide plenty of examples of what to do ;)
Like I said though, it was simply an example....
Quoted 13 lines
Quoted 7 lines
And as to the pinning problem - if you have a problem with too much memory in the I/O queues, then the I/O queues are too deep or they need to be throttled based on the amount of data in them as well as the number of queued requests. It's the method or configuration of the I/O scheduler being used to throttle requests that is deficient here, not the fact that a filesystem is building large I/Os.
Quoted 5 lines
It seems to me that you've crippled the block layer to solve very specific problems that most people don't see. I haven't seen pinning problems since the cfq request queue depth was reduced from 8192 to 128 and all the I/O latency problems I see are to do with multiple small I/Os being issued rather than a single large I/O....
Quoted 5 lines
I haven't crippled anything, in fact it's a lot more flexible now. I don't know why you are whining, you have the exact same possibilities to do large ios as you did before. Up
max_sectors_kb.8192 requests was nasty.
I still have the scars.....
Quoted 2 lines
And guess what, any recent ide or sata drive
should have 32768 asmax_sectors_kbvalue.
Please correct me if I'm wrong - my understanding is that a max_sectors_kb this large is mostly irrelevant because the maximum size of an single I/O linux can support:
#define BIO_MAX_PAGES (256)
On a 4k page machine, we've already got a maximum of 1MB for an I/O.
And looking at at max_hw_segments or max_phys_segments, they both restrict the size of the bio. Seems that most devices set them to 128 or 256 as well.
That means that on a 4k page machine 512k or 1MB are the most common largest possible I/O sizes and hence the maximum amount of memory pinned is 32 or 64x smaller than your original claim.
Also If that is correct, then capping max_sectors to 512KB can't do much for reducing I/O latency or the amount of pinned memory on these systems because they couldn't issue I/O much larger than this.
So I'm still not understanding the rationale behind the new default setting of max_sectors....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner R&D Software Enginner SGI Australian Software Group
Quoted 23 lines · David Chinner
Quoted 19 lines · Jens Axboe
Quoted 10 lines · David Chinner
Quoted 6 lines · Jens Axboe
The change was needed to safely split
max_sectorsinto two sane parts:- The soft value, ->max_sectors, that holds a sane default of maximum io size. The main issue we want to prevent is filling the queue with huge amounts of io, both from a pinning POV but also from user latency reasons.
Got any data that you can share with us?
Quoted 7 lines
Wrt latency, is the problem to do with large requests causing short term latency? I thought that latency minimisation is the job of the I/O scheduler, so if this is the case, doesn't this indicate a deficiency of the I/O scheduler? e.g. the I/o scheduler could split large requests to reduce latency, just like you merge adjacent requests to reduce the number of I/Os and keep overall latency low...
Quoted 2 lines
What would be the point of allowing you to build these large ios only to split them up again?
Quoted 7 lines
Filesystems have good reasons to issue large I/Os. Large I/Os for XFS mean less locking and filesystem structure traversal (i.e CPU usage) for a given load, we execute fewer and larger allocations, we get better parallelism and scalability, less transactions are required which means less log writes to the device, etc. There's a few second order filesystem effects that I can think of like this that result in reduced filesystem I/O load on the block device...
It's pretty clear that where you are coming from - the big server camp. And yes, for those cases you pretty much always want huge ios. And yes, you can get those, just set the exposed variable and be done with it. I don't know why you are still making an issue of this or debating it...
Quoted 6 lines
AFAICT, the latency issue you talk about is not a filesystem issue but a block layer issue, so my question is why you saw this as something the filesystems were doing wrong rather than a deficiency with the I/O scheduler. I'm not advocating you do what I said, I was illustrating a possible alternate approach to reducing latency in the block layer when the filesystem issues large I/Os.
It's actually a colaboration between the block layer / io scheduler and the vm. The vm to some extent relies on the block layer throttling to not have gigabytes or io in progress. I'm not too fond of that, but that's the way it still is.
Quoted 6 lines
FWIW, I'd much prefer to do 2 concurrent 2MB I/Os than issue 2x(4x512KB) I/Os and hope that we don't get 8 seeks instead of 2. Larger I/Os give far more consistent performance than the equivalent throughput in small I/Os. On loads where latency matters, consistent, predictable latency is preferable even if it means a higher baseline latency.
I agree. For most people it doesn't matter though, and the usecs spent in completion is more important as it gives them skipless audio and whatnot.
Quoted 10 lines
Quoted 3 lines
it's also
tricky to do since it requires extra allocations and no good place to do it.Allocation fails - means OOM, latency is already shot to pieces and you can simply ship it to disk as it stands. And i'd think that the place to split it would be in the merge function. My guess is that dm and md would provide plenty of examples of what to do ;)
Like I said though, it was simply an example....
It was a joke, there's no way on earth I'd ever add something like this. It's pointless. Splitting is actually pretty tricky to get right (I'm sure if you'd ever looked at dm splitting you would not be making such suggestions). And since we are building these large ios in the first place, the way to go is naturally to _disallow_ building something we would split up later.
Take a look at how the buildup works, dm/md have hooks to say yay or nay to adding another page to a bio.
Quoted 10 lines
Quoted 2 lines
And guess what, any recent ide or sata drive
should have 32768 asmax_sectors_kbvalue.Please correct me if I'm wrong - my understanding is that a
max_sectors_kbthis large is mostly irrelevant because the maximum size of an single I/O linux can support:#define
BIO_MAX_PAGES(256)On a 4k page machine, we've already got a maximum of 1MB for an I/O.
Not quite true. It's the largest size of a bio, but you can have multiple bios tied to a request.
Quoted 3 lines
And looking at at
max_hw_segmentsormax_phys_segments, they both restrict the size of the bio. Seems that most devices set them to 128 or 256 as well.
We can never exceed the driver given value of course. If the hardware can do more than it advertises, then it would be a good idea to adjust those values. Don't forget that a segment may be larger than a page.
Quoted 7 lines
That means that on a 4k page machine 512k or 1MB are the most common largest possible I/O sizes and hence the maximum amount of memory pinned is 32 or 64x smaller than your original claim.
Also If that is correct, then capping
max_sectorsto 512KB can't do much for reducing I/O latency or the amount of pinned memory on these systems because they couldn't issue I/O much larger than this.
Most drivers don't set a true max_hw_sectors yet, what I stated was for what SATA (or lba48 on IDE, but that is limited by 256 segments anyway) can support. The reason they don't set it yet is because this soft/hard value split up is still quite fresh and drivers were told in the past to not set huge max_sectors values because of this.
Quoted 2 lines
So I'm still not understanding the rationale behind the new default setting of
max_sectors....
Sorry I'm not making myself clearer. You still seem to not understand exactly how the request buildup or limitations work, perhaps you should start there. As to the problem in general, seems to me you are making a big deal out of a small problem - a problem that can easily be rectified in user space by just setting the right value.
Signature · 1 line
Jens Axboe
Quoted 2 lines · David Chinner
Got any data that you can share with us?
The thread was from July 2004 and was called:
Re: [linux-audio-dev] Re: [announce]
[patch] Voluntary Kernel Preemption
Patch
Also some info in thread:
Re: [patch]
voluntary-preempt-2.6.8-rc2-M5
Quoted 8 lines
Wrt latency, is the problem to do with large requests causing short term latency? I thought that latency minimisation is the job of the I/O scheduler, so if this is the case, doesn't this indicate a deficiency of the I/O scheduler? e.g. the I/o scheduler could split large requests to reduce latency, just like you merge adjacent requests to reduce the number of I/Os and keep overall latency low...
I think you are talking past each other - Jens is referring to scheduler latency, not IO latency.
Lee