Direct I/O bio size regression
Quoted 57 lines · David Chinner
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 22 lines
Index:
2.6.x-xfs-new/fs/bio.c
===================================================================
---2.6.x-xfs-new.orig/fs/bio.c2006-02-06 11:57:50.000000000
+1100 +++2.6.x-xfs-new/fs/bio.c2006-04-24
15:46:16.849484424 +1000 @@ -304,7 +304,7 @@ int bio_get_nr_vecs(structblock_devicerequest_queue_t*q =bdev_get_queue(bdev);
intnr_pages;-
nr_pages= ((q->max_sectors<< 9) +PAGE_SIZE- 1) >>PAGE_SHIFT;
+nr_pages= ((q->max_hw_sectors<< 9) +PAGE_SIZE- 1) >>PAGE_SHIFT; if (nr_pages>q->max_phys_segments)nr_pages = q->max_phys_segments;if (
nr_pages>q->max_hw_segments)(quoted diff) +1
-1@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) { struct request_queue *q = bdev_get_queue(bio->bi_bdev); - return __bio_add_page(q, bio, page, len, offset, q->max_sectors); + return __bio_add_page(q, bio, page, len, offset, q->max_hw_sectors); }Quoted 6 lines
Quoted 3 lines
Review int bio_add_page(struct bio *bio, struct · line 452
Quoted 1 line
(quoted diff) +0
-0struct bio_map_data {Clearly correct, I'll make sure this gets merged right away.
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O..... FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
Quoted 3 lines
That's why we
have abio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.
bio_add_pc_page()requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio fromsg_io(). The comment forbio_add_pc_page()says for use withREQ_PCqueues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?Also, it seems to me that using
bio_add_pc_page()in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.
Quoted 4 lines
Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.You mean
/sys/block/sd*/max_sector_kb?
On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.
Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.
Tuning max_sectors_kb to ~192 only stabilizes this situation.
Would you think that this points to some underlying bio/queue problem?
Thanks!
--
Al
Quoted 61 lines · Al Boldi
Quoted 60 lines · David Chinner
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 22 lines
Index:
2.6.x-xfs-new/fs/bio.c
===================================================================
---2.6.x-xfs-new.orig/fs/bio.c2006-02-06 11:57:50.000000000
+1100 +++2.6.x-xfs-new/fs/bio.c2006-04-24
15:46:16.849484424 +1000 @@ -304,7 +304,7 @@ int bio_get_nr_vecs(structblock_devicerequest_queue_t*q =bdev_get_queue(bdev);
intnr_pages;-
nr_pages= ((q->max_sectors<< 9) +PAGE_SIZE- 1) >>PAGE_SHIFT;
+nr_pages= ((q->max_hw_sectors<< 9) +PAGE_SIZE- 1) >>PAGE_SHIFT; if (nr_pages>q->max_phys_segments)nr_pages = q->max_phys_segments;if (
nr_pages>q->max_hw_segments)(quoted diff) +1
-1@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct unsigned int offset) { struct request_queue *q = bdev_get_queue(bio->bi_bdev); - return __bio_add_page(q, bio, page, len, offset, q->max_sectors); + return __bio_add_page(q, bio, page, len, offset, q->max_hw_sectors); }Quoted 6 lines
Quoted 3 lines
Review int bio_add_page(struct bio *bio, struct · line 452
Quoted 1 line
(quoted diff) +0
-0struct bio_map_data {Clearly correct, I'll make sure this gets merged right away.
Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.
I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O..... FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....
Quoted 3 lines
That's why we
have abio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.
bio_add_pc_page()requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio fromsg_io(). The comment forbio_add_pc_page()says for use withREQ_PCqueues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?Also, it seems to me that using
bio_add_pc_page()in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.Quoted 2 lines
The best approach is probably to tune
max_sectorson the system itself. That's why it is exposed, after all.
Quoted 8 lines
Quoted 1 line
You mean
/sys/block/sd*/max_sector_kb?On my system
max_hw_sectors_kbis fixed at 1024, andmax_sectors_kbdefaults to 512, which leads to terribly fluctuating thruput.Setting
max_sectors_kb=max_hw_sectors_kbmakes things even worse.Tuning
max_sectors_kbto ~192 only stabilizes this situation.
That sounds pretty strange. Do you have a test case?
Signature · 1 line
Jens Axboe
Quoted 9 lines · Jens Axboe
Quoted 6 lines · Al Boldi
On my system
max_hw_sectors_kbis fixed at 1024, andmax_sectors_kbdefaults to 512, which leads to terribly fluctuating thruput.Setting
max_sectors_kb=max_hw_sectors_kbmakes things even worse.Tuning
max_sectors_kbto ~192 only stabilizes this situation.That sounds pretty strange. Do you have a test case?
I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:
1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat /dev/hda > /dev/null &
4. # cat /dev/hda > /dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.
Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.
Setting max_sectors_kb to ~192 seems to inhibit this problem.
Thanks!
--
Al
Quoted 16 lines · Al Boldi
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Al Boldi
On my system
max_hw_sectors_kbis fixed at 1024, andmax_sectors_kbdefaults to 512, which leads to terribly fluctuating thruput.Setting
max_sectors_kb=max_hw_sectors_kbmakes things even worse.Tuning
max_sectors_kbto ~192 only stabilizes this situation.That sounds pretty strange. Do you have a test case?
I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:
Quoted 8 lines
1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat/dev/hda>/dev/null&
4. # cat/dev/hda>/dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.
*random* disk access?
What io scheduler are you using? Can you try with as?
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 24 lines · Nick Piggin
Quoted 23 lines · Al Boldi
Quoted 9 lines · Jens Axboe
Quoted 6 lines · Al Boldi
On my system
max_hw_sectors_kbis fixed at 1024, andmax_sectors_kbdefaults to 512, which leads to terribly fluctuating thruput.Setting
max_sectors_kb=max_hw_sectors_kbmakes things even worse.Tuning
max_sectors_kbto ~192 only stabilizes this situation.That sounds pretty strange. Do you have a test case?
I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:
1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat/dev/hda>/dev/null&
4. # cat/dev/hda>/dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.
Quoted 6 lines
Quoted 2 lines
Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.
*random* disk access?
What io scheduler are you using? Can you try with as?
Same w/ deadline, as, and cfq.
Thanks!
--
Al