Skip to contentSkip to conversation

Direct I/O bio size regression

PATCH Al Boldi 3 participants 5 messages Apr 24, 2006 – Apr 25, 2006

lore
Share & tools
lore

Al Boldi

author
#0 lore raw
Quoted 57 lines · David Chinner
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 22 lines

Index: 2.6.x-xfs-new/fs/bio.c
===================================================================
--- 2.6.x-xfs-new.orig/fs/bio.c 2006-02-06 11:57:50.000000000
+1100 +++ 2.6.x-xfs-new/fs/bio.c 2006-04-24
15:46:16.849484424 +1000 @@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_device request_queue_t *q = bdev_get_queue(bdev);
int nr_pages;

- nr_pages = ((q->max_sectors << 9) + PAGE_SIZE - 1) >> PAGE_SHIFT;
+ nr_pages = ((q->max_hw_sectors << 9) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (nr_pages > q->max_phys_segments)

          nr_pages = q->max_phys_segments;

if (nr_pages > q->max_hw_segments)

(quoted diff) +1 -1
@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct
           unsigned int offset)
 {
  struct request_queue *q = bdev_get_queue(bio->bi_bdev);
- return __bio_add_page(q, bio, page, len, offset, q->max_sectors);
+ return __bio_add_page(q, bio, page, len, offset,
 q->max_hw_sectors); }
 
Quoted 6 lines
Quoted 3 lines

Review int bio_add_page(struct bio *bio, struct · line 452

Quoted 1 line
(quoted diff) +0 -0
 struct bio_map_data {

Clearly correct, I'll make sure this gets merged right away.

Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.

I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O..... FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....

Quoted 3 lines

That's why we
have a bio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.

bio_add_pc_page() requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio from sg_io(). The comment for bio_add_pc_page() says for use with REQ_PC queues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?

Also, it seems to me that using bio_add_pc_page() in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.

Quoted 4 lines
Quoted 2 lines

The best approach is probably to tune max_sectors on the system itself. That's why it is exposed, after all.

You mean /sys/block/sd*/max_sector_kb?

On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.

Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.

Tuning max_sectors_kb to ~192 only stabilizes this situation.

Would you think that this points to some underlying bio/queue problem?

Thanks!

--
Al

#1 lore raw
Quoted 61 lines · Al Boldi
Quoted 60 lines · David Chinner
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 22 lines

Index: 2.6.x-xfs-new/fs/bio.c
===================================================================
--- 2.6.x-xfs-new.orig/fs/bio.c 2006-02-06 11:57:50.000000000
+1100 +++ 2.6.x-xfs-new/fs/bio.c 2006-04-24
15:46:16.849484424 +1000 @@ -304,7 +304,7 @@ int bio_get_nr_vecs(struct block_device request_queue_t *q = bdev_get_queue(bdev);
int nr_pages;

- nr_pages = ((q->max_sectors << 9) + PAGE_SIZE - 1) >> PAGE_SHIFT;
+ nr_pages = ((q->max_hw_sectors << 9) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (nr_pages > q->max_phys_segments)

          nr_pages = q->max_phys_segments;

if (nr_pages > q->max_hw_segments)

(quoted diff) +1 -1
@@ -446,7 +446,7 @@ int bio_add_page(struct bio *bio, struct
           unsigned int offset)
 {
  struct request_queue *q = bdev_get_queue(bio->bi_bdev);
- return __bio_add_page(q, bio, page, len, offset, q->max_sectors);
+ return __bio_add_page(q, bio, page, len, offset,
 q->max_hw_sectors); }
 
Quoted 6 lines
Quoted 3 lines

Review int bio_add_page(struct bio *bio, struct · line 452

Quoted 1 line
(quoted diff) +0 -0
 struct bio_map_data {

Clearly correct, I'll make sure this gets merged right away.

Spoke too soon... The last part is actually on purpose, to prevent really huge requests as part of normal file system IO.

I don't understand why this was considered necessary. It doesn't appear to be explained in any of the code so can you explain the problem that large filesystem I/Os pose to the block layer? We _need_ to be able to drive really huge requests from the filesystem down to the disks, especially for direct I/O..... FWIW, we've just got XFS to the point where we could issue large I/Os (up to 8MB on 16k pages) with a default configuration kernel and filesystem using md+dm on an Altix. That makes an artificial 512KB filesystem I/O size limit a pretty major step backwards in terms of performance for default configs.....

Quoted 3 lines

That's why we
have a bio_add_pc_page(). The first hunk may cause things to not work optimally then if we don't apply the last hunk.

bio_add_pc_page() requires a request queue to be passed to it. It's called only from scsi layers in the context of mapping pages into a bio from sg_io(). The comment for bio_add_pc_page() says for use with REQ_PC queues only, and that appears to only be used by ide-cd cdroms. Is that comment correct?

Also, it seems to me that using bio_add_pc_page() in a filesystem or in the generic direct i/o code seems like a gross layering violation to me because they are supposed to know nothing about request queues.

Quoted 2 lines

The best approach is probably to tune max_sectors on the system itself. That's why it is exposed, after all.

Quoted 8 lines
Quoted 1 line

You mean /sys/block/sd*/max_sector_kb?

On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.

Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.

Tuning max_sectors_kb to ~192 only stabilizes this situation.

That sounds pretty strange. Do you have a test case?

Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 9 lines · Jens Axboe
Quoted 6 lines · Al Boldi

On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.

Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.

Tuning max_sectors_kb to ~192 only stabilizes this situation.

That sounds pretty strange. Do you have a test case?

I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:

1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat /dev/hda > /dev/null &
4. # cat /dev/hda > /dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.

Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.

Setting max_sectors_kb to ~192 seems to inhibit this problem.

Thanks!

--
Al

#3 lore raw
Quoted 16 lines · Al Boldi
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Al Boldi

On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.

Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.

Tuning max_sectors_kb to ~192 only stabilizes this situation.

That sounds pretty strange. Do you have a test case?

I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:

Quoted 8 lines

1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat /dev/hda > /dev/null &
4. # cat /dev/hda > /dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.

Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.

*random* disk access?

What io scheduler are you using? Can you try with as?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
author
#4 lore raw
Quoted 24 lines · Nick Piggin
Quoted 23 lines · Al Boldi
Quoted 9 lines · Jens Axboe
Quoted 6 lines · Al Boldi

On my system max_hw_sectors_kb is fixed at 1024, and max_sectors_kb defaults to 512, which leads to terribly fluctuating thruput.

Setting max_sectors_kb = max_hw_sectors_kb makes things even worse.

Tuning max_sectors_kb to ~192 only stabilizes this situation.

That sounds pretty strange. Do you have a test case?

I would think that, if you could get your hands on some hw that defaults to the same values, you may easily see the same problem by doing this:

1. # vmstat 1 (or some other bio mon)
2. < change vt >
3. # cat /dev/hda > /dev/null &
4. # cat /dev/hda > /dev/null
Let this second cat run for a sec, then ^C.
Depending on your hw specifics the bio should either go up or down by a factor of 2 (on my system 25mb/s-48mb/s). You may have to repeat step 4 a few times to aggravate the situation.

Quoted 6 lines
Quoted 2 lines

Note that this is not specific to cat, but can also be observed during normal random disk access, although not in a controlled manner.

*random* disk access?

What io scheduler are you using? Can you try with as?

Same w/ deadline, as, and cfq.

Thanks!

--
Al