Skip to contentSkip to conversation

unfair io behaviour for high load interactive use still present in 2.6.31

DISCUSSION Daniel J Blueman 2 participants 3 messages Sep 15, 2009 – Sep 26, 2009 5 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 7 lines · Tobias Oetiker

Experts,

We run several busy NFS file servers with Areca HW Raid + LVM2 + ext3

We find that the read bandwidth falls dramatically as well as the response times going up to several seconds as soon as the system comes under heavy write strain.

It's worthwhile checking:
- that the ext3 filesystem starts at a stripe-aligned offset
- that the ext3 filesystem was created with the correct stripe-width
and stride (chunk) size
- due to the larger amount of memory, ext4 may be a big win (due to
delayed allocate), if you'll stay with a newer kernel
- if you have battery backup at the right levels:
- performance may be better mounting the ext3 filesystem with
'barrier=0'
- performance may improve mounting 'data=writeback'

Thanks,
Daniel
--
Daniel J Blueman

#2 lore raw

Hi Daniel,

Quoted 5 lines · Yesterday Daniel J Blueman
Quoted 4 lines · Tobias Oetiker

Experts,

We run several busy NFS file servers with Areca HW Raid + LVM2 + ext3

Quoted 7 lines
Quoted 3 lines

We find that the read bandwidth falls dramatically as well as the response times going up to several seconds as soon as the system comes under heavy write strain.

It's worthwhile checking:

- that the ext3 filesystem starts at a stripe-aligned offset

yep

Quoted 2 lines

- that the ext3 filesystem was created with the correct stripe-width and stride (chunk) size

yep

Quoted 2 lines

- due to the larger amount of memory, ext4 may be a big win (due to delayed allocate), if you'll stay with a newer kernel

ext4 does not seem to change much

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util

dm-18             0.00     0.00    0.00  316.60     0.00     1.24     8.00  2590.22   99.29   3.16 100.00
dm-18             0.00     0.00    1.60 1211.60     0.01     4.73     8.00   692.64 2925.31   0.76  92.00
dm-18             0.00     0.00    2.00 4541.40     0.01    17.74     8.00  1858.21  413.23   0.22  99.52
dm-18             0.00     0.00    0.00 1341.80     0.00     5.24     8.00  1084.59  239.59   0.68  90.96
dm-18             0.00     0.00   10.00 3739.80     0.04    14.61     8.00   914.32  447.29   0.25  93.12
dm-18             0.00     0.00    2.60 2474.20     0.01     9.66     8.00   537.33   23.88   0.36  89.36
dm-18             0.00     0.00    2.00 2569.00     0.01    10.04     8.00  1215.49  658.95   0.33  85.92
Quoted 3 lines

- if you have battery backup at the right levels:
- performance may be better mounting the ext3 filesystem with
'barrier=0'

the results look pretty much the same

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util

dm-18             0.00     0.00    1.40 9272.00     0.01    36.22     8.00   413.57  111.78   0.10  88.64
dm-18             0.00     0.00    4.20   71.80     0.02     0.28     8.00    78.24  471.06  12.83  97.52
dm-18             0.00     0.00    0.60 4183.20     0.00    16.34     8.00   988.32  216.66   0.24 100.00
dm-18             0.00     0.00    3.60  792.80     0.01     3.10     8.00  1098.60 1535.50   0.93  74.16
dm-18             0.00     0.00    1.60 6161.20     0.01    24.07     8.00   407.17   42.46   0.16  99.60
dm-18             0.00     0.00    0.00 2128.60     0.00     8.31     8.00  2497.56  368.99   0.47  99.92
dm-18             0.00     0.00    2.60 2657.80     0.01    10.38     8.00  1507.78  937.57   0.34  91.28
dm-18             0.00     0.00    5.80 8872.20     0.02    34.66     8.00   455.43  130.94   0.09  79.20
dm-18             0.00     0.00    2.20 4981.20     0.01    19.46     8.00  1058.84  245.39   0.19  92.24
Quoted 1 line

- performance may improve mounting 'data=writeback'

here the effect is that writes get delayed much longer ... it almost seems as if they got delayed until the reads were done ...

What I am really interested is not perfomance (the total throughput of the system is pretty ok) the problem is just that there are such big read delays under heavy load which makes interactive use pretty hard.

cheers
tobi

Signature · 2 lines
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch [email protected] ++41 62 775 9902 / sb: -9900
#7 lore raw
In reply to a message not in the archive
Quoted 34 lines · Tobias Oetiker
Quoted 11 lines · Today Corrado Zoccolo
Quoted 9 lines · Tobias Oetiker

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
---------------------------------------------------------------------------------------------------------
dm-18             0.00     0.00    0.00 2566.80     0.00    10.03     8.00  2224.74  737.62   0.39 100.00 dm-18             0.00     0.00    9.60  679.00     0.04     2.65     8.00   400.41 1029.73   1.35  92.80 dm-18             0.00     0.00    0.00 2080.80     0.00     8.13     8.00   906.58  456.45   0.48 100.00 dm-18             0.00     0.00    0.00 2349.20     0.00     9.18     8.00  1351.17  491.44   0.43 100.00 dm-18             0.00     0.00    3.80  665.60     0.01     2.60     8.00   906.72 1098.75   1.39  93.20 dm-18             0.00     0.00    0.00 1811.20     0.00     7.07     8.00  1008.23  725.34   0.55 100.00 dm-18             0.00     0.00    0.00 2632.60     0.00    10.28     8.00  1651.18  640.61   0.38 100.00

Quoted 6 lines

Good.
The high await is normal for writes, especially since you get so many queued requests.
can you post the output of "grep -r .  /sys/block/_device_/queue/" and iostat for your real devices?
This should not affect reads, that will preempt writes with cfq.

/sys/block/sdc/queue/nr_requests:128 /sys/block/sdc/queue/read_ahead_kb:128 /sys/block/sdc/queue/max_hw_sectors_kb:2048 /sys/block/sdc/queue/max_sectors_kb:512 /sys/block/sdc/queue/scheduler:noop anticipatory deadline [cfq] /sys/block/sdc/queue/hw_sector_size:512 /sys/block/sdc/queue/logical_block_size:512 /sys/block/sdc/queue/physical_block_size:512 /sys/block/sdc/queue/minimum_io_size:512 /sys/block/sdc/queue/optimal_io_size:0 /sys/block/sdc/queue/rotational:1 /sys/block/sdc/queue/nomerges:0 /sys/block/sdc/queue/rq_affinity:0 /sys/block/sdc/queue/iostats:1

Quoted 1 line

/sys/block/sdc/queue/iosched/quantum:4

As Mike points out [1], it may be worth trying a small quantum, though the benefit may depend on the behaviour of your SCSI RAID controller.

--- [1]

http://groups.google.com/group/linux.kernel/browse_thread/thread/49d2360a22af8898/60022a7e287af096#60022a7e287af096
--
Daniel J Blueman