Skip to contentSkip to conversation

weird throughput on write to SATA disk

DISCUSSION kenneth johansson 4 participants 12 messages Oct 13, 2008 – Oct 14, 2008 libata

lore
Share & tools
lore
#0 lore raw

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

This happens on two different SATA disk one samsung and one seagate and on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external esata connector. Controller is 00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)

What could be the reason for this ?? reading is no problem.

#1 lore raw
Quoted 4 lines · kenneth johansson

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

Is the application using "O_DIRECT"?
If not, could be issue with how VM writeback is (not) working.

Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)

grant

Quoted 10 lines

This happens on two different SATA disk one samsung and one seagate and on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external esata connector. Controller is 00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)

What could be the reason for this ?? reading is no problem.

#2 lore raw
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

Is the application using "O_DIRECT"?

no just open then write no fancy stuff

Quoted 1 line

If not, could be issue with how VM writeback is (not) working.

the access pattern is the simplest possible. Is there any knobs I could try to adjust.

Quoted 2 lines

Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)

dd is having the same effect.

#3 lore raw
Quoted 13 lines · kenneth johansson
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

Is the application using "O_DIRECT"?

no just open then write no fancy stuff

Quoted 1 line

If not, could be issue with how VM writeback is (not) working.

Quoted 7 lines

the access pattern is the simplest possible. Is there any knobs I could try to adjust.

Quoted 2 lines

Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)

dd is having the same effect.

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

Or try with "sgp_dd"? It's part of sg3-utils package.

thanks,
grant

#4 lore raw

On Mon, Oct 13, 2008 at 2:34 PM, Grant Grundler <[email protected]> wrote:
...

Quoted 1 line

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

sorry...meant to type "if=/dev/zero"...I don't recall if /dev/null will work the same.

grant

#5 lore raw
In reply to Grant Grundler
Quoted 17 lines · Grant Grundler
Quoted 16 lines · kenneth johansson
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

Is the application using "O_DIRECT"?

no just open then write no fancy stuff

Quoted 1 line

If not, could be issue with how VM writeback is (not) working.

the access pattern is the simplest possible. Is there any knobs I could try to adjust.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)

dd is having the same effect.

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time. the disk is supposed to have 105 115 sustained data rate.

#6 lore raw

On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...

Quoted 4 lines
Quoted 1 line

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time.

That's low but it could be worse. Many things can contribute to slow disks. Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger). sgp_dd is better since it is multi-threaded. fio has all the right knobs to test disk perf and measure it properly.

Quoted 2 lines

the disk is supposed to have 105 115 sustained
data rate.

Where did 105-115 number come from?

The best I've seen to date with 7200 rpm drives was 108MB/s on the outside diameter. That degrades slowly until about 60-70% towards the inside diameter and then drops off dramatically (down to something like 50MB/s).

hth,
grant

#7 lore raw
Quoted 9 lines · Grant Grundler

On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...

Quoted 4 lines
Quoted 1 line

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time.

That's low but it could be worse. Many things can contribute to slow disks.

Well my main problem was that it was fluctuating so much that can't be right.

Quoted 4 lines

Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger).

All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using O_DIRECT. I'm doing a full disk write now to get a reference plot. Not using O_DIRECT should be almost identical.

Quoted 4 lines
Quoted 2 lines

the disk is supposed to have 105 115 sustained
data rate.

Where did 105-115 number come from?

datasheet . they listed two drives in the same column so that was not the range it was sustained OD and the model I have max out at 105.

#8 lore raw
Quoted 21 lines · kenneth johansson
Quoted 7 lines · Grant Grundler

On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...

Quoted 3 lines
Quoted 1 line

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time.

That's low but it could be worse. Many things can contribute to slow disks.

Well my main problem was that it was fluctuating so much that can't be right.

Quoted 4 lines

Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger).

All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using O_DIRECT. I'm doing a full disk write now to get a reference plot. Not using O_DIRECT should be almost identical.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

the disk is supposed to have 105 115 sustained
data rate.

Where did 105-115 number come from?

datasheet . they listed two drives in the same column so that was not the range it was sustained OD and the model I have max out at 105.

Can you try deadline scheduler?

Signature · 1 line
tejun
#9 lore raw
Quoted 4 lines · Tejun Heo
Quoted 3 lines · kenneth johansson
Quoted 2 lines · Grant Grundler

On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time.

That's low but it could be worse. Many things can contribute to slow disks.

Well my main problem was that it was fluctuating so much that can't be right.

Can you try deadline scheduler?

I did but it was so bad I restarted my computer and did it again. Same result complete disaster. so fas O_DIRECT is the only one that is as it should be.

Alan mentioned a patch by Arjan van de Ven anyone knows how to find that one?

#10 lore raw
In reply to kenneth johansson

On Tue, Oct 14, 2008 at 2:14 AM, kenneth johansson <[email protected]> wrote:
...

Quoted 3 lines

All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using O_DIRECT.

Ok. Good.

Quoted 1 line

I'm doing a full disk write now to get a reference plot.

See http://hdperf.sourceforge.net/ and http://nathan.laredo.name/hdperf/
Simple tool to (relatively) quickly measure the perf across the disk platter.

Quoted 1 line

Not using O_DIRECT should be almost identical.

*should* - but there are known problems with writeback.
Some of them are described nicely by Dave Chinner as "Random Thought #3":
http://iou.parisc-linux.org/lsf2008/linux_storage_scalability-Dave_Chinner.odp

Look for "writeback" for a summary of the discussion:

    http://iou.parisc-linux.org/lsf2008/SUMMARY-Storage.txt

hth,
grant

#11 lore raw
In reply to Grant Grundler
Quoted 8 lines · On Mon, 13 Oct 2008 14:07:15 -0700 "Grant Grundler" <[email protected]> wrote:
Quoted 4 lines · kenneth johansson

When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.

Is the application using "O_DIRECT"?
If not, could be issue with how VM writeback is (not) working.

Or just mashing up the I/O horribly. Would also be worth turning the CFQ scheduler off or applying Arjan van de Ven's patches to fix the ioprio of the kernel writeout threads. That last patch makes a huge difference here.

Alan