weird throughput on write to SATA disk
When writing data to /dev/sdb (that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.
This happens on two different SATA disk one samsung and one seagate and on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external esata connector. Controller is 00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)
What could be the reason for this ?? reading is no problem.
Quoted 4 lines · kenneth johansson
When writing data to
/dev/sdb(that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.
Is the application using "O_DIRECT"?
If not, could be issue with how VM writeback is (not) working.
Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)
grant
Quoted 10 lines
This happens on two different SATA disk one samsung and one seagate and on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external esata connector. Controller is 00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)
What could be the reason for this ?? reading is no problem.
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson
When writing data to
/dev/sdb(that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.Is the application using "
O_DIRECT"?
no just open then write no fancy stuff
Quoted 1 line
If not, could be issue with how VM writeback is (not) working.
the access pattern is the simplest possible. Is there any knobs I could try to adjust.
Quoted 2 lines
Can you reproduce this using fio or even "dd"?
(Seegit://git.kernel.dk/fio)
dd is having the same effect.
Quoted 13 lines · kenneth johansson
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson
When writing data to
/dev/sdb(that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.Is the application using "
O_DIRECT"?no just open then write no fancy stuff
Quoted 1 line
If not, could be issue with how VM writeback is (not) working.
Quoted 7 lines
the access pattern is the simplest possible. Is there any knobs I could try to adjust.
Quoted 2 lines
Can you reproduce this using fio or even "dd"?
(Seegit://git.kernel.dk/fio)dd is having the same effect.
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
Or try with "sgp_dd"? It's part of sg3-utils package.
thanks,
grant
On Mon, Oct 13, 2008 at 2:34 PM, Grant Grundler <[email protected]> wrote:
...
Quoted 1 line
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
sorry...meant to type "if=/dev/zero"...I don't recall if /dev/null will work the same.
grant
Quoted 17 lines · Grant Grundler
Quoted 16 lines · kenneth johansson
Quoted 7 lines · Grant Grundler
Quoted 4 lines · kenneth johansson
When writing data to
/dev/sdb(that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.Is the application using "
O_DIRECT"?no just open then write no fancy stuff
Quoted 1 line
If not, could be issue with how VM writeback is (not) working.
the access pattern is the simplest possible. Is there any knobs I could try to adjust.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Can you reproduce this using fio or even "dd"?
(Seegit://git.kernel.dk/fio)dd is having the same effect.
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
I changed to use O_DIRECT and it's much more consistent now. 69-75 with 74 about 95% of the time. the disk is supposed to have 105 115 sustained data rate.
On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...
Quoted 4 lines
Quoted 1 line
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
I changed to use
O_DIRECTand it's much more consistent now. 69-75 with 74 about 95% of the time.
That's low but it could be worse. Many things can contribute to slow disks. Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger). sgp_dd is better since it is multi-threaded. fio has all the right knobs to test disk perf and measure it properly.
Quoted 2 lines
the disk is supposed to have 105 115 sustained
data rate.
Where did 105-115 number come from?
The best I've seen to date with 7200 rpm drives was 108MB/s on the outside diameter. That degrades slowly until about 60-70% towards the inside diameter and then drops off dramatically (down to something like 50MB/s).
hth,
grant
Quoted 9 lines · Grant Grundler
On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...Quoted 4 lines
Quoted 1 line
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
I changed to use
O_DIRECTand it's much more consistent now. 69-75 with 74 about 95% of the time.That's low but it could be worse. Many things can contribute to slow disks.
Well my main problem was that it was fluctuating so much that can't be right.
Quoted 4 lines
Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger).
All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using O_DIRECT. I'm doing a full disk write now to get a reference plot. Not using O_DIRECT should be almost identical.
Quoted 4 lines
Quoted 2 lines
the disk is supposed to have 105 115 sustained
data rate.Where did 105-115 number come from?
datasheet . they listed two drives in the same column so that was not the range it was sustained OD and the model I have max out at 105.
Quoted 21 lines · kenneth johansson
Quoted 7 lines · Grant Grundler
On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...Quoted 3 lines
Quoted 1 line
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
I changed to use
O_DIRECTand it's much more consistent now. 69-75 with 74 about 95% of the time.That's low but it could be worse. Many things can contribute to slow disks.
Well my main problem was that it was fluctuating so much that can't be right.
Quoted 4 lines
Favorites are overtemp (See SMART field 194) and vibration (no measurement possible w/o special equipment). "dd" isn't exactly a performance application until one uses really big block sizes (1MB or larger).
All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using
O_DIRECT. I'm doing a full disk write now to get a reference plot. Not usingO_DIRECTshould be almost identical.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
the disk is supposed to have 105 115 sustained
data rate.Where did 105-115 number come from?
datasheet . they listed two drives in the same column so that was not the range it was sustained OD and the model I have max out at 105.
Can you try deadline scheduler?
Signature · 1 line
tejun
Quoted 4 lines · Tejun Heo
Quoted 3 lines · kenneth johansson
Quoted 2 lines · Grant Grundler
On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <[email protected]> wrote:
...
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
I changed to use
O_DIRECTand it's much more consistent now. 69-75 with 74 about 95% of the time.That's low but it could be worse. Many things can contribute to slow disks.
Well my main problem was that it was fluctuating so much that can't be right.
Can you try deadline scheduler?
I did but it was so bad I restarted my computer and did it again. Same result complete disaster. so fas O_DIRECT is the only one that is as it should be.
Alan mentioned a patch by Arjan van de Ven anyone knows how to find that one?
On Tue, Oct 14, 2008 at 2:14 AM, kenneth johansson <[email protected]> wrote:
...
Quoted 3 lines
All my numbers comes from using 1MB blocks. but I'm not using dd. After removing the logic that actually put some data into the buffers I do get about 105 MB/sec using
O_DIRECT.
Ok. Good.
Quoted 1 line
I'm doing a full disk write now to get a reference plot.
See http://hdperf.sourceforge.net/ and http://nathan.laredo.name/hdperf/
Simple tool to (relatively) quickly measure the perf across the disk platter.
Quoted 1 line
Not using
O_DIRECTshould be almost identical.
*should* - but there are known problems with writeback.
Some of them are described nicely by Dave Chinner as "Random Thought #3":
http://iou.parisc-linux.org/lsf2008/linux_storage_scalability-Dave_Chinner.odp
Look for "writeback" for a summary of the discussion:
http://iou.parisc-linux.org/lsf2008/SUMMARY-Storage.txt
hth,
grant
Quoted 8 lines · On Mon, 13 Oct 2008 14:07:15 -0700 "Grant Grundler" <[email protected]> wrote:
Quoted 4 lines · kenneth johansson
When writing data to
/dev/sdb(that is the whole disk and no filesystem) it starts out ok around 100MB/sec but soon end up doing considerably worse. For prolonged times lasting several minutes it hovers in the 20-30MB/sec.Is the application using "
O_DIRECT"?
If not, could be issue with how VM writeback is (not) working.
Or just mashing up the I/O horribly. Would also be worth turning the CFQ scheduler off or applying Arjan van de Ven's patches to fix the ioprio of the kernel writeout threads. That last patch makes a huge difference here.
Alan