Very poor latency when using hard drive (raid1)
There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.
When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.
Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.
Did anyone seen such problem, where too look, what to test?
What could solve it (other then ionice on applications that I expect to use hard drive)?
Quoted 7 lines · On 15/04/13 11:59, [email protected] wrote:
There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.
When I write into them e.g. using dd from
/dev/zeroto a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.
Quoted 7 lines
Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.
Did anyone seen such problem, where too look, what to test?
What could solve it (other then ionice on applications that I expect to use hard drive)?
I got reply (by e-mail) suggesting to use XFS.
Thanks, possible for other/next server.
But I fell this should work correctly as well on ext4.
Quoted 13 lines · On Tue, 2013-04-16 at 08:49 +0200, [email protected] wrote:
Quoted 12 lines · On 15/04/13 11:59, [email protected] wrote:
There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.
When I write into them e.g. using dd from
/dev/zeroto a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.
Did anyone seen such problem, where too look, what to test?
Quoted 7 lines
Quoted 2 lines
What could solve it (other then ionice on applications that I expect to use hard drive)?
I got reply (by e-mail) suggesting to use XFS.
Thanks, possible for other/next server.But I fell this should work correctly as well on ext4.
It should not starve readers that badly, something is wrong.
You can try setting low_latency for the devices if you're using CFQ ioscheduler. For my box, that would be..
echo 1 > /sys/devices/pci0000:00/0000:00:1f.2/ata2/host0/target0:0:0/0:0:0:0/block/sda/queue/iosched/low_latency
Or, you can try a different scheduler. cat (blabla)/sda/queue/scheduler to see which choices are available, and echo your choice back to the file to select it.
-Mike
15.04.2013 13:59, [email protected] пишет:
Quoted 11 lines
There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.
When I write into them e.g. using dd from
/dev/zeroto a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.
Did anyone seen such problem, where too look, what to test?
This is typical, known for many years, issue.
Your dds are run against buffer cache, the same as used by all other regular accesses. So once it fills up, cached directories and the like are thrown away to make room for new cache space. So once you need something else, that something needs to be read from disk, which is busy together with the buffer cache.
Quoted 2 lines
What could solve it (other then ionice on applications that I expect to use hard drive)?
Just don't mix these two workloads. Or, if you really need to transfer large amount of data, use direct I/O (O_DIRECT) -- for dd it is iflag=direct or oflag=direct (depending on the I/O direction).
ionice wont help much.
Thanks,
/mjt
Quoted 2 lines · On Mon 15-04-13 11:59:59, [email protected] wrote:
There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.
Any possibility in trying a newer kernel? Like 3.8 / 3.9?
Quoted 3 lines
When I write into them e.g. using dd from
/dev/zeroto a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.
That shouldn't really happen. I presume you use the default IO scheduler?
i.e. CFQ?
Quoted 4 lines
Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.
Did anyone seen such problem, where too look, what to test?
Not in such a simple setting as yours.
Quoted 2 lines
What could solve it (other then ionice on applications that I expect to use hard drive)?
ionice wouldn't help because dd writes the data into page cache and from
there flusher thread writes them to disk. So IO happens in flusher context which won't be ioniced.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR