Skip to contentSkip to conversation

Very poor latency when using hard drive (raid1)

DISCUSSION [email protected] 4 participants 5 messages Apr 15, 2013 – Apr 19, 2013

lore
Share & tools
lore
#0 lore raw

There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.

When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.

Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.

Did anyone seen such problem, where too look, what to test?

What could solve it (other then ionice on applications that I expect to use hard drive)?

#1 lore raw
Quoted 7 lines · On 15/04/13 11:59, [email protected] wrote:

There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.

When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.

Quoted 7 lines

Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.

Did anyone seen such problem, where too look, what to test?

What could solve it (other then ionice on applications that I expect to use hard drive)?

I got reply (by e-mail) suggesting to use XFS.
Thanks, possible for other/next server.

But I fell this should work correctly as well on ext4.

#2 lore raw
Quoted 13 lines · On Tue, 2013-04-16 at 08:49 +0200, [email protected] wrote:
Quoted 12 lines · On 15/04/13 11:59, [email protected] wrote:

There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.

When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.

Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.

Did anyone seen such problem, where too look, what to test?

Quoted 7 lines
Quoted 2 lines

What could solve it (other then ionice on applications that I expect to use hard drive)?

I got reply (by e-mail) suggesting to use XFS.
Thanks, possible for other/next server.

But I fell this should work correctly as well on ext4.

It should not starve readers that badly, something is wrong.

You can try setting low_latency for the devices if you're using CFQ ioscheduler. For my box, that would be..

echo 1 > /sys/devices/pci0000:00/0000:00:1f.2/ata2/host0/target0:0:0/0:0:0:0/block/sda/queue/iosched/low_latency

Or, you can try a different scheduler. cat (blabla)/sda/queue/scheduler to see which choices are available, and echo your choice back to the file to select it.

-Mike

#3 lore raw
In reply to [email protected]

15.04.2013 13:59, [email protected] пишет:

Quoted 11 lines

There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.

When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.

Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.

Did anyone seen such problem, where too look, what to test?

This is typical, known for many years, issue.

Your dds are run against buffer cache, the same as used by all other regular accesses. So once it fills up, cached directories and the like are thrown away to make room for new cache space. So once you need something else, that something needs to be read from disk, which is busy together with the buffer cache.

Quoted 2 lines

What could solve it (other then ionice on applications that I expect to use hard drive)?

Just don't mix these two workloads. Or, if you really need to transfer large amount of data, use direct I/O (O_DIRECT) -- for dd it is iflag=direct or oflag=direct (depending on the I/O direction).

ionice wont help much.

Thanks,

/mjt

#4 lore raw
In reply to [email protected]
Quoted 2 lines · On Mon 15-04-13 11:59:59, [email protected] wrote:

There are 2 hard drives (normal, magnetic) in software raid 1 on 3.2.41 kernel.

Any possibility in trying a newer kernel? Like 3.8 / 3.9?

Quoted 3 lines

When I write into them e.g. using dd from /dev/zero to a local file (ext4 on default settings), running 2 dd at once (writing two files) it starves all other programs that try to use the disk.

That shouldn't really happen. I presume you use the default IO scheduler?
i.e. CFQ?

Quoted 4 lines

Running ls on any directory on same disk (same fs btw), takes over half minute to execute, same for any other disk touching action.

Did anyone seen such problem, where too look, what to test?

Not in such a simple setting as yours.

Quoted 2 lines

What could solve it (other then ionice on applications that I expect to use hard drive)?

ionice wouldn't help because dd writes the data into page cache and from
there flusher thread writes them to disk. So IO happens in flusher context which won't be ioniced.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR