Skip to contentSkip to conversation

md raid6: slow resync if fs is mounted rw

DISCUSSION Jakob Unterwurzacher 3 participants 4 messages Feb 25, 2014 – Feb 26, 2014 MD RAID

lore
Share & tools
lore

Jakob Unterwurzacher

author
#0 lore raw

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel 3.12. Initial resync goes nice and fast at 130MB/s:

Quoted 10 lines
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]

      29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write! Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

I dumped the /sys/block/md1/md tree with the filesystem mounted ro and rw, the diff looks like this:

sync_speed 2014-02-25 17:33:31.250019324 +0100 line 1

Quoted 13 lines
# diff -u md.rw md.ro
[...]
stripe_cache_active 2014-02-25 17:33:31.250019324 +0100 +1 -1
@@ -1 +1 @@
-0
+256

diff -u md.rw/sync_speed md.ro/sync_speed

sync_speed 2014-02-25 17:33:31.250019324 +0100 +1 -1
@@ -1 +1 @@
-1052
+126244

So, stripe_cache_active is zero if the fs is mounted rw. Hmm.

Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.

Any idea what is going on?

Best regards,
Jakob Unterwurzacher

#1 lore raw
Quoted 39 lines · On Tue, 25 Feb 2014 23:55:15 +0100 Jakob Unterwurzacher <[email protected]>…

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel 3.12. Initial resync goes nice and fast at 130MB/s:

Quoted 10 lines
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]

      29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write! Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

I dumped the /sys/block/md1/md tree with the filesystem mounted ro and rw, the diff looks like this:

sync_speed 2014-02-25 17:33:31.250019324 +0100 line 1

Quoted 13 lines
# diff -u md.rw md.ro
[...]
stripe_cache_active 2014-02-25 17:33:31.250019324 +0100 +1 -1
@@ -1 +1 @@
-0
+256

diff -u md.rw/sync_speed md.ro/sync_speed

sync_speed 2014-02-25 17:33:31.250019324 +0100 +1 -1
@@ -1 +1 @@
-1052
+126244

So, stripe_cache_active is zero if the fs is mounted rw. Hmm.

Quoted 4 lines

Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.

Any idea what is going on?

Simplest explanation would be that ext4 is writing to the array occasionally. Maybe 1 block per second. That would be enough for md to decide that the array is busy and to back-off the resync.

Try
blktrace -w 20 /dev/md1

while the filesystem is mounted read-write, then
blkparse md1.blktrace*

do we if anything writes are going to the array.

NeilBrown

#2 lore raw
In reply to Jakob Unterwurzacher
Quoted 11 lines · Jakob Unterwurzacher

Dear list,

I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel 3.12. Initial resync goes nice and fast at 130MB/s:

Quoted 6 lines
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
       7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
       [=>...................]  resync =  7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec
Quoted 9 lines
Quoted 4 lines

md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]

       29280128 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>

...unless the ext4 filesystem on the array is mounted read-write! Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.

You are probably expeririencing an effect of this :

http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

Regards,
Brad

author
#3 lore raw
Quoted 3 lines · Brad Campbell

You are probably expeririencing an effect of this :

http://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Lazy_Initialization

That's it. ext4lazyinit shows up in the blktrace and also flashes up every now and then in iotop - I did not notice it before. I reformatted with "mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0" and the issue is gone. Of course, just waiting a bit with the filesystem mounted rw would have done it, too.

Thanks a lot Neil and Brad!
Jakob