Skip to contentSkip to conversation

mismatch_cnt and Raid6

DISCUSSION Andrew Falgout 4 participants 5 messages Apr 21, 2011 MD RAID

lore
Share & tools
lore

Andrew Falgout

author
#0 lore raw

I got an error last week from a new raid6 array about a mismatch_cnt. I did some reading online, performed a repair action on the array, performed a check action, and checked for the mismatch_cnt again. The number was greatly reduced, but it was still there. According to mdadm, everything appears to be working fine. All the drives are passing short tests on smartctl.

What is mismatch_cnt really? Should I even be concerned about this? The array is giving me 25-30MB/sec performance on an sshfs mount over the network. With a local copy I can see speeds of 50 to 60MB/sec.

Thanks,
Andrew Falgout

-----> inserting boring details here <-----------
==> checked for mismatch_cnt
cat /sys/block/md1/md/mismatch_cnt
7752
==> performed the repair this way:
echo "repair" >/sys/block/md1/md/sync_action
==> performed the check this way:
echo "check" >/sys/block/md1/md/sync_action
==> Array Details
Array Details as follows:
/dev/md1:

        Version : 1.2

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Thu Apr 21 07:35:08 2011
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : nas.falgout.lan:1 (local to host nas.falgout.lan) UUID : df03c833:90129ffd:123abdca:6b30c319
Events : 75072

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 81 1 active sync /dev/sdf1
2 8 97 2 active sync /dev/sdg1
3 8 113 3 active sync /dev/sdh1
4 8 129 4 active sync /dev/sdi1
5 8 145 5 active sync /dev/sdj1

Disk Details:
/dev/sda1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6735ec05:80890fa1:86e03b80:1f5de847

Internal Bitmap : 8 sectors from superblock
Update Time : Thu Apr 21 07:45:17 2011
Checksum : 99fdeb3f - correct
Events : 75072

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdf1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e18ce851:efc426b3:cadca5f9:df898fa7

Internal Bitmap : 8 sectors from superblock
Update Time : Thu Apr 21 07:45:17 2011
Checksum : d122a72e - correct
Events : 75072

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdg1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4cff4ab7:8e7c4312:1445828a:f0be7e89

Internal Bitmap : 8 sectors from superblock
Update Time : Thu Apr 21 07:45:17 2011
Checksum : 86d6e94 - correct
Events : 75072

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdh1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 05a949c4:ab94a3f4:ae25fe55:8034dbfb

Internal Bitmap : 8 sectors from superblock

    Update Time : Thu Apr 21 07:45:17 2011
       Checksum : 35a48696 - correct
         Events : 75072

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdi1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : ff0881e7:7e22804f:d0b57eb7:60872ff0

Internal Bitmap : 8 sectors from superblock

    Update Time : Thu Apr 21 07:45:17 2011
       Checksum : 98d5766 - correct
         Events : 75072

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdj1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : df03c833:90129ffd:123abdca:6b30c319
           Name : nas.falgout.lan:1  (local to host nas.falgout.lan)

Creation Time : Fri Mar 25 20:16:33 2011
Raid Level : raid6
Raid Devices : 6

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)

     Array Size : 15628095488 (7452.06 GiB 8001.58 GB)

Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9a200b64:483f52d0:a3d97d13:39ff95be

Internal Bitmap : 8 sectors from superblock

    Update Time : Thu Apr 21 07:45:17 2011
       Checksum : 314f2777 - correct
         Events : 75072

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing)

#1 lore raw
Quoted 8 lines · On Thu, 21 Apr 2011 08:00:39 -0500 Andrew Falgout <[email protected]>…

I got an error last week from a new raid6 array about a mismatch_cnt. I did some reading online, performed a repair action on the array, performed a check action, and checked for the mismatch_cnt again. The number was greatly reduced, but it was still there. According to mdadm, everything appears to be working fine. All the drives are passing short tests on smartctl.

What is mismatch_cnt really? Should I even be concerned about this?

Yes, you should be concerned.
mismatch_cnt is a count of sectors where the parity blocks don't match the data blocks.

The code doesn't check every sector individually. For raid5/6 it checks 4K at a time, so divide by 8, and that many 4K blocks are in doubt.

So something if going wrong somewhere.

I would run 'check' a few time and see if the number changes. If it goes down at all, then it looks like you occasionally get bad reads from a device.
If it only ever increases, then you are presumably getting bad writes sometimes.

You could:
- stop the array
- run sha1sum on each member disk, several times.
- if any one disk has an unstable result - check cabling, or replace the disk
- if more than one disk has an unstable result, replace the controller maybe.
- if all results are stable it must be a write-only problem - much harder to work with.

NeilBrown

Quoted 5 lines

The array is giving me 25-30MB/sec performance on an sshfs mount over the network. With a local copy I can see speeds of 50 to 60MB/sec.

Thanks,
Andrew Falgout

author
#2 lore raw

It takes about 23 hours per check if I don't do anything to the array, so it will be a while before I can get back to you with results. But thanks for the quick response.

./Andrew

Quoted 43 lines · NeilBrown
Quoted 8 lines · On Thu, 21 Apr 2011 08:00:39 -0500 Andrew Falgout<[email protected]>…

I got an error last week from a new raid6 array about a mismatch_cnt. I did some reading online, performed a repair action on the array, performed a check action, and checked for the mismatch_cnt again. The number was greatly reduced, but it was still there. According to mdadm, everything appears to be working fine. All the drives are passing short tests on smartctl.

What is mismatch_cnt really? Should I even be concerned about this?

Yes, you should be concerned.
mismatch_cnt is a count of sectors where the parity blocks don't match the data blocks.

The code doesn't check every sector individually. For raid5/6 it checks 4K at a time, so divide by 8, and that many 4K blocks are in doubt.

So something if going wrong somewhere.

I would run 'check' a few time and see if the number changes. If it goes down at all, then it looks like you occasionally get bad reads from a device.
If it only ever increases, then you are presumably getting bad writes sometimes.

You could:
- stop the array
- run sha1sum on each member disk, several times.
- if any one disk has an unstable result - check cabling, or replace the disk
- if more than one disk has an unstable result, replace the controller maybe.
- if all results are stable it must be a write-only problem - much harder to work with.

NeilBrown

Quoted 5 lines

The array is giving me 25-30MB/sec performance on an sshfs mount over the network. With a local copy I can see speeds of 50 to 60MB/sec.

Thanks,
Andrew Falgout

#3 lore raw
Quoted 3 lines · Andrew Falgout

It takes about 23 hours per check if I don't do anything to the array, so it will be a while before I can get back to you with results. But thanks for the quick response.

It may also be instructive to look at the full output of smartctl from all your drives - a short test may pass and the overall status be "PASSED" even on a very sick drive. You should also look in your system logs for ata errors to see which (if any) drive has given any read errors.

Cheers,

John.

#4 lore raw
In reply to Andrew Falgout
Quoted 10 lines · On Thu, 21 Apr 2011 08:00:39 -0500 Andrew Falgout <[email protected]>…

I got an error last week from a new raid6 array about a mismatch_cnt. I did some reading online, performed a repair action on the array, performed a check action, and checked for the mismatch_cnt again. The number was greatly reduced, but it was still there. According to mdadm, everything appears to be working fine. All the drives are passing short tests on smartctl.

What is mismatch_cnt really? Should I even be concerned about this? The array is giving me 25-30MB/sec performance on an sshfs mount over the network. With a local copy I can see speeds of 50 to 60MB/sec.

Hello,

What kind of SATA cards/controllers do you use?

Signature · 2 lines
With respect,
Roman