Skip to contentSkip to conversation

What exactly does echo check > /sys/block/mdX/md/sync_action do?

DISCUSSION Christian Schmidt 2 participants 5 messages Jan 9, 2011 – Jan 10, 2011 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi all,

As the subject says, I'm wondering what issuing the "check" command to a raid array does.
The wiki says it starts a full read of the raid array. However I wonder if all members, especially the parts of the drives containing the redundancy information, will be read, and possibly the validity of the redundancy data will be checked?

A possibly related question is: why did this member turn into "spare" role? The system was fully functional and in daily use for about a year. It was declared to be a four drive raid 5 with no spares. If I remember level 5 correctly there is no single drive for the redundancy data to avoid bottlenecks, right?

alpha md # mdadm --examine --verbose /dev/sdh2
/dev/sdh2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 172eb49b:03e62242:614d7ed3:1fb25f65

    Update Time : Sun Jan  9 19:55:09 2011
       Checksum : a991f168 - correct
         Events : 34

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)

Too bad that 1.2 superblocks don't contain the full array information like 0.90 did.

Regards,
Christian

#1 lore raw
Quoted 8 lines · On Sun, 09 Jan 2011 23:48:05 +0100 Christian Schmidt <[email protected]> wrote:

Hi all,

As the subject says, I'm wondering what issuing the "check" command to a raid array does.
The wiki says it starts a full read of the raid array. However I wonder if all members, especially the parts of the drives containing the redundancy information, will be read, and possibly the validity of the redundancy data will be checked?

May I suggest
man 4 md

and search for 'check'
???

md/sync_action
This can be used to monitor and control the resync/recovery pro- cess of MD. In particular, writing "check" here will cause the array to read all data block and check that they are consistent (e.g. parity is correct, or all mirror replicas are the same). Any discrepancies found are NOT corrected.

A count of problems found will be stored in md/mismatch_count.

Alternately, "repair" can be written which will cause the same check to be performed, but any errors will be corrected.

Finally, "idle" can be written to stop the check/repair process.

Does that answer your question?

A more recent man page says:

md arrays can be scrubbed by writing either check or repair to the file md/sync_action in the sysfs directory for the device.

Requesting a scrub will cause md to read every block on every device in the array, and check that the data is consistent. For RAID1 and RAID10, this means checking that the copies are identical. For RAID4, RAID5, RAID6 this means checking that the parity block is (or blocks are) correct.

If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block.

If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch.

Quoted 6 lines

A possibly related question is: why did this member turn into "spare" role? The system was fully functional and in daily use for about a year. It was declared to be a four drive raid 5 with no spares. If I remember level 5 correctly there is no single drive for the redundancy data to avoid bottlenecks, right?

One would need to see the history of the whole array, not just the current state of a single device, to be able to guess the reason for the current state.

And yes: RAID5 distributes the parity blocks to avoid bottlenecks.

Quoted 27 lines

alpha md # mdadm --examine --verbose /dev/sdh2
/dev/sdh2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 172eb49b:03e62242:614d7ed3:1fb25f65

    Update Time : Sun Jan  9 19:55:09 2011
       Checksum : a991f168 - correct
         Events : 34

         Layout : left-symmetric
     Chunk Size : 512K
Quoted 5 lines

Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)

Too bad that 1.2 superblocks don't contain the full array information like 0.90 did.

The extra information that 0.90 stored was not (and could not be) reliable.

This device thinks that that the array is functioning correctly with no failed devices, and that this device is a spare - presumably a 5th device? It doesn't know the names of the other devices (and if it thought it did, it could easily be wrong as names changed). What do the other devices think of the state of the array?

NeilBrown

#2 lore raw
Quoted 5 lines · NeilBrown
Quoted 2 lines · On Sun, 09 Jan 2011 23:48:05 +0100 Christian Schmidt <[email protected]> wrote:

Hi all,

Quoted 7 lines
Quoted 2 lines

As the subject says, I'm wondering what issuing the "check" command to a raid array does.

May I suggest
man 4 md

Does that answer your question?

Yes, indeed. Thanks.

Quoted 45 lines
Quoted 5 lines

A possibly related question is: why did this member turn into "spare" role? The system was fully functional and in daily use for about a year. It was declared to be a four drive raid 5 with no spares. If I remember level 5 correctly there is no single drive for the redundancy data to avoid bottlenecks, right?

One would need to see the history of the whole array, not just the current state of a single device, to be able to guess the reason for the current state.

And yes: RAID5 distributes the parity blocks to avoid bottlenecks.

Quoted 27 lines

alpha md # mdadm --examine --verbose /dev/sdh2
/dev/sdh2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 172eb49b:03e62242:614d7ed3:1fb25f65

    Update Time : Sun Jan  9 19:55:09 2011
       Checksum : a991f168 - correct
         Events : 34

         Layout : left-symmetric
     Chunk Size : 512K
Quoted 5 lines

Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)

Too bad that 1.2 superblocks don't contain the full array information like 0.90 did.

Quoted 7 lines

The extra information that 0.90 stored was not (and could not be) reliable.

This device thinks that that the array is functioning correctly with no failed devices, and that this device is a spare - presumably a 5th device? It doesn't know the names of the other devices (and if it thought it did, it could easily be wrong as names changed). What do the other devices think of the state of the array?

[~]>mdadm -Q --detail /dev/md3
/dev/md3:

        Version : 1.02

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Array Size : 5857390080 (5586.04 GiB 5997.97 GB)
Used Dev Size : 1952463360 (1862.01 GiB 1999.32 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jan 10 00:38:00 2011
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 68% complete

Name : sysresccd:1
UUID : fa8fb033:6312742f:0524501d:5aa24a28
Events : 34

Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 82 2 active sync /dev/sdf2
4 8 114 3 active sync /dev/sdh2

So just "check" turns the array into rebuild mode and one of the drives into a spare? That's unexpected.

Thanks,
Christian

#3 lore raw
Quoted 33 lines · On Mon, 10 Jan 2011 01:28:07 +0100 Christian Schmidt <[email protected]> wrote:
Quoted 5 lines

This device thinks that that the array is functioning correctly with no failed devices, and that this device is a spare - presumably a 5th device? It doesn't know the names of the other devices (and if it thought it did, it could easily be wrong as names changed). What do the other devices think of the state of the array?

[~]>mdadm -Q --detail /dev/md3
/dev/md3:

        Version : 1.02

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Array Size : 5857390080 (5586.04 GiB 5997.97 GB)
Used Dev Size : 1952463360 (1862.01 GiB 1999.32 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jan 10 00:38:00 2011
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 68% complete

Name : sysresccd:1
UUID : fa8fb033:6312742f:0524501d:5aa24a28
Events : 34

Quoted 8 lines

Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 82 2 active sync /dev/sdf2
4 8 114 3 active sync /dev/sdh2

So just "check" turns the array into rebuild mode and one of the drives into a spare? That's unexpected.

I very much doubt writing "check" is all that happened. Maybe seeing some kernel logs would help.
What does
cat /proc/mdstat

show (assuming the check/recovery/whatever hasn't finished yet). It should say "recovering" as I think the key word is copied into the 'State:' line above.

But writing "check" should not cause any drive to become a 'spare', and should not trigger a 'rebuild' - just a 'check'.

NeilBrown

#4 lore raw
Quoted 43 lines · NeilBrown
Quoted 39 lines · On Mon, 10 Jan 2011 01:28:07 +0100 Christian Schmidt <[email protected]> wrote:
Quoted 5 lines

This device thinks that that the array is functioning correctly with no failed devices, and that this device is a spare - presumably a 5th device? It doesn't know the names of the other devices (and if it thought it did, it could easily be wrong as names changed). What do the other devices think of the state of the array?

[~]>mdadm -Q --detail /dev/md3
/dev/md3:

        Version : 1.02

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Array Size : 5857390080 (5586.04 GiB 5997.97 GB)
Used Dev Size : 1952463360 (1862.01 GiB 1999.32 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jan 10 00:38:00 2011
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 68% complete

Name : sysresccd:1
UUID : fa8fb033:6312742f:0524501d:5aa24a28
Events : 34

Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 82 2 active sync /dev/sdf2
4 8 114 3 active sync /dev/sdh2

Quoted 5 lines
Quoted 2 lines

So just "check" turns the array into rebuild mode and one of the drives into a spare? That's unexpected.

I very much doubt writing "check" is all that happened. Maybe seeing some kernel logs would help.

Here they are:

[  235.503895] md: md3 stopped.
[  235.505428] md: bind<sdd2>
[  235.505557] md: bind<sdf2>
[  235.505673] md: bind<sdh2>
[  235.505804] md: bind<sdc2>
[  235.510288] md/raid:md3: device sdc2 operational as raid disk 0
[  235.510292] md/raid:md3: device sdh2 operational as raid disk 3
[  235.510294] md/raid:md3: device sdf2 operational as raid disk 2
[  235.510296] md/raid:md3: device sdd2 operational as raid disk 1
[  235.510569] md/raid:md3: allocated 4280kB
[  235.510604] md/raid:md3: raid level 5 active with 4 out of 4 devices,
algorithm 2
[  235.510607] RAID conf printout:
[  235.510609]  --- level:5 rd:4 wd:4
[  235.510611]  disk 0, o:1, dev:sdc2
[  235.510613]  disk 1, o:1, dev:sdd2
[  235.510614]  disk 2, o:1, dev:sdf2
[  235.510616]  disk 3, o:1, dev:sdh2
[  235.510652] md3: detected capacity change from 0 to 5997967441920
[  236.204947]  md3: unknown partition table
[ 1347.192343] md: data-check of RAID array md3
[ 1347.192346] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 1347.192347] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for data-check.
[ 1347.192352] md: using 128k window, over a total of 1952463360 blocks.

Actually I rebooted the machine after a kernel update, which turned out to change the drive names (I left an unrelated drive in a hotswap bay). Also, I had an erroneous /etc/mdadm.conf which was still referring to the old drive naming. When I realized this drive array wasn't started I completely renamed the config file and ran
mdadm -A --scan
after which the array was found. I have some issues opening crypto volumes on the LVM though and tried to figure out whether I forgot the key for one and never created the other, or something's wrong on the underlying layer, so I started a check.

Quoted 2 lines

What does
cat /proc/mdstat

It says:

md3 : active raid5 sdc2[0] sdh2[4] sdf2[2] sdd2[1]

      5857390080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
[UUUU]
      [=================>...]  check = 85.3% (1667391744/1952463360)

finish=57.5min speed=82511K/sec

Quoted 6 lines

show (assuming the check/recovery/whatever hasn't finished yet). It should say "recovering" as I think the key word is copied into the 'State:' line above.

But writing "check" should not cause any drive to become a 'spare', and should not trigger a 'rebuild' - just a 'check'.

Well... so what is this raid actually doing? mdstat says check, mdam -q --detail says recovering, and mdadm --examine on one of the drives says spare (while no spare are listed at any other point).

mdadm --examine:

/dev/sdc2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 801bb0ab:256d6f57:7e53e467:62094362

    Update Time : Mon Jan 10 01:43:39 2011
       Checksum : 5f661441 - correct
         Events : 35

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)

/dev/sdd2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d14e0126:4c8be6cd:418165b2:24bba827

    Update Time : Mon Jan 10 01:43:39 2011
       Checksum : 6015453f - correct
         Events : 35

         Layout : left-symmetric
     Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)

/dev/sdf2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)

Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3b8a4934:40a3270d:7e285e98:07aec354

Update Time : Mon Jan 10 01:43:39 2011
Checksum : c0b232bd - correct
Events : 35

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)

/dev/sdh2:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa8fb033:6312742f:0524501d:5aa24a28
           Name : sysresccd:1

Creation Time : Sat Jul 17 02:57:27 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 3904927887 (1862.01 GiB 1999.32 GB)

     Array Size : 11714780160 (5586.04 GiB 5997.97 GB)

Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 172eb49b:03e62242:614d7ed3:1fb25f65

Update Time : Mon Jan 10 01:43:39 2011
Checksum : a8d4425a - correct
Events : 35

Layout : left-symmetric
Chunk Size : 512K

Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)