Unable to re-add a disk after a reboot.

Hi,

I just finished converting a 3-disk raid5 to 4-disk raid6. After a
reboot to start clean, I noticed that one of the disk (the new one I just added) was missing in /proc/partitions. This was disk 4 in my /dev/md0. Assuming some cable issue, I powered off, wiggled the cables and restarted and the device was found by kernel. However, md0 shows device missing and array degraded

lata [rramesh] 280 > cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdb1[0] sdd1[3] sdc1[1]

          3906763776 blocks super 1.2 level 6, 512k chunk, algorithm 2

[4/3] [UUU_]

unused devices: <none>

However my attempt to --re-add does not work.

lata [rramesh] 277 > sudo mdadm /dev/md0 --verbose --re-add /dev/sde1
mdadm: --re-add for /dev/sde1 to /dev/md0 is not possible lata [rramesh] 278 > sudo mdadm -E /dev/sde1
/dev/sde1:

              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 730051d9:f4c58e0c:504fd1d9:798a84a4
               Name : lata:0  (local to host lata)
      Creation Time : Sun Oct  6 16:41:01 2013
         Raid Level : raid6
       Raid Devices : 4

     Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
         Array Size : 3906763776 (3725.78 GiB 4000.53 GB)
      Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 03898148:47c40cc2:f365082e:9f7f06cf

        Update Time : Thu Aug 14 08:53:16 2014
           Checksum : 346e9226 - correct
             Events : 1191488

             Layout : left-symmetric
         Chunk Size : 512K

       Device Role : Active device 3
       Array State : AAAA ('A' == active, '.' == missing)

lata [rramesh] 279 > fgrep UUID /etc/mdadm/mdadm.conf # ARRAY /dev/md/0 metadata=1.2 UUID=0e9f76b5:4a89171a:a930bccd:78749144 name=zym:0 ARRAY /dev/md0 metadata=1.2 spares=1 name=lata:0 UUID=730051d9:f4c58e0c:504fd1d9:798a84a4

I checked the SMART and it shows a lot of reallocated_sector_ct errors also. So, the disk is dying, but I am not able understand why mdadm would not add.

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE

      1 Raw_Read_Error_Rate     0x000b   091   091   016 Pre-fail 
   Always       -       53
      2 Throughput_Performance  0x0005   100   100   054 Pre-fail 
   Offline      -       0
      3 Spin_Up_Time            0x0007   135   135   024 Pre-fail 
   Always       -       426 (Average 425)
      4 Start_Stop_Count        0x0012   100   100   000 Old_age  
   Always       -       59
   *5 Reallocated_Sector_Ct   0x0033   001   001   005 Pre-fail 
   Always   FAILING_NOW 330*
      7 Seek_Error_Rate         0x000b   098   098   067 Pre-fail 
   Always       -       2
      8 Seek_Time_Performance   0x0005   100   100   020 Pre-fail 
   Offline      -       0
      9 Power_On_Hours          0x0012   100   100   000 Old_age  
   Always       -       3445
     10 Spin_Retry_Count        0x0013   100   100   060 Pre-fail 
   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000 Old_age  

Always - 59
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 548
193 Load_Cycle_Count 0x0012 100 100 000 Old_age
Always - 548
194 Temperature_Celsius 0x0002 153 153 000 Old_age
Always - 39 (Min/Max 21/43)
196 Reallocated_Event_Count 0x0032 001 001 000 Old_age
Always - 17604
197 Current_Pending_Sector 0x0022 001 001 000 Old_age
Always - 13256
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age
Always - 0

Any recommendations while I am waiting to get a replacement.

Ramesh

"re-add" only makes sense when you have a write-indent bitmap which you don't have.
So you need to "--add" which marks the device as a spare and then starts a complete rebuild.

It will "add". It just wont "re-add".

NeilBrown

Thanks. Did not know that. I thought it will add without rebuild. This means if a cable accidentally came off or if I booted without one disk by mistake, my arrays are dead. This looks too restrictive. I must be wrong in my conclusion. Please help me see this. Is there a add with assume clean?

Anyway, there is no point in rebuilding (or adding) it after it failed this miserably (has 17K reallocated event count, whatever that means) . I will let the array be degraded until I find a replacement.

I thought write-intent bitmap was not a good idea. May be I did not research enough. This brings me to the next (probably more important) question. How do I replace a old drive that has not died without having to rebuild? If I did a dd image xfer will it accept the replacement?

Ramesh

If you have a fairly recent kernel and mdadm, there is mdadm --replace.

https://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array

Mikael Abrahamsson    email: [email protected]

Thanks. If I may, I like to ask one related question. I have a disk that is already kicked out. Will adding a bitmap to degraded array help in -re_add the device? I doubt it, but I rather ask before trying as I am paranoid after the disk failure.

Ramesh

No, it wont.

Mikael Abrahamsson    email: [email protected]